#selfdoc.tokenizer

#selfdoc.tokenizer

Standalone Markdown block tokenizer.

Splits Markdown source into a flat list of typed block tokens. Each token carries 1-based start and end line numbers so the caller knows exactly which source lines produced it.

No imports from selfdoc -- this module is designed for reuse outside the project.

#CodeBlock

#Heading

#Table

#UnorderedList

#OrderedList

#Blockquote

#DefinitionList

#ThematicBreak

#BlankLine

#Directive

#Paragraph

#tokenize

def tokenize(content: str) -> list[Block]

Tokenize Markdown content into a list of block tokens.

Line numbers on every token are 1-based (first line of the file is 1). The tokens cover every line exactly once -- no gaps and no overlaps.