click_extra.markdown package

Markdown text tooling, free of Sphinx and docutils.

click_extra.markdown.blocks walks the fences of a Markdown source and refreshes the self-updating blocks it holds. click_extra.markdown.myst converts reST to MyST, in Python docstrings and in sphinx-apidoc pages.

Submodules

click_extra.markdown.blocks module

Offline self-updating block toolkit for Markdown sources.

Fence-aware Markdown scanning, the <!-- name … --> / <!-- name-end --> marker grammar, and the walk-rewrite-write loop behind the click-extra refresh-directives command.

Every scanner of the package that must not read inside a fence walks the document through walk_fences(), or rebuilds it through rewrite_fences(): a fence is one opaque unit, whose body is never read as live markup.

The toolkit backs the :matrix: directive ({mod}``click_extra.sphinx.matrix``), the python:render :mirror: flag and the click:run :screenshot: mirror, but it depends only on re and pathlib: nothing here imports Sphinx or docutils. Living outside click_extra.sphinx lets a release pipeline or a standalone documentation script reuse replace_region and update_blocks without pulling in the sphinx extra.

class click_extra.markdown.blocks.FenceSpan(start, close, end)

Bases: NamedTuple

A top-level fence in a Markdown source, as line indices.

Create new instance of FenceSpan(start, close, end)

start: int

Index of the opening fence line.

close: int | None

Index of the closing fence line, or None when unterminated.

end: int

Index just past the fence: past its closing line, or the line count of the source when the fence never closes.

property body: slice

The lines between the opening fence line and the closing one, which run to the end of the source when the fence never closes.

click_extra.markdown.blocks.fence_spans(lines, *, chars='`')

Map each top-level fence’s opening line index to its span.

Fences are consumed as opaque units: a fence line inside an outer fence (a documented example wrapped in a longer code-block fence) never starts a span of its own. A close requires a bare run of the same character, at least as long as the opener, at the same indentation or deeper. An unterminated fence spans to the end of the file with close=None.

Parameters:

chars (str) – The characters a fence is made of: backticks by default, : for colon fences alone, or both, see _fence_open_re.

Return type:

dict[int, FenceSpan]

click_extra.markdown.blocks.walk_fences(lines, spans=None, start=0)

Walk lines from start, a line at a time, stepping over each fence whole.

Yields (index, None) for each line outside the top-level fences, and (index, span) for each fence, from its opening line. The walk then resumes at span.end: the body of a fence is never visited, and an unterminated fence ends the walk.

Parameters:

• lines (Sequence[str]) – the Markdown source, split in lines.

• spans (Mapping[int, FenceSpan] | None) – the fences of lines, from fence_spans(). Computed when not given.

• start (int) – the index to start from, outside any fence body.

Return type:

Iterator[tuple[int, FenceSpan | None]]

click_extra.markdown.blocks.rewrite_fences(lines, visit, spans=None)

Rebuild lines along walk_fences(), letting visit rewrite them.

visit(index, span) sees each step of the walk. It answers None to keep that line, or that whole fence, verbatim: an unterminated fence keeps the rest of the source. Or it answers (replacement, resume), to emit the replacement lines in place of the source lines up to resume, which must lie past index and outside any fence body.

Parameters:

• lines (Sequence[str]) – the Markdown source, split in lines.

• visit (Callable[[int, FenceSpan | None], tuple[list[str], int] | None]) – decides what replaces each line or fence.

• spans (Mapping[int, FenceSpan] | None) – the fences of lines, from fence_spans(). Computed when not given.

Return type:

list[str]

Returns:

the rebuilt lines.

click_extra.markdown.blocks.marker_res(name)

Build the (open, close) regexes of a <!-- name --> region.

The grammar is shared by every self-updating marker region: the opening comment is <!-- name [args] --> (args optional, whitespace separated), the closing comment is <!-- name-end -->. Both capture their leading indentation as indent.

Cached: a region name maps to one pair of compiled patterns, however many refreshers read it.

Return type:

tuple[Pattern[str], Pattern[str]]

click_extra.markdown.blocks.replace_region(text, name, content, *, pad=True)

Return text with the body of a <!-- name --> region swapped for content.

Finds the <!-- name [args] --> opening and <!-- name-end --> closing markers (the grammar of marker_res()) and replaces everything between them with content. The markers themselves are preserved, so the region round-trips: a second call with the same content is a no-op. A marker quoted inside a code fence, like in a page documenting the grammar, is never taken for a live one.

When either marker is missing the text is returned unchanged, so the call is safe to fan out over every file of an update_blocks() rewrite even when only some carry the region. This is the generic counterpart to the fence-driven refreshers of click_extra.sphinx.matrix and the python:render :mirror: flag: use it when the content is produced outside the document (a registry dump, an external generator) rather than by an inline directive.

Parameters:

pad (bool) – With the default True, content is padded by one blank line on each side, the usual layout for a Markdown block. Set it to False to keep the blank line after the opening marker (Markdown needs it to start a fresh block) but drop the one before the closing marker, so the region ends flush against it. That flush layout is what mdformat-footnote requires: it strips an HTML comment sitting on its own line right after a footnote definition (executablebooks/mdformat-footnote#11), so a region wrapping footnotes must place its closing marker on the line immediately below the last body line. An empty content collapses to a single blank line between the markers whatever the padding.

Return type:

str

click_extra.markdown.blocks.region_markers(name)

The (open, close) marker lines of a <!-- name --> region.

The writing-side twin of marker_res(): <!-- name --> opens the region and <!-- name-end --> closes it, and every refresher writes both through here so they cannot drift from what that reader matches.

Return type:

tuple[str, str]

click_extra.markdown.blocks.split_options(lines)

Split a fence body into its leading :key: value options and its content.

Stops at the first line that is not an option, so a body whose Python happens to start with a colon is never mistaken for one. A single blank line separating the options from the content goes with the options: the content comes back exactly as the directive would receive it.

Parameters:

lines (Sequence[str]) – the lines between a fence’s opening and closing markers.

Return type:

tuple[dict[str, str], list[str]]

Returns:

the options, in declaration order, and the content lines.

click_extra.markdown.blocks.skip_region(lines, index, name, spans=None)

Return the index just past the <!-- name --> region opening at index.

Skips leading blank lines, then a whole region if one opens there. Returns index unchanged when no region follows, so the content below a first-time block is never consumed as if it were a stale region. The search for the closing marker steps over fences: a <!-- name-end --> quoted in a code example the region holds does not end it.

Parameters:

spans (Mapping[int, FenceSpan] | None) – the fences of lines, from fence_spans(). Computed when not given.

Return type:

int

click_extra.markdown.blocks.rewrite_fenced_regions(text, name, fence_open, content)

Refresh the <!-- name --> region kept below every matching fence of text.

Walks the document through rewrite_fences(), so an example nested inside a longer code-block fence is copied verbatim, never read as a live block. A top-level fence whose opening line matches fence_open hands its inner lines, options included, to content, which answers the lines the region holds, or None to leave the fence with no region at all. The region is inserted directly below the fence on first sight and replaced afterwards, blank-line padded so mdformat never ping-pongs on the markers. Idempotent: an unchanged block round-trips to the same text.

The python:render :mirror: and the click:run :screenshot: regions are both maintained through here, each with its own content.

Parameters:

• text (str) – the Markdown source.

• name (str) – the region’s marker name, see region_markers().

• fence_open (Pattern[str]) – matches the opening line of a fence carrying a region.

• content (Callable[[list[str]], list[str] | None]) – computes the region’s lines from the fence’s inner lines.

Return type:

str

Returns:

the rewritten source.

click_extra.markdown.blocks.iter_markdown_files(paths)

Yield the Markdown sources under paths (files as-is, dirs recursed).

Return type:

Iterator[Path]

click_extra.markdown.blocks.rewrite_file(path, rewrite, *, check=False)

Apply rewrite to the text of path, and write it back when it changed.

The one write-if-changed step of every source rewriter: the block refreshers through update_blocks(), and the reST-to-MyST docstring converter.

Parameters:

check (bool) – report the change without writing it.

Return type:

bool

Returns:

whether the text changed, or would have under check.

click_extra.markdown.blocks.update_blocks(paths, rewrite, *, check=False)

Rewrite self-updating blocks in the Markdown sources under paths.

Walks paths (files, or directories recursed for *.md), applies rewrite(text, path) to each, and writes the file back when its content changed. In check mode nothing is written; the return value still lists the files that would change, so a caller can exit non-zero to flag stale documentation in CI.

Return type:

list[Path]

Returns:

the files whose blocks were (or, under check, would be) updated.

click_extra.markdown.myst module

Convert reST to MyST: the docstrings of Python sources, and sphinx-apidoc pages.

convert_source() transforms the reST markup of docstrings and comments to MyST markdown. The companion Sphinx extension click_extra.sphinx.myst_docstrings converts the MyST back to reST at build time, so sphinx.ext.autodoc still works.

Conversions applied (in order):

1. Cross-references: a reST role, its name between colons, becomes a MyST role, its name in braces.

2. Named links: a reST hyperlink embedding its URL becomes a Markdown link.

3. Inline code: double-backtick literals become single-backtick code spans.

4. #: comment blocks: strip prefix, convert directives, re-wrap.

5. Directives: a reST directive and its indented body become a backtick directive fence.

Only docstrings (bare string-expression statements, located with ast) and comments (located with tokenize) are transformed. String literals, f-strings, and every other piece of runtime code pass through byte-for-byte: a regex pattern or an error message that happens to contain reST markup is not documentation and must not be rewritten.

Safe to re-run: already-converted MyST syntax does not match the reST patterns, so the script is idempotent.

Note

f-string exclusion: Cross-reference and inline-code regexes exclude targets containing { so that interpolation-style placeholders (like {self.id} in a documented format template) are untouched.

Note

Nested fences stay as reST: A directive whose body already contains a triple-backtick fence is left in reST. Converting it would nest two same-level fences, which markdown cannot delimit, and the build-time extension passes reST through unchanged anyway.

Note

Nested directives stay as reST: A .. code-block:: inside a converted backtick-fenced warning directive is emitted as-is. The hook handles this correctly because it re-indents the body when converting back to reST.

Note

Link labels lose backticks: reST has no nested markup, so the hook drops the backticks of a code span in a link label before emitting the reST link.

convert_apidoc_rst_to_myst() handles only the narrow reST subset that sphinx-apidoc generates: section headings (title + underline), automodule directives with indented options, and structural headers like Submodules.

Note

Autodoc directives cannot be used as native MyST directives because they perform internal rST nested parsing that requires an rST parser context only {eval-rst} provides. See MyST-Parser #587.

The patterns are compiled on first use, through the cache of re.

click_extra.markdown.myst.convert_xrefs(text)

Convert reST cross-references to MyST syntax.

Return type:

str

click_extra.markdown.myst.convert_links(text)

Convert reST named hyperlinks to markdown links.

Return type:

str

click_extra.markdown.myst.convert_inline_code(text)

Convert reST double-backtick literals to single-backtick.

Return type:

str

click_extra.markdown.myst.convert_directives(text)

Convert reST directives to MyST backtick fences in a single pass.

Body lines are collected by indentation (deeper than the .. line) and dedented to the fence level. Trailing blank lines between consecutive directives are preserved as a single separator.

Nested reST directives (like .. code-block:: inside .. warning::) are emitted as-is in the fence body. The hook re-indents them during the reST round-trip.

A directive whose body contains a triple-backtick fence is left in reST entirely: converting it would produce two same-level fences that markdown cannot tell apart.

Symmetrically, existing fences are opaque: a reST directive inside a fence body is exactly what a previous conversion of nested directives produces (the inner one stays reST by design), so re-scanning it would break idempotency and nest same-level fences.

Todo

Convert a directive whose body holds a fence to a longer outer fence, instead of leaving it in reST: the build-time extension click_extra.sphinx.myst_docstrings reads fences of any length.

Return type:

str

click_extra.markdown.myst.convert_comment_blocks(text)

Convert #: Sphinx comment docstrings.

Consecutive #: lines are collected, the prefix is stripped, all conversions are applied to the extracted content, and the prefix is re-added.

Return type:

str

click_extra.markdown.myst.convert_source(source)

Convert reST markup to MyST in a Python module’s docstrings and comments.

Docstrings get the full pipeline, in an order that matters: inline constructs (cross-references, links, inline code) run before directives so that directive bodies are already converted when they are dedented into fences. Comments get the inline conversions only, except consecutive full-line #: comments, whose directives are also converted through convert_comment_blocks(). Everything outside docstrings and comments passes through byte-for-byte.

Return type:

str

click_extra.markdown.myst.convert_file(filepath)

Apply all conversions to a single Python file.

Parameters:

filepath (Path) – The Python file to convert in place.

Return type:

bool

Returns:

True if the file was modified.

click_extra.markdown.myst.convert_directory(directory)

Convert all Python files in a directory from reST to MyST docstrings.

Parameters:

directory (Path) – Directory to process recursively.

Return type:

list[Path]

Returns:

List of files that were modified.

click_extra.markdown.myst.convert_apidoc_rst_to_myst(content)

Convert sphinx-apidoc RST to MyST markdown with {eval-rst} blocks.

Parameters:

content (str) – RST content produced by sphinx-apidoc.

Return type:

str

Returns:

Equivalent MyST markdown.

click_extra.markdown.myst.convert_rst_files_in_directory(directory)

Convert sphinx-apidoc RST files to MyST markdown in the given directory.

For each .rst file containing .. automodule:: directives:

• If a .md file with the same stem exists, delete the .rst (the existing markdown takes precedence).

• Otherwise, convert the RST content to MyST and write a .md file, then delete the .rst.

Parameters:

directory (Path) – Directory to scan for .rst files.

Return type:

list[Path]

Returns:

List of newly created .md file paths.