CLI testing

Todo

Write example and tutorial.

click_extra.testing API

        classDiagram
  AssertionError <|-- RegexLineMismatch
  CliRunner <|-- CliRunner
  Result <|-- Result
    

CLI testing and simulation of their execution.

click_extra.testing.TNestedArgs

Type for arbitrary nested CLI arguments.

Arguments can be str, pathlib.Path objects or None values.

alias of Iterable[str | Path | None | Iterable[TNestedArgs]]

click_extra.testing.PROMPT = '$ '

Prompt used to simulate the CLI execution.

Hint

Use ASCII characters to avoid issues with Windows terminals.

click_extra.testing.INDENT = '  '

Constants for rendering of CLI execution.

click_extra.testing.OUTPUT_LABEL = '<output>'

Label for the merged stream, where stdout and stderr are interleaved.

click_extra.testing.STDOUT_LABEL = '<stdout>'

Label for the standard output stream.

click_extra.testing.STDERR_LABEL = '<stderr>'

Label for the standard error stream.

click_extra.testing.EXIT_CODE_LABEL = '<exit_code>'

Label for the process exit code.

click_extra.testing.STREAM_FIELDS = {'output_': ('<output>', 'output'), 'stderr_': ('<stderr>', 'stderr'), 'stdout_': ('<stdout>', 'stdout')}

Maps a test-case field prefix to its stream label and StreamView attribute.

output_* directives target the merged stream; stdout_* and stderr_* target the separate streams. Both render_cli_run() and click_extra.test_plan.CLITestCase.run_cli_test() read this single table so the rendered trace and the assertion loop agree on labels and stream selection.

class click_extra.testing.StreamView(stdout='', stderr='', output='', exit_code=None)

Bases: object

Normalized view of a CLI run’s captured streams and exit code.

Both runners produce one of these so the renderer and the assertion loop read a single shape, regardless of whether the run was driven in-process (Click’s click.testing.Result) or as a black-box subprocess (subprocess.CompletedProcess).

A run captures either the merged stream (output) or the separate stdout and stderr streams, never both: the unused fields stay empty.

stdout: str = ''

Captured standard output, or empty when the merged stream was captured.

stderr: str = ''

Captured standard error, or empty when the merged stream was captured.

output: str = ''

Captured merged stream (stdout and stderr interleaved), or empty when the separate streams were captured.

exit_code: int | None = None

Process exit code, or None when unavailable.

classmethod from_result(result)

Build a view from an in-process click.testing.Result.

Click always exposes stdout, stderr and the interleaved output together, so all three are carried over verbatim.

Return type:

StreamView

classmethod from_completed_process(result)

Build a view from a black-box subprocess.CompletedProcess.

A subprocess run with stderr merged into stdout (stderr=STDOUT) reports result.stderr as None: that case is rendered as the interleaved output stream. Otherwise the two streams are kept separate.

Return type:

StreamView

click_extra.testing.args_cleanup(*args)

Flatten recursive iterables, remove all None, and cast each element to strings.

Helps serialize pathlib.Path and other objects.

It also allows for nested iterables and None values as CLI arguments for convenience. We just need to flatten and filters them out.

Return type:

tuple[str, ...]

click_extra.testing.render_cli_run(args, result, env=None)

Generates the full simulation of CLI execution, including output.

Mostly used to print debug traces to user or in test results.

Return type:

str

click_extra.testing.INVOKE_ARGS = {'args', 'catch_exceptions', 'cli', 'color', 'env', 'input', 'self'}

Parameter IDs of click.testing.CliRunner.invoke().

We need to collect them to help us identify which extra parameters passed to invoke() collides with its original signature.

Warning

This has been reported upstream to Click project but has been rejected and not considered an issue worth fixing.

class click_extra.testing.Result(runner, stdout_bytes, stderr_bytes, output_bytes, return_value, exit_code, exception, exc_info=None)

Bases: Result

A Result subclass with automatic traceback formatting.

Enhances __repr__ so that pytest assertion failures show the full traceback instead of just the exception type.

property formatted_exception: str | None

Full formatted traceback, or None if no exception occurred.

class click_extra.testing.CliRunner(charset='utf-8', env=None, echo_stdin=False, catch_exceptions=True, capture='sys')

Bases: CliRunner

Augment click.testing.CliRunner with extra features and bug fixes.

force_color: bool = False

Global class attribute to override the color parameter in invoke.

invoke(cli, *args, input=None, env=None, catch_exceptions=True, color=None, **extra)

Same as click.testing.CliRunner.invoke() with extra features.

• The first positional parameter is the CLI to invoke. The remaining positional parameters of the function are the CLI arguments. All other parameters are required to be named.

• The CLI arguments can be nested iterables of arbitrary depth. This is useful for argument composition of test cases with @pytest.mark.parametrize.

• Allow forcing of the color property at the class-level via force_color attribute.

• Adds a special case in the form of color="forced" parameter, which allows colored output to be kept, while forcing the initialization of Context.color = True. This is not allowed in current implementation of click.testing.CliRunner.invoke() because of colliding parameters.

• Strips all ANSI codes from results if color was explicirely set to False.

• Always prints a simulation of the CLI execution as the user would see it in its terminal. Including colors.

• Pretty-prints a formatted exception traceback if the command fails.

Parameters:

• cli (Command) – CLI to invoke.

• args (str | Path | None | Iterable[str | Path | None | Iterable[Iterable[str | Path | None | Iterable[TNestedArgs]]]]) – can be nested iterables composed of str, pathlib.Path objects and None values. The nested structure will be flattened and None values will be filtered out. Then all elements will be casted to str. See args_cleanup() for details.

• input (str | bytes | IO | None) – same as click.testing.CliRunner.invoke().

• env (Mapping[str, str | None] | None) – same as click.testing.CliRunner.invoke().

• catch_exceptions (bool) – same as click.testing.CliRunner.invoke().

• color (bool | Literal['forced'] | None) – If a boolean, the parameter will be passed as-is to click.testing.CliRunner.isolation(). If "forced", the parameter will be passed as True to click.testing.CliRunner.isolation() and an extra color=True parameter will be passed to the invoked CLI.

• extra (Any) – same as click.testing.CliRunner.invoke(), but colliding parameters are allowed and properly passed on to the invoked CLI.

Return type:

Result

click_extra.testing.unescape_regex(text)

De-obfuscate a regex for better readability.

This is like the reverse of re.escape().

Return type:

str

exception click_extra.testing.RegexLineMismatch(regex_line, content_line, line_number)

Bases: AssertionError

Raised when a regex line does not match the corresponding content line.

click_extra.testing.REGEX_NEWLINE = '\\n'

Newline token used to split a multi-line regex pattern for line-by-line matching.

click_extra.testing.regex_fullmatch_line_by_line(regex, content)

Check that the content matches the given regex.

If the regex does not fully match the content, raise an AssertionError, with a message showing the first mismatching line.

This is useful when comparing large walls of text, such as CLI output.

Return type:

None