CLI testing¶

CliRunner runs a Click CLI in-process: the command object is imported, its callback executes inside the test process, and its streams are captured in memory. It is a drop-in replacement for click.testing.CliRunner, and what the runner and invoke fixtures hand to a test.

Tip

For the black-box counterpart, which spawns the command as a subprocess and never imports it, see test suites. Both render a run with the same execution trace.

Invoking a command¶

Take a CLI reporting the temperature of each city it is given:

from click_extra import argument, command, echo, option


@command
@option("--unit", default="celsius")
@argument("cities", nargs=-1)
def forecast(unit, cities):
    """Print the temperature of each city."""
    if not cities:
        echo("No city to report on.", err=True)
    for city in cities:
        echo(f"{city}: 21 {unit}")

Hand it to a runner. The command is the first positional argument, and everything after it is a command-line argument:

from click_extra.testing import CliRunner

runner = CliRunner()
result = runner.invoke(forecast, "--unit", "fahrenheit", "Oslo", "Lisbon")

assert result.exit_code == 0
assert result.stdout == "Oslo: 21 fahrenheit\nLisbon: 21 fahrenheit\n"

$ forecast --unit fahrenheit Oslo Lisbon
<output> stream:
  Oslo: 21 fahrenheit
  Lisbon: 21 fahrenheit

<stdout> stream:
  Oslo: 21 fahrenheit
  Lisbon: 21 fahrenheit

<exit_code>: 0

Passing the arguments as a list under an args keyword, the way vanilla Click expects them, works too and can be mixed with the positional form.

Every invocation prints the execution trace reproduced above, pass or fail. It is what a failing test shows in its captured output, so a broken assertion comes with the session that produced it instead of a bare exit code.

Composing arguments¶

Positional arguments are flattened, so a nested structure of lists and tuples is spelled out as-is. None values are dropped, and every remaining item is cast to a string, which lets a Path travel unconverted:

from pathlib import Path

result = runner.invoke(forecast, ["Oslo", None, ["Lisbon", (Path("Kyoto"),)]])

assert result.exit_code == 0
assert result.stdout == "Oslo: 21 celsius\nLisbon: 21 celsius\nKyoto: 21 celsius\n"

$ forecast Oslo Lisbon Kyoto
<output> stream:
  Oslo: 21 celsius
  Lisbon: 21 celsius
  Kyoto: 21 celsius

<stdout> stream:
  Oslo: 21 celsius
  Lisbon: 21 celsius
  Kyoto: 21 celsius

<exit_code>: 0

That is args_cleanup() doing the work, and it is what makes a parametrized test readable: each case contributes its own fragment of the command line, and an optional fragment collapses to None instead of forcing the test to assemble the list itself.

test_forecast.py¶

import pytest


@pytest.mark.parametrize("unit_flag", (None, ("--unit", "fahrenheit")))
@pytest.mark.parametrize("cities", (("Oslo",), ("Oslo", "Lisbon")))
def test_forecast(invoke, unit_flag, cities):
    result = invoke(forecast, unit_flag, cities)
    assert result.exit_code == 0

Reading the result¶

Result carries the exit code, the captured streams and whatever exception escaped the callback:

Attribute	Content
`exit_code`	Process exit status.
`stdout`	Standard output alone.
`stderr`	Standard error alone.
`output`	Both streams, interleaved in the order they were written.
`exception`	The exception that escaped the callback, if any.
`formatted_exception`	Its full traceback, as a string, or `None`.

The two streams stay addressable separately, which is how a test pins a diagnostic to the stream it belongs on:

result = runner.invoke(forecast)

assert result.exit_code == 0
assert result.stdout == ""
assert result.stderr == "No city to report on.\n"

$ forecast
<output> stream:
  No city to report on.

<stderr> stream:
  No city to report on.

<exit_code>: 0

When the callback raises, the runner catches the exception, sets a non-zero exit_code, and prints the traceback below the trace. formatted_exception keeps that traceback available, and repr(result) embeds it, so an assert result.exit_code == 0 that fails reports where the CLI died rather than which exception class it was. Pass catch_exceptions=False to let the exception propagate into the test instead.

Colors¶

ANSI codes survive the trip only when asked for. color accepts one more value than Click’s:

`color=`	Captured streams	`Context.color`
`None`	Stripped	`None`
`False`	Stripped	`None`
`True`	Kept	`None`
`"forced"`	Kept	`True`

color=True keeps the codes in the captured output, but the invoked CLI still sees an uncolored context and takes its uncolored branch. color="forced" covers both: it keeps the codes and initializes Context.color to True, which vanilla Click cannot express because the two meanings collide on one parameter name (pallets/click#2110). Click Extra routes the second one through a patched main() call, so any other Context keyword named like an invoke() parameter gets through as well.

import click

from click_extra import style


@click.command
@click.pass_context
def report(ctx):
    click.echo(style("Sunny", fg="yellow") + " in Lisbon.")
    click.echo(f"Context.color is {ctx.color!r}")

result = runner.invoke(report, color="forced")

assert "\x1b[33mSunny\x1b[0m in Lisbon.\n" in result.stdout
assert "Context.color is True\n" in result.stdout

$ report
<output> stream:
  Sunny in Lisbon.
  Context.color is True

<stdout> stream:
  Sunny in Lisbon.
  Context.color is True

<exit_code>: 0

Setting color=False goes one step further than Click’s stripping and scrubs the result bytes, so a CLI writing raw escape sequences past Click’s own machinery still yields clean text. To keep the codes on every invocation of a suite without touching each call, set the force_color class attribute on CliRunner: it pins every run to the color=True row above.

Note

The command above is a plain Click one on purpose: nothing but the runner decides its colors. A Click Extra command owns that decision itself, through its --color/--no-color options and the NO_COLOR and FORCE_COLOR environment variables, which have the last word over whatever the runner was told.

Rendering a run¶

render_cli_run() produces the trace on its own, from either an in-process click.testing.Result or a subprocess.CompletedProcess. Both are normalized into a StreamView first, so a black-box run and an in-process one read identically:

import subprocess
import sys

from click_extra.testing import render_cli_run

process = subprocess.run(
    (sys.executable, "-c", "print('Lisbon: 21 celsius')"),
    capture_output=True,
    text=True,
    encoding="utf-8",
)

print(render_cli_run(("forecast", "Lisbon"), process, env={"FORECAST_UNIT": "celsius"}))

$ FORECAST_UNIT=celsius forecast Lisbon
<stdout> stream:
  Lisbon: 21 celsius

<exit_code>: 0

The first line is the command as the user would have typed it, environment assignments included. Each captured stream then gets its own labelled, indented block, and the exit code closes the trace. A stream that captured nothing is left out entirely: a subprocess run with stderr=STDOUT reports a single interleaved <output> stream, while separate streams give the <stdout> and <stderr> pair.

Matching output against a regex¶

Comparing a wall of terminal output to an expected string reports the whole wall on failure, and leaves the reader to find the offending character. regex_fullmatch_line_by_line() matches the pattern one line at a time and reports the first line that disagrees:

from click_extra.testing import RegexLineMismatch, regex_fullmatch_line_by_line

try:
    regex_fullmatch_line_by_line(
        r"Lisbon: \d+ celsius\nOslo: \d+ fahrenheit\n",
        "Lisbon: 21 celsius\nOslo: 9 celsius\n",
    )
except RegexLineMismatch as ex:
    print(ex)

Line #2 does not match.
Regex : 'Oslo: \\d+ fahrenheit\\n'
Output: 'Oslo: 9 celsius\n'

The pattern is split on its \n tokens, so it is written as one raw string mirroring the expected output, not as a list of per-line patterns. A pattern matching in full short-circuits the loop, and only a mismatch pays for the line-by-line pass.

The reported pattern goes through unescape_regex(), the inverse of re.escape(), which strips the backslashes an escaped literal is littered with so the two sides of the report line up visually:

from click_extra.testing import unescape_regex

assert unescape_regex(r"Usage: forecast \[OPTIONS\] \[CITIES\]\.\.\.") == (
    "Usage: forecast [OPTIONS] [CITIES]..."
)
print(unescape_regex(r"Usage: forecast \[OPTIONS\] \[CITIES\]\.\.\."))

Usage: forecast [OPTIONS] [CITIES]...

The assert_output_regex fixture wraps that comparison into an assertion carrying a character-level diff, and the ready-made patterns shipped with it cover the help screen and debug logs Click Extra adds to every CLI.

`click_extra.testing` API¶

        classDiagram
  AssertionError <|-- RegexLineMismatch
  CliRunner <|-- CliRunner
  Result <|-- Result

CLI testing and simulation of their execution.

click_extra.testing.OUTPUT_LABEL = '<output>': Label for the merged stream, where stdout and stderr are interleaved.

click_extra.testing.STDOUT_LABEL = '<stdout>': Label for the standard output stream.

click_extra.testing.STDERR_LABEL = '<stderr>': Label for the standard error stream.

click_extra.testing.EXIT_CODE_LABEL = '<exit_code>': Label for the process exit code.

click_extra.testing.STREAM_FIELDS = {'output_': ('<output>', 'output'), 'stderr_': ('<stderr>', 'stderr'), 'stdout_': ('<stdout>', 'stdout')}

Maps a test-case field prefix to its stream label and StreamView attribute.

output_* directives target the merged stream; stdout_* and stderr_* target the separate streams. Both render_cli_run() and click_extra.test_suite.CLITestCase.run_cli_test() read this single table so the rendered trace and the assertion loop agree on labels and stream selection.

class click_extra.testing.StreamView(stdout='', stderr='', output='', exit_code=None)[source]

Bases: object

Normalized view of a CLI run’s captured streams and exit code.

Both runners produce one of these so the renderer and the assertion loop read a single shape, regardless of whether the run was driven in-process (Click’s click.testing.Result) or as a black-box subprocess (subprocess.CompletedProcess).

A run captures either the merged stream (output) or the separate stdout and stderr streams, never both: the unused fields stay empty.

stdout: str = '': Captured standard output, or empty when the merged stream was captured.

stderr: str = '': Captured standard error, or empty when the merged stream was captured.

output: str = '': Captured merged stream (stdout and stderr interleaved), or empty when the separate streams were captured.

exit_code: int | None = None: Process exit code, or None when unavailable.

classmethod from_result(result)[source]

Build a view from an in-process click.testing.Result.

Click always exposes stdout, stderr and the interleaved output together, so all three are carried over verbatim.

Return type:: StreamView

classmethod from_completed_process(result)[source]

Build a view from a black-box subprocess.CompletedProcess.

A subprocess run with stderr merged into stdout (stderr=STDOUT) reports result.stderr as None: that case is rendered as the interleaved output stream. Otherwise the two streams are kept separate.

Return type:: StreamView

click_extra.testing.render_cli_run(args, result, env=None)[source]

Generates the full simulation of CLI execution, including output.

Mostly used to print debug traces to user or in test results.

Return type:: str

click_extra.testing.INVOKE_ARGS = {'args', 'catch_exceptions', 'cli', 'color', 'env', 'input', 'self'}

Parameter IDs of click.testing.CliRunner.invoke().

We need to collect them to help us identify which extra parameters passed to invoke() collides with its original signature.

Warning

This has been reported upstream to Click project but has been rejected and not considered an issue worth fixing.

class click_extra.testing.Result(runner, stdout_bytes, stderr_bytes, output_bytes, return_value, exit_code, exception, exc_info=None)[source]

Bases: Result

A Result subclass with automatic traceback formatting.

Enhances __repr__ so that pytest assertion failures show the full traceback instead of just the exception type.

property formatted_exception: str | None[source]: Full formatted traceback, or None if no exception occurred.

class click_extra.testing.CliRunner(charset='utf-8', env=None, echo_stdin=False, catch_exceptions=True, capture='sys')[source]

Bases: CliRunner

Augment click.testing.CliRunner with extra features and bug fixes.

force_color: bool = False: Global class attribute to override the color parameter in invoke.

invoke(cli, *args, input=None, env=None, catch_exceptions=True, color=None, **extra)[source]

Same as click.testing.CliRunner.invoke() with extra features.

The first positional parameter is the CLI to invoke. The remaining positional parameters of the function are the CLI arguments. All other parameters are required to be named.
The CLI arguments can be nested iterables of arbitrary depth. This is useful for argument composition of test cases with @pytest.mark.parametrize.
Allow forcing of the color property at the class-level via force_color attribute.
Adds a special case in the form of color="forced" parameter, which allows colored output to be kept, while forcing the initialization of Context.color = True. This is not allowed in current implementation of click.testing.CliRunner.invoke() because of colliding parameters.
Strips all ANSI codes from results if color was explicitly set to False.
Always prints a simulation of the CLI execution as the user would see it in its terminal. Including colors.
Pretty-prints a formatted exception traceback if the command fails.

Parameters:

cli (Command) – CLI to invoke.
args (str | Path | None | Iterable[str | Path | None | Iterable[Iterable[str | Path | None | Iterable[TNestedArgs]]]]) – can be nested iterables composed of str, pathlib.Path objects and None values. The nested structure will be flattened and None values will be filtered out. Then all elements will be cast to str. See args_cleanup() for details.
input (str | bytes | IO | None) – same as click.testing.CliRunner.invoke().
env (Mapping[str, str | None] | None) – same as click.testing.CliRunner.invoke().
catch_exceptions (bool) – same as click.testing.CliRunner.invoke().
color (bool | Literal['forced'] | None) – If a boolean, the parameter will be passed as-is to click.testing.CliRunner.isolation(). If "forced", the parameter will be passed as True to click.testing.CliRunner.isolation() and an extra color=True parameter will be passed to the invoked CLI.
extra (Any) – same as click.testing.CliRunner.invoke(), but colliding parameters are allowed and properly passed on to the invoked CLI.

Return type:

Result

click_extra.testing.unescape_regex(text)[source]

De-obfuscate a regex for better readability.

This is like the reverse of re.escape().

Return type:: str

exception click_extra.testing.RegexLineMismatch(regex_line, content_line, line_number)[source]

Bases: AssertionError

Raised when a regex line does not match the corresponding content line.

click_extra.testing.REGEX_NEWLINE = '\\n': Newline token used to split a multi-line regex pattern for line-by-line matching.

click_extra.testing.regex_fullmatch_line_by_line(regex, content)[source]

Check that the content matches the given regex.

If the regex does not fully match the content, raise an AssertionError, with a message showing the first mismatching line.

This is useful when comparing large walls of text, such as CLI output.

Return type:: None

CLI testing¶

Invoking a command¶

Composing arguments¶

Reading the result¶

Colors¶

Rendering a run¶

Matching output against a regex¶

click_extra.testing API¶

`click_extra.testing` API¶