Test plans¶

A test plan is a declarative list of CLI invocations and the results each one should produce. Click Extra runs the plan against any command or binary as separate subprocesses, checking exit codes and output. It is the black-box, subprocess-level complement to CliRunner, which drives a CLI in-process: a test plan never imports the target, so it works just as well against a compiled binary, a shell command, or a CLI written in another language.

Important

Parsing a plan from YAML needs the optional pyyaml dependency. Install it with the yaml extra:

{code-block} shell-session $ pip install click-extra[yaml]

The engine itself (building CLITestCase objects and running them) has no such requirement: only parse_test_plan does.

Writing a plan¶

A plan is a YAML list. Each entry is one case: the parameters to append to the command, plus the expectations to check. A case with no expectation only asserts that the command ran.

- cli_parameters: --version
  exit_code: 0

- cli_parameters: forecast --city paris
  stdout_contains: Sunny
  timeout: 5

- cli_parameters: --help
  stdout_regex_matches:
    - Usage:.+
  skip_platforms:
    - windows

The directives map one-to-one onto CLITestCase fields:

cli_parameters: arguments appended to the command (a string is split, a list is used as-is).
exit_code: the expected process exit code.
stdout_contains / stderr_contains: substrings that must appear.
stdout_regex_matches / stderr_regex_matches: regexes that must each match somewhere.
stdout_regex_fullmatch / stderr_regex_fullmatch: a regex that must fully match, line by line.
strip_ansi: strip ANSI escapes before matching.
timeout: seconds before the case fails as a timeout.
skip_platforms / only_platforms: extra_platforms identifiers (linux, macos, windows, group IDs) controlling where the case runs.

Running from the command line¶

The click-extra test-plan subcommand runs a plan against a target. Point it at a command on the PATH, a command line, or a path to a binary:

$ click-extra test-plan --command weather --plan-file plan.yaml
Running 3 test cases across 7 workers (os.cpu_count()=8).
Test plan results - Total: 3, Skipped: 0, Failed: 0

Cases run in parallel by default, one fewer than the available logical CPUs (see --jobs). Pass --jobs max to use every core, or --jobs 1 for sequential execution, which lets --exit-on-error stop on the first failure. On an interactive terminal a spinner reports progress; it is silent in pipes and CI logs, and --no-progress turns it off.

Configuring the plan¶

Rather than passing --plan-file every time, a project can declare its plan once under [tool.click-extra.test-plan], and click-extra test-plan picks it up when no plan is given on the command line:

[tool.click-extra.test-plan]
file = "tests/cli-test-plan.yaml"  # default; a path to the YAML plan
# inline = "- cli_parameters: --version"  # or embed the plan directly
# timeout = 30  # default per-case timeout in seconds

The resolution precedence is: --plan-file/--plan-envvar, then [tool.click-extra.test-plan] inline, then its file, then a built-in default plan that exercises --version and --help. The config maps onto the TestPlanConfig schema (wrapped by ClickExtraConfig).

Running from Python¶

parse_test_plan() turns YAML into cases, and run_test_plan() runs them, returning a Counter of total, skipped, and failed:

from click_extra import parse_test_plan, run_test_plan

cases = list(parse_test_plan(open("plan.yaml").read()))
counter = run_test_plan("weather", cases, jobs=4)
if counter["failed"]:
    raise SystemExit(1)

Build cases directly when a plan is computed rather than read from YAML (this path needs no yaml extra):

from click_extra import CLITestCase, run_test_plan

cases = [
    CLITestCase(cli_parameters="--version", exit_code=0),
    CLITestCase(cli_parameters="forecast --city lyon", stdout_contains="Cloudy"),
]
run_test_plan("weather", cases)

`click_extra.test_plan` API¶

        classDiagram
  Exception <|-- SkippedTest

Declarative, black-box CLI test plans.

A test plan is a list of CLITestCase invocations: each runs a target command (a name, a command line, or a path to a binary) once with extra parameters, then checks its exit code and stdout/stderr against literal, substring, or regex expectations. Cases carry their own platform skip/only rules, so one plan runs across operating systems unchanged.

Plans are usually written as YAML and loaded with parse_test_plan(), which needs the optional click-extra[yaml] extra. run_test_plan() drives a list of cases against a target, parallelized per the resolved --jobs count (see click_extra.execution.run_jobs()) and reporting live progress through a click_extra.spinner.Spinner.

This is the black-box, subprocess-level complement to click_extra.testing.CliRunner, which drives a CLI in-process.

exception click_extra.test_plan.SkippedTest[source]¶

Bases: Exception

Raised when a test case should be skipped.

class click_extra.test_plan.CLITestCase(cli_parameters=<factory>, skip_platforms=<factory>, only_platforms=<factory>, timeout=None, exit_code=None, strip_ansi=False, output_contains=<factory>, stdout_contains=<factory>, stderr_contains=<factory>, output_regex_matches=<factory>, stdout_regex_matches=<factory>, stderr_regex_matches=<factory>, output_regex_fullmatch=None, stdout_regex_fullmatch=None, stderr_regex_fullmatch=None, execution_trace=None)[source]¶

Bases: object

A single CLI test case: how to invoke the command and what to expect.

Each case runs the command-under-test once with cli_parameters appended, then checks the captured result against the expectation directives below. A case with no expectation only asserts the command ran (plus exit_code, if set).

cli_parameters: tuple[str, ...] | str¶

Arguments and options appended to the command-under-test.

A plain string is split into arguments (on spaces on Windows, with shlex elsewhere); a list or tuple is used as-is.

Platforms (or platform-group IDs) on which to skip this case.

Accepts extra_platforms identifiers such as linux, macos, windows, in any case, mixed freely with group IDs.

Restrict this case to these platforms; skip it everywhere else.

The mirror image of skip_platforms, using the same identifiers.

timeout: float | str | None = None¶

Seconds before the command is killed and the case fails as a timeout.

Falls back to the command’s –timeout default, then to no limit.

exit_code: int | str | None = None¶: Expected process exit code; the case fails on any other code.

strip_ansi: bool = False¶: Strip ANSI escape sequences from the captured output before matching.

output_contains: tuple[str, ...] | str¶

Reserved: combined stdout/stderr matching is not implemented yet.

Setting any output_* directive raises at runtime; use the stdout_* and stderr_* variants instead.

stdout_contains: tuple[str, ...] | str¶: Substrings that must all be present in stdout.

stderr_contains: tuple[str, ...] | str¶: Substrings that must all be present in stderr.

output_regex_matches: tuple[Pattern | str, ...] | str¶: Reserved: see output_contains.

stdout_regex_matches: tuple[Pattern | str, ...] | str¶: Regexes that must each match somewhere in stdout (searched, re.DOTALL).

stderr_regex_matches: tuple[Pattern | str, ...] | str¶: Regexes that must each match somewhere in stderr (searched, re.DOTALL).

output_regex_fullmatch: Pattern | str | None = None¶: Reserved: see output_contains.

stdout_regex_fullmatch: Pattern | str | None = None¶: Regex that must fully match stdout, line by line.

stderr_regex_fullmatch: Pattern | str | None = None¶: Regex that must fully match stderr, line by line.

execution_trace: str | None = None¶

Rendering of the command execution and its output.

Populated after the case runs, for inspection on failure; not a directive you set in a test plan.

run_cli_test(command, additional_skip_platforms, default_timeout)[source]¶

Run a CLI command and check its output against the test case.

The provided command can be either:

a path to a binary or script to execute;
a command name to be searched in the PATH,
a command line with arguments to be parsed and executed by the shell.

`{todo} Add support for environment variables. `

`{todo} Add support for proper mixed <stdout>/<stderr> stream as a single, intertwined output. `

click_extra.test_plan.parse_test_plan(plan_string)[source]¶

Return type:: Generator[CLITestCase, None, None]

class click_extra.test_plan.TestPlanConfig(file='./tests/cli-test-plan.yaml', inline=None, timeout=None)[source]¶

Bases: object

Config schema for a project’s test plan, read from [tool.<cli>.test-plan].

The test-plan CLI command resolves its cases from this config when no plan is given on the command line. Map it onto an app’s config section with a field carrying metadata={"click_extra.config_path": "test-plan"}.

file: str = './tests/cli-test-plan.yaml'¶: Path to a YAML test plan file, resolved relative to the project root.

inline: str | None = None¶: Inline YAML test plan, an alternative to file. Takes precedence.

timeout: int | None = None¶

Default timeout (seconds) for each case that does not set its own.

None leaves cases unbounded unless --timeout is passed.

class click_extra.test_plan.ClickExtraConfig(test_plan=<factory>)[source]¶

Bases: object

Schema for the [tool.click-extra] configuration section.

Currently carries only the test-plan sub-table, letting a project point click-extra test-plan at its own plan without repeating it on the command line. It is the config_schema of the test-plan CLI command.

test_plan: TestPlanConfig¶: The [tool.click-extra.test-plan] sub-table (file/inline/timeout).

click_extra.test_plan.run_test_plan(command, cases, *, jobs=1, select_test=None, skip_platform=None, timeout=None, exit_on_error=False, show_trace_on_error=True, stats=True, show_progress=True)[source]¶

Run a list of test cases against a target command and tally the results.

Cases are parallelized per jobs (see click_extra.execution.run_jobs()): at one worker they run sequentially and lazily, so exit_on_error can stop before the rest start; otherwise they run in a thread pool and every case runs to completion. Either way outcomes are tallied in submission order. On an interactive terminal a click_extra.spinner.Spinner reports progress unless show_progress is false.

Parameters:

command (Path | str) – The target to test: a command name, a command line, or a path to a binary or script.
cases (Sequence[CLITestCase]) – The test cases to run.
jobs (int) – Number of parallel workers; 1 runs sequentially.
select_test (Sequence[int] | None) – 1-based case numbers to run; others are skipped.
skip_platform (Trait | Group | str | None | Iterable[Trait | Group | str | None | Iterable[Trait | Group | str | None | Iterable[Trait | Group | str | None | Iterable[_TNestedReferences]]]]) – Extra platforms (or group IDs) to skip every case on.
timeout (float | None) – Default per-case timeout in seconds when a case sets none.
exit_on_error (bool) – Stop at the first failure (sequential runs only).
show_trace_on_error (bool) – Echo the execution trace of each failed case.
stats (bool) – Echo a one-line worker summary up front and a result tally.
show_progress (bool) – Allow the progress spinner on an interactive terminal.

Return type:

Counter

Returns:

A collections.Counter with total, skipped, and failed keys. A non-zero failed count signals the caller to exit with an error.

Test plans¶

Writing a plan¶

Running from the command line¶

Configuring the plan¶

Running from Python¶

click_extra.test_plan API¶

`click_extra.test_plan` API¶