Configuration#

The structure of the configuration file is automatically derived from the parameters of the CLI and their types. There is no need to manually produce a configuration data structure to mirror the CLI.

Standalone option#

The @config_option decorator provided by Click Extra can be used as-is with vanilla Click:

from click import group, option, echo

from click_extra import config_option

@group(context_settings={"show_default": True})
@option("--dummy-flag/--no-flag")
@option("--my-list", multiple=True)
@config_option
def my_cli(dummy_flag, my_list):
    echo(f"dummy_flag    is {dummy_flag!r}")
    echo(f"my_list       is {my_list!r}")

@my_cli.command
@option("--int-param", type=int, default=10)
def subcommand(int_param):
    echo(f"int_parameter is {int_param!r}")

The code above is saved into a file named my_cli.py.

It produces the following help screen:

$ my-cli --help
Usage: my-cli [OPTIONS] COMMAND [ARGS]...

Options:
  --dummy-flag / --no-flag  [default: no-flag]
  --my-list TEXT
  -C, --config CONFIG_PATH  Location of the configuration file. Supports glob
                            pattern of local path and remote URL.  [default:
                            ~/.config/my-cli/*.{toml,yaml,yml,json,ini,xml}]
  --help                    Show this message and exit.

Commands:
  subcommand

See there the explicit mention of the default location of the configuration file. This improves discoverability, and makes sysadmins happy, especially those not familiar with your CLI.

A bare call returns:

$ my-cli subcommand
dummy_flag    is False
my_list       is ()
int_parameter is 10

With a simple TOML file in the application folder, we will change the CLI’s default output.

Here is what ~/.config/my-cli/config.toml contains:

# My default configuration file.
top_level_param = "is_ignored"

[my-cli]
extra_value = "is ignored too"
dummy_flag = true   # New boolean default.
my_list = ["item 1", "item #2", "Very Last Item!"]

[garbage]
# An empty random section that will be skipped.

[my-cli.subcommand]
int_param = 3
random_stuff = "will be ignored"

In the file above, pay attention to:

  • the default configuration base path (~/.config/my-cli/ here on Linux) which is OS-dependant;

  • the app’s folder (/my-cli/) which is built from the script’s name (my_cli.py);

  • the top-level config section ([my-cli]), based on the CLI’s group ID (def my_cli());

  • all the extra comments, sections and values that will be silently ignored.

Now we can verify the configuration file is properly read and change the defaults:

$ my-cli subcommand
dummy_flag    is True
my_list       is ('item 1', 'item #2', 'Very Last Item!')
int_parameter is 3

Precedence#

The configuration loader fetch values according the following precedence:

  • CLI parameters

    • ↖ Configuration file

      • ↖ Environment variables

        • ↖ Defaults

The parameter will take the first value set in that chain.

See how inline parameters takes priority on defaults from the previous example:

$ my-cli subcommand --int-param 555
dummy_flag    is True
my_list       is ('item 1', 'item #2', 'Very Last Item!')
int_parameter is 555

Get configuration values#

After gathering all the configuration from the different sources, and assembling them together following the precedence rules above, the configuration values are merged back into the Context’s default_map. But only the values that are matching the CLI’s parameters are kept and passed as defaults. All others are silently ignored.

You can still access the full configuration by looking into the context’s meta attribute:

from click_extra import option, echo, pass_context, command, config_option


@command
@option("--int-param", type=int, default=10)
@config_option
@pass_context
def my_cli(ctx, int_param):
    echo(f"Configuration location: {ctx.meta['click_extra.conf_source']}")
    echo(f"Full configuration: {ctx.meta['click_extra.conf_full']}")
    echo(f"Default values: {ctx.default_map}")
    echo(f"int_param is {int_param!r}")

[my-cli]
int_param = 3
random_stuff = "will be ignored"

[garbage]
dummy_flag = true

$ my-cli --config ./conf.toml --int-param 999
Load configuration matching ./conf.toml
Configuration location: /home/me/conf.toml
Full configuration: {'my-cli': {'int_param': 3, 'random_stuff': 'will be ignored'}, 'garbage': {'dummy_flag': True}}
Default values: {'int_param': 3}
int_parameter is 999

Hint

Variables in meta are presented in their original Python type:

  • click_extra.conf_source is either a normalized Path or URL object

  • click_extra.conf_full is a dict whose values are either str or richer types, depending on the capabilities of each format

Strictness#

As you can see in the first example above, all unrecognized content is ignored.

If for any reason you do not want to allow any garbage in configuration files provided by the user, you can use the strict argument.

Given this cli.toml file:

[cli]
int_param = 3
random_param = "forbidden"

The use of strict=True parameter in the CLI below:

from click import command, option, echo

from click_extra import config_option

@command
@option("--int-param", type=int, default=10)
@config_option(strict=True)
def cli(int_param):
    echo(f"int_parameter is {int_param!r}")

Will raise an error and stop the CLI execution on unrecognized random_param value:

$ cli --config "cli.toml"
Load configuration matching cli.toml
(...)
ValueError: Parameter 'random_param' is not allowed in configuration file.

Excluding parameters#

The excluded_params argument allows you to block some of your CLI options to be loaded from configuration. By setting this argument, you will prevent your CLI users to set these parameters in their configuration file.

It defaults to the value of ParamStructure.DEFAULT_EXCLUDED_PARAMS.

You can set your own list of option to ignore with the excluded_params argument:

from click import command, option, echo

from click_extra import config_option

@command
@option("--int-param", type=int, default=10)
@config_option(excluded_params=["my-cli.non_configurable_option", "my-cli.dangerous_param"])
def my_cli(int_param):
    echo(f"int_parameter is {int_param!r}")

Attention

You need to provide the fully-qualified ID of the option you’re looking to block. I.e. the dot-separated ID that is prefixed by the CLI name. That way you can specify an option to ignore at any level, including subcommands.

If you have difficulties identifying your options and their IDs, run your CLI with the --show-params option for introspection.

Formats#

Several dialects are supported:

  • TOML

  • YAML

  • JSON, with inline and block comments (Python-style # and Javascript-style //, thanks to commentjson)

  • INI, with extended interpolation, multi-level sections and non-native types (list, set, 
)

  • XML

TOML#

See the example in the top of this page.

YAML#

The example above, given for a TOML configuration file, is working as-is with YAML.

Just replace the TOML file with the following configuration at ~/.config/my-cli/config.yaml:

# My default configuration file.
top_level_param: is_ignored

my-cli:
  extra_value: is ignored too
  dummy_flag: true   # New boolean default.
  my_list:
    - point 1
    - 'point #2'
    - Very Last Point!

  subcommand:
    int_param: 77
    random_stuff: will be ignored

garbage: >
  An empty random section that will be skipped

$ my-cli --config "~/.config/my-cli/config.yaml" subcommand
dummy_flag    is True
my_list       is ('point 1', 'point #2', 'Very Last Point!')
int_parameter is 77

JSON#

Again, same for JSON:

{
  "top_level_param": "is_ignored",
  "garbage": {},
  "my-cli": {
    "dummy_flag": true,
    "extra_value": "is ignored too",
    "my_list": [
      "item 1",
      "item #2",
      "Very Last Item!"
    ],
    "subcommand": {
      "int_param": 65,
      "random_stuff": "will be ignored"
    }
  }
}

$ my-cli --config "~/.config/my-cli/config.json" subcommand
dummy_flag    is True
my_list       is ('item 1', 'item #2', 'Very Last Item!')
int_parameter is 65

INI#

INI configuration files are allowed to use ExtendedInterpolation by default.

Todo

Write example.

XML#

Todo

Write example.

Pattern matching#

The configuration file is searched based on a wildcard-based pattern.

By default, the pattern is /<app_dir>/*.{toml,yaml,yml,json,ini,xml}, where:

  • <app_dir> is the default application folder (see section below)

  • *.{toml,yaml,yml,json,ini,xml} is any file in that folder with any of .toml, .yaml, .yml, .json , .ini or .xml extension.

See also

There is a long history about the choice of the default application folder.

For Unix, the oldest reference I can track is from the Where Configurations Live chapter of The Art of Unix Programming by Eric S. Raymond.

The XDG Base Directory Specification is the latest iteration of this tradition on Linux. This long-due guidelines brings lots of benefits to the platform. This is what Click Extra is implementing by default.

But there is still a lot of cases for which the XDG doesn’t cut it, like on other platforms (macOS, Windows, 
) or for legacy applications. That’s why Click Extra allows you to customize the way configuration is searched and located.

Default folder#

The configuration file is searched in the default application path, as defined by click.get_app_dir().

Like the latter, the @config_option decorator and ConfigOption class accept a roaming and force_posix argument to alter the default path:

Platform

roaming

force_posix

Folder

macOS (default)

-

False

~/Library/Application Support/Foo Bar

macOS

-

True

~/.foo-bar

Unix (default)

-

False

~/.config/foo-bar

Unix

-

True

~/.foo-bar

Windows (default)

True

-

C:\Users\<user>\AppData\Roaming\Foo Bar

Windows

False

-

C:\Users\<user>\AppData\Local\Foo Bar

Let’s change the default base folder in the following example:

from click import command

from click_extra import config_option

@command(context_settings={"show_default": True})
@config_option(force_posix=True)
def cli():
    pass

See how the default to --config option has been changed to ~/.cli/*.{toml,yaml,yml,json,ini,xml}:

$ cli --help
Usage: cli [OPTIONS]

Options:
  -C, --config CONFIG_PATH  Location of the configuration file. Supports glob
                            pattern of local path and remote URL.  [default:
                            ~/.cli/*.{toml,yaml,yml,json,ini,xml}]
  --help                    Show this message and exit.

Custom pattern#

If you’d like to customize the pattern, you can pass your own to the default parameter.

Here is how to look for an extension-less YAML dotfile in the home directory, with a pre-defined .commandrc name:

from click import command

from click_extra import config_option
from click_extra.config import Formats

@command(context_settings={"show_default": True})
@config_option(default="~/.commandrc", formats=Formats.YAML)
def cli():
    pass

$ cli --help
Usage: cli [OPTIONS]

Options:
  -C, --config CONFIG_PATH  Location of the configuration file. Supports glob
                            pattern of local path and remote URL.  [default:
                            ~/.commandrc]
  --help                    Show this message and exit.

Pattern specifications#

Patterns provided to @config_option:

Default extensions#

The extensions that are used for each dialect to produce the default file pattern matching are encoded by the Formats Enum:

Format

Extensions

TOML

*.toml

YAML

*.yaml, *.yml

JSON

*.json

INI

*.ini

XML

*.xml

Multi-format matching#

The default behavior consist in searching for all files matching the default *.{toml,yaml,yml,json,ini,xml} pattern.

A parsing attempt is made for each file matching the extension pattern, in the order of the table above.

As soon as a file is able to be parsed without error and returns a dict, the search stops and the file is used to feed the CLI’s default values.

Forcing formats#

If you know in advance the only format you’d like to support, you can use the formats argument on your decorator like so:

from click import command, option, echo

from click_extra import config_option
from click_extra.config import Formats

@command(context_settings={"show_default": True})
@option("--int-param", type=int, default=10)
@config_option(formats=Formats.JSON)
def cli(int_param):
    echo(f"int_parameter is {int_param!r}")

Notice how the default search pattern gets limited to files with a .json extension:

$ cli --help
Usage: cli [OPTIONS]

Options:
  --int-param INTEGER       [default: 10]
  -C, --config CONFIG_PATH  Location of the configuration file. Supports glob
                            pattern of local path and remote URL.  [default:
                            ~/.config/cli/*.json]
  --help                    Show this message and exit.

This also works with a subset of formats:

from click import command, option, echo

from click_extra import config_option
from click_extra.config import Formats

@command(context_settings={"show_default": True})
@option("--int-param", type=int, default=10)
@config_option(formats=[Formats.INI, Formats.YAML])
def cli(int_param):
    echo(f"int_parameter is {int_param!r}")

$ cli --help
Usage: cli [OPTIONS]

Options:
  --int-param INTEGER       [default: 10]
  -C, --config CONFIG_PATH  Location of the configuration file. Supports glob
                            pattern of local path and remote URL.  [default:
                            ~/.config/cli/*.{ini,yaml,yml}]
  --help                    Show this message and exit.

Remote URL#

Remote URL can be passed directly to the --config option:

$ my-cli --config "https://example.com/dummy/configuration.yaml" subcommand
dummy_flag    is True
my_list       is ('point 1', 'point #2', 'Very Last Point!')
int_parameter is 77

click_extra.config API#

classDiagram Enum <|-- Formats ExtraOption <|-- ConfigOption ParamStructure <|-- ConfigOption

Utilities to load parameters and options from a configuration file.

class click_extra.config.Formats(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: Enum

Supported configuration formats and the list of their default extensions.

The default order set the priority by which each format is searched for the default configuration file.

TOML = ('toml',)#
YAML = ('yaml', 'yml')#
JSON = ('json',)#
INI = ('ini',)#
XML = ('xml',)#
class click_extra.config.ConfigOption(param_decls=None, metavar='CONFIG_PATH', type=STRING, help='Location of the configuration file. Supports glob pattern of local path and remote URL.', is_eager=True, expose_value=False, formats=(<Formats.TOML: ('toml', )>, <Formats.YAML: ('yaml', 'yml')>, <Formats.JSON: ('json', )>, <Formats.INI: ('ini', )>, <Formats.XML: ('xml', )>), roaming=True, force_posix=False, excluded_params=None, strict=False, **kwargs)[source]#

Bases: ExtraOption, ParamStructure

A pre-configured option adding --config/-C option.

Takes as input a glob pattern or an URL.

Glob patterns must follow the syntax of wcmatch.glob.

  • is_eager is active by default so the config option’s callback gets the opportunity to set the default_map values before the other options use them.

  • formats is the ordered list of formats that the configuration file will be tried to be read with. Can be a single one.

  • roaming and force_posix are fed to click.get_app_dir() to setup the default configuration folder.

  • excluded_params is a list of options to ignore by the configuration parser. Defaults to ParamStructure.DEFAULT_EXCLUDED_PARAMS.

  • strict

    • If True, raise an error if the configuration file contain unrecognized content.

    • If False, silently ignore unsupported configuration option.

formats: Sequence[Formats]#
roaming: bool#
force_posix: bool#
strict: bool#
default_pattern()[source]#

Returns the default pattern used to search for the configuration file.

Defaults to /<app_dir>/*.{toml,yaml,yml,json,ini,xml}. Where <app_dir> is produced by the clickget_app_dir() method. The result depends on OS and is influenced by the roaming and force_posix properties of this instance.

In that folder, we’re looking for any file matching the extensions derived from the self.formats property: :rtype: str

  • a simple *.ext pattern if only one format is set

  • an expanded *.{ext1,ext2,...} pattern if multiple formats are set

get_help_record(ctx)[source]#

Replaces the default value by the pretty version of the configuration matching pattern.

Return type:

tuple[str, str] | None

search_and_read_conf(pattern)[source]#

Search on local file system or remote URL files matching the provided pattern.

pattern is considered an URL only if it is parseable as such and starts with http:// or https://.

Returns an iterator of the normalized configuration location and its textual content, for each file/URL matching the pattern.

Return type:

Iterable[tuple[Path | URL, str]]

parse_conf(conf_text)[source]#

Try to parse the provided content with each format in the order provided by the user.

A successful parsing in any format is supposed to return a dict. Any other result, including any raised exception, is considered a failure and the next format is tried.

Return type:

dict[str, Any] | None

read_and_parse_conf(pattern)[source]#

Search for a configuration file matching the provided pattern.

Returns the location and parsed content of the first valid configuration file that is not blank, or (None, None) if no file was found.

Return type:

tuple[Path | URL, dict[str, Any]] | tuple[None, None]

default: t.Union[t.Any, t.Callable[[], t.Any]]#
type: types.ParamType#
is_flag: bool#
is_bool_flag: bool#
flag_value: t.Any#
name: t.Optional[str]#
opts: t.List[str]#
secondary_opts: t.List[str]#
load_ini_config(content)[source]#

Utility method to parse INI configuration file.

Internal convention is to use a dot (., as set by self.SEP) in section IDs as a separator between levels. This is a workaround the limitation of INI format which doesn’t allow for sub-sections.

Returns a ready-to-use data structure.

Return type:

dict[str, Any]

recursive_update(a, b)[source]#

Like standard dict.update(), but recursive so sub-dict gets updated.

Ignore elements present in b but not in a.

Return type:

dict[str, Any]

merge_default_map(ctx, user_conf)[source]#

Save the user configuration into the context’s default_map.

Merge the user configuration into the pre-computed template structure, which will filter out all unrecognized options not supported by the command. Then cleans up blank values and update the context’s default_map.

Return type:

None

load_conf(ctx, param, path_pattern)[source]#

Fetch parameters values from configuration file and sets them as defaults.

User configuration is merged to the context’s default_map, like Click does.

By relying on Click’s default_map, we make sure that precedence is respected. And direct CLI parameters, environment variables or interactive prompts takes precedence over any values from the config file.

Return type:

None