CLI¶
Help screen¶
$ mdedup --help
Usage: mdedup [OPTIONS] MAIL_SOURCE_1 MAIL_SOURCE_2 ...
Deduplicate mails from multiple sources.
Process:
- Step #1: load mails from their sources.
- Step #2: compute the canonical hash of each mail based on their headers (and
optionally their body), and regroup mails sharing the same hash.
- Step #3: apply a selection strategy on each subset of duplicate mails.
- Step #4: perform an action on all selected mails.
- Step #5: report statistics.
Positional arguments:
MAIL_SOURCE_1 MAIL_SOURCE_2 ...
Mail sources to deduplicate. Can be a single mail box or a list of mails.
Mail sources (step #1):
-i, --input-format [maildir|mbox|mh|babyl|mmdf|eml]
Force all provided mail sources to be parsed in the specified format. If
not set, auto-detect the format of sources independently. Auto-detection
only supports maildir, mbox and eml formats. Use this option to open up
other box formats, or bypass unreliable detection.
-u, --force-unlock
Remove the lock on mail source opening if one is found.
Hashing (step #2):
-h, --hash-header Header-ID
Headers to use to compute each mail's hash. Must be repeated multiple times
to set an ordered list of headers. Header IDs are case-insensitive.
Repeating entries are ignored. [default: Date, From, To, Subject, MIME-
Version, Content-Type, Content-Disposition, User-Agent, X-Priority,
Message-ID]
-b, --hash-body [skip|raw|normalized]
Method used to hash the body of mails. Defaults to skip, which doesn't hash
the body at all: it is the fastest method and header-based hash should be
sufficient to determine duplicate set. raw use the body as it is (slow).
normalized pre-process the body before hashing, by removing all line breaks
and spaces (slowest). [default: skip]
-H, --hash-only
Compute and display the internal hashes used to identify duplicates. Do not
performs any selection or action.
--cache / --no-cache
Reuse the hashes computed by previous runs, so unchanged mails are not read
nor parsed again. An entry is only trusted while the size and modification
time of the file backing its mail are unchanged, and the whole cache is
discarded as soon as any option feeding the hashes changes. Mails of a
file-based box (mbox, babyl, mmdf) all share the box's file, so editing it
invalidates every one of them at once. Off by default. [default: no-cache]
--cache-path CACHE_DB_PATH
Location of the hash cache database. Implies --cache. Defaults to
/home/runner/.cache/mdedup/hashes.db
Deduplication (step #3):
Process each set of mails sharing the same hash and apply the selection
--strategy. Fine-grained checks on size and content are performed beforehand.
Mails differing above safety levels are set aside so the rest can still be
deduplicated, and the set is skipped if fewer than 2 remain. Limits can be set
via the --size-threshold and --content-threshold options.
-s, --strategy [select-older|select-oldest|select-newer|select-newest|discard-newest|discard-newer|discard-oldest|discard-older|select-smaller|select-smallest|select-bigger|select-biggest|discard-biggest|discard-bigger|discard-smallest|discard-smaller|select-matching-path|select-non-matching-path|discard-non-matching-path|discard-matching-path|select-one|select-all-but-one|discard-all-but-one|discard-one]
Selection strategy to apply within a subset of duplicates. Can be repeated
multiple times to set an ordered list of fallback strategies: each
duplicate set is handed over to the next strategy each time a strategy
fails to discriminate its mails, by selecting all of them, none of them, or
by missing the timestamps to compare them. Repeating entries are ignored,
including aliases of strategies already listed. If not set, duplicates will
be grouped and counted but all be skipped, selection will be empty, and no
action will be performed. Description of each strategy is available further
down that help screen.
-t, --time-source [date-header|ctime]
Source of a mail's time reference used in time-sensitive strategies.
[default: date-header]
-r, --regexp REGEXP
Regular expression on a mail's file path. Applies to individual mail
location for folder-based boxes (maildir, mh, eml). But for file-based
boxes (mbox, babyl, mmdf), applies to the whole box's path, as all mails
are packed into one single file. Required in discard-matching-path,
discard-non-matching-path, select-matching-path and select-non-matching-
path strategies.
-S, --size-threshold BYTES
Maximum difference allowed in size between mails sharing the same hash.
Mails in an offending pair are set aside until the rest all pass. The
subset is skipped if fewer than 2 remain. Set to 0 to enforce strictness
and apply selection strategy on the subset only if all mails are exactly
the same. Set to -1 to allow any difference and apply the strategy whatever
the differences. [default: 512; x>=-1]
-C, --content-threshold BYTES
Maximum difference allowed in content between mails sharing the same hash.
Mails in an offending pair are set aside until the rest all pass. The
subset is skipped if fewer than 2 remain. Set to 0 to enforce strictness
and apply selection strategy on the subset only if all mails are exactly
the same. Set to -1 to allow any difference and apply the strategy whatever
the differences. [default: 768; x>=-1]
-d, --show-diff
Show the unified diff of duplicates not within thresholds.
Action (step #4):
-a, --action [copy-selected|copy-discarded|move-selected|move-discarded|delete-selected|delete-discarded|hardlink-discarded]
Action performed on the selected mails. Defaults to copy-selected as it is
the safest: it only reads the mail sources and create a brand new mail box
with the selection results. [default: copy-selected]
-E, --export MAIL_BOX_PATH
Location of the destination mail box to where to copy or move deduplicated
mails. Required in copy-selected, copy-discarded, move-selected and move-
discarded actions.
-e, --export-format [maildir|mbox|mh|babyl|mmdf|eml]
Format of the mail box to which deduplication mails will be exported to.
Only affects copy-selected, copy-discarded, move-selected and move-
discarded actions. [default: mbox]
--export-append
If destination mail box already exists, add mails into it instead of
interrupting (default behavior). Affect copy-selected, copy-discarded,
move-selected and move-discarded actions.
--hardlink-differing
Hardlink discarded mails whose content differs byte for byte from the copy
they are linked to, instead of leaving them alone. Their own content is
then swapped for that copy's, so whatever was unique to them, like the
headers a mail collects on its way to one account, is lost. Only affects
the hardlink-discarded action.
-n, --dry-run
Do not perform any action but act as if it was, and report which action
would have been performed otherwise.
Other options:
--time / --no-time
Measure and print elapsed execution time. [default: no-time]
--config CONFIG_PATH
Location of the configuration file. Supports local path with glob patterns
or remote URL. [default:
~/.config/mdedup/{*.toml,*.yaml,*.yml,*.json,*.ini,pyproject.toml}]
--no-config
Ignore all configuration files and only use command line parameters and
environment variables.
--validate-config FILE
Validate the configuration file and exit.
--export-config FORMAT
Export the configuration in the selected format to <stdout>, then exit.
--accessible
Accessibility mode: disable colors and render tables in a plain, screen-
reader-friendly format.
--color [auto|always|never]
Colorize the output. A bare --color is the same as --color=always.
[default: auto]
--no-color
Disable colorization (alias of --color=never).
--progress / --no-progress
Show progress indicators during long operations. Disabled for non-
interactive output (pipes, dumb terminals, CI) and by --accessible.
[default: progress]
--theme [dark|dracula|light|manpage|monokai|nord|solarized_dark]
Color theme used for help screens. [default: dark]
--params
Show all CLI parameters, their provenance, defaults and value, then exit.
--table-format [aligned|asciidoc|colon-grid|csv|csv-excel|csv-excel-tab|csv-unix|double-grid|double-outline|fancy-grid|fancy-outline|github|grid|heavy-grid|heavy-outline|hjson|html|jira|json|json5|jsonc|latex|latex-booktabs|latex-longtable|latex-raw|mediawiki|mixed-grid|mixed-outline|moinmoin|orgtbl|outline|pipe|plain|presto|pretty|psql|rounded-grid|rounded-outline|rst|simple|simple-grid|simple-outline|textile|toml|tsv|unsafehtml|vertical|xml|yaml|youtrack]
Rendering style of tables. [default: rounded-outline]
--verbosity LEVEL
Either CRITICAL, ERROR, WARNING, INFO, DEBUG. [default: INFO]
-v, --verbose
Increase the default WARNING verbosity by one level for each additional
repetition of the option. [default: 0]
-q, --quiet
Decrease the default WARNING verbosity by one level for each additional
repetition of the option. [default: 0]
--tree
Show the tree of nested subcommands and exit.
--man
Show the command's man page (roff) and exit.
--version
Show the version and exit.
--jobs [auto|max|INTEGER]
Number of parallel jobs used to hash mails (step #2) and settle duplicate
sets (step #3). Accepts an integer, 'auto' (one fewer than the host's
logical CPUs) or 'max'. Defaults to 1 (sequential). Only folder-based boxes
fan out; higher values speed up --hash-body raw/normalized and boxes
holding many duplicate sets. [default: 1]
--help
Show this message and exit.
Constraints:
{--regexp}
all required if -s/--strategy is one of discard-matching-path, discard-non-
matching-path, select-matching-path, select-non-matching-path
{--regexp}
all forbidden if -s/--strategy is none of discard-matching-path, discard-
non-matching-path, select-matching-path, select-non-matching-path
{--export}
all required if -a/--action is one of copy-discarded, copy-selected, move-
discarded, move-selected
{--export}
all forbidden if -a/--action is none of copy-discarded, copy-selected,
move-discarded, move-selected
Available strategies:
[select-all-but-one|discard-one]
Randomly discard one duplicate, and select all others.
[select-bigger|discard-smallest]
Select all bigger duplicates. Discards the smallests, i.e. the subset
sharing the smallest size.
[select-biggest|discard-smaller]
Select all the biggest duplicates. Discards the smallers, i.e. all mail of
the duplicate set but those sharing the biggest size.
[select-matching-path|discard-non-matching-path]
Select all duplicates whose file path match the regular expression provided
via the --regexp parameter.
[select-newer|discard-oldest]
Select all newer duplicates. Discards the oldest, i.e. the subset sharing
the most ancient timestamp.
[select-newest|discard-older]
Select all the newest duplicates. Discards the olders, i.e. all mail of the
duplicate set but those sharing the newest timestamp.
[select-non-matching-path|discard-matching-path]
Select all duplicates whose file path doesn't match the regular expression
provided via the --regexp parameter.
[select-older|discard-newest]
Select all older duplicates. Discards the newests, i.e. the subset sharing
the most recent timestamp.
[select-oldest|discard-newer]
Select all the oldest duplicates. Discards the newers, i.e. all mail of the
duplicate set but those sharing the oldest timestamp.
[select-one|discard-all-but-one]
Randomly select one duplicate, and discards all others.
[select-smaller|discard-biggest]
Select all smaller duplicates. Discards the biggests, i.e. the subset
sharing the biggest size.
[select-smallest|discard-bigger]
Select all the smallest duplicates. Discards the biggers. i.e. all mail of
the duplicate set but those sharing the smallest size.
Command |
Description |
|---|---|
Deduplicate mail boxes |