Installation¶

Quick start¶

Mail Deduplicate is distributed on PyPI.

Install the latest stable release with uv:

$ uv tool install mail-deduplicate

Try it now¶

You can try Mail Deduplicate right now in your terminal, without installing any dependency or virtual env thanks to uvx:

Latest release

$ uvx --from mail-deduplicate -- mdedup --help

$ mdedup --help
Usage: mdedup [OPTIONS] MAIL_SOURCE_1 MAIL_SOURCE_2 ...

  Deduplicate mails from multiple sources.

  Process:
  - Step #1: load mails from their sources.
  - Step #2: compute the canonical hash of each mail based on their headers (and
             optionally their body), and regroup mails sharing the same hash.
  - Step #3: apply a selection strategy on each subset of duplicate mails.
  - Step #4: perform an action on all selected mails.
  - Step #5: report statistics.

Positional arguments:
  MAIL_SOURCE_1 MAIL_SOURCE_2 ...
     Mail sources to deduplicate. Can be a single mail box or a list of mails.

Preset:
  --preset [conservative|default|aggressive]
     Start from the option values of a preset. conservative treats as copies
     only the mails whose bytes are identical, and stands for --raw-headers
     --hash-body raw --size-threshold -1 --content-threshold -1 --strategy
     select-one. default keeps the built-in defaults. aggressive trusts the
     Message-ID header alone to group copies, and stands for --hash-header
     Message-ID. An option set on the command line, in the environment or in a
     configuration file overrides the value the preset gives it. conservative
     turns the thresholds off only with its own --raw-headers and --hash-body
     raw. When another value or a --hash-header list replaces either one, both
     thresholds keep their defaults.  [default: default]

Mail sources (step #1):
  -i, --input-format [maildir|mbox|mh|babyl|mmdf|eml]
     Force all provided mail sources to be parsed in the specified format. If
     not set, auto-detect the format of sources independently. Auto-detection
     only supports maildir, mbox and eml formats. Use this option to open up
     other box formats, or bypass unreliable detection.

  -u, --force-unlock
     Remove the lock on mail source opening if one is found.

Hashing (step #2):
  -h, --hash-header Header-ID
     Headers to use to compute each mail's hash. Must be repeated multiple times
     to set an ordered list of headers. Header IDs are case-insensitive.
     Repeating entries are ignored.  [default: Date, From, To, Subject, MIME-
     Version, Content-Type, Content-Disposition, User-Agent, X-Priority,
     Message-ID]

  --raw-headers / --no-raw-headers
     Hash the header block of each mail as its box stores it, line endings
     aside: every header, in its order, with no normalization. Replaces the
     --hash-header list, so the two are refused together, and no minimal number
     of headers is required.  [default: no-raw-headers]

  -b, --hash-body [skip|raw|decoded|normalized]
     Method used to hash the body of mails. Defaults to skip, which doesn't hash
     the body at all: it is the fastest method and header-based hash should be
     sufficient to determine duplicate set. raw hashes the body bytes as the box
     holds them, with no decoding. decoded undoes the transfer encodings and
     decodes the text first, so copies that differ in their encoding alone hash
     the same (slow). normalized also strips all whitespace (slowest).
     [default: skip]

  -H, --hash-only
     Compute and display the internal hashes used to identify duplicates. Do not
     performs any selection or action.

  --cache / --no-cache
     Reuse the hashes computed by previous runs, so unchanged mails are not read
     nor parsed again. An entry is only trusted while the size and modification
     time of the file backing its mail are unchanged, and the whole cache is
     discarded as soon as any option feeding the hashes changes. Mails of a
     file-based box (mbox, babyl, mmdf) all share the box's file, so editing it
     invalidates every one of them at once. Off by default.  [default: no-cache]

  --cache-path CACHE_DB_PATH
     Location of the hash cache database. Implies --cache. Defaults to
     /home/runner/.cache/mdedup/hashes.db

Deduplication (step #3):
  Process each set of mails sharing the same hash and apply the selection
  --strategy. Fine-grained checks on size and content are performed beforehand.
  Mails differing above safety levels are set aside so the rest can still be
  deduplicated, and the set is skipped if fewer than 2 remain. Limits can be set
  via the --size-threshold and --content-threshold options.
  -s, --strategy [select-older|select-oldest|select-newer|select-newest|discard-newest|discard-newer|discard-oldest|discard-older|select-smaller|select-smallest|select-bigger|select-biggest|discard-biggest|discard-bigger|discard-smallest|discard-smaller|select-matching-path|select-non-matching-path|discard-non-matching-path|discard-matching-path|select-one|select-all-but-one|discard-all-but-one|discard-one]
     Selection strategy to apply within a subset of duplicates. Can be repeated
     multiple times to set an ordered list of fallback strategies: each
     duplicate set is handed over to the next strategy each time a strategy
     fails to discriminate its mails, by selecting all of them, none of them, or
     by missing the timestamps to compare them. Repeating entries are ignored,
     including aliases of strategies already listed. If not set, every duplicate
     set is skipped: none of its mails is selected or discarded. The mails
     without duplicates stay selected, and a copy or a move still carries them.
     Description of each strategy is available further down that help screen.

  -t, --time-source [date-header|ctime]
     Source of a mail's time reference used in time-sensitive strategies.
     [default: date-header]

  -r, --regexp REGEXP
     Regular expression on a mail's file path. Applies to individual mail
     location for folder-based boxes (maildir, mh, eml). But for file-based
     boxes (mbox, babyl, mmdf), applies to the whole box's path, as all mails
     are packed into one single file. Required in discard-matching-path,
     discard-non-matching-path, select-matching-path and select-non-matching-
     path strategies.

  -S, --size-threshold BYTES
     Maximum difference allowed in size between mails sharing the same hash, in
     bytes of their decoded bodies. Mails in an offending pair are set aside
     until the rest all pass. The subset is skipped if fewer than 2 remain. Set
     to 0 to enforce strictness and apply selection strategy on the subset only
     if all mails are exactly the same. Set to -1 to allow any difference and
     apply the strategy whatever the differences.  [default: 512; x>=-1]

  -C, --content-threshold BYTES
     Maximum difference allowed in content between mails sharing the same hash,
     in bytes of the diff of their decoded bodies. Mails in an offending pair
     are set aside until the rest all pass. The subset is skipped if fewer than
     2 remain. Set to 0 to enforce strictness and apply selection strategy on
     the subset only if all mails are exactly the same. Set to -1 to allow any
     difference and apply the strategy whatever the differences.  [default: 768;
     x>=-1]

  -d, --show-diff
     Show the unified diff of duplicates not within thresholds.

Action (step #4):
  -a, --action [copy-selected|copy-discarded|move-selected|move-discarded|delete-selected|delete-discarded|hardlink-discarded]
     Action performed on the selected mails. Defaults to copy-selected as it is
     the safest: it only reads the mail sources and create a brand new mail box
     with the selection results. Mails that have no duplicate are part of the
     selection, but delete-selected leaves them in place, as deleting one
     removes the only copy of it there is.  [default: copy-selected]

  -E, --export MAIL_BOX_PATH
     Location of the destination mail box to where to copy or move deduplicated
     mails. Required in copy-selected, copy-discarded, move-selected and move-
     discarded actions.

  -e, --export-format [maildir|mbox|mh|babyl|mmdf|eml]
     Format of the mail box to which deduplication mails will be exported to.
     Only affects copy-selected, copy-discarded, move-selected and move-
     discarded actions.  [default: mbox]

  --export-append
     If destination mail box already exists, add mails into it instead of
     interrupting (default behavior). Affect copy-selected, copy-discarded,
     move-selected and move-discarded actions.

  --hardlink-differing
     Hardlink discarded mails whose content differs byte for byte from the copy
     they are linked to, instead of leaving them alone. Their own content is
     then swapped for that copy's, so whatever was unique to them, like the
     headers a mail collects on its way to one account, is lost. Only affects
     the hardlink-discarded action.

  -n, --dry-run
     Do not perform any action but act as if it was, and report which action
     would have been performed otherwise.

Other options:
  --jobs [auto|max|INTEGER]
     Number of parallel jobs used to hash mails (step #2) and settle duplicate
     sets (step #3). Accepts an integer, 'auto' (the host's logical CPUs, minus
     one when there are three or more) or 'max'. Defaults to 1 (sequential).
     Only folder-based boxes fan out; higher values speed up --hash-body
     decoded/normalized and boxes holding many duplicate sets.  [default: 1]

  --help
     Show this message and exit.

Configuration options:
  --config LOCATION
     Location of the configuration file. Supports local path with glob patterns
     or remote URL.  [default: ~/.config/mdedup/]

  --no-config
     Ignore all configuration files and only use command line parameters and
     environment variables.

  --validate-config LOCATION
     Validate the configuration file and exit.

  --export-config FORMAT
     Export the configuration in the selected format to <stdout>, then exit.

Output options:
  --accessible
     Accessibility mode: disable colors and render tables in a borderless,
     screen-reader-friendly format.

  --color [auto|always|never]
     Colorize the output. A bare --color is the same as --color=always.
     [default: auto]

  --no-color
     Disable colorization (alias of --color=never).

  --progress / --no-progress
     Show progress indicators during long operations. Disabled for non-
     interactive output (pipes, dumb terminals, CI) and by --accessible.
     [default: progress]

  --theme [auto|dark|dracula|light|manpage|monokai|nord|solarized-dark]
     Color theme used for help screens.  [default: dark]

  --table-format FORMAT
     Rendering style of tables.  [default: rounded-outline]

Logging options:
  --verbosity LEVEL
     Either CRITICAL, ERROR, WARNING, INFO, DEBUG.  [default: INFO]

  -v, --verbose
     Increase the default WARNING verbosity by one level for each additional
     repetition of the option.  [default: 0]

  -q, --quiet
     Decrease the default WARNING verbosity by one level for each additional
     repetition of the option.  [default: 0]

  --debug
     Shorthand for --verbosity DEBUG.

Introspection options:
  --time / --no-time
     Measure and print elapsed execution time.  [default: no-time]

  --params
     Show all CLI parameters, their provenance, defaults and value, then exit.

  --tree
     Show the tree of nested subcommands and exit.

  --man
     Read the command's manual page and exit.

  --help-format [carapace|json|json-full|man|markdown|markdown-full]
     Render the command in the given format and exit.

  --version
     Show the version and exit.

Constraints:
  {--regexp}
     all required if -s/--strategy is one of discard-matching-path, discard-non-
     matching-path, select-matching-path, select-non-matching-path

  {--regexp}
     all forbidden if -s/--strategy is none of discard-matching-path, discard-
     non-matching-path, select-matching-path, select-non-matching-path

  {--export}
     all required if -a/--action is one of copy-discarded, copy-selected, move-
     discarded, move-selected

  {--export}
     all forbidden if -a/--action is none of copy-discarded, copy-selected,
     move-discarded, move-selected

Available strategies:
  [select-all-but-one|discard-one]
     Discard the duplicate whose path sorts first, and select all others.

  [select-bigger|discard-smallest]
     Select all bigger duplicates. Discards the smallests, i.e. the subset
     sharing the smallest size.

  [select-biggest|discard-smaller]
     Select all the biggest duplicates. Discards the smallers, i.e. all mail of
     the duplicate set but those sharing the biggest size.

  [select-matching-path|discard-non-matching-path]
     Select all duplicates whose file path matches the --regexp expression.

  [select-newer|discard-oldest]
     Select all newer duplicates. Discards the oldest, i.e. the subset sharing
     the most ancient timestamp.

  [select-newest|discard-older]
     Select all the newest duplicates. Discards the olders, i.e. all mail of the
     duplicate set but those sharing the newest timestamp.

  [select-non-matching-path|discard-matching-path]
     Select all duplicates whose file path doesn't match the --regexp
     expression.

  [select-older|discard-newest]
     Select all older duplicates. Discards the newests, i.e. the subset sharing
     the most recent timestamp.

  [select-oldest|discard-newer]
     Select all the oldest duplicates. Discards the newers, i.e. all mail of the
     duplicate set but those sharing the oldest timestamp.

  [select-one|discard-all-but-one]
     Select the duplicate whose path sorts first, and discard all others.

  [select-smaller|discard-biggest]
     Select all smaller duplicates. Discards the biggests, i.e. the subset
     sharing the biggest size.

  [select-smallest|discard-bigger]
     Select all the smallest duplicates. Discards the biggers. i.e. all mail of
     the duplicate set but those sharing the smallest size.

Specific version

$ uvx --from mail-deduplicate@9.6.0 -- mdedup

Development version

$ uvx --from git+https://github.com/kdeldycke/mail-deduplicate -- mdedup

This will download mail-deduplicate (the package), and run mdedup, the CLI included in the package.

Installation methods¶

Easiest way is to install uv, then install mail-deduplicate system-wide, with the uv tool command:

$ uv tool install mail-deduplicate

Then you can run mdedup directly:

$ mdedup --version

pip

You can install the latest stable release and its dependencies with a simple pip call:

$ python -m pip install mail-deduplicate

Other variations includes:

$ pip install mail-deduplicate

$ pip3 install mail-deduplicate

If you have difficulties to use pip, see pip’s own installation instructions.

pipx

pipx is a great way to install Python applications globally:

$ pipx install mail-deduplicate

Arch Linux

An mdedup package is available on AUR and can be installed with any AUR helper:

$ pacaur -S mail-deduplicate

$ paru -S mail-deduplicate

$ yay -S mail-deduplicate

Homebrew

Mail Deduplicate is available as an Homebrew formula, so you just need to:

$ brew install mail-deduplicate

Support envelope¶

The compatibility table below keeps the support envelope of mail-deduplicate. Outside of it, these are obsolete:

mail-deduplicate releases before 6.1.0 (2021-01-26)
Python versions before 3.6 (2016-12-23)

Python compatibility¶

Each cell holds one of three states:

✅: the release declares support for that Python.
❌: the requires-python of the release rules that Python out.
–: the release neither claimed nor forbade that Python, usually because that Python was not out yet when the release declared its requirements.

See three states, two sources. The table keeps the support envelope of mail-deduplicate, so the oldest releases drop off as the oldest supported Python moves up.

`mail-deduplicate` ↴ \ Python →	`3.15`	`3.14`	`3.13`	`3.12`	`3.11`	`3.10`	`3.9`	`3.8`	`3.7`	`3.6`
`9.5.x` → `9.x`	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
`9.4.0`	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌
`8.0.x` → `9.3.x`	–	✅	✅	✅	✅	✅	❌	❌	❌	❌
`7.6.x`	–	–	✅	✅	✅	✅	❌	❌	❌	❌
`7.5.0`	–	–	–	✅	✅	✅	✅	❌	❌	❌
`7.3.x` → `7.4.x`	–	–	–	✅	✅	✅	✅	✅	❌	❌
`7.0.x` → `7.2.x`	–	–	–	✅	✅	✅	✅	✅	✅	❌
`6.1.x` → `6.2.x`	–	–	–	–	✅	✅	✅	✅	✅	✅

Binaries¶

Binaries are compiled at each release, so you can skip the installation process above and download the standalone executables directly.

This is the preferred way of testing mdedup without polluting your machine. They also offer the possibility of running the CLI on older systems not supporting the minimal Python version required by mdedup.

Platform	`arm64`	`x86_64`
Linux	Download `mdedup-linux-arm64.bin`	Download `mdedup-linux-x64.bin`
macOS	Download `mdedup-macos-arm64.bin`	Download `mdedup-macos-x64.bin`
Windows	Download `mdedup-windows-arm64.exe`	Download `mdedup-windows-x64.exe`

All links above point to the latest released version of mdedup.

Binaries of all past releases, with their VirusTotal analyses, are cataloged on the binaries page.

Release verification¶

Each release signs its binaries with a build provenance attestation. After you download a binary, verify it with the gh CLI:

$ gh attestation verify mdedup-linux-arm64.bin --repo kdeldycke/mail-deduplicate --signer-repo kdeldycke/repomatic

--signer-repo kdeldycke/repomatic is required: the release runs in the reusable _release-engine.yaml workflow of kdeldycke/repomatic, whose identity signs the attestation. Each release attaches every binary twice, once with the version in its name: both copies hold the same bytes, and verify the same way.

The PyPI distributions carry their own PEP 740 attestations, which the PyPI project page shows and verifies.

Python module usage¶

Mail Deduplicate should now be available system-wide:

$ mdedup --version

If not, you can directly execute the module from Python:

$ python -m mail_deduplicate --version

Shell completion¶

Completion for popular shell rely on Click feature.

Bash

Add this to ~/.bashrc:

eval "$(_MDEDUP_COMPLETE=bash_source mdedup)"

Zsh

Add this to ~/.zshrc:

eval "$(_MDEDUP_COMPLETE=zsh_source mdedup)"

Fish

Add this to ~/.config/fish/completions/mdedup.fish:

eval (env _MDEDUP_COMPLETE=fish_source mdedup)

Alternatively, export the generated completion code as a static script to be executed:

Bash

$ _MDEDUP_COMPLETE=bash_source mdedup > ~/.mdedup-complete.bash

Then source it from ~/.bashrc:

. ~/.mdedup-complete.bash

Zsh

$ _MDEDUP_COMPLETE=zsh_source mdedup > ~/.mdedup-complete.zsh

Then source it from ~/.zshrc:

. ~/.mdedup-complete.zsh

Fish

_MDEDUP_COMPLETE=fish_source mdedup > ~/.config/fish/completions/mdedup.fish

Man pages¶

mdedup has a --man option, which shows its manual in a pager, the way man does:

$ mdedup --man

To get the roff source of the page instead, ask for the man rendering of --help-format:

$ mdedup --help-format man > mdedup.1

The page is also published as a static mdedup.1 file:

Bundled as mail-deduplicate-manpages.tar.gz on every GitHub release. Extract it, then copy man/mdedup.1 into a man1 folder of your MANPATH, like /usr/local/share/man/man1/.
Rendered as HTML on the man pages page of this site.