Tutorial¶
This page walks through a complete deduplication run on two tiny mail boxes built for the occasion, so you can rehearse every command in a sandbox before pointing mdedup at your precious archives. Every mdedup output below is captured live while building this page, so what you read is exactly what the documented version does: the commands run from a scratch folder, whose absolute path shows up in the outputs. The one picture on this page, in the safety nets section, is a recording kept in the repository.
Where duplicates come from¶
Duplicate mails are rarely typed twice: they are copies accumulated by the tools around them.
Backups of the same account taken at different times, each holding a copy of the older mails.
Per-folder or per-label exports, where a mail filed under several labels lands in several boxes.
Archives consolidated from multiple clients or machines over the years.
IMAP synchronization accidents, where an interrupted or misconfigured sync re-uploads mails.
Mailing lists reflecting back mails you also keep in your sent box, slightly altered by footers or subject prefixes.
The first four mostly leave copies that are identical byte for byte, which the first pass below catches without interpreting anything. The last one is why mdedup does not stop at comparing bytes: it can also group mails by a hash of a curated set of headers, so two copies can differ in transport details and still be recognized as the same mail.
Build a playground¶
Save this script as make_playground.py. It creates two overlapping mbox archives, mimicking two backups of the same account taken a year apart:
import mailbox
from email.message import EmailMessage
def make_mail(subject, date, msg_id, body):
mail = EmailMessage()
mail["From"] = "news@example.com"
mail["To"] = "me@example.com"
mail["Subject"] = subject
mail["Date"] = date
mail["Message-ID"] = msg_id
mail.set_content(body)
return mail
# Two mails present in both archives: these are the duplicates.
shared = [
make_mail(
"Weekly digest",
"Mon, 05 Feb 2024 09:00:00 +0000",
"<digest-2024-02-05@example.com>",
"This week's news.\n",
),
make_mail(
"Server maintenance window",
"Wed, 03 Apr 2024 17:30:00 +0000",
"<maintenance-2024-04-03@example.com>",
"The server will be down on Friday.\n",
),
]
# One mail kept twice, as it was sent and as a mailing list delivered it back:
# a copy too, with a subject prefix, the list's header and a footer.
sent = make_mail(
"Garden club digest",
"Fri, 10 May 2024 10:00:00 +0000",
"<garden-2024-05-10@example.com>",
"This month: plums and quinces.\n",
)
reflected = make_mail(
"[garden-club] Garden club digest",
"Fri, 10 May 2024 10:00:00 +0000",
"<garden-2024-05-10@example.com>",
"This month: plums and quinces.\n"
"\n--\nUnsubscribe: https://lists.example.com/garden-club\n",
)
reflected["List-Id"] = "<garden-club.lists.example.com>"
# Mails unique to each archive.
only_2024 = [
make_mail(
"Invoice January",
"Wed, 31 Jan 2024 12:00:00 +0000",
"<invoice-2024-01@example.com>",
"Your invoice is attached.\n",
),
]
only_2025 = [
make_mail(
"Weekly digest",
"Mon, 03 Feb 2025 09:00:00 +0000",
"<digest-2025-02-03@example.com>",
"This week's news, a year later.\n",
),
make_mail(
"Password reset",
"Tue, 11 Mar 2025 08:15:00 +0000",
"<reset-2025-03-11@example.com>",
"Click the link to reset your password.\n",
),
]
for path, mails in (
("archive-2024.mbox", shared + only_2024 + [sent]),
("archive-2025.mbox", shared + only_2025 + [reflected]),
):
box = mailbox.mbox(path)
for mail in mails:
box.add(mailbox.mboxMessage(mail))
box.flush()
box.close()
print(f"{path}: {len(mails)} mails")
Run it from an empty working folder:
$ python make_playground.py
archive-2024.mbox: 4 mails
archive-2025.mbox: 5 mails
That is 9 mails in total, of which only 6 are distinct. Two traps are laid for naive matchers. The two archives each contain a different “Weekly digest” mail, sent a year apart: a deduplicator keying on subjects alone would wrongly collapse them. And the garden club digest is kept twice, as sent and as the list delivered it back: a copy that no comparison of bytes recognizes.
First pass: the conservative preset¶
Start with --preset conservative. It treats as copies only the mails whose bytes are identical, line endings aside: no header is normalized, no body is decoded, no threshold applies. A preset is an alias of other options, which the run lists as it starts. Identical copies are interchangeable, so it keeps the copy of each mail whose path sorts first. The default copy-selected action then writes every mail that survives into the --export box. The sources are only read, so this run needs no rehearsal:
$ mdedup --preset conservative --export first-pass.mbox archive-2024.mbox archive-2025.mbox
info: Resolved --jobs to 1 (4 logical CPUs).
info: Apply the options of the conservative preset:
info: --raw-headers --hash-body raw --size-threshold -1 --content-threshold -1 --strategy select-one
● Step #1 - Load mails
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox ...
info: mbox detected.
info: 4 mails found.
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox ...
info: mbox detected.
info: 5 mails found.
● Step #2 - Compute hashes and group duplicates
info: Use every header, as stored, to compute hashes.
● Step #3 - Select mails in each group
info: select-one strategy will be applied on each duplicate set to select candidates.
info: Skip checking for size differences.
info: Skip checking for content differences.
● Step #4 - Perform action on selected mails
info: Perform copy-selected action...
info: 7 mails selected for action.
info: Creating new mbox box at /tmp/mdedup-tutorial-z932v0og/first-pass.mbox ...
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:0> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:1> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:2> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:3> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox:2> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox:3> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox:4> copied
✓ Copied 7/7 mails
● Step #5 - Report and statistics
╭────────────────────┬────────┬────────────────────────────────────────────────╮
│ Mails │ Metric │ Description │
├────────────────────┼────────┼────────────────────────────────────────────────┤
│ Found │ 9 │ Total number of mails encountered from all │
│ │ │ mail sources. │
│ Rejected │ 0 │ Number of mails rejected individually because │
│ │ │ they could not be read. │
│ Unhashed │ 0 │ Number of mails with too few of the hashed │
│ │ │ headers to compute a hash. They joined no │
│ │ │ duplicate set, and were kept in the final │
│ │ │ selection like unique mails. │
│ Failed │ 0 │ Number of mails set aside, and left untouched, │
│ │ │ because hashing them failed on an unexpected │
│ │ │ error. The bug report after this table tells │
│ │ │ how to report it. │
│ Retained │ 9 │ Number of valid mails parsed and retained for │
│ │ │ deduplication. │
│ Hashes │ 7 │ Number of unique hashes. │
│ Unique │ 5 │ Number of unique mails (which were │
│ │ │ automatically added to selection). │
│ Duplicates │ 4 │ Number of duplicate mails (sum of mails in all │
│ │ │ duplicate sets with at least 2 mails). │
│ Skipped │ 0 │ Number of mails ignored in the selection step │
│ │ │ because the whole set they belong to was │
│ │ │ skipped. │
│ Discarded │ 2 │ Number of mails discarded from the final │
│ │ │ selection. │
│ Selected │ 2 │ Number of mails kept in the final selection on │
│ │ │ which the action will be performed. │
│ Copied │ 7 │ Number of mails copied from their original │
│ │ │ mailbox to another. │
│ Moved │ 0 │ Number of mails moved from their original │
│ │ │ mailbox to another. │
│ Deleted │ 0 │ Number of mails deleted from their mailbox in- │
│ │ │ place. │
│ Hardlinked │ 0 │ Number of mails replaced in-place by a │
│ │ │ hardlink to the copy kept in their duplicate │
│ │ │ set. │
│ Hardlink - Skipped │ 0 │ Number of mails left untouched by the │
│ │ │ hardlinking action, because they could not be │
│ │ │ linked to their copy or already were. │
╰────────────────────┴────────┴────────────────────────────────────────────────╯
╭─────────────────────┬────────┬───────────────────────────────────────────────╮
│ Duplicate sets │ Metric │ Description │
├─────────────────────┼────────┼───────────────────────────────────────────────┤
│ Total │ 7 │ Total number of duplicate sets. │
│ Single │ 5 │ Total number of sets containing only a single │
│ │ │ mail with no applicable strategy. They were │
│ │ │ automatically kept in the final selection. │
│ Skipped - Size │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ size. │
│ Skipped - Content │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ content. │
│ Skipped - Timestamp │ 0 │ Number of sets skipped from the selection │
│ │ │ process because a timestamp could not be │
│ │ │ derived for some of their mails. │
│ Skipped - Strategy │ 0 │ Number of sets skipped from the selection │
│ │ │ process because the strategy could not be │
│ │ │ applied. │
│ Skipped - Error │ 0 │ Number of sets skipped from the selection │
│ │ │ process because reading or comparing their │
│ │ │ mails failed on an error. │
│ Deduplicated │ 2 │ Number of valid sets on which the selection │
│ │ │ strategy was successfully applied. │
╰─────────────────────┴────────┴───────────────────────────────────────────────╯
Read the report bottom-up: the two shared mails were found in both archives, the copy of each in archive-2024.mbox was selected and the other discarded, and the 2 selected mails were copied to first-pass.mbox together with the 5 mails that have no identical copy. 9 mails in, 7 out, and nothing lost: each copy discarded here is the same bytes as the copy kept.
$ grep --count "^From " first-pass.mbox
7
The garden club digest is still there twice, as its two copies differ: the list added a subject prefix, a header and a footer. Whatever a conservative pass leaves behind differs somewhere, and the rest of this page is about seeing through such differences.
Tip
The export box defaults to the mbox format. Pass --export-format maildir to produce a maildir folder instead, ready to be dropped on an IMAP server or opened by any client.
Second pass: when copies differ¶
The conservative preset set five options for you. A run without --preset takes the default preset, which leaves each of them at its default:
Which headers identify a mail:
--raw-headershashes every header as stored. By default, the hash reads the--hash-headerlist instead, ten headers each reduced to a canonical form:Dateis kept to its day, andSubjectloses itsRe:and[list]prefixes.Whether the body counts:
--hash-body rawhashes its bytes. By default, the body is left out.How far two copies may differ before they are left alone:
--size-thresholdand--content-threshold, measured on the decoded bodies. The conservative preset turns both off, as identical bytes cannot differ.Which copy of a set to keep:
--strategy select-one. By default, no strategy runs.
Under those defaults, the two copies of the garden club digest hash alike, and their bodies differ by a footer the thresholds allow. The price is interpretation: a run that sees through differences can be wrong about one, so it deserves a rehearsal before any destructive action, which is what --dry-run is for.
--preset aggressive hashes Message-ID alone. It also groups the copies whose other headers a gateway rewrote, with the thresholds as the only check, so rehearse it too.
Choose a strategy¶
A strategy decides, within each set of copies, which mails are selected and which are discarded. Without one, mdedup refuses to guess: every duplicate set is skipped, and only the mails without copies reach the export box. Strategies come in mirrored pairs (select-oldest is discard-newer, and so on), and in a few families:
Time-based:
select-oldest,select-newestand friends, comparing dates.Size-based:
select-smallest,select-biggestand friends, comparing mail sizes.Path-based:
select-matching-pathand friends, testing mail locations against--regexp.Quantity-based:
select-onekeeps the copy whose path sorts first, andselect-all-but-onediscards it, for when copies are indistinguishable.
The full list, with the exact semantics of each, sits at the bottom of mdedup --help and in the CLI parameters page.
Backup copies of the same mail are usually byte-identical: same Date header, same size. Time-based and size-based strategies cannot tell such copies apart, and skip the set as a whole rather than acting on it. That makes select-one the right strategy for merging identical copies, and the one the conservative preset applies on its own. It picks the same copy on every run, so a --dry-run names the copies the real run keeps.
Strategies can also be chained into a fallback cascade by repeating the option: --strategy select-oldest --strategy select-one keeps the oldest copy in the sets where dates differ, and falls back to the copy whose path sorts first in the sets where they do not.
Merge into a clean box¶
--strategy select-one picks one copy per set, and the default action (copy-selected) writes every selected mail to the --export box, leaving the sources untouched:
$ mdedup --strategy select-one --export merged.mbox archive-2024.mbox archive-2025.mbox
info: Resolved --jobs to 1 (4 logical CPUs).
● Step #1 - Load mails
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox ...
info: mbox detected.
info: 4 mails found.
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox ...
info: mbox detected.
info: 5 mails found.
● Step #2 - Compute hashes and group duplicates
info: Use [date, from, to, subject, mime-version, content-type, content-disposition, user-agent, x-priority, message-id] headers to compute hashes.
● Step #3 - Select mails in each group
info: select-one strategy will be applied on each duplicate set to select candidates.
● Step #4 - Perform action on selected mails
info: Perform copy-selected action...
info: 6 mails selected for action.
info: Creating new mbox box at /tmp/mdedup-tutorial-z932v0og/merged.mbox ...
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:0> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:1> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:2> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:3> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox:2> copied
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox:3> copied
✓ Copied 6/6 mails
● Step #5 - Report and statistics
╭────────────────────┬────────┬────────────────────────────────────────────────╮
│ Mails │ Metric │ Description │
├────────────────────┼────────┼────────────────────────────────────────────────┤
│ Found │ 9 │ Total number of mails encountered from all │
│ │ │ mail sources. │
│ Rejected │ 0 │ Number of mails rejected individually because │
│ │ │ they could not be read. │
│ Unhashed │ 0 │ Number of mails with too few of the hashed │
│ │ │ headers to compute a hash. They joined no │
│ │ │ duplicate set, and were kept in the final │
│ │ │ selection like unique mails. │
│ Failed │ 0 │ Number of mails set aside, and left untouched, │
│ │ │ because hashing them failed on an unexpected │
│ │ │ error. The bug report after this table tells │
│ │ │ how to report it. │
│ Retained │ 9 │ Number of valid mails parsed and retained for │
│ │ │ deduplication. │
│ Hashes │ 6 │ Number of unique hashes. │
│ Unique │ 3 │ Number of unique mails (which were │
│ │ │ automatically added to selection). │
│ Duplicates │ 6 │ Number of duplicate mails (sum of mails in all │
│ │ │ duplicate sets with at least 2 mails). │
│ Skipped │ 0 │ Number of mails ignored in the selection step │
│ │ │ because the whole set they belong to was │
│ │ │ skipped. │
│ Discarded │ 3 │ Number of mails discarded from the final │
│ │ │ selection. │
│ Selected │ 3 │ Number of mails kept in the final selection on │
│ │ │ which the action will be performed. │
│ Copied │ 6 │ Number of mails copied from their original │
│ │ │ mailbox to another. │
│ Moved │ 0 │ Number of mails moved from their original │
│ │ │ mailbox to another. │
│ Deleted │ 0 │ Number of mails deleted from their mailbox in- │
│ │ │ place. │
│ Hardlinked │ 0 │ Number of mails replaced in-place by a │
│ │ │ hardlink to the copy kept in their duplicate │
│ │ │ set. │
│ Hardlink - Skipped │ 0 │ Number of mails left untouched by the │
│ │ │ hardlinking action, because they could not be │
│ │ │ linked to their copy or already were. │
╰────────────────────┴────────┴────────────────────────────────────────────────╯
╭─────────────────────┬────────┬───────────────────────────────────────────────╮
│ Duplicate sets │ Metric │ Description │
├─────────────────────┼────────┼───────────────────────────────────────────────┤
│ Total │ 6 │ Total number of duplicate sets. │
│ Single │ 3 │ Total number of sets containing only a single │
│ │ │ mail with no applicable strategy. They were │
│ │ │ automatically kept in the final selection. │
│ Skipped - Size │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ size. │
│ Skipped - Content │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ content. │
│ Skipped - Timestamp │ 0 │ Number of sets skipped from the selection │
│ │ │ process because a timestamp could not be │
│ │ │ derived for some of their mails. │
│ Skipped - Strategy │ 0 │ Number of sets skipped from the selection │
│ │ │ process because the strategy could not be │
│ │ │ applied. │
│ Skipped - Error │ 0 │ Number of sets skipped from the selection │
│ │ │ process because reading or comparing their │
│ │ │ mails failed on an error. │
│ Deduplicated │ 3 │ Number of valid sets on which the selection │
│ │ │ strategy was successfully applied. │
╰─────────────────────┴────────┴───────────────────────────────────────────────╯
Three duplicate sets this time: the two the first pass found, and the garden club digest, whose two copies now share a hash. One copy of each was selected, and the 3 selected mails were copied to merged.mbox together with the 3 unique mails. 9 mails in, 6 mails out:
$ grep --count "^From " merged.mbox
6
Both “Weekly digest” mails made it through, as their different Date and Message-ID headers put them in different sets.
Delete duplicates in place¶
Copying is the safe default, but sometimes you want to prune the originals: say the mails in archive-2025.mbox are the canonical ones, and any copy of them lingering in archive-2024.mbox should go.
Path-based strategies handle this preference: select-matching-path keeps the copies whose location matches --regexp, so the delete-discarded action removes the copies living elsewhere. Destructive actions deserve a rehearsal first, and --dry-run reports what the run would do without touching a mail:
$ mdedup --strategy select-matching-path --regexp 2025 --action delete-discarded --dry-run archive-2024.mbox archive-2025.mbox
info: Resolved --jobs to 1 (4 logical CPUs).
● Step #1 - Load mails
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox ...
info: mbox detected.
info: 4 mails found.
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox ...
info: mbox detected.
info: 5 mails found.
● Step #2 - Compute hashes and group duplicates
info: Use [date, from, to, subject, mime-version, content-type, content-disposition, user-agent, x-priority, message-id] headers to compute hashes.
● Step #3 - Select mails in each group
info: select-matching-path strategy will be applied on each duplicate set to select candidates.
● Step #4 - Perform action on selected mails
info: Perform delete-discarded action...
info: 6 mails selected for action.
warning: DRY RUN: 3 mails would be acted upon, but none will be altered.
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:0> would be deleted
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:1> would be deleted
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:3> would be deleted
✓ DRY RUN: would have Deleted 3/3 mails
● Step #5 - Report and statistics
╭────────────────────┬────────┬────────────────────────────────────────────────╮
│ Mails │ Metric │ Description │
├────────────────────┼────────┼────────────────────────────────────────────────┤
│ Found │ 9 │ Total number of mails encountered from all │
│ │ │ mail sources. │
│ Rejected │ 0 │ Number of mails rejected individually because │
│ │ │ they could not be read. │
│ Unhashed │ 0 │ Number of mails with too few of the hashed │
│ │ │ headers to compute a hash. They joined no │
│ │ │ duplicate set, and were kept in the final │
│ │ │ selection like unique mails. │
│ Failed │ 0 │ Number of mails set aside, and left untouched, │
│ │ │ because hashing them failed on an unexpected │
│ │ │ error. The bug report after this table tells │
│ │ │ how to report it. │
│ Retained │ 9 │ Number of valid mails parsed and retained for │
│ │ │ deduplication. │
│ Hashes │ 6 │ Number of unique hashes. │
│ Unique │ 3 │ Number of unique mails (which were │
│ │ │ automatically added to selection). │
│ Duplicates │ 6 │ Number of duplicate mails (sum of mails in all │
│ │ │ duplicate sets with at least 2 mails). │
│ Skipped │ 0 │ Number of mails ignored in the selection step │
│ │ │ because the whole set they belong to was │
│ │ │ skipped. │
│ Discarded │ 3 │ Number of mails discarded from the final │
│ │ │ selection. │
│ Selected │ 3 │ Number of mails kept in the final selection on │
│ │ │ which the action will be performed. │
│ Copied │ 0 │ Number of mails copied from their original │
│ │ │ mailbox to another. │
│ Moved │ 0 │ Number of mails moved from their original │
│ │ │ mailbox to another. │
│ Deleted │ 3 │ Number of mails deleted from their mailbox in- │
│ │ │ place. │
│ Hardlinked │ 0 │ Number of mails replaced in-place by a │
│ │ │ hardlink to the copy kept in their duplicate │
│ │ │ set. │
│ Hardlink - Skipped │ 0 │ Number of mails left untouched by the │
│ │ │ hardlinking action, because they could not be │
│ │ │ linked to their copy or already were. │
╰────────────────────┴────────┴────────────────────────────────────────────────╯
╭─────────────────────┬────────┬───────────────────────────────────────────────╮
│ Duplicate sets │ Metric │ Description │
├─────────────────────┼────────┼───────────────────────────────────────────────┤
│ Total │ 6 │ Total number of duplicate sets. │
│ Single │ 3 │ Total number of sets containing only a single │
│ │ │ mail with no applicable strategy. They were │
│ │ │ automatically kept in the final selection. │
│ Skipped - Size │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ size. │
│ Skipped - Content │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ content. │
│ Skipped - Timestamp │ 0 │ Number of sets skipped from the selection │
│ │ │ process because a timestamp could not be │
│ │ │ derived for some of their mails. │
│ Skipped - Strategy │ 0 │ Number of sets skipped from the selection │
│ │ │ process because the strategy could not be │
│ │ │ applied. │
│ Skipped - Error │ 0 │ Number of sets skipped from the selection │
│ │ │ process because reading or comparing their │
│ │ │ mails failed on an error. │
│ Deduplicated │ 3 │ Number of valid sets on which the selection │
│ │ │ strategy was successfully applied. │
╰─────────────────────┴────────┴───────────────────────────────────────────────╯
Three deletions planned, as expected: the two shared mails and the garden club digest, in their archive-2024.mbox incarnation. Do not let the 6 mails selected line worry you: it counts the mails that will survive, while the action itself only touches the 3 discarded copies. Drop --dry-run to proceed:
$ mdedup --strategy select-matching-path --regexp 2025 --action delete-discarded archive-2024.mbox archive-2025.mbox
info: Resolved --jobs to 1 (4 logical CPUs).
● Step #1 - Load mails
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox ...
info: mbox detected.
info: 4 mails found.
info: Opening /tmp/mdedup-tutorial-z932v0og/archive-2025.mbox ...
info: mbox detected.
info: 5 mails found.
● Step #2 - Compute hashes and group duplicates
info: Use [date, from, to, subject, mime-version, content-type, content-disposition, user-agent, x-priority, message-id] headers to compute hashes.
● Step #3 - Select mails in each group
info: select-matching-path strategy will be applied on each duplicate set to select candidates.
● Step #4 - Perform action on selected mails
info: Perform delete-discarded action...
info: 6 mails selected for action.
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:0> deleted
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:1> deleted
✓ <mboxDedupMail /tmp/mdedup-tutorial-z932v0og/archive-2024.mbox:3> deleted
✓ Deleted 3/3 mails
● Step #5 - Report and statistics
╭────────────────────┬────────┬────────────────────────────────────────────────╮
│ Mails │ Metric │ Description │
├────────────────────┼────────┼────────────────────────────────────────────────┤
│ Found │ 9 │ Total number of mails encountered from all │
│ │ │ mail sources. │
│ Rejected │ 0 │ Number of mails rejected individually because │
│ │ │ they could not be read. │
│ Unhashed │ 0 │ Number of mails with too few of the hashed │
│ │ │ headers to compute a hash. They joined no │
│ │ │ duplicate set, and were kept in the final │
│ │ │ selection like unique mails. │
│ Failed │ 0 │ Number of mails set aside, and left untouched, │
│ │ │ because hashing them failed on an unexpected │
│ │ │ error. The bug report after this table tells │
│ │ │ how to report it. │
│ Retained │ 9 │ Number of valid mails parsed and retained for │
│ │ │ deduplication. │
│ Hashes │ 6 │ Number of unique hashes. │
│ Unique │ 3 │ Number of unique mails (which were │
│ │ │ automatically added to selection). │
│ Duplicates │ 6 │ Number of duplicate mails (sum of mails in all │
│ │ │ duplicate sets with at least 2 mails). │
│ Skipped │ 0 │ Number of mails ignored in the selection step │
│ │ │ because the whole set they belong to was │
│ │ │ skipped. │
│ Discarded │ 3 │ Number of mails discarded from the final │
│ │ │ selection. │
│ Selected │ 3 │ Number of mails kept in the final selection on │
│ │ │ which the action will be performed. │
│ Copied │ 0 │ Number of mails copied from their original │
│ │ │ mailbox to another. │
│ Moved │ 0 │ Number of mails moved from their original │
│ │ │ mailbox to another. │
│ Deleted │ 3 │ Number of mails deleted from their mailbox in- │
│ │ │ place. │
│ Hardlinked │ 0 │ Number of mails replaced in-place by a │
│ │ │ hardlink to the copy kept in their duplicate │
│ │ │ set. │
│ Hardlink - Skipped │ 0 │ Number of mails left untouched by the │
│ │ │ hardlinking action, because they could not be │
│ │ │ linked to their copy or already were. │
╰────────────────────┴────────┴────────────────────────────────────────────────╯
╭─────────────────────┬────────┬───────────────────────────────────────────────╮
│ Duplicate sets │ Metric │ Description │
├─────────────────────┼────────┼───────────────────────────────────────────────┤
│ Total │ 6 │ Total number of duplicate sets. │
│ Single │ 3 │ Total number of sets containing only a single │
│ │ │ mail with no applicable strategy. They were │
│ │ │ automatically kept in the final selection. │
│ Skipped - Size │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ size. │
│ Skipped - Content │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ content. │
│ Skipped - Timestamp │ 0 │ Number of sets skipped from the selection │
│ │ │ process because a timestamp could not be │
│ │ │ derived for some of their mails. │
│ Skipped - Strategy │ 0 │ Number of sets skipped from the selection │
│ │ │ process because the strategy could not be │
│ │ │ applied. │
│ Skipped - Error │ 0 │ Number of sets skipped from the selection │
│ │ │ process because reading or comparing their │
│ │ │ mails failed on an error. │
│ Deduplicated │ 3 │ Number of valid sets on which the selection │
│ │ │ strategy was successfully applied. │
╰─────────────────────┴────────┴───────────────────────────────────────────────╯
$ grep --count "^From " archive-2024.mbox
1
$ grep --count "^From " archive-2025.mbox
5
archive-2024.mbox is down to its single unique mail, and archive-2025.mbox was not touched. Mails without duplicates are never deleted, whatever the strategy and whichever of the two delete actions runs: only members of a duplicate set are.
Caution
For folder-based boxes (maildir, mh, eml), --regexp is tested against the path of each individual mail file. For file-based boxes (mbox, babyl, mmdf), all mails share the path of the box itself.
Link duplicates instead of deleting them¶
Deleting a discarded copy frees its space, but the mail also leaves the folder it was in. Sometimes every copy has a reason to stay: the same message delivered to two accounts you sync side by side, or a mail filed under several labels. The hardlink-discarded action covers that case. Each discarded mail keeps its name and its folder, and only the file behind it changes: it becomes a hardlink to the copy that was selected, so the copies share a single file and the space the others took is given back.
This needs boxes where each mail owns a file, so maildir rather than mbox. Reusing the playground’s mails, in the same folder:
import mailbox
# The two shared mails delivered to two accounts synced side by side, each
# account also holding a mail of its own.
for account, mails in (
("account-a", shared + only_2024),
("account-b", shared + only_2025),
):
box = mailbox.Maildir(account)
for mail in mails:
box.add(mail)
box.close()
That is 7 mails, each stored in a file of its own:
$ find account-a account-b -type f | wc --lines
7
$ find account-a account-b -type f -printf '%i\n' | sort --unique | wc --lines
7
The path-based strategy from the previous section applies here too: --regexp account-a keeps the copies stored under account-a, so the account-b copies are the ones linked back to them. Rehearse with --dry-run as before, then run it for real:
$ mdedup --strategy select-matching-path --regexp account-a --action hardlink-discarded account-a account-b
info: Resolved --jobs to 1 (4 logical CPUs).
● Step #1 - Load mails
info: Opening /tmp/mdedup-tutorial-z932v0og/account-a ...
info: maildir detected.
info: 3 mails found.
info: Opening /tmp/mdedup-tutorial-z932v0og/account-b ...
info: maildir detected.
info: 4 mails found.
● Step #2 - Compute hashes and group duplicates
info: Use [date, from, to, subject, mime-version, content-type, content-disposition, user-agent, x-priority, message-id] headers to compute hashes.
● Step #3 - Select mails in each group
info: select-matching-path strategy will be applied on each duplicate set to select candidates.
● Step #4 - Perform action on selected mails
info: Perform hardlink-discarded action...
info: 5 mails selected for action.
✓ <MaildirDedupMail /tmp/mdedup-tutorial-z932v0og/account-b/new/1794023760.M392751P2647Q4.runnervmwuk1x> hardlinked
✓ <MaildirDedupMail /tmp/mdedup-tutorial-z932v0og/account-b/new/1794023760.M393332P2647Q5.runnervmwuk1x> hardlinked
✓ Hardlinked 2/2 mails, reclaiming 550 B
● Step #5 - Report and statistics
╭────────────────────┬────────┬────────────────────────────────────────────────╮
│ Mails │ Metric │ Description │
├────────────────────┼────────┼────────────────────────────────────────────────┤
│ Found │ 7 │ Total number of mails encountered from all │
│ │ │ mail sources. │
│ Rejected │ 0 │ Number of mails rejected individually because │
│ │ │ they could not be read. │
│ Unhashed │ 0 │ Number of mails with too few of the hashed │
│ │ │ headers to compute a hash. They joined no │
│ │ │ duplicate set, and were kept in the final │
│ │ │ selection like unique mails. │
│ Failed │ 0 │ Number of mails set aside, and left untouched, │
│ │ │ because hashing them failed on an unexpected │
│ │ │ error. The bug report after this table tells │
│ │ │ how to report it. │
│ Retained │ 7 │ Number of valid mails parsed and retained for │
│ │ │ deduplication. │
│ Hashes │ 5 │ Number of unique hashes. │
│ Unique │ 3 │ Number of unique mails (which were │
│ │ │ automatically added to selection). │
│ Duplicates │ 4 │ Number of duplicate mails (sum of mails in all │
│ │ │ duplicate sets with at least 2 mails). │
│ Skipped │ 0 │ Number of mails ignored in the selection step │
│ │ │ because the whole set they belong to was │
│ │ │ skipped. │
│ Discarded │ 2 │ Number of mails discarded from the final │
│ │ │ selection. │
│ Selected │ 2 │ Number of mails kept in the final selection on │
│ │ │ which the action will be performed. │
│ Copied │ 0 │ Number of mails copied from their original │
│ │ │ mailbox to another. │
│ Moved │ 0 │ Number of mails moved from their original │
│ │ │ mailbox to another. │
│ Deleted │ 0 │ Number of mails deleted from their mailbox in- │
│ │ │ place. │
│ Hardlinked │ 2 │ Number of mails replaced in-place by a │
│ │ │ hardlink to the copy kept in their duplicate │
│ │ │ set. │
│ Hardlink - Skipped │ 0 │ Number of mails left untouched by the │
│ │ │ hardlinking action, because they could not be │
│ │ │ linked to their copy or already were. │
╰────────────────────┴────────┴────────────────────────────────────────────────╯
╭─────────────────────┬────────┬───────────────────────────────────────────────╮
│ Duplicate sets │ Metric │ Description │
├─────────────────────┼────────┼───────────────────────────────────────────────┤
│ Total │ 5 │ Total number of duplicate sets. │
│ Single │ 3 │ Total number of sets containing only a single │
│ │ │ mail with no applicable strategy. They were │
│ │ │ automatically kept in the final selection. │
│ Skipped - Size │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ size. │
│ Skipped - Content │ 0 │ Number of sets skipped from the selection │
│ │ │ process because they were too dissimilar in │
│ │ │ content. │
│ Skipped - Timestamp │ 0 │ Number of sets skipped from the selection │
│ │ │ process because a timestamp could not be │
│ │ │ derived for some of their mails. │
│ Skipped - Strategy │ 0 │ Number of sets skipped from the selection │
│ │ │ process because the strategy could not be │
│ │ │ applied. │
│ Skipped - Error │ 0 │ Number of sets skipped from the selection │
│ │ │ process because reading or comparing their │
│ │ │ mails failed on an error. │
│ Deduplicated │ 2 │ Number of valid sets on which the selection │
│ │ │ strategy was successfully applied. │
╰─────────────────────┴────────┴───────────────────────────────────────────────╯
$ find account-a account-b -type f | wc --lines
7
$ find account-a account-b -type f -printf '%i\n' | sort --unique | wc --lines
5
Still 7 mails on disk, now backed by 5 files: each account-b copy shares the file of its account-a twin. Both accounts still list everything they held, and each copy keeps its own maildir flags, which live in the file name rather than in the file, so the same mail can stay read in one account and unread in the other. Running the command again changes nothing, as copies already sharing a file are reported as such and left alone.
Caution
Only copies identical byte for byte are linked by default. Two copies of a mail that reached you by different routes usually differ, if only by a Received header, and linking those swaps one copy’s content for the other’s. --hardlink-differing allows it, as a deliberate choice. Mails from mbox, babyl and mmdf boxes are always left alone, having no file of their own to link.
Tip
--preset conservative --action hardlink-discarded reclaims space without a rehearsal: the conservative preset only ever discards identical copies, which are the ones this action links by default.
Safety nets¶
Several safeguards run before any mail is acted upon, each detailed in the design page. Unless you override its hash options, the first two have nothing to check under the conservative preset, whose duplicate sets hold identical bytes by construction.
A mail with too few of the hashed headers gets no hash, as a hash resting on so little cannot be trusted. It joins no duplicate set, and is kept like a mail without duplicates.
Mails differing too much in size or content from the rest of their set are considered suspicious: they are set aside, and only the mails that all match each other are deduplicated. The set is skipped as a whole when no such core remains. The
--size-thresholdand--content-thresholdoptions tune these limits, and--show-diffprints the offending differences.A strategy that would select all mails of a set, or none, achieves nothing: the set is handed over to the next
--strategyif one was chained, and left untouched otherwise.Unique mails, and mails with no hash, are always part of the final selection, so a merge never drops them.
A mail or a set that hits an unexpected error is set aside and left untouched, and the run ends with a bug report to paste into an issue. See errors and bug reports.
--show-diff prints the difference that made a set fail the content threshold. Here two mails share every hashed header, Message-ID included, but one holds a soup recipe and the other a weather log:
$ mdedup --show-diff --strategy select-one --export merged.mbox new.mbox old.mbox
When nothing gets deduplicated¶
A run that changes nothing is the number one source of confusion, so map the statistics report to its cause:
The conservative preset found nothing: no two mails share their bytes. Copies can still be there, differing by a header a server or a client added, a subject prefix or a footer. The second pass sees through such differences, and
--hash-onlyshows what each mail hashes to under any preset.Copied,MovedorDeletedat zero: the action found nothing to act on. That row is the symptom, so read the rows above it for the cause. With--action delete-selected, zero is also what a box holding no duplicate at all reports, as mails without a copy are never deleted.Duplicates: 0: no two mails shared a hash. A single box that was never merged or re-synced typically holds no duplicates:mdedupshines on piles of overlapping boxes. If you are certain copies are in there, they may differ in the hashed headers: mails re-delivered or forwarded can get a newMessage-IDfor instance. Narrow the matching down with repeated--hash-headeroptions, or inspect what each mail hashes to with--hash-only.Unhashedabove zero: these mails have too few of the hashed headers to get a hash. They joined no duplicate set, so no strategy ruled on them. The conservative preset still groups their identical copies: it hashes every header, with no minimal number.Duplicatesabove zero butSkipped - Strategycounting sets: either no--strategywas given, or the chosen criterion cannot split the copies apart (identical copies share the same date and size). Switch toselect-oneor a path-based strategy, or chain one as a fallback with a repeated--strategyoption.Skipped - Timestampcounting sets: a time-based strategy could not compare some mails because they lack a parseableDateheader. Chain a fallback with a repeated--strategyoption so another criterion takes over, or pick a strategy that does not depend on time, likeselect-one.Skipped - SizeorSkipped - Contentcounting sets, orSet aside ... mails too dissimilarwarnings in the logs: mails grouped under the same hash differ more than the thresholds allow. The usual cause is one mail archived twice in two shapes. A newsletter kept once as plain text and once with its base64 attachment or its HTML part puts thousands of bytes between two genuine copies. Inspect the difference with--show-diff. Then raise--size-thresholdand--content-thresholddeliberately, or set both to-1to drop the check. Dropping it leaves the hash as the only evidence that two mails are copies, so do not narrow--hash-headerat the same time. Each header left out of the hash lets more mails share it.
Going further¶
Every option demonstrated here is described in the CLI parameters page.
Recurring options can be saved in a configuration file.
The design page details hashing, header normalization and the safeguards.
Header-based hashing can be complemented by body hashing for stricter matching: see
--hash-body.The performance page covers
--cache, which reuses the hashes of previous runs, and--jobs, which spreads both the hashing and the selection over several processes.