Contribution guide¶

Development commands¶

# Run the test suite (test dependencies live in the `test` group).
$ uv run --group test -- pytest

# Run a single test.
$ uv run --group test -- pytest tests/test_strategy.py::test_name

# Type-check at the minimum Python version, with the mypy that the newest
# repomatic release past the cooldown pins. With no target, the runner checks
# what CI checks, `docs/` scripts included.
$ uvx --exclude-newer '1 week' repomatic run mypy

# Build the documentation into the gitignored output directory. Like the CI
# build, it fails on any warning.
$ uv run --group docs -- sphinx-build -b html --fail-on-warning --jobs auto ./docs ./docs/_build/html

# Run the CLI from the working tree.
$ uv run -- mdedup --help

`AGENTS.md` file¶

This file provides guidance to Claude Code when working with code in this repository.

Project overview¶

Mail Deduplicate (mdedup) is a CLI that finds and removes duplicate mails across mail boxes. It reads maildir, mbox, MH, Babyl, MMDF and eml boxes, groups mails into duplicate sets by a hash of selected headers, applies a user-chosen strategy to pick which copies to keep, then copies, moves, deletes or hardlinks the rest.

Upstream conventions¶

This repository uses the reusable workflows and pyproject.toml configuration from kdeldycke/repomatic and follows the conventions established there. Since 7.14.0, repomatic no longer syncs instructions into a consuming repository: the generic conventions for code style, typing, documentation, testing and design live in the machine-wide instructions file, and this file records only what is specific to mail-deduplicate.

Contributing upstream: if you spot a gap or improvement in the reusable workflows or shared conventions, propose it at kdeldycke/repomatic.

Source of truth hierarchy¶

This file defines the rules; the codebase and CI are what those rules are measured against. When they disagree, fix the code to match. If a rule itself is wrong, fix this file.

Keeping this file lean¶

Record only conventions, rationale and non-obvious rules that cannot be discovered by reading the code. Do not paste the module tree, source snippets or general Python knowledge here: reference the source instead.

Commands¶

The contribution guide (docs/contributing.md) lists the commands that run the tests, check the types, build the documentation and run the CLI.

Architecture¶

mdedup runs a four-step pipeline, orchestrated by the Deduplicate class in deduplicate.py. Each step is documented in depth in docs/design.md:

Load the source boxes and read their mails (mail_box.py, mail.py).
Hash mails into DuplicateSets keyed by a hash of selected headers (deduplicate.py).
Select which mails to keep within each set, via a selection strategy (strategy.py).
Act on the selected or discarded mails: copy, move, delete or hardlink (action.py).

Module	Responsibility
`cache.py`	Cross-run SQLite cache of mail hashes, its invalidation and its default location.
`cli.py`	The `mdedup` Click command, its `Config`, and option groups.
`deduplicate.py`	`Deduplicate` orchestrator, `DuplicateSet`, worker plumbing, statistics (`Stat`).
`mail.py`	`DedupMailMixin`: a mail wrapped with dedup-relevant properties (hash, date, size).
`mail_box.py`	Box formats (`BoxFormat`), autodetection, opening, locking, subfolders.
`strategy.py`	Selection strategies (oldest/newest, size, matching-path, …).
`action.py`	Actions applied to the selected or discarded mails.
`bug_report.py`	Mail fingerprints, failure capture and the bug report of unexpected errors.

Non-obvious rules¶

DedupMailMixin is mixed into each mailbox.Message subclass at runtime by make_dedup_mail(), so every box format shares one dedup implementation. Keep format-specific code in mail_box.py; keep format-agnostic dedup logic on the mixin.
After the hashing step every retained mail is dehydrated: it keeps only its box reference, identity and memoized scalars (timestamp, size), not its parsed message. The content-dependent properties (body_lines, canonical_headers, parsed_date) transparently re-read the message from its source box. Any new post-hashing consumer of mail content must go through them (or call hydrate()), and dehydrate afterwards to keep memory flat.
Add a mail to a box with add_mail(), never box.add(): it writes the mail as the bytes its source box holds, and carries into a box of the same format the metadata those bytes leave out (maildir folder, flags and modification time, MH sequences, and the envelope line, which also travels between mbox and MMDF). Python’s email generator rebuilds a parsed message instead of writing it back, and drops some malformed mails on the way.
An unexpected error in the hashing or selection step puts its mail or its duplicate set aside. Only the action step stops on such an error. A new expected error, one with a cause the user can fix, gets its own except before the catch-all, so it never reaches the bug report. The report and the fingerprint never print mail content or a path: keep any new field to structure, as bug_report.py explains.
In tests, hand a stdlib Mailbox.add() LF bytes only. It rewrites each LF as the separator of the platform, so CRLF bytes come out with doubled CRs on Windows, the headers end after the first one, and the mail has too few headers to get a hash. Write a CRLF fixture straight to a file, or into an EML box, which writes bytes as they are.
Three safeguards make destructive runs safer: an automatic minimal-headers floor (derived as min(4, number of --hash-header values), no option) plus a size and a content threshold, both on by default and only dropped when set to -1. They are described in docs/design.md; run mdedup --help for the threshold option names. When changing selection or hashing, re-read those safeguards: they exist to avoid deleting mails that only look like duplicates. The conservative preset sidesteps all three, as --raw-headers with --hash-body raw hashes every byte of a mail. Keep that preset free of any interpretation: the tutorial sells it as the pass that needs no rehearsal.