mail_deduplicate package

Expose package-wide elements.

Submodules

mail_deduplicate.action module

Actions performed once the selection is settled: copy, move, delete or hardlink.

Each action ID pairs an operation verb with the subset of mails it applies to, and Action.perform() routes one to the other.

mail_deduplicate.action.export_box(dedup)

Context manager for export box operations.

Return type:

Generator[Mailbox | None, None, None]

mail_deduplicate.action.dry_run_prefix(dedup)

Marks a summary as describing what a run would have done.

A dry run reports through the same trail as a real one, and this prefix does for the trail’s closing summary what outcome() does for each of its lines.

Return type:

str

mail_deduplicate.action.outcome(dedup, done)

Words what happened to a mail, for its line of the trail.

A dry run touches no mail, so it says what the action would have done to each of them, where a real run reports it as done.

Return type:

str

mail_deduplicate.action.copy_mails(dedup, mails)

Copy provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.move_mails(dedup, mails)

Move provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.delete_mails(dedup, mails)

Remove provided mails in-place, from their original boxes.

Return type:

None

mail_deduplicate.action.LINK_TEMP_PREFIX = '.mdedup-hardlink-'

Prefix of the temporary link a mail is replaced through.

Every folder-based format skips dot-prefixed files when listing its mails, so a temporary left behind by an interrupted run is never read back as one.

mail_deduplicate.action.has_own_file(mail)

Whether the mail is backed by a file holding it alone.

File-based boxes pack all their mails into the box’s single file, which is what path returns for each of them: there would be nothing to link but the whole box.

Return type:

bool

mail_deduplicate.action.hardlink_blocker(dedup, mail, target)

Explain why a mail cannot be replaced by a hardlink to its target.

Returns the reason as a sentence ready to be logged, or None when the link can go ahead. Nothing is touched along the way, so a dry run reaches the same verdicts a real run acts on.

Return type:

str | None

mail_deduplicate.action.replace_by_hardlink(mail_path, target_path)

Point a mail’s own path at the file backing another mail.

The link is created under a temporary name in the mail’s own directory, then renamed over it: the rename is atomic and shares the mail’s filesystem by construction, so the mail is never missing from its box, whatever interrupts the run. The mail also keeps its file name, which is where maildir records the per-folder flags that make the same mail read in one folder and unread in another.

Return type:

None

mail_deduplicate.action.hardlink_mails(dedup, mails)

Replace provided mails in-place by a hardlink to the copy kept in their set.

The mails stay right where they are, under their own name: only the content they are backed by is shared with the copy that survived the selection, so the disk space they took is reclaimed.

Return type:

None

mail_deduplicate.action.OPERATIONS: dict[str, Callable[[Deduplicate, Collection[DedupMailMixin]], None]] = {'copy': <function copy_mails>, 'delete': <function delete_mails>, 'hardlink': <function hardlink_mails>, 'move': <function move_mails>}

The operation functions above, keyed by the verb half of an action ID.

All share the same signature: the deduplication they report to, and the mails they apply to.

class mail_deduplicate.action.Action(*values)

Bases: StrEnum

Define all available action IDs.

An action ID joins an operation verb to the subset of mails it applies to: the *-selected actions act on the mails kept by the selection, the *-discarded ones on the mails it discarded.

The mails that have no duplicate belong to the selection, and so do the mails with too few headers to be hashed: the copy and move actions carry them into the box they build. spares_unique explains why the delete one leaves them behind.

COPY_SELECTED = 'copy-selected'

COPY_DISCARDED = 'copy-discarded'

MOVE_SELECTED = 'move-selected'

MOVE_DISCARDED = 'move-discarded'

DELETE_SELECTED = 'delete-selected'

DELETE_DISCARDED = 'delete-discarded'

HARDLINK_DISCARDED = 'hardlink-discarded'

property verb: str

The operation half of the action ID, keying into OPERATIONS.

property acts_on_discarded: bool

Whether the action applies to the discarded mails, not the selected ones.

property spares_unique: bool

Whether the action must leave the mails that have no duplicate alone.

A mail alone in its duplicate set is kept, so it belongs to the selection every *-selected action targets. So does a mail with too few headers to be hashed, which joins no set at all. Copying and moving it is what makes the exported box hold the whole deduplicated corpus rather than its duplicates only, and neither loses it. Deleting it does: no strategy ever ruled on that mail, nothing was written anywhere else, and there is no other copy of it to fall back on. So the deletion applies to the mails a strategy really picked. See: https://github.com/kdeldycke/mail-deduplicate/issues/1053

targets(dedup)

The subset of mails this action applies to.

Return type:

set[DedupMailMixin]

perform(dedup)

Perform the action on the subset of mails it targets.

Return type:

None

mail_deduplicate.bug_report module

Bug reports for the errors mdedup has no rule for.

Such an error is a bug in mdedup, or a mail shape it does not handle yet. To fix it, a maintainer needs two facts: the code that raised the error, and the shape of the mail that triggered it. This module collects both, and renders them as a block that a user can paste into an issue.

Caution

A report never holds mail content. Mails carry the personal data of their senders and recipients, and issues are public. So fingerprint() describes a mail by its structure alone. The traceback keeps module names and line numbers, and drops the file paths, which hold the name of the user on most systems.

Todo

Contribute the crash handler, the environment and options lines and the report invitation to click-extra, so every Click Extra CLI can invite bug reports. mdedup then keeps only the mail fingerprint and the quarantine of the failing mails.

mail_deduplicate.bug_report.ISSUES_URL = 'https://github.com/kdeldycke/mail-deduplicate/issues'

Where users file bug reports.

Repeats the Issues URL of pyproject.toml, and a test keeps the two in sync: a binary built by Nuitka ships no package metadata to read it from.

mail_deduplicate.bug_report.REPORT_FORM = 'bug-report.yml'

File name of the issue form a report links to, in .github/ISSUE_TEMPLATE/.

mail_deduplicate.bug_report.REPORT_START = '----- 8< ----- mdedup bug report ----- 8< -----'

First line of the block a user pastes into an issue.

mail_deduplicate.bug_report.REPORT_END = '----- 8< ----- end of bug report ----- 8< -----'

Last line of the block a user pastes into an issue.

mail_deduplicate.bug_report.REPORTS_FOLDER = 'reports'

Folder of the cache directory where save_report() writes each bug report.

mail_deduplicate.bug_report.FAILURE_ATTRIBUTE = 'mdedup_failure'

Attribute of an exception that holds the Failure attached by blame().

mail_deduplicate.bug_report.FINGERPRINTED_MAILS = 2

Mails of one failure that get a fingerprint.

A failing duplicate set does not tell which of its mails caused the error, and its first two mails usually include the culprit.

mail_deduplicate.bug_report.MAX_FINGERPRINT_LINES = 60

Cap on the lines of a fingerprint, for mails with hundreds of headers or parts.

mail_deduplicate.bug_report.MAX_FRAMES = 12

Innermost frames kept from a traceback. A RecursionError has a thousand.

mail_deduplicate.bug_report.MAX_KINDS = 5

Kinds of failure detailed in a report. The warnings of the run name the others.

mail_deduplicate.bug_report.MAX_VALUE_LENGTH = 200

Cap on the characters of an error message or of a header value shown verbatim.

mail_deduplicate.bug_report.PARAMETER_HEADERS = frozenset({'content-disposition', 'content-type'})

Headers a fingerprint shows with their structure only: see structure().

mail_deduplicate.bug_report.SAFE_PARAMETERS = frozenset({'charset', 'delsp', 'format', 'micalg', 'protocol', 'report-type', 'type'})

Parameters of PARAMETER_HEADERS shown as they are: they tell how to read a part.

structure() names the SHAPED_PARAMETERS without their value, and hides the name of any other parameter too.

mail_deduplicate.bug_report.SAMPLES = 3

Failures of each kind that a report details, with the fingerprint of their mails.

More samples of the same bug add length to a report, not information, and each fingerprint costs one more read and parse of its mail.

mail_deduplicate.bug_report.SHAPED_PARAMETERS = frozenset({'boundary', 'filename', 'name'})

Parameters of PARAMETER_HEADERS a fingerprint names, with their shape alone.

They can hold content: a file name, or a boundary built from a sender address.

mail_deduplicate.bug_report.VERBATIM_HEADERS = frozenset({'content-transfer-encoding', 'date', 'mime-version'})

Headers a fingerprint shows as they are: they hold structure, not content.

Their comments are free text, so mask_comments() shows each one by its shape. Date caused several of the past crashes, by its parsing or by the bytes of its comment. See: https://github.com/kdeldycke/mail-deduplicate/issues/600, https://github.com/kdeldycke/mail-deduplicate/issues/954 and https://github.com/kdeldycke/mail-deduplicate/issues/1117

mail_deduplicate.bug_report.CONTROL_BYTES = re.compile(b'[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f]')

C0 control bytes and DEL, except the tab and the line breaks of a mail.

mail_deduplicate.bug_report.ENCODED_WORD = re.compile('=\\?([^?\\s]+)\\?([bBqQ])\\?')

Start of an RFC 2047 encoded word: its charset and its encoding.

mail_deduplicate.bug_report.TOKEN = re.compile("[!#$%&'*+.^_`|~0-9A-Za-z-]+(?:/[!#$%&'*+.^_`|~0-9A-Za-z-]+)?")

An RFC 2045 token, or a type/subtype pair of them.

mail_deduplicate.bug_report.plural(count, singular, several=None)

Count things in words, or say nothing about a count of zero.

Return type:

str

mail_deduplicate.bug_report.to_bytes(text)

Recover the bytes a parsed value came from.

Parsing a mail from bytes keeps each non-ASCII byte as a lone surrogate, which surrogateescape turns back into that byte. Other surrogates come from no byte, so surrogatepass writes them as the invalid UTF-8 they are.

Return type:

bytes

mail_deduplicate.bug_report.byte_class(value)

Describe a value by its length and the kind of its bytes, never by its bytes.

Return type:

str

mail_deduplicate.bug_report.shape(value)

Describe a value by its bytes, its folding and its encoded words.

Return type:

str

mail_deduplicate.bug_report.verbatim(value)

Show a value as it is, with its folding collapsed and its length capped.

A value holding non-ASCII bytes renders as bytes, so each byte shows as an escape rather than as a lone surrogate.

Return type:

str

mail_deduplicate.bug_report.mask_comments(value)

Replace the text of each comment of a header value by its shape.

A comment is free text in parentheses, which RFC 5322 allows between the tokens of a structured header. Some mail clients write the name of their time zone there, after a Date, and some their own name, after MIME-Version. A comment can hold others, and a backslash quotes the character after it. A comment left open runs to the end of the value.

Return type:

str

mail_deduplicate.bug_report.split_parameters(value)

Split a header value on its semicolons, except the ones in a quoted string.

Return type:

list[str]

mail_deduplicate.bug_report.structure(value)

Show a Content-Type or Content-Disposition value by its structure alone.

The leading type/subtype or disposition shows as it is, and so do the SAFE_PARAMETERS. The SHAPED_PARAMETERS keep their name, and any other parameter shows as ?, all of them with the shape of their value only.

Return type:

str

mail_deduplicate.bug_report.header_value(name, value)

Show a header value as its name allows: verbatim, by structure, or by shape.

A verbatim value still shows its comments by their shape.

Return type:

str

mail_deduplicate.bug_report.line_endings(raw)

Name the line endings of a mail, with their counts when they are mixed.

Return type:

str

mail_deduplicate.bug_report.first_raw(part, name)

The raw value of the first header of a part with that name.

A parsed header holding 8-bit bytes reads back as a Header object, and its raw value as a string with lone surrogates, which to_bytes() turns back to bytes.

Return type:

str | None

mail_deduplicate.bug_report.part_lines(part, depth=0)

One line per MIME part, with the subparts of a multipart indented under it.

Return type:

Iterator[str]

mail_deduplicate.bug_report.fingerprint(raw)

Describe a mail by its structure, with none of its content.

The description shows the size and line endings of the mail, then every header name, in order. Headers in VERBATIM_HEADERS show their value but for their comments, those in PARAMETER_HEADERS their structure, and any other one only the shape of its value: length, kind of bytes, folding and encoded words. Then comes the MIME tree, with the type, charset, transfer encoding and kind of bytes of each part, the preamble and epilogue of each multipart, and the defects the parser found.

Never raises: a mail too broken to describe gets a line saying so.

Return type:

str

mail_deduplicate.bug_report.fingerprint_mail(mail)

The fingerprint() of a mail, read again from its source.

Never raises: a mail that cannot be read again gets a line saying so.

Return type:

str

mail_deduplicate.bug_report.box_format_of(item)

The format of a box, or of the box a mail comes from.

Return type:

str

mail_deduplicate.bug_report.describe(expt)

The class and message of an exception, on one line of capped length.

Return type:

str

mail_deduplicate.bug_report.compact_frames(tb)

A traceback as module:line in function entries, outermost first.

Module names say as much as file paths about the code, and nothing about where it is installed, so they are what the report shows. A method shows with its class, like Maildir.add.

Return type:

tuple[str, ...]

mail_deduplicate.bug_report.culprit_function(tb)

The name of the innermost function of mdedup in a traceback, or nothing.

It names the code to fix, even for an error that Python or a library raised from a call of mdedup. A generator expression, a comprehension or a lambda has no name of its own, so the named function nearest to it in the traceback stands for it.

Return type:

str

class mail_deduplicate.bug_report.Failure(step, error, frames, function='', culprits=(), formats=(), fingerprints=(), trace='')

Bases: object

An error mdedup has no rule for, reduced to strings.

Strings can leave a worker process, and keep nothing alive: the mails, boxes and frames of the error are dropped as soon as it is captured.

step: str

What the run was doing: hashing, selecting, deleting, and so on.

error: str

The class and message of the exception.

frames: tuple[str, ...]

The traceback, as compact_frames() renders it.

function: str = ''

The innermost function of mdedup the error went through, as culprit_function() finds it.

culprits: tuple[str, ...] = ()

The repr() of each mail involved. They hold paths, so they never reach the report block.

formats: tuple[str, ...] = ()

The box format of each mail involved.

fingerprints: tuple[str, ...] = ()

The fingerprint of the first mails involved, or nothing for a failure that is not a sample.

trace: str = ''

The full traceback, for the debug log, or nothing for a failure that is not a sample.

property kind: tuple[str, str, str]

Failures of one kind share their step, exception class and raising line.

property title: str

The title of an issue reporting the failure.

It names the exception class, the innermost function of mdedup in the traceback and the step. Reports of the same bug get the same title, so GitHub lists the issue already open for it while the next user types theirs. The function tells apart two bugs raising the same class in the same step, and unlike a line number, it does not change between releases.

mail_deduplicate.bug_report.capture(expt, step, mails=(), sample=True)

Reduce an error to a Failure.

Only a sample gets the fingerprints of its first mails and the full traceback: both cost far more than the rest.

Return type:

Failure

mail_deduplicate.bug_report.failure_lines(samples, title)

The lines of a report detailing one kind of failure.

Return type:

Iterator[str]

class mail_deduplicate.bug_report.FailureLog

Bases: object

The failures of a run: each one counted, the first SAMPLES of a kind kept.

Memory stays flat however many mails fail the same way, which a single bug can make all of them do.

counts: Counter[tuple[str, str, str]]

Number of failures of each kind.

samples: dict[tuple[str, str, str], list[Failure]]

The first failures of each kind, in the order they happened.

capture(expt, step, mails=())

Reduce an error to a Failure, sampled while its kind has room left.

Return type:

Failure

add(failure)

Count a failure, and keep it if its kind still has room for a sample.

Return type:

None

headline()

The first line of the report, saying how much the failures set aside.

It also says that the run continued, as the word error reads to some users as a crash that stopped everything.

Return type:

str

title()

The title of the issue: that of the most frequent kind of failure.

Return type:

str

traces()

The full traceback of each sample, for the debug log.

Return type:

Iterator[str]

report_lines()

The report lines detailing each kind of failure, most frequent first.

Return type:

Iterator[str]

mail_deduplicate.bug_report.option_words(param, value)

Render one parameter as typed on the command line, with its values masked.

A path or a regular expression can name the folders and correspondents of the user, so each shows as a placeholder.

Return type:

list[str]

mail_deduplicate.bug_report.options_summary()

The parameters of the run that do not come from their default, masked.

The parameters of Click Extra keep no value of their own in the context, and none of them changes what mdedup does to a mail, except --jobs.

Return type:

str

mail_deduplicate.bug_report.mdedup_version()

The version of mdedup, as mdedup --version prints it.

Click Extra appends to a development version the short hash of its commit, which it reads from git, a hash baked in at build time, a git archive export, or the record of an install from a Git URL, like a uvx --from git+... run.

Return type:

str

mail_deduplicate.bug_report.report_url(title)

The URL of the bug report form, with its title filled in.

Return type:

str

mail_deduplicate.bug_report.render_report(headline, title, details, sources)

The message inviting the user to file a bug report, with the block to paste.

It links to the bug report form, with title filled in. The block opens with the version, the platform, the options and the formats of the sources, then gives the details of the error.

Return type:

str

mail_deduplicate.bug_report.save_report(report)

Write a bug report to a new file in the REPORTS_FOLDER of the cache.

A user who loses the output of a long run, to a closed terminal or a short scrollback, still finds the report there. The file holds the report as it is printed, with no path and no mail content, so it can be attached to an issue whole.

Each report gets a new file, named after the time it was saved. mkstemp() adds random characters to the name and never reuses one, so two reports of the same second, like a run’s failures and its statistics check, keep a file each.

Returns None if the file cannot be written: the printed report then stands alone.

Return type:

Path | None

mail_deduplicate.bug_report.log_report(level, headline, title, details, sources)

Log the report render_report() builds, and save it with save_report().

The path of the saved file comes last, alone on its line, so it is the last thing a user sees of the run, and easy to copy. It stays out of the file.

Return type:

None

mail_deduplicate.bug_report.blame(mail, step)

Attach the mail being processed to any error escaping the block.

The error goes on to stop the run, and report_unexpected_errors() names the mail. It also fingerprints the mail in the bug report, except for an OSError, which gets no bug report.

Return type:

Generator[None, None, None]

mail_deduplicate.bug_report.report_unexpected_errors(dedup)

End the run with a one-line error or a bug report, whatever escapes the block.

The exceptions of Click pass through: they are how a command exits, or refuses an input. An OSError comes from the system, like a missing permission or a full disk, so the user can fix it: it ends the run with a one-line error. Any other error is a bug. Its traceback is logged with the report inviting the user to file it, and the run exits with code 1.

Return type:

Generator[None, None, None]

mail_deduplicate.cache module

Persistent cache of mail hashes, so a second run does not re-parse unchanged mails.

Hashing dominates a run, and it is pure work: the same mail, hashed with the same settings, always produces the same result. Storing that result lets a later run skip opening and parsing every mail it has already seen. See: https://github.com/kdeldycke/mail-deduplicate/issues/87

Correctness rests on two independent guards:

• A fingerprint of every setting feeding the hash. Anything else invalidates the whole database, as no entry produced under different settings can be trusted.

• A per-mail staleness key of (size, mtime), taken from the file backing the mail before it is read, so a mail modified mid-run is re-hashed by the next one instead of being trusted.

mail_deduplicate.cache.HASHING_VERSION = '1'

Bumped whenever a change to the hashing, or to the size it memoizes, makes the entries of earlier versions wrong.

Such an entry reads as valid, and would group its mail differently from the mails hashed afresh: the whole database is discarded instead, as for a change of settings.

mail_deduplicate.cache.SCHEMA_VERSION = '1'

Bumped whenever the table layout changes, to discard databases of older shapes.

mail_deduplicate.cache.CACHED_SETTINGS = ('hash_headers', 'hash_body', 'raw_headers', 'time_source')

Configuration keys whose value changes what gets cached.

hash_headers, hash_body and raw_headers change the hash itself. time_source changes the timestamp memoized alongside it. The minimal-headers floor is derived from the number of hash headers, so it is covered by them.

class mail_deduplicate.cache.StaleKey(size, mtime_ns)

Bases: NamedTuple

Identifies the version of the file backing a mail.

Create new instance of StaleKey(size, mtime_ns)

size: int

Alias for field number 0

mtime_ns: int

Alias for field number 1

class mail_deduplicate.cache.CacheEntry(mail_hash, timestamp, mail_size)

Bases: NamedTuple

What is worth keeping about a hashed mail.

The hash is the point. The two scalars are what dehydrate() memoizes to spare the later steps a re-read, so restoring them is what actually lets a cached mail stay unparsed. mail_size is only known when the body was hashed, and timestamp is None for a mail whose Date header could not be parsed.

Create new instance of CacheEntry(mail_hash, timestamp, mail_size)

mail_hash: str

Alias for field number 0

timestamp: float | None

Alias for field number 1

mail_size: int | None

Alias for field number 2

mail_deduplicate.cache.default_cache_dir()

Location of the cache database, following each platform’s conventions.

See click_extra.carapace.get_cache_dir() for the location on each platform.

Return type:

Path

mail_deduplicate.cache.default_cache_path()

Full path of the cache database.

Return type:

Path

mail_deduplicate.cache.settings_fingerprint(conf)

Digest of every setting that changes what a cached entry would hold.

The two versions go in with the settings, so a database of an earlier release is discarded the same way when either moved.

Return type:

str

mail_deduplicate.cache.open_cache(conf)

Opens the hash cache the configuration asks for, if it asks for one at all.

A cache only ever saves work, so a database that cannot be opened, on a read-only or full filesystem for instance, is reported and skipped instead of taking the whole run down with it.

Return type:

HashCache | None

class mail_deduplicate.cache.HashCache(path, fingerprint)

Bases: object

A mail hash cache backed by SQLite, keyed by mail identity.

Opened for the whole run and committed once at the end, so an interrupted run leaves the database as it found it.

WRITE_BATCH: int = 512

Rows buffered before being handed to SQLite in one executemany().

Inserting one row at a time costs more in statement overhead than in storage.

LOCK_TIMEOUT: float = 30.0

Seconds a run waits for another one to release the database.

Runs sharing a database only ever contend on the single write burst each of them performs at the end, so waiting it out beats failing. See commit() for what happens when the wait is not enough.

lookup(box, mail_id)

Returns the entry cached for a mail, if it is still valid.

The staleness key is taken here, before the caller reads the mail, so a mail modified between this check and its hashing is caught by the next run rather than trusted.

Return type:

CacheEntry | None

store(source_path, mail_id, entry)

Record the hash of a freshly read mail, under the key taken by lookup().

Silently skipped for a mail that was never looked up, as there is then no staleness key that provably predates the read.

Return type:

None

prune()

Drops the entries of mails and boxes that are no longer there.

Two kinds of leftovers accumulate. A box that was deleted or moved keeps every one of its entries, and a mail deleted from a box that is still around keeps its own. Both are dropped here, and how many were removed is recorded as pruned for the run to report.

Only the boxes visited by this run are considered for their individual mails: every mail of a box that was not opened is missing from seen for the plain reason that nobody looked, which is not evidence that it is gone.

Return type:

None

forget(source_path, mail_id)

Drop the staleness key of a mail that will not be recorded.

Return type:

None

commit()

Flush everything recorded during the run, in one transaction.

Returns whether it went through. By this point the deduplication itself is done, so a database another run is holding, or a disk that filled up while we worked, costs the next run its head start and nothing more: it must not take down a run that has already produced its results.

Return type:

bool

close()

Release the database handle, without ever failing the run over it.

Return type:

None

mail_deduplicate.cli module

The mdedup command: its options, their validation and the run Config.

mail_deduplicate.cli.DEFAULT_HASH_HEADERS: tuple[str, ...] = ('Date', 'From', 'To', 'Subject', 'MIME-Version', 'Content-Type', 'Content-Disposition', 'User-Agent', 'X-Priority', 'Message-ID')

Default ordered list of headers to use to compute the unique hash of a mail.

By default we choose to exclude:

CC

Since mailman apparently sometimes trims list members from the CC header to avoid sending duplicates. Which means that copies of mail reflected back from the list server will have a different CC to the copy saved by the MUA at send-time.

BCC

Because copies of the mail saved by the MUA at send-time will have BCC, but copies reflected back from the list server won’t.

Reply-To

Since a mail could be CC’d to two lists with different Reply-To munging options set.

mail_deduplicate.cli.DEFAULT_MINIMAL_HEADERS = 4

Cap on the number of headers that must be present in a mail to compute a solid hash.

The per-mail floor is min(DEFAULT_MINIMAL_HEADERS, len(hash_headers)). A near-empty or corrupted mail whose hash would rest on too few headers gets no hash, and the run keeps it like a mail without duplicates. The floor relaxes automatically when --hash-header narrows the hash to fewer headers than this cap.

class mail_deduplicate.cli.Preset(*values)

Bases: StrEnum

Named sets of option values, each a starting point for a run.

A preset is an alias of the options PRESET_OPTIONS lists for it.

CONSERVATIVE = 'conservative'

Treats as copies only the mails whose bytes are identical.

DEFAULT = 'default'

Keeps the built-in defaults.

AGGRESSIVE = 'aggressive'

Trusts the Message-ID header alone to group copies.

mail_deduplicate.cli.PRESET_OPTIONS: dict[Preset, tuple[tuple[str, str | bool], ...]] = {Preset.AGGRESSIVE: (('--hash-header', 'Message-ID'),), Preset.CONSERVATIVE: (('--raw-headers', True), ('--hash-body', 'raw'), ('--size-threshold', '-1'), ('--content-threshold', '-1'), ('--strategy', 'select-one')), Preset.DEFAULT: ()}

The options each preset stands for, spelled as on the command line.

apply_preset() gives each of them its value, unless the user set that option. A value that relies on other values of the same preset applies only while all of them apply: see DEPENDENT_OPTIONS.

The conservative preset hashes the bytes of the whole mail, which only identical copies share. The thresholds then have nothing left to catch, and turning them off spares a second read of each mail. When the user replaces one of its two hashing options, copies that differ can share a hash, so the thresholds stay on. Identical copies are interchangeable, so the preset also keeps one of each.

The aggressive preset trusts Message-ID alone, so the copies whose other headers a mailing list or a gateway rewrote still share a hash. The thresholds stay the only check against two mails that reuse an ID. Its copies can differ, so which one to keep stays the user’s choice.

mail_deduplicate.cli.LINKED_OPTIONS: tuple[frozenset[str], ...] = (frozenset({'hash_headers', 'raw_headers'}),)

Options that make one choice between them, by their name.

--raw-headers hashes every header, so a --hash-header list has nothing left to select. The user setting either option decides the headers, so a preset value for the other one gives way too.

mail_deduplicate.cli.DEPENDENT_OPTIONS: dict[str, frozenset[str]] = {'content_threshold': frozenset({'hash_body', 'raw_headers'}), 'size_threshold': frozenset({'hash_body', 'raw_headers'})}

Options whose preset value relies on the preset values of others, by their name.

A preset value of such an option applies only while each option it relies on keeps the value of the same preset: the preset gives that value, or the user gives the same one. When one of them gives way, the option gives way too, and keeps its default or the user’s value. The rule ignores an option that the preset does not set.

The thresholds can stay off only while the hash covers every byte of a mail, which --raw-headers and --hash-body raw do together.

mail_deduplicate.cli.QUIET_OVERRIDES = frozenset({'strategies'})

Options the user can set over a preset with no warning, by their name.

The strategy decides which copy stays, not what counts as a copy.

mail_deduplicate.cli.preset_word(flag, value)

One option of a preset, as typed on the command line.

Return type:

str

mail_deduplicate.cli.preset_words(preset)

The options a preset stands for, as typed on the command line.

Return type:

str

class mail_deduplicate.cli.Config

Bases: TypedDict

Holds global configuration.

input_format: BoxFormat | None

force_unlock: bool

raw_headers: bool

hash_headers: tuple[str, ...]

minimal_headers: int

hash_body: BodyHasher

hash_only: bool

cache: bool

cache_path: Path | None

size_threshold: int

content_threshold: int

show_diff: bool

strategies: tuple[Strategy, ...]

time_source: TimeSource

regexp: Pattern | None

action: Action

export: Path | None

export_format: BoxFormat

export_append: bool

hardlink_differing: bool

dry_run: bool

mail_deduplicate.cli.normalize_headers(ctx, param, value)

Validate headers provided as parameters to the CLI.

Headers are case-insensitive in Python implementation, so we normalize them to lower-case.

We then deduplicate them, while preserving order.

Mail headers are expected to be composed of one or more ASCII characters between 33 and 126 (both inclusive) according to RFC-5322.

Return type:

tuple[str, ...]

mail_deduplicate.cli.unique_strategies(ctx, param, value)

Deduplicate strategies provided as parameters to the CLI, preserving order.

Strategies are deduplicated by the selection function they point to, so repeating a strategy already listed under one of its aliases is ignored too.

Return type:

tuple[Strategy, ...]

mail_deduplicate.cli.compile_regexp(ctx, param, value)

Validate and compile regular expression provided as parameters to the CLI.

Return type:

Pattern[str] | None

mail_deduplicate.cli.PATH_STRATEGIES = (Strategy.DISCARD_MATCHING_PATH, Strategy.DISCARD_NON_MATCHING_PATH, Strategy.SELECT_MATCHING_PATH, Strategy.SELECT_NON_MATCHING_PATH)

Strategies relying on the -r/--regexp parameter.

mail_deduplicate.cli.EXPORT_ACTIONS = (Action.COPY_SELECTED, Action.COPY_DISCARDED, Action.MOVE_SELECTED, Action.MOVE_DISCARDED)

Actions relying on the -E/--export parameter.

mail_deduplicate.cli.echo_step(title)

Print the heading of a pipeline step, after a blank line.

The blank line is printed on its own, outside the heading’s style. Styled as part of the heading, its escapes open the style on the line above the text, so any tool reading the output line by line (a pager, grep, a capture cut with --head) finds a style open on a line that closes it nowhere.

Parameters:

title (str) – the step’s name, after the bullet.

Return type:

None

mail_deduplicate.cli.human_join(items)

Renders IDs as an English enumeration, for the help screen.

Both tuples above are the single source of truth of which strategies and actions take an extra parameter, so the options mentioning them read the list from there instead of spelling it out again and drifting from it.

Return type:

str

class mail_deduplicate.cli.AnyValueIn(param_name, targets, label, negate=False)

Bases: Predicate

Condition that is true when any of a parameter’s values is in a target set.

The parameter’s resolved value is inspected, so defaults count, not only user-provided values. Single values and multiple=True tuples are normalized to a common shape. Set negate to invert the membership test: true when none of the values is in the target set.

Whatever negate says, the condition silently evaluates to false when no mail source is provided: the CLI then only prints its help screen and exits, so no parameter combination deserves a validation error.

The same applies in -H/--hash-only mode: selection and action steps never run there, so options they require (like --export for the default copy-selected action) must not be demanded. Options from those steps are separately reported as ignored by the hash-only code path.

description(ctx)

Succinct description of the predicate (alias: desc).

Return type:

str

class mail_deduplicate.cli.MdedupCommand(*args, version_fields=None, config_schema=None, config_strict=False, schema_strict=False, fallback_sections=(), config_validators=(), included_params=None, excluded_params=None, config_cascade=False, extra_option_at_end=True, option_priorities=None, populate_auto_envvars=True, extra_keywords=None, excluded_keywords=None, examples=(), **kwargs)

Bases: Command

Like cloup.Command.__init__, with Click Extra’s defaults and parameters.

Parameters:

• version_fields (dict[str, Any] | None) – dictionary of VersionOption template field overrides forwarded to the version option. Accepts any field from VersionOption.template_fields (like prog_name, version, git_branch). Lets you customize --version output from the command decorator without replacing the default params list.

• config_strict (bool) – forwarded to the default ConfigOption’s strict setting: configuration keys not matching any CLI parameter raise an error instead of being silently ignored. Like the other config_* and *_params forwards, it spares you from replacing the whole default params list to customize the config option.

• config_cascade (bool) – forwarded to the default ConfigOption’s cascade setting: auto-discovery layers every configuration file it finds, the most local one winning on key lookup, instead of stopping at the first.

• excluded_params (Sequence[str] | None) – additional parameter IDs to block from configuration files, merged into the default ConfigOption’s excluded_params blocklist. Additive, unlike the option-level excluded_params which replaces the default blocklist entirely. Items are fully-qualified parameter IDs (like mycli.mail_sources). Mutually exclusive with included_params.

• extra_keywords (HelpKeywords | None) – a HelpKeywords instance whose entries are merged into the auto-collected keyword set. Use this to inject additional strings for help screen highlighting.

• excluded_keywords (HelpKeywords | None) – a HelpKeywords instance whose entries are removed from the auto-collected keyword set. Use this to suppress highlighting of specific strings.

• examples (Sequence[Sequence[str]]) – a sequence of (description, command) string pairs showing the command in use. They are rendered in an Examples: section of the help screen, in the man page, and in every –help-format rendering. A malformed pair raises TypeError here, at command construction, rather than on the first --help a user runs.

• extra_option_at_end (bool) – reorders all parameters attached to the command, by moving all instances of ExtraOption at the end of the parameter list. The original order of the options is preserved among themselves.

• option_priorities (Mapping[str, float] | None) – maps an option to its priority in the help screen, relative to DEFAULT_PRIORITY, lowest shown first. Keys are matched against each parameter’s long and short flags first, then its destination name, so the --config / --no-config pair (which shares the config destination) stays addressable one flag at a time. Presentation only: self.params, and with it the order callbacks are evaluated in, is left alone. Positional arguments are never reordered, their sequence being part of the command’s grammar.

• populate_auto_envvars (bool) – forces all parameters to have their auto-generated environment variables registered. This addresses the shortcoming of click which only evaluates them dynamically. Once registered, the auto-generated environment variables get displayed in the help screen, fixing click#2483 issue. On Windows, environment variable names are case-insensitive, so they are normalized to uppercase.

By default, these Click context settings are applied:

• auto_envvar_prefix = self.name (Click feature)

  Auto-generate environment variables for all options, using the command ID as prefix. The prefix is normalized to be uppercased and all non-alphanumerics replaced by underscores.

• help_option_names = ("--help", "-h") (Click feature)

  Allow help screen to be invoked with either –help or -h options.

• show_default = True (Click feature)

  Show all default values in help screen.

Additionally, these Cloup context settings are set:

• align_option_groups = True (Cloup feature)

  Aligns option groups in help screen.

  Every command carries the sections of DEFAULT_OPTION_GROUPS, so leaving each to compute its own column width would step the help text left and right from one section to the next. Aligned, a command’s own options keep the column click-extra’s widest option sets, which is what the default_options_*_help fixtures of click_extra.pytest match against.

• show_constraints = True (Cloup feature)

  Show all constraints in help screen.

• show_subcommand_aliases = True (Cloup feature)

  Show all subcommand aliases in help screen.

Click Extra also adds its own context_settings:

• show_choices = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally show or hide choices when prompting a user for input. Only makes sense for options whose prompt property is set.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_choices setting.

• show_envvar = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally enable or disable the display of environment variables in help screen.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_envvar setting. The rationale being that discoverability of environment variables is enabled by the --params option, which is active by default on extra commands. So there is no need to surcharge the help screen.

  This addresses the click#2313 issue.

To override these defaults, you can pass your own settings with the context_settings parameter:

@command(
    context_settings={
        "show_default": False,
        ...
    }
)

format_help(ctx, formatter)

Extend the help screen with the description of all available strategies.

Return type:

None

mail_deduplicate.cli.flag_names(param)

Every spelling of an option, joined as the help screen lists them.

Return type:

str

mail_deduplicate.cli.user_set(ctx, name)

Whether the user gave a parameter its value, from any source but its default.

A value from a configuration file or the environment counts as set.

Return type:

bool

mail_deduplicate.cli.apply_preset(ctx, preset)

The values a preset gives the options the user left at their default.

An option the user set keeps its value. The run warns about each one that overrides the preset with another value, unless QUIET_OVERRIDES lists it. The options of a LINKED_OPTIONS group give way together. A value that relies on other values of the preset, as DEPENDENT_OPTIONS lists, gives way with any of them, and the run names the options that then keep their defaults.

Parameters:

• ctx (Context) – the context of the run, holding the values of its parameters.

• preset (Preset) – the preset to apply.

Return type:

dict[str, Any]

Returns:

the value of each option the preset sets, by parameter name.

mail_deduplicate.cli.ignored_step_options(ctx)

User-provided options from the steps -H/--hash-only never runs.

Collects the options attached to the sections of steps after #2 whose value does not come from a default, so the run can report them as ignored.

Return type:

list[str]

mail_deduplicate.deduplicate module

The deduplication pipeline: group mails by hash, settle each set, keep score.

The Deduplicate orchestrator drives the whole run. Both of its expensive steps, hashing and selection, can fan out across worker processes: the module-level _-prefixed functions are the halves that run inside a worker.

mail_deduplicate.deduplicate.step_progressbar(**kwargs)

A progress bar for one step of the run, drawn on a terminal only.

Off a terminal, Click prints a bar’s label once instead of the bar. Each step already opens on a heading saying the same thing, so in a log, a pipe or a capture the label is a bare line repeating it. Hiding the bar there drops that line, and leaves a terminal the bar --progress asks for.

The bar draws on sys.stderr, as click_extra.progressbar() makes it, unless file names another stream: that is the stream checked.

Parameters:

kwargs (Any) – what click_extra.progressbar() takes.

Return type:

ProgressBar[Any]

mail_deduplicate.deduplicate.path_order(mail)

Sort key ranking mails by path, then by ID among the mails of one file.

A set of mails iterates in an order that changes from one run to the next. Sorted by this key, the same mails come out in the same order on every run.

Return type:

tuple[str, str]

class mail_deduplicate.deduplicate.Stat(*values)

Bases: Enum

All tracked statistics.

The member’s name carries the category as its MAIL_/SET_ prefix, and its value the description shown in the final report.

MAIL_FOUND = 'Total number of mails encountered from all mail sources.'

MAIL_REJECTED = 'Number of mails rejected individually because they could not be read.'

MAIL_UNHASHED = 'Number of mails with too few of the hashed headers to compute a hash. They joined no duplicate set, and were kept in the final selection like unique mails.'

MAIL_FAILED = 'Number of mails set aside, and left untouched, because hashing them failed on an unexpected error. The bug report after this table tells how to report it.'

MAIL_RETAINED = 'Number of valid mails parsed and retained for deduplication.'

MAIL_HASHES = 'Number of unique hashes.'

MAIL_UNIQUE = 'Number of unique mails (which were automatically added to selection).'

MAIL_DUPLICATES = 'Number of duplicate mails (sum of mails in all duplicate sets with at least 2 mails).'

MAIL_SKIPPED = 'Number of mails ignored in the selection step because the whole set they belong to was skipped.'

MAIL_DISCARDED = 'Number of mails discarded from the final selection.'

MAIL_SELECTED = 'Number of mails kept in the final selection on which the action will be performed.'

MAIL_COPIED = 'Number of mails copied from their original mailbox to another.'

MAIL_MOVED = 'Number of mails moved from their original mailbox to another.'

MAIL_DELETED = 'Number of mails deleted from their mailbox in-place.'

MAIL_HARDLINKED = 'Number of mails replaced in-place by a hardlink to the copy kept in their duplicate set.'

MAIL_HARDLINK_SKIPPED = 'Number of mails left untouched by the hardlinking action, because they could not be linked to their copy or already were.'

SET_TOTAL = 'Total number of duplicate sets.'

SET_SINGLE = 'Total number of sets containing only a single mail with no applicable strategy. They were automatically kept in the final selection.'

SET_SKIPPED_SIZE = 'Number of sets skipped from the selection process because they were too dissimilar in size.'

SET_SKIPPED_CONTENT = 'Number of sets skipped from the selection process because they were too dissimilar in content.'

SET_SKIPPED_TIMESTAMP = 'Number of sets skipped from the selection process because a timestamp could not be derived for some of their mails.'

SET_SKIPPED_STRATEGY = 'Number of sets skipped from the selection process because the strategy could not be applied.'

SET_SKIPPED_ERROR = 'Number of sets skipped from the selection process because reading or comparing their mails failed on an error.'

SET_DEDUPLICATED = 'Number of valid sets on which the selection strategy was successfully applied.'

property description: str

The description of the statistic, shown in the final report.

property category: str

Whether the statistic counts mails or sets, read off the member’s name.

exception mail_deduplicate.deduplicate.SizeDiffAboveThreshold

Bases: Exception

Difference in mail size is greater than threshold.

exception mail_deduplicate.deduplicate.ContentDiffAboveThreshold

Bases: Exception

Difference in mail content is greater than threshold.

exception mail_deduplicate.deduplicate.MissingTimestamps

Bases: Exception

Some mails of a set have no timestamp for time-based strategies to compare.

Happens for mails without a parseable Date header, when the timestamp is sourced from it.

class mail_deduplicate.deduplicate.BodyHasher(*values)

Bases: StrEnum

Enumeration of available body hashing methods.

Each value interprets the body one step further than the one before it. skip leaves the body out. raw hashes its bytes as the box holds them, with line endings reduced to LF. decoded undoes the transfer encodings and decodes the text, so copies that differ in their encoding alone hash the same. normalized also strips all whitespace first.

SKIP = 'skip'

RAW = 'raw'

DECODED = 'decoded'

NORMALIZED = 'normalized'

property function: Callable[[DedupMailMixin], str]

The callable producing this member’s body hash for one mail.

class mail_deduplicate.deduplicate.DuplicateSet(hash_key, mail_set, conf)

Bases: object

A set of mails sharing the same hash.

Implements all the safety checks required before we can apply any selection strategy.

Load-up the duplicate set of mail and freeze pool.

Once loaded-up, the pool of parsed mails is considered frozen for the rest of the duplicate set’s life. This allows aggressive caching of lazy instance attributes depending on the pool content.

hash_key: str

selection: set[DedupMailMixin]

Mails selected after application of selection strategy.

unique: set[DedupMailMixin]

The subset of the selection no strategy ever ruled on.

A mail alone in its set has no copy to be compared to, so it is kept without any criterion picking it over another. Held apart from the rest of the selection so a destructive action can leave it alone. See: https://github.com/kdeldycke/mail-deduplicate/issues/1053

discard: set[DedupMailMixin]

Mails discarded after application of selection strategy.

conf

Configuration shared from the main deduplication process.

pool: frozenset[DedupMailMixin]

Pool referencing all duplicated mails and their attributes.

stats: Counter[Stat]

Set metrics. Unset statistics naturally read as zero.

property size: int

Returns the number of mails in the duplicate set.

property timestamps: tuple[float, ...]

Returns the timestamps of all mails in the set.

Raises MissingTimestamps if a timestamp could not be derived for some mails, naming them so users can locate and fix them. See: https://github.com/kdeldycke/mail-deduplicate/issues/132

property newest_timestamp: float

Returns the newest timestamp among all mails in the set.

property oldest_timestamp: float

Returns the oldest timestamp among all mails in the set.

property biggest_size: int

Returns the biggest size among all mails in the set.

property smallest_size: int

Returns the smallest size among all mails in the set.

check_differences()

Checks the mails of the set against each other for size and content.

Their differences must stay within the limits the thresholds set.

Instead of rejecting the whole set on the first offending pair, the mails involved in the most offending pairs are greedily set aside until every remaining pair passes the thresholds. This keeps a single outlier from preventing the deduplication of the true copies sharing its set. See: https://github.com/kdeldycke/mail-deduplicate/issues/851

Returns the mails to set aside, empty if the whole pool already passes.

Raises SizeDiffAboveThreshold or ContentDiffAboveThreshold if fewer than 2 mails would remain, in which case there is no coherent core of duplicates and the whole set is to be skipped, as before.

Return type:

set[DedupMailMixin]

diff(mail_a, mail_b)

Return the difference between two mails’ decoded body, in bytes.

The unified diff of the two bodies is measured in UTF-8, the unit of --content-threshold and of size.

Todo

Rewrite the diff algorithm to not rely on naive unified diff result parsing.

Return type:

int

pretty_diff(mail_a, mail_b)

Returns a verbose unified diff between two mails’ decoded body.

The body lines carry no line ending, so no diff line gets one either: they are joined with one instead.

Each line is styled by the active theme, the way git diff colors its own: a removed line as an error, an added one as a success, the hunk range as a range, and the two file headers as structure. The log handler strips the styling wherever color is off.

Return type:

str

skip_set(reason, stat)

Mark the entire set as skipped.

Return type:

None

select()

Settle which mails of the set are selected and which are discarded.

Run preliminary checks, then apply the strategies to the pool of mails, each in turn until one produces a proper selection.

The process results in two subsets of mails: the selected and the discarded.

Return type:

None

select_or_skip(failures)

Run select(), and skip the whole set if it raises an error.

Returns the error, ready for the bug report, or None when the selection went through or the error needs no report. See skip_on_error().

Return type:

Failure | None

skip_on_error(expt, pool, failures)

Skip the whole set after an error, as a set failing a safeguard is skipped.

Whatever the interrupted selection recorded is dropped, and the full pool is accounted for, so none of its mails is acted upon. An OSError means a mail could not be read again, which needs no bug report. Any other error is captured for the report and returned.

Return type:

Failure | None

class mail_deduplicate.deduplicate.HashedMail(mail_hash, timestamp, mail_size, rejection, failure=None, unhashed=None)

Bases: NamedTuple

What a hashing worker sends back for one mail.

Deliberately nothing but scalars, and the strings of a Failure for a mail that hit an unexpected error: the parsed message stays in the worker and dies with the task, so a few dozen bytes cross the process boundary instead of the whole mail. Which mail an answer belongs to is not carried either, as fan_out() hands every answer back alongside the task it came from. The parent rebuilds the same dehydrated stub it would have produced itself, exactly as it does for a mail restored from the cache.

Create new instance of HashedMail(mail_hash, timestamp, mail_size, rejection, failure, unhashed)

mail_hash: str | None

Alias for field number 0

timestamp: float | None

Alias for field number 1

mail_size: int | None

Alias for field number 2

rejection: str | None

Alias for field number 3

failure: Failure | None

Alias for field number 4

unhashed: str | None

Alias for field number 5

class mail_deduplicate.deduplicate.HashOutcome(mail, mail_hash, rejection=None, failure=None, unhashed=None)

Bases: NamedTuple

What the hashing step makes of one mail, ready to be merged into the run.

At most one of mail_hash, rejection, failure and unhashed is set: a mail is retained, rejected for a reason that needs no bug report, set aside after an error that does, or kept without a hash, as it has too few headers to compute one.

Create new instance of HashOutcome(mail, mail_hash, rejection, failure, unhashed)

mail: DedupMailMixin

Alias for field number 0

mail_hash: str | None

Alias for field number 1

rejection: str | None

Alias for field number 2

failure: Failure | None

Alias for field number 3

unhashed: str | None

Alias for field number 4

mail_deduplicate.deduplicate.unreadable(expt)

Words the rejection of a mail whose file the system refused to read.

Return type:

str

class mail_deduplicate.deduplicate.MailMeta(source_path, mail_id, path, timestamp, mail_size)

Bases: NamedTuple

All a worker needs to rebuild one mail of a duplicate set from its own file.

Create new instance of MailMeta(source_path, mail_id, path, timestamp, mail_size)

source_path: str

Alias for field number 0

mail_id: str

Alias for field number 1

path: str

Alias for field number 2

timestamp: float | None

Alias for field number 3

mail_size: int | None

Alias for field number 4

class mail_deduplicate.deduplicate.SelectedSet(selected, unique, discarded, stats, records, failure=None)

Bases: NamedTuple

What a selection worker sends back for one duplicate set.

Mails are named rather than returned: the parent already holds them, and shipping them back would mean pickling every message it worked so hard not to keep.

Create new instance of SelectedSet(selected, unique, discarded, stats, records, failure)

selected: tuple[tuple[str, str], ...]

Alias for field number 0

unique: tuple[tuple[str, str], ...]

Alias for field number 1

discarded: tuple[tuple[str, str], ...]

Alias for field number 2

stats: dict[Stat, int]

Alias for field number 3

records: tuple[tuple[int, str], ...]

Alias for field number 4

failure: Failure | None

Alias for field number 5

class mail_deduplicate.deduplicate.Deduplicate(conf)

Bases: object

Load mails, search for duplicates, apply the strategies and perform the action.

Similar messages sharing the same hash are grouped together in a DuplicateSet.

CHUNK_SIZE: int = 200

Mails handed to a hashing worker in one go.

Each queue round-trip costs far more than hashing a single mail, so tasks travel in chunks. Large enough to make that cost disappear, small enough that the last worker to finish does not hold up the others.

SET_CHUNK_SIZE: int = 32

Duplicate sets handed to a selection worker in one go.

Smaller than the hashing chunk because a set is several mails’ worth of work, so fewer of them already amortize the same round-trip.

sources: dict[str, Mailbox]

Index of mail sources by their full, normalized path. So we can refer to them in Mail instances. Also have the nice side effect of natural deduplication of sources themselves.

mails: dict[str, list[DedupMailMixin]]

All mails grouped by hashes.

Grouped in lists rather than sets: mails carry no value equality, so a set only ever deduplicated by object identity, which the single pass over each box already rules out. A one-element list also costs 64 bytes where a set costs 216, and most hashes group a single mail.

selection: set[DedupMailMixin]

Mails selected after application of selection strategy.

unique: set[DedupMailMixin]

The subset of the selection no strategy ever ruled on.

It holds the mails that have no duplicate, which each duplicate set contributes here: see DuplicateSet.unique. It also holds the mails with too few headers to be hashed, which join no duplicate set at all.

discard: set[DedupMailMixin]

Mails discarded after application of selection strategy.

link_targets: dict[DedupMailMixin, DedupMailMixin]

Maps each discarded mail to the selected mail it can be hardlinked to.

Left empty unless the configured action consumes it: see track_link_targets.

conf

Configuration shared across the deduplication process.

track_link_targets: bool

Whether each discarded mail has to be paired with a selected one.

Only the hardlinking action needs that pairing, and it costs one dictionary entry per discarded mail, so it is settled once here and only recorded when something reads it.

stats: Counter[Stat]

Deduplication statistics. Unset statistics naturally read as zero.

failures

The errors no rule explains, each of which put a mail or a duplicate set aside.

step

What the run is doing, named by the bug report of an error that stops it.

source_formats: Counter[str]

Number of boxes opened in each format, for the bug report.

Counted as the boxes open, since closing them empties sources.

cache: HashCache | None

Cross-run cache of mail hashes, when the user opted in with --cache and the database could be opened.

restore_cached(box, mail_id, entry)

Rebuild a hashed mail from its cache entry, without opening its file.

Produces the same dehydrated stub the hashing step would have left behind, with the scalars the later steps rely on already memoized. Anything else those steps need is re-read from the box on demand, as for any other mail.

Return type:

HashOutcome

blank_stub(box, mail_id)

An empty mail carrying only its identity, ready to be filled in.

Stands in for a mail this process never parsed, because its content came from the cache or from a worker that has already thrown it away.

Return type:

DedupMailMixin

parallel_boxes()

The sources whose mails a worker process could hash on its own.

Only folder-based boxes qualify. Their mails each own a file, which a worker opens by path, sharing nothing. Mails of a file-based box are byte ranges of one file that a single handle seeks through, so handing them out would mean several processes seeking the same descriptor.

Returns an empty list unless every source qualifies, so a run is either wholly parallel or wholly sequential rather than silently half of each.

Return type:

list[Mailbox]

property jobs: int

Worker processes the --jobs option resolved to, read off the context.

worker_pool(jobs, initializer, initargs, verb, doing)

Hand out a pool of worker processes prepared by the initializer.

The pool shuts down however the block it wraps ends.

Yields None instead when the pool cannot be started, which is the case in some frozen or sandboxed environments: the initializer is then run right here, so the worker functions find the state they expect, and the caller degrades to sequential calls instead of dying.

Return type:

Generator[ProcessPoolExecutor | None, None, None]

fan_out(pool, worker, tasks, window, chunksize)

Yields each task back alongside its worker’s answer.

Tasks pair what the parent keeps with the payload its worker receives, and are handed out a window at a time rather than all at once: map() consumes whatever it is given immediately, so passing the whole corpus would hold a task per mail, undoing the flat memory the rest of the run maintains. The window is wide enough that every worker always has chunks queued behind it.

With or without a pool, answers come back in submission order, so whatever the caller aggregates comes out identical to a sequential run at any number of workers.

Return type:

Iterator[tuple]

uncached(boxes, absorb, progress)

Yields the identity of every mail the cache cannot answer for.

A mail the cache can restore is absorbed here and never even opened, which is the whole point of keeping the cache. Lazy, and driven from the main process alone: neither box objects nor the cache are safe for concurrent access.

Return type:

Iterator[tuple[Mailbox, str]]

hash_in_parallel(jobs, absorb, progress)

Hash every uncached mail across a pool of worker processes.

Threads cannot do this job: what hashing spends itself on is Python-level work that the interpreter lock serializes, so fanning it out across threads only adds contention. Processes sidestep the lock, and the mail never has to travel: a worker opens its own file and sends back a hash and two scalars.

Return type:

None

adopt_hashed(box, mail_id, result)

Turn a worker’s answer back into the mail stub this process works with.

Return type:

HashOutcome

add_source(source_path)

Registers a source of mails, validates and opens it.

Duplicate sources of mails are not allowed, as when we perform the action, we use the path as a unique key to tie back a mail from its source.

A source that cannot be opened or listed raises MailSourceError: the error comes from the source or its format, which the user can fix.

Return type:

None

hash_all()

Read the mails of all sources, compute their hashes and group them by hash.

Displays a progress bar as the operation might be slow.

Each mail is dehydrated as soon as it is hashed, so whatever the size of the corpus, only a lightweight stub of every mail is retained. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

Hashing fans out across worker processes when --jobs resolves above 1 and every source is a folder-based box; otherwise mails stream through this process one at a time, which is also the lowest-memory path. Box listing and the hash cache always stay in this process, as neither is safe for concurrent access, so the speedup is largest where the hashing itself is the cost, with --hash-body decoded/normalized.

Return type:

None

build_sets()

Build the selected and discarded sets from each duplicate set.

The selection is settled one duplicate set at a time, to keep the memory footprint low and make the log easier to read.

Return type:

None

settle(hash_key, mail_set)

Settle one set in this process, and merge its verdict into the run.

The set goes through the thresholds and the strategies.

Return type:

None

log_set_heading(hash_key, mail_count)

Announce a duplicate set, at a level reflecting whether it holds copies.

Styling the heading is not free, and most sets hold a single mail and so report at debug level, where the result is thrown away: only pay for it when it is actually logged.

Return type:

None

release(mail_set)

Drop the content the thresholds and strategies pulled back from the boxes.

Iterating on the original set covers skipped sets and set-aside mails too, which land in neither selection nor discard. See: https://github.com/kdeldycke/mail-deduplicate/issues/362

Return type:

None

record_link_targets(selection, discard)

Pair every discarded mail of a set with a selected mail of that same set.

Hardlinking a discarded mail only makes sense against a copy that survives the very set both were found in, and that pairing is the one thing the flat selection and discard sets no longer say once every set has been settled.

A strategy is free to keep several mails, so the target is the one with the lowest path: a run then links to the same copy however the sets came back, which matters when the selection is spread over a pool of processes.

Iterables are only walked once the run is known to need them, so a call from a non-hardlinking run costs nothing beyond the arguments themselves.

Return type:

None

select_in_parallel(jobs, progress)

Apply the selection to every duplicate set across a pool of processes.

Duplicate sets share nothing with one another, so a set is the natural unit of work. Only sets holding copies are handed out: a set of one is settled without reading anything, and would cost more to ship than to decide.

Return type:

None

describe(mail)

Everything a worker needs to rebuild a mail, and nothing more.

Return type:

MailMeta

adopt_selection(hash_key, mail_set, result)

Merge a worker’s verdict on one set back into this process.

The heading and the worker’s own messages are said here, in that order, so the log reads as it would have from a sequential run however the sets were spread out.

Return type:

None

close_all(flush=True)

Close all open boxes, and the hash cache if one was opened.

With flush off, the pending changes of the boxes stay unwritten: see release_box(). Closed boxes and cache are dropped, so calling this again does nothing.

Return type:

None

report()

Returns a text report of user-friendly statistics and metrics.

Return type:

str

report_failures()

Log the bug report of the mails and sets the run set aside.

Returns whether there were any, so the run can end with an error code.

Return type:

bool

assert_stats(first, operator, second)

Render failed stats assertions in plain English.

Statistics that do not add up are a bug: the run logs a bug report and exits with code 115.

Return type:

None

check_stats()

Perform some high-level consistency checks on metrics.

Helps users reports tricky edge-cases.

mail_deduplicate.mail module

A mail wrapped with the properties deduplication needs.

They are the canonical hash, the normalized headers, the timestamp, the size, and the dehydration that keeps memory flat.

exception mail_deduplicate.mail.TooFewHeaders

Bases: Exception

Not enough headers were found to produce a solid hash.

class mail_deduplicate.mail.TimeSource(*values)

Bases: StrEnum

Enumeration of all supported mail timestamp sources.

DATE_HEADER = 'date-header'

Timestamp sourced from the message’s Date header.

CTIME = 'ctime'

Timestamp is from the email’s file on the filesystem.

Attention

Only meaningful for sources storing one mail per file, like maildir and eml.

mail_deduplicate.mail.ADDRESS_HEADERS = frozenset({'bcc', 'cc', 'delivered-to', 'disposition-notification-to', 'envelope-to', 'from', 'original-recipient', 'reply-to', 'resent-bcc', 'resent-cc', 'resent-from', 'resent-reply-to', 'resent-sender', 'resent-to', 'return-path', 'sender', 'to', 'x-envelope-from', 'x-envelope-to', 'x-original-to'})

Headers that contain email addresses.

Hint

Headers from which quotes should be discarded, so "Bob" <bob@example.com> hashes to the same thing as Bob <bob@example.com>.

Attention

These IDs should be kept lower-case, because they are compared to the IDs provided to the -h/--hash-header option, carried by the hash_headers entry of the configuration.

mail_deduplicate.mail.HEADER_LINE = re.compile(b'\\A(From |[!-9;-~]*:|[\\t ])')

A line of the header block, as the parser reads one: an envelope line, a header or a folded continuation.

The parser ends the headers on the first blank line, or on the first line that is none of these when the blank line is missing. split_raw() follows the same rule.

mail_deduplicate.mail.LINE = re.compile(b'[^\\r\\n]*[\\r\\n]|[^\\r\\n]+\\Z')

One line of a mail with LF line endings, with its line break when it has one.

A bare CR also ends a line, as it does for the parser.

class mail_deduplicate.mail.DedupMailMixin(message=None)

Bases: Message

Message with deduplication-specific properties and utilities.

Extends standard library’s mailbox.Message, and shouldn’t be used directly, but composed with mailbox.Message sub-classes.

Initialize a Message instance.

CONTENT_CACHES: tuple[str, ...] = ('body_lines', 'canonical_headers', 'hash_decoded_body', 'hash_normalized_body', 'hash_raw_body')

Memoized properties derived from the message content.

They are only needed while computing hashes, and are dropped by dehydrate() alongside the parsed message itself.

resolve_path: Callable[[Mailbox, str], str]

Derives the real filesystem location of a mail from its box.

Set per box format by make_dedup_mail(), as folder-based boxes give each mail its own file while file-based ones pack them all into the box’s single file.

defects: list = ()

Fallback for the parsing defects dropped by dehydrate().

An empty tuple rather than a list, so it can be shared by every dehydrated mail instead of costing one throw-away empty list each: reads still work, and the appends only the parser performs fail loudly on a mail that has no message.

PARSED_MESSAGE_ATTRS: tuple[str, ...] = ('_headers', '_payload', '_unixfrom', 'preamble', 'epilogue', 'defects')

Attributes carrying the parsed message, as populated by email.parser.

They hold the bulk of a mail’s memory footprint, and are the ones dropped by dehydrate() and restored by hydrate().

box: Mailbox | None

The box this message was read from, kept to re-fetch its content on demand after dehydration.

source_path: str | None

Normalized path to the mailbox this message originates from.

mail_id: str | None

Mail ID used to uniquely refers to it in the context of its source.

conf: Config

Global configuration

property path: str

Real filesystem location of the mail.

Returns the individual mail’s file for folder-based box types (maildir & co.), but returns the whole box path for file-based boxes (mbox & co.). Used by regexp-based selection strategies and to render the mail’s repr.

Derived on access from the box rather than stored, so a retained mail does not carry a copy of its own absolute path for the whole run. Raises AttributeError before the box metadata is attached, so getattr(mail, “path”, None) still reads as absent.

add_box_metadata(box, mail_id)

Attach to the mail some metadata derived from its parent box.

Called right after the __init__() constructor.

This allows the mail to carry its own information on its origin box and index.

Return type:

None

property is_hydrated: bool

Whether the mail still carries its parsed message.

dehydrate()

Reduce the mail to the lightweight metadata needed by the next steps.

Memoizes the scalar properties consumed by selection strategies (timestamp, and size when the decoded body is at hand), then drops the parsed message and every cached copy of its content. This cuts the resident footprint of a retained mail from the full size of its message to a few hundred bytes, whatever the number of mails processed. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

No-op if the mail is already dehydrated. The dropped content is re-read from the source box by hydrate() when a later step needs it again.

Return type:

None

hydrate()

Restore the full parsed message dropped by dehydrate().

Re-reads the mail from its source box, through the same parsing path that produced it in the first place, so the restored content is identical. No-op if the mail still carries its parsed message.

Return type:

None

raw_bytes()

Read the bytes of the mail again, from its own file or from its box.

The parser consumed them without keeping them. A mail a worker opened by path is read from that file. A mail of the main process is read from its box, which hands it out without the envelope line and the state its format keeps beside each mail.

Return type:

bytes

hydrated()

Borrow the full parsed message for the duration of the block.

Restores the message on the way in and releases it on the way out, so a step needing the whole mail after the hashing one keeps its content resident only while it uses it, and memory stays flat across a loop of mails.

Return type:

Generator[DedupMailMixin, None, None]

property parsed_date: float | None

Parse the mail’s date header into float timestamp.

Returns None if the mail has no valid date header, or one whose year no timestamp can hold.

Self-hydrating: re-reads the message from its box if it was dehydrated.

property timestamp: float | None

Compute the normalized canonical timestamp of the mail.

Sourced from the message’s Date header by default. In the case of maildir, can be sourced from the email’s file from the filesystem.

Warning

ctime does not refer to creation time on POSIX systems, but rather the time of the last metadata change.

Todo

Investigate what mailbox.MaildirMessage.get_date() does and if we can use it.

property size: int

Returns the canonical size of the mail, in bytes.

The size is the byte length, in UTF-8, of the decoded body: the payload of the mail stripped of its headers, its transfer encodings undone and its text decoded, never the size of the mail on disk. Copies of a mail that differ in their transfer encoding alone measure the same, and --size-threshold compares them in the unit it names.

Todo

Allow customization of the way the size is computed, by getting the file size instead, for example with os.path.getsize(mail_file).

property body_lines: list[str]

Return the lines of the decoded body of the message.

Each text part has its transfer encoding undone and its text decoded with its charset, so copies of a mail that differ in their encoding alone read the same. A part that is not text is kept as the mail carries it. The preamble and epilogue of a multipart are read as UTF-8.

Self-hydrating: re-reads the message from its box if it was dehydrated.

decode_part(part)

Decode a single message part to string.

Return type:

str

undo_surrogateescape(text, errors='replace')

Decode as UTF-8 the non-ASCII bytes that parsing left as lone surrogates.

A mail parsed from bytes keeps each non-ASCII byte as a lone surrogate, and encoding such a string to UTF-8 raises UnicodeEncodeError. get_payload() undoes this for each part, with the charset of the part. The preamble and the epilogue have no charset, so they go through here instead. Bytes that are not valid UTF-8 become U+FFFD, as get_payload() does with its charset, unless errors names another way out, like backslashreplace.

Encoding with utf-8 instead of ascii keeps the non-ASCII characters of a mail parsed from a string, which ascii rejects.

Return type:

str

hash_key()

Returns the canonical hash of a mail.

Under --raw-headers, the hash is that of the header block, as raw_headers() reads it: every header takes part, so no minimal number of them applies.

Caution

This method hasn’t been made explicitly into a cached property in order to reduce the overall memory footprint.

Return type:

str

split_raw()

The bytes of the mail as the box holds them, cut into headers and body.

Line endings are reduced to LF, as add_mail() writes them: every box format rewrites them to the separator of the platform, and the mbox reader reduces them on Windows, so a mail would otherwise never hash like its own export, and a hash would depend on the platform. Nothing else is interpreted. A header a server or a client added, an mbox >From quote and a transfer encoding all count.

The header block ends with the first blank line, which it keeps. When that blank line is missing, it ends before the first line that is neither a header nor a continuation, where the parser starts the body too. A mail of headers alone has an empty body. The two halves always join back into the whole mail, so --raw-headers with --hash-body raw hashes every byte.

Return type:

tuple[bytes, bytes]

raw_headers()

The header block of the mail, as split_raw() cuts it.

Return type:

bytes

raw_body()

The body bytes of the mail, as split_raw() cuts them.

Return type:

bytes

property hash_raw_body: str

Returns the hash of the body bytes, as the box holds them.

Nothing but the line endings is interpreted, see raw_body(): a transfer encoding, a charset, a bare CR and an mbox >From quote all count, where hash_decoded_body reads through the first two.

property hash_decoded_body: str

Returns the hash of the decoded body of a mail, line by line.

property hash_normalized_body: str

Returns the hash of the decoded body of a mail, stripped of whitespace.

property canonical_headers: tuple[tuple[str, str], ...]

Returns the canonical header names and values, ready for hashing.

Under --raw-headers, it returns every header instead, in its order and unnormalized, for --hash-only to show. The hash itself reads the bytes of the header block: see raw_headers().

Self-hydrating: re-reads the message from its box if it was dehydrated.

pretty_canonical_headers()

Renders a table of headers names and values used to produce the mail’s hash.

Caution

This method hasn’t been explicitly made into a cached property in order to reduce the overall memory footprint.

Returns a string ready to be printed.

Return type:

str

serialized_headers()

Serialize the canonical headers into a single string ready to be hashed.

At this point we should have an absolute minimum of headers.

Caution

This method hasn’t been explicitly made into a cached property in order to reduce the overall memory footprint.

Return type:

bytes

header_text(value)

The text of one header value, as the mail holds it.

A header holding raw 8-bit bytes is read as those bytes: decoded as UTF-8 where they are valid UTF-8, and escaped byte by byte elsewhere. get() and get_all() hand such a header over as a Header, which string functions like email.utils.parsedate_tz() reject. Its text also replaces every such byte with U+FFFD, under which two headers differing in those bytes alone read, and hash, alike.

Return type:

str

header_values(header_id)

Returns the text of each occurrence of a header, in the order of the mail.

Each value reads as header_text() reads it.

Return type:

Iterator[str]

normalized_header_values(header_id)

Returns all normalized values of a header.

Values are cleaned-up into their canonical form, from the text header_values() reads.

Return type:

Iterator[str]

normalize_subject(subject)

Strip Re:/Fwd: and [list-name] prefixes from Subject.

This cleans up prefixes automatically added by mailing list software, since the mail could have been CC’d to multiple lists, in which case it will receive a different prefix for each.

Return type:

str

normalize_content_type(value)

Normalize Content-Type by stripping parameters.

Removes everything after the semicolon, keeping only the MIME type. E.g., text/plain; charset=utf-8 becomes text/plain.

Apparently list servers actually munge Content-Type e.g. by stripping the quotes from charset="us-ascii". Section 5.1 of RFC2045 says that either form is valid (and they are equivalent).

Additionally, with multipart/mixed, boundary delimiters can vary by recipient. We need to allow for duplicates coming from multiple recipients, since for example you could be signed up to the same list twice with different addresses. Or maybe someone bounces you a load of mail some of which is from a mailing list you’re both subscribed to - then it’s still useful to be able to eliminate duplicates.

Return type:

str

normalize_date(value)

Normalize Date to YYYY-MM-DD format.

Date timestamps can differ by seconds or hours for various reasons, so only the date is honoured, normalized to the UTC timezone.

Todo

Revisit the day-level granularity, and whether the time of the day should take part in the hash.

Return type:

str

normalize_address_header(value)

Normalize address headers by removing quotes and collapsing whitespace.

E.g., "Bob" <bob@example.com> becomes Bob <bob@example.com>.

Remove quotes in any headers that contain addresses to ensure a quoted name is hashed to the same value as an unquoted one.

Danger

This may not be the cleanest way to normalize email addresses. E.g. "Robert \"Bob\"" becomes Robert \Bob\, but this shouldn’t matter for hashing purposes as we’re just trying to get a good heuristic. Refs: #846 and #847.

Return type:

str

normalize_message_id(value)

Normalize Message-ID header by stripping angle brackets.

E.g., <unique-id@example.com> becomes unique-id@example.com.

Return type:

str

strip_angle_brackets(value)

Strip angle brackets from a value if it’s a single bracketed item.

Only strips if the value matches <something> with no commas.

Note

Sometimes email.parser strips the <> brackets from a To: header which has a single address. I have seen this happen for only one mail in a duplicate pair. I’m not sure why (presumably the parser uses email.utils.unquote somewhere in its code path which was only triggered by that mail and not its sister mail), but to be safe, we should always strip the <> brackets to avoid this difference preventing duplicate detection.

Return type:

str

mail_deduplicate.mail_box module

Utilities to read and write mail boxes in various formats.

Based on Python’s standard library mailbox module.

exception mail_deduplicate.mail_box.MailSourceError(message)

Bases: ClickException

A mail source that cannot be used as given.

Raised for a problem the user can fix: a folder with no structure mdedup recognizes, a box another program locked, a source given twice. Click prints it as a one-line Error:, with no traceback and no invitation to report a bug.

mail_deduplicate.mail_box.maildir_subpath(box, key)

Path of a maildir mail below its box, read from the table of contents.

A maildir key drops the :2,<flags> suffix the file name carries, so the name cannot be rebuilt from the key alone. The box refreshed its table of contents to hand out the mail in the first place, so it is only re-validated on a miss.

Return type:

str

mail_deduplicate.mail_box.maildir_mail_path(box, key)

Location of a maildir mail, read from the box’s table of contents.

Return type:

str

mail_deduplicate.mail_box.keyed_mail_path(box, key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

mail_deduplicate.mail_box.box_file_path(box, key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

mail_deduplicate.mail_box.iter_mail_ids(box)

Yields the key of every mail held by a box.

Maildir.iterkeys() confirms that each key still resolves to a file, one stat per mail, on top of the directory listing it has just built. Every caller here goes on to stat or open that same file anyway, and copes with its disappearance, so the check is paid for twice and needed once: reading the refreshed table of contents directly skips it.

The other formats list their mails without that extra round, and are left to their own iterator.

Return type:

Iterator[str]

mail_deduplicate.mail_box.resolve_mail_path(box, key)

Location of a mail in its box, without instantiating the mail.

Lets a caller reach a mail’s file before deciding to read it, which is how the hash cache checks whether a mail changed without paying for its parsing.

Return type:

str

mail_deduplicate.mail_box.make_dedup_mail(name, base, path_resolver)

Create a DedupMail class for a mailbox message type.

Deriving a mail’s own location from its box is format-specific, so the resolver is baked into the class here instead of being branched on at runtime.

Return type:

type

class mail_deduplicate.mail_box.MaildirDedupMail(message=None)

Bases: DedupMailMixin, MaildirMessage

Initialize a MaildirMessage instance.

static resolve_path(box, key)

Location of a maildir mail, read from the box’s table of contents.

Return type:

str

class mail_deduplicate.mail_box.mboxDedupMail(message=None)

Bases: DedupMailMixin, mboxMessage

Initialize an mboxMMDFMessage instance.

static resolve_path(box, key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.MHDedupMail(message=None)

Bases: DedupMailMixin, MHMessage

Initialize an MHMessage instance.

static resolve_path(box, key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

class mail_deduplicate.mail_box.BabylDedupMail(message=None)

Bases: DedupMailMixin, BabylMessage

Initialize a BabylMessage instance.

static resolve_path(box, key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.MMDFDedupMail(message=None)

Bases: DedupMailMixin, MMDFMessage

Initialize an mboxMMDFMessage instance.

static resolve_path(box, key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.EMLDedupMail(message=None)

Bases: DedupMailMixin, Message

Initialize a Message instance.

static resolve_path(box, key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

class mail_deduplicate.mail_box.EML(dirname, factory=None, create=True)

Bases: Mailbox

A folder of loose .eml files, walked recursively.

Supports mail archives exported as individual RFC 5322 files, one mail per file, as produced by Outlook PST/OST conversion tools for instance. See: https://github.com/kdeldycke/mail-deduplicate/issues/760

Keys are the paths of the mail files, relative to the folder’s root. Files without the .eml extension (case-insensitive) are ignored, as well as hidden files and directories.

Follows the interface of Python’s mailbox.Mailbox. Like maildir, the one-file-per-mail storage needs no locking.

Initialize a Mailbox instance.

iterkeys()

Return an iterator over keys.

get_file(key)

Return a file-like representation or raise a KeyError.

get_bytes(key)

Return a byte string representation or raise a KeyError.

Return type:

bytes

get_message(key)

Return a Message representation or raise a KeyError.

add(message)

Add message and return assigned key.

Return type:

str

remove(key)

Remove the keyed message; raise KeyError if it doesn’t exist.

Return type:

None

list_folders()

No subfolder objects: the recursive walk covers nested directories.

Return type:

list[str]

get_folder(folder)

flush()

Mails are written straight to the filesystem: nothing to flush.

Return type:

None

lock()

One-file-per-mail storage needs no locking.

Return type:

None

unlock()

One-file-per-mail storage needs no locking.

Return type:

None

close()

No resource is kept open between operations.

Return type:

None

class mail_deduplicate.mail_box.BoxStructure(*values)

Bases: Enum

Box structures can be file-based or folder-based.

FOLDER = 1

FILE = 2

class mail_deduplicate.mail_box.BoxFormat(*values)

Bases: Enum

IDs of all the supported box formats and their metadata.

Each entry is associated to:

• their original base class,

• the structure they implement (file-based or folder-based),

• the custom message factory class to use.

From these, we can derive the proper constructor with our own custom DedupMail factory.

Todo

Add support for other mailbox formats and sources, like Gmail accounts, IMAP servers, etc.

MAILDIR = (<class 'mailbox.Maildir'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.MaildirDedupMail'>)

MBOX = (<class 'mailbox.mbox'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.mboxDedupMail'>)

MH = (<class 'mailbox.MH'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.MHDedupMail'>)

BABYL = (<class 'mailbox.Babyl'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.BabylDedupMail'>)

MMDF = (<class 'mailbox.MMDF'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.MMDFDedupMail'>)

EML = (<class 'mail_deduplicate.mail_box.EML'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.EMLDedupMail'>)

property constructor

Return a constructor for this box format with our custom message factory.

mail_deduplicate.mail_box.FOLDER_FORMATS = (BoxFormat.MAILDIR, BoxFormat.MH, BoxFormat.EML)

Box formats implementing a folder-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.FILE_FORMATS = (BoxFormat.MBOX, BoxFormat.BABYL, BoxFormat.MMDF)

Box formats implementing a file-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.FOLDER_FORMAT_CLASSES = (<class 'mailbox.Maildir'>, <class 'mailbox.MH'>, <class 'mail_deduplicate.mail_box.EML'>)

Base classes of folder-based box formats, as a tuple ready for isinstance.

mail_deduplicate.mail_box.MAILDIR_SUBDIRS = frozenset({'cur', 'new', 'tmp'})

List of required sub-folders defining a properly structured maildir.

mail_deduplicate.mail_box.MBOX_FAMILY = (<class 'mailbox.mbox'>, <class 'mailbox.MMDF'>)

Formats keeping an envelope From line before each mail.

The standard library reads and writes both with the same code, so the line travels from one to the other.

mail_deduplicate.mail_box.is_maildir(path)

Returns True when the path holds all the sub-directories of a maildir.

Return type:

bool

mail_deduplicate.mail_box.contains_maildir(path)

Returns True when the path is a maildir or holds one at any depth.

Allows the discovery of nested maildir folders stored as plain directories, as produced by isync/mbsync’s Verbatim naming style. See: https://github.com/kdeldycke/mail-deduplicate/issues/973

Dot-prefixed directories are ignored, as they are covered by the Maildir++ folder convention. The mail-holding sub-directories of maildirs are not explored either.

Return type:

bool

mail_deduplicate.mail_box.contains_eml(path)

Returns True when the path holds at least one .eml file, at any depth.

Hidden files and directories are ignored, and the extension is matched case-insensitively, mirroring the walk of EML boxes.

Return type:

bool

mail_deduplicate.mail_box.autodetect_box_type(path)

Auto-detect the format of the mailbox located at the provided path.

If the path is a file, then it is considered as an mbox. Else, if the provided path is a folder and features the MAILDIR_SUBDIRS sub-directories, or holds nested maildir folders at any depth, it is parsed as a maildir. A folder holding loose .eml files instead is parsed as an eml source.

Todo

Future finer autodetection heuristics should be implemented here. Some ideas:

• single mail from a maildir

• plain text mail content

• other mailbox formats supported in Python’s standard library:

  • MH

  • Babyl

  • MMDF

Return type:

BoxFormat

mail_deduplicate.mail_box.open_box(path, box_format=None, force_unlock=False)

Open a mail box.

Returns a list of boxes, one per sub-folder. All are locked, ready for operations.

If box_format is provided, forces the opening of the box in the specified format. Else, defaults to autodetection.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.lock_box(box, force_unlock)

Lock an opened box and allows for forced unlocking.

Returns the locked box. Raises MailSourceError if another lock holds the box and force_unlock is not set.

Return type:

Mailbox

mail_deduplicate.mail_box.release_box(box)

Unlock a box and close its file, without writing its pending changes.

Mailbox.close() first writes the pending changes of a single-file box, which rewrites the whole file when a mail was removed. A run cut short by an error or an interrupt calls this instead: its single-file boxes stay as the run found them, and an interrupt does not start a rewrite.

Unlike MH.close(), this also closes the .mh_sequences file that a refused MH.lock() leaves open.

Return type:

None

mail_deduplicate.mail_box.open_subfolders(box, force_unlock)

Browse recursively the subfolder tree of a box.

Returns a list of opened and locked boxes, each for one subfolder. If a subfolder cannot be locked, because another program holds it for instance, the boxes locked before it are unlocked again and the error is raised: the next run then needs no --force-unlock for them.

Skips box types not supporting subfolders. For maildir, both the Maildir++ convention (dot-prefixed folders) and Verbatim-style layouts (nested plain directories, each a maildir of its own) are browsed. A directory without the maildir structure only acts as a container of nested folders and carries no mail of its own.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.carry_maildir_metadata(source, key, target, new_key)

Give a copied maildir mail the folder, flags and time of its original.

A maildir keeps the state of a mail outside of its content: the new or cur folder says whether a client saw it, the :2, suffix of its file name holds its flags, and the modification time of the file is its delivery time, which IMAP servers like Dovecot serve as the internal date of the mail. Maildir.add() takes them from a MaildirMessage alone, so a mail added as bytes lands in new with no flags and the time of the run: it is renamed and re-dated afterwards.

Return type:

None

mail_deduplicate.mail_box.carry_mh_sequences(source, key, target, new_key)

Enroll a copied MH mail in the sequences of its original.

MH keeps the state of a mail (unseen, replied, flagged) in the .mh_sequences file of its folder, by the number of the mail. MH.add() takes them from an MHMessage alone, so they are written here for a mail added as bytes, the way MH.add() writes them.

Return type:

None

mail_deduplicate.mail_box.add_mail(box, mail)

Add a mail to a box, as the bytes its source box holds.

Every box format writes a mail handed over as a message with BytesGenerator, from the message parsed out of it. The generator does not write the mail back: it rebuilds it. It redraws each multipart boundary, closes one left open, turns a bare \r into a line break, and cannot write at all a multipart part parsed with no subpart, which python/cpython#136686 tracks. Deduplicating a mail is no reason to repair it, so every mail goes in as bytes, which each format writes as they are. See: https://github.com/kdeldycke/mail-deduplicate/issues/1090

Line endings are the one conversion left. They are reduced to LF before the write, as every box format but eml writes \n as the line separator of the platform, and would double the CR of a CRLF ending on Windows. An mbox or MMDF box also quotes the body lines starting with From, as its format requires.

The bytes do not include what a format keeps beside each mail: the envelope From line of an mbox or MMDF mail, the folder, flags and modification time of a maildir mail, the sequences of an MH mail, the labels of a Babyl mail. The first three travel with the mail into a box of the same format, and the envelope line also travels between mbox and MMDF, which keep it alike. A box of another format has no place for them, and nothing is translated: a maildir mail copied into an mbox gets no Status header. Babyl labels are not carried, as the standard library takes them from a BabylMessage alone.

Return type:

None

mail_deduplicate.mail_box.create_box(path, box_format, export_append=False)

Creates a brand new box from scratch.

Return type:

Mailbox

mail_deduplicate.strategy module

Strategy definitions.

mail_deduplicate.strategy.SelectionFunc: TypeAlias

mail_deduplicate.strategy.log_selection(message_template)

Decorator to log selection criteria.

mail_deduplicate.strategy.select_older(duplicates)

Select all older duplicates.

Discards the newests, i.e. the subset sharing the most recent timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_oldest(duplicates)

Select all the oldest duplicates.

Discards the newers, i.e. all mail of the duplicate set but those sharing the oldest timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_newer(duplicates)

Select all newer duplicates.

Discards the oldest, i.e. the subset sharing the most ancient timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_newest(duplicates)

Select all the newest duplicates.

Discards the olders, i.e. all mail of the duplicate set but those sharing the newest timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_smaller(duplicates)

Select all smaller duplicates.

Discards the biggests, i.e. the subset sharing the biggest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_smallest(duplicates)

Select all the smallest duplicates.

Discards the biggers. i.e. all mail of the duplicate set but those sharing the smallest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_bigger(duplicates)

Select all bigger duplicates.

Discards the smallests, i.e. the subset sharing the smallest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_biggest(duplicates)

Select all the biggest duplicates.

Discards the smallers, i.e. all mail of the duplicate set but those sharing the biggest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_matching_path(duplicates)

Select all duplicates whose file path matches the –regexp expression.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_non_matching_path(duplicates)

Select all duplicates whose file path doesn’t match the –regexp expression.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.first_by_path(duplicates)

The mail of a set that select_one() keeps and select_all_but_one() drops.

Neither strategy compares the mails, so the order of their paths decides: the order the action trail lists them in. Both strategies then rule on the same mail at every run. A dry run names the copy that the real run acts on, and a --jobs worker picks the copy that the main process would.

Return type:

DedupMailMixin

mail_deduplicate.strategy.select_one(duplicates)

Select the duplicate whose path sorts first, and discard all others.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_all_but_one(duplicates)

Discard the duplicate whose path sorts first, and select all others.

Return type:

set[DedupMailMixin]

class mail_deduplicate.strategy.Strategy(*values)

Bases: Enum

Selection strategies to apply on a set of duplicate mails.

Each strategy in the Enum points to the function implementing the selection logic, by way of the function property.

Strategies whose member value is a string are simply aliases to other strategies, pointing to the name of the function implementing the logic. The other members have integer values, to indicate their function ID is to be derived from the member name. This arrangement lets each member exist on its own instead of being hidden by the aliasing mechanism of Enum.

Aliases are great usability features to represent inverse operations. They help users reason about the selection operators in whichever direction matches their mental model.

Note

True Enum aliases (DISCARD_NEWEST = SELECT_OLDER) were considered and turned down. An alias is then the very same object as its canonical member, so it cannot be told apart at runtime, and three properties go away:

1. str() stops round-tripping the spelling the user typed: a run started with --strategy=discard-newest would report select-older in its logs.

2. PATH_STRATEGIES in cli.py collapses from four members to two, so the --regexp help line and the AnyValueIn error message would name only half the strategies that require it.

3. tests/test_strategy.py feeds f"--strategy={strategy_id}", so pairs like [SELECT_OLDER, DISCARD_NEWEST] would stringify identically and the alias spellings would no longer be exercised end to end.

Todo

Revisit the aliasing arrangement once the Python floor reaches 3.13, whose Enum._add_alias_() accepts non-identifier names, so SELECT_OLDER._add_alias_("discard-newest") puts the CLI spelling straight into __members__, where EnumChoice(Strategy, choice_source="name", show_aliases=True) picks it up. EnumChoice also grew a transform argument (click-extra 8.9.0) to kebab-case the raw identifiers that show_aliases otherwise yields. Neither addresses the three losses listed in the note above, which is why the current arrangement stands.

SELECT_OLDER = 1

SELECT_OLDEST = 2

SELECT_NEWER = 3

SELECT_NEWEST = 4

DISCARD_NEWEST = 'select_older'

DISCARD_NEWER = 'select_oldest'

DISCARD_OLDEST = 'select_newer'

DISCARD_OLDER = 'select_newest'

SELECT_SMALLER = 5

SELECT_SMALLEST = 6

SELECT_BIGGER = 7

SELECT_BIGGEST = 8

DISCARD_BIGGEST = 'select_smaller'

DISCARD_BIGGER = 'select_smallest'

DISCARD_SMALLEST = 'select_bigger'

DISCARD_SMALLER = 'select_biggest'

SELECT_MATCHING_PATH = 9

SELECT_NON_MATCHING_PATH = 10

DISCARD_NON_MATCHING_PATH = 'select_matching_path'

DISCARD_MATCHING_PATH = 'select_non_matching_path'

SELECT_ONE = 11

SELECT_ALL_BUT_ONE = 12

DISCARD_ALL_BUT_ONE = 'select_one'

DISCARD_ONE = 'select_all_but_one'

property function: Callable[[DuplicateSet], set[DedupMailMixin]]

The selection function this member stands for.

Alias members carry the function’s name as their value; canonical members derive it from their own name.

apply(duplicates)

Perform the selection strategy on the provided duplicate set.

Returns the set of selected mail objects.

Return type:

set[DedupMailMixin]