mail_deduplicate package

Expose package-wide elements.

class mail_deduplicate.StrEnum(new_class_name, /, names, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: str, ReprEnum

Enum where members are also (and must be) strings

Submodules

mail_deduplicate.action module

Actions performed once the selection is settled: copy, move, delete or hardlink.

Each action ID pairs an operation verb with the subset of mails it applies to, and Action.perform() routes one to the other.

mail_deduplicate.action.export_box(dedup)

Context manager for export box operations.

Return type:

Iterator[Mailbox | None]

mail_deduplicate.action.dry_run_prefix(dedup)

Marks a summary as describing what a run would have done.

A dry run reports through the same trail as a real one, so what it did not do is said once at the end rather than warned about for every single mail.

Return type:

str

mail_deduplicate.action.copy_mails(dedup, mails)

Copy provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.move_mails(dedup, mails)

Move provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.delete_mails(dedup, mails)

Remove provided mails in-place, from their original boxes.

Return type:

None

mail_deduplicate.action.LINK_TEMP_PREFIX = '.mdedup-hardlink-'

Prefix of the temporary link a mail is replaced through.

Every folder-based format skips dot-prefixed files when listing its mails, so a temporary left behind by an interrupted run is never read back as one.

mail_deduplicate.action.has_own_file(mail)

Whether the mail is backed by a file holding it alone.

File-based boxes pack all their mails into the box’s single file, which is what path returns for each of them: there would be nothing to link but the whole box.

Return type:

bool

mail_deduplicate.action.hardlink_blocker(dedup, mail, target)

Explain why a mail cannot be replaced by a hardlink to its target.

Returns the reason as a sentence ready to be logged, or None when the link can go ahead. Nothing is touched along the way, so a dry run reaches the same verdicts a real run acts on.

Return type:

str | None

mail_deduplicate.action.replace_by_hardlink(mail_path, target_path)

Point a mail’s own path at the file backing another mail.

The link is created under a temporary name in the mail’s own directory, then renamed over it: the rename is atomic and shares the mail’s filesystem by construction, so the mail is never missing from its box, whatever interrupts the run. The mail also keeps its file name, which is where maildir records the per-folder flags that make the same mail read in one folder and unread in another.

Return type:

None

mail_deduplicate.action.hardlink_mails(dedup, mails)

Replace provided mails in-place by a hardlink to the copy kept in their set.

The mails stay right where they are, under their own name: only the content they are backed by is shared with the copy that survived the selection, so the disk space they took is reclaimed.

Return type:

None

mail_deduplicate.action.OPERATIONS: dict[str, Callable[[Deduplicate, Collection[DedupMailMixin]], None]] = {'copy': <function copy_mails>, 'delete': <function delete_mails>, 'hardlink': <function hardlink_mails>, 'move': <function move_mails>}

The operation functions above, keyed by the verb half of an action ID.

All share the same signature: the deduplication they report to, and the mails they apply to.

class mail_deduplicate.action.Action(*values)

Bases: StrEnum

Define all available action IDs.

An action ID joins an operation verb to the subset of mails it applies to: the *-selected actions act on the mails kept by the selection, the *-discarded ones on the mails it discarded.

COPY_SELECTED = 'copy-selected'

COPY_DISCARDED = 'copy-discarded'

MOVE_SELECTED = 'move-selected'

MOVE_DISCARDED = 'move-discarded'

DELETE_SELECTED = 'delete-selected'

DELETE_DISCARDED = 'delete-discarded'

HARDLINK_DISCARDED = 'hardlink-discarded'

property verb: str

The operation half of the action ID, keying into OPERATIONS.

property acts_on_discarded: bool

Whether the action applies to the discarded mails rather than the selected ones.

targets(dedup)

The subset of mails this action applies to.

Return type:

set[DedupMailMixin]

perform(dedup)

Perform the action on the subset of mails it targets.

Return type:

None

mail_deduplicate.cache module

Persistent cache of mail hashes, so a second run does not re-parse unchanged mails.

Hashing dominates a run, and it is pure work: the same mail, hashed with the same settings, always produces the same result. Storing that result lets a later run skip opening and parsing every mail it has already seen. See: https://github.com/kdeldycke/mail-deduplicate/issues/87

Correctness rests on two independent guards:

• A fingerprint of every setting feeding the hash. Anything else invalidates the whole database, as no entry produced under different settings can be trusted.

• A per-mail staleness key of (size, mtime), taken from the file backing the mail before it is read, so a mail modified mid-run is re-hashed by the next one instead of being trusted.

mail_deduplicate.cache.SCHEMA_VERSION = '1'

Bumped whenever the table layout changes, to discard databases of older shapes.

mail_deduplicate.cache.CACHED_SETTINGS = ('hash_headers', 'hash_body', 'time_source')

Configuration keys whose value changes what gets cached.

hash_headers and hash_body change the hash itself. time_source changes the timestamp memoized alongside it. The minimal-headers floor is derived from the number of hash headers, so it is covered by the first one.

class mail_deduplicate.cache.StaleKey(size: int, mtime_ns: int)

Bases: NamedTuple

Identifies the version of the file backing a mail.

Create new instance of StaleKey(size, mtime_ns)

size: int

Alias for field number 0

mtime_ns: int

Alias for field number 1

class mail_deduplicate.cache.CacheEntry(mail_hash: str, timestamp: float | None, mail_size: int | None)

Bases: NamedTuple

What is worth keeping about a hashed mail.

The hash is the point. The two scalars are what dehydrate() memoizes to spare the later steps a re-read, so restoring them is what actually lets a cached mail stay unparsed. mail_size is only known when the body was hashed, and timestamp is None for a mail whose Date header could not be parsed.

Create new instance of CacheEntry(mail_hash, timestamp, mail_size)

mail_hash: str

Alias for field number 0

timestamp: float | None

Alias for field number 1

mail_size: int | None

Alias for field number 2

mail_deduplicate.cache.default_cache_dir()

Location of the cache database, following each platform’s conventions.

~/Library/Caches/mdedup on macOS, $XDG_CACHE_HOME/mdedup on other POSIX systems, and %LOCALAPPDATA%\mdedup\Cache on Windows.

appauthor is turned off because there is no vendor to namespace under: leaving it unset would have Windows fall back to the application name and nest the cache one level deeper, under mdedup\mdedup\Cache.

Return type:

Path

mail_deduplicate.cache.default_cache_path()

Full path of the cache database.

Return type:

Path

mail_deduplicate.cache.settings_fingerprint(conf)

Digest of every setting that changes what a cached entry would hold.

Return type:

str

mail_deduplicate.cache.open_cache(conf)

Opens the hash cache the configuration asks for, if it asks for one at all.

A cache only ever saves work, so a database that cannot be opened, on a read-only or full filesystem for instance, is reported and skipped instead of taking the whole run down with it.

Return type:

HashCache | None

class mail_deduplicate.cache.HashCache(path, fingerprint)

Bases: object

A mail hash cache backed by SQLite, keyed by mail identity.

Opened for the whole run and committed once at the end, so an interrupted run leaves the database as it found it.

WRITE_BATCH: int = 512

Rows buffered before being handed to SQLite in one executemany().

Inserting one row at a time costs more in statement overhead than in storage.

LOCK_TIMEOUT: float = 30.0

Seconds a run waits for another one to release the database.

Runs sharing a database only ever contend on the single write burst each of them performs at the end, so waiting it out beats failing. See commit() for what happens when the wait is not enough.

lookup(box, mail_id)

Returns the entry cached for a mail, if it is still valid.

The staleness key is taken here, before the caller reads the mail, so a mail modified between this check and its hashing is caught by the next run rather than trusted.

Return type:

CacheEntry | None

store(source_path, mail_id, entry)

Record the hash of a freshly read mail, under the key taken by lookup().

Silently skipped for a mail that was never looked up, as there is then no staleness key that provably predates the read.

Return type:

None

prune()

Drops the entries of mails and boxes that are no longer there.

Two kinds of leftovers accumulate. A box that was deleted or moved keeps every one of its entries, and a mail deleted from a box that is still around keeps its own. Both are dropped here, and how many were removed is recorded as pruned for the run to report.

Only the boxes visited by this run are considered for their individual mails: every mail of a box that was not opened is missing from seen for the plain reason that nobody looked, which is not evidence that it is gone.

Return type:

None

forget(source_path, mail_id)

Drop the staleness key of a mail that will not be recorded.

Return type:

None

commit()

Flush everything recorded during the run, in one transaction.

Returns whether it went through. By this point the deduplication itself is done, so a database another run is holding, or a disk that filled up while we worked, costs the next run its head start and nothing more: it must not take down a run that has already produced its results.

Return type:

bool

close()

Release the database handle, without ever failing the run over it.

Return type:

None

mail_deduplicate.cli module

The mdedup command: its options, their validation, and the Config mapping carried through the whole run.

mail_deduplicate.cli.DEFAULT_HASH_HEADERS: tuple[str, ...] = ('Date', 'From', 'To', 'Subject', 'MIME-Version', 'Content-Type', 'Content-Disposition', 'User-Agent', 'X-Priority', 'Message-ID')

Default ordered list of headers to use to compute the unique hash of a mail.

By default we choose to exclude:

CC

Since mailman apparently sometimes trims list members from the CC header to avoid sending duplicates. Which means that copies of mail reflected back from the list server will have a different CC to the copy saved by the MUA at send-time.

BCC

Because copies of the mail saved by the MUA at send-time will have BCC, but copies reflected back from the list server won’t.

Reply-To

Since a mail could be CC’d to two lists with different Reply-To munging options set.

mail_deduplicate.cli.DEFAULT_MINIMAL_HEADERS = 4

Cap on the number of headers that must be present in a mail to compute a solid hash.

The per-mail floor is min(DEFAULT_MINIMAL_HEADERS, len(hash_headers)): it rejects near-empty or corrupted mails whose hash would rest on too few headers, while relaxing automatically when the hash is narrowed to fewer headers than this cap via --hash-header.

class mail_deduplicate.cli.Config

Bases: TypedDict

Holds global configuration.

input_format: BoxFormat | None

force_unlock: bool

hash_headers: tuple[str, ...]

minimal_headers: int

hash_body: BodyHasher

hash_only: bool

cache: bool

cache_path: Path | None

size_threshold: int

content_threshold: int

show_diff: bool

strategies: tuple[Strategy, ...]

time_source: TimeSource

regexp: Pattern | None

action: Action

export: Path | None

export_format: BoxFormat

export_append: bool

hardlink_differing: bool

dry_run: bool

mail_deduplicate.cli.normalize_headers(ctx, param, value)

Validate headers provided as parameters to the CLI.

Headers are case-insensitive in Python implementation, so we normalize them to lower-case.

We then deduplicate them, while preserving order.

Mail headers are expected to be composed of ASCII characters between 33 and 126 (both inclusive) according to RFC-5322.

Return type:

tuple[str, ...]

mail_deduplicate.cli.unique_strategies(ctx, param, value)

Deduplicate strategies provided as parameters to the CLI, preserving order.

Strategies are deduplicated by the selection function they point to, so repeating a strategy already listed under one of its aliases is ignored too.

Return type:

tuple[Strategy, ...]

mail_deduplicate.cli.compile_regexp(ctx, param, value)

Validate and compile regular expression provided as parameters to the CLI.

Return type:

Pattern[str] | None

mail_deduplicate.cli.PATH_STRATEGIES = (Strategy.DISCARD_MATCHING_PATH, Strategy.DISCARD_NON_MATCHING_PATH, Strategy.SELECT_MATCHING_PATH, Strategy.SELECT_NON_MATCHING_PATH)

Strategies relying on the -r/--regexp parameter.

mail_deduplicate.cli.EXPORT_ACTIONS = (Action.COPY_SELECTED, Action.COPY_DISCARDED, Action.MOVE_SELECTED, Action.MOVE_DISCARDED)

Actions relying on the -E/--export parameter.

mail_deduplicate.cli.human_join(items)

Renders IDs as an English enumeration, for the help screen.

Both tuples above are the single source of truth of which strategies and actions take an extra parameter, so the options mentioning them read the list from there instead of spelling it out again and drifting from it.

Return type:

str

class mail_deduplicate.cli.AnyValueIn(param_name, targets, label, negate=False)

Bases: Predicate

Condition that is true when any of a parameter’s values is in a target set.

The parameter’s resolved value is inspected, so defaults count, not only user-provided values. Single values and multiple=True tuples are normalized to a common shape. Set negate to invert the membership test: true when none of the values is in the target set.

Whatever negate says, the condition silently evaluates to false when no mail source is provided: the CLI then only prints its help screen and exits, so no parameter combination deserves a validation error.

The same applies in -H/--hash-only mode: selection and action steps never run there, so options they require (like --export for the default copy-selected action) must not be demanded. Options from those steps are separately reported as ignored by the hash-only code path.

description(ctx)

Succinct description of the predicate (alias: desc).

Return type:

str

class mail_deduplicate.cli.MdedupCommand(*args, version_fields=None, config_schema=None, config_strict=False, schema_strict=False, fallback_sections=(), config_validators=(), included_params=None, excluded_params=None, extra_option_at_end=True, populate_auto_envvars=True, extra_keywords=None, excluded_keywords=None, **kwargs)

Bases: Command

List of extra parameters:

Parameters:

• version_fields (dict[str, Any] | None) – dictionary of VersionOption template field overrides forwarded to the version option. Accepts any field from VersionOption.template_fields (like prog_name, version, git_branch). Lets you customize --version output from the command decorator without replacing the default params list.

• config_strict (bool) – forwarded to the default ConfigOption’s strict setting: configuration keys not matching any CLI parameter raise an error instead of being silently ignored. Like the other config_* and *_params forwards, it spares you from replacing the whole default params list to customize the config option.

• excluded_params (Sequence[str] | None) – additional parameter IDs to block from configuration files, merged into the default ConfigOption’s excluded_params blocklist. Additive, unlike the option-level excluded_params which replaces the default blocklist entirely. Items are fully-qualified parameter IDs (like mycli.mail_sources). Mutually exclusive with included_params.

• extra_keywords (HelpKeywords | None) – a HelpKeywords instance whose entries are merged into the auto-collected keyword set. Use this to inject additional strings for help screen highlighting.

• excluded_keywords (HelpKeywords | None) – a HelpKeywords instance whose entries are removed from the auto-collected keyword set. Use this to suppress highlighting of specific strings.

• extra_option_at_end (bool) – reorders all parameters attached to the command, by moving all instances of ExtraOption at the end of the parameter list. The original order of the options is preserved among themselves.

• populate_auto_envvars (bool) – forces all parameters to have their auto-generated environment variables registered. This address the shortcoming of click which only evaluates them dynamically. By forcing their registration, the auto-generated environment variables gets displayed in the help screen, fixing click#2483 issue. On Windows, environment variable names are case-insensitive, so we normalize them to uppercase.

By default, these Click context settings are applied:

• auto_envvar_prefix = self.name (Click feature)

  Auto-generate environment variables for all options, using the command ID as prefix. The prefix is normalized to be uppercased and all non-alphanumerics replaced by underscores.

• help_option_names = ("--help", "-h") (Click feature)

  Allow help screen to be invoked with either –help or -h options.

• show_default = True (Click feature)

  Show all default values in help screen.

Additionally, these Cloup context settings are set:

• align_option_groups = False (Cloup feature)

  Aligns option groups in help screen.

• show_constraints = True (Cloup feature)

  Show all constraints in help screen.

• show_subcommand_aliases = True (Cloup feature)

  Show all subcommand aliases in help screen.

Click Extra also adds its own context_settings:

• show_choices = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally show or hide choices when prompting a user for input. Only makes sense for options whose prompt property is set.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_choices setting.

• show_envvar = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally enable or disable the display of environment variables in help screen.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_envvar setting. The rationale being that discoverability of environment variables is enabled by the --params option, which is active by default on extra commands. So there is no need to surcharge the help screen.

  This addresses the click#2313 issue.

To override these defaults, you can pass your own settings with the context_settings parameter:

@command(
    context_settings={
        "show_default": False,
        ...
    }
)

format_help(ctx, formatter)

Extend the help screen with the description of all available strategies.

Return type:

None

mail_deduplicate.cli.ignored_step_options(ctx)

User-provided options from the steps -H/--hash-only never runs.

Collects the options attached to the sections of steps after #2 whose value does not come from a default, so the run can report them as ignored.

Return type:

list[str]

mail_deduplicate.deduplicate module

The deduplication pipeline: group mails by hash, settle each duplicate set, keep score.

The Deduplicate orchestrator drives the whole run. Both of its expensive steps, hashing and selection, can fan out across worker processes: the module-level _-prefixed functions are the halves that run inside a worker.

class mail_deduplicate.deduplicate.Stat(*values)

Bases: Enum

All tracked statistics.

The member’s name carries the category as its MAIL_/SET_ prefix, and its value the description shown in the final report.

MAIL_FOUND = 'Total number of mails encountered from all mail sources.'

MAIL_REJECTED = 'Number of mails rejected individually because they were unparsable or did not have enough metadata to compute hashes.'

MAIL_RETAINED = 'Number of valid mails parsed and retained for deduplication.'

MAIL_HASHES = 'Number of unique hashes.'

MAIL_UNIQUE = 'Number of unique mails (which were automatically added to selection).'

MAIL_DUPLICATES = 'Number of duplicate mails (sum of mails in all duplicate sets with at least 2 mails).'

MAIL_SKIPPED = 'Number of mails ignored in the selection step because the whole set they belong to was skipped.'

MAIL_DISCARDED = 'Number of mails discarded from the final selection.'

MAIL_SELECTED = 'Number of mails kept in the final selection on which the action will be performed.'

MAIL_COPIED = 'Number of mails copied from their original mailbox to another.'

MAIL_MOVED = 'Number of mails moved from their original mailbox to another.'

MAIL_DELETED = 'Number of mails deleted from their mailbox in-place.'

MAIL_HARDLINKED = 'Number of mails replaced in-place by a hardlink to the copy kept in their duplicate set.'

MAIL_HARDLINK_SKIPPED = 'Number of mails left untouched by the hardlinking action, because they could not be linked to their copy or already were.'

SET_TOTAL = 'Total number of duplicate sets.'

SET_SINGLE = 'Total number of sets containing only a single mail with no applicable strategy. They were automatically kept in the final selection.'

SET_SKIPPED_ENCODING = 'Number of sets skipped from the selection process because they had encoding issues.'

SET_SKIPPED_SIZE = 'Number of sets skipped from the selection process because they were too dissimilar in size.'

SET_SKIPPED_CONTENT = 'Number of sets skipped from the selection process because they were too dissimilar in content.'

SET_SKIPPED_TIMESTAMP = 'Number of sets skipped from the selection process because a timestamp could not be derived for some of their mails.'

SET_SKIPPED_STRATEGY = 'Number of sets skipped from the selection process because the strategy could not be applied.'

SET_DEDUPLICATED = 'Number of valid sets on which the selection strategy was successfully applied.'

property description: str

The description of the statistic, shown in the final report.

property category: str

Whether the statistic counts mails or sets, read off the member’s name.

exception mail_deduplicate.deduplicate.SizeDiffAboveThreshold

Bases: Exception

Difference in mail size is greater than threshold.

exception mail_deduplicate.deduplicate.ContentDiffAboveThreshold

Bases: Exception

Difference in mail content is greater than threshold.

exception mail_deduplicate.deduplicate.MissingTimestamps

Bases: Exception

Some mails of a duplicate set have no timestamp, so they cannot be compared by time-based strategies.

Happens for mails without a parseable Date header, when the timestamp is sourced from it.

class mail_deduplicate.deduplicate.BodyHasher(*values)

Bases: StrEnum

Enumeration of available body hashing methods.

SKIP = 'skip'

RAW = 'raw'

NORMALIZED = 'normalized'

property function: Callable[[DedupMailMixin], str]

The callable producing this member’s body hash for one mail.

class mail_deduplicate.deduplicate.DuplicateSet(hash_key, mail_set, conf)

Bases: object

A set of mails sharing the same hash.

Implements all the safety checks required before we can apply any selection strategy.

Load-up the duplicate set of mail and freeze pool.

Once loaded-up, the pool of parsed mails is considered frozen for the rest of the duplicate set’s life. This allows aggressive caching of lazy instance attributes depending on the pool content.

hash_key: str

selection: set[DedupMailMixin]

Mails selected after application of selection strategy.

discard: set[DedupMailMixin]

Mails discarded after application of selection strategy.

conf

Configuration shared from the main deduplication process.

pool: frozenset[DedupMailMixin]

Pool referencing all duplicated mails and their attributes.

stats: Counter[Stat]

Set metrics. Unset statistics naturally read as zero.

property size: int

Returns the number of mails in the duplicate set.

property timestamps: tuple[float, ...]

Returns the timestamps of all mails in the set.

Raises MissingTimestamps if a timestamp could not be derived for some mails, naming them so users can locate and fix them. See: https://github.com/kdeldycke/mail-deduplicate/issues/132

property newest_timestamp: float

Returns the newest timestamp among all mails in the set.

property oldest_timestamp: float

Returns the oldest timestamp among all mails in the set.

property biggest_size: int

Returns the biggest size among all mails in the set.

property smallest_size: int

Returns the smallest size among all mails in the set.

check_differences()

Checks all mails of the set against each other, for size and content differences within the limits imposed by the thresholds.

Instead of rejecting the whole set on the first offending pair, the mails involved in the most offending pairs are greedily set aside until every remaining pair passes the thresholds. This keeps a single outlier from preventing the deduplication of the true copies sharing its set. See: https://github.com/kdeldycke/mail-deduplicate/issues/851

Returns the mails to set aside, empty if the whole pool already passes.

Raises SizeDiffAboveThreshold or ContentDiffAboveThreshold if fewer than 2 mails would remain, in which case there is no coherent core of duplicates and the whole set is to be skipped, as before.

Return type:

set[DedupMailMixin]

diff(mail_a, mail_b)

Return difference in bytes between two mails’ normalized body.

Todo

Rewrite the diff algorithm to not rely on naive unified diff result parsing.

Return type:

int

pretty_diff(mail_a, mail_b)

Returns a verbose unified diff between two mails’ normalized body.

Return type:

str

skip_set(reason, stat)

Mark the entire set as skipped.

Return type:

None

select()

Settle which mails of the set are selected and which are discarded.

Run preliminary checks, then apply the strategies to the pool of mails, each in turn until one produces a proper selection.

The process results in two subsets of mails: the selected and the discarded.

Return type:

None

class mail_deduplicate.deduplicate.HashedMail(mail_hash: str | None, timestamp: float | None, mail_size: int | None, rejection: str | None)

Bases: NamedTuple

What a hashing worker sends back for one mail.

Deliberately nothing but scalars: the parsed message stays in the worker and dies with the task, so a few dozen bytes cross the process boundary instead of the whole mail. Which mail an answer belongs to is not carried either, as fan_out() hands every answer back alongside the task it came from. The parent rebuilds the same dehydrated stub it would have produced itself, exactly as it does for a mail restored from the cache.

Create new instance of HashedMail(mail_hash, timestamp, mail_size, rejection)

mail_hash: str | None

Alias for field number 0

timestamp: float | None

Alias for field number 1

mail_size: int | None

Alias for field number 2

rejection: str | None

Alias for field number 3

class mail_deduplicate.deduplicate.MailMeta(source_path: str, mail_id: str, path: str, timestamp: float | None, mail_size: int | None)

Bases: NamedTuple

All a worker needs to rebuild one mail of a duplicate set from its own file.

Create new instance of MailMeta(source_path, mail_id, path, timestamp, mail_size)

source_path: str

Alias for field number 0

mail_id: str

Alias for field number 1

path: str

Alias for field number 2

timestamp: float | None

Alias for field number 3

mail_size: int | None

Alias for field number 4

class mail_deduplicate.deduplicate.SelectedSet(selected: tuple[tuple[str, str], ...], discarded: tuple[tuple[str, str], ...], stats: dict[Stat, int], records: tuple[tuple[int, str], ...])

Bases: NamedTuple

What a selection worker sends back for one duplicate set.

Mails are named rather than returned: the parent already holds them, and shipping them back would mean pickling every message it worked so hard not to keep.

Create new instance of SelectedSet(selected, discarded, stats, records)

selected: tuple[tuple[str, str], ...]

Alias for field number 0

discarded: tuple[tuple[str, str], ...]

Alias for field number 1

stats: dict[Stat, int]

Alias for field number 2

records: tuple[tuple[int, str], ...]

Alias for field number 3

class mail_deduplicate.deduplicate.Deduplicate(conf)

Bases: object

Load-up messages, search for duplicates, apply selection strategy and perform the action.

Similar messages sharing the same hash are grouped together in a DuplicateSet.

CHUNK_SIZE: int = 200

Mails handed to a hashing worker in one go.

Each queue round-trip costs far more than hashing a single mail, so tasks travel in chunks. Large enough to make that cost disappear, small enough that the last worker to finish does not hold up the others.

SET_CHUNK_SIZE: int = 32

Duplicate sets handed to a selection worker in one go.

Smaller than the hashing chunk because a set is several mails’ worth of work, so fewer of them already amortize the same round-trip.

sources: dict[str, Mailbox]

Index of mail sources by their full, normalized path. So we can refer to them in Mail instances. Also have the nice side effect of natural deduplication of sources themselves.

mails: dict[str, list[DedupMailMixin]]

All mails grouped by hashes.

Grouped in lists rather than sets: mails carry no value equality, so a set only ever deduplicated by object identity, which the single pass over each box already rules out. A one-element list also costs 64 bytes where a set costs 216, and most hashes group a single mail.

selection: set[DedupMailMixin]

Mails selected after application of selection strategy.

discard: set[DedupMailMixin]

Mails discarded after application of selection strategy.

link_targets: dict[DedupMailMixin, DedupMailMixin]

Maps each discarded mail to the selected mail it can be hardlinked to.

Left empty unless the configured action consumes it: see track_link_targets.

conf

Configuration shared across the deduplication process.

track_link_targets: bool

Whether each discarded mail has to be paired with a selected one.

Only the hardlinking action needs that pairing, and it costs one dictionary entry per discarded mail, so it is settled once here and only recorded when something reads it.

stats: Counter[Stat]

Deduplication statistics. Unset statistics naturally read as zero.

cache: HashCache | None

Cross-run cache of mail hashes, when the user opted in with --cache and the database could be opened.

restore_cached(box, mail_id, entry)

Rebuild a hashed mail from its cache entry, without opening its file.

Produces the same dehydrated stub the hashing step would have left behind, with the scalars the later steps rely on already memoized. Anything else those steps need is re-read from the box on demand, as for any other mail.

Return type:

tuple[DedupMailMixin, str, None]

blank_stub(box, mail_id)

An empty mail carrying only its identity, ready to be filled in.

Stands in for a mail this process never parsed, because its content came from the cache or from a worker that has already thrown it away.

Return type:

DedupMailMixin

parallel_boxes()

The sources whose mails a worker process could hash on its own.

Only folder-based boxes qualify. Their mails each own a file, which a worker opens by path, sharing nothing. Mails of a file-based box are byte ranges of one file that a single handle seeks through, so handing them out would mean several processes seeking the same descriptor.

Returns an empty list unless every source qualifies, so a run is either wholly parallel or wholly sequential rather than silently half of each.

Return type:

list[Mailbox]

property jobs: int

Worker processes the --jobs option resolved to, read off the context.

worker_pool(jobs, initializer, initargs, verb, doing)

Hand out a pool of worker processes prepared by the initializer, and shut it down however the block it wraps ends.

Yields None instead when the pool cannot be started, which is the case in some frozen or sandboxed environments: the initializer is then run right here, so the worker functions find the state they expect, and the caller degrades to sequential calls instead of dying.

Return type:

Iterator[ProcessPoolExecutor | None]

fan_out(pool, worker, tasks, window, chunksize)

Yields each task back alongside its worker’s answer.

Tasks pair what the parent keeps with the payload its worker receives, and are handed out a window at a time rather than all at once: map() consumes whatever it is given immediately, so passing the whole corpus would hold a task per mail, undoing the flat memory the rest of the run maintains. The window is wide enough that every worker always has chunks queued behind it.

With or without a pool, answers come back in submission order, so whatever the caller aggregates comes out identical to a sequential run at any number of workers.

Return type:

Iterator[tuple]

uncached(boxes, absorb, progress)

Yields the identity of every mail the cache cannot answer for.

A mail the cache can restore is absorbed here and never even opened, which is the whole point of keeping the cache. Lazy, and driven from the main process alone: neither box objects nor the cache are safe for concurrent access.

Return type:

Iterator[tuple[Mailbox, str]]

hash_in_parallel(jobs, absorb, progress)

Hash every uncached mail across a pool of worker processes.

Threads cannot do this job: what hashing spends itself on is Python-level work that the interpreter lock serializes, so fanning it out across threads only adds contention. Processes sidestep the lock, and the mail never has to travel: a worker opens its own file and sends back a hash and two scalars.

Return type:

None

adopt_hashed(box, mail_id, result)

Turn a worker’s answer back into the mail stub this process works with.

Return type:

tuple[DedupMailMixin, str | None, TooFewHeaders | None]

add_source(source_path)

Registers a source of mails, validates and opens it.

Duplicate sources of mails are not allowed, as when we perform the action, we use the path as a unique key to tie back a mail from its source.

Return type:

None

hash_all()

Browse all mails from all registered sources, compute hashes and group mails by hash.

Displays a progress bar as the operation might be slow.

Each mail is dehydrated as soon as it is hashed, so whatever the size of the corpus, only a lightweight stub of every mail is retained. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

Hashing fans out across worker processes when --jobs resolves above 1 and every source is a folder-based box; otherwise mails stream through this process one at a time, which is also the lowest-memory path. Box listing and the hash cache always stay in this process, as neither is safe for concurrent access, so the speedup is largest where the hashing itself is the cost, with --hash-body raw/normalized.

Return type:

None

build_sets()

Build the selected and discarded sets from each duplicate set.

The selection is settled one duplicate set at a time, to keep the memory footprint low and make the log easier to read.

Return type:

None

settle(hash_key, mail_set)

Run one duplicate set through the thresholds and the strategies, right in this process, and merge its verdict into the run.

Return type:

None

log_set_heading(hash_key, mail_count)

Announce a duplicate set, at a level reflecting whether it holds copies.

Styling the heading is not free, and most sets hold a single mail and so report at debug level, where the result is thrown away: only pay for it when it is actually logged.

Return type:

None

release(mail_set)

Drop the content the thresholds and strategies pulled back from the boxes.

Iterating on the original set covers skipped sets and set-aside mails too, which land in neither selection nor discard. See: https://github.com/kdeldycke/mail-deduplicate/issues/362

Return type:

None

record_link_targets(selection, discard)

Pair every discarded mail of a set with a selected mail of that same set.

Hardlinking a discarded mail only makes sense against a copy that survives the very set both were found in, and that pairing is the one thing the flat selection and discard sets no longer say once every set has been settled.

A strategy is free to keep several mails, so the target is the one with the lowest path: a run then links to the same copy however the sets came back, which matters when the selection is spread over a pool of processes.

Iterables are only walked once the run is known to need them, so a call from a non-hardlinking run costs nothing beyond the arguments themselves.

Return type:

None

select_in_parallel(jobs, progress)

Apply the selection to every duplicate set across a pool of processes.

Duplicate sets share nothing with one another, so a set is the natural unit of work. Only sets holding copies are handed out: a set of one is settled without reading anything, and would cost more to ship than to decide.

Return type:

None

describe(mail)

Everything a worker needs to rebuild a mail, and nothing more.

Return type:

MailMeta

adopt_selection(hash_key, mail_set, result)

Merge a worker’s verdict on one set back into this process.

The heading and the worker’s own messages are said here, in that order, so the log reads as it would have from a sequential run however the sets were spread out.

Return type:

None

close_all()

Close all open boxes, and the hash cache if one was opened.

Return type:

None

report()

Returns a text report of user-friendly statistics and metrics.

Return type:

str

assert_stats(first, operator, second)

Render failed stats assertions in plain English.

Return type:

None

check_stats()

Perform some high-level consistency checks on metrics.

Helps users reports tricky edge-cases.

mail_deduplicate.mail module

A mail wrapped with the deduplication-specific properties: canonical hash, normalized headers, timestamp, size, and the dehydration machinery keeping memory flat.

exception mail_deduplicate.mail.TooFewHeaders

Bases: Exception

Not enough headers were found to produce a solid hash.

class mail_deduplicate.mail.TimeSource(*values)

Bases: StrEnum

Enumeration of all supported mail timestamp sources.

DATE_HEADER = 'date-header'

Timestamp sourced from the message’s Date header.

CTIME = 'ctime'

Timestamp is from the email’s file on the filesystem.

Attention

Only meaningful for sources storing one mail per file, like maildir and eml.

mail_deduplicate.mail.ADDRESS_HEADERS = frozenset({'bcc', 'cc', 'delivered-to', 'disposition-notification-to', 'envelope-to', 'from', 'original-recipient', 'reply-to', 'resent-bcc', 'resent-cc', 'resent-from', 'resent-reply-to', 'resent-sender', 'resent-to', 'return-path', 'sender', 'to', 'x-envelope-from', 'x-envelope-to', 'x-original-to'})

Headers that contain email addresses.

Hint

Headers from which quotes should be discarded, so "Bob" <bob@example.com> hashes to the same thing as Bob <bob@example.com>.

Attention

These IDs should be kept lower-case, because they are compared to the IDs provided to the -h/--hash-header option, carried by the hash_headers entry of the configuration.

class mail_deduplicate.mail.DedupMailMixin(message=None)

Bases: Message

Message with deduplication-specific properties and utilities.

Extends standard library’s mailbox.Message, and shouldn’t be used directly, but composed with mailbox.Message sub-classes.

Initialize a Message instance.

CONTENT_CACHES: tuple[str, ...] = ('body_lines', 'canonical_headers', 'hash_raw_body', 'hash_normalized_body')

Memoized properties derived from the message content.

They are only needed while computing hashes, and are dropped by dehydrate() alongside the parsed message itself.

resolve_path: Callable[[Mailbox, str], str]

Derives the real filesystem location of a mail from its box.

Set per box format by make_dedup_mail(), as folder-based boxes give each mail its own file while file-based ones pack them all into the box’s single file.

defects: list = ()

Fallback for the parsing defects dropped by dehydrate().

An empty tuple rather than a list, so it can be shared by every dehydrated mail instead of costing one throw-away empty list each: reads still work, and the appends only the parser performs fail loudly on a mail that has no message.

PARSED_MESSAGE_ATTRS: tuple[str, ...] = ('_headers', '_payload', '_unixfrom', 'preamble', 'epilogue', 'defects')

Attributes carrying the parsed message, as populated by email.parser.

They hold the bulk of a mail’s memory footprint, and are the ones dropped by dehydrate() and restored by hydrate().

box: Mailbox | None

The box this message was read from, kept to re-fetch its content on demand after dehydration.

source_path: str | None

Normalized path to the mailbox this message originates from.

mail_id: str | None

Mail ID used to uniquely refers to it in the context of its source.

conf: Config

Global configuration

property path: str

Real filesystem location of the mail.

Returns the individual mail’s file for folder-based box types (maildir & co.), but returns the whole box path for file-based boxes (mbox & co.). Used by regexp-based selection strategies and to render the mail’s repr.

Derived on access from the box rather than stored, so a retained mail does not carry a copy of its own absolute path for the whole run. Raises AttributeError before the box metadata is attached, so getattr(mail, “path”, None) still reads as absent.

add_box_metadata(box, mail_id)

Post-instantiation utility to attach to mail some metadata derived from its parent box.

Called right after the __init__() constructor.

This allows the mail to carry its own information on its origin box and index.

Return type:

None

property is_hydrated: bool

Whether the mail still carries its parsed message.

dehydrate()

Reduce the mail to the lightweight metadata needed by the next steps.

Memoizes the scalar properties consumed by selection strategies (timestamp, and size when the decoded body is at hand), then drops the parsed message and every cached copy of its content. This cuts the resident footprint of a retained mail from the full size of its message to a few hundred bytes, whatever the number of mails processed. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

No-op if the mail is already dehydrated. The dropped content is re-read from the source box by hydrate() when a later step needs it again.

Return type:

None

hydrate()

Restore the full parsed message dropped by dehydrate().

Re-reads the mail from its source box, through the same parsing path that produced it in the first place, so the restored content is identical. No-op if the mail still carries its parsed message.

Return type:

None

hydrated()

Borrow the full parsed message for the duration of the block.

Restores the message on the way in and releases it on the way out, so a step needing the whole mail after the hashing one keeps its content resident only while it uses it, and memory stays flat across a loop of mails.

Return type:

Iterator[DedupMailMixin]

property parsed_date: float | None

Parse the mail’s date header into float timestamp.

Returns None if the mail has no valid date header.

Self-hydrating: re-reads the message from its box if it was dehydrated.

property timestamp: float | None

Compute the normalized canonical timestamp of the mail.

Sourced from the message’s Date header by default. In the case of maildir, can be sourced from the email’s file from the filesystem.

Warning

ctime does not refer to creation time on POSIX systems, but rather the time of the last metadata change.

Todo

Investigate what mailbox.MaildirMessage.get_date() does and if we can use it.

property size: int

Returns canonical mail size.

Size is computed as the length of the message body, i.e. the payload of the mail stripped of all its headers, not from the mail file persisting on the file- system.

Todo

Allow customization of the way the size is computed, by getting the file size instead, for example with os.path.getsize(mail_file).

property body_lines: list[str]

Return a normalized list of lines from message’s body.

Self-hydrating: re-reads the message from its box if it was dehydrated.

decode_part(part)

Decode a single message part to string.

Return type:

str

hash_key()

Returns the canonical hash of a mail.

Caution

This method hasn’t been made explicitly into a cached property in order to reduce the overall memory footprint.

Return type:

str

property hash_raw_body: str

Returns the canonical body hash of a mail.

property hash_normalized_body: str

Returns the normalized body hash of a mail.

property canonical_headers: tuple[tuple[str, str], ...]

Returns the full list of all canonical headers names and values in preparation for hashing.

Self-hydrating: re-reads the message from its box if it was dehydrated.

pretty_canonical_headers()

Renders a table of headers names and values used to produce the mail’s hash.

Caution

This method hasn’t been explicitly made into a cached property in order to reduce the overall memory footprint.

Returns a string ready to be printed.

Return type:

str

serialized_headers()

Serialize the canonical headers into a single string ready to be hashed.

At this point we should have an absolute minimum of headers.

Caution

This method hasn’t been explicitly made into a cached property in order to reduce the overall memory footprint.

Return type:

bytes

normalized_header_values(header_id)

Returns all normalized values of a header.

Values are cleaned-up into their canonical form.

Return type:

Iterator[str]

normalize_subject(subject)

Strip Re:/Fwd: and [list-name] prefixes from Subject.

This cleans up prefixes automatically added by mailing list software, since the mail could have been CC’d to multiple lists, in which case it will receive a different prefix for each.

Return type:

str

normalize_content_type(value)

Normalize Content-Type by stripping parameters.

Removes everything after the semicolon, keeping only the MIME type. E.g., text/plain; charset=utf-8 becomes text/plain.

Apparently list servers actually munge Content-Type e.g. by stripping the quotes from charset="us-ascii". Section 5.1 of RFC2045 says that either form is valid (and they are equivalent).

Additionally, with multipart/mixed, boundary delimiters can vary by recipient. We need to allow for duplicates coming from multiple recipients, since for example you could be signed up to the same list twice with different addresses. Or maybe someone bounces you a load of mail some of which is from a mailing list you’re both subscribed to - then it’s still useful to be able to eliminate duplicates.

Return type:

str

normalize_date(value)

Normalize Date to YYYY-MM-DD format.

Date timestamps can differ by seconds or hours for various reasons, so let’s only honour the date for now and normalize them to UTC timezone.

Return type:

str

normalize_address_header(value)

Normalize address headers by removing quotes and collapsing whitespace.

E.g., "Bob" <bob@example.com> becomes Bob <bob@example.com>.

Remove quotes in any headers that contain addresses to ensure a quoted name is hashed to the same value as an unquoted one.

Danger

This may not be the cleanest way to normalize email addresses. E.g. "Robert \"Bob\"`` becomes Robert \Bob\, but this shouldn’t matter for hashing purposes as we’re just trying to get a good heuristic. Refs: #847 and #846.

Return type:

str

normalize_message_id(value)

Normalize Message-ID header by stripping angle brackets.

E.g., <unique-id@example.com> becomes unique-id@example.com.

Return type:

str

strip_angle_brackets(value)

Strip angle brackets from a value if it’s a single bracketed item.

Only strips if the value matches <something> with no commas.

Note

Sometimes email.parser strips the <> brackets from a To: header which has a single address. I have seen this happen for only one mail in a duplicate pair. I’m not sure why (presumably the parser uses email.utils.unquote somewhere in its code path which was only triggered by that mail and not its sister mail), but to be safe, we should always strip the <> brackets to avoid this difference preventing duplicate detection.

Return type:

str

mail_deduplicate.mail_box module

Utilities to read and write mail boxes in various formats.

Based on Python’s standard library mailbox module.

mail_deduplicate.mail_box.maildir_mail_path(box, key)

Location of a maildir mail, read from the box’s table of contents.

A maildir key drops the :2,<flags> suffix the file name carries, so the name cannot be rebuilt from the key alone. The box refreshed its table of contents to hand out the mail in the first place, so it is only re-validated on a miss.

Return type:

str

mail_deduplicate.mail_box.keyed_mail_path(box, key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

mail_deduplicate.mail_box.box_file_path(box, key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

mail_deduplicate.mail_box.iter_mail_ids(box)

Yields the key of every mail held by a box.

Maildir.iterkeys() confirms that each key still resolves to a file, one stat per mail, on top of the directory listing it has just built. Every caller here goes on to stat or open that same file anyway, and copes with its disappearance, so the check is paid for twice and needed once: reading the refreshed table of contents directly skips it.

The other formats list their mails without that extra round, and are left to their own iterator.

Return type:

Iterator[str]

mail_deduplicate.mail_box.resolve_mail_path(box, key)

Location of a mail in its box, without instantiating the mail.

Lets a caller reach a mail’s file before deciding to read it, which is how the hash cache checks whether a mail changed without paying for its parsing.

Return type:

str

mail_deduplicate.mail_box.make_dedup_mail(name, base, path_resolver)

Create a DedupMail class for a mailbox message type.

Deriving a mail’s own location from its box is format-specific, so the resolver is baked into the class here instead of being branched on at runtime.

Return type:

type

class mail_deduplicate.mail_box.MaildirDedupMail(message=None)

Bases: DedupMailMixin, MaildirMessage

Initialize a MaildirMessage instance.

static resolve_path(key)

Location of a maildir mail, read from the box’s table of contents.

A maildir key drops the :2,<flags> suffix the file name carries, so the name cannot be rebuilt from the key alone. The box refreshed its table of contents to hand out the mail in the first place, so it is only re-validated on a miss.

Return type:

str

class mail_deduplicate.mail_box.mboxDedupMail(message=None)

Bases: DedupMailMixin, mboxMessage

Initialize an mboxMMDFMessage instance.

static resolve_path(key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.MHDedupMail(message=None)

Bases: DedupMailMixin, MHMessage

Initialize an MHMessage instance.

static resolve_path(key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

class mail_deduplicate.mail_box.BabylDedupMail(message=None)

Bases: DedupMailMixin, BabylMessage

Initialize a BabylMessage instance.

static resolve_path(key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.MMDFDedupMail(message=None)

Bases: DedupMailMixin, MMDFMessage

Initialize an mboxMMDFMessage instance.

static resolve_path(key)

Location of a mail from a file-based box: the box’s own single file.

Every mail of an mbox, babyl or mmdf box is packed into it, so they all share this path and are told apart by their mail ID.

Return type:

str

class mail_deduplicate.mail_box.EMLDedupMail(message=None)

Bases: DedupMailMixin, Message

Initialize a Message instance.

static resolve_path(key)

Location of an MH or eml mail, whose file is named after its key.

Return type:

str

class mail_deduplicate.mail_box.EML(dirname, factory=None, create=True)

Bases: Mailbox

A folder of loose .eml files, walked recursively.

Supports mail archives exported as individual RFC 5322 files, one mail per file, as produced by Outlook PST/OST conversion tools for instance. See: https://github.com/kdeldycke/mail-deduplicate/issues/760

Keys are the paths of the mail files, relative to the folder’s root. Files without the .eml extension (case-insensitive) are ignored, as well as hidden files and directories.

Follows the interface of Python’s mailbox.Mailbox. Like maildir, the one-file-per-mail storage needs no locking.

Initialize a Mailbox instance.

iterkeys()

Return an iterator over keys.

get_file(key)

Return a file-like representation or raise a KeyError.

get_bytes(key)

Return a byte string representation or raise a KeyError.

Return type:

bytes

get_message(key)

Return a Message representation or raise a KeyError.

add(message)

Add message and return assigned key.

Return type:

str

remove(key)

Remove the keyed message; raise KeyError if it doesn’t exist.

Return type:

None

list_folders()

No dedicated subfolder objects: the recursive walk covers nested directories.

Return type:

list[str]

get_folder(folder)

flush()

Mails are written straight to the filesystem: nothing to flush.

Return type:

None

lock()

One-file-per-mail storage needs no locking.

Return type:

None

unlock()

One-file-per-mail storage needs no locking.

Return type:

None

close()

No resource is kept open between operations.

Return type:

None

class mail_deduplicate.mail_box.BoxStructure(*values)

Bases: Enum

Box structures can be file-based or folder-based.

FOLDER = 1

FILE = 2

class mail_deduplicate.mail_box.BoxFormat(base_class, structure, message_class)

Bases: Enum

IDs of all the supported box formats and their metadata.

Each entry is associated to:

• their original base class,

• the structure they implement (file-based or folder-based),

• the custom message factory class to use.

From these, we can derive the proper constructor with our own custom DedupMail factory.

Hint

This could be extended in the future to add support for other mailbox formats and sources, like Gmail accounts, IMAP servers, etc.

MAILDIR = (<class 'mailbox.Maildir'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.MaildirDedupMail'>)

MBOX = (<class 'mailbox.mbox'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.mboxDedupMail'>)

MH = (<class 'mailbox.MH'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.MHDedupMail'>)

BABYL = (<class 'mailbox.Babyl'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.BabylDedupMail'>)

MMDF = (<class 'mailbox.MMDF'>, BoxStructure.FILE, <class 'mail_deduplicate.mail_box.MMDFDedupMail'>)

EML = (<class 'mail_deduplicate.mail_box.EML'>, BoxStructure.FOLDER, <class 'mail_deduplicate.mail_box.EMLDedupMail'>)

property constructor

Return a constructor for this box format with our custom message factory.

mail_deduplicate.mail_box.FOLDER_FORMATS = (BoxFormat.MAILDIR, BoxFormat.MH, BoxFormat.EML)

Box formats implementing a folder-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.FILE_FORMATS = (BoxFormat.MBOX, BoxFormat.BABYL, BoxFormat.MMDF)

Box formats implementing a file-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.FOLDER_FORMAT_CLASSES = (<class 'mailbox.Maildir'>, <class 'mailbox.MH'>, <class 'mail_deduplicate.mail_box.EML'>)

Base classes of folder-based box formats, as a tuple ready for isinstance.

mail_deduplicate.mail_box.MAILDIR_SUBDIRS = frozenset({'cur', 'new', 'tmp'})

List of required sub-folders defining a properly structured maildir.

mail_deduplicate.mail_box.is_maildir(path)

Returns True when the path holds all the sub-directories of a properly structured maildir.

Return type:

bool

mail_deduplicate.mail_box.contains_maildir(path)

Returns True when the path is a maildir or holds one at any depth.

Allows the discovery of nested maildir folders stored as plain directories, as produced by isync/mbsync’s Verbatim naming style. See: https://github.com/kdeldycke/mail-deduplicate/issues/973

Dot-prefixed directories are ignored, as they are covered by the Maildir++ folder convention. The mail-holding sub-directories of maildirs are not explored either.

Return type:

bool

mail_deduplicate.mail_box.contains_eml(path)

Returns True when the path holds at least one .eml file, at any depth.

Hidden files and directories are ignored, and the extension is matched case-insensitively, mirroring the walk of EML boxes.

Return type:

bool

mail_deduplicate.mail_box.autodetect_box_type(path)

Auto-detect the format of the mailbox located at the provided path.

If the path is a file, then it is considered as an mbox. Else, if the provided path is a folder and features the MAILDIR_SUBDIRS sub-directories, or holds nested maildir folders at any depth, it is parsed as a maildir. A folder holding loose .eml files instead is parsed as an eml source.

Todo

Future finer autodetection heuristics should be implemented here. Some ideas:

• single mail from a maildir

• plain text mail content

• other mailbox formats supported in Python’s standard library:

  • MH

  • Babyl

  • MMDF

Return type:

BoxFormat

mail_deduplicate.mail_box.open_box(path, box_format=None, force_unlock=False)

Open a mail box.

Returns a list of boxes, one per sub-folder. All are locked, ready for operations.

If box_format is provided, forces the opening of the box in the specified format. Else, defaults to autodetection.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.lock_box(box, force_unlock)

Lock an opened box and allows for forced unlocking.

Returns the locked box.

Return type:

Mailbox

mail_deduplicate.mail_box.open_subfolders(box, force_unlock)

Browse recursively the subfolder tree of a box.

Returns a list of opened and locked boxes, each for one subfolder.

Skips box types not supporting subfolders. For maildir, both the Maildir++ convention (dot-prefixed folders) and Verbatim-style layouts (nested plain directories, each a maildir of its own) are browsed. A directory without the maildir structure only acts as a container of nested folders and carries no mail of its own.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.create_box(path, box_format, export_append=False)

Creates a brand new box from scratch.

Return type:

Mailbox

mail_deduplicate.strategy module

Strategy definitions.

mail_deduplicate.strategy.log_selection(message_template)

Decorator to log selection criteria.

mail_deduplicate.strategy.select_older(duplicates)

Select all older duplicates.

Discards the newests, i.e. the subset sharing the most recent timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_oldest(duplicates)

Select all the oldest duplicates.

Discards the newers, i.e. all mail of the duplicate set but those sharing the oldest timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_newer(duplicates)

Select all newer duplicates.

Discards the oldest, i.e. the subset sharing the most ancient timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_newest(duplicates)

Select all the newest duplicates.

Discards the olders, i.e. all mail of the duplicate set but those sharing the newest timestamp.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_smaller(duplicates)

Select all smaller duplicates.

Discards the biggests, i.e. the subset sharing the biggest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_smallest(duplicates)

Select all the smallest duplicates.

Discards the biggers. i.e. all mail of the duplicate set but those sharing the smallest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_bigger(duplicates)

Select all bigger duplicates.

Discards the smallests, i.e. the subset sharing the smallest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_biggest(duplicates)

Select all the biggest duplicates.

Discards the smallers, i.e. all mail of the duplicate set but those sharing the biggest size.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_matching_path(duplicates)

Select all duplicates whose file path match the regular expression provided via the –regexp parameter.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_non_matching_path(duplicates)

Select all duplicates whose file path doesn’t match the regular expression provided via the –regexp parameter.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_one(duplicates)

Randomly select one duplicate, and discards all others.

Return type:

set[DedupMailMixin]

mail_deduplicate.strategy.select_all_but_one(duplicates)

Randomly discard one duplicate, and select all others.

Return type:

set[DedupMailMixin]

class mail_deduplicate.strategy.Strategy(*values)

Bases: Enum

Selection strategies to apply on a set of duplicate mails.

Each strategy in the Enum points to the function implementing the selection logic, by way of the function property.

Strategies whose member value is a string are simply aliases to other strategies, pointing to the name of the function implementing the logic. The other members have integer values, to indicate their function ID is to be derived from the member name. This arrangement lets each member exist on its own instead of being hidden by the aliasing mechanism of Enum.

Aliases are great usability features to represent inverse operations. They help users reason about the selection operators in whichever direction matches their mental model.

SELECT_OLDER = 1

SELECT_OLDEST = 2

SELECT_NEWER = 3

SELECT_NEWEST = 4

DISCARD_NEWEST = 'select_older'

DISCARD_NEWER = 'select_oldest'

DISCARD_OLDEST = 'select_newer'

DISCARD_OLDER = 'select_newest'

SELECT_SMALLER = 5

SELECT_SMALLEST = 6

SELECT_BIGGER = 7

SELECT_BIGGEST = 8

DISCARD_BIGGEST = 'select_smaller'

DISCARD_BIGGER = 'select_smallest'

DISCARD_SMALLEST = 'select_bigger'

DISCARD_SMALLER = 'select_biggest'

SELECT_MATCHING_PATH = 9

SELECT_NON_MATCHING_PATH = 10

DISCARD_NON_MATCHING_PATH = 'select_matching_path'

DISCARD_MATCHING_PATH = 'select_non_matching_path'

SELECT_ONE = 11

SELECT_ALL_BUT_ONE = 12

DISCARD_ALL_BUT_ONE = 'select_one'

DISCARD_ONE = 'select_all_but_one'

property function: Callable[[DuplicateSet], set[DedupMailMixin]]

The selection function this member stands for.

Alias members carry the function’s name as their value; canonical members derive it from their own name.

apply(duplicates)

Perform the selection strategy on the provided duplicate set.

Returns the set of selected mail objects.

Return type:

set[DedupMailMixin]