mail_deduplicate package

Expose package-wide elements.

mail_deduplicate.HASH_HEADERS: tuple[str, ...] = ('Date', 'From', 'To', 'Subject', 'MIME-Version', 'Content-Type', 'Content-Disposition', 'User-Agent', 'X-Priority', 'Message-ID')

Default ordered list of headers to use to compute the unique hash of a mail.

By default we choose to exclude:

CC

Since mailman apparently sometimes trims list members from the CC header to avoid sending duplicates. Which means that copies of mail reflected back from the list server will have a different CC to the copy saved by the MUA at send-time.

BCC

Because copies of the mail saved by the MUA at send-time will have BCC, but copies reflected back from the list server won’t.

Reply-To

Since a mail could be CC’d to two lists with different Reply-To munging options set.

mail_deduplicate.ADDRESS_HEADERS = frozenset({'bcc', 'cc', 'delivered-to', 'disposition-notification-to', 'envelope-to', 'from', 'original-recipient', 'reply-to', 'resent-bcc', 'resent-cc', 'resent-from', 'resent-reply-to', 'resent-sender', 'resent-to', 'return-path', 'sender', 'to', 'x-envelope-from', 'x-envelope-to', 'x-original-to'})

Headers that contain email addresses.

Danger

These IDs should be kept lower-case, because they are compared to the one provided to those provided to the -h/--hash-header option, that is carried by the hash_headers property of the configuration.

mail_deduplicate.QUOTE_DISCARD_HEADERS = frozenset({'bcc', 'cc', 'delivered-to', 'disposition-notification-to', 'envelope-to', 'from', 'original-recipient', 'reply-to', 'resent-bcc', 'resent-cc', 'resent-from', 'resent-reply-to', 'resent-sender', 'resent-to', 'return-path', 'sender', 'to', 'x-envelope-from', 'x-envelope-to', 'x-original-to'})

Headers from which quotes should be discarded.

E.g. "Bob" <bob@example.com> should hash to the same thing as Bob <bob@example.com>.

mail_deduplicate.MINIMAL_HEADERS_COUNT = 4

Below this value, we consider not having enough headers to compute a solid hash.

mail_deduplicate.DEFAULT_SIZE_THRESHOLD = 512

Default size threshold in bytes.

Since we’re ignoring the Content-Length header by default because of mailing-list effects, we introduced a limit on the allowed difference between the sizes of the message payloads.

If this is exceeded, a warning is issued and the messages are not considered duplicates, because this could point to message corruption somewhere, or a false positive.

Note

Headers are not counted towards this threshold, because many headers can be added by mailing list software such as mailman, or even by the process of sending the mail through various MTAs.

One copy could have been stored by the sender’s MUA prior to sending, without any Received headers, and another copy could be reflected back via a CC-to-self mechanism or mailing list server.

This threshold has to be large enough to allow for footers added by mailing list servers.

mail_deduplicate.DEFAULT_CONTENT_THRESHOLD = 768

Default content threshold in bytes.

As above, we similarly generates unified diffs of duplicates and ensure that the diff is not greater than a certain size to limit false-positives.

exception mail_deduplicate.TooFewHeaders

Bases: Exception

Not enough headers were found to produce a solid hash.

exception mail_deduplicate.SizeDiffAboveThreshold

Bases: Exception

Difference in mail size is greater than threshold..

exception mail_deduplicate.ContentDiffAboveThreshold

Bases: Exception

Difference in mail content is greater than threshold..

Submodules

mail_deduplicate.action module

mail_deduplicate.action.copy_mails(dedup, mails)

Copy provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.move_mails(dedup, mails)

Move provided mails to a brand new box or an existing one.

Return type:

None

mail_deduplicate.action.delete_mails(dedup, mails)

Remove provided mails in-place, from their original boxes.

Return type:

None

mail_deduplicate.action.copy_selected(dedup)

Copy all selected mails to a brand new box.

Return type:

None

mail_deduplicate.action.copy_discarded(dedup)

Copy all discarded mails to a brand new box.

Return type:

None

mail_deduplicate.action.move_selected(dedup)

Move all selected mails to a brand new box.

Return type:

None

mail_deduplicate.action.move_discarded(dedup)

Move all discarded mails to a brand new box.

Return type:

None

mail_deduplicate.action.delete_selected(dedup)

Remove in-place all selected mails, from their original boxes.

Return type:

None

mail_deduplicate.action.delete_discarded(dedup)

Remove in-place all discarded mails, from their original boxes.

Return type:

None

class mail_deduplicate.action.Action(*values)

Bases: Enum

Define all available action IDs.

COPY_SELECTED = 'copy-selected'

COPY_DISCARDED = 'copy-discarded'

MOVE_SELECTED = 'move-selected'

MOVE_DISCARDED = 'move-discarded'

DELETE_SELECTED = 'delete-selected'

DELETE_DISCARDED = 'delete-discarded'

action_function()

Return the action function associated with this action.

Return type:

callable

perform_action(dedup)

Performs the action on selected mail candidates.

Return type:

None

mail_deduplicate.cli module

class mail_deduplicate.cli.Config

Bases: TypedDict

Holds global configuration.

input_format: BoxFormat | None

force_unlock: bool

hash_headers: tuple[str, ...]

hash_body: BodyHasher

hash_only: bool

size_threshold: int

content_threshold: int

show_diff: bool

strategy: str | None

time_source: TimeSource

regexp: re.Pattern | None

action: Action

export: Path | None

export_format: BoxFormat

export_append: bool

dry_run: bool

mail_deduplicate.cli.normalize_headers(ctx, param, value)

Validate headers provided as parameters to the CLI.

Headers are case-insensitive in Python implementation, so we normalize them to lower-case.

We then deduplicate them, while preserving order.

Mail headers are expected to be composed of ASCII characters between 33 and 126 (both inclusive) according to RFC-5322.

Return type:

tuple[str, ...]

mail_deduplicate.cli.compile_regexp(ctx, param, value)

Validate and compile regular expression provided as parameters to the CLI.

Return type:

Pattern[str] | None

class mail_deduplicate.cli.MdedupCommand(*args, version=None, extra_option_at_end=True, populate_auto_envvars=True, **kwargs)

Bases: ExtraCommand

List of extra parameters:

Parameters:

• version (str | None) – allows a version string to be set directly on the command. Will be passed to the first instance of ExtraVersionOption parameter attached to the command.

• extra_option_at_end (bool) – reorders all parameters attached to the command, by moving all instances of ExtraOption at the end of the parameter list. The original order of the options is preserved among themselves.

• populate_auto_envvars (bool) – forces all parameters to have their auto-generated environment variables registered. This address the shortcoming of click which only evaluates them dynamiccaly. By forcing their registration, the auto-generated environment variables gets displayed in the help screen, fixing click#2483 issue. On Windows, environment variable names are case-insensitive, so we normalize them to uppercase.

By default, these Click context settings are applied:

• auto_envvar_prefix = self.name (Click feature)

  Auto-generate environment variables for all options, using the command ID as prefix. The prefix is normalized to be uppercased and all non-alphanumerics replaced by underscores.

• help_option_names = ("--help", "-h") (Click feature)

  Allow help screen to be invoked with either –help or -h options.

• show_default = True (Click feature)

  Show all default values in help screen.

Additionally, these Cloup context settings are set:

• align_option_groups = False (Cloup feature)

  Aligns option groups in help screen.

• show_constraints = True (Cloup feature)

  Show all constraints in help screen.

• show_subcommand_aliases = True (Cloup feature)

  Show all subcommand aliases in help screen.

Click Extra also adds its own context_settings:

• show_choices = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally show or hide choices when prompting a user for input. Only makes sense for options whose prompt property is set.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_choices setting.

• show_envvar = None (Click Extra feature)

  If set to True or False, will force that value on all options, so we can globally enable or disable the display of environment variables in help screen.

  Defaults to None, which will leave all options untouched, and let them decide of their own show_envvar setting. The rationale being that discoverability of environment variables is enabled by the --show-params option, which is active by default on extra commands. So there is no need to surcharge the help screen.

  This addresses the click#2313 issue.

To override these defaults, you can pass your own settings with the context_settings parameter:

@extra_command(
    context_settings={
        "show_default": False,
        ...
    }
)

format_help(ctx, formatter)

Extend the help screen with the description of all available strategies.

Return type:

None

mail_deduplicate.deduplicate module

mail_deduplicate.deduplicate.STATS_DEF = {'mail_copied': 'Number of mails copied from their original mailbox to another.', 'mail_deleted': 'Number of mails deleted from their mailbox in-place.', 'mail_discarded': 'Number of mails discarded from the final selection.', 'mail_duplicates': 'Number of duplicate mails (sum of mails in all duplicate sets with at least 2 mails).', 'mail_found': 'Total number of mails encountered from all mail sources.', 'mail_hashes': 'Number of unique hashes.', 'mail_moved': 'Number of mails moved from their original mailbox to another.', 'mail_rejected': 'Number of mails rejected individually because they were unparsable or did not have enough metadata to compute hashes.', 'mail_retained': 'Number of valid mails parsed and retained for deduplication.', 'mail_selected': 'Number of mails kept in the final selection on which the action will be performed.', 'mail_skipped': 'Number of mails ignored in the selection step because the whole set they belong to was skipped.', 'mail_unique': 'Number of unique mails (which where automatically added to selection).', 'set_deduplicated': 'Number of valid sets on which the selection strategy was successfully applied.', 'set_single': 'Total number of sets containing only a single mail with no applicable strategy. They were automatically kept in the final selection.', 'set_skipped_content': 'Number of sets skipped from the selection process because they were too dissimilar in content.', 'set_skipped_encoding': 'Number of sets skipped from the selection process because they had encoding issues.', 'set_skipped_size': 'Number of sets skipped from the selection process because they were too dissimilar in size.', 'set_skipped_strategy': 'Number of sets skipped from the selection process because the strategy could not be applied.', 'set_total': 'Total number of duplicate sets.'}

All tracked statistics and their definition.

class mail_deduplicate.deduplicate.BodyHasher(*values)

Bases: Enum

Enumeration of available body hashing methods.

SKIP = 'skip'

RAW = 'raw'

NORMALIZED = 'normalized'

hash_function()

Returns the hashing function corresponding to the body hasher.

class mail_deduplicate.deduplicate.DuplicateSet(hash_key, mail_set, conf)

Bases: object

A set of mails sharing the same hash.

Implements all the safety checks required before we can apply any selection strategy.

Load-up the duplicate set of mail and freeze pool.

Once loaded-up, the pool of parsed mails is considered frozen for the rest of the duplicate set’s life. This allows aggressive caching of lazy instance attributes depending on the pool content.

property size: int

Returns the number of mails in the duplicate set.

property newest_timestamp

Returns the newest timestamp among all mails in the set.

property oldest_timestamp

Returns the oldest timestamp among all mails in the set.

property biggest_size

Returns the biggest size among all mails in the set.

property smallest_size

Returns the smallest size among all mails in the set.

check_differences()

Ensures all mail differs in the limits imposed by size and content thresholds.

Compare all mails of the duplicate set with each other, both in size and content. Raise an error if we’re not within the limits imposed by the threshold settings.

diff(mail_a, mail_b)

Return difference in bytes between two mails’ normalized body.

Todo

Rewrite the diff algorithm to not rely on naive unified diff result parsing.

pretty_diff(mail_a, mail_b)

Returns a verbose unified diff between two mails’ normalized body.

categorize_candidates()

Process the list of duplicates for action.

Run preliminary checks, then apply the strategy to the pool of mails.

The process results in two subsets of mails: the selected and the discarded.

class mail_deduplicate.deduplicate.Deduplicate(conf)

Bases: object

Load-up messages, search for duplicates, apply selection strategy and perform the action.

Similar messages sharing the same hash are grouped together in a DuplicateSet.

add_source(source_path)

Registers a source of mails, validates and opens it.

Duplicate sources of mails are not allowed, as when we perform the action, we use the path as a unique key to tie back a mail from its source.

Return type:

None

hash_all()

Browse all mails from all registered sources, compute hashes and group mails by hash.

Displays a progress bar as the operation might be slow.

build_sets()

Build the selected and discarded sets from each duplicate set.

We apply the selection strategy one duplicate set at a time to keep memory footprint low and make the log easier to read.

close_all()

Close all open boxes.

report()

Returns a text report of user-friendly statistics and metrics.

assert_stats(first, operator, second)

Render failed stats assertions in plain English.

..hint ::

If inconsistent metrics are detected, the CLI will exit with a code numbered 115.

This has been arbitrarily chosen in PR #842, to make it unlikely to conflict with other exit codes. Users can rely on 115 meaning that the statistics checks failed.

check_stats()

Perform some high-level consistency checks on metrics.

Helps users reports tricky edge-cases.

mail_deduplicate.mail module

class mail_deduplicate.mail.TimeSource(*values)

Bases: Enum

Enumeration of all supported mail timestamp sources.

DATE_HEADER = 'date-header'

Timestamp sourced from the message’s Date header.

CTIME = 'ctime'

Timestamp is from the email’s file on the filesystem.

Attention

Only available for maildir sources.

class mail_deduplicate.mail.DedupMail(message)

Bases: object

Message with deduplication-specific properties and utilities.

Extends standard library’s mailbox.Message, and shouldn’t be used directly, but composed with mailbox.Message sub-classes.

Initialize a pre-parsed Message instance the same way the default factory in Python’s mailbox module does.

add_box_metadata(box, mail_id)

Post-instantiation utility to attach to mail some metadata derived from its parent box.

Called right after the __init__() constructor.

This allows the mail to carry its own information on its origin box and index.

Return type:

None

property uid: tuple[str | None, str | None]

Unique ID of the mail.

property timestamp: float | None

Compute the normalized canonical timestamp of the mail.

Sourced from the message’s Date header by default. In the case of maildir, can be sourced from the email’s file from the filesystem.

Warning

ctime does not refer to creation time on POSIX systems, but rather the last time the inode data changed.

Todo

Investigate what mailbox.MaildirMessage.get_date() does and if we can use it.

property size: int

Returns canonical mail size.

Size is computed as the length of the message body, i.e. the payload of the mail stripped of all its headers, not from the mail file persisting on the file- system.

Todo

Allow customization of the way the size is computed, by getting the file size instead for example: `python size = os.path.getsize(mail_file) `

property body_lines: list[str]

Return a normalized list of lines from message’s body.

property subject: str

Normalized subject.

Only used for debugging and human-friendly logging.

hash_key()

Returns the canonical hash of a mail.

Caution

This method hasn’t been made explicitly into a cached property in order to reduce the overall memory footprint.

Return type:

str

property hash_raw_body: str

Returns the canonical body hash of a mail.

property hash_normalized_body: str

Returns the normalized body hash of a mail.

property canonical_headers: tuple[tuple[str, str], ...]

Returns the full list of all canonical headers names and values in preparation for hashing.

pretty_canonical_headers()

Renders a table of headers names and values used to produce the mail’s hash.

Caution

This method hasn’t been made explicitly into a cached property in order to reduce the overall memory footprint.

Returns a string ready to be printed.

Return type:

str

serialized_headers()

Serialize the canonical headers into a single string ready to be hashed.

At this point we should have at an absolute minimum of headers.

Caution

This method hasn’t been made explicitly into a cached property in order to reduce the overall memory footprint.

Return type:

bytes

static normalize_header_value(header_id, value)

Normalize and clean-up header value into its canonical form.

Always returns a unicode string.

Return type:

str

mail_deduplicate.mail_box module

Utilities to read and write mail boxes in various formats.

Based on Python’s standard library mailbox module.

class mail_deduplicate.mail_box.BoxStructure(*values)

Bases: Enum

Box structures can be file-based or folder-based.

FOLDER = 1

FILE = 2

class mail_deduplicate.mail_box.BoxFormat(base_class, structure)

Bases: Enum

IDs of all the supported box formats and their metadata.

Each entry is associated to their original base class, and the structure they implement (file-based or folder-based).

From these, we can derive the proper constructor with our own custom DedupMail factory.

Hint

This could be extended in the future to add support for other mailbox formats and sources, like Gmail accounts, IMAP servers, etc.

MAILDIR = (<class 'mailbox.Maildir'>, BoxStructure.FOLDER)

MBOX = (<class 'mailbox.mbox'>, BoxStructure.FILE)

MH = (<class 'mailbox.MH'>, BoxStructure.FOLDER)

BABYL = (<class 'mailbox.Babyl'>, BoxStructure.FILE)

MMDF = (<class 'mailbox.MMDF'>, BoxStructure.FILE)

property constructor

Wrap a subclass of mailbox.Message with our own DedupMail class.

mail_deduplicate.mail_box.FOLDER_FORMATS = (BoxFormat.MAILDIR, BoxFormat.MH)

Box formats implementing a folder-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.FILE_FORMATS = (BoxFormat.MBOX, BoxFormat.BABYL, BoxFormat.MMDF)

Box formats implementing a file-based structure.

Is a tuple to keep natural order defined by BoxFormat.

mail_deduplicate.mail_box.MAILDIR_SUBDIRS = frozenset({'cur', 'new', 'tmp'})

List of required sub-folders defining a properly structured maildir.

mail_deduplicate.mail_box.autodetect_box_type(path)

Auto-detect the format of the mailbox located at the provided path.

Returns a box type as indexed in the BOX_TYPES dictionary above.

If the path is a file, then it is considered as an mbox. Else, if the provided path is a folder and feature the expecteed sub-directories, it is parsed as a maildir.

Todo

Future finer autodetection heuristics should be implemented here. Some ideas:

• single mail from a maildir

• plain text mail content

• other mailbox formats supported in Python’s standard library:

  • MH

  • Babyl

  • MMDF

Return type:

BoxFormat

mail_deduplicate.mail_box.open_box(path, box_format=None, force_unlock=False)

Open a mail box.

Returns a list of boxes, one per sub-folder. All are locked, ready for operations.

If box_format is provided, forces the opening of the box in the specified format. Else, defaults to autodetection.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.lock_box(box, force_unlock)

Lock an opened box and allows for forced unlocking.

Returns the locked box.

Return type:

Mailbox

mail_deduplicate.mail_box.open_subfolders(box, force_unlock)

Browse recursively the subfolder tree of a box.

Returns a list of opened and locked boxes, each for one subfolder.

Skips box types not supporting subfolders.

Return type:

list[Mailbox]

mail_deduplicate.mail_box.create_box(path, box_format, export_append=False)

Creates a brand new box from scratch.

Return type:

Mailbox

mail_deduplicate.strategy module

Strategy definitions.

mail_deduplicate.strategy.select_older(duplicates)

Select all older duplicates.

Discards the newests, i.e. the subset sharing the most recent timestamp.

mail_deduplicate.strategy.select_oldest(duplicates)

Select all the oldest duplicates.

Discards the newers, i.e. all mail of the duplicate set but those sharing the oldest timestamp.

mail_deduplicate.strategy.select_newer(duplicates)

Select all newer duplicates.

Discards the oldest, i.e. the subset sharing the most ancient timestamp.

mail_deduplicate.strategy.select_newest(duplicates)

Select all the newest duplicates.

Discards the olders, i.e. all mail of the duplicate set but those sharing the newest timestamp.

mail_deduplicate.strategy.select_smaller(duplicates)

Select all smaller duplicates.

Discards the biggests, i.e. the subset sharing the biggest size.

mail_deduplicate.strategy.select_smallest(duplicates)

Select all the smallest duplicates.

Discards the biggers. i.e. all mail of the duplicate set but those sharing the smallest size.

mail_deduplicate.strategy.select_bigger(duplicates)

Select all bigger duplicates.

Discards the smallests, i.e. the subset sharing the smallest size.

mail_deduplicate.strategy.select_biggest(duplicates)

Select all the biggest duplicates.

Discards the smallers, i.e. all mail of the duplicate set but those sharing the biggest size.

mail_deduplicate.strategy.select_matching_path(duplicates)

Select all duplicates whose file path match the regular expression provided via the –regexp parameter.

mail_deduplicate.strategy.select_non_matching_path(duplicates)

Select all duplicates whose file path doesn’t match the regular expression provided via the –regexp parameter.

mail_deduplicate.strategy.select_one(duplicates)

Randomly select one duplicate, and discards all others.

mail_deduplicate.strategy.select_all_but_one(duplicates)

Randomly discard one duplicate, and select all others.

mail_deduplicate.strategy.SELECT_NEWEST = 'select-newest'

Time-based strategies.

mail_deduplicate.strategy.SELECT_BIGGEST = 'select-biggest'

Size-based strategies.

mail_deduplicate.strategy.SELECT_NON_MATCHING_PATH = 'select-non-matching-path'

Location-based strategies.

mail_deduplicate.strategy.SELECT_ALL_BUT_ONE = 'select-all-but-one'

Quantity-based strategies.

mail_deduplicate.strategy.STRATEGY_ALIASES = frozenset({('select-all-but-one', 'discard-one'), ('select-bigger', 'discard-smallest'), ('select-biggest', 'discard-smaller'), ('select-matching-path', 'discard-non-matching-path'), ('select-newer', 'discard-oldest'), ('select-newest', 'discard-older'), ('select-non-matching-path', 'discard-matching-path'), ('select-older', 'discard-newest'), ('select-oldest', 'discard-newer'), ('select-one', 'discard-all-but-one'), ('select-smaller', 'discard-biggest'), ('select-smallest', 'discard-bigger')})

Groups strategy aliases and their definitions.

Aliases are great usability features as it helps users to better reason about the selection operators depending on their mental models.

mail_deduplicate.strategy.get_method_id(strategy_id)

Transform strategy ID to its method ID.

mail_deduplicate.strategy.build_method_mapping()

Precompute the mapping of all strategy IDs to their preferred method name, including aliases as fallbacks.

mail_deduplicate.strategy.apply_strategy(strategy_id, duplicates)

Perform the selection strategy on the provided duplicate set.

Returns a set of selected mails objects.