tests package

Submodules

tests.conftest module

tests.conftest.BOX_TYPES = (<class 'mailbox.Maildir'>, <class 'mailbox.mbox'>, <class 'mailbox.MH'>, <class 'mailbox.Babyl'>, <class 'mailbox.MMDF'>, <class 'mail_deduplicate.mail_box.EML'>)

All box classes the make_box/check_box helpers know how to build and read.

Covers every format in BoxFormat, so tests can parametrize over the whole set.

tests.conftest.ALL_FORMATS = MarkDecorator(mark=Mark(name='parametrize', args=('box_format', (<BoxFormat.MAILDIR: (<class 'mailbox.Maildir'>, <BoxStructure.FOLDER: 1>, <class 'mail_deduplicate.mail_box.MaildirDedupMail'>)>, <BoxFormat.MBOX: (<class 'mailbox.mbox'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.mboxDedupMail'>)>, <BoxFormat.MH: (<class 'mailbox.MH'>, <BoxStructure.FOLDER: 1>, <class 'mail_deduplicate.mail_box.MHDedupMail'>)>, <BoxFormat.BABYL: (<class 'mailbox.Babyl'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.BabylDedupMail'>)>, <BoxFormat.MMDF: (<class 'mailbox.MMDF'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.MMDFDedupMail'>)>, <BoxFormat.EML: (<class 'mail_deduplicate.mail_box.EML'>, <BoxStructure.FOLDER: 1>, <class 'mail_deduplicate.mail_box.EMLDedupMail'>)>)), kwargs={'ids': <class 'str'>}))

Sweeps a test over every BoxFormat, to check a behavior shared by all of them.

tests.conftest.DEDUP_ARGS = ('--strategy=select-newest', '--action=delete-discarded', '--dry-run')

A run that exercises every step without touching the mails on disk.

tests.conftest.cache_home(tmp_path)

Point the cache directory of mdedup to a folder of the test.

A run that ends with a bug report saves it in the cache directory. Pointing XDG_CACHE_HOME and LOCALAPPDATA here keeps every test out of the cache of the user running the suite: the tests inside the runner fixture, the tests that call the code directly, and the subprocesses they start.

The variables go through a MonkeyPatch of its own. The monkeypatch fixture would then outlive the runner fixture of each test, and restore a working directory that the runner already removed.

tests.conftest.invoke(runner)

tests.conftest.config()

A fully-populated Config carrying the CLI defaults.

Lets unit tests instantiate Deduplicate and friends without going through the Click command. Mutate the returned mapping to exercise a specific setting.

Return type:

Config

class tests.conftest.MailFactory(**custom_fields)

Bases: object

Create fake mail messages to serve as unittest fixtures.

Help production of either random, customized or deterministic mail message.

Init the mail with custom fields.

You can bypass data normalization by passing the pre-formatted date string with date_rfc2822 custom field instead of date.

render()

Returns the full, rendered content of the mail.

No blank line separates the headers from the body: the parser then ends the headers on the first body line and records a MissingHeaderBodySeparatorDefect, so every fixture built here exercises that path. A test that needs the separator writes its bytes by hand.

static random_string(length)

tests.conftest.make_box(tmp_path)

A generic fixture to produce a temporary box of mails.

The mail container can be created in any format supported by Python’s standard library, plus the custom eml format, via the box_type parameter. Supported values are listed in BOX_TYPES.

tests.conftest.metrics(output)

Extracts the name and value of every metric of a run’s report tables.

Return type:

dict[str, str]

tests.conftest.mail_files(box_path)

Every mail file of a folder-based box, in a stable order.

Dot-prefixed files are left out, as every folder-based format skips them when listing its mails: the temporary link the hardlinking action goes through is one of them.

Return type:

list[Path]

tests.conftest.check_box(box_path, box_type, content=None)

Check the content of a mail box, in any of the BOX_TYPES formats.

Compares the bytes of each mail, as its box hands them out, with the bytes the expected mail was rendered from: a mail rewritten on its way into the box fails the check, where comparing parsed messages would hide the rewrite. An mbox ends every mail with a line break, so an expected mail lacking one gets it.

Counts the occurrences of each mail, so a copy missing from the box or added to it fails the check, where comparing sets would silently deduplicate them.

tests.conftest.assert_one_line_error(result, *messages)

Check that a problem the user can fix ended the run with a one-line error.

The stderr of the run holds each of the messages. It holds no traceback and no invitation to file a bug report, which belong to the errors that are bugs.

Return type:

None

tests.test_action module

tests.test_action.BASE_HEADERS = b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\n'

Headers of a mail written by hand, with CRLF line endings, up to its body type.

tests.test_action.PLAIN_HEADERS = b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n'

Headers of a plain text mail.

tests.test_action.PARTLESS_HEADERS = b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary="Kiwi-Boundary"\r\n\r\n'

Headers of a multipart mail whose body never opens with its boundary, so it parses with no subpart.

tests.test_action.REWRITTEN_BY_THE_GENERATOR = (((b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary="Kiwi-Boundary"\r\n\r\nTaipei: \xe6\x99\xb4\xe5\xa4\xa9, 28 \xc2\xb0C.\r\nTainan: \xe5\xa4\x9a\xe9\x9b\xb2.\r\n',), (), 'partless-8bit'), ((b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary="Kiwi-Boundary"\r\n\r\nSunny.\r\n--Kiwi-Boundary--\r\nRain.\r\n',), (), 'partless-closed'), ((b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary="Kiwi-Boundary"\r\n\r\nPreamble.\r\n--Kiwi-Boundary  \r\nContent-Type: text/plain\r\n\r\nSunny.\r\n--Kiwi-Boundary\r\nContent-Type: text/html\r\n\r\n<p>Sunny.</p>\r\n',), (), 'padded-boundary-left-open'), ((b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nTaipei: sunny\rTainan: cloudy\r\nKaohsiung: rain\r\n',), (), 'bare-cr'), ((b'Date: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nFrom the north came hail.\r\nFrom the south, sun.\r\n',), (), 'from-lines'), ((b'Received: from mx.orchard.example by mail.orchard.example with ESMTPS id abc123 for <quince@orchard.example>; Thu, 11 Oct 2018 09:30:01 +0800\r\nDate: Thu, 11 Oct 2018 09:30:00 +0800\r\nFrom: Clementine <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Weather in Taipei\r\nMessage-ID: <taipei-weather@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nRain.\r\n',), (), 'long-header'))

Mails that Python’s email generator would not write back as they are.

It drops the content of a multipart parsed with no subpart, or fails on it; strips the padding after a boundary and closes a boundary left open; turns a bare CR into a line break; and refolds a header past 78 columns.

tests.test_action.test_action_definition(action)

Test duplicate action definitions.

tests.test_action.SELECTION = [<tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>]

Mails the selection resolves to: the unique mail plus one surviving copy.

tests.test_action.DISCARD = [<tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>]

Mails the discard resolves to: the two other copies.

tests.test_action.DISCARD_AND_UNIQUE = [<tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>]

What a delete-selected run leaves behind.

The mail that has no duplicate belongs to the selection, but deleting it would leave no copy of it anywhere, so only the copy the strategy picked is removed. See: https://github.com/kdeldycke/mail-deduplicate/issues/1053

tests.test_action.test_action_matrix(invoke, make_box, action, needs_export, source_after, export_after, dry_run)

Every action lands the right mails in the right box.

The stats self-check stays consistent, in real and dry-run mode alike.

Copy actions leave the source intact; move and delete actions strip it down to whichever subset they did not act on, with delete-selected also leaving the mail that has no duplicate. Export actions also cover the re-reading of each mail’s bytes from the source boxes, as mails are dehydrated to lightweight stubs once hashed: a regression there would export empty or truncated mails.

tests.test_action.test_trail_lines_tell_a_dry_run_apart(invoke, make_box, action, needs_export, box_type, outcome, dry_run)

Each mail’s line reports what happened to it in a real run only.

A dry run touches no mail, so its lines say what would have happened instead of reading like the record of a run that went through.

tests.test_action.test_trail_lists_mails_in_a_stable_order(invoke, make_box, box_type)

The action trail lists its mails in the same order on every run.

The targets of an action are a set, whose iteration order changes from one process to the next: left unsorted, two runs on the same boxes printed the same lines in different orders, which no diff of two runs survives. The order the run sorts them by is the one a reader compares two runs with: the box path, then the mail’s ID within it.

tests.test_action.test_delete_selected_spares_mails_without_duplicates(invoke, make_box, jobs)

A unique mail survives a delete-selected run that has nothing else to act on.

The two copies here differ by far more than the default size threshold, as a mail carrying an attachment differs from the same mail without one. Their set is skipped, which leaves the selection made of the mail that has no duplicate alone. Deleting the whole selection then wipes the one mail the run was never asked to touch, and keeps the two it was. Covers both the sequential and the parallel selection paths, as each merges its own verdict back. See: https://github.com/kdeldycke/mail-deduplicate/issues/1053

tests.test_action.test_assert_stats_consistent_passes(config)

A satisfied assertion returns quietly, without exiting.

tests.test_action.inodes(box_path)

The distinct files backing the mails of a folder-based box.

One inode per mail before any linking, one per group of linked copies after.

Return type:

set[int]

tests.test_action.test_hardlink_discarded_collapses_copies_onto_one_file(invoke, make_box, jobs)

Discarded copies keep their own name and place, backed by the kept copy’s file.

The unique mail keeps a file to itself.

Runs over both selection paths: pairing each discarded mail with the copy it is linked to is recorded per duplicate set, which a parallel run settles in worker processes and merges back afterwards.

tests.test_action.test_hardlink_discarded_dry_run_leaves_files_alone(invoke, make_box)

A dry run reports on the links it would create without creating any.

tests.test_action.DIFFERING_COPIES = [<tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>]

Two mails sharing a hash, as the body is left out of it by default, but backed by files differing byte for byte.

tests.test_action.test_hardlink_discarded_gates_on_byte_equality(invoke, make_box, extra_args, distinct_files)

Copies that are not byte-for-byte identical are only linked when asked for.

Linking swaps the discarded mail’s own content for the kept copy’s.

tests.test_action.test_hardlink_discarded_is_idempotent(invoke, make_box)

A second run finds the copies already sharing a file and leaves them be.

tests.test_action.test_hardlink_that_cannot_be_created_leaves_the_mail(invoke, make_box, monkeypatch)

A link that cannot replace its mail leaves the mail as it was.

It removes the temporary link it went through, and counts the mail as skipped.

tests.test_action.test_hardlink_discarded_leaves_file_based_box_alone(invoke, make_box)

A mail packed into the box’s own file has nothing of its own to link.

The box comes out byte for byte as it went in.

tests.test_action.test_export_keeps_mail_bytes(invoke, tmp_path, raw, action, export_format)

A mail reaches every export format as the bytes its source holds.

Only its line endings are reduced to LF, and an mbox or mmdf box quotes the body lines starting with From, as its format requires.

See: https://github.com/kdeldycke/mail-deduplicate/issues/1090

tests.test_action.DELIVERY_TIME = 1600000000

Modification time of a mail file, far enough from now to tell the two apart.

tests.test_action.test_export_keeps_maildir_metadata(invoke, tmp_path, action, subdir, info)

A maildir mail keeps its folder, flags and delivery time in a maildir export.

The stdlib files a mail given as bytes in new, with no flags and the time of the run, which would mark every exported mail unread and delivered today.

tests.test_action.test_export_keeps_mbox_envelope(invoke, tmp_path, action, export_format)

An mbox mail keeps its envelope From line in an mbox or mmdf export.

The stdlib fabricates one naming MAILER-DAEMON and the time of the run for a mail that comes without it.

tests.test_action.test_export_keeps_mh_sequences(invoke, tmp_path, action)

An MH mail keeps its sequences in an MH export.

The stdlib enrolls a mail given as bytes in no sequence.

tests.test_action.test_export_translates_no_metadata_across_formats(invoke, tmp_path)

A maildir mail copied into an mbox carries its bytes and nothing else.

Its flags have no place in an mbox but a Status header, which would be an addition to the mail: they are dropped, not translated.

tests.test_action.test_unique_only_source_is_exported(invoke, make_box)

A source made only of unique mails still exports all of them.

A set of one mail is auto-selected, not skipped as “all mails were selected”.

See: https://github.com/kdeldycke/mail-deduplicate/issues/843 and https://github.com/kdeldycke/mail-deduplicate/issues/599

tests.test_bug_report module

tests.test_bug_report.DOOMED_ID = '<doomed@orchard.example>'

Message-ID of the test mail that the patched code fails on.

tests.test_bug_report.MAIL_FACTORY_CONTENT = ('foo@bar.com', 'A duplicate mail', 'Да, они летят.')

Header values and body of every MailFactory mail, which no report may show.

tests.test_bug_report.PERSONAL = ('Clementine Orchard', 'clementine@orchard.example', 'quince@orchard.example', 'Picnic in Lyon', 'pear-123', 'Fig-Boundary-77', 'pommes', 'plum tart', 'harvest-plan.pdf')

Names, addresses and words the test mails spell out, and no fingerprint shows.

tests.test_bug_report.PREAMBLE_MAIL = b'From: Clementine Orchard <clementine@orchard.example>\r\nTo: quince@orchard.example\r\nSubject: Picnic in Lyon\r\nDate: Thu, 11 Oct 2018 09:30:00 +0800\r\nMessage-ID: <pear-123@orchard.example>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary="Fig-Boundary-77"\r\n\r\nG\xc3\xa2teau aux pommes pr\xc3\xa9par\xc3\xa9.\r\n--Fig-Boundary-77\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Transfer-Encoding: 8bit\r\n\r\nBring the plum tart.\r\n--Fig-Boundary-77\r\nContent-Type: application/pdf; name="harvest-plan.pdf"\r\nContent-Disposition: attachment; filename="harvest-plan.pdf"\r\nContent-Transfer-Encoding: base64\r\n\r\nJVBERi0xLjQgYXBwbGVzIGFuZCBwZWFycw==\r\n--Fig-Boundary-77--\r\n'

A multipart mail with 8-bit UTF-8 bytes in its preamble, as in https://github.com/kdeldycke/mail-deduplicate/issues/1085

tests.test_bug_report.EIGHT_BIT_MAIL = b'From: Clementine Orchard <clementine@orchard.example>\nTo: quince@orchard.example\nSubject: Picnic in Lyon, \xc3\xa0 midi\nDate: [mail_date]\nMessage-ID: <pear-123@orchard.example>\nContent-Type: text/plain; charset=us-ascii\n\nBring the plum tart, \xc3\xa0 bient\xc3\xb4t.\n'

A mail with 8-bit header values, a body its charset cannot decode and a Date that is no date, as in https://github.com/kdeldycke/mail-deduplicate/issues/954

tests.test_bug_report.report_block(output)

The lines between the two markers of the bug report in a run’s output.

Return type:

str

tests.test_bug_report.saved_report(output)

The content of the file that the last line of a run’s output names.

The file sits in the reports folder of the cache, and holds the report as it is printed, from its headline to its end marker, without its own path.

Return type:

str

tests.test_bug_report.fail_on_doomed(monkeypatch, method, error)

Make a method of the mails raise error on the mail carrying DOOMED_ID.

Return type:

None

tests.test_bug_report.worker_state(monkeypatch)

Isolate the state a worker function sets up when called in the test process.

tests.test_bug_report.test_fingerprint_shows_no_content(raw)

A fingerprint shows no name, address, subject, body or file name.

tests.test_bug_report.test_fingerprint_shows_mime_structure()

The structure of a multipart mail shows, down to its preamble.

The preamble is what https://github.com/kdeldycke/mail-deduplicate/issues/1085 tripped on.

tests.test_bug_report.test_fingerprint_shows_8bit_values()

8-bit header values and an undecodable body show by their kind of bytes.

A Date shows as it is, even when it is no date.

tests.test_bug_report.test_fingerprint_describes_broken_mails(raw, expected)

A broken mail still gets a fingerprint, which shows what is wrong with it.

tests.test_bug_report.test_fingerprint_shows_header_values(raw, expected)

Header values show by their shape, and a Content-Type by its structure.

A Date shows as it is, but for its comments, up to a capped length.

tests.test_bug_report.test_fingerprint_masks_comments(name)

Each header shown as it is shows its comments by their shape alone.

tests.test_bug_report.test_fingerprint_failure_is_reported(monkeypatch)

A mail the fingerprint cannot describe gets a line saying so.

The bug report still renders.

tests.test_bug_report.test_bytes_recovered_from_surrogates()

The surrogate parsing leaves for a non-ASCII byte turns back into that byte.

A surrogate from no byte shows as the invalid UTF-8 it is.

tests.test_bug_report.test_fingerprint_length_is_capped()

A mail with hundreds of headers does not flood the report.

tests.test_bug_report.test_issues_url_matches_package_metadata()

The URL the report gives is the Issues URL of the package.

tests.test_bug_report.test_traceback_holds_no_path()

The traceback of a report names modules and lines, and no file.

tests.test_bug_report.test_traceback_starts_in_the_blamed_block()

The frame of blame() that catches the error stays out of the traceback.

The traceback names a method with its class.

tests.test_bug_report.dive(depth)

Raise an error from depth calls further down.

Return type:

None

tests.test_bug_report.test_traceback_keeps_the_innermost_frames()

A deep traceback keeps the frames closest to the error, and counts the rest.

tests.test_bug_report.test_failure_describes_its_error(error, expected)

A failure names the class of its error, then its message, on one capped line.

The module of the class shows, outside the builtins.

tests.test_bug_report.test_unknown_box_format_shows_its_class()

A mail from no known box format shows in the report by its class name.

tests.test_bug_report.test_failure_log_keeps_a_few_samples()

Every failure is counted, but only the first ones of a kind are kept.

The kept ones carry their traceback.

tests.test_bug_report.test_title_names_the_innermost_function_of_mdedup(monkeypatch, config, owner, name, function)

The title names the innermost function of mdedup the error went through.

The frames of Python, of a library and of a generator expression do not count, as in the traceback of https://github.com/kdeldycke/mail-deduplicate/issues/1117

tests.test_bug_report.KINDS_OF_ERROR = (<class 'ValueError'>, <class 'KeyError'>, <class 'TypeError'>, <class 'IndexError'>, <class 'ZeroDivisionError'>, <class 'OverflowError'>, <class 'AttributeError'>)

Error classes raised from one line, each of which makes a kind of failure.

tests.test_bug_report.test_failure_log_details_a_few_kinds(kinds, summary)

The report details the most frequent kinds of failure and counts the rest.

tests.test_bug_report.test_report_version_matches_version_option()

The report gives the version mdedup --version prints.

That version names the commit of a development build.

A real run, as --version resolves the CLI from the stack, which a CliRunner invoking the command from another module misleads.

tests.test_bug_report.test_agents_ask_their_user_first(monkeypatch, agent)

An AI agent is asked to show the report to its user before filing it.

tests.test_bug_report.test_hashing_error_sets_the_mail_aside(invoke, make_box, monkeypatch, box_format)

A mail that fails hashing is set aside and left untouched.

The others are deduplicated, and the run ends with a bug report.

tests.test_bug_report.test_mail_failing_to_read_is_rejected_or_set_aside(invoke, make_box, monkeypatch, failing, error, outcome, exit_code, counted)

A mail that cannot be read is rejected on a system error, or else set aside.

Either way, it is left untouched while the others are deduplicated.

tests.test_bug_report.test_hash_only_reports_failures(invoke, make_box, monkeypatch)

--hash-only prints the hashes it computed, then reports the failed mails.

tests.test_bug_report.test_debug_log_holds_traceback_of_samples(invoke, make_box, monkeypatch)

At the debug level, the log holds the full traceback of each failure.

These are the failures the report details.

tests.test_bug_report.test_report_masks_paths_and_regexps(invoke, make_box, monkeypatch)

The options of a report show the values that drive the run.

Paths and regular expressions, which can name folders and people, are replaced.

tests.test_bug_report.test_selection_error_skips_the_set(invoke, make_box, monkeypatch, box_type)

A set that fails selection is skipped, and none of its mails is acted upon.

tests.test_bug_report.test_action_error_names_its_mail(invoke, make_box, monkeypatch, tmp_path, action, patched, step, function)

An unexpected error in an action stops the run with a bug report.

The report names and describes the mail the action was working on.

tests.test_bug_report.test_action_system_error_is_one_line(invoke, make_box, monkeypatch)

An OSError in an action is a system problem, which the user can fix.

The run ends with a one-line error naming the mail, and no bug report.

tests.test_bug_report.test_hashing_worker_rejects_or_sets_aside_its_mail(config, make_box, monkeypatch, worker_state, failing, error, rejection)

A worker process answers with strings, so one failing mail spares the pool.

It rejects a mail it cannot read or hash for a reason the user can fix, as this process does, and sends any other error back as a failure.

tests.test_bug_report.test_hashing_worker_leaves_a_mail_with_too_few_headers_unhashed(config, make_box, monkeypatch, worker_state)

A worker sends back why a mail has no hash, and the parent keeps the mail.

The verdict travels in a field of its own, apart from a rejection, as the parent keeps this mail in the selection where it drops a rejected one.

tests.test_bug_report.test_selection_worker_skips_its_set(config, make_box, monkeypatch, worker_state, failing)

A worker skips a set it cannot settle, and accounts for all of its mails.

The parent merges that verdict back, and keeps the failure of a bug for the report.

tests.test_bug_report.test_statistics_self_check_reports_a_bug(config, caplog, operator, found, retained)

Statistics that do not add up, or that the check cannot compare, end the run.

The run exits with code 115 and a bug report.

tests.test_bug_report.test_saved_reports_never_overwrite()

Each report gets a file of its own, even two saved within the same second.

tests.test_bug_report.test_report_stands_alone_when_it_cannot_be_saved(caplog)

A report the cache directory cannot hold is printed all the same.

tests.test_cache module

tests.test_cache.count_hashes(monkeypatch)

Counts the mails actually hashed, which a cache hit must not be.

Restoring a mail from the cache skips its parsing entirely, so hash_key() is the sharpest signal that the file was read rather than served from the database.

tests.test_cache.cache_db(tmp_path)

Path of the hash cache database of a test.

tests.test_cache.cached_run(invoke, cache_db)

Runs DEDUP_ARGS with the hash cache of the test.

Takes the options to add, then the boxes to deduplicate.

tests.test_cache.test_cache_is_off_by_default(invoke, make_box, tmp_path, monkeypatch)

No database is created unless the user opts in.

tests.test_cache.test_default_cache_dir(monkeypatch, platform, xdg_cache_home, local_app_data, expected)

The locations docs/performance.md lists for each platform.

click-extra tests the edge cases of get_cache_dir(), like a relative XDG_CACHE_HOME.

tests.test_cache.test_default_cache_path_sits_under_the_cache_dir()

The database lands in the platform’s cache location, not next to the mails.

tests.test_cache.test_warm_run_matches_uncached_run(invoke, cached_run, make_box, box_format, jobs, count_hashes)

A run served from the cache must decide exactly what an uncached run does.

The report accounts for every mail through the four steps, so a stale hash would surface as a different grouping and different metrics. Swept over every format, as each one locates its mails differently, and over the sequential and parallel hashing paths, as only the former reads the cache one mail at a time.

MH, Babyl and MMDF are never autodetected, so the format is forced.

tests.test_cache.test_settings_change_discards_the_cache(cached_run, make_box, count_hashes, changed_option)

Every option feeding a cached value invalidates the whole database.

An entry produced under different hashing settings cannot be told apart from a valid one by its staleness key alone, so none of them may survive.

tests.test_cache.test_modified_mail_is_rehashed(cached_run, make_box, count_hashes)

A mail whose file changed must not be served from the cache.

Maildir keys survive a rewrite, so only the staleness key stands between a modified mail and a hash computed from its previous content.

tests.test_cache.test_edited_file_based_box_invalidates_every_mail(cached_run, make_box, count_hashes)

Editing an mbox invalidates all of its mails, not just the ones that moved.

Mails of an mbox share the box’s file and are keyed by byte offsets, which shift as soon as a mail is added or removed.

tests.test_cache.test_mail_vanishing_mid_run_is_rejected(invoke, make_box, cache_db, monkeypatch, cache_args)

A mail removed between the box listing it and the run reading it is rejected.

Regression test: the hashing step used to walk the box with iteritems(), which swallows that race. Walking the keys instead, so a cached mail is never read, has to keep handling it rather than dying on a KeyError. Rejecting the mail keeps it in the statistics, as a --jobs worker does for a file that is gone.

tests.test_cache.test_unusable_cache_does_not_abort_the_run(invoke, make_box, tmp_path)

A database that cannot be opened is reported and skipped.

The cache only ever saves work, so an unwritable location, a read-only home or a full disk must not take the deduplication down with it.

tests.test_cache.test_corrupt_cache_is_skipped_and_closed(cached_run, cache_db, make_box, monkeypatch)

A file that is not a database is reported and skipped.

Its connection is closed.

SQLite opens any file, and only fails on the first statement it runs there, so the connection exists by then and must not stay open for the rest of the run.

tests.test_cache.test_unhashed_mails_are_not_cached(invoke, make_box, cache_db)

A mail with too few headers has no hash to store, so it is read every run.

tests.test_cache.cached_rows(cache_db)

Number of mail entries held by a cache database.

The connection of sqlite3 commits on leaving a with block, but stays open: closing() is what closes it.

Return type:

int

tests.test_cache.test_deleted_mails_are_pruned(cached_run, cache_db, make_box, monkeypatch, write_batch)

Mails removed from a box must not keep their entry forever.

A cache nobody prunes only grows, holding hashes for mails that no longer exist. Batches of one flush the writes and the sightings of a run as they come, as a corpus bigger than a batch does.

tests.test_cache.test_vanished_box_is_pruned(cached_run, cache_db, make_box)

A box that was deleted or moved takes all of its entries with it.

tests.test_cache.test_unvisited_boxes_are_left_alone(cached_run, cache_db, make_box)

Only the boxes a run opens are pruned mail by mail.

Every mail of a box nobody opened is absent from the run’s sightings for the plain reason that nobody looked, which is no evidence that it is gone. Pruning on that basis would empty the cache of every box not passed on the command line.

tests.test_cache.test_run_survives_a_database_held_by_another_run(cached_run, cache_db, make_box)

A database another run is writing must not take this one down.

Runs sharing a cache contend on the write burst each performs at the end. By then the deduplication is done, so losing that race may only cost the next run its head start. Simulated by holding the write lock for longer than the loser is willing to wait.

tests.test_cache.test_cache_survives_a_corpus_growing_between_runs(cached_run, make_box)

Mails added to a folder-based box after a run join the cached ones.

tests.test_cli module

tests.test_cli.acted_upon(result)

The lines of a dry run’s action trail, one per mail it would act on.

Return type:

list[str]

tests.test_cli.test_bare_call(invoke)

tests.test_cli.test_no_progress_bar_label_off_a_terminal(invoke, make_box, label)

Off a terminal, a run prints no progress bar, and so none of its labels.

Click prints a hidden bar’s label once instead of the bar, which put a bare line repeating the step heading in every log, pipe and capture.

tests.test_cli.test_no_log_line_is_left_empty(invoke, make_box)

Every log line carries its message on the line of its level prefix.

tests.test_cli.test_early_export_file_check(invoke, make_box, tmp_path)

Ensures the export file is tested for existence before any process is ran.

See: https://github.com/kdeldycke/mail-deduplicate/issues/119

tests.test_cli.test_theme_styles_runtime_output(invoke, make_box, theme_id)

The active --theme must style runtime output, not just the help screen.

Regression test: the theme used to be captured once at import time via get_default_theme(), so --theme was ignored everywhere but --help.

tests.test_cli.test_cli_test_suite()

Run the TOML black-box suite (cli-test-suite.toml) against the installed mdedup.

Each case is executed as a subprocess by click-extra’s test-suite runner, so this exercises the real entry point (version reporting, help screen rendering).

Marked once: the CI matrix drives the same TOML suite in every cell through click-extra’s runner, so this Python-level wrapper only needs a single executor.

tests.test_cli.test_parallel_run_matches_sequential(invoke, make_box, strategy)

A run with –jobs > 1 must decide what the sequential default decides.

Both the hashing and the selection are handed to worker processes, and both yield their results in submission order, so the grouping, the statistics and the report must come out identical at any job count.

tests.test_cli.test_memory_stays_bounded_while_hashing(invoke, make_box, jobs)

Peak memory must track the few mails in flight, not the whole corpus.

Regression test for the out-of-memory conditions reported on big boxes: mails used to be retained fully parsed for the whole run, so memory grew linearly with the corpus. They are now dehydrated as soon as they are hashed, and the parallel path materializes mails in bounded batches instead of the whole corpus. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

tests.test_cli.test_hash_only_prints_headers(invoke, make_box, box_type)

--hash-only must print each mail’s canonical headers and hash, not crash.

Regression test: the display loop referenced a non-existent mail.pretty_headers attribute, so --hash-only died with AttributeError on the first mail. See: https://github.com/kdeldycke/mail-deduplicate/issues/1004

tests.test_cli.test_single_hash_header_needs_no_minimal_flag(invoke, make_box)

A single --hash-header must work without a separate minimal-headers flag.

Regression test: narrowing the hash below four headers used to raise “Provided number of headers to hash (1) is less than the minimal required number of headers (4)” and then reject every mail. The floor is now derived as min(4, number of --hash-header values), so a lone header is enough. See: https://github.com/kdeldycke/mail-deduplicate/issues/974

tests.test_cli.test_too_few_headers_leaves_mail_unhashed(invoke, make_box)

A mail carrying fewer hash headers than the floor gets no hash.

The warning renders the headers that were found, so the mail can be inspected.

The floor is derived as min(4, number of --hash-header values), so asking for four headers the mail does not carry leaves it one below the floor.

tests.test_cli.applied_options(result, preset)

The options a run says it applies for a preset.

They sit on the log line after the one announcing the preset.

Return type:

str

tests.test_cli.preset_args(preset)

The options a preset stands for, as arguments of a command line.

Return type:

list[str]

tests.test_cli.test_preset_is_an_alias_of_its_options(invoke, make_box, preset, duplicates)

A run with a preset decides what a run with the options it stands for decides.

The box holds what tells the presets apart: two identical copies, a near copy sharing every hashed header, and a copy a gateway dated two days later. Each preset groups more of them than the one before.

tests.test_cli.test_preset_yields_to_an_option_set_by_the_user(invoke, make_box, args, env, overriding, dropped)

An option the user set keeps its value over the one its preset gives it.

The run names each option that overrides the preset, and leaves the preset’s value for it out of the options it applies.

tests.test_cli.test_preset_takes_some_options_without_a_warning(invoke, make_box)

Two kinds of options set over a preset raise no warning.

A value equal to the preset’s changes nothing. The strategy decides which copy stays, not what counts as a copy, so it replaces the preset’s in silence.

tests.test_cli.test_preset_thresholds_stay_on_without_its_hashing(invoke, tmp_path)

Replacing a hashing option of the conservative preset keeps the thresholds on.

Under --hash-body skip, the hash covers the header block alone, so two mails with the same headers and different bodies share it. The preset turns the thresholds off only with its own hashing options, so here they stay on and skip the set.

tests.test_cli.test_preset_thresholds_rely_on_its_hashing(invoke, make_box, args, kept)

The thresholds of the conservative preset apply only with its hashing options.

Replacing a hashing option, directly or through the option linked to it, leaves the thresholds at their defaults. A user value equal to the preset’s keeps them.

tests.test_cli.test_raw_headers_refuse_a_header_list(invoke, make_box)

--raw-headers and --hash-header set together contradict each other.

tests.test_cli.test_conservative_preset_keeps_near_copies(invoke, make_box)

Only identical copies are deduplicated: a near copy is left untouched.

The same three mails collapse into one set under the default hashing, as the near copy shares every hashed header and differs by a footer the thresholds allow. The footer sits on the body’s single line: MailFactory indents the first line of a body alone, so a second line would break the mail apart.

tests.test_cli.test_raw_headers_hash_a_mail_lacking_headers(invoke, tmp_path)

No minimal number of headers applies to the raw headers.

The same mail gets no hash under the default hashing: see test_mail_with_too_few_headers_is_kept.

tests.test_cli.test_raw_headers_hash_only_shows_every_header(invoke, make_box)

Under --raw-headers, --hash-only lists every header of a mail.

tests.test_cli.test_conservative_preset_parallel_run_matches_sequential(invoke, make_box)

The conservative preset picks the same copies in a worker process.

tests.test_cli.test_headers_table_rendered_only_when_logged(invoke, make_box, monkeypatch)

The canonical-headers table is not built at the default verbosity.

Rendering it goes through tabulate and costs more than the hash itself, yet any level above debug discards it.

tests.test_cli.test_invalid_option_value_rejected(invoke, make_box, args, message)

An option value that cannot work is refused at parse time.

tests.test_cli.test_duplicate_source_rejected(invoke, make_box)

The same mail source given twice is refused.

The source path is the key that ties a mail back to its origin.

tests.test_cli.test_hash_only_warns_about_ignored_options(invoke, make_box)

–hash-only ignores the options of the selection and action steps it skips.

Any of them the user set is reported.

tests.test_cli.test_main_entrypoint_reports_version(monkeypatch, capsys)

The main() indirection runs the CLI end to end: --version exits cleanly.

tests.test_cli.test_main_refuses_to_run_in_a_worker(monkeypatch, enter_worker)

main() exits before it imports the CLI, in both states a worker reaches it.

tests.test_cli.UNGUARDED_LAUNCHER = 'import multiprocessing\nimport sys\n\nimport mail_deduplicate.__main__ as user_main\n\n# Each spawn worker imports this file again, which calls main() in it.\nmultiprocessing.set_start_method("spawn", force=True)\nsys.exit(user_main.main())\n'

A launcher shaped like the one Nuitka writes for --main-entry-point, with no __name__ == "__main__" guard around its call.

tests.test_cli.test_unguarded_launcher_stops_in_each_worker(tmp_path)

A launcher calling main() on import stops in each --jobs worker.

It stops before the CLI there opens a box. The parent then fails on its broken pool.

The run cannot multiply processes, guard or not: CPython refuses to start a process from a worker that is still importing its parent’s main module.

tests.test_config module

Configuration file loading, validation and export.

tests.test_config.make_config(tmp_path)

Write a TOML configuration file and return its path as a string.

tests.test_config.test_config_file_drives_run(invoke, make_box, make_config)

Strategy, action and dry-run picked up from a configuration file.

tests.test_config.test_config_kebab_case_keys(invoke, make_box, make_config)

Kebab-case keys, the TOML convention matching the CLI flags, are accepted.

tests.test_config.test_config_unknown_key_rejected(invoke, make_box, make_config)

A typo in the configuration file aborts the run instead of being ignored.

tests.test_config.test_config_preset(invoke, make_box, make_config)

A preset picked up from a configuration file applies its options.

tests.test_config.test_config_value_overrides_a_preset(invoke, make_box, make_config)

A configuration file value overrides the preset, and the run says so.

A forgotten key would otherwise change a run that reads as conservative.

tests.test_config.test_config_mail_sources_blocked(invoke, make_box, make_config)

Mail sources are command-line only: a config file cannot point at boxes.

tests.test_config.test_config_values_checked(invoke, make_box, make_config, setting, message)

Config-sourced values go through the checks of command-line ones.

tests.test_config.test_cli_overrides_config(invoke, make_config)

Command-line values take precedence over configuration file values.

tests.test_config.test_pyproject_autodiscovery(invoke, make_box, monkeypatch, tmp_path)

A pyproject.toml with a [tool.mdedup] section is discovered from cwd.

tests.test_config.test_params_reflects_autodiscovered_config(invoke, monkeypatch, tmp_path)

–params shows values sourced from an autodiscovered configuration file.

tests.test_config.test_export_config_toml_roundtrip(invoke, make_config)

The TOML export names every configurable option and loads back cleanly.

Loading the exported file back under strict mode is the round-trip guarantee: every key the export emits is recognized by the CLI.

tests.test_docs module

Checks on the docs pages that list the test suite, and on the readme assets.

tests.test_docs.TESTS_DIR = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate/tests')

Directory of the test suite.

tests.test_docs.PROJECT_ROOT = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate')

Repository root.

tests.test_docs.TESTS_PAGE = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate/docs/tests.md')

Documentation page with one section per module of the test suite.

tests.test_docs.ASSETS_DIR = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate/docs/assets')

Directory holding the pictures the readme and the docs embed.

tests.test_docs.CAPTURE_SCRIPT = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate/docs/cli_screenshots_update.py')

Script drawing the CLI captures the readme and the docs embed.

tests.test_docs.DESIGN_PAGE = PosixPath('/home/runner/work/mail-deduplicate/mail-deduplicate/docs/design.md')

Documentation page with the table of the presets.

tests.test_docs.NO_DOCS = 'The documentation sources are not part of this tree.'

Why a test reading docs/ skips in the sdist, which ships the tests alone.

tests.test_docs.test_design_page_spells_out_each_preset(preset)

docs/design.md lists the options each preset stands for.

The table of the page is written by hand, so a preset changing its options has to change the table too.

tests.test_docs.test_every_test_module_has_a_section()

docs/tests.md has one section per module of the test suite, and no other.

No job generates those sections: a module added to tests/ stays out of the published page until someone writes its section.

tests.test_docs.test_every_cli_capture_is_embedded()

Each capture docs/cli_screenshots_update.py writes is embedded in a page.

That page is the readme or a docs page, and no SVG in docs/assets/ outlives the capture that produced it.

The stems of the captures are the file names the pages link, so renaming one in the script breaks a page silently: GitHub serves a 404 where the readme’s picture was, and nothing in a docs build reads that URL. The reverse direction rots the same way: a capture dropped from the script leaves its SVG in the tree, unreferenced and never refreshed again.

tests.test_docs.test_pages_show_the_command_their_picture_runs()

A page embedding a capture shows the command line the capture ran.

The picture and the command sit side by side, and a reader compares them: an option changed in one and not the other reads as the picture disagreeing with the instructions. Nothing compares the two, since the command is prose to every other check here.

A capture forcing an error is the exception: its command reproduces nothing on a reader’s boxes, so its page explains the picture instead.

tests.test_docs.test_cli_capture_prompts_fit_their_window()

Every captured command line fits its window on one row.

docs/cli_screenshots_update.py rejects an over-long invocation before it draws one, but the guard only runs when someone recaptures by hand. This runs the same guard over the committed captures instead, so an option added to a capture and not to its picture cannot land unnoticed.

A wrapped prompt reads as two commands, and the second one opens mid-option: the picture would teach an invocation nobody can type.

tests.test_docs.test_captures_run_outside_any_agent(monkeypatch)

The captures run without any variable that makes mdedup detect an agent.

The bug report asks a detected agent to show it to its user, a line no person running mdedup sees. Each agent extra-platforms detects reads a variable of its own, so a new release can name one the capture script does not drop.

tests.test_docs.test_self_referencing_urls_name_a_file_the_tree_holds()

A raw.githubusercontent.com URL to this repository names a file it carries.

This covers the URLs that point at the main branch.

Nothing else resolves these: they are fetched by a reader far from any test, on GitHub and on PyPI, where the readme is the whole page. Renaming a file under docs/assets/ therefore breaks every surface at once, with no build and no link check in between.

A URL pinned to a tag names a historical tree and is skipped, the working copy being no evidence about what that tag held.

tests.test_mail module

tests.test_mail.create_mail_from_bytes(raw)

Parse raw bytes into a DedupMailMixin, body and all.

Keeps the parsed payload, so the body-derived properties (body_lines, size, the body hashes) can be exercised.

Return type:

DedupMailMixin

tests.test_mail.create_mail_with_headers(*headers)

Helper to create a DedupMailMixin object with custom headers.

Parameters:

headers (tuple[str, str | bytes | Header]) – (name, value) tuples to set as mail headers. Values can be strings, bytes, or email.header.Header objects.

Return type:

DedupMailMixin

tests.test_mail.test_body_hashes_distinguish_and_normalize()

hash_decoded_body reflects any difference in the decoded body.

hash_normalized_body ignores whitespace, but still separates different bodies.

tests.test_mail.test_raw_body_hash_reads_the_bytes_of_the_box(tmp_path, config)

hash_raw_body hashes the body bytes as stored, line endings aside.

Two copies of a text in different transfer encodings hash apart under raw and alike under decoded. Two copies differing in their line endings alone hash alike under both, as every box format rewrites line endings.

tests.test_mail.test_raw_headers_hash_the_header_block(tmp_path, config)

Under --raw-headers, hash_key() digests the header block as stored.

Line endings are reduced to LF first. No header is required: a mail with none of the hashed headers still gets a hash, where the default hashing gives it none.

tests.test_mail.test_raw_bytes_come_from_the_file_or_the_box(tmp_path)

A worker reads a mail’s bytes from the path it opened, the parent from the box.

A mail attached to neither cannot be read again.

tests.test_mail.test_raw_body_starts_where_the_parser_ends_the_headers(tmp_path, raw, body)

raw_body() starts after the first blank line, or where a header line ends.

The parser treats a line that is neither a header nor a continuation as the start of the body when the blank line is missing, and so does the raw body. A bare CR ends a line for both, so the body of a mail with CR line endings is not lost. The header block holds the rest: the two join back into the mail.

tests.test_mail.test_size_counts_the_bytes_of_the_decoded_body()

size is the UTF-8 byte length of the decoded body, not its character count.

Copies of a mail that differ in their transfer encoding alone measure the same.

tests.test_mail.test_body_lines_gathers_multipart_preamble_and_epilogue()

body_lines collects the preamble, every leaf part and the epilogue.

It skips the multipart container itself.

tests.test_mail.multipart_mail(preamble=b'Forecast.', part=b'Sunny.', epilogue=b'End of forecast.')

Wrap one 8-bit text/plain part in a multipart/mixed mail.

Return type:

bytes

tests.test_mail.test_body_lines_decodes_each_part(raw, expected)

Every part of a mail is decoded into its body lines, whatever its bytes.

Both body hashes are computed from them.

The parser keeps each non-ASCII byte as a lone surrogate. get_payload() decodes the parts, and body_lines decodes the preamble and epilogue as UTF-8, instead of leaving surrogates that crash both body hashes with a UnicodeEncodeError. A byte that is not valid UTF-8 becomes U+FFFD.

A non-text part is passed through undecoded. A text part with no charset is tried as ASCII, then as UTF-8. A text part its charset cannot decode comes back as its raw payload instead of crashing.

See: https://github.com/kdeldycke/mail-deduplicate/issues/1085

tests.test_mail.test_header_normalization(header_name, values, expected)

Test header value normalization.

tests.test_mail.test_raw_8bit_headers_hash_apart()

Two headers that differ in their raw 8-bit bytes alone hash apart.

get_all() hands such a header over with every non-ASCII byte replaced by U+FFFD, under which Noël and Noêl in latin-1 read alike.

tests.test_mail.test_raw_8bit_date_parses()

A Date holding raw 8-bit bytes parses like its ASCII spelling.

get() hands such a header over as a Header, which parsedate_tz() cannot split. See: https://github.com/kdeldycke/mail-deduplicate/issues/1117

tests.test_mail.test_normalizers_read_raw_8bit_bytes(header_id)

Every header with its own normalization reads a value holding raw 8-bit bytes.

A normalizer that reads its header again with get() gets a Header, which string functions reject: header_values() gives the text instead.

tests.test_mail.test_unparsable_date_returns_none(header)

A missing, unparsable or out-of-range Date gives no timestamp, not a crash.

tests.test_mail.test_mail_path_derived_from_box(make_box, box_format)

A mail’s location comes from its box’s own index, not from opening its file.

Its repr renders that location, ready to be copy-pasted for inspection.

Folder-based boxes give every mail its own file, so their mails carry distinct paths below the box’s directory, which their repr shows. File-based boxes pack every mail into the box’s single file, so their mails all share its path, and their repr adds the mail ID to tell them apart.

See: https://github.com/kdeldycke/mail-deduplicate/issues/157

tests.test_mail.load_single_mail(make_box, config, box_format)

Helper returning the single mail of a fresh test box, ready for dedup work.

tests.test_mail.test_dehydrate_hydrate_roundtrip(make_box, config, box_format)

Dehydration drops the parsed message, but keeps the scalars of the strategies.

Content-dependent properties must transparently restore the exact same content from the source box.

See: https://github.com/kdeldycke/mail-deduplicate/issues/761

tests.test_mail.test_dehydrated_mail_defers_body_reads(make_box, config, box_format)

A mail dehydrated before its body was decoded re-reads its message on access.

The first body-dependent access reads it from the source box.

tests.test_mail_box module

tests.test_mail_box.stdlib_box_types()

Yields all mailbox types defined in the standard library.

Only collect direct subclasses of the mailbox.Mailbox interface. Ignore mailbox.Mailbox itself and all others starting with an underscore.

Return type:

list[type[Mailbox]]

tests.test_mail_box.LOCKING_FORMATS = MarkDecorator(mark=Mark(name='parametrize', args=('box_format', (<BoxFormat.MBOX: (<class 'mailbox.mbox'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.mboxDedupMail'>)>, <BoxFormat.MH: (<class 'mailbox.MH'>, <BoxStructure.FOLDER: 1>, <class 'mail_deduplicate.mail_box.MHDedupMail'>)>, <BoxFormat.BABYL: (<class 'mailbox.Babyl'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.BabylDedupMail'>)>, <BoxFormat.MMDF: (<class 'mailbox.MMDF'>, <BoxStructure.FILE: 2>, <class 'mail_deduplicate.mail_box.MMDFDedupMail'>)>)), kwargs={'ids': <class 'str'>}))

Sweeps a test over the formats that lock their box: maildir and eml take no lock.

tests.test_mail_box.test_box_format_definition(box_format)

Ensures each box format is correctly defined.

tests.test_mail_box.test_box_formats_cover_stdlib()

Box formats cover every box type of the standard library, plus custom ones.

Custom formats like eml come on top, and each format has one structure.

tests.test_mail_box.test_box_roundtrip(make_box, box_format, mails)

Mails survive a build/read round-trip in each format.

Each format opens through its own custom DedupMail factory.

The duplicates of the mixed box check that no layer collapses identical mails.

tests.test_mail_box.test_dedup_across_formats(invoke, make_box, box_format)

A full dedup run succeeds on every supported format.

MH, Babyl and MMDF are never autodetected, so the format is forced. Babyl also guards a regression: its get_file() hands back a nameless in-memory buffer, which used to crash the extraction of a mail’s path metadata.

tests.test_mail_box.test_run_cut_short_unlocks_boxes(invoke, make_box, monkeypatch, box_format, error)

A run cut short by an error or an interrupt still unlocks its source box.

The next run needs no --force-unlock.

tests.test_mail_box.STOP_POINT = 'Removals queued, waiting to be stopped.'

Line printed by STOPPABLE_RUN once its action has queued its removals.

The ID of the interpreter process follows it, on the same line.

tests.test_mail_box.STOPPABLE_RUN = "\nimport os\nimport sys\nimport time\n\nfrom mail_deduplicate.action import Action\nfrom mail_deduplicate.cli import mdedup\n\nperform = Action.perform\n\n\ndef perform_then_wait(self, dedup):\n    perform(self, dedup)\n    print('Removals queued, waiting to be stopped.', os.getpid(), flush=True)\n    time.sleep(60)\n\n\nAction.perform = perform_then_wait\nmdedup(sys.argv[1:])\n"

An mdedup run that waits to be stopped while it holds the locks of its boxes.

tests.test_mail_box.test_killed_run_needs_force_unlock(invoke, make_box, box_format, stop)

A run stopped by SIGTERM or SIGKILL exits before any cleanup.

Its box stays locked, and only --force-unlock lets the next run through.

On Windows, os.kill() calls TerminateProcess() for any signal, which skips the cleanup too.

tests.test_mail_box.test_refused_lock_keeps_no_file_open(make_box, box_format)

A box refused for an existing lock keeps no file open.

MH.lock() opens .mh_sequences before it takes the lock, and MH.close() only closes that file for a locked box.

tests.test_mail_box.test_refused_subfolder_lock_unlocks_the_folders_before_it(invoke, make_box)

A subfolder locked by another program stops the run.

The folders locked before it are unlocked again: the next run needs no --force-unlock.

tests.test_mail_box.test_create_box_refuses_to_clobber_existing(tmp_path)

create_box refuses an existing path unless append is requested.

This guards a destination box even when reached directly.

tests.test_mail_box.test_nonexistent_path(invoke, source)

tests.test_mail_box.test_unusable_source(invoke, args, messages)

A source that cannot be opened in its format stops the run.

No mail is touched.

tests.test_mail_box.test_verbatim_nested_maildirs(invoke, tmp_path)

Nested plain-directory maildir folders (isync’s Verbatim style) are discovered.

They are found at any depth, without requiring the root to be a maildir itself.

See: https://github.com/kdeldycke/mail-deduplicate/issues/973

tests.test_mail_box.test_verbatim_forced_maildir_format(invoke, tmp_path)

Forcing maildir on a Verbatim-style tree does not crash on the mail-less root.

tests.test_mail_box.test_mixed_maildir_folder_conventions(invoke, tmp_path)

Maildir++ dot-folders and Verbatim plain folders coexist under one root.

Each opens exactly once. Guards against subfolders being skipped so only the root INBOX is processed.

See: https://github.com/kdeldycke/mail-deduplicate/issues/123

tests.test_mail_box.test_eml_autodetect_and_dedup(invoke, tmp_path)

A folder of loose .eml files is autodetected and deduplicated.

Nested directories are included.

See: https://github.com/kdeldycke/mail-deduplicate/issues/760

tests.test_mail_box.test_eml_export_format(invoke, tmp_path)

Deduplicated mails can be exported as a folder of .eml files.

tests.test_mail_box.test_eml_add_keeps_header_folding(tmp_path)

EML.add() writes a message with its headers folded as they were parsed.

Message.as_bytes() refolds every header past 78 columns, where the stdlib boxes write them as they are.

tests.test_mail_box.test_eml_nonexistent_box_raises(tmp_path)

Opening a missing eml folder without creating it fails cleanly.

tests.test_mail_box.test_eml_membership(tmp_path)

in matches existing .eml keys only, and short-circuits on other names.

tests.test_mail_box.test_eml_missing_key_raises_keyerror(tmp_path)

Every accessor raises KeyError for an unknown key, like the stdlib boxes.

tests.test_mail_box.test_eml_folders_unsupported(tmp_path)

eml folders are walked recursively, so the subfolder API is inert.

tests.test_mail_integration module

End-to-end behavior of mail parsing, timestamps and hashing, through mdedup.

Unit-level tests of the mail object live in test_mail.py.

tests.test_mail_integration.DATELESS_HASH_HEADERS = ('--hash-header=message-id', '--hash-header=from', '--hash-header=to', '--hash-header=subject')

Four headers every test mail carries, and no Date: mails that differ by their date alone land in the same duplicate set.

tests.test_mail_integration.test_invalid_date_parsing(invoke, make_box, hash_headers, remaining)

Mails with odd non-standard dates are parsed and grouped into duplicate sets.

tests.test_mail_integration.undated_mail = <tests.conftest.MailFactory object>

A mail whose Date header cannot be parsed into a timestamp.

tests.test_mail_integration.dated_mail = <tests.conftest.MailFactory object>

A mail with a valid Date header.

tests.test_mail_integration.test_missing_date_header_skips_time_strategy(invoke, tmp_path)

Duplicates with no Date header are skipped by time-based strategies.

Some clients save such mails into mbox files. The run does not crash, and names the offending mail rather than failing the whole mailbox silently.

See: https://github.com/kdeldycke/mail-deduplicate/issues/600 and https://github.com/kdeldycke/mail-deduplicate/issues/954

tests.test_mail_integration.test_unparsable_date_skips_time_strategy(invoke, make_box, mails, args)

Time-based strategies skip sets holding mails with no parseable Date header.

They name the offending mails instead of crashing.

See: https://github.com/kdeldycke/mail-deduplicate/issues/132

tests.test_mail_integration.test_unparsable_date_show_diff(invoke, make_box)

Rendering the diff of mails with no parseable Date header does not crash.

tests.test_mail_integration.test_hash_body(invoke, make_box, hash_body, other_body, deduplicated)

–hash-body=skip groups mails by their headers alone, whatever their bodies.

=raw folds every byte of the body into the hash, =decoded folds in its decoded text, and =normalized folds it in but ignores whitespace.

tests.test_mail_integration.test_ctime_time_source(invoke, make_box)

–time-source=ctime dates each mail by the inode change time of its file.

A maildir with distinct file ctimes deduplicates by them.

tests.test_mail_integration.test_mail_with_too_few_headers_is_kept(invoke, tmp_path, action, left_in_box, exported, jobs)

A mail with fewer than the minimal hashable headers gets no hash, and is kept.

It joins no duplicate set, so no strategy rules on it: every action treats it as a mail without duplicates. A copy or a move carries it into the export box, and no deletion removes it. Covers the hashing in worker processes too, which sends the verdict back as a string.

See: https://github.com/kdeldycke/mail-deduplicate/issues/27

tests.test_mail_integration.test_crlf_and_lf_bodies_are_duplicates(invoke, tmp_path, args)

Two mails that differ only by LF or CRLF line endings are duplicates.

Line endings must not count toward size or content differences, nor toward the bytes the conservative preset compares.

See: https://github.com/kdeldycke/mail-deduplicate/issues/844 (fixed by PR 845).

tests.test_performance module

Guards against performance regressions.

Wall-clock timings are too noisy to assert on in CI, so these pin the quantities that actually drive them and are deterministic: how much work a run does per mail, and how much memory it keeps per mail. The figures documented in docs/performance.md and the readme are derived from the same properties, at a scale no test suite should run.

tests.test_performance.RETAINED_BYTES_CEILING = 2000

Upper bound on the bytes the hash index keeps per mail, once hashing is done.

Deliberately loose. A mail stub is mostly its instance dictionary, whose size depends on the interpreter: the same corpus measures 630 to 675 bytes per mail here on Python 3.14, and has been seen at 1,343 on a CI runner of the very version that gives 675 locally. A ceiling tight enough to pin any of those is a ceiling that fails somewhere else.

So this only catches a stub that grew by an order of magnitude, which is what retaining parsed messages would do. The exact guards on what a stub may carry are test_retained_mails_carry_neither_path_nor_payload, which names the attributes, and test_retained_memory_stays_flat_as_the_corpus_grows, which compares the same interpreter against itself. See: https://github.com/kdeldycke/mail-deduplicate/issues/87

tests.test_performance.deep_size(obj, seen=None)

Recursive size of an object graph, counting each object once.

Follows the containers the hash index is made of, and the instance dictionary of the mail stubs it holds. Shared references, like the box and the configuration every mail points at, are naturally counted a single time.

Return type:

int

tests.test_performance.index_footprint(monkeypatch)

Captures what the hash index retains, right after hashing.

tests.test_performance.count_hashing_reads(monkeypatch)

Counts the mail files opened by the hashing step alone.

Later steps legitimately re-read mails they need the body of, so only the reads performed while hashing are counted.

tests.test_performance.test_hashing_reads_each_mail_once(invoke, make_box, count_hashing_reads)

Hashing must open each mail’s file exactly once: the read that parses it.

Regression test: a mail’s path used to be read off a second box.get_file() handle, costing an extra open() and stat() on every single mail on top of the read that already parsed it. The path now comes from the box’s own index. See: https://github.com/kdeldycke/mail-deduplicate/issues/87

tests.test_performance.test_retained_memory_stays_flat_as_the_corpus_grows(invoke, make_box, index_footprint)

What the index keeps per mail must not grow with the number of mails.

Measured against itself on one interpreter, so it holds wherever it runs: a per-mail figure that climbs with the corpus means something is accumulating that should not be. The fixed costs amortize as the corpus grows, so the larger one is allowed to come out slightly lower, never higher.

tests.test_performance.test_retained_memory_does_not_track_mail_size(invoke, make_box, index_footprint)

A corpus of huge mails must retain no more than a corpus of small ones.

Mails are dehydrated once hashed.

Regression test for the out-of-memory reports on big boxes: mails used to be kept fully parsed for the whole run. See: https://github.com/kdeldycke/mail-deduplicate/issues/761

tests.test_performance.test_retained_mails_carry_neither_path_nor_payload(invoke, make_box, index_footprint)

A retained mail keeps its identity and its memoized scalars, nothing else.

Its location is derived from its box on access rather than stored, so burying a box under a long path does not inflate every one of its mails. Its parsed message is gone entirely, which is what keeps memory flat.

tests.test_performance.test_parallel_hashing_only_claims_folder_boxes(make_box, config, box_format)

Only boxes whose mails own a file can be handed to worker processes.

Mails of a file-based box are byte ranges of one file that a single handle seeks through, so several processes would be seeking the same descriptor.

tests.test_performance.test_step_falls_back_when_its_pool_will_not_start(invoke, make_box, monkeypatch, step, verb, initializer)

A pool that cannot be started must not cost the run its results.

Frozen and sandboxed environments can refuse to spawn processes. The step then runs in this one, which is slower and entirely correct.

tests.test_performance.test_single_mail_sets_are_not_handed_to_workers(invoke, make_box, monkeypatch)

A set of one is settled here, without reading anything.

Shipping it to a worker would cost more than deciding it here.

tests.test_performance.test_large_duplicate_sets_do_not_diff_every_pair(invoke, make_box, monkeypatch)

The cost of a duplicate set must not grow with the square of its size.

Regression test: the thresholds were confirmed pair by pair, diffing bodies each time, so a set of 200 copies of one mail ran 19,900 diffs and took longer than 20,000 mails arranged in pairs. Mails carrying the same body are now compared once between them, and the thresholds are settled over the whole set first. See: https://github.com/kdeldycke/mail-deduplicate/issues/87

tests.test_strategy module

tests.test_strategy.test_strategy_enum_has_no_hidden_alias()

The Strategy enum defines no member under a second name or value.

tests.test_strategy.test_strategy_definition(strategy)

Each strategy has a lowercase ID matching its name.

It runs a documented function named after one of the strategies.

tests.test_strategy.test_maildir_dry_run(invoke, make_box, strategy_id, params)

Check no mail is removed in dry-run mode.

tests.test_strategy.test_maildir_strategy(invoke, make_box, strategy, mailbox_input, mailbox_results)

Generic test to check the result of a selection strategy.

tests.test_strategy.test_strategy_is_deterministic(make_box, config, strategy)

Each strategy selects the same mails from the same set, at every pass.

A dry run must name the mails that the real run acts on, and a --jobs worker must decide what the main process decides. Each pass loads the mails anew, as a worker does.

tests.test_strategy.test_quantity_strategies_rule_on_the_first_path(invoke, make_box, strategy, kept_by_first, jobs)

select-one keeps the first copy by path, and select-all-but-one drops it.

The order of the sources on the command line plays no part, and a --jobs worker rules on the same copy.

tests.test_strategy.undated_mail = <tests.conftest.MailFactory object>

A mail with an unparsable Date header, from which no timestamp can be derived.

tests.test_strategy.test_strategy_fallback_resolves_identical_copies(invoke, make_box)

Time-based strategies cannot discriminate byte-identical copies.

The next strategy of the cascade takes over the sets they fail on.

See: https://github.com/kdeldycke/mail-deduplicate/issues/647

tests.test_strategy.test_strategy_fallback_on_missing_timestamp(invoke, make_box)

A set whose mails have no parseable timestamp goes to the next strategy.

The set is not skipped.

tests.test_strategy.test_strategy_cascade_exhausted(invoke, make_box)

A set failing every strategy of the cascade is skipped as a whole.

tests.test_strategy.test_time_strategy_alone_cannot_split_same_timestamp_set(invoke, make_box)

A time-based strategy that selects the whole set skips it, and deletes no mail.

This happens when every mail shares a timestamp, which discriminates nothing. A fallback strategy is the documented way to resolve such sets.

See: https://github.com/kdeldycke/mail-deduplicate/issues/270 and https://github.com/kdeldycke/mail-deduplicate/issues/647

tests.test_strategy.test_strategy_cascade_dedup_aliases(invoke, make_box)

Repeated strategies are collapsed into one, even under their aliases.

tests.test_strategy.PATH_STRATEGIES = 'discard-matching-path, discard-non-matching-path, select-matching-path, select-non-matching-path'

The strategies taking a --regexp, as the constraint errors list them.

tests.test_strategy.test_strategy_cascade_regexp(invoke, make_box, args, messages)

–regexp is required as soon as a path-based strategy is part of the cascade.

It is rejected when no strategy of the cascade makes use of it.

tests.test_strategy.outlier_mail = <tests.conftest.MailFactory object>

A mail whose body size and content exceed the default thresholds against the other fixtures.

tests.test_strategy.test_outlier_eviction_keeps_core_dedup(invoke, make_box)

An outlier mail no longer prevents the deduplication of the true copies.

The outlier exceeds the thresholds against its set.

See: https://github.com/kdeldycke/mail-deduplicate/issues/851

tests.test_strategy.test_dissimilar_pair_still_skipped(invoke, make_box)

A set with no coherent core of at least 2 similar mails is skipped as a whole.

This is the behavior from before outlier eviction.

tests.test_strategy.test_eviction_is_not_transitive(invoke, make_box)

A chain of mails, each within threshold of the next, is not one group.

The mails are not all within threshold of each other. An endpoint is set aside, and the coherent remainder is deduplicated.

tests.test_strategy.test_threshold_checks_disabled(invoke, make_box, extra_args, size_skip_logged, content_skip_logged)

A threshold set to -1 turns off its similarity check, and logs that it did.

The identical pair deduplicates whatever the thresholds; the flags only govern whether the size and content guards run at all.

See: https://github.com/kdeldycke/mail-deduplicate/issues/97

tests.test_strategy.test_no_strategy_skips_duplicate_sets(invoke, make_box)

With no –strategy, duplicate sets are grouped and counted, then skipped whole.

Every mail stays in place.

tests.test_strategy.DISSIMILAR_PAIR = (<tests.conftest.MailFactory object>, <tests.conftest.MailFactory object>)

Two mails sharing every hashed header, with bodies of the same length.

The equal lengths keep the size check quiet, so the content check is what trips.

tests.test_strategy.test_show_diff_starts_from_the_lowest_path(invoke, make_box)

The diff of two mails always runs from the one with the lowest path.

The pairs were walked in the order of a set, which changes from one run to the next, so two runs on the same box printed the same diff in either direction.

tests.test_strategy.test_show_diff_is_colored_by_the_theme(invoke, make_box)

Each line of the diff takes the theme slot its kind reads as.

This is how git diff colors its own.

tests.test_strategy.test_content_difference_counts_bytes(config)

The content difference of two mails is measured in bytes, not characters.

Two bodies of the same length in characters weigh more bytes when one holds non-ASCII characters, and the difference grows by that many bytes.

tests.test_strategy.test_thresholds_read_no_body_the_hash_already_covers(config, monkeypatch, hash_body, reads)

A set hashed on its decoded body is settled without a read.

Every mail of such a set shares the decoded body, from which the size and the diff both follow, so the checks have nothing to compare. A set hashed on headers alone, on its bytes, or on a body stripped of its whitespace still gets its bodies read and compared.

tests.test_strategy.test_size_check_alone_reads_no_body(config, monkeypatch)

With the content check off, the sizes the hashing step kept settle the set.

No body is read again to compare contents that nothing compares.

tests.test_strategy.test_raw_hash_keeps_the_size_safeguard(invoke, tmp_path, hash_body)

The size check still compares mails that share their body bytes.

The same bytes decode to different texts under different transfer encodings, and no hash header covers the encoding by default. Hashed on their headers alone, the control, or on their bytes, both copies land in one set, which their decoded sizes keep from being deduplicated.

tests.test_strategy.test_content_dissimilar_pair_skipped(invoke, make_box, show_diff)

A content-dissimilar set is skipped whole.

–show-diff prints the unified diff of its mails.