This CLI reads all mails from a variety of mailbox formats (
mmdf), then automatically detects, regroup and
act on duplicates.
Duplicate detection is done by cherry-picking certain headers, in some cases doing some minor tweaking of the values to reduce them to a canonical form, and then computing a hash of those headers concatenated together.
In theory, we could rely on the mail’s
Message-ID as a key to identify duplicate mails.
The reality is messier. There is no guarantee that
Message-ID is unique
or even present. Yes, certain broken mail servers which must remain
nameless are guilty of this. 😩
That is why
mdedup propose to identify uniqueness of mails based on an ordered hashed list of
The list of headers to consider can be set with the
The default parameters of the CLI, especially the list of default headers have been crafted to limit the effects of mailing-lists on both the mail headers and body.
Mailing lists effects includes:
having an extra footer in the mail’s body, thus changing the
mails with a new path described by the
Receivedheaders which would not be contained in any copy of the mail saved locally at the time it was sent to the list;
Reply-Toheader even though it’s a bad idea;
adding plenty of other random headers which a copy saved locally at sending-time would not have, such as
Errors-To, and so on;
adding a prefix to the
For added protection against accidentally removing mails due to false positives, duplicates are verified by comparing body sizes and also diff’ing the contents. If the sizes or contents differ by more than a threshold, they are not counted as duplicates.