cypherpunks-legacy archives 1992-2013

Back in July there was discussion about preserving the available archives. I have now made the available archives from 1992-2013 available via Mailman, and placed copies of the available mbox files within. Visit them here: https://lists.cpunks.org/pipermail/cypherpunks-legacy/ The main info page, which links to the sources and a description of what was involved in getting them into Mailman, is here: https://lists.cpunks.org/mailman/listinfo/cypherpunks-legacy Finally, here is the README-processing file that describes what I did, and why I decided to not try to incorporate those older archives with the current list archives at https://lists.cpunks.org/pipermail/cypherpunks/. Suggestions welcome. -------- Processing of Cypherpunks Archives Available archives of the Cypherpunks email list are incomplete, and in fact there is evidence they have been tampered with and/or redacted over the years. This project was to do some basic clean-up of available archives, which are in mbox format, so that they could be ingested and be viewed within Mailman. The archives contain many poorly formed messages, and Mailman defaults to the current date (December 2019, at the time of this writing) when it encounters a problem with the date. So, a separate 'cypherpunks-legacy' list was deployed to make the archives available without overlapping with the current active 'cypherpunks' list, which goes back to July 20 2013. Otherwise, the legacy archive messages would have been peppered into the current archives, in ways that would be difficult to predict or undo. There were especially many anomalies from the larger source, spanning 1999-2015. In addition to many poorly formed messages (i.e., messages that, in one way or another, could not be cleanly ingested with the Mailman 'arch') comment, there were invalid dates, and lines that had an errant "From " at the start. To ready the sources for ingestion to Mailman two automated tools were utilized, followed by some ad hoc edits and changes: 1. 'sortmbox.py' uses a Python library to put messages in by-date order. This proved to be less confusing to Mailman (i.e., fewer messages were inserted to the current month). 2. 'cleanarch' (/var/lib/mailman/bin/cleanarch) is part of the Mailman package. It fixes errant "From " entries at the start of lines. 3. I also used 'sed' to replace invalid dates with valid ones that were in the same ballpark day+time as the message. I found these either when 'arch' complained (such as for dates before the Unix epoch), or when Mailman was showing messages in the future: 's/ 0101 / 1999 /' | sed 's/ 0102 / 1999 /' 's/Thu Dec 31 22:40:39 1903/Thu Jul 5 22:40:39 2018/g' 's/Jan 1904/Jul 2018/' 's/Date: Sun, 1 Apr 2029 03:07:16 +0200/Date: Sat, 31 Mar 2001 15:59:46 -0800/' 's/Date: Fri, 3457 Jan 4 61400:2064:61300 +0200/Wed May 29 15:00:02 2013/' There might have been a few other small edits made within the files, which I didn't record, simply to help Mailman to do a better job of creating browsable archives. 4. I then concatenated all the mbox files (one each for 1992-1998, plus one larger file for 1999-2015), and reran sortmbox.py and cleanarch. 5. I edited the resulting file to remove everything after the new list archives were set up, on July 20 2013. To do the work above, I created a temporary Mailman list, and repeatedly used the 'arch' command to ingest the archives and fix problems. This was an iterative process. Once 'arch' was giving sane output, I created a new Mailman list, 'cypherpunks-legacy.' I put the single unified + fixed mbox file where Mailman tools would find it: /var/lib/mailman/archives/public/cypherpunks-legacy.mbox/cypherpunks-legacy.mbox At this point, I could use the Mailman to slurp the mbox files in, and create the browsable structure. The 'arch' command: /var/lib/mailman/bin/arch --wipe cypherpunks-legacy This served to populate the list archives, which are browsable here: https://lists.cpunks.org/pipermail/cypherpunks-legacy The single large mbox file resulting from the steps above is linked at the top of the Archives page. Here is a direct link: https://lists.cpunks.org/pipermail/cypherpunks-legacy.mbox/cypherpunks-legac... (615MB, containing approximately 180149 messages) Please be aware that Mailman's placement of messages by author, date, subject and thread - including the correct by-month folder - is not perfect. Messages sometimes end up in the wrong place, and sometimes threads are split across different years. Also note that these mbox files did not include attachments. The current Mailman archive does include attachments, but this legacy archive does not. It seems they were not included with the archive input sources (though it's possible some messages have MIME-encoded attachments within them). Anyone interested in doing a serious dig into the archive should also consult the original mbox files. These can be ingested into any capable email client program, and viewed as separate messages. They may be sorted and searched, just like any other email folder. The same sorts of issues as those described above will likely be evident in any email client, and clients will even show a different total number of messages. The types of sorting, editing, and displaying described above would have somewhat different results, if a different toolset is used. These archives are freely available, and the effort to make them available via Mailman is freely given. - gbn -------- Earlier thread on this concluded below: Subject: Re: newsflash! cypherpunks mailing list is behind cloudflare-NSA On Fri, Jul 12, 2019 at 06:34:07PM -0400, grarpamp wrote:

On 7/12/19, Greg Newby wrote:

...

Newsflash! This happened in April, and was announced here: https://lists.cpunks.org/pipermail/cypherpunks/2019-April/045250.html We have been on Cloudflare's DNS since then for the email lists.

Use of CF or any other CDN was not mentioned in the announcement, whether for DNS, or HTTPS. The entire internet is NSA anyway.

If CDN for HTTPS, consider multihoming on I2P or Tor so users can still access when CDN javascript captcha or otherwise arbitrarily blocks them or goes down.

As to caching bandwidth and archives...

You really should fork that 335MiB mbox file off now or no later than year end, and compress it, and then once yearly thereafter, and sign them all. People will eventually seed them into IPFS, etc.

Try using a modern unix compression tool like zstd, they are faster, smaller, available for all systems...

https://github.com/facebook/zstd https://facebook.github.io/zstd/ https://code.fb.com/core-data/zstandard/ https://en.wikipedia.org/wiki/Zstandard