The mail server is back
News, Technical ·Friday November 16, 2012 @ 17:02 EST (link)
Most of the rest of the mail files have been recovered and internal DNS and the mail system is mostly back—Courier-IMAP, Qmail, SpamAssassin—except for an internal webmail system (Horde/IMP), which is giving trouble and has always been a bletcherous piece of PHP crap anyway. It's nice to be receiving and sending mail locally again. Given that it was down for two weeks I may have missed some mail that hasn't been resent, or been bounced from some mailing lists; others are amazingly resilient (or perhaps, "stubborn") and have kept on sending to me. All configuration files were recovered, so it was just a matter of merging them with the reinstalled Gentoo packages, and tracking down a few unexpected errors via log files and adding the daemons to system startup. Now that it's stable it's been backed up (rsync) and will be periodically.
Recovery of some mail files was laborious; a group of inodes were destroyed covering some mail directories and files; the directories were recovered by a disk scan: first, the . entry points to its own inode, and second, by looking for blocks that look like directories containing distinctive maildir files (starting with the Unix time) and subtracting off known/reachable blocks to yield orphans (much like a mark and sweep garbage collector). Another similar scan was done for files that look like mail messages (with a Delivered-To: header in the first 256 bytes). I also recovered a few from Mutt's cache, removing dupes using a python program mapping file digests to filenames. All in all, I wrote about 1000 lines of C++ (saved some work by using libext2fs) and 500 lines of Python (to process block lists etc.).
