• RSS
  • Twitter
  • FaceBook

Security Forums

Log in

FAQ | Search | Usergroups | Profile | Register | RSS | Posting Guidelines | Recent Posts

Hypermail - convert mail to HTML archive

Users browsing this topic:0 Security Fans, 0 Stealth Security Fans
Registered Security Fans: None
Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> General Software

View previous topic :: View next topic  
Author Message
capi
SF Senior Mod
SF Senior Mod


Joined: 21 Sep 2003
Posts: 16777097
Location: Portugal

Offline

PostPosted: Sat Feb 26, 2005 2:59 pm    Post subject: Hypermail - convert mail to HTML archive Reply with quote

Disclaimer: I am not in any way affiliated with the developers of this program, but I thought it might be useful to the general public. I have created a patch to port Hypermail to Windows, which I am releasing in this post for anyone who wishes to use it.
______________________


Hypermail
Convert your mail to an HTML archive


Hypermail is an open source program that lets you create an HTML archive of your emails, complete with an index with various sorting options, links to attachments, and so on - much like what you see in online archives of mailing lists (see Hypermail's Mailing List Archives for one possible example).

To quote from the Hypermail FAQ:
Quote:
Hypermail is a program that takes a file of mail messages in UNIX mailbox format and generates a set of cross-referenced HTML documents. Each file that is created represents a separate message in the mail archive and contains links to other articles, so that the entire archive can be browsed in a number of ways by following links. Archives generated by Hypermail can be incrementally updated, and Hypermail is set by default to only update archives when changes are detected.

As mentioned, the program works with the UNIX mailbox format (i.e. pine for example). This means it will also work with Mozilla Thunderbird mail folders, which use that format as well. All you need is to take the mail folder file (usually present inside your Thunderbird profile, with the same name as the folder - i.e Inbox or Sent, etc) and run it through Hypermail.

For usage information please visit Hypermail's homepage (linked to at the beginning of this post) and/or read their FAQ (linked above). The source code can be obtained from the Hypermail Download Page, in order to compile it.

The program is originally meant mainly for Unix and Unix-like environments such as GNU/Linux. However, it is possible to compile it for Windows as well. One way to accomplish this is by using Cygwin, for which instructions are available on the site. The other method, which has the advantage of producing a native binary which will run on any Windows platform without need of third-party DLLs, is to build it using MinGW.

In its current state as can be downloaded from the site, however, Hypermail will not compile under MinGW, as it has some *nix specific code. To that effect, I have created a patch to port Hypermail to Windows (and fix a couple of minor bugs in the process as well). By applying it to the sources, it is possible to compile and build Hypermail under MinGW, thus creating a native executable that can be run from any normal Windows computer without the need for MinGW to be installed there.

I made the patch in diff -ru format against Hypermail version 2.1.8, it can be downloaded Gzip compressed from here:
2.1.8-ilugo.patch.gz
md5: f8d07847e6767c668d3f12fa63eafba8

To apply the patch, extract it to the same directory where you extracted the source code (hypermail-2.1.8.tar.gz) , enter the directory and type:
Code:
patch -p1 < 2.1.8-ilugo.patch

The code will then be patched, after which you can build it as normal.

For more detailed instructions on the whole process of building and using Hypermail, please read my guide to Building Hypermail for Windows using MinGW.

For convenience, I have created a pre-compiled binary of Hypermail for Windows, configured with GDBM enabled, and made it available for download. Just choose one of the files below and extract it to a directory of its own. Both files are equivalent and have the same content, just different compression formats:


Keywords: Hypermail Windows port MinGW patch open source email archive HTML mbox mailbox UNIX native binary archiving tool e-mail backup compiling building tutorial


Last edited by capi on Sat Oct 28, 2006 8:46 pm; edited 6 times in total
Back to top
View user's profile Send private message
Stormhawk
Trusted SF Member
Trusted SF Member


Joined: 26 Aug 2003
Posts: 31
Location: Warwickshire, England, UK

Offline

PostPosted: Wed Mar 02, 2005 4:52 am    Post subject: Reply with quote

This sounds kinda useful. I have a lot of mail stored in Thunderbird that I could do with storing in some archive format. I was just going to pull it out of the mail folder and tar/bz2 it, but hey, this looks like it might be more useful to me!

Well found, capi!
I'd check your patch, but since its you I've no doubt its 200% perfect in every way imaginable, and some ways I couldn't even dream of.
Back to top
View user's profile Send private message Visit poster's website
capi
SF Senior Mod
SF Senior Mod


Joined: 21 Sep 2003
Posts: 16777097
Location: Portugal

Offline

PostPosted: Wed Mar 02, 2005 6:32 am    Post subject: Reply with quote

Oh, it is very useful indeed - at least for my needs. Works wonders for archiving those old emails that have been eating up disk space. The fact that it will also archive the attachments along with the emails (with a link to each attachment from its respective email page) is just the icing on the cake.

Funny you should mention my patch being perfect hehe - I just found a small bug in it, which has been promptly fixed (I've already uploaded the new patch version and edited the above post to include the correct MD5 of this new version).

The two new changes were to src/file.c:
  • I added a prototype at the beginning of the file to declare the dirname_w32() function I had created (more elegant plus it helps GCC behave better);

  • dirname_w23()'s sanity check for a NULL argument was broken. Calling dirname_w32(NULL), would cause the function to sigsegv, contrary to what its description said in the source.

Fortunately, none of these bugs actually had any practical consequence, since as it stands dirname_w32() will never be called with a NULL argument. Nevertheless, the bugs were there, and have been fixed.

The rest of the changes made by my patch should be in working order - any bugs found, please post them here so that I can fix them.

Upon looking at the added code, you may notice that in some places the route taken may not have been the most elegant. This is explained by the fact that when I created this patch, I did so with the intention of trying to change (almost) as little as possible. I did not (and at least for now, don't) intend to do a complete rewrite of Hypermail just in order to port it to Windows, so in some places my code had to adapt to the existing code structure.

Now, for a simple guide into building Hypermail on MinGW and using it in Windows, check the next post.
Back to top
View user's profile Send private message
capi
SF Senior Mod
SF Senior Mod


Joined: 21 Sep 2003
Posts: 16777097
Location: Portugal

Offline

PostPosted: Wed Mar 02, 2005 3:19 pm    Post subject: Building Hypermail for Windows using MinGW Reply with quote

Building Hypermail for Windows using MinGW
Instructions by Israel G. Lugo <israel.lugo@lugosys.com>



Installing MinGW


First, in order to build Hypermail, you will need a working MinGW installation. Teaching how to install MinGW would make for a somewhat larger post, so instead I will just refer you to Building GnuPG for Win32 using MinGW (written by Carlo Luciano Bianco) for a nice guide on how to install MinGW from scratch. Just follow the instructions contained there to install MinGW, up to, and not including, the point where it tells you to download the GnuPG sources, which you obviously will not need for this project.


_____________________________


Downloading and patching Hypermail


  1. First, download the Hypermail sources from the official site. My patch was made for version 2.1.8, so that's what you should preferably get, as it may or may not work for future versions: hypermail-2.1.8.tar.gz

  2. Extract the sources, by typing, from within MinGW and in the same directory where you placed the hypermail-2.1.8.tar.gz file:
    Code:
    tar xzvf hypermail-2.1.8.tar.gz

  3. Now, download my patch: 2.1.8-ilugo.patch.gz, and save it to the same directory where you saved hypermail-2.1.8.tar.gz.

  4. Extract the patch, by typing, from within MinGW and in the same directory where you saved it:
    Code:
    gunzip 2.1.8-ilugo.patch.gz

  5. Now, still from within MinGW and still from the same directory, type the following two commands, in order:
    Code:
    cd hypermail-2.1.8
    patch -p1 < ../2.1.8-ilugo.patch

  6. All done, the source has now been patched.


_____________________________


Building Hypermail



Example 1:

Say you wanted to compile this and install it to "C:\Hypermail". From within MinGW, you would run:
Code:
configure --prefix="c:/Hypermail" --exec-prefix="c:/Hypermail" --with-httpddir='c:/Hypermail/apache'

The --with-httpddir part is of course subject to being changed to an actual location where apache exists (or just leave it there if you have no apache installed, at least you'll get some nice documentation).

You would then just have to do make followed by make install.



Example 2:

Expanding on the above, say you wanted to optimize the program for an Intel Pentium 4 (or equivalent Celeron). Instead of the above configure line, you would want to run something like:
Code:
configure CFLAGS='-O2 -march=pentium4 -mmmx -msse -msse2' --prefix="c:/Hypermail" --exec-prefix="c:/Hypermail" --with-httpddir='c:/Hypermail/apache'

This is only one command in one line, it may break lines on this page due to screen width but it should be entered in one single line.



Example 3:

Same as above, only to optimize for an Intel Pentium MMX this time:
Code:
configure CFLAGS='-O2 -march=pentium-mmx -mmmx' --prefix="c:/Hypermail" --exec-prefix="c:/Hypermail" --with-httpddir='c:/Hypermail/apache'


Again this is only one command in one line. And so on, you get the point Wink. Check the GCC Manual's Hardware Models and Configurations section for a full list of architecture-specific compiler flags.



Example 4:

Now, say you want to include support for the GDBM library. One possible reason to do this would be to be able to do incremental updates on archives created with the "folder_by_date" option.

First, you would need to properly install GDBM on MinGW:
  1. Go to the GnuWin32 file list on sourceforge, and download the following files (the actual versions can vary, just grab the most recent ones):

  2. Then, unzip the contents of gdbm-1.8.3-1-bin.zip and gdbm-1.8.3-1-lib.zip to your MinGW folder.
  3. Finally, open gdbm-1.8.3-1-src.zip and extract gdbm-dll.h into your /mingw/include directory.

This last step of extracting gdbm-dll.h may or may not be necessary on future versions of the library. It was necessary for me on the above mentioned versions of the files, as gdbm-1.8.3-1-lib.zip is broken and does not contain the necessary gdbm-dll.h file included by its own gdbm.h header file.

Ok, so now that you have installed GDBM on your MinGW system, here is how to configure Hypermail to use it:
Code:
configure --prefix="c:/Hypermail" --exec-prefix="c:/Hypermail" --with-httpddir='c:/Hypermail/apache' --with-gdbm=/mingw

You can of course add any optimizations you like via CFLAGS, from the above examples or your own experience. You would then issue the usual make followed by make install, after which there is just one final step to do: copy C:\MinGW\bin\gdbm3.dll (or wherever you installed MinGW) to C:\Hypermail\bin.


_____________________________


Using Hypermail


The first thing you may want to do once you've installed Hypermail is to add it to the PATH environmental variable, so that you can use it from any directory. To do this in Windows 2000:
  1. Go to the Control Panel, then open System.
  2. In the Advanced tab, click on the Environmental Variables button.
  3. In the window that appears, there are two lists. Look in the bottom one (System variables) for the entry called PATH (or Path or path or whatever, letter case is unimportant), select it and click on the Edit button.
  4. In the dialog box that appears, where it says Variable Value, append C:\Hypermail\bin to what is already there, separating this new entry from the existing ones with a semi-comma (;). Naturally, you should replace C:\Hypermail with whatever directory you installed Hypermail to.



Example 1:

To create a default configuration file for Hypermail, run, from the Windows command line:
Code:
hypermail -v > .hmrc

This will create a file called .hmrc in the current directory, containing all the default options. You can edit it with any text editor to suit your needs. Hypermail will look for this file in the path specified by a environmental variable called HOME, or in the binary's own directory if there is no such environmental variable.

You can also download a sample configuration file provided by me: hmconfig.gz

It's a modified version of the configuration, the one I'm currently using for that matter. With this configuration file, you will need to have compiled Hypermail with GDBM support in order to perform incremental updates, as it is set to use the folder_by_date option (which puts emails into folders by date, makes for a somewhat neater archive). See above for more on this.



Example 2:

To archive a mailbox by the name of Inbox, run, from the Windows command line:
Code:
hypermail -m Inbox -d C:\path\to\new\archive

Where C:\path\to\new\archive should obviously be replaced with the path where you want hypermail to create the archive (e.g. C:\email_archives\inbox).



Example 3:

One suggestion I would make is to join all related mailboxes in one big mailbox, then archive the whole lot. For example, joining Inbox and Sent in one big mailbox then archiving the whole thing would give you the benefit of having the incoming emails along with their respective replies in the same indexed archive. Each email would have a link to its reply (if any), each reply would have a link to its originating message, and so on.

There are various ways to join together different mailboxes. If you're using Thunderbird, you can just move all the emails from one mail folder into the other from within Thunderbird, then archive the whole thing with Hypermail. For the general case (which is still perfectly valid when using Thunderbird as well), you can just concatenate the two mailbox files together into some third temporary file, then archive that one. Using MinGW this would be as trivial as running, from the directory where the mailboxes are located:
Code:
cat Inbox Sent > all_my_emails

You can also do it using a text editor, as the mailbox files are just plain-text. You would open one, open the other, then copy and paste so that it's all together in one big file.



Example 4:

To incrementally update an existing archive with new emails:
Code:
hypermail -u -m Inbox -d C:\path\to\existing\archive

This would add all the emails inside the Inbox mailbox to the archive. If you created this archive using the folder_by_date option, you will need to have compiled Hypermail with GDBM support in order to perform an incremental update.


_____________________________


Miscellaneous information


Hmm, what else, what else... Well, a few heads ups:
  • Expect to get a warning about not finding the en_US locale when you run Hypermail. This happens because the locale files are not being installed anywhere outside of MinGW and thus cannot be found from the Windows command line. I've had no problems with Hypermail despite this fact, if someone does experience problems (maybe people using non-ASCII characters?) please report back here, or better yet if you have a solution for this I'd be glad to include it in the patch;

  • By default, Hypermail discards (as in, it does not archive) email messages which do not contain a Message-ID. Also, it will discard email messages with duplicate Message-IDs. This behavior can be altered by editing the .hmrc configuration file (after you have created one by following my example several lines above). The parameters to alter are, respectively, require_msgids and discard_dup_msgids.

  • The part of my patch that ports the msg2archive tool (included in the Hypermail package) to MinGW is broken. As it is, msg2archive will compile fine, but does not function. I'll have to rewrite parts of the program's code to bypass the need for POSIX functions (specifically fork, waitpid and pipe) and adapt it to use the normal Win32 API. This does not affect any of the other programs in the Hypermail package, though. In particular, hypermail itself is still in perfect working order, ready and waiting to archive those emails! Smile


And well, that's about as much as I can remember off hand right now. I'll post any future updates here, and shall try to answer any questions posted. Keep in mind however that I was not the original creator of the program Wink

Keywords: Hypermail Windows port MinGW patch open source email archive HTML mbox mailbox UNIX native binary archiving tool e-mail backup compiling building tutorial


Last edited by capi on Wed Dec 09, 2009 11:39 pm; edited 6 times in total
Back to top
View user's profile Send private message
Ocumo
Just Arrived
Just Arrived


Joined: 14 Jun 2004
Posts: 0


Offline

PostPosted: Sun Mar 06, 2005 7:45 pm    Post subject: Reply with quote

Thank you for this great document. Very Happy Very Happy Very Happy Very Happy Very Happy

I had been looking for something like this, ever since I read Linux Magazine's issue 52 cover story "Hypermail - Convert your email folder into an indexed collection of HTML documents" ( www.linux-magazine.com/issue/52 ).

I have just got the link for this article.

I have tried before a procedure to port hypermail to Windows that is referenced in www.hypermail.org, but I have to say, I quit, because (with all due respect) I found that procedure quite confusing and not satisfactory, since one had to use Cygwin DLLs in order to run Hypermail. Moreover, the examples given there don't help and seem in fact to have something wrong.

I would like to see this link in Hypermail website. I wonder if they are aware of this, and that Linux Magazine wrote a cover story on their work!

Thank you and congratulations for an excellent, very professional document, Capi.

Ocumo
Back to top
View user's profile Send private message
capi
SF Senior Mod
SF Senior Mod


Joined: 21 Sep 2003
Posts: 16777097
Location: Portugal

Offline

PostPosted: Wed Apr 06, 2005 2:35 pm    Post subject: Reply with quote

Thank you very much for your feedback, it is good to see my tutorial was of help to someone Very Happy

I did try to make the Hypermail developers aware of my patch, but failed to contact them at any of the email addresses mentioned in their site - it would seem that the site itself is no longer being maintained, or at least that's the impression that I was left with.

Nevertheless, Hypermail is still a very useful program for anyone with a big set of emails who wants to create a nice HTML archive of them. I use it myself regularly, and the Windows port makes it all the more practical for even more people Smile
Back to top
View user's profile Send private message
Display posts from previous:   

Post new topic   Reply to topic   Printer-friendly version    Networking/Security Forums Index -> General Software All times are GMT + 2 Hours
Page 1 of 1


 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Community Area

Log in | Register