Moving emails accounts to a new host, imapcopy with actual encryption

Tunnelling Imapcopy traffic over TLS and everything that went wrong for me.

My hosting provider’s IP reputation went to trash and I was not able to send emails out to microsoft (hotmail, office365) users. My host had given up trying to fix it and instead told me to pay for external mail forwarding services. This was my hint that I needed to move to a new host :)

Moving emails from an old host to a new host is always an interesting step, there are three methods I know of:

1. Copy the emails over in your email client (Thunderbird in my case)
2. Copy the emails using a dedicated imap-to-imap tool (imapcopy)
3. Copy the maildir folders across (requires file-level access on both hosts)

Copying them myself in Thunderbird

This was a disaster.

For some folders it works fine.

For other folders Thunderbird decides that it hasn’t got the emails downloaded yet. This means it interleaves downloading 1 message and then copying 1 message. This is really slow. I am not sure why TB does not always fetch emails in folders that you are Subscribed to, perhaps renaming and moving folders confuses it.

After completing the copy process: I had less emails in my new account than my old. Sometimes by a small margin (a few emails), other times by a large margin (more than 50% of the emails).

Automate it with Imapcopy

This was also a disaster (but with workarounds).

I suspect this tool is what most shared hosting support staff use. It’s dedicated to the task, it looks nice and it’s probably very reliable.

Some hosts provide a web UI for this tool, in my case I had already asked for ssh access (I need this for my site) so I cobbled together my own copy and used it there (more on this below). Running this tool on the destination host itself has the advantage of speed: you’re copying emails directly from host to host, rather than middle-manning them with your desktop computer’s limited internet connection speed.

What’s encryption?

Imapcopy has no SSL/TLS support at all. Your email login details are sent over plaintext imap and all of your emails are fetched & synchronised over plaintext imap. This is just stupid.

Yes I know lots of emails today are sent in plaintext. But not all of them, a lot of services encrypt their SMTP traffic. We should be encouraging better behaviour, not going “oh no everyone does it so I won’t bother either” and “some of these emails will have been leaked as plaintext so I may as well publicly flash all of them and my password too”.

I found some guides on the web mentioning the use of stunnel to proxy the traffic over SSL/TLS, but I tried their incantations and stunnel was stunned. I suspect the entire commandline syntax for stunnel underwent a wash cycle at some point and all of these copypasta articles are old.

Socat came to my rescue. I chose a random port number of localhost to listen to (unencrypted) and then connected the other end to my source (old) mailserver:

./socat -d TCP4-LISTEN:2891,fork,bind=127.0.0.1 OPENSSL:myoldemailhost.net:993

I then configured imapcopy to source the emails from localhost:2891 and write them to localhost:143 (I ran this on the mailserver itself).

1 User processed, 11718 Messages copied, 0 Error(s)
17 Folder(s) created, 0 Folder create errors, 0 Folder not copied

Hooray! And it was much faster than the Thunderbird copy attempt too.

How do I get these tools onto my shared host?

You will definitely need SSH access. Many hosting providers will allow ssh if you politely ask and explain why you want it.

Upstream imapcopy hasn’t been updated since 2009. Debian maintains a bunch of patches, so I went with them.

I ended up downloading the precompiled .deb files for imapcopy, socat and libwrap (a dependency that my server didn’t have). I extracted just the parts I needed into a folder and copied that across to the shared host:

ImapCopy.cfg
imapcopy*
libwrap.so.0
libwrap.so.0.7.6
socat*

The ImapCopy.cfg file is provided in the /usr/doc directory of the imapcopy package.

Make sure to chmod ImapCopy.cfg before editing it, otherwise other users on your host might be able to see your passwords you put into it. I changed my passwords after using it anyway as a precaution, I had already accidentally submitted them to the old host thanks to Thunderbird account shenanigans.

You will also need to export LD_LIBRARY_PATH=“$(pwd)” so that socat notices and loads any object files (like libwrap.so) that you have in your folder.

In general: stealing precompiled packages from debian stable and running their binaries on random linux platforms works remarkably well. I’ve done it before for old equipment too, often one of the older Debian releases is “close enough” to the libc and kernel of your target system for things to work. Ideally you want statically compiled binaries, but those are few and far between (it would be amazing if there were a distro somewhere with 10 years of repositories full of them).

Copying the maildir across

I’ve never tried this method and I didn’t try it this time. This would be fastest if done host-to-host, eg by sshing from one host to the other (sshfs).

What else goes wrong?

Thunderbird. Lots of Thunderbird.

I didn’t delete all of the emails from my first copy attempt (Thunderbird copy) before attempting the second (imapcopy). I ended up with about 6500 duplicate emails in my Sent box. Thunderbird Duplicate Messages addon came to the rescue, but only after changing its settings (by default it won’t touch folders that it deems “special”).

I then had to manually re-subscribe to these folders in Thunderbird (by right-clicking the account and choosing “Subscribe” from the menu). Interestingly folders that only contained other folders (no emails) are greyed out and untickable, perhaps imapcopy doesn’t copy these. This seems to be a harmless problem anyway.

Adding new accounts on Thunderbird is always fraught by the settings autodetection system. I don’t like what it does:

1. It defaults to STARTTLS instead of TLS. Yes it’s probably just as secure, but people doing audits often flag it as unencrypted traffic and I’m not sure why anyone would bother with it as anything other than a legacy option.
2. It overwrites EVERY MANUAL SETTING YOU HAVE ENTERED if you enter your password after entering server details, and then immediately sends your login credentials to whatever server settings it auto-guesses.

WTF. Now my old host has my new credentials. Thanks.

(I always configure my email accounts to use the shared hosting provider’s domain name rather than my domain name for smtp and imap. This way all of my accounts keep working even if I’m changing what my domain points to. It’s a lot less hassle and fiddly to do it this way, especially during changeover periods like this. Additionally it means Thunderbird stores your emails in folders with the name of your hosting provider, so dealing with those is dramatically easier too.)

Do you have any better methods?

I’d in particular like to hear if anyone has success with copying Maildirs, I’m not sure if it’s more sane (you can do it encrypted) or less (you have to ssh/sshfs/sftp/scp from one shared host into another shared host).