It's April 2005 and I'm currently working on migrating a couple of email servers from mbox to maildir format. This is a collection of my notes on the process. I didn't try to fully explain all the background, but I will try to go back and fill in gaps as I find time.
Mbox format stores email folders as one file while maildir uses a distinct file for every email message. This has serious implications for filesystem setup and there for requires some research before proceeding.
Mbox format stores email folders as one file while maildir uses a distinct file for every message, so it is important to know how many messages will be migrating and their approximate size. Knowing this will allow us to create a filesystem that won't run out of inodes before it runs out of space.
I will be moving one system which has approximately 250 users consuming 35GB with about 2.6 million messages. This works out to an average of 13KB per message. Since I'm sure there are large attachments skewing the average upwards I wasn't satisfied with this number.
To arrive at this number I used the following script: count-mbox-msg.sh.
#!/bin/bash # Dale BewleyWed Apr 27 14:21:47 PDT 2005 # quick script to count the number of messages found on a system in # ~/mbox, ~/mail, and /var/mail SPOOL=/var/mail HOMEROOT=/export/home/raid echo user,spool,mbox,folders for spool in $SPOOL/*; do user=`basename $spool` # count messages in spool if [ -f "$SPOOL/$user" ]; then spool_cnt=`grep -c '^From ' $SPOOL/$user` fi # count messages in mbox (pine) if [ -f "$HOMEROOT/$user/mbox" ]; then mbox_cnt=`grep -c '^From ' $HOMEROOT/$user/mbox` fi # count messages in folders # i want to know how many folders and how big they are, but not the names # there are 2 users who have SO MANY folders that this output will split the lines! if [ -d "$HOMEROOT/$user/mail" ]; then folder_cnt=`find $HOMEROOT/$user/mail/ -type f -exec grep -c '^From ' {} \;` fi echo $user,$spool_cnt,$mbox_cnt,$folder_cnt done
After a test conversion to Maildir I used this script to count the messages by size.
#!/bin/bash
# count-maildir-msg.sh - count email messages in Maildir format
# at least as Dovecot saves them...
# Dale Bewley
#
# based on: http://www.mostlygeek.com/node/11 by phrax
echo "user,<2k,<4k,<8k,<16k,<32k,<64k,>64k,Total"
for user in /home/*; do
echo -n `basename $user`,
ls -lRAaF $user/Maildir | grep '^-' | grep 'mbox:\|newmail:' | \
awk '{ \
num+=1; \
if ($5 <= 2048) c2k+=1; \
else if ($5 <= 4096) c4k+=1; \
else if ($5 <= 8192) c8k+=1; \
else if ($5 <= 16384) c16k+=1; \
else if ($5 <= 32768) c32k+=1; \
else if ($5 <= 65535) c64k+=1; \
else cBig+=1; \
endif; \
} END { \
printf ("%d,%d,%d,%d,%d,%d,%d,%d\n",c2k,c4k,c8k, \
c16k,c32k,c64k,cBig,num) \
}'
done
Not all users are converted, but I just wanted to pull out a distribution. This gave me the following count.
<2k <4k <8k <16k <32k <64k >64k Total Total 430596 534266 282273 96848 32388 27873 25959 1430203 Percent 30.11% 37.36% 19.74% 6.77% 2.26% 1.95% 1.82% Avg 1743.3 2163.02 1142.81 392.1 131.13 112.85 105.1 5790.3 Max 95909 123354 40086 18398 3396 3051 2593 255597 Blk 2k 860402 Blk 4k 1720804
mbox format stores email folders as one file while maildir uses a distinct file for every email message. This has serious implications for filesystem setup and there for requires some research before proceeding.
In main.cf set the following. You can leave the -p out, but I use it to support TMDA.
home_mailbox = Maildir/ mailbox_command = /usr/bin/procmail -p -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir
On Fedora anyway, the default /etc/dovecot.conf should automatically detect the Maildir format and use it. See the default_mail_env setting.
This isn't really related to migration to Maildir, but I'm adding it at the same time.
################################################################################ # this is still under testing # still to do http://www.securitysage.com/antispam/sasltls.html smtpd_sasl_auth_enable = yes smtpd_sasl_security_options = noanonymous #smtpd_sasl_local_domain = $myhostname smtpd_sasl_local_domain = smtp.bewley.net # You only need the following line if supporting old mail clients #broken_sasl_auth_clients = yes smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination # This will only allow authentication of users once TLS has been # started and information being transferred is encrypted. smtpd_tls_auth_only = yes #In order to use your certificate, you will need to make the following additions to main.cf: smtp_use_tls = yes smtpd_use_tls = yes smtp_tls_note_starttls_offer = yes smtpd_tls_key_file = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_cert_file = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_CAfile = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_loglevel = 1 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s tls_random_source = dev:/dev/urandom # this is still under testing ################################################################################
Also in master.cf you need to enable the submission port (587) if you want Outlook to work apparently. I guess it can't just do TLS on port 25.
submission inet n - n - - smtpd -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes -o smtpd_etrn_restrictions=reject -o local_header_rewrite_clients=permit_sasl_authenticated
I have users who have been on pine for eons and don't want to change. I'm kinda one of them. Unfortunately, pine can't read Maildir. To work around that, just tell pine to use IMAP rather than the local filesystem. Dovecot will act as the go between. It works quite well actually. Here's a little hack to fix up all your users' ~/.pinerc files. Here is a sample .pinerc you can use.
Notice you won't find pine in the Fedora distribution though. You'll have to get a copy from somewhere like Dag's repository.
#!/bin/bash ################################################################################ # pinemaildir - Dale BewleyFri Apr 22 09:34:47 PDT 2005 #------------------------------------------------------------------------------- # Configure pine to use IMAP to localhost for the purposes of supporting # Maildir folders. The IMAPd (courier, dovecot, etc) will retrieve and # store messages in maildir format since pine only understands mbox. # # Thanks to http://www.neverending.org/~ftobin/resources/pine-maildir/ ################################################################################ ################################################################################ # Settings we need to effect # if you cahnge the default-fcc then the monthly folder prune will fail SETTINGS="inbox-path folder-collections feature-list" # where are the users' .pinerc's? PINERCS=/home/*/.pinerc ################################################################################ # look through ~/.pinerc's and find users who have # changed these settings rather than the empty defaults for pinerc in $PINERCS; do echo -n "Checking $pinerc..." if [ ! -f $pinerc ]; then echo " not a file. skipping" next; fi default=1 for setting in $SETTINGS; do current=`grep "^$setting=" $pinerc | cut -d= -f2` if [ -n "$current" ]; then echo " $setting is customized to: $current" default=0 fi current='' done if [ "$default" -lt 1 ]; then echo " settings customized" skip="$skip $pinerc" else TMPFILE=`mktemp` || exit 1 sed -e 's!^inbox-path=$!inbox-path={localhost/ssl/novalidate-cert}inbox!' \ -e 's!^folder-collections=$!folder-collections="Local IMAP" {localhost/ssl/novalidate-cert}[]!' \ -e 's!^feature-list=$!feature-list=enable-lame-list-mode,\n\tcombined-subdirectory-display,\n\tcombined-folder-display,\n\tquell-empty-directories!' \ < $pinerc > $TMPFILE echo mv $TMPFILE $pinerc fi echo done echo "You should check these users by hand:" for u in $skip; do echo $u; done
You want your new users to hit the ground running, so create Maildir in /etc/skell. You might also put a fixed up .pinerc in there.
mkdir -p /etc/skel/Maildir/new mkdir -p /etc/skel/Maildir/cur mkdir -p /etc/skel/Maildir/tmp