Background

Introduction

It's April 2005 and I'm currently working on migrating a couple of email servers from mbox to maildir format. This is a collection of my notes on the process. I didn't try to fully explain all the background, but I will try to go back and fill in gaps as I find time.

Mbox format stores email folders as one file while maildir uses a distinct file for every email message. This has serious implications for filesystem setup and there for requires some research before proceeding.

Research

Message Count

Mbox format stores email folders as one file while maildir uses a distinct file for every message, so it is important to know how many messages will be migrating and their approximate size. Knowing this will allow us to create a filesystem that won't run out of inodes before it runs out of space.

I will be moving one system which has approximately 250 users consuming 35GB with about 2.6 million messages. This works out to an average of 13KB per message. Since I'm sure there are large attachments skewing the average upwards I wasn't satisfied with this number.

To arrive at this number I used the following script: count-mbox-msg.sh.

#!/bin/bash
# Dale Bewley  Wed Apr 27 14:21:47 PDT 2005
# quick script to count the number of messages found on a system in
# ~/mbox, ~/mail, and /var/mail

SPOOL=/var/mail
HOMEROOT=/export/home/raid

echo user,spool,mbox,folders

for spool in $SPOOL/*; do
	user=`basename $spool`

        # count messages in spool
        if [ -f "$SPOOL/$user" ]; then
                spool_cnt=`grep -c '^From ' $SPOOL/$user`
        fi

        # count messages in mbox (pine)
        if [ -f "$HOMEROOT/$user/mbox" ]; then
                mbox_cnt=`grep -c '^From ' $HOMEROOT/$user/mbox`
        fi

        # count messages in folders
	# i want to know how many folders and how big they are, but not the names
	# there are 2 users who have SO MANY folders that this output will split the lines!
        if [ -d "$HOMEROOT/$user/mail" ]; then
                folder_cnt=`find $HOMEROOT/$user/mail/ -type f -exec grep -c '^From ' {} \;`
        fi

        echo $user,$spool_cnt,$mbox_cnt,$folder_cnt
done

After a test conversion to Maildir I used this script to count the messages by size.

#!/bin/bash
# count-maildir-msg.sh - count email messages in Maildir format
# at least as Dovecot saves them...
# Dale Bewley
#
# based on: http://www.mostlygeek.com/node/11 by phrax

echo "user,<2k,<4k,<8k,<16k,<32k,<64k,>64k,Total"
for user in /home/*; do
    echo -n `basename $user`,
    ls -lRAaF $user/Maildir | grep '^-' | grep  'mbox:\|newmail:' | \
    awk '{ \
			num+=1; \
			if ($5 <= 2048) c2k+=1; \
			else if ($5 <= 4096) c4k+=1; \
			else if ($5 <= 8192) c8k+=1; \
			else if ($5 <= 16384) c16k+=1; \
			else if ($5 <= 32768) c32k+=1; \
			else if ($5 <= 65535) c64k+=1; \
			else cBig+=1; \
			endif; \
	} END { \
			printf ("%d,%d,%d,%d,%d,%d,%d,%d\n",c2k,c4k,c8k, \
				  c16k,c32k,c64k,cBig,num) \
    }'
done

Not all users are converted, but I just wanted to pull out a distribution. This gave me the following count.

	<2k	<4k	<8k	<16k	<32k	<64k	>64k	Total
Total	430596	534266	282273	96848	32388	27873	25959	1430203
Percent	30.11%	37.36%	19.74%	6.77%	2.26%	1.95%	1.82%	
Avg	1743.3	2163.02	1142.81	392.1	131.13	112.85	105.1	5790.3
Max	95909	123354	40086	18398	3396	3051	2593	255597

Blk 2k	860402	
Blk 4k	1720804

Inodes

mbox format stores email folders as one file while maildir uses a distinct file for every email message. This has serious implications for filesystem setup and there for requires some research before proceeding.

MTA and MDA

Postfix and Procmail

In main.cf set the following. You can leave the -p out, but I use it to support TMDA.

home_mailbox = Maildir/ mailbox_command = /usr/bin/procmail -p -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

Dovecot

On Fedora anyway, the default /etc/dovecot.conf should automatically detect the Maildir format and use it. See the default_mail_env setting.

User Managment

MUA - Mail User Agent

Authenticated SMTP

This isn't really related to migration to Maildir, but I'm adding it at the same time.

################################################################################ # this is still under testing # still to do http://www.securitysage.com/antispam/sasltls.html smtpd_sasl_auth_enable = yes smtpd_sasl_security_options = noanonymous #smtpd_sasl_local_domain = $myhostname smtpd_sasl_local_domain = smtp.bewley.net # You only need the following line if supporting old mail clients #broken_sasl_auth_clients = yes smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination # This will only allow authentication of users once TLS has been # started and information being transferred is encrypted. smtpd_tls_auth_only = yes #In order to use your certificate, you will need to make the following additions to main.cf: smtp_use_tls = yes smtpd_use_tls = yes smtp_tls_note_starttls_offer = yes smtpd_tls_key_file = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_cert_file = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_CAfile = $config_directory/ssl/smtp.bewley.net.pem smtpd_tls_loglevel = 1 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s tls_random_source = dev:/dev/urandom # this is still under testing ################################################################################

Also in master.cf you need to enable the submission port (587) if you want Outlook to work apparently. I guess it can't just do TLS on port 25.

submission inet n - n - - smtpd -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes -o smtpd_etrn_restrictions=reject -o local_header_rewrite_clients=permit_sasl_authenticated

Pine

I have users who have been on pine for eons and don't want to change. I'm kinda one of them. Unfortunately, pine can't read Maildir. To work around that, just tell pine to use IMAP rather than the local filesystem. Dovecot will act as the go between. It works quite well actually. Here's a little hack to fix up all your users' ~/.pinerc files. Here is a sample .pinerc you can use.

Notice you won't find pine in the Fedora distribution though. You'll have to get a copy from somewhere like Dag's repository.

#!/bin/bash
################################################################################
# pinemaildir - Dale Bewley  Fri Apr 22 09:34:47 PDT 2005
#-------------------------------------------------------------------------------
# Configure pine to use IMAP to localhost for the purposes of supporting
# Maildir folders. The IMAPd (courier, dovecot, etc) will retrieve and
# store messages in maildir format since pine only understands mbox.
#
# Thanks to http://www.neverending.org/~ftobin/resources/pine-maildir/
################################################################################

################################################################################
# Settings we need to effect
# if you cahnge the default-fcc then the monthly folder prune will fail
SETTINGS="inbox-path
	folder-collections
	feature-list"
# where are the users' .pinerc's?
PINERCS=/home/*/.pinerc
################################################################################

# look through ~/.pinerc's and find users who have 
# changed these settings rather than the empty defaults
for pinerc in $PINERCS; do
	echo -n "Checking $pinerc..."
	if [ ! -f $pinerc ]; then
		echo " not a file. skipping"
		next;
	fi

	default=1
	for setting in $SETTINGS; do
		current=`grep "^$setting=" $pinerc | cut -d= -f2`
		if [ -n "$current" ]; then
			echo "	$setting is customized to: $current"
			default=0
		fi
		current=''
	done

	if [ "$default" -lt 1 ]; then
		echo "	settings customized"
		skip="$skip $pinerc"
	else
		TMPFILE=`mktemp` || exit 1
		sed     -e 's!^inbox-path=$!inbox-path={localhost/ssl/novalidate-cert}inbox!' \
			-e 's!^folder-collections=$!folder-collections="Local IMAP" {localhost/ssl/novalidate-cert}[]!' \
			-e 's!^feature-list=$!feature-list=enable-lame-list-mode,\n\tcombined-subdirectory-display,\n\tcombined-folder-display,\n\tquell-empty-directories!' \
		< $pinerc > $TMPFILE
		echo mv $TMPFILE $pinerc
	fi

	echo
done

echo "You should check these users by hand:"
for u in $skip; do echo $u; done

Account Creation

You want your new users to hit the ground running, so create Maildir in /etc/skell. You might also put a fixed up .pinerc in there.

mkdir -p /etc/skel/Maildir/new mkdir -p /etc/skel/Maildir/cur mkdir -p /etc/skel/Maildir/tmp