What is procmail and why is it using up all my memory?

I recently came across a problem that took longer to resolve than I would have liked. I had a box that was responsible for running a Java process and nothing else. When the Java process mysteriously died one day, I went to restart it and it failed immediately due to lack of memory. That was odd... the box had far more memory than I was requesting. When I started looking at top, I saw that there were dozens of procmail processes chewing up all available memory. Dropping out of top and looking at the process list, I saw that it was more than dozens - over 10,000 procmail processes were taking over the entire system. So... where did these come from and what were they doing? (And how should I get rid of them?)

The first order of business was figuring out just what the heck "procmail" was in the first place. I figured from the name that it was related to Linux's mail subsystem, but it wasn't a process I'd ever encountered in the past. After a bit of research, I discovered that procmail was apparently Unix's equivalent to Outlook's "rules" folder: it automatically categorizes messages into appropriate folders based on mail headers. It was pretty obviously something that I had just gotten for "free" with the CentOS distribution; it was being invoked on each mail forward and what do you know, there was a corresponding sendmail process for each procmail, with the counts of each going up.

Googling "too many procmail processes" didn't help, but I did come across a hit that suggested checking /var/log/maillog. I did, and found: 

timeout waiting for input from local during Draining Input

Now, this was something I could Google, and it led me here. 

If e-mail is delivered to a program which generates too much
output, then sendmail may issue an error:

timeout waiting for input from local during Draining Input

Make sure that the program does not generate output beyond a
status message (corresponding to the exit status).  This may
require a wrapper around the actual program to redirect output
to /dev/null.

So what would be generating mail messages? Well, by default, when a cron job runs, it mails the results to the owning user. Sure enough, looking at the user's crontab, there was a cron process that ran every minute. If cron was mailing the user every minute, there should be... 

$ mail
"/var/spool/mail/ec2-user": 110263 messages 387 new 110261 unread

Ah-ha... 110,263 unread mail messages. The mail agent was taking ages to start up, too. After a delete * in the mail console, the problematic procmail processes started disappearing and enough memory was freed up to run my daemon process. Still, the mail spool continued to fill up — now that I had the system in a stable state, I could check the mail to see what, exactly, it was so upset ahbout. It turned out that there was a problem with one of the cron scripts. A quick tweak of the misbehaving script cleared up the output problem and I was back up and running.

Add a comment:

Completely off-topic or spam comments will be removed at the discretion of the moderator.

You may preserve formatting (e.g. a code sample) by indenting with four spaces preceding the formatted line(s)

Name: Name is required
Email (will not be displayed publicly):
Comment:
Comment is required
My Book

I'm the author of the book "Implementing SSL/TLS Using Cryptography and PKI". Like the title says, this is a from-the-ground-up examination of the SSL protocol that provides security, integrity and privacy to most application-level internet protocols, most notably HTTP. I include the source code to a complete working SSL implementation, including the most popular cryptographic algorithms (DES, 3DES, RC4, AES, RSA, DSA, Diffie-Hellman, HMAC, MD5, SHA-1, SHA-256, and ECC), and show how they all fit together to provide transport-layer security.

My Picture

Joshua Davies

Past Posts