[Tech] Filesystem /var full of spam!

[Edmund R. MacKenty]

I have been receiving dozens of mailer errors from green-rainbow.org
indicating that postfix was unable to queue messages.  The /var filesystem
was completely full.  The system has been pretty much overloaded by spam,
which is not surprising because the recent increase in spam levels has been
using methods which the old SpamAssassin rulesets do not detect.  Because
the situation was causing *all* incomming mail to fail, I decided I had to
do something drastic: I deleted about 14MB of mail from the deferred queue
that had the "Host not found" message associated with them.

I ordinarily never access the body of a queued mail message, because
opening a user's mail is just not done.  But it was pretty clear that all
the messages that had been deferred because their destination hostname does
not exist were spam bounces.  To be sure, I peeked at the headers of about
12% of them and they were all bounced spam.  I looked at the bodies of about
20 messages just to make sure.

The messages removed accounted for only about 20% of the queued messages.
I have not looked at any messages that are deferred for other reasons
(timeouts and so on).  Because deferred mail can sit in the queue for up to
a week (perhaps more, I didn't check the config), the current high rate of
incomming spam will continue to fill the /var filesystem and repeatedly
cause this problem.

To fix it, I think we should upgrade our SpamAssassin rules with some new
ones that recognize the newer spam styles and flag them.  I don't know
enough about SpamAssassin rules yet to do this; does anyone else?

For the record, here's what I did to clean things up.  Under
/var/spool/postfix, the "defer" directory contains messages about why mail
has been deferred, and the "deferred" directory contains the messages
themselves.  So I wrote a pipe that gets the queue pathnames of the
undeliverable mail and removes that entry from both directories.  Here's
the commands I used:

# Get into the spool directory:
cd /var/spool/postfix

# Look at some of the bounced spam messages.  The tr(1) is needed because
# postfix stores the messages with some strange header separators.  This
# didn't get everything, but it did limit the output to just the headers.
# Note that the grep only looks at 2 of the 16 second-level subdirectories.
(cd defer; grep -l 'Host not found' */[5A]/*) | \
	while read f; do \
		echo $f; \
		tr '[:cntrl:]' '\n' < $f | egrep '^(To|From|Subject):'; \
	done | less

# This is what actually removes them all:
(cd defer; grep -l 'Host not found' */*/*) | \
	while read f; do echo defer/$f deferred/$f; done | \
	xargs rm

I highly recommend that people do *NOT* use this as a "normal" way to clean
things up, as this has the potential to delete real mail if you are not
careful.  I'm just documenting what I did.

While writing this message, about 1MB of more mail has been enqueued.  The
mail queue still contains some 2750 messages (and rising).  We need to get
our spam filter updated ASAP!
	- MacK.
-----
Edmund R. MacKenty

"We demand rigidly defined areas of doubt and uncertainty!" -- Broomfondle.