[Tech] Filesystem /var full of spam!

[Daniel Melnechuk]

THanks MacK!

We also have another problem... the IP net block our address is in is  
now listed as follows:


http://www.completewhois.com/cgi2/rbl_lookup.cgi? 
query=38.113.6.73&display=webtable

spews-level2

Level 2 - Spam Supporting Networks  (monitored blocks)

38.113.0.0/20 --> [2] neomill/rackpoint, see http://spews.org/ask.cgi? 
S2888


So some hosts in the IP address block are thought to be purchased by  
spammers or are supporting spammers.

I have contacted our co-location provider to ask them to escalate the  
problem.

As for spam, our spamassassin is installed but it is not really  
working i think. So upgrading it and making it work is a great idea.  
MacK, have you done that before?

Peace,
Dan

On Nov 14, 2006, at 11:22 PM, Edmund R. MacKenty wrote:

> I have been receiving dozens of mailer errors from green-rainbow.org
> indicating that postfix was unable to queue messages.  The /var  
> filesystem
> was completely full.  The system has been pretty much overloaded by  
> spam,
> which is not surprising because the recent increase in spam levels  
> has been
> using methods which the old SpamAssassin rulesets do not detect.   
> Because
> the situation was causing *all* incomming mail to fail, I decided I  
> had to
> do something drastic: I deleted about 14MB of mail from the  
> deferred queue
> that had the "Host not found" message associated with them.
>
> I ordinarily never access the body of a queued mail message, because
> opening a user's mail is just not done.  But it was pretty clear  
> that all
> the messages that had been deferred because their destination  
> hostname does
> not exist were spam bounces.  To be sure, I peeked at the headers  
> of about
> 12% of them and they were all bounced spam.  I looked at the bodies  
> of about
> 20 messages just to make sure.
>
> The messages removed accounted for only about 20% of the queued  
> messages.
> I have not looked at any messages that are deferred for other reasons
> (timeouts and so on).  Because deferred mail can sit in the queue  
> for up to
> a week (perhaps more, I didn't check the config), the current high  
> rate of
> incomming spam will continue to fill the /var filesystem and  
> repeatedly
> cause this problem.
>
> To fix it, I think we should upgrade our SpamAssassin rules with  
> some new
> ones that recognize the newer spam styles and flag them.  I don't know
> enough about SpamAssassin rules yet to do this; does anyone else?
>
> For the record, here's what I did to clean things up.  Under
> /var/spool/postfix, the "defer" directory contains messages about  
> why mail
> has been deferred, and the "deferred" directory contains the messages
> themselves.  So I wrote a pipe that gets the queue pathnames of the
> undeliverable mail and removes that entry from both directories.   
> Here's
> the commands I used:
>
> # Get into the spool directory:
> cd /var/spool/postfix
>
> # Look at some of the bounced spam messages.  The tr(1) is needed  
> because
> # postfix stores the messages with some strange header separators.   
> This
> # didn't get everything, but it did limit the output to just the  
> headers.
> # Note that the grep only looks at 2 of the 16 second-level  
> subdirectories.
> (cd defer; grep -l 'Host not found' */[5A]/*) | \
> 	while read f; do \
> 		echo $f; \
> 		tr '[:cntrl:]' '\n' < $f | egrep '^(To|From|Subject):'; \
> 	done | less
>
> # This is what actually removes them all:
> (cd defer; grep -l 'Host not found' */*/*) | \
> 	while read f; do echo defer/$f deferred/$f; done | \
> 	xargs rm
>
> I highly recommend that people do *NOT* use this as a "normal" way  
> to clean
> things up, as this has the potential to delete real mail if you are  
> not
> careful.  I'm just documenting what I did.
>
> While writing this message, about 1MB of more mail has been  
> enqueued.  The
> mail queue still contains some 2750 messages (and rising).  We need  
> to get
> our spam filter updated ASAP!
> 	- MacK.
> -----
> Edmund R. MacKenty
>
> "We demand rigidly defined areas of doubt and uncertainty!" --  
> Broomfondle.
> _______________________________________________
> Tech mailing list
> Tech at green-rainbow.org
> http://www.green-rainbow.org/mailman/listinfo/tech
>