Anti-SPAM Revisited

I use cbl.abuseat.org, bl.spamcop.net, dynamic.spamlists.tqmcube.com and a return code of 127.0.0.10 from dnsbl.sorbs.net. If you want to accept direct to MX mail from dynamic IPs remove the last two. cbl.abuseat.org would be a good list to add to your current checks either way.

Another thing to concider is greylisting. This gives a message a temp failure the first time delivery is attempted and will let it pass on reattempts after a configurable delay (eg 10, 15, 20 minutes or so). Many spam trojans on infected machines will not reattempt at all. Those that do will have given the blocking lists you use an extra period of time to list their IPs. This is as easy to implement in your setup as "apt-get install postgrey".

Lastly, configure your mailsever to be more strict about what it accepts for a HELO value and in enforcement of other RFC standards. For example quite a few pieces of spam get rejected by mail server for failing to wait for the SMTP banner before saying HELO.

I know a lot of people hate it, because they feel that it is to restrictive, and breaks the free flowing nature of email, but I have been using TMDA (Tagged Message Delivery Agent) for several years with great results. TMDA uses whitelists. When I send an email to someone they are added to my whitelist and can send me email unhindered. If someone sends me email out of the blue, they get a reply asking for confirmation. When they confirm they are added to the whitelist and the original message is released.

I can also create dated email address with TMDA on the fly which only work for several weeks, then require confirmation. These addresses work well say when I am registering for a website and want to receive the password, but not be bothered later with marketing crap.

TMDA may not be for everyone, but it works well for me and the few users that I support.

@drware:

I know a lot of people hate it, because they feel that it is to restrictive, and breaks the free flowing nature of email, but I have been using TMDA (Tagged Message Delivery Agent) for several years with great results. TMDA uses whitelists. When I send an email to someone they are added to my whitelist and can send me email unhindered. If someone sends me email out of the blue, they get a reply asking for confirmation. When they confirm they are added to the whitelist and the original message is released.

I can also create dated email address with TMDA on the fly which only work for several weeks, then require confirmation. These addresses work well say when I am registering for a website and want to receive the password, but not be bothered later with marketing crap.

TMDA may not be for everyone, but it works well for me and the few users that I support.

I would suggest rereading the Linode Terms of Service and Acceptable Use Policy. Since most spam uses forged from addresses, and since some of these addresses do exist, you are sending unsolicited bulk mail. We treat reports about Challenge/Response systems just as we treat any other spam report, account termination is not out of the question.

I added cbl.abuseat.org after my other two RBLs. No hits to that one yet, but its new. :)

I installed the postgrey package, but I haven't enabled it yet. I'm extremely conservative with my mail server because I have some clients that won't put up with e-mail not working (or bounces or whatever).

Should I build a pretty comprehensive whitelist_clients.local file for postgrey?

Should I enable auto-whitelisting? It seems like a good idea to me, but it doesn't seem to be enabled by default.

(I've always thought TMDA was excellent in theory, but the inability to determine an e-mail's true sender with 100% accuracy renders it not just useless, but harmful. I'd disable it unless you manually monitor a lot of its activities which kind of defeats its purpose anyway.)

@untitled9:

I added cbl.abuseat.org after my other two RBLs. No hits to that one yet, but its new. :)

Yeah, I have it checked first in my case, it will have entries from your other dnsbls but in my tests seemed to catch stuff others miss.

@untitled9:

I installed the postgrey package, but I haven't enabled it yet. I'm extremely conservative with my mail server because I have some clients that won't put up with e-mail not working (or bounces or whatever).

Should I build a pretty comprehensive whitelist_clients.local file for postgrey?

Should I enable auto-whitelisting? It seems like a good idea to me, but it doesn't seem to be enabled by default.

I use greylistd with exim personally. It has a feature that acts what I assume auto-whitelisting is in postgrey enabled by default. I think it is worthwhile, especially if old data is removed from the whitelist when it hasn't been used in some amount of time.

@untitled9:

(I've always thought TMDA was excellent in theory, but the inability to determine an e-mail's true sender with 100% accuracy renders it not just useless, but harmful. I'd disable it unless you manually monitor a lot of its activities which kind of defeats its purpose anyway.)

Right, I think comunism/socialism is great in theory… ;-)

@untitled9:

Basically: What's the spam blocking flavor of the month?

sbl-xbl.spamhaus.org on the mail server. Just that one DNSBL seems to catch about 60% of my spam.

Plus SpamAssassin with a few custom rules from rulesemporium.com.

That catches about 60% of whats left.

SpamAssassin is a resource hog, but if you don't get thousands of mails a day it should not be a problem.

I'd stay well clear of TMDA, it's a solution thats as bad as the problem.

> SpamAssassin is a resource hog, but if you don't get thousands of mails a day it should not be a problem.

And therein lies the problem. My server handles over 2,000 e-mails a day, over 60% of which are spam along with about a dozen relatively low traffic web sites. (So I guess SpamAssassin would only see < 1,000/day.)

Spam bombs and the occassional 5mb e-mail to 30 co-workers bring the server to a crawl while ClamAV scans everything. I'm afraid adding another resource hog that scans every e-mail onto the chain will really bring down the server.

Should I just quit whining and install SpamAssassin?

@untitled9:

> SpamAssassin is a resource hog, but if you don't get thousands of mails a day it should not be a problem.

And therein lies the problem. My server handles over 2,000 e-mails a day, over 60% of which are spam along with about a dozen relatively low traffic web sites. (So I guess SpamAssassin would only see < 1,000/day.)

Spam bombs and the occassional 5mb e-mail to 30 co-workers bring the server to a crawl while ClamAV scans everything. I'm afraid adding another resource hog that scans every e-mail onto the chain will really bring down the server.

Should I just quit whining and install SpamAssassin?

A very rough estimate of how much mail a linode 80 could handle:

Mail that gets rejected by the MTA due to a DNSBL wastes almost no time. I'm ignoring those.

A big HTML mail thats comes in, gets scanned by spamassassin, and gets forwared externally takes a total of 13 seconds end to end to process. Lets say 13 seconds is average for all mail.

Thats 4.6 mails a minute, 276 an hour, 6624 a day.

I have never run ClamAV so don't have any idea how long that takes so I'm guessing it doubles the processing time. I don't think it does anything on mails without attachments so lets say it averages out at another 13 seconds

per mail. Thats 3312 mails a day.

I'm not taking into account the fact that you can process two mails at once with very little slowdown, or the fact that your mail isn't going to come in evenly spaced. The bulk of it will likely come in during the daytime. It's a very rough estimate, but your linode should be able to handle running spamassassin as well.

The only real way to tell is to try it and see what happens. You can always turn spamassassin off if your linode starts swap thrashing.

@mikegrb:

I would suggest rereading the Linode Terms of Service and Acceptable Use Policy. Since most spam uses forged from addresses, and since some of these addresses do exist, you are sending unsolicited bulk mail. We treat reports about Challenge/Response systems just as we treat any other spam report, account termination is not out of the question.

Your policy, or rather this application of it, seems a little harsh. What happens about out-of-office autoresponders replying to spam that leaks through the filters? They seem to fit your criteria for ubm, yet they are a legitimate business practice that has been screwed up by spammers. Half the crap in my postmaster inbox is the result of this sort of thing.

@pclissold:

Your policy, or rather this application of it, seems a little harsh. What happens about out-of-office autoresponders replying to spam that leaks through the filters? They seem to fit your criteria for ubm, yet they are a legitimate business practice that has been screwed up by spammers. Half the crap in my postmaster inbox is the result of this sort of thing.

Out of office autoresponders aren't purposefully exploited by spammers like challenge response systems are though. That said, if we received a legitimate complaint about an autoresponder, we would suggest that it be disabled.

@mikegrb:

Out of office autoresponders aren't purposefully exploited by spammers like challenge response systems are though.

Point taken.

Note to self: Think before posting :? .

@untitled9:

What does everyone else use?

A LOT of procmail and perl and shellscripts centered around a several years old version of bogofilter.

(I use procmail on outgoing e-mails also.)

Due to the spamtraps and all the automatic whitelisting it's been working way beyond great, for a number of years.

Now I'm moving/porting it all to my server here, so I've started to rewrite it all; starting with adding a cpl of rbls (relays.ordb.org sbl-xbl.spamhaus.org) at smtp-level.

Thanks to everyone for the great responses. cbl.abuseat.org is now catching a respectable amount of spam every day that the other 2 DNSBLs miss, but its obvious that DNSBLs can only do so much.

I have postgrey installed and configured but not enabled because I really don't like the idea of breaking SMTP to block spam. I know technically it shouldn't break anything, but a quick glance at the long list of exceptions in postgrey's whitelist was enough to make me nervous.

I think its time to bite the bullet and install SpamAssassin. If it starts thrashing my swap too much, I guess I'll just have to upgrade my Linode 160. I've also been thinking about moving our non-critical web sites to a super cheap host like GoDaddy, but I'm not sure that would free up very many resources.

Thanks again for your help!

1. Do as much rejecting as possible at SMTP-level; using DNSBLs is probably the easiest way to do this with most mailservers. Be very careful about what DNSBLs you use, you rather want 10'000 spam extra to pass than 1 ham get rejected here. (Remember, you don't even have a spam-folder where you can find the ones that got rejected.)

2. Whitelist as much as possible; check the outgoing e-mails and automagically whitelist every person that someone at your company writes to. Don't forget to also keep track of all message-ids, that way you can also whitelist all replies to all sent e-mails even if they don't come from the address it was sent to; you also whitelist e-mails (e-mailaddresses and/or message-ids) that are replies to sent e-mails… that way you whitelist whole threads on mailinglists even if you don't whitelist the mailinglist itself.

3. Blacklist as much as possible… Here you probably want to use something like procmail, making it possible for you to block exactly the kinds of e-mails that you NEVER get as ham, like e-mails written in chinese and things like that.

4. Use the white- and blacklisted e-mails to train whatever bayesian(ish) filter it is that you're using, let it be something like bogofilter or spamassassin.

5. Here's where you take what's left and let the software "guess" if it's spam or not. Just don't forget that if the mail is for a whole company there are some people into things which others are not (marketingpeople getting lots of flashy HTML-emails, techguys might only get HTML in spam)… So if you're filter of choice is bayesian(ish)oriented you problably want individual databases, or at least one for each group/department.

Just my 2 c…

I have tried installing postgrey, but it just appears to be blocking all access to my server, is there a good howto anywhere on how to install postgrey with postfix? Or does anyone have any tips?

Cheers.

@cherring:

I have tried installing postgrey, but it just appears to be blocking all access to my server, is there a good howto anywhere on how to install postgrey with postfix? Or does anyone have any tips?

Well, it does work by blocking e-mails, just not blocking them forever… so maybe you just need to wait a lil bit longer before you see it working as it should. Just a thought.

Or maybe the database's corrupted, try to delete it and start over.

Or use the perl at the end of this page to write your own solution:

http://www.postfix.org/SMTPDPOLICYREADME.html

Well I'm not sure what it was doing but I am trying to get it going from scratch again. Does postgrey do any logging that I can check to see exactly how it is handling the incoming mail, I am handling it on a tcp port, do I need to open a port on my firewall?

Cheers.

EDIT:

Decided against postgrey in favour of sqlgrey and am very impressed, easy to configure and get running, only problem was it isn't part of gentoo portage, but there is a link to an ebuild on the sqlgrey website and the gentoo wiki has a great sqlgrey howto.

Highly Recommended :D

Cheers Guys.

@cherring:

[…]

Decided against postgrey in favour of sqlgrey and am very impressed, easy to configure and get running, only problem was it isn't part of gentoo portage, but there is a link to an ebuild on the sqlgrey website and the gentoo wiki has a great sqlgrey howto.

Highly Recommended :D

Cheers Guys.

Thanks :-) Just do a "tcptraceroute sqlgrey.bouton.name 80" to look up where SQLgrey default whitelists are hosted :-)

Nice to read that on the forum.

gyver, SQLgrey's dev.

@gyver:

Thanks :-) Just do a "tcptraceroute sqlgrey.bouton.name 80" to look up where SQLgrey default whitelists are hosted :-)

Nice to read that on the forum.

gyver, SQLgrey's dev.

Mate it's a pleasure to give something a rap when it's been so helpful. I detest spam and was receiving enough for SpamAssassin to take up a lot of resources. Sqlgrey is blocking heaps of spam so spam assassin only has to scan the stuff that passes grey lisiting, but not much does, freed up tons of resources.

I am highly Impressed and would recommend to it to anyone.

Cheers. :D

@untitled9:

I know there are a number of old forum discussions about preferred anti-spam measures, but I thought I'd just see what the current anti-spam flavor of the month is.

My flavor of the month is DEA (Disposable Email Addresses). Never use your real email address for anything, and it will never get any spam. Front your REA with DEA's and delete them if and when they start forwarding spam to your REA.

As its creator, I'm naturally partial to www.e4ward.com. (And it's got a great host provider:)

@areider1:

My flavor of the month is DEA (Disposable Email Addresses). Never use your real email address for anything, and it will never get any spam.

Just use a strange enough looking address to avoid dictionary-attacks and it works, but that "solution" just doesn't work if you want to build some sort of e-life… spend years on mailinglists, have friends contact you even though you haven't talked online for months or years… or get updates regarding software that you use and so on…

Sure, it's possible to use different e-mailaddresses at all the places, only kill the ones that start to get spam; but in the long run it's just so much easier to get a good spamfilter set up and then use the same address year after year after year without worring if the spammers are going to get it or not.

I find the key is to get rid of them as early in the transaction as possible. Running Postfix, I have the following:

In /etc/postfix/helo_checks:

REJECT You are not in # Somebody HELO'ing with our IP address?

REJECT You are not # Somebody HELO'ing as "localhost?" Impossible, we're "localhost"

localhost REJECT You are not me

Judging from the logs, this catches almost a third of my spam.

Add to that a couple specific offshore ISP blocks ending in .tw and .pl, and I've tossed a large fraction before I even get to the RBL's.

I get a couple pump and dump stock spams a week. I'm still hunting for a good solution to these. The multipart ones that include the ad in text or html are relatively easy. You can block those with a body check like:

/Recommendation.*[B,b][u,U][Y,y]/ REJECT Stock Spam.

/5 Day Target price\:/ REJECT Stock Spam.

But this is an expensive filter, and not real accurate. The gibberish with the ad as a gif is tough. You need to OCR it to inspect the content, and you're already way late in the transaction. You've already received it, taken the bandwidth hit, and now you're consuming CPU/memory on it.

Postgrey works wonders.

I use the following.

• qmail supervised under daemontools

• Clamav (clamdscan+qscanq)

• RBL: sbl-xbl.spamhaus.org

• Drop connections from hosts with no PTR record and warn with a URL

• Delay connections from hosts sending mail with bare LFs.

• DSPAM for spam filtering (very, very effective)

I'm not sure if self promotion is allowed around these 'ere parts so feel free to slap me otherwise.

I have an extremely efficient spam detection setup and i recently came to Linode as many of my customers were asking about dedicated spam setups. Well most dont do enough e-mail volume to require their own dedicated server so hence why i'm at Linode.com

It seems like SPAM is a big problem for alot of people here so I'm offering a deal for Linode users. £5/month to scan 1 domain, regardless of users or mail volume. If that sounds expensive well my smallest customer is paying me 25 times that to scan 200 mail boxes. Why? Because it works and cuts almost 100% of spam with 0 false positives!

If you want a free trial feel free to PM me and i can set you up for a couple weeks for free, all you have to do is point your MX record towards my server and give me the IP you want clean spam to go to.