IP Reputation

My current primary spam related interest is my new public free combined whitelist and blacklist. I need your data.

Past

In February 2010, I thought it would be good to block spam with SPF. Then I became aware of its forwarding breaking problem, a barrior to adoption. So I created MTX, which was also then not adopted.

Stuff from 2006

The mail server software (MTA) I use is Postfix.

Greylisting, SpamAssassin, SpamProbe, Image Spam, DNSWL, and Viruses

It's not necessary to do this all at once. That would probably be overwhelming. I recommend setting up one piece at a time.
Greylisting
I use Postgrey for greylisting. Basically if an email server which I have not heard from before tries to deliver email to me, it is immediately given a temporary error and asked to try again later. Well behaved mail servers will all try again, many spammers won't.
SpamAssassin
SpamAssassin is rule based, and uses a number of online services. It does things like "This includes a big html font and the sender address has been reported for spamming, therefore this is spam."
SpamProbe
SpamProbe is a (multi-word token bayesian) pattern learning filter. You tell it "here are the spams I'm getting, and here are the non-spams I'm getting", and it figures out on its own what the differences in the patterns are.
Image Spam
All image spams are blocked by postfix body_checks
DNSWL
DNSWL is a list of mail servers known not to send spam (White List), accessible in useful ways (including DNS).
Viruses
Viruses are filtered with ClamAV

How

  1. Anything that might be image spam is rejected by my postfix body_checks.
  2. Anything ranked high or medium by DNSWL.org skips all other filtering and goes right to my inbox. (Postfix Configuration)
  3. The rest is greylisted via postgrey from postfix.
  4. Procmail filters out viruses via ClamAv.
  5. Procmail filters remaining spam via SpamAssassin and SpamProbe. If they agree it's spam the email goes in my spam folder and I never look at it. If they agree it's not spam it goes to my inbox. If they disagree it goes in other folders which I check.
  6. After verifying everything is in the right place I run my retraining script.

Image spams are, I believe, a uniquely effective way to circumvent filtering, because all of their spamminess is contained in an inlined attached image. So I reject all images containing (basically) 'src="cid:' using the Postfix body_checks regex (pattern definition):
/\bsrc\s*=(?:3D)?\s*["']?cid:/ REJECT Your email was rejected because you embedded an attached image in the body.
This method ensures that legitimate senders will get an error message and will not result in error messages being sent to forged sender addresses from spam.

The postfix body_checks are implemented by putting this line in your main.cf: body_checks = pcre:/etc/postfix/body_checks # image spam regex goes in this file
Then in the file /etc/postfix/body_checks you put the line:
/\bsrc\s*=(?:3D)?\s*["']?cid:/ REJECT Your email was rejected because you embedded an attached image in the body.
The regex is designed to match things that look similar to '<img src="cid:spam.jpg">'. The "pcre" part of the main.cf line indicates this is a Perl Compatible Regular Expression which means you can look it up in the perlre man page. The meaning of each piece is as follows:
/ beginning of the regex
\b Matches "word boundaries", the point between the whitespace before "src" and the beginning of "src"
src the "src" part of the img tag
\s* any amount of whitespace (spaces, tabs, etc.), or none
= the "=" in the img tag
(?:3D)? quoted printable email encoding can replace an "=" with "=3D", this handles it
\s* any amount of whitespace (spaces, tabs, etc.), or none
["']? single or double quote, "?" allows for it to be missing
cid: "cid:" the part of an img tag url that replaces http: and means it's an attached file, not hosted on a webserver
/ end of the regex
"REJECT Your email was rejected because you embedded an attached image in the body." defines the error message that will be sent only to legitimate senders.




my ~/.procmailrc config file

The relevant part of my postfix main.cf: 
smtpd_recipient_restrictions = 
       permit_mynetworks
       permit_sasl_authenticated
       reject_unauth_destination
       check_client_access cidr:/home/darxus/dnswl/postfix-dnswl-header # add X-DNSWL headers
       check_client_access cidr:/home/darxus/dnswl/postfix-dnswl-permit # skip greylisting
       check_policy_service inet:127.0.0.1:60000 # greylisting

body_checks = pcre:/etc/postfix/body_checks # image spam regex goes in this file


To get SpamAssassin to pay attention to DNSWL ranks I added these rules to /etc/spamassassin/local.cf: 
header   RCVD_IN_DNSWL        X-DNSWL =~ /^none/
score    RCVD_IN_DNSWL        -0.1
describe RCVD_IN_DNSWL        Sender listed at http://www.dnswl.org/, no trust

header   RCVD_IN_DNSWL_LOW    X-DNSWL =~ /^low/
score    RCVD_IN_DNSWL_LOW    -1
describe RCVD_IN_DNSWL_LOW    Sender listed at http://www.dnswl.org/, low trust

header   RCVD_IN_DNSWL_MED    X-DNSWL =~ /^med/
score    RCVD_IN_DNSWL_MED    -4
describe RCVD_IN_DNSWL_MED    Sender listed at http://www.dnswl.org/, medium trust

header   RCVD_IN_DNSWL_HI     X-DNSWL =~ /^hi/
score    RCVD_IN_DNSWL_HI     -8
describe RCVD_IN_DNSWL_HI     Sender listed at http://www.dnswl.org/, high trust

header   RCVD_IN_DNSWL_NO     X-DNSWL =~ /^No$/
score    RCVD_IN_DNSWL_NO     0.1
describe RCVD_IN_DNSWL_NO     Sender *not* listed at http://www.dnswl.org/


SpamAssasin rules for use without Postfix (causes more network load for everyone): 
header   RCVD_IN_DNSWL        eval:check_rbl('dnswl-firsttrusted', 'list.dnswl.org.')
score    RCVD_IN_DNSWL        -0.1
describe RCVD_IN_DNSWL        Sender listed at http://www.dnswl.org/, no trust

header   RCVD_IN_DNSWL_LOW    eval:check_rbl_sub('dnswl-firsttrusted', '127.0.\d+.1')
score    RCVD_IN_DNSWL_LOW    -1
describe RCVD_IN_DNSWL_LOW    Sender listed at http://www.dnswl.org/, low trust

header   RCVD_IN_DNSWL_MED    eval:check_rbl_sub('dnswl-firsttrusted', '127.0.\d+.2')
score    RCVD_IN_DNSWL_MED    -4
describe RCVD_IN_DNSWL_MED    Sender listed at http://www.dnswl.org/, medium trust

header   RCVD_IN_DNSWL_HI     eval:check_rbl_sub('dnswl-firsttrusted', '127.0.\d+.3')
score    RCVD_IN_DNSWL_HI     -8
describe RCVD_IN_DNSWL_HI     Sender listed at http://www.dnswl.org/, high trust

meta     RCVD_IN_DNSWL_NO     !RCVD_IN_DNSWL
score    RCVD_IN_DNSWL_NO     0.1
describe RCVD_IN_DNSWL_NO     Sender *not* listed at http://www.dnswl.org/


To get SpamProbe to pay attention to the DNSWL I ran this command (which will overwrite your config file): 
spamprobe -H+x-dnswl create-config  # needs to be lowercase


To install all relevant packages on a Debian based system, run:
aptitude update;aptitude install postgrey spamassassin spamprobe clamav
Versions on Debian Stable are typically pretty ancient. To keep up with spam it can be pretty important to verify you're running a reasonably recent version of this software.
How do I keep my spammers off my network?
Submit your mail server to DNSWL.org to be whitelisted.
Comment on this page.
Return to Darxus' home page.
Sat Apr 16 11:25:46 EDT 2011