The Fight Against Spam

Spam: Death By A Million Paper Cuts
The organization I work for receives 19-20 UCEs (Unsolicited Commercial Email) per second, 1.7-2 million potential UCEs per day, 11.7 million UCEs per week. I only have 13,000 email users. These users were desperate and email for many was unusable. Me and another co-worker had 3 months to implement a plan to get rid of most of it. I had to document every step we took and I had one shot to accomplish this. At the end of that time it had to work and it had to be noticeable.

I am not an expert on spam. However, I have learned many things about UCEs and what can be done to fight it and how to adjust to UCEs' dynamic nature.

Our goal was to:
* Replace our aging Linux/Sendmail gateway
* Use a sane and stable MTA (i.e Postfix, Exim, Qmail etc)
* Prevent spammer dictionary attacks
* Block certain countries (country DNSBL) from sending spam and make our domain invisible to new spammers in those countries.
* Accept only email that is RFC 821 compliant
* Use two or three DNSBLs (PSBL, Spamcop and Spamhaus) via datafeeds and local DNS lookups.
* Implement NoListing and Greylisting.
* Minimize false postives and keep them to a manageable level.
* Block as much spam as possible BEFORE any DATA was sent to keep network and server loads sane.
What follows is how we accomplished it. This is not a howto on the subject but I hope it will be useful to anyone that runs a mail server regardless of the size of the organization.

The Problem?: RFC 821
The problem with email/UCEs is the SMTP protocol is very trusting. In fact, it is one of the only protocols that does not require authentication. Later RFCs have enhanced SMTP to have authentication but largely this is not done. Simple Mail Transfer Protocol is simple. Below is all that is required to send an email. First two servers must open a connection. Anyone can do this. Note: S = Sender and R = Receiver.

The Sender opens a TCP connection to port 25 the SMTP port.
telnet mx.example.com 25
R: 220 mx.example.com Simple Mail Transfer Service Ready
S: HELO mx.myexample.com
R: 250 mx.example.com


Once this is completed successfully, the sender begins the rest of the transmission.
S: MAIL FROM:
R: 250 OK
S: RCPT TO:
R: 250 OK
S: DATA
R: 354 Start mail input; end with .
S: This is where all the email message/body goes
S: ...etc. etc. etc.
S: .
R: 250 OK
Once this happens the Sender is ready to close the transmission.
S: QUIT
R: 221 mx.example.com Service closing transmission channel
Now let's look at an example SMTP transmission on Postfix
Connected to mx.example.com.
Escape character is '^]'.
R: 220 mx1.example.com ESMTP Postfix
S: HELO myexample.com
R: 250 myexample.com
S: MAIL FROM:
R: 250 2.1.0 Ok
S: RCPT TO:
R: 250 2.1.5 Ok
S: DATA
R: 354 End data with .
S: Subject: Testing
S: this is a test
S: .
R: 250 2.0.0 Ok: queued as A277539820
S: QUIT
R: 221 2.0.0 Bye
Connection closed by foreign host.
The email is then sent on its way.

RFC 821 was replaced by RFC 1651 which extended the old RFC. Most if not all mail exchangers use the extensions even though the new Standard is backward compatible to the old RFC 821.
R: 220 example.com ESMTP Postfix
S: EHLO myexample.com
R: 250-myexample.com
R: 250-SIZE 10240000
R: 250-ETRN
R: 250-ENHANCEDSTATUSCODES
R: 250-8BITMIME
R: 250 DSN
S: MAIL FROM:
R: 250 2.1.0 Ok
S: RCPT TO:
R: 250 2.1.5 Ok
S: DATA
R: 354 End data with .
S: Subject: this is a test
S: This is a test
S: .
R: 250 2.0.0 Ok: queued as 87E9639828
S: QUIT
R: 221 2.0.0 Bye
Connection closed by foreign host.
Why go through all this?
The purpose of showing the above is to show where you want to stop most of the UCEs from entering your network; just before or just after the RCPT TO: command before you receive any data or the bulk of the email. The data stream up to that point would probably never get more than a couple hundred bytes. But once the data stream enters the data command the stream will explode to approximately 3000 bytes without an attachment. If you can stop UCEs before they send their payload you have saved costs in terms of bandwidth and server and network resources. Postfix makes this very easy to do.

The MTA (Mail Transfer Agent)
I have worked for years with sendmail and for the most part hated it. Sendmail is time consuming. The only satisfaction I would get was to get something configured correctly. But sendmail is still a difficult MTA to configure. When Postfix hit version 2.0 I converted to that. Why? Although sendmail is an excellent MTA and for years a "standard" workhorse that moved billions of emails over the Internet, Postfix is much more flexible and configuring it is a more sane task. So for me it was not a difficult decision to go with Postfix. I recommend Postfix or any other MTA over sendmail to anyone. The amount of time you spend learning it is far more rewarding than an equal amount of time learning sendmail. This is probably true with the other non-sendmail MTAs.

In Postfix at the very end of the file but just before smtpd_recipient restrictions add the following two lines. They help prevent spamming by slowing down dictionary attacks and making sure the sender is an 821 compliant mail system. Many spammers and zombie mailers are not. This will knock a few of them out.

smtpd_helo_required = yes - Sender must send a HELO command
disable_vrfy_command = yes - Sender cannot verify that an email address is valid

Postfix comes with built-in anti-UCE mechanisms. I will go over the important ones. There are many and it is not necessary to use all of them. The important ones are the ones placed under the smtpd_recipient_restrictions
. When you place options here order is important. If you mess up here you can cause your system to be an open relay. While using these you can test each one by using the warn_if_reject before each command, like so: warn_if_reject,
reject_unauth_destination,
Then you are now able to look in your maillog files and see reject_warnings to see the effects it would have had they been in effect.

Here are the basic anti-UCE controls Postfix uses:
reject_invalid_hostname - many non-legit senders issue nonsensical hostnames in the helo or ehlo stage so get rid of them. On the other hand, RFC 821 compliant mailers announce exactly who they are so we'll let them through.
reject_non_fqdn_hostname - This also checks RFC compliance. It should look like mx.example.com.
reject_non_fqdn_sender - Reject the request when the MAIL FROM address is not in fully-qualified domain form, as required by the RFC.
reject_non_fqdn_recipient - Reject the request when the RCPT TO address is not in fully-qualified domain form, as required by the RFC.
reject_unknown_sender_domain - Reject the request when Postfix is not final destination for the sender address, and the MAIL FROM address has no DNS A or MX record, or when it has a malformed MX record such as a record with a zero-length MX hostname. All legitimate mail exchangers should have MX records. Many spammers and zombies do not.
reject_unknown_recipient_domain - Similar to the one above, reject the request when Postfix is not final destination for the recipient address, and the RCPT TO address has no DNS A or MX record, or when it has a malformed MX record such as a record with a zero-length MX hostname.
permit_mynetworks - Now it is OK if it is from anywhere in my network.
reject_unauth_destination - * Postfix is mail forwarder: the resolved RCPT TO address matches $relay_domains or a subdomain thereof, and contains no sender-specified routing (user@elsewhere@domain), * Postfix is the final destination: the resolved RCPT TO address matches $mydestination, $inet_interfaces, $proxy_interfaces, $virtual_alias_domains, or $virtual_mailbox_domains, and contains no sender-specified routing (user@elsewhere@domain).

If you noticed and have referred to the legitimate email transmission above, you will see that all the above controls will reject email before it sends its payload when it hits the DATA command. On days when I get blasted by spammers the above directives kill up to 10% of the spam alone. Obviously, this is not enough but it is low cost and necessary.

The next restrictions you want to use are block lists. This can be a very slippery area and not one to take lightly.

Why Block Lists?
I once hated them. They produced many false positives and for the most part were operated by questionable people. Getting off a list was nearly impossible for some and seemed irrational. I had at one time been a victim of spamcop.net. In short, I hated block lists. Some time ago Al Iverson started dnsbl.com and is an excellent resource reviewing and analyzing the various block lists. His criteria is simple; percentage of accuracy and percentage of false positives. The higher the accuracy and the lower the false positives the better the overall rating of the list.

An RBL (Realtime Block List) is like having a staff that does nothing but checks reports of spam. Very large commercial ISPs do in fact hire a number of people that do just that. Google, Yahoo, MSN and AOL have people on duty that check spam reports. I don't have that luxury and probably 99% of you don't either.

There is a downside to using a block list. There are lots of them. You have to consider what will happen if you use them. Even though Spamhaus and lately Spamcop have extremely low false positives you may still have a problem or two occaisionally. Don't overly rely on them. They will block lots of spam but unless you use extremely aggressive lists spam will get through. No one method works but a combination of methods to block spam will get rid of most of it. Don't rely entirely on them.

I chose only 3 block lists...none are real aggressive: Spamhaus Zen (includes sbl, xbl and pbl), Spamcop and PSBL(Passive Spam Block List). I chose these because of their extremely low false postives and their high percentage of accuracy, according to dnsbl.com. Because of the volume of the email that we receive, I had to subscribe to the above lists in particular Spamhaus. If your email volume is not more than 1000s per day then it should not be necessary. In any case it is a bargain.

In Postfix in the section smtpd_recipient_restrictions and after the permit statement, we place the RBLs like so:
reject_rbl_client bl.spamcop.net,
reject_rbl_client zen.dnsbl,

The first statement above will look up the IP address of the sender at bl.spamcop.net. If it is on their list then it gets blocked after the MAIL FROM command. The second statement above however will not work until you install and configure BIND 9.x and rbldnsd. More about this in a minute.

Blocking Countries: YMMV
This is real controversial for a lot of email administrators and many administrators do not do it. I do not recommend it unless you do a lot of analysis of your log files and do a comprehensive check of your business rules. If you do not take the time to do this analysis then don't block countries at all or you will be sorry. This is especially true if e-commerce is a critical part of your business. YOU HAVE BEEN WARNED.

I spent about a month analyzing log files to determine where most of our spam was coming from and I developed a "Top 10" list of countries and checked them against the Whois registry.

Next I wanted to minimize my bandwidth to the Internet so I installed BIND 9.x and rbldnsd to do local lookups of my RBLs and Blocked countries list that I had compiled. I wanted all the RBLs and blocked countries to be looked up locally and I accomplished this by creating new zones. The idea is to use named to do all lookups and for the dnsbl zones (spamhaus, spamcop and countries I wanted to block) named then forwarded the lookup to the rbldnsd name server. This required that I install two name servers. However, rbldnsd is very low on server resources, fast and efficient. It is perfect for these kinds of look ups.

At this point, everything is complete for Postfix. You can put at the very end of the restrictions the following:
permit - Everything else moves on to the next stage.

So far, Postfix and block lists have done a majority of the work. But we still have a couple more methods to consider and use.

Nolisting and Greylisting
I decided that I could try nolisting or what some call "poor man's greylisting". Nolisting and Greylisting relies on your public DNS and its MX records. In my DNS I had the following in my BIND data file:
example.com. IN MX 10 mx1.example.com.
example.com. IN MX 20 mx2.example.com.
What the above means is that a sending mail server will first try to send mail to the the mx1.example.com. server first. If it can not do this it will try to send mail to mx2.example.com. Nolisting means that one or the other mail servers is never available. In fact, the server that is the nolisting server doesn't accept mail delivery and it doesn't need to exist. With nolisting you have to conduct some tests to see which server should be the nolisting server will be the most effective. There is some debate whether spammers skip the primary mail server and send right away to the secondary mail server. The thinking is that the secondary mail server will be a less protected server and spam will have easier entrance into your network. For this reason, I have greylisting on the secondary server.

Greylisting is a method of stopping spam by refusing the sender the first time it tries to send email to any user and the receiving server requesting that the sender send it again at a later time. Here is how it works, it is called a 'triplet'.

1. The IP address of the host attempting the delivery
2. The envelope sender address
3. The envelope recipient address

If receiving MTA has never seen this triplet before, then it refuses this delivery and any others that may come within a certain period of time with a temporary failure. This works very well and generally is not noticed by the administrator or user. It is VERY effective against attacks from zombies.

The basic greylisting software for Postfix is Postgrey. Postgrey uses the DBM database and I used it for several months. It is easy to set up and use and comes with a nice report feature called Postgreyreport. There are several packages that you can use for Greylisting. I chose SQLGrey because it has more options to configure and to look at. It uses either MySQL, Posgresql or SQLite. This allows for a lot of flexibility.

I decided to make my primary server a nolisting server and have the secondary mail server (running Postfix) use Greylisting. As a general rule zombie mailers and a lot of spammers will try one and never come back to try again. According to the RFC if a mail server is down the sending mail server should try again some time in the future. Many spammers won't do this because it is not efficient for them to do so. Greylisting takes advantage of the fact that spammers want to spew as much spam to as many users as possible. Retrying to send email is not efficient for them and greylisting takes advantage of this.

What Comes Next?: Spamassassin and ClamAV
At this point you have stopped 70-80 percent of UCEs coming into your network. Even though this is a great improvement it is not good enough...not even close. Our internal groupware servers consisted of a mail hub and 3 "post offices" that the hub routes users' email to. We had an excellent 3rd party commercial application that had heuristics for spam and also did virus scanning.

What you do next depends on your resources. ClamAV. Spamassassin and the alternative Spambouncer will be needed for two reasons: to get rid of embedded URIs that carry dangerous payloads, virus laden email attachments, for heuristics and Bayesian filtering.

You have a choice here you can send the mail on to an internal mail hub or if your server is beefy enough you can process mail that gets through with Spamassassin. Postfix handles this very well. The only problem is Spamassassin is a resource hog on your server. Keep this in mind. For awhile I put Spamassassin and ClamAV on a mail hub inside our network and processed the mail before it was sent on its way. Later I had a gateway server outside our network that I ran it on an it only gets stressed when there are spam blasts.

With heuristics we are able to get rid of 90-95 percent of the UCEs that enter our network. This makes UCEs manageable. I still strive for 100% UCE free.

Other anti-UCE measures
Powerful anti-UCE tools of interest that you might consider using.
SPF-Sender Policy Framework
: The main aim of SPF is to prevent forged email. This is done using DNS TXT resource records. It determines if a mail server is authorized to send email to your domain or not. For a small to medium sized business where you have lots of control over your users AND you have lots of UCEs this is an excellent option. For us it is still out of the question and would require formal training for each user. In my situation this is not possible.
DomainKeys
DomainKeys does not prevent abuse but makes it easier track. That fact alone kept me from considering it. It verifies the source and content. It is a form of authentication.
HashCash
I didn't consider this one either. Although like the one above if most MTAs used it I would as well. With the volume of mail we receive it would require considerable computational resources I didn't want to expend. May be useful to a smaller number of users.

I am sure there are others I have forgotten or didn't seriously consider.

Image and currently pdf spam has not been a great problem but one that I need to address. I do that on the mail hub.

Log files and Reports
If you do all the above and then forget about your work, chances are good that eventually spam will start to build up again. Log files and reports are the tools that will help make adjustments as spam changes. Without them I would be lost. Looking through Gigabytes of files that my maillog generates would be mind numbing. I have installed pflogsumm which generates a very large and detailed file and also wrote a script that gives me exactly what I want. Remember you will always be a step behind.

A Typical But Light Day
249158 Turkey
206287 Poland
126605 SpamHaus xbl
63588 SpamHaus pbl
40026 Germany
32108 Russian Federation
25596 GREYLISTED
22245 Korea
20887 RFC - Need fully qualified hostname
17376 France
16844 Brazil
16693 China
15808 Message accepted
11987 Taiwan
11833 Spain
7957 Argentina
7954 Israel
7564 Italy
7108 Czech Republic
7060 SpamCop bl
6599 Netherlands
6109 Hungary
5390 Romania
5309 RFC - Domain not found
5106 Japan
4744 Surriel bl
3230 Chile
3109 Bulgaria
3015 Vietnam
2403 Belgium
1776 Ukraine
1587 Slovenia
1548 United Arab Emirates
1495 RFC - Helo Invalid Name
1433 SpamHaus sbl
1432 Greece
1040 South Africa
618 Relay access denied
608 Senegal
567 Estonia
544 Ivory Coast
435 RFC - Need fully-qualified address
392 Saudi Arabia
293 Malta
106 RFC - Malformed DNS Server
70 RFC - Improper pipelining
36 Marketers
13 Nigeria
8 Benin
6 Kenya
5 SURBL
1 Greenland
1 Botswana


Henny-penny...
A couple of days after we cut spam down to almost nothing, I was walking down the hall, returning to my office to troll on Kuro5hin and met a co-worker from another department.

"Is the mail server down?" she inquired. She had a worried look on her face. "I haven't received much email today."

I smiled to myself when I realized that she was no longer spending half the morning deleting spam from her inbox.

"No", I replied, "The sky's not falling."