Experimental and Theoretical ways to get rid of Spam
This page is for the development of ideas and projects to move SpamAssassin and related spam fighting projects forward. The idea here is to write about ideas and projects the represent the cutting edge ideas that are in development related to spam fighting.
New Black/White/Yellow List Technologies
This article describes experimental technologies used at [http://www.junkemailfilter.com Junk Email Filter]. I (Marc Perkel) am writing this in the hopes that other people will pick up on these ideas and improve them. Although it is working very well for us so far, it can be improved and expanded. It is my hope that this will inspire others to work on this and make it far better than what we have developed. Information about our Host Karma lists can be found [http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists here]
We are all familiar with black lists. Black lists are DNS lists that where IP addresses of spammer are looked up. You send a request to the list with the IP and if it is in the blacklist then you bounce the email. Sounds simple enough except for the false positives.
White lists help reduce false positives. If an IP address is white listed then you can just pass it and not look at black lists.
Yellow lists are lists of hosts that send a mix of spam and non-spam. Email services like Yahoo, Gmail, and Hotmail are examples of mixed source servers. These servers should never be either black or white listed. These are servers where the IP address yields no useful information as to whether the message is spam or ham.
Multi-Color List Processing Logic
When looking at lists from a black/white/yellow perspective there is an order to the list processing. First the yellow lists are tested. If the message is yellow listed then checking black and white lists isn't necessary. The the white lists are checked. If the message is white listed it can be passed without having to run it through SA or SA can short circuit the tests and declare the message ham. Then the black lists are checked and if it is in a trusted blacklist or several less trusted blacklists the message can be rejected.
Reducing Lookups by using multiple return codes to indicate the result
Many DNS lists return and code to indicate yes and nothing to indicate no. This was fine for only black list lookups but if you are looking up multiple states you would have to make a separate DNS call for each one.
yellowlist.junkemailfilter.com whitelist.junkemailfilter.com blacklist.junkemailfilter.com
But why do three lookups when you can do one? That's the way the HostKarma list works. It returns a different value to indicate black/white/yellow.
127.0.0.1 white 127.0.0.2 black 127.0.0.3 yellow
Forward Confirmed rDNS (FCrDNS)
Forward Confirmed reverse DNS is an important concept for the ideas I'm about to introduce below. One of the ways to separate spam from ham is to find things that spammers can't spoof. One of those things is forward confirmed rDNS.
Reverse DNS is straight forward. An IP has a PTR record so that when you look up the IP it returns a name associated with the IP. Unfortunately a spammer can put any name they want in a PTR record. But when that name is looked up, it points somewhere else.
Forward Confirmed rDNS means that when the name returned by a PTR lookup is checked it will point back to the original IP address that was looked up.
1.2.3.4 -> mail.mydomain.com mail.mydomain.com -> 1.2.3.4
Spammers can't spoof this because even though PTR can be faked the spammer can't create an A record to point back to the original IP because they don't have control over the faked domain.
FCrDNS is very reliable for detecting white and yellow domains. It can also be used for black domains but generally spammers aren't that stable. So this is mostly for actively detecting ham and avoiding false positives than detecting spam.
Host Name based Lists based on FCrDNS host names
In addition to IP based black/white/yellow lists the [http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists HostKarma lists] also contain host names that are also listed. A host name lookup would look like this:
dig mydomain.com.hostkarma.junkemailfilter.com
The FCrDNS of the sending host name (which can't be spoofed) is looked up in the same database that is used for IP based lookups. The Junk Email Filter HostKarma DNS list supports name based lookups as well as IP based lookups. If the lookup succeeds the IP base tests need not be done. The name based test are actually more powerful than IP based tests for white and yellow listed servers where IP based lookups are better for black listed hosts.
For example, if the FCrDNS of the sending host resolves to yahoo.com then no other DNS tests need to be done. Yahoo is neither a certified spam or ham domain and once the name returns yellow no other lookups need be done. The message then can go to content testing to figure out if it is spam or ham. If you get a message from your bank, like Wells Fargo Bank, and you see that the sending host is mail.wellsfargo.com, then it will look up as white listed and the message can be declared ham without any other testing.