Sunday, November 19, 2006
One more way spammers damage the email system: SMTP Callbacks
A group I work with was refusing lots of legitimate email, and didn't know it. It turned out they are using a spam defense measure called "Sender Address Verification" or "SMTP callbacks," but their DNS wasn't set up quite right.
When your Internet service provider (ISP) tries to send a legitimate email to theirs, theirs puts that conversation on hold while it tries to verify that your sending address is valid. They do that by starting to send an email message to it. When your ISP's email server says okay, I'll accept that, they break off their transmission, and allow your incoming email to continue. That test message they almost sent is called a probe message or SMTP callback.
But if your ISP's spam defense interprets their probe message as possible spam, for any reason, and refuses or defers it, they won't get your legitimate message.
SMTP Callback is a controversial technique because it generates spurious email traffic. If it catches on, ISPs are going to have to invest in more equipment to handle that extra traffic, and email service will cost more.
We use a very common spam defense technique. We defer messages that come from Internet Protocol (IP) Addresses (IPAs) that have not been given a name. We defer messages that come from IPAs whose names are not defined. That's called "reverse DNS verification." It's cheap, fast, not abusive, and very effective. It works because places that are expected to send legitimate email are given valid names in the global Domain Name Service.
Unfortunately, our friends in Sacramento were sending probes from an IPA whose name was something like unknown.host.example.net. If you looked up that name, it was not defined. It says unknown right in its name! So of course we were deferring their probe, and their callback test was failing, and they were refusing our email. They would refuse email from anybody who uses the Postfix feature reject_unknown_client.
Before the spam crisis, none of this was necessary. Spammers are making email more expensive and less reliable, in unexpected ways.
When your Internet service provider (ISP) tries to send a legitimate email to theirs, theirs puts that conversation on hold while it tries to verify that your sending address is valid. They do that by starting to send an email message to it. When your ISP's email server says okay, I'll accept that, they break off their transmission, and allow your incoming email to continue. That test message they almost sent is called a probe message or SMTP callback.
But if your ISP's spam defense interprets their probe message as possible spam, for any reason, and refuses or defers it, they won't get your legitimate message.
SMTP Callback is a controversial technique because it generates spurious email traffic. If it catches on, ISPs are going to have to invest in more equipment to handle that extra traffic, and email service will cost more.
We use a very common spam defense technique. We defer messages that come from Internet Protocol (IP) Addresses (IPAs) that have not been given a name. We defer messages that come from IPAs whose names are not defined. That's called "reverse DNS verification." It's cheap, fast, not abusive, and very effective. It works because places that are expected to send legitimate email are given valid names in the global Domain Name Service.
Unfortunately, our friends in Sacramento were sending probes from an IPA whose name was something like unknown.host.example.net. If you looked up that name, it was not defined. It says unknown right in its name! So of course we were deferring their probe, and their callback test was failing, and they were refusing our email. They would refuse email from anybody who uses the Postfix feature reject_unknown_client.
Before the spam crisis, none of this was necessary. Spammers are making email more expensive and less reliable, in unexpected ways.
Comments:
<< Home
I came across your posts in the google groups thread Verizon.net joins the 450-forever parade. I, and several others have the same problem, except we're using gmail for domains. That thread is here http://groups.google.com/group/hosted-mail-delivery/browse_thread/thread/bf4f30eae37178db/1d6b532eaebf4fde?q=verizon&lnk=ol&
When I do a report at www.dnsstuff.com
I get the following error message about reverse dns ERROR: The IP of one or more of your mail server(s) have no reverse DNS (PTR) entries/* (if you see "Timeout" below, it may mean that your DNS servers did not respond fast enough)*/. RFC1912 2.1 says you should have a reverse DNS for all your mail servers. It is strongly urged that you have them, as many mailservers will not accept mail from mailservers with no reverse DNS entry. You can double-check using the 'Reverse DNS Lookup' tool at the DNSstuff site if you recently changed your reverse DNS entry (it contacts your servers in real time; the reverse DNS lookups in the DNS report use our local caching DNS server). The problem MX records are:
27.247.14.72.in-addr.arpa [No reverse DNS entry (rcode: 3 ancount: 0) (check it)]
And also about invalid host names:
WARNING: One or more of your mailservers is claiming to be a host other than what it really is (the SMTP greeting should be a 3-digit code, followed by a space or a dash, then the host name). If your mailserver sends out E-mail using this domain in its EHLO or HELO, your E-mail might get blocked by anti-spam software. This is also a technical violation of RFC821 4.3 (and RFC2821 4.3.1). Note that the hostname given in the SMTP greeting should have an A record pointing back to the same server. Note that this one test may use a cached DNS record.
ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP a22si31878712pye
ALT2.ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP 36si35451209agc
ALT1.ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP l21si101521430nfc
The way I see this, google has there DNS servers misconfigured and there's nothing I can do about it. But you seem to have a little better understanding, so I was wondering what your take is on this.
Thanx
When I do a report at www.dnsstuff.com
I get the following error message about reverse dns ERROR: The IP of one or more of your mail server(s) have no reverse DNS (PTR) entries/* (if you see "Timeout" below, it may mean that your DNS servers did not respond fast enough)*/. RFC1912 2.1 says you should have a reverse DNS for all your mail servers. It is strongly urged that you have them, as many mailservers will not accept mail from mailservers with no reverse DNS entry. You can double-check using the 'Reverse DNS Lookup' tool at the DNSstuff site if you recently changed your reverse DNS entry (it contacts your servers in real time; the reverse DNS lookups in the DNS report use our local caching DNS server). The problem MX records are:
27.247.14.72.in-addr.arpa [No reverse DNS entry (rcode: 3 ancount: 0) (check it)]
And also about invalid host names:
WARNING: One or more of your mailservers is claiming to be a host other than what it really is (the SMTP greeting should be a 3-digit code, followed by a space or a dash, then the host name). If your mailserver sends out E-mail using this domain in its EHLO or HELO, your E-mail might get blocked by anti-spam software. This is also a technical violation of RFC821 4.3 (and RFC2821 4.3.1). Note that the hostname given in the SMTP greeting should have an A record pointing back to the same server. Note that this one test may use a cached DNS record.
ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP a22si31878712pye
ALT2.ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP 36si35451209agc
ALT1.ASPMX.L.GOOGLE.COM claims to be non-existent host mx.google.com:
220 mx.google.com ESMTP l21si101521430nfc
The way I see this, google has there DNS servers misconfigured and there's nothing I can do about it. But you seem to have a little better understanding, so I was wondering what your take is on this.
Thanx
It seems to me teraforce has diagnosed this issue correctly. Google seems to be sending SMTP callbacks from an IP Address with a PTR Resource Record that contains a name which has no corresponding Address Resource Record in DNS.
Now Google is trying to be a generally higher quality operation than Yahoo Mail or Hotmail. They're making enough money on other operations that they can afford to run Gmail at a loss, the way Microsoft runs Hotmail. They're also trying to maintain a reputation as the "best and brightest" in their field. That means they employ competent technical staff in adequate numbers. Unlike the lowball consumer broadband sellers. They don't have the handicap of trying to run the operation on Microsoft's software. If this is all true, they'll figure out their little DNS error (missing A records) pretty fast and fix it.
Meanwhile, Verizon will keep sending callbacks from IP Addresses with no corresponding Pointer Records in the in-addr.arpa domain.
LOOK HERE! We have two big-name providers who can't send to each other because both have made DNS mistakes in their SMTP callback implementations! This is a free clue to anybody considering SMTP callback: BAD IDEA. Consider greylisting instead. Consider more aggressive block listing. At least subscribe your server to the zen.spamhaus.org public DNSBL. Then see if that didn't solve the problem you wanted callbacks to solve.
Post a Comment
Now Google is trying to be a generally higher quality operation than Yahoo Mail or Hotmail. They're making enough money on other operations that they can afford to run Gmail at a loss, the way Microsoft runs Hotmail. They're also trying to maintain a reputation as the "best and brightest" in their field. That means they employ competent technical staff in adequate numbers. Unlike the lowball consumer broadband sellers. They don't have the handicap of trying to run the operation on Microsoft's software. If this is all true, they'll figure out their little DNS error (missing A records) pretty fast and fix it.
Meanwhile, Verizon will keep sending callbacks from IP Addresses with no corresponding Pointer Records in the in-addr.arpa domain.
LOOK HERE! We have two big-name providers who can't send to each other because both have made DNS mistakes in their SMTP callback implementations! This is a free clue to anybody considering SMTP callback: BAD IDEA. Consider greylisting instead. Consider more aggressive block listing. At least subscribe your server to the zen.spamhaus.org public DNSBL. Then see if that didn't solve the problem you wanted callbacks to solve.
<< Home
