Saturday, March 31, 2007
blocking vs filtering
A poster on Techrepublic boasted that his workstation security suite (for MS-Windows) "blocks" spam.
AVG Internet Security does a lot of good things. I recommend it to my customers who still use Windoze. We prefer it to Symantec or McAfee. But it doesn't block spam. Nor do its competitors.
If you're using the typical consumer setup where you download your email via POP3 from your ISP's mailbox server, your workstation doesn't see the spam until it's already been delivered.
AVG Internet Security and its competitors filter spam. That is, they analyze and sort it. One good optimization you can do with POP3 is pull all the message headers, analyze them, and delete the obvious spam from the mailbox before downloading the whole messages. I'd be surprised if they don't do that, at least as an option. But that's not available if you want to download all the spam into a local spam "folder" to look for false positives.
Only your email service provider can block spam. That's because blocking happens before the SMTP server (receiving system) has accepted the message. The SMTP server has to consider the source while the wanna-be SMTP client (sender) is waiting to connect, or analyze the message on the fly while the client is waiting for a response.There are two significant differences between blocking and filtering.
- Blocked spam from spamware just disappears. (Spamware is the specialized software criminal spammers use for sending. Most of it is installed on PCs the criminals have broken into, using trojans or rootkits or the like. A lot of it just connects and blasts away, without paying any attention to the responses from the SMTP server.) But blocked spam from a "legitimate" sender piles up in the sender's outbound queue or gets returned. That gives the sender feedback that he's sending unwanted mail and/or the address is bad. In the case of a legitimate sender exploited by a criminal spammer, it gives him feedback that his security is compromised. Filtered spam appears to the sender to have been delivered. The "legitimate" spammers (the minority who try to comply with CAN-SPAM et. al.) are deprived of the chance to clean bad addresses off their lists. Those criminal spammers who pay any attention to SMTP responses at all are told your address is deliverable, which makes it more valuable to sell to other spamemrs. In the end, filtering is a way of (partially) automating the process of "just hitting delete." It adds to the overall problem.
- Blocked spam doesn't cost the non-recipient anything to store or download. And blocking before body content analysis is a whole lot cheaper than filtering.
Friday, March 30, 2007
Danger of reporting spam
This morning I got a spam from a hobby server on DSLextreme.com. I'm a DSL Extreme customer. Their customer service is great, and they don't tolerate spam. In fact, the last time I reported one of these to them, they misinterpreted the report and blocked me. Shoot first, ask questions later.
So today I copied the spam sample to my server at Explosive.net and sent the report from there. Explosive is really great, too. But their Internet Protocol Address (IPA) space is on Speakeasy.net, and Speakeasy's just as mean to spammers as DSL Extreme is. So I'm still taking a chance.
That's what it's come to. The well-run retail ISPs are few and far between. You don't want to be anywhere else. But the well-run ISPs are on such a hair-trigger you have to think twice about sending legitimate email that could be mistaken for spam. Argh.
Meanwhile, I'm shopping for a low-cost, well run virtual private server (VPS) in squeaky-clean IPA space. My users want to host video clips and I can't do it from the colocation at Explosive. Speakeasy and DSLExtreme don't offer VPS. I considered GPLhost but they're on PCCW, which doesn't seem to handle abuse complaints competently. Drop me a line if you've got any ideas. Charlie Lima Sierra at Truffula dot Sierra Juliet dot Charlie Alpha dot Uniform Sierra. Geez, think they'll harvest that?
So today I copied the spam sample to my server at Explosive.net and sent the report from there. Explosive is really great, too. But their Internet Protocol Address (IPA) space is on Speakeasy.net, and Speakeasy's just as mean to spammers as DSL Extreme is. So I'm still taking a chance.
That's what it's come to. The well-run retail ISPs are few and far between. You don't want to be anywhere else. But the well-run ISPs are on such a hair-trigger you have to think twice about sending legitimate email that could be mistaken for spam. Argh.
Meanwhile, I'm shopping for a low-cost, well run virtual private server (VPS) in squeaky-clean IPA space. My users want to host video clips and I can't do it from the colocation at Explosive. Speakeasy and DSLExtreme don't offer VPS. I considered GPLhost but they're on PCCW, which doesn't seem to handle abuse complaints competently. Drop me a line if you've got any ideas. Charlie Lima Sierra at Truffula dot Sierra Juliet dot Charlie Alpha dot Uniform Sierra. Geez, think they'll harvest that?
Wednesday, March 28, 2007
Where to report spam
I've seen estimates that less than one in a million spams results in a well-directed complaint.
Almost every week I see bad advice about where to report incoming spam.
Never reply to a spam message. The reply address is probably bogus, and if it's real, you just made your address more valuable to other spammers. You can't mailbomb them. You can't exhaust their web servers with repeated requests, either.
Don't report spam if you're not computer literate enough to save a spam into a plain text file and look at the headers. That means the lines in the message headers that begin with the word "Received:." If the files you save don't have those, don't bother. If you do not know the difference between a plain text file and an MS-Word document with the font set to Courier, don't bother. But if you can include the message in-line, not as an attachment, without destroying the headers or adding word processor crap, go for it. Report it to:
Your email service provider. That's nice. Sometimes it helps "educate" or "train" their filters. AOL and Yahoo! Mail do that. It does approximately nothing to the spammer. Your ISP is probably not going to contact his ISPs.
Spamcop. That's nice too. It helps ISPs who subscribe to Spamcop's block list block more spam from the same source. Don't bother if the spam is more than an hour old. Unfortunately Spamcop also offers a "personal" software product that's supposed to analyze the spam and help you generate a report. But it's not very accurate, and a lot of ISPs, maybe most, ignore those reports.
The FTC. You can forward spam with complete headers to spam@uce.gov. They keep statistics.
The SEC. You can forward stock spam to enforcement@sec.gov. They bust some criminals sometimes. If it's "image" spam, put the stock symbol that's being promoted in your subject line.
news.admin.net-abuse.sightings. That's a Usenet newsgroup for posting spam samples. People use it to research spam patterns. If you can't post the contents of a plain text file, in-line, to a newsgroup, don't bother.
The owner of the exploited equipment. Almost all spam is sent through computers the spammers don't own. Spammers break into servers through leaky Web applications. Or they steal or guess weak passwords. They break into PCs in people's homes, on DSL or Cable, through "virus" infected email and malware infected Web pages.
Look at the Received: header line where your service provider receives the message from someplace that's not your service provider. (If you can't read, don't bother.) Maybe it's a cable company you've heard of. Look up that company's abuse reporting address. There's a service for doing that, at abuse.net. You can query abuse.net with your whois program (e.g., whois -h whois.abuse.net hotmail.com), or use its web site. The DSL or cable company will (sometimes) contact the owner of the compromised computer.
The giveaway for those DSL or cable senders is a so-called "generic address." I'll pick two examples from today's incoming spam. The hostname wsip-70-183-84-39.dl.dl.cox.net is generic. It's got numbers in it that are the same as its IP address. The hostname mercury1.networknoc.com is not generic. If it's not a generic name, it's not one of those home machines. It's either web hosting or a small business. You can figure out who the ISP is with your traceroute or tcptraceroute program. You'll never figure out who the owners of the individual cable/DSL zombies are. But their ISPs know. You can try calling the owners of the exploited web servers themselves. But if you can't talk about the spam they're sending authoritatively, they'll think you're just harassing them or trying to sell something. It's easier to just send a spam sample to the abuse address at their ISP.
Sometimes spammers break into other people's computers to host their name servers or Web servers or both. Never go to the URL in a spam message. If you use MS-Outlook [Express] or Thunderbird don't even open your email with image display enabled. But you can trace the name in the URL. And you can trace the name servers. They're named in the domain's Whois entry or you can look them up with your host or nslookup or dig commands. If the servers are on cable/DSL or at hosting places in western Europe, Australia, or North America, report them. Elsewhere, it's probably not worth the trouble. If it's in China, South Korea, Russia, or Bulgaria, sad to say, don't bother. As far as I know, all ISPs in those countries are spammer-friendly. You can look up the ISP at Spamhaus.org if you're not sure. There is no point in reporting spam to a spammer-friendly ISP.
"Free" email providers. A certain type of spammer prefers to use throwaway accounts at Yahoo Mail, Hotmail, Excite.com, etc. Those are the "advance fee fraud" or "Nigeria 419" scammers. If they're fresh, report these to the abuse address at the email company. If you received it more than 24 hours ago, don't bother. Notice that the "Reply to" address is hardly ever the same as the "From" in these things. Sometimes the Reply to address is repeated in the message body. Those are the ones that are worth reporting. The From address had already been discarded by the time you saw the spam.
Notice that abuse@hotmail.com does not work and never has. Hotmail (Microsoft) thinks the rules of the Internet don't apply to them, and their special abuse address is report_spam@hotmail.com. Also, the abuse address for Yahoo Mail is always abuse@yahoo.com, even for the country domains like yahoo.co.uk.
That's all. If you're savvy enough to find other assets in the spammer's network, you already knew all this stuff and didn't have to read this far. Experts go after the credit card processors. Uber-experts sometimes take legal action. But this is not work for amateurs. Remember spammers are criminals. Spam is international organized crime. You don't want to provoke these people if you don't know what you're doing.
Almost every week I see bad advice about where to report incoming spam.
Never reply to a spam message. The reply address is probably bogus, and if it's real, you just made your address more valuable to other spammers. You can't mailbomb them. You can't exhaust their web servers with repeated requests, either.
Don't report spam if you're not computer literate enough to save a spam into a plain text file and look at the headers. That means the lines in the message headers that begin with the word "Received:." If the files you save don't have those, don't bother. If you do not know the difference between a plain text file and an MS-Word document with the font set to Courier, don't bother. But if you can include the message in-line, not as an attachment, without destroying the headers or adding word processor crap, go for it. Report it to:
Your email service provider. That's nice. Sometimes it helps "educate" or "train" their filters. AOL and Yahoo! Mail do that. It does approximately nothing to the spammer. Your ISP is probably not going to contact his ISPs.
Spamcop. That's nice too. It helps ISPs who subscribe to Spamcop's block list block more spam from the same source. Don't bother if the spam is more than an hour old. Unfortunately Spamcop also offers a "personal" software product that's supposed to analyze the spam and help you generate a report. But it's not very accurate, and a lot of ISPs, maybe most, ignore those reports.
The FTC. You can forward spam with complete headers to spam@uce.gov. They keep statistics.
The SEC. You can forward stock spam to enforcement@sec.gov. They bust some criminals sometimes. If it's "image" spam, put the stock symbol that's being promoted in your subject line.
news.admin.net-abuse.sightings. That's a Usenet newsgroup for posting spam samples. People use it to research spam patterns. If you can't post the contents of a plain text file, in-line, to a newsgroup, don't bother.
The owner of the exploited equipment. Almost all spam is sent through computers the spammers don't own. Spammers break into servers through leaky Web applications. Or they steal or guess weak passwords. They break into PCs in people's homes, on DSL or Cable, through "virus" infected email and malware infected Web pages.
Look at the Received: header line where your service provider receives the message from someplace that's not your service provider. (If you can't read, don't bother.) Maybe it's a cable company you've heard of. Look up that company's abuse reporting address. There's a service for doing that, at abuse.net. You can query abuse.net with your whois program (e.g., whois -h whois.abuse.net hotmail.com), or use its web site. The DSL or cable company will (sometimes) contact the owner of the compromised computer.
The giveaway for those DSL or cable senders is a so-called "generic address." I'll pick two examples from today's incoming spam. The hostname wsip-70-183-84-39.dl.dl.cox.net is generic. It's got numbers in it that are the same as its IP address. The hostname mercury1.networknoc.com is not generic. If it's not a generic name, it's not one of those home machines. It's either web hosting or a small business. You can figure out who the ISP is with your traceroute or tcptraceroute program. You'll never figure out who the owners of the individual cable/DSL zombies are. But their ISPs know. You can try calling the owners of the exploited web servers themselves. But if you can't talk about the spam they're sending authoritatively, they'll think you're just harassing them or trying to sell something. It's easier to just send a spam sample to the abuse address at their ISP.
Sometimes spammers break into other people's computers to host their name servers or Web servers or both. Never go to the URL in a spam message. If you use MS-Outlook [Express] or Thunderbird don't even open your email with image display enabled. But you can trace the name in the URL. And you can trace the name servers. They're named in the domain's Whois entry or you can look them up with your host or nslookup or dig commands. If the servers are on cable/DSL or at hosting places in western Europe, Australia, or North America, report them. Elsewhere, it's probably not worth the trouble. If it's in China, South Korea, Russia, or Bulgaria, sad to say, don't bother. As far as I know, all ISPs in those countries are spammer-friendly. You can look up the ISP at Spamhaus.org if you're not sure. There is no point in reporting spam to a spammer-friendly ISP.
"Free" email providers. A certain type of spammer prefers to use throwaway accounts at Yahoo Mail, Hotmail, Excite.com, etc. Those are the "advance fee fraud" or "Nigeria 419" scammers. If they're fresh, report these to the abuse address at the email company. If you received it more than 24 hours ago, don't bother. Notice that the "Reply to" address is hardly ever the same as the "From" in these things. Sometimes the Reply to address is repeated in the message body. Those are the ones that are worth reporting. The From address had already been discarded by the time you saw the spam.
Notice that abuse@hotmail.com does not work and never has. Hotmail (Microsoft) thinks the rules of the Internet don't apply to them, and their special abuse address is report_spam@hotmail.com. Also, the abuse address for Yahoo Mail is always abuse@yahoo.com, even for the country domains like yahoo.co.uk.
That's all. If you're savvy enough to find other assets in the spammer's network, you already knew all this stuff and didn't have to read this far. Experts go after the credit card processors. Uber-experts sometimes take legal action. But this is not work for amateurs. Remember spammers are criminals. Spam is international organized crime. You don't want to provoke these people if you don't know what you're doing.
Thursday, March 22, 2007
Advice to an unwilling spammer host
I reported a spam to an email admin at an Indian reservation. He replied with a nice thank you note. His technician has been trying to stop the spam coming out of their MSFT system for a few days, with no success. I offer general advice:
Hi Justin, thanks.
I hope you won't mind some unsolicited general advice about the problem.
You're using a Microsoft system for your exposed email server. That's going to be an ongoing headache. Believe it or not, and despite everything you have read in the trade press and heard from Microsoft's sales force, their operating system is not designed to be exposed (on a "routable" address) directly to the Internet.
The customers Microsoft listens to, that they design their system for, are the Fortune 500 corporations. Consumers, small business, and distributors like Dell and Gateway, are taken for granted, because they have been taught they "have no choice." Their (our) needs are not considered in Microsoft's design decisions. Fortune 500 corporations do not expose Microsoft systems to the Internet. They hide them behind layers of protection: proxy servers, firewalls, "policy servers," and other equipment.
You would be wise to start thinking about placing some non-Microsoft system between your exposed address and your internal Microsoft email system, to relay email in and out, and be a firewall. PCs are very cheap now. You can put a PC running FreeBSD or GNU+Linux between the Internet and your private network for less than you spend on "anti-virus" junk for a few MSFT machines. The PC you retired because it wasn't fast enough to run Windows XP very well will usually do. You can stop the "virus" email and 90% of the incoming spam with it, as well as the criminals who compromised your current system.
It takes a bigger PC to run today's comprehensive spam and virus filters, but even a serious compute engine only costs a few hundred bucks these days, and all the software you need to do it is truly free and trustworthy.
A painless and risk-free first step down this road is to try a couple of "Live Linux" CDs. These let you temporarily run a fully functional computer system on your current PC, directly off the CD, without disturbing your current software installation and without installing anything. I recommend Knoppix.net, but Ubuntulinux.org is more popular. If you have an older, smaller PC, you might try damnsmalllinux.org instead.
--
Best wishes,
Me in San José
http://greens.org/nnn/
Hi Justin, thanks.
I hope you won't mind some unsolicited general advice about the problem.
You're using a Microsoft system for your exposed email server. That's going to be an ongoing headache. Believe it or not, and despite everything you have read in the trade press and heard from Microsoft's sales force, their operating system is not designed to be exposed (on a "routable" address) directly to the Internet.
The customers Microsoft listens to, that they design their system for, are the Fortune 500 corporations. Consumers, small business, and distributors like Dell and Gateway, are taken for granted, because they have been taught they "have no choice." Their (our) needs are not considered in Microsoft's design decisions. Fortune 500 corporations do not expose Microsoft systems to the Internet. They hide them behind layers of protection: proxy servers, firewalls, "policy servers," and other equipment.
You would be wise to start thinking about placing some non-Microsoft system between your exposed address and your internal Microsoft email system, to relay email in and out, and be a firewall. PCs are very cheap now. You can put a PC running FreeBSD or GNU+Linux between the Internet and your private network for less than you spend on "anti-virus" junk for a few MSFT machines. The PC you retired because it wasn't fast enough to run Windows XP very well will usually do. You can stop the "virus" email and 90% of the incoming spam with it, as well as the criminals who compromised your current system.
It takes a bigger PC to run today's comprehensive spam and virus filters, but even a serious compute engine only costs a few hundred bucks these days, and all the software you need to do it is truly free and trustworthy.
A painless and risk-free first step down this road is to try a couple of "Live Linux" CDs. These let you temporarily run a fully functional computer system on your current PC, directly off the CD, without disturbing your current software installation and without installing anything. I recommend Knoppix.net, but Ubuntulinux.org is more popular. If you have an older, smaller PC, you might try damnsmalllinux.org instead.
--
Best wishes,
Me in San José
http://greens.org/nnn/
Monday, March 19, 2007
Spamassassin and Amavis
After resisting for years, I've taken the second step down the slippery slope of content filtering. My first lines of spam defense will continue to be source blocking and SMTP mistake-catching. But that only gets you so far. The criminals who break into legitimate web hosts get through. The only way to get them is analyze the messages.
The first step was Postfix' header_checks and body_checks. They stop some of the most obvious stuff. But Postfix warns you not to get carried away, and you can't combine different checks. "If it says it's from Paypal but it wasn't sent from their IP space" is too complex.
The second step is a big one. We set up a special local server, Amavis-new, that Postfix can consult as it decides whether to accept a message. This evaluation has to happen fast, while the sender (client) is waiting for the receiver's (server's) decision. Once you accept the message into your delivery queue, it's too late to refuse it. You can't return it, because once it's yours you don't really know where it came from. The client is long-gone, and the "From:" address in spam is always a lie.
Amavis-new's biggest module is Spamassassin, a collection of thousands of little "tests" that can be intricately selected, combined, and scored. Amavis-new considers Spamassassin's opinion of the message and advises Postfix to refuse the spammiest. It leaves marks on the messages it accepts, so that the final recipients can sort them as they're delivered. A very cool contraption. Each part is carefully and independently maintained.
It's software for professionals; the "documentation" is great reference material but scant tutorial. And there are lots of ways to put the pieces together. The maintainers of each piece have rather little to say about all those ways. They're responsible for their respective pieces, but you're responsible for your contraption. I have the O'Reilly Spamassassin book and the No Starch Postfix book (they're both pretty good) and I still had to ask for help. Someone on the Debian-ISPs list sent me exactly the clue I needed, immediately. Somewhere in Amavis-new's documentation they tell you that amavisd will only mark up messages destined for "local" recipients. That's what the @local_domains_maps variable is about. It's in the sample config file.
The first step was Postfix' header_checks and body_checks. They stop some of the most obvious stuff. But Postfix warns you not to get carried away, and you can't combine different checks. "If it says it's from Paypal but it wasn't sent from their IP space" is too complex.
The second step is a big one. We set up a special local server, Amavis-new, that Postfix can consult as it decides whether to accept a message. This evaluation has to happen fast, while the sender (client) is waiting for the receiver's (server's) decision. Once you accept the message into your delivery queue, it's too late to refuse it. You can't return it, because once it's yours you don't really know where it came from. The client is long-gone, and the "From:" address in spam is always a lie.
Amavis-new's biggest module is Spamassassin, a collection of thousands of little "tests" that can be intricately selected, combined, and scored. Amavis-new considers Spamassassin's opinion of the message and advises Postfix to refuse the spammiest. It leaves marks on the messages it accepts, so that the final recipients can sort them as they're delivered. A very cool contraption. Each part is carefully and independently maintained.
It's software for professionals; the "documentation" is great reference material but scant tutorial. And there are lots of ways to put the pieces together. The maintainers of each piece have rather little to say about all those ways. They're responsible for their respective pieces, but you're responsible for your contraption. I have the O'Reilly Spamassassin book and the No Starch Postfix book (they're both pretty good) and I still had to ask for help. Someone on the Debian-ISPs list sent me exactly the clue I needed, immediately. Somewhere in Amavis-new's documentation they tell you that amavisd will only mark up messages destined for "local" recipients. That's what the @local_domains_maps variable is about. It's in the sample config file.
@local_domains_maps list of lookup tables are used in deciding whether a
recipient is local or not, or in other words, if the message is outgoing
or not. This affects inserting spam-related headers for local recipients,
limiting recipient virus notifications (if enabled) to local recipients,
in deciding if address extension may be appended, and in SQL lookups
for non-fqdn addresses. Set it up correctly if you need features
that rely on this setting (or just leave empty otherwise).