15,000+ spam attempts an hour...

[RJM62]

They were many different addresses, almost all of which were addressed to muiltiple recipients. All of the senders and recipients were in [many domains].tw or hinet.net.

I got in touch with the senior sysadmin at the hosting company, and he believes that the attacks actually started before I assumed possession of the IP addresses on December 31. Apparently the previous user of the IPs on the VPS had an open relay, and every spammer in Taiwan was using it. (This is the second time in a month that I inherited the problems of the previous user of an IP; the first was on another server when I was assigned an IP that had been used by a spammer who had been blacklisted by EVERYONE.)

As far as I can determine, when I set up the VPS, I changed the hostname from the hosting company's temporary one to one of my own domains, which borked the cPanel license, which wouldn't reactivate because of a firewall issue, which was mishandled by VPS support. (In fairness, mirrored systems are usually handled by dedicated support, so he didn't know that the firewalling was done differently. He was just trying to expeditiously correct the problem.)

Once I got cPanel back, I hardened the Exim configuration and did a few other security-related tweaks, but the result was that Exim and spamd were now consuming vast resources trapping the spam, which should have gotten that far to begin with. Classic "error chain" sort of thing.

In any case, I contacted the sysadmin this morning, and he looked at the setup and made a few tweaks; and everything seems fine now except for lfd, which keeps crashing and beeping me. That will probably be corrected by a system reboot later on.

Just to make things interesting, however, once the system was secured, there were also a series of successive failed SSH logins from an IP address in India. I don't know if there is any relation...

-Rich

 

[jesse]

They were many different addresses, almost all of which were addressed to muiltiple recipients. All of the senders and recipients were in [many domains].tw or hinet.net.

I got in touch with the senior sysadmin at the hosting company, and he believes that the attacks actually started before I assumed possession of the IP addresses on December 31. Apparently the previous user of the IPs on the VPS had an open relay, and every spammer in Taiwan was using it. (This is the second time in a month that I inherited the problems of the previous user of an IP; the first was on another server when I was assigned an IP that had been used by a spammer who had been blacklisted by EVERYONE.)

As far as I can determine, when I set up the VPS, I changed the hostname from the hosting company's temporary one to one of my own domains, which borked the cPanel license, which wouldn't reactivate because of a firewall issue, which was mishandled by VPS support. (In fairness, mirrored systems are usually handled by dedicated support, so he didn't know that the firewalling was done differently. He was just trying to expeditiously correct the problem.)

Once I got cPanel back, I hardened the Exim configuration and did a few other security-related tweaks, but the result was that Exim and spamd were now consuming vast resources trapping the spam, which should have gotten that far to begin with. Classic "error chain" sort of thing.

In any case, I contacted the sysadmin this morning, and he looked at the setup and made a few tweaks; and everything seems fine now except for lfd, which keeps crashing and beeping me. That will probably be corrected by a system reboot later on.

Just to make things interesting, however, once the system was secured, there were also a series of successive failed SSH logins from an IP address in India. I don't know if there is any relation...

-Rich

Rich,

Check and see if you're accepting e-mail for domains or users that you do not host. *If* you are you need to get this fixed which would have prevented this incident.

An example..
I do not host e-mail for billg@microsoft.com

telnet 10.0.1.55 25
Trying 10.0.1.55...
Connected to [REMOVED].
Escape character is '^]'.
220 [REMOVED] ESMTP Postfix
helo jesseangell.com
250 [REMOVED]
mail from: <>
250 2.1.0 Ok
rcpt to: billg@microsoft.com
554 5.7.1 <billg@microsoft.com>: Relay access denied
quit
221 2.0.0 Bye
Connection closed by foreign host.

Notice how it killed my SMTP session with an error and didn't accept a message?

Now for an e-mail I do host:

telnet 10.0.1.55 25
Trying 10.0.1.55...
Connected to [REMOVED]
Escape character is '^]'.
220 [REMOVED] ESMTP Postfix
helo jesseangell.com
250 [REMOVED]
mail from: <>
250 2.1.0 Ok
rcpt to: sjobs@apple.com
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
Hey Steve! Ignore this test.
.
250 2.0.0 Ok: queued as 3D3E7290356
quit
221 2.0.0 Bye
Connection closed by foreign host.

Configure your mail server to terminate a SMTP session (WITH A GOOD ERROR CODE) if you don't host e-mail for that address/domain. If you do this that SPAM would have never even hit your queue.

 

[jmaynard]

Just to make things interesting, however, once the system was secured, there were also a series of successive failed SSH logins from an IP address in India. I don't know if there is any relation...

Maybe, maybe not. There's a botnet out there that's trying ssh dictionary attacks on lots of systems. I installed DenyHosts to cut it off at the knees on my box.

 

[RJM62]

Rich,

Check and see if you're accepting e-mail for domains or users that you do not host. *If* you are you need to get this fixed which would have prevented this incident.

An example..
I do not host e-mail for billg@microsoft.com

Notice how it killed my SMTP session with an error and didn't accept a message?

Now for an e-mail I do host:


Configure your mail server to terminate a SMTP session (WITH A GOOD ERROR CODE) if you don't host e-mail for that address/domain. If you do this that SPAM would have never even hit your queue.

Thank you, Jesse. I think the sysadmin is still poking around the system just to make sure everything's okay (he's a little obsessive), so I want to wait until he's finished.

I ran a few relay tests today (njabl.org's, for example), and everything came up okay. But I respect your knowledge and value your advice; so I will check, and if necessary, make the changes.

By the way... I'm working on something else you may be interested in looking at. It's based on the SpamFreeText site I built about a year ago, but which much improved code (thanks largely to you) and intended for contact form filtering. I'll send you a message when it's baked. (It's only half-baked now: The spam-filtering works, but I'm still building the user administration area and a few other odds and ends, in between chasing Taiwanese spammers. )

-Rich

 

[jesse]

Thank you, Jesse. I think the sysadmin is still poking around the system just to make sure everything's okay (he's a little obsessive), so I want to wait until he's finished.

Some of these people are really sneaky. The most interesting one I saw recently was this:

Attacker scans for proxy servers (like Squid).
Once attacker finds proxy server he uses the proxy server to send mail *THROUGH* the mail server running on the proxy server. Since the spammer is going through the proxy server it looks like 127.0.0.1 to the mail server so it delivers the spam.

The above resulted in a backed up queue with well over a million messages.

I ran a few relay tests today (njabl.org's, for example), and everything came up okay.

Good deal. The big question is: How did 15,000 messages bound for other domains you don't host get into *YOUR* mail queue. It is important that you figure out the answer to that--because something has to be *not right* for that to occur.

By the way... I'm working on something else you may be interested in looking at. It's based on the SpamFreeText site I built about a year ago, but which much improved code (thanks largely to you) and intended for contact form filtering. I'll send you a message when it's baked. (It's only half-baked now: The spam-filtering works, but I'm still building the user administration area and a few other odds and ends, in between chasing Taiwanese spammers. )

-Rich

Cool. Let me know.

 

[flyingcheesehead]

SBC outright won't respond to our requests, not even an acknowledgement, despite the fact that one of our lead subcontractors (who is an ex-Microsoft forensics employee) is on their system and has similarly made requests.

Maybe your requests are just being accepted and sent to /dev/null.

 

[wsuffa]

Maybe your requests are just being accepted and sent to /dev/null.

For those two operations, probably so.

Then again, we've also tried the phone thing to no avail, too....

 

[Neal Howard]

I've got vast swaths of the APNIC netblocks completely filtered out at my border router. Traffic coming in from any of those netblocks, regardless of port number, never even makes it to my mailserver.

 

[jason]

Ya know, Jesse, I understand the implications.

Frankly, I'm tired of the 30,000 (and counting) backscatter messages that took down a server. The only reasonable way to deal with something like that is to /dev/null the error messages.

Likewise, I have zero, zip, nada compassion for spam servers in China and elsewhere that send enormous amounts of spam. If one handles little or no legit traffic from those servers, I have no issue with "blackholing" or /dev/nulling the incoming *from those servers*. It may not be a legit technique for *all* addresses (especially the dynblocks), but it is for something that has 99.9999999% of the messages as spam.

Don't even start about troubleshooting email. The likes of AT&T (Prodigy) and SBC have irritated me by their unwillingness to remove my business email server from their private block lists. The business email server in question is under our full control sends about 200 legit emails a day, and is outright incapable (bandwidth limitation) of being a spam source. Yet AT&T had the IP blocked privately, which required days of aggravation to remove (and nearly cost me a business consulting client) and SBC has been outright unresponsive to 6 months of requests... SBC outright won't respond to our requests, not even an acknowledgement, despite the fact that one of our lead subcontractors (who is an ex-Microsoft forensics employee) is on their system and has similarly made requests.

For all intents and purposes, there is no "troubleshooting" - just outright arrogance. YOU may be responsive - but others aren't.

SpamAssassin may not be the right solution for a Yahoo, Gmail, or even a university. It is one of many solutions for others.

Ugh. I'm glad that I've not been called in to troubleshoot a problem sending to your server. Nothing like finding out that you've wasted half a day because somebody had a "better way" (or thought that there was no other way).

There are other ways to deal with backscatter. Set up an SPF record on your domain. Tell servers reading SPF (including your server) to hard fail from unknown sources. It'll deny it as part of the smtp transaction so you don't even have to read the message in.

Spam filtering isn't 100% accurate. Until such a time that it is no server admins should black hole mail without notification to the sender or the recipient.

 

[RJM62]

Sometimes my own stupidity shocks me...

I ran every Web-based tool I know of to find open relays, and the server passed them all.

I also logged in remotely via SSH, disabled csf, and ran the telnet-based tests from the server. Again, passed them all.

But when I telneted in from here, I was able to successfully send mail between two addresses, neither of which are on domains hosted on that server.

As I sat here pulling my hair out, my lady friend's granddaughter called me. She just got her first cell phone today, and she's been calling and texting me most of the afternoon and into the night. I figure the next time she's here I'll have to take her to the hospital to have it surgically removed from her ear.

She asked what I was doing, and I explained in terms that a child could understand that the server wasn't supposed to be delivering mail for domains it doesn't host. "Well, silly," she said, "It's delivering the mail because it already knows who you are!"

...

DOH!

My mail client checks my mail every ten minutes, and I have antirelayd running on the server; so in effect, as long as I'm at the same IP address, I'm already authenticated. So I shut down the email client, chatted with the kid for a while longer, made some coffee, and voila: no open relay.

But like Jesse said, those mails came from somewhere, and the only thing I've established for sure is that they weren't coming from a script on the server. I'll keep poing about; but for now, all is quiet, and life is good.

-Rich

 

[wsuffa]

Ugh. I'm glad that I've not been called in to troubleshoot a problem sending to your server. Nothing like finding out that you've wasted half a day because somebody had a "better way" (or thought that there was no other way).

There are other ways to deal with backscatter. Set up an SPF record on your domain. Tell servers reading SPF (including your server) to hard fail from unknown sources. It'll deny it as part of the smtp transaction so you don't even have to read the message in.

Spam filtering isn't 100% accurate. Until such a time that it is no server admins should black hole mail without notification to the sender or the recipient.

Jason,

Unless you're operating a server in China, Russia, or certain blocks in South America, you're not likely to have any trouble with my server. At all. If you'd read what I wrote before you criticize, you'd see that those are the only areas I blackhole, with the EXCEPTION of excessive backscatter that threatens to bring the server down. 2 hours of 15,000 backscatters per minute is justified in blackholing the BACKSCATTER ONLY.

My goal is 100% pass of legitimate traffic. That's pretty hard to do if I blackhole everything.

I'm sorry I even bothered to mention it.