So What's a Reasonable Amount of Spam?

[RJM62]

I have a client who's threatening to bail because of excessive spam. He said that in the last 24 hours, it had been "unbearable."

So I ran a mail delivery report and found that out of 847 mails his users received during that period, all but 27 were from addresses to which users had sent email. This was easy to do because some of the users have BoxTrapper enabled, which whitelists every address to which the user sends mail (along with any CCs).

Even assuming that all 27 messages that made it through the multiple levels of spam filtering and weren't in the whitelists were spam, we're only talking about 3.2 percent of the total received messages. I don't think that's "unbearable," especially since the client insists on using easy-to-guess email addresses like "bob," "dan," "office," and so forth. And again, I counted any mail from non-whitelisted addresses as spam.

Also, he has 26 users, and half a dozen of them account for almost all the presumed spam that was received. They're all old accounts that probably should be changed because they've gotten on spam lists over the years, but he refuses to change them.

Frankly, I may fire this client before he gets the chance to bail because he's a pain in my ass. It's a typical father-son situation: The father was a pleasure to do business with, but the son is an obnoxious little twerp who needs to be slapped upside the head a few times. But that's neither here nor there.

So what do you think a "reasonable" amount of spam slipping past the filters would be?

Rich

 

[PaulS]

None? Don't really understand your business, so I don't understand how this is your problem.

 

[RJM62]

None? Don't really understand your business, so I don't understand how this is your problem.

Well, I'm the mail provider, so I do acknowledge a responsibility to try to filter out the spam. But I also know that false positives are unacceptable. That's especially true in this case because the client insists on auto-deleting rather than subject-line re-writing or other sorts of filtering that would preserve the messages in case they were falsely identified as spam.

"None" is unrealistic. Short of human-reading every mail, you'll never achieve perfection in spam filtering (nor anything else, for that matter). My contact form spam filter is about 99.5 percent effective and I still get complaints.

Rich

 

[PaulS]

Well, I'm the mail provider, so I do acknowledge a responsibility to try to filter out the spam. But I also know that false positives are unacceptable. That's especially true in this case because the client insists on auto-deleting rather than subject-line re-writing or other sorts of filtering that would preserve the messages in case they were falsely identified as spam.

"None" is unrealistic. Short of human-reading every mail, you'll never achieve perfection in spam filtering (nor anything else, for that matter). My contact form spam filter is about 99.5 percent effective and I still get complaints.

Rich

99.5 % is pretty damn good, a human reader would not be as good.

 

[RJM62]

99.5 % is pretty damn good, a human reader would not be as good.

Thanks. But that just works on form submissions. It basically looks for robot-like behavior, known-bad IPs, and submissions that didn't come from the form itself. There's very little content filtering and no CAPTCHAs, math problems, pictures of cats, or that sort of nonsense. I like it.

It's not perfect, though. Human spammers from IPs that aren't in the RBLs can get past it. But human-submitted form spam is still a small percentage of the whole, so it makes life a lot easier for clients.

Rich

 

[jesse]

I built out a fairly high volume spam processing cluster years ago to replace a commercial system. For several years it performed quite well but spam is by far worse today. Constantly deal with the very issue you're facing. At this point just looking to get out of the email game and migrate people to Google Apps. They do a far better job than we can with our resources.

 

[denverpilot]

Generally agree with Jesse but we're looking to even front end the Google stuff with something that gives each user full control over their level of tolerance for false positives and ability to recover anything dropped themselves with no help from anyone.

Internet email is a continuing train wreck.

 

[TazzyTazzy]

Generally agree with Jesse but we're looking to even front end the Google stuff with something that gives each user full control over their level of tolerance for false positives and ability to recover anything dropped themselves with no help from anyone.

Internet email is a continuing train wreck.

I thought everyone moved to SMS, fb, Instagram, or Snapchat? Anyone wana see a pic of my...cat?

Email is where it is because no one can agree on a new standard to fight spam.

 

[Bob Noel]

So what do you think a "reasonable" amount of spam slipping past the filters would be?

I have little zero tolerance for spam that is obvious forgery or obvious crap.

At work, the spam filters fail to stop some spam that is so obvious that I'm wondering what the heck the filters are looking at. However, the failure rate of less than 1% overall is acceptable for me at work.

 

[jsstevens]

Jesse and Nate,

Do you or your users have any concerns about Google mining the email and docs content? Or do you have a way to prevent that?

John

 

[EdFred]

We put in an untangle box, and holy crap did that cut down on the amount of spam we got. 50-100 a day to less than 5 a day.

 

[denverpilot]

Jesse and Nate,



Do you or your users have any concerns about Google mining the email and docs content? Or do you have a way to prevent that?



John

Pretty much a concern about any cloud service. Have to read the terms of service and know what you're agreeing to. If you work somewhere big enough, let the corporate attorney review it.

(Make it the Legal department's problem. )

We may not move to Google, we may maintain the existing mail server (even though I hate it) and still front end it with an anti-spam service.

Same possible problem with anti spam services too, by the way, if you think about it.

As one guy's signature line on the CentOS mailing lists says,

"Email is not a Secure Channel."

^^ That has to be beat into people's heads. Unless they're encrypting the content, nothing should be put in an email you don't want someone else reading. Period.

 

[wsuffa]

Pretty much a concern about any cloud service. Have to read the terms of service and know what you're agreeing to. If you work somewhere big enough, let the corporate attorney review it.

(Make it the Legal department's problem. )

We may not move to Google, we may maintain the existing mail server (even though I hate it) and still front end it with an anti-spam service.

Same possible problem with anti spam services too, by the way, if you think about it.

As one guy's signature line on the CentOS mailing lists says,

"Email is not a Secure Channel."

^^ That has to be beat into people's heads. Unless they're encrypting the content, nothing should be put in an email you don't want someone else reading. Period.

There is a huge difference between "not a secure channel" and "actively mining your data to send you more advertising or present more ads on services (including pay services) that you use".

I am running Untangle plus desktop apps. They take >99.5% of the spam out.

 

[Palmpilot]

Yahoo's spam filter is very effective, except that I do have to scan it for false positives, which happen somewhere between once a week and once a month. Fortunately, the majority of spammers seem to put the subject in the "from" field, so in most cases, looking at that field alone makes it immediately obvious which is which.

 

[denverpilot]

There is a huge difference between "not a secure channel" and "actively mining your data to send you more advertising or present more ads on services (including pay services) that you use".



I am running Untangle plus desktop apps. They take >99.5% of the spam out.

Sure but email through a third party vendor unencrypted is guaranteed to be available for viewing, parsing, whatever.

99.5% won't be nearly high enough soon, if it is now. I'm still seeing a doubling of spam annually if not more.

The best solution so far has been to completely block APNIC IP blocks. We don't have customers in Asia and even if we did, I'd open only their servers or suggest they find a different server to use somewhere that gives a damn about their local public network hygiene.

 

[Palmpilot]

If spam is doubling every year, it sounds like The Tragedy of the Commons will soon swallow the Internet whole.

 

[jsstevens]

Pretty much a concern about any cloud service. Have to read the terms of service and know what you're agreeing to. If you work somewhere big enough, let the corporate attorney review it.

(Make it the Legal department's problem. )

We may not move to Google, we may maintain the existing mail server (even though I hate it) and still front end it with an anti-spam service.

Same possible problem with anti spam services too, by the way, if you think about it.

As one guy's signature line on the CentOS mailing lists says,

"Email is not a Secure Channel."

^^ That has to be beat into people's heads. Unless they're encrypting the content, nothing should be put in an email you don't want someone else reading. Period.

Yeah, we're maintaining our own infrastructure here due to DoD security concerns. They're big on you being able to point to a box and say "The data is on there and it's unplugged." Then you can start cleaning it up. My biggest concern with it is somebody accidentally (I hope!) sending classified data in a email, which requires using approved cleaning techniques on all affected devices. (iPhone for instance doesn't have one other than shredding it.) If I catch somebody forwarding email (which we do allow) to a google account, I politely remind them that it's actively mined and our proprietary data should never be on there. We're small enough and tech savvy enough I'm willing to live with that for now. Someday it'll probably bite us though...

I don't say "email is not a secure channel", I say "Email is FOREVER." But it amounts to a similar sentiment. Don't say it in email if you don't want someone else to read it.

Thanks. I wondered if you had such discussion with Google. They are really convenient and there's the rub. The cartoon could applies to google too...

John

 

[denverpilot]

Nope. No discussions with them other than when they call annually trying to sell us more than the geocoding stuff we outsource to them. And my response is always, "We don't use all the features. Price is too high for what we use it for but we prefer your accuracy over [insert biggest competitor here]. Got a $10000 discount for that feedback last year.

 

[denverpilot]

^^^ which means they want the data they're mining about our customers more than they care if we pay for the service.

 

[jsstevens]

^^^ which means they want the data they're mining about our customers more than they care if we pay for the service.

Yep. It's all about the money...

Oh and "Don't be evil". Yeah.

 

[wsuffa]

Sure but email through a third party vendor unencrypted is guaranteed to be available for viewing, parsing, whatever.

99.5% won't be nearly high enough soon, if it is now. I'm still seeing a doubling of spam annually if not more.

The best solution so far has been to completely block APNIC IP blocks. We don't have customers in Asia and even if we did, I'd open only their servers or suggest they find a different server to use somewhere that gives a damn about their local public network hygiene.

Agree on all points. I don't care to make it any easier on Google to mine the inbound email that arrives from a variety of different points. Understood and accepted that they're going to get whatever flows through their servers/networks/routers anyway.

99.5% is not adequate now. My number is higher than that, but I've not bothered to compute it for a while.

I've got a manual block list on the server that rejects connect requests from certain addresses and blocks. That list is hundreds of lines long and takes out much of the AP region and some of Russia. It's not perfect, but it helps. There are some addresses in that region that I care about and I do send them to a gmail forwarding box or except them in the access file.

 

[jesse]

Personally I don't care if Google mines my work email. I don't write anything in an email at work that I expect any degree of confidentiality. As a company we barely use email for internal stuff, primarily use chat. Only purpose for email to me is for conversing with other companies and I'm always quite careful about what I write.

My personal email I'm considerably more bothered by it being indexed. I will be moving it to Amazons email service once it comes out of tech preview assuming their spam filtering holds up. I don't get spam whatsoever on Google Apps. Move my oldest domain to basically any other email service and I get hundreds a day. I'm hoping Amazons offering will be good with blocking spam.

 

[RJM62]

Personally I don't care if Google mines my work email. I don't write anything in an email at work that I expect any degree of confidentiality. As a company we barely use email for internal stuff, primarily use chat. Only purpose for email to me is for conversing with other companies and I'm always quite careful about what I write.

My personal email I'm considerably more bothered by it being indexed. I will be moving it to Amazons email service once it comes out of tech preview assuming their spam filtering holds up. I don't get spam whatsoever on Google Apps. Move my oldest domain to basically any other email service and I get hundreds a day. I'm hoping Amazons offering will be good with blocking spam.

I'd consider an Amazon service. But Google, no. I don't care how good their spam filtering is. I'd rather lose the client.

Rich

 

[Ghery]

Yahoo's spam filter is very effective, except that I do have to scan it for false positives, which happen somewhere between once a week and once a month. Fortunately, the majority of spammers seem to put the subject in the "from" field, so in most cases, looking at that field alone makes it immediately obvious which is which.

I have to check the Spam folder on m Comcrap e-mail now and then, too. False positives, but so far they have been e-mails that wouldn't be the end of the world if I missed.

 

[Palmpilot]

One time I had to fish an email from my brother out of there!

 

[JohnAJohnson]

If spam is doubling every year, it sounds like The Tragedy of the Commons will soon swallow the Internet whole.

In the case of two identical apartment complexes... In one, each apartment has its own electric meter while the other offers free electricity to its tenants. Which complex consumes the most electricity monthly?

 

[denverpilot]

In the case of two identical apartment complexes... In one, each apartment has its own electric meter while the other offers free electricity to its tenants. Which complex consumes the most electricity monthly?

Whichever one has the grow lamps for the local pot dealer.

 

[RJM62]

The only solution I can think of to the greater spam problem would require a fee for each email sent over a very small number that would be included with an account. It doesn't have a prayer's chance in hell of happening.

Rich

 

[jesse]

The only solution I can think of to the greater spam problem would require a fee for each email sent over a very small number that would be included with an account. It doesn't have a prayer's chance in hell of happening.

Rich

They'd just start paying with stolen credit cards

 

[James331]

99.5 % is pretty damn good, a human reader would not be as good.

Agree

Plus who knows where the users are submitting their email addresses /domain.

With my .aero domain I get next to no spam, only junk I get is after submitting my email for a warranty card, or opening a new cell phone line or something, one click on un subscribe and that's that.

I think some folks bring spam upon themselves.

 

[denverpilot]

The worst "spam" ever was in registering one of my SIP phone numbers in an online "411" database. That thing gets fifteen unwanted phone calls per day. And I kinda like the phone number, but I got tired of the calls and dropped it from the home Asterisk box. Now
My SIP provider sends me an email every time they can't route a call to it (and charge me to route it) and they don't charge for the email warning, so I leave the number active on their system just to marvel at how bad it is and use it as an example of why NEVER to register via any SIP provider's 411 service.

 

[wsuffa]

The only solution I can think of to the greater spam problem would require a fee for each email sent over a very small number that would be included with an account. It doesn't have a prayer's chance in hell of happening.

Rich

"We send for you free email if you let us wire inherited funds to you from Nigeria"

 

[EdFred]

Agree

Plus who knows where the users are submitting their email addresses /domain.

With my .aero domain I get next to no spam, only junk I get is after submitting my email for a warranty card, or opening a new cell phone line or something, one click on un subscribe and that's that.

I think some folks bring spam upon themselves.

Or when one of your vendors puts your company email link naked on their webpage. Thanks for those 13,000 emails from Botswana that I would have gotten were it not for untangle/spamhaus.