Well....that was dumb

[SkyHog]

Last night, I was browsing the internet, and things started going quite slow...like mollases slow. Took almost 5 minutes to load a page off PoA. Decided I better start digging.

Let me explain how I have my stuff set up at home.

I have a file server, a media server, a web server, and a "multi-purpose" server which does everything the others do, and then some. Why so many? Why not?

The "multi-purpose" server is my DNS server, my DHCP server, and has Apache running on it for a specilized website that I run internally, with an instance of MySQL specific to that one website. It also contained most of my important documents that I backed up for when I moved across the country. It also serves as the portal to my entire network, as I run VNC on it and an X-Server, for easy remote control.

At some point in the past, I am not sure when or why, I enabled port forwarding on my router, forwarding a bunch of ports to that IP address. Among those ports were the SSH port, the VNC port the MySQL port, and port 80. Still, no clue why I enabled this.

About 2 days ago, I was testing out TightVNC in lieu of the default Vino VNC that comes with Ubuntu. I neglected to set a password on this instance of VNC.

You can see the accident chain building up, can't you....

Now that the background is out of the way, back to the original story. I noticed the slowdown, VNCed into my "multi-purpose" server, and found Ubuntu Update Manager running, password window open, with a long password typed in and "Incorrect Password" on the screen. I turned to my wife and said "Stupid cats," since they like to play on the keyboard for the server sometimes, and I don't always disconnect it.

But then I closed the update window and saw that a spreadsheet was opened, with a dump of the data that is contained within the MySQL Server (financial information), and "You got pwned!!!" in cell A1.

Oh crap.

I immediately went into snoop mode. Found that nearly 100% of my CPU power was being taken up by 2 perl scripts, there were 4 instances of TightVNC running, and 2 active connections to IRC servers. It appears that connections to my other servers had not occurred, thankfully, despite them being wide open to that machine in question.

I dug through auth.log and found out that there have been attempts to break in going on for 2 days using the open SSH port and username 'root' (thank God that's disabled by default in Ubuntu). This was a brute force attempt that persisted for 2 days, every 2-3 seconds a different attempt was tried. auth.log was filled with "POSSIBLE BREAK IN ATTEMPT."

But then the attempts stopped in auth.log. And a bunch of cron jobs were created, which were set to restart the perl scripts in question above. The entire history of what this guy was doing was present in auth.log.

Then I found an IRC log...with some DCC file transfer in it that didn't look pretty. Unfortunately, I don't know where on my file system that these files were stored, because I can't seem to find them. I also am not missing a large amount of disk space, so I am not sure the files are even there anymore.

Rather then mess around too much more, its time to restore a backup.

Wow, I'm dumb.

 

[jesse]

Best off to just close off everything and use a VPN solution to get into your network instead of publicly accessible ports. If everybody did that there would be a lot less security incidents.

 

[JeffDG]

D'oh!

Sorry, man.

 

[JeffDG]

Best off to just close off everything and use a VPN solution to get into your network instead of publicly accessible ports. If everybody did that there would be a lot less security incidents.

+1

 

[Dean]

That was totally Greek to me.

 

[tonycondon]

That was totally Geek to me.

FTFY.

 

[SkyHog]

Best off to just close off everything and use a VPN solution to get into your network instead of publicly accessible ports. If everybody did that there would be a lot less security incidents.

I generally don't access any of it from outside my network, so I don't even know why I had it open to the outside world.

Blegh.

 

[Kelvin]

It was probably that drunk down the road...who can't drive...

 

[Tom-D]

That was so far above my head, I don't even think it left a con trail.

 

[wsuffa]

Best off to just close off everything and use a VPN solution to get into your network instead of publicly accessible ports. If everybody did that there would be a lot less security incidents.

+1. OpenVPN is your friend. IPSec endpoint routers are your friend (and cheap). OpenVPN can be free. DD-WRT w/OpenVPN turned on runs exceedingly well on various wireless routers.

The only publically accessible stuff should be Port 25 if you're running an inbound mail server and Port 80 if you're running a web server. Even at that there's some risk. OpenVPN and IPSec on a VPN endpoint router keep the open ports isolated in the box. Even my Ooma box is behind a firewall (w/QoS turned on).

 

[pilot_dude]

The only thing dumb about this thread is this poster's knowledge of what is being discussed.

 

[SkyHog]

Oh, I should point out as well, once the SSH attempts stopped, almost immediately afterward, a series of new TightVNC instances started, which leads me to believe that the little bugger gave up on trying to brute force, and instead simply used a VNC client and connected, which gave him unfettered access to the machine, and which is when things actually started happening.

Wah.

 

[Jaybird180]

At some point in the past, I am not sure when or why, I enabled port forwarding on my router, forwarding a bunch of ports to that IP address. Among those ports were the SSH port, the VNC port the MySQL port, and port 80. Still, no clue why I enabled this.

Maybe you were doing some testing and just forgot...it happens. Two person integrity would have saved you in this case.

About 2 days ago, I was testing out TightVNC in lieu of the default Vino VNC that comes with Ubuntu. I neglected to set a password on this instance of VNC.

See my above point?

But then I closed the update window and saw that a spreadsheet was opened, with a dump of the data that is contained within the MySQL Server (financial information), and "You got pwned!!!" in cell A1.

And why do you have financials on a 'general' server? Next time, add SSN mother's maiden name and blood type. j/k

I dug through auth.log and found out that there have been attempts to break in going on for 2 days using the open SSH port and username 'root' (thank God that's disabled by default in Ubuntu).

Does it really matter at this point? They already got the goods via path of least resistance. Why bother cracking root? It's pointless, unless they just wanted to prove a point.

 

[SkyHog]

Maybe you were doing some testing and just forgot...it happens. Two person integrity would have saved you in this case.

Yep. But at home, I am the only sysadmin around, and the only one with the knowledge to do this stuff.

See my above point?

Yep. But, the port forwarding wasn't touched at that point. That was all about remoting from within the network, not outside.

And why do you have financials on a 'general' server? Next time, add SSN mother's maiden name and blood type. j/k

LOL. That's something that's going to change. Its a webapp that I wrote that we use for budget purposes. Its going to get its own dedicated server now, even though that's overkill. The funny part is that

Does it really matter at this point? They already got the goods via path of least resistance. Why bother cracking root? It's pointless, unless they just wanted to prove a point.

Nah, the point was that they tried for a long time to crack root before even trying VNC. Like 2 days. If they had tried VNC first, they'd have had me much easier than they did. That's actually the only good thing about this.

 

[JeffDG]

Go find yourself a nice aggressive little honeypot and set up your port forwarding to it...

 

[rpadula]

Is this the point in the thread where someone should say "Get a Mac?"

 

[SkyHog]

Is this the point in the thread where someone should say "Get a Mac?"

LOL! Same exact thing would have happened on a mac if I had left it this open. VNC works on a Mac, as does SSH, IRC, and everything else that was used.

Except I wouldn't know how to check the logs

 

[JeffDG]

LOL! Same exact thing would have happened on a mac if I had left it this open. VNC works on a Mac, as does SSH, IRC, and everything else that was used.

Except I wouldn't know how to check the logs

And when you did find them, the logs would consist of "Everything's fine, you don't need to worry about the details"

 

[flyingcheesehead]

LOL! Same exact thing would have happened on a mac if I had left it this open. VNC works on a Mac, as does SSH, IRC, and everything else that was used.

Except I wouldn't know how to check the logs

Well, depending on what logs you're looking for, they'll either be in /Library/Logs or /var/log.

And when you did find them, the logs would consist of "Everything's fine, you don't need to worry about the details"

 

[TMetzinger]

LOL! Same exact thing would have happened on a mac if I had left it this open. VNC works on a Mac, as does SSH, IRC, and everything else that was used.

Except I wouldn't know how to check the logs

And when you found them they'd say:

01/19/2011 20:00:01 GMT - Move along, nothing to see here.
01/19/2011 20:00:02 GMT - Move along, nothing to see here.
01/19/2011 20:00:03 GMT - Move along, nothing to see here.
01/19/2011 20:00:04 GMT - Move along, nothing to see here.
01/19/2011 20:00:05 GMT - Move along, nothing to see here.
01/19/2011 20:00:06 GMT - Move along, nothing to see here.
01/19/2011 20:00:07 GMT - You need more Apple Products.
01/19/2011 20:00:08 GMT - Move along, nothing to see here.
01/19/2011 20:00:09 GMT - Move along, nothing to see here.
01/19/2011 20:00:10 GMT - Move along, nothing to see here.
.
.
.
Edit: I see that great minds think alike.

 

[Jaybird180]

The cracker should have inserted that into the logs...now THAT would have really been funny.

Have you tried any reverse tracing? Perhaps scare them with 'computer fraud and abuse act'. Then maybe the gov't will stop harrassing the ATT/iPad guys.

 

[SkyHog]

The cracker should have inserted that into the logs...now THAT would have really been funny.

Have you tried any reverse tracing? Perhaps scare them with 'computer fraud and abuse act'. Then maybe the gov't will stop harrassing the ATT/iPad guys.

Everything points to Belgium. Whether proxied or truly Belgum, that's as far as I went.

Not a whole lot I can say to a hacker like that.

"Oh, dude, you know that system I practically begged a hacker to take? You shouldn't have hacked it."

 

[denverpilot]

I once got the "joy" of being chatted with (to?) by a hacker in South America (or coming through a South American proxy, anyway) who broke into a DNS server that management wouldn't let me patch with an emergency remote root exploit (BIND 4 days...) that was on the public Internet.

After we got done patching all the OTHER servers with a 1 AM phone call to the COO, and doing all the drive pulling and forensics required by the legal department (which they never asked us to do again, ever... wasn't worth it), the night was over.

Patch released on a Friday around 2PM, requested permission, denied because "Engineering" hadn't reviewed it, hacker in the box at midnight, tripped a number of defenses I had set up on the machines, but mainly screwed up and stopped DNS services by accident, which set off alarms in the NOC, NOC calls, I log in... he sends a wall message... "You caught me."

Not a fun night, and a really crappy way to start the weekend, which I proceeded to mostly sleep through all of Saturday then after getting done around 5AM.

Fun post-trauma conversation with the bosses on Monday in a meeting about how I probably knew better than they did what "remote root exploit" meant.

 

[Dave Theisen]

As I read the OP, I swear I heard the teacher from Charlie Brown. If you are young enough to understand Nick's computer talk, then you probably aren't old enough to get my reference. :wink2:

 

[denverpilot]

As I read the OP, I swear I heard the teacher from Charlie Brown. If you are young enough to understand Nick's computer talk, then you probably aren't old enough to get my reference. :wink2:

I get both. I'm in that "lost in the middle" older Gen X crowd... the non-slacker variety. :wink2:

 

[rpadula]

As I read the OP, I swear I heard the teacher from Charlie Brown. If you are young enough to understand Nick's computer talk, then you probably aren't old enough to get my reference. :wink2:

I get both. I'm in that "lost in the middle" older Gen X crowd... the non-slacker variety. :wink2:

Yeah, me too.

 

[JeffDG]

Yeah, me too.

three

 

[gprellwitz]

three

four!