My sshd was bruteforced!

Few minutes ago my sshd was bruteforced by one of your accounts, namely li123-111.members.linode.com.

Looks like it has been hacked.

Here's a snipplet of auth.log (notice, that log time is gmt+10):

Feb  6 14:18:04 samolet sshd[10763]: Invalid user students from 69.164.208.111                                                                                                      
Feb  6 14:18:04 samolet sshd[10763]: pam_unix(sshd:auth): check pass; user unknown                                                                                                  
Feb  6 14:18:04 samolet sshd[10763]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com                           
Feb  6 14:18:06 samolet sshd[10763]: Failed password for invalid user students from 69.164.208.111 port 47634 ssh2                                                                  
Feb  6 14:18:08 samolet sshd[10765]: Invalid user students from 69.164.208.111                                                                                                      
Feb  6 14:18:08 samolet sshd[10765]: pam_unix(sshd:auth): check pass; user unknown                                                                                                  
Feb  6 14:18:08 samolet sshd[10765]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com                           
Feb  6 14:18:10 samolet sshd[10765]: Failed password for invalid user students from 69.164.208.111 port 49052 ssh2                                                                  
Feb  6 14:18:12 samolet sshd[10767]: Invalid user students from 69.164.208.111                                                                                                      
Feb  6 14:18:12 samolet sshd[10767]: pam_unix(sshd:auth): check pass; user unknown                                                                                                  
Feb  6 14:18:12 samolet sshd[10767]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com                           
Feb  6 14:18:14 samolet sshd[10767]: Failed password for invalid user students from 69.164.208.111 port 50710 ssh2
Feb  6 14:18:17 samolet sshd[10769]: Invalid user students from 69.164.208.111
Feb  6 14:18:17 samolet sshd[10769]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:17 samolet sshd[10769]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:19 samolet sshd[10769]: Failed password for invalid user students from 69.164.208.111 port 52243 ssh2
Feb  6 14:18:21 samolet sshd[10771]: Invalid user squid from 69.164.208.111
Feb  6 14:18:21 samolet sshd[10771]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:21 samolet sshd[10771]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:23 samolet sshd[10771]: Failed password for invalid user squid from 69.164.208.111 port 53818 ssh2
Feb  6 14:18:25 samolet sshd[10773]: Invalid user squid from 69.164.208.111
Feb  6 14:18:25 samolet sshd[10773]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:25 samolet sshd[10773]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:27 samolet sshd[10773]: Failed password for invalid user squid from 69.164.208.111 port 55409 ssh2
Feb  6 14:18:29 samolet sshd[10775]: Invalid user support from 69.164.208.111
Feb  6 14:18:29 samolet sshd[10775]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:29 samolet sshd[10775]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:31 samolet sshd[10775]: Failed password for invalid user support from 69.164.208.111 port 56912 ssh2
Feb  6 14:18:33 samolet sshd[10777]: Invalid user support from 69.164.208.111
Feb  6 14:18:33 samolet sshd[10777]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:33 samolet sshd[10777]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:35 samolet sshd[10777]: Failed password for invalid user support from 69.164.208.111 port 58437 ssh2
Feb  6 14:18:37 samolet sshd[10779]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com  user=sys
Feb  6 14:18:39 samolet sshd[10779]: Failed password for sys from 69.164.208.111 port 60046 ssh2
Feb  6 14:18:41 samolet sshd[10781]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com  user=sys
Feb  6 14:18:43 samolet sshd[10781]: Failed password for sys from 69.164.208.111 port 33356 ssh2
Feb  6 14:18:45 samolet sshd[10783]: Invalid user sysadmin from 69.164.208.111
Feb  6 14:18:45 samolet sshd[10783]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:45 samolet sshd[10783]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:47 samolet sshd[10783]: Failed password for invalid user sysadmin from 69.164.208.111 port 34917 ssh2
Feb  6 14:18:49 samolet sshd[10787]: Invalid user sysadmin from 69.164.208.111
Feb  6 14:18:49 samolet sshd[10787]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:18:49 samolet sshd[10787]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:18:51 samolet sshd[10787]: Failed password for invalid user sysadmin from 69.164.208.111 port 36397 ssh2
Feb  6 14:18:53 samolet sshd[10789]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com  user=sync
Feb  6 14:18:56 samolet sshd[10789]: Failed password for sync from 69.164.208.111 port 37983 ssh2
Feb  6 14:18:58 samolet sshd[10791]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com  user=sync
Feb  6 14:18:59 samolet sshd[10791]: Failed password for sync from 69.164.208.111 port 39625 ssh2
Feb  6 14:19:02 samolet sshd[10793]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com  user=sync
Feb  6 14:19:03 samolet sshd[10793]: Failed password for sync from 69.164.208.111 port 41021 ssh2
Feb  6 14:19:06 samolet sshd[10795]: Invalid user tech from 69.164.208.111
Feb  6 14:19:06 samolet sshd[10795]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:19:06 samolet sshd[10795]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:19:08 samolet sshd[10795]: Failed password for invalid user tech from 69.164.208.111 port 42470 ssh2
Feb  6 14:19:10 samolet sshd[10797]: Invalid user tech from 69.164.208.111
Feb  6 14:19:10 samolet sshd[10797]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:19:10 samolet sshd[10797]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:19:12 samolet sshd[10797]: Failed password for invalid user tech from 69.164.208.111 port 44090 ssh2
Feb  6 14:19:14 samolet sshd[10799]: Invalid user telnetd from 69.164.208.111
Feb  6 14:19:14 samolet sshd[10799]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:19:14 samolet sshd[10799]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:19:15 samolet sshd[10799]: Failed password for invalid user telnetd from 69.164.208.111 port 45447 ssh2
Feb  6 14:19:18 samolet sshd[10804]: Invalid user telnetd from 69.164.208.111
Feb  6 14:19:18 samolet sshd[10804]: pam_unix(sshd:auth): check pass; user unknown
Feb  6 14:19:18 samolet sshd[10804]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=li123-111.members.linode.com
Feb  6 14:19:18 samolet sshd[1228]: Received signal 15; terminating.

30 Replies

You should send this info to abuse@linode.com

People still setup SSH to use passwords???

The abuse email should be more prominent on the main site, the Contact Us page seems like a logical choice. There have been a few posts in the forums in recent weeks that should have really gone directly to abuse@linode.com.

The abuse contact is exactly where you'd expect it to be:

mwalling@youtoo:~$ whois 69.164.208.111 | grep -i abuse
RAbuseHandle: LAS12-ARIN
RAbuseName:   Linode Abuse Support
RAbusePhone:  +1-609-593-7103
RAbuseEmail:  abuse@linode.com
OrgAbuseHandle: LAS12-ARIN
OrgAbuseName:   Linode Abuse Support
OrgAbusePhone:  +1-609-593-7103
OrgAbuseEmail:  abuse@linode.com

Don't know how many other people do this, but on any new install the first thing is change the SSH port from 22 to something else ( 22222 for example ).

[Edit] Sorry, misread that post as from a linode customer, not other way aroud … my gaff :P

@mooseday:

Don't know how many other people do this, but on any new install the first thing is change the SSH port from 22 to something else ( 22222 for example ).

[Edit] Sorry, misread that post as from a linode customer, not other way aroud … my gaff :P

Thats exactly what i do but they still find a way to figure out the port hence why i use DenyHost :)

Ive installed DenyHost yesterday as i was reading on the linode forum that people were doing "back yard" attacks where they bruted machines on the same network!

I dont actually see the point with brute forcing.. Two of our old server were bruted into before we looked into DenyHost.. Why do people actually brute force do they actually get anything out of it?

Brute forcing is very attractive to attackers because it works. Not only has my employer fallen victim to a brute force attack, I have talked to a lot of security professionals who have experienced the same problem. Users create easily guessed passwords all the time. I used to perform password cracking at my primary place of employment (with permission) and you would be amazed by the passwords used. Want samples? Here are some off the top of my head.

Password1

Football1

Dolphin1

Zzzzzzzz

Abcd1234

Abcdefg1

While there are lots of reasons that these shouldn't even be allowed as passwords, it illustrates that users will generally choose simplicity over complexity.

@Key:

Ive installed DenyHost yesterday as i was reading on the linode forum that people were doing "back yard" attacks where they bruted machines on the same network!
This is why I firewall off my private interface. :)

That said, when computers are compromised, it is quite common that the attacker will take a look at the interfaces and then go for any other devices they can see – with an emphasis on machines on the same LAN as the compromised host. That way, if the admin cleans one machine, they still have another… and it is likely that the admin will leave the same hole as they did previously.

@Key:

I dont actually see the point with brute forcing.. Two of our old server were bruted into before we looked into DenyHost.. Why do people actually brute force do they actually get anything out of it?

Yes. Because people are lazy and configure accounts with dumb names and weak passwords. They don't need root to DDoS a site, just basic connectivity. When you think about it, you can do quite a lot with a regular account.

Yeh, we had a new employee set up a test linux mailserver on a spare public IP and set root as "letmein" .. got hacked via SSH within 3 days and was spamming the beans out of everything. Only detected it as our network started dying. His response was "it's linux .. it perfectly safe from hacking and viruses" … sigh.

Oh i see.. The once back in september we got bruted and they took down the apache and mysql users and then started uploading documents that didnt look 100% legal.. Its amazing how they (well it) didnt delete any of our websites and what not

Brute forcers make me angry we now have to install extra software like DenyHost or completely disable the service to stop them attempting to break in!

There should be a way to report and take them down for good!

I've just installed this, pretty simple to install.

@Key:

Brute forcers make me angry we now have to install extra software like DenyHost or completely disable the service to stop them attempting to break in!

There should be a way to report and take them down for good!

The business I run off of Linode is a computer security business. I am slowly working on scripts that will watch for brute force attempts and centralize the source IP addresses the attempts are coming from. This data will be available (not sure if it will be free or a small subscription fee) so that customers can block hosts using Netfilter/iptables based on the information gathered. Essentially a Spamhaus for SSH brute force attacks.

If you run DenyHosts, you can optionally make it sync with a central database. Check the config file.

@vonskippy:

People still setup SSH to use passwords???

This.

No reason not to have SSH configured to allow only PK authentication.

@kmweber:

@vonskippy:

People still setup SSH to use passwords???

This.

No reason not to have SSH configured to allow only PK authentication.

+1

Even helps you out if you have to manage X boxes, you can use single key, some ssh-agent magick and ssh in and out of them without ever needed to invent, remember or type any passwords.

@kmweber:

No reason not to have SSH configured to allow only PK authentication.
I don't carry a storage device around with me at all times, so having credentials that can be memorized is useful to me.

@Vance:

@kmweber:

No reason not to have SSH configured to allow only PK authentication.
I don't carry a storage device around with me at all times, so having credentials that can be memorized is useful to me.

Pssh, you dont have your 4096 bit key memorized?

@Vance:

I don't carry a storage device around with me at all times, so having credentials that can be memorized is useful to me.
You'd rather use a untrusted system with a password then carry around a teeny tiny thumbdrive?

If I'm not at my desktop (home or work) or I'm on the road and I don't have my smartphone or netbook or notebook, then I don't log in.

@vonskippy:

@Vance:

I don't carry a storage device around with me at all times, so having credentials that can be memorized is useful to me.
You'd rather use a untrusted system with a password then carry around a teeny tiny thumbdrive?

If I'm not at my desktop (home or work) or I'm on the road and I don't have my smartphone or netbook or notebook, then I don't log in.

+1 * 1e400

Totally agree.

@vonskippy:

You'd rather use a untrusted system with a password then carry around a teeny tiny thumbdrive?
Exactly what benefit do you think a private key will give you if you plug it into an untrusted system? All I need to do is copy off the data on your thumbdrive (eh; just copy files that are used) and perform keyboard logging, and I now have your key and passphrase.

(And that assumes you have your own ssh client on the thumbdrive and aren't using an insecure one on the machine).

Untrusted systems are, by definition, untrusted.

SelfishMan told me about http://www.yubico.com/products/yubikey/

I've been using it with its PAM module for a couple months now, and it works very well. It shows up as a HID keyboard, and when plugged into a virgin port on a Windows machine, is ready to go in ~ 5 seconds. It does mean that you still have to accept password logins over SSH, but you can remove passwd from PAM for the ssh service (or something like that. I come from Slackware where PAM is the root of all evil, so it is a little bit voodoo magic).

@vonskippy:

If I'm not at my desktop (home or work) or I'm on the road and I don't have my smartphone or netbook or notebook, then I don't log in.
Okay. That's the decision you've made.

The assertion was that there was "no reason" to use password authentication. You may disagree with my reason, but it's still a reason.

As Mr. Schneier is fond of saying, security is about making tradeoffs. In some cases I have looked at the value of what is being protected, the convenience of being able to log in via password, and the countermeasures taken to prevent illegitimate logins, and decided password authentication is worthwhile.

If you are going to login from an untrusted computer - then your only hope is actually a key based login - and a very hard to break root password - though even then it's a matter of local brute force attack.

So just don't use untrusted computers to login to your server…

That being said, it's not a good argument against key based login at all.

The way to secure the SSHD is:

1. move the port from default 22 to something random

2. create an everyday account that you will use to login

3. disable "sudo su" if you are using distro that is set up that way by default (like Ubuntu).

4. disable remote root login

5. setup key based login

5. setup private/public key (with password) and make sure you can login as non-root user to the server

5. disable login by just password - hence leaving PK as only option

That means from that point on you'll be logging in to your server as non-root, password protected key file will mean that you've got both the private key file and password, and you'll need to become root (and have root password) before you can do any damage.

So it's :

one root password

  • vs -

1. find sshd port

2. somehow steal the private key file (which is the hardest part)

3. brute force the password used on the private key file

4. brute force the root password

Yeah, bad news is that you'll need to remember 2 passwords, 1. for key, and 2. for root. Though if you really really want to remember just one password - then it's better to have a passwordless private key file for non-root account + really strong root password.

@mwalling:

SelfishMan told me about http://www.yubico.com/products/yubikey/
Interesting device, but inadequate by itself for security; what happens if it's stolen? It's good as part of a two-factor authentication system (eg password plus yubikey OTP).

I'm using it in 2 factor mode for sshd (which is different from my /etc/passwd password)

Hmmm, passwords are good enough for my bank account, my retirement fund, and oh yeah LISH and LPM.

In 2+ years of running sshd on port 22 of my linodes, there has not been a single attempt that even supplied a valid username.

Paranoia is great, but well-chosen passwords are still plenty secure for my purposes.

> In 2+ years of running sshd on port 22 of my linodes, there has not been a single attempt that even supplied a valid username.

Do you change root to something else then?

In ~5 minutes of setting up a fresh Linode and having sshd on port 22, I had 100s of hits trying to guess the root password.

In sshd_config:

AllowUsers someusername
PermitRootLogin no

@vonskippy:

People still setup SSH to use passwords???

It amazes me too. There is no excuse for falling to simple username/password dictionary attacks.

Here is a clue people:

Use SSH public key authentication

Limit keys to certain source ips if practical ( yes it does that )

Limit keys to certain commands if practical ( yes it does that too )

Disable root authentication or disable root password authentication

Don't use predictable usernames

Implement a decent password policy

Implement SSH connection rate limiting in iptables

Changing the SSH port isn't security. At best it reduces the rate of attacks

If you know enough to argue with the above ( like the people who say denyhosts is better than iptables rate limiting ) then do what you know to be right.

You might prefer OSSEC over DenyHosts. Lots of useful features that DenyHosts doesn't have and of course it blocks brute-force attempts. Takes about five minutes to install..

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct