Linode - sudden high CPU - MySQL - hacked?
Hi,
I created a linode about a month ago - I haven't finished setting up my websites so it's not in production.
I suddenly got a CPU warning email. About three hours ago the MySQL instance was going crazy - 94% of useage.
Looking at the auth.log I can see lots of login attempts from various IPs which looks to me like someone attempting to hack the server but I can't see a successful login.
Tracert to these IP addresses do seem to be random Indian IPs I don't recognise.
Why would MySQL suddenly max out - my websites are now not able to login to MySQL and I can't log in via the shell to see any processes running. Is it possible I've left the port open to the outside world and hackers are brute forcing mysql?
I installed all updates (thinking it might be a stuck upgrade) and rebooted the server but it comes up the same.
I'm shutting the server down for now (which is a nice luxury to have) but what have I failed to setup correctly that's left me vulnerable?
Any hints appreciated.
Steve
4 Replies
Trying to connect to the MySQL instance results in:
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
If I kill the Mysql process it instantly starts back up again and starts hitting 94% cpu?!
I am seeing some errors about a failed upgrade. Not sure how to resolve it.
@mdssteve writes:
Looking at the auth.log I can see lots of login attempts from various IPs which looks to me like someone attempting to hack the server but I can't see a successful login.
Welcome to the open internet. You better get used to this. fail2ban(1) will solve this problem.
If I kill the Mysql process it instantly starts back up again and starts hitting 94% cpu?!
You have the monitor process enabled. You should see two mysqld processes -- mysqld_safe & mysqld -- you need to kill them both…and probably shut off the systemd thingie:
sudo systemctl stop mysql
This will probably fail because you write:
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
…but at least, after the failure, it won't try to restart mysqld for you.
So… stop the systemd thing then kill all the mysqld processes:
- mysqld_safe
- mysqld
Kill the mysqld_safe process first.
You write:
I am seeing some errors about a failed upgrade. Not sure how to resolve it.
Once you've killed all the mysqld processes, use your package manager to remove mysql. Once that's done, install MariaDB instead (a completely backward-compatible alternative to mysql that doesn't have any of Oracle's baggage). I don't do a lot of database stuff, but when I have, I've found MariaDB works a lot better.
A failed upgrade is probably a more-likely explanation for the behavior you're seeing than your mysql server getting "hacked". However, you need to mitigate those bogus ssh(1) logins before you move forward. These are the ever-present dictionary attacks from the Chinese & Russians and their various proxies around the world. They're as common as traffic noise. As I said before, check out fail2ban(1)…
-- sw
Thanks sw - I'm pretty sure I installed fail2ban. The IP addresses in the log do seem to change but I will check.
Here's some info from my jail.local file:
## ban times are for 30 days -- adjust to your taste
#
bantime = 30d
...
## ban action -- this one uses ipset and an associated rule to ban all
## members of the set.
#
banaction = iptables-ipset-proto6-allports[blocktype=DROP, bantime=30d]
## There's an equivalent for one-rule-per-ip bans.
#
#banaction = iptables-multiport.conf
...
## Configuration for ssh jail. This will ban an intruder after finding
## ONE attempt in 5 minutes. Change 'maxretry' and 'findtime' to suit
## your taste.
#
[sshd]
enabled = true
mode = extra
port = ssh
filter = sshd[mode=aggressive]
logpath = /var/log/auth.log
maxretry = 1
# ...in 5 minutes
#
findtime = 300
...
-- sw