Why is my Linode suddenly using 100% of its CPU?
Linode
Linode Staff
All of a sudden my Linode is using all of its available CPU. Why is this happening and how do I fix it?
1 Reply
hphillips
Linode Staff
There could be many reasons why your Linode is would suddenly using more CPU.
Common Causes
Reboot configuration changes
Many times while trying to configure a server, changes are made to configuration files that were expected to go into effect immediately, but in reality just sit and wait until a service is restarted. When a server reboots, this is usually the time when these changes go into effect, so you are left wondering why your server no longer works in the same way it did before the reboot. This guide may help:
Cron Jobs
Some programs like search index programs or virus scanning software are triggered by a scheduled cron job. When these programs run, they may push your system to its limit. You can take a look at your /etc/crontab file to see when an what may be scheduled to run.
Denial of Service Attacks
Sometimes a malicious actor may use a botnet or swarm of compromised servers across the internet to send requests to your system faster than it can process them. When this occurs your server is unable to keep up with the demand.
Process Accumulation
Some software creates new processes to deal with requests as they come in and may not properly get rid of the processes when they are no longer needed. When this happens, these processes accumulate and together they start to need more CPU time than is available. In these cases, rebooting your Linode will allow you server to go back to normal.
Resource Bottlenecks
When your server starts to run out of memory, your Linode will start to swap memory into and off of the disk. When this happens your server will start to spend lots of its CPU time performing the task of swapping memory and leaving less time to do actual work. The free -m command will let you see how much memory your Linode is using.
If your Linode is running out of disk space it may constantly stop and restart programs that are trying to use disk space. You can check your disk space usage with the df -h command.
IO Wait
Some programs such as a database will request data to be read from the disk and the process will sit and wait while the data is fetched. If enough processes are sitting and waiting for data, other processes that are fetching the data will have less time to run. This creates a situation that is similar to grid lock with traffic where cars block the intersection and no one can move in either direction
Other Users
In some cases if you have more than one user account on your server, you may have other users who are running programs on your server that you are not aware of. The other users may using all of the CPU and you may be unaware of it.
Compromise or Malware
A common sign that your Linode may have been hacked is a sudden spike in CPU usage. Hackers often run software on the systems they break into.
If you believe that your Linode has been compromised, you can start troubleshooting by auditing the following log files and writable directories:
- /var/log/auth.log : Check this log file for signs of unauthorized access and brute-force attempts. Use the ‘last’ command to cross reference recent account logins with this file.
- /tmp : This directory is often used by malicious parties to store files
- Web server logs: There may be a vulnerable script or web application. The location of these log files depends on your web server (apache, nginx, etc.) configuration.
- ps aux : Use this command to audit running processes for foreign processes
Troubleshooting Techniques
top
The top command is very useful to see what processes are using the most CPU time.
Linode Longview
If you take the time to install the Linode Longview client on your Linode, you will be able to look at your resource utilization from within your Linode Manager and identify what processes are using the most CPU.
iostat
if you run the command iostat 1 10 the command will check each second for ten seconds to see how much disk activity is occurring and how much time your CPU is waiting for data from disk.
who
The who command will let you see who else may be using your Linode and could possibly be using up your CPU behind your back.
Things to try
Upgrading your Plan
Upgrading your Linode to a higher plan will give you additional virtual CPUs allowing your Linode to handle larger processing loads.
Switching to a Dedicated CPU Plan
Linode offers plans that provide dedicated CPU resources which will prevent your server from being effected by other Linodes activities.