Unexplained CPU Jump

My CPU usage jumped from an average of 4% to 104% yesterday, and has been sitting around 104% ever since.

I've got one rails app running on there (ubuntu, nginx, passenger), which has really low traffic (this hasn't changed - about a couple of hundred requests/day).

I haven't touched anything on the server in the past week either.

Has anyone experienced anything similar?

4 Replies

Well, does ''htop', 'top' or 'ps' say anything interesting?

104% is a very suspicious number. I bet the 100% is from something single-threaded chewing as much CPU as it can, and the other 4% is everything else.

Thanks mnordhoff - Yep - I have discovered that the 100% was coming from a single perl process. How it got there I have no idea. I also discovered a whole lot more. Lots of request for phpmyadmin and other setup scripts in the log files, and requests for odd domains.

Also a persistent IRCD connection from a atw.hu domain.

In short - the server has been compromised. Backed everything up, and about to rebuild.

Before I delete everything - I'd like to know how they got in though. Do you know where I could find the tell tale signs?

@RayS:

Lots of request for phpmyadmin and other setup scripts in the log files, and requests for odd domains.
Well that's normal. Any public IP suffers a lot of attacks; the important part is whether or not they succeed.

@RayS:

Also a persistent IRCD connection from a atw.hu domain.

In short - the server has been compromised. Backed everything up, and about to rebuild.
OK, that's definitely not normal! Yikes.

(You're sure it's really compromised, not just an attacker attempting to connect or something?)

@RayS:

Before I delete everything - I'd like to know how they got in though. Do you know where I could find the tell tale signs?
Sorry, that's not something I know much about.

1. Back up the entire node (linode's backup service would be ideal for this)

2. Clone the node from the backup

3. Boot the node using finnix so it can't talk to the outside world

4. Check your log files

That's a good start.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct