It sounds like it COULD be an attack, but you can never be 100% positive. It could be a poorly written crawler/script or it could be malicious. Either way, blocking the IP as you did is probably your best course of action.

It may be an attack of some sort, but, as it's only coming from one IP, it's not a DDoS - a Distributed Denial of Service attack, by definition, comes from multiple IPs.

Unless you have family/friends/business in Norway - just ban (drop) that IP and then you have no worries.

Happened again today from: 83.109.195.192

> tail -f -n 0 error.log

[Mon Jan 10 17:29:14 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:16 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:17 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:18 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:19 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:20 2011] [error] [client 83.109.195.192] request failed: error reading the headers

[Mon Jan 10 17:29:22 2011] [error] [client 83.109.195.192] request failed: error reading the headers

Is there anything I can do to prevent it from taking down Apache? Some type of measure to throttle a specific ip if it's requesting too much in a short amount of time? Should I tweak my Apache settings? This is a problem if I'm not around the current second.

Should I setup monin to restart apache automatically? Anyone have some tips/solutions so this doesn't take down the server for a hour+ until I get back to computer?

So in summary, server seems to be getting hit same time every night from a different IP but all related to this "norway isp" telenor.

Even if you get hit with a ton of these it really shouldn't take down Apache. At worst it should just make your site slow while they are happening, but it certainly shouldn't interfere with basic server function or access (e.g., over ssh).

If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.

– David

@db3l:

Even if you get hit with a ton of these it really shouldn't take down Apache. At worst it should just make your site slow while they are happening, but it certainly shouldn't interfere with basic server function or access (e.g., over ssh).

If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.

– David

David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.

This is my config:

> StartServers 1

MinSpareServers 3

MaxSpareServers 6

ServerLimit 12

MaxClients 40

MaxRequestsPerChild 3000

I'm on 756 MB of memory.

I would change the MaxClients to 20 but I don't want to change it arbitrarily without testing it to see if it makes a difference.

I tried to use a "ab" benchmark to see if it would bring the server down but no luck. The server kept responding promptly. Should I leave it on longer? Any ideas?

What would be a good ab benchmark test?

I tried doing the following:

ab -n 5000 -c 10

Which actually ended up closing my local wifi/internet connection. I originally tried -n 1000 -c 5. No luck on slowing down server.

Any advice would be greatly appreciated! Thank you!

ab -c 10 would only make 10 simultaneous connections. Since you have MaxClients set to 40, it would take ab -c 40 to stress test your server properly. You'll probably run out of memory on the server if you're using mod_php.

@arachn1d:

David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.
Well, it sounds like you have an almost nightly occurrence via external sources, so even if you can't generate the necessary load manually, tweak and then just wait a day :-)

Though as hybinet points out, the crucial point is how many simultaneous requests you initiate, not really the total number, since that's what causes all the Apache clients to be instantiated, and likely overloading your memory. You ought to be able to generate quite a few simultaneous connection attempts even over a slow client link.

It depends on your application stack, but MaxClients of 40 feels too high for a 768. Personally I'd have no worries about dropping it to 20 and then watching behavior. Odds are your normal usage won't be affected at all.

There are a number of Apache tuning threads here in the forums you might reference for other ways to benchmark your system. One at http://forum.linode.com/viewtopic.php?t=6272 for example that covers some of the steps to take to monitor and test changes.

BTW, I had similar log patterns on an Apache server of minte in the past, and I've been wracking my brain trying to remember what I identified it as, as it didn't turn out to be malicious. Finally remembered…

In my case it ended up being incoming BitTorrent traffic, where a client behind the same firewall as the server had ended up choosing the same (non-standard) port as was being forwarded to Apache. So when the client stopped, for a period of time thereafter, I got incoming probes from BitTorrent clients looking for my peer, but now landing on Apache.

In general, your logs just say that some client using a binary protocol is connecting to the port Apache is listening on. So your case could be completely different, or could in fact be malicious, but perhaps this will help you think of other possibilities, especially if there's any chance that Apache is listening on a port that may be getting other uses, or proxy's or whatever.

-- David

The error message seems to suggest that whoever is sending those requests isn't even bothering to write correct HTTP headers. Maybe it's not talking HTTP at all, as db3l says.

In that case, you could set up nginx on port 80 and have it proxy all requests to Apache (maybe except some static files, which are better served by nginx). Ideally, nginx will see that the request is not valid HTTP, and drop the connection without passing it on to Apache. Also, a few hundred additional connections on nginx will hardly use any resources, while a few dozen additional connections on Apache can easily bring a VPS to its knees.