Google bot spamming my site for admin links?

The last several days I have noticed a marked decrease in my sites performance. I hadn't changed anything so for the life of me I couldn't figure out what was happening. I have a drupal site running nginx that while a very "heavy" site usually only gets low traffic but has readily handled all traffic spikes up to 2000 visitors a day with 1 second page loads. Suddenly I'm seeing page loads of over 7 seconds? In checking my logs I see a bunch of 404 errors so thinking that may be the problem I install the drupal fast_404 module and things improve but not back to the level they should be. I also notice most of my hits are from google bot.

Today I am stunned to find my access log is 17mb (just for today! They usually run under 500kb) with traffic almost entirely form googlebot. The IP addresses listed actually are google IP's. The pages trying to be accessed are almost entirely from equine/admin/* type links which are no longer in use on my site so are generating 404's but would have been inaccessible previously (access denied). I checked webmaster tools and almost all the 404 errors to my site for the last couple of days are to these links and I mean PAGES of them.

So my question is why is google spamming my site trying to find links that aren't there? The changes I made to my sites structure have been in place since March so it's not like I changed things last week. I did add equine/admin as denied on my robots.txt but is there anything else I should do?

Thanks everyone!

1 Reply

Thought I'd drop in with an answer as no-one else had.

google follows links. If spammy links have been posted to your site by a bot then there's nothing you can do about it. 2000 visitors a day is not high traffic and any linode should easily be able to cater for this.

Are you using a caching plugin or better yet APC/Memcached? if not, i'd advise getting this set up. You could always set up apache/nginx to block the /enquine/admin/* pages, this would reduce your load as requests aren't going back to your database. You could also block bots from accessing your site via apache/nginx config files but this is NOT recommended.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct