High CPU Usage due to Web Crawlers

Linode Staff


We are seeing high CPU usage on or Linode due to web crawlers. I am looking for a way to mitigate this?

1 Reply


I suggest using a robot.txt file. To set what crawlers you allow on your site. Place the robot.txt file in the directory of your site.

Add the following to your robot.txt to block all bots from a single directory please not that the / is the directory you want to exclude.

User-agent: *
Disallow: /

Add the below if you would like to block certain bots.

User-agent: Botname
Disallow: /


Please enter an answer

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct