High CPU Usage due to Web Crawlers
Linode
Linode Staff
Hello,
We are seeing high CPU usage on or Linode due to web crawlers. I am looking for a way to mitigate this?
1 Reply
Hello,
I suggest using a robot.txt file. To set what crawlers you allow on your site. Place the robot.txt file in the directory of your site.
Add the following to your robot.txt to block all bots from a single directory please not that the / is the directory you want to exclude.
User-agent: *
Disallow: /
Add the below if you would like to block certain bots.
User-agent: Botname
Disallow: /