Let that be a lesson to me!
Wow, my personal web site has 1500 hits already. I migrated the site to the linode on 16th May and for that half of the month there was 800 hits total.
More research… Ah hah… 3 or 4 search engines decided to reindex my site at the beginning of the month, and I have 3 cgi scripts that generate 2300 links to data views…
I think I need a robots.txt file to prevent those cgi's from being indexed!
Let that be a lesson to me
2 Replies
I've been told by reputable software engineers that the robots.txt file is ignored by most search engines's robot web-crawlers.
Search engines will ignore an improperly constructed, improperly located or otherwise unusable robots.txt file. All of Google, Bing, Duck Duck Go and Yahoo! will honor them (if they can).
As far as I can tell, the Russian (Yandex) and Chinese (Baidu) crawlers do not honor them. A good reason to put all their networks in blacklists…
The response from @normangdrum78 was spam. I reported it.
-- sw