I manage a small private forum, and I’ve noticed that sometimes it slows to a stop. In fact, something is hammering on my server and running so much ASP script that my whole website becomes slow. Then I figured out why:

Yahoo Search Robot! It was logged into my forum 11 times. I turned off guest read access, and everything instantly became normal and speedy again. But the Yahoo search server was going nuts. 20 guests (as seen above). Then 30 guests! All getting access denied.

I also turned on robot exclusion on my site (robots.txt and the META tags), but this is not stopping the Yahoo Search Robot from trying to crawl my forum in the most obnoxious possible manner.

Advertisements