Administrator Guide: Friendly Crawling
ES.NET 2004 includes a feature to prevent hammering Web sites. This
feature will prevent the spider from requesting more than one page per minute
per domain.
If your spider will be crawling the Web, or crawling another Web site than your
own, it’s highly recommended that you enable Friendly Crawling.
<< Back To Table Of Contents
|