Limit crawl frequency

If you occasionally get high traffic from MSNBot, you can specify a crawl delay parameter in the robots.txt file to specify how often, in seconds, MSNBot can access your website. To do this, add this syntax to your robots.txt file:

User-agent: msnbot
Crawl-delay: 120

When you contact us about an issue, include the following information so that we can help you more quickly:

• The target website address that MSNBot put in the robots.txt file
• The date range when the issue occurred
• The access logs