Blog Advertising - Get Paid to Blog

Saturday, April 26, 2008

Restrict indexing and link crawling within your website

You can block MSNBot from crawling specific file types linked to your website by specifying MSNBot as the user-agent for a Disallow tag that specifies the file types to exclude.
To do this: Use this syntax: Examples
Restrict MSNBot from indexing specific file types

User-agent: msnbot
Disallow: /*.[file extension]$


(the "$" is required)

User-agent: msnbot
Disallow: /*.PDF$
Disallow: /*.jpeg$

Disallow: /*.exe$

No comments:

 
ss_blog_claim=636b3226b9752525e0b4d3dc79b20607 6 27