You can block MSNBot from crawling specific file types linked to your website by specifying MSNBot as the user-agent for a Disallow tag that specifies the file types to exclude.
To do this: Use this syntax: Examples
Restrict MSNBot from indexing specific file types
User-agent: msnbot
Disallow: /*.[file extension]$
(the "$" is required)
User-agent: msnbot
Disallow: /*.PDF$
Disallow: /*.jpeg$
Disallow: /*.exe$
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment