Use metadata tags to control page indexing and link crawling

You can allow MSNBot or Google to crawl your website and still restrict access to specific web pages and documents by using the noindex and nofollow meta tags within the page code. The tag allows the web page to be retrieved by MSNBot, but blocks indexing of its content. The tag blocks the web crawler from following links in the web page that go to other web pages or documents. Note that not all web crawling robots obey these tags.

If you want to set access and indexing restrictions for your website, replace the user-agent name robotswith msnbotor "*".msnbot in the tag syntax examples below. You can use each tag alone or combine both tags into a single meta tag.