In a nutshell
You can use a special HTML <META> tag to tell robots not to index the content of a page, and/or not scan it for links to follow.
For example:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
There are two important considerations when using the robots <META> tag:
* robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.
Don't confuse this NOFOLLOW with the rel="nofollow" link attribute.
The details
Like the /robots.txt, the robots META tag is a de-facto standard. It originated from a "birds of a feather" meeting at a 1996 distributed indexing workshop, and was described in meeting notes.
The META tag is also described in the HTML 4.01 specification, Appendix B.4.1.
The rest of this page gives an overview of how to use the robots <META> tags in your pages, with some simple recipes. To learn more see also the FAQ.
How to write a Robots Meta Tag
Where to put it
Like any <META> tag it should be placed in the HEAD section of an HTML page, as in the example above. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.
What to put into it
The "NAME" attribute must be "ROBOTS".
Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW". Multiple comma-separated values are allowed, but obviously only some combinations make sense. If there is no robots <META> tag, the default is "INDEX,FOLLOW", so there's no need to spell that out. That leaves:
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Comments
Post a Comment