Meta robots tags are used to pass instructions to search engines on how they should index or crawl sections of a website. This article gives an overview of the different values that can be used in a meta robots tag and what their purpose is.
Meta robots tag values
The following parameters or values can be used on their own, or combined with others in the content attribute of the tag (comma separated). They will specify how search engines should interact with a page.
all
Shortcut for follow, index and assumed to be the default on all pages. It does nothing when specified.
follow
Allows search engines to follow links on the page to find other pages and is assumed to be the default on all pages. This parameter normally does not have to be specified.
index
Allow search engines to start website indexation by submitting needed page to Google index. This parameter normally does not have to be specified.
noarchive
Search engines may not show a cached copy of this page in search results when this value is used.
nocache
Only used by MSN/Live as an alternative to noarchive.
Search engines robots may not follow any links on a page that uses this parameter.
Some search engines apply this to all links, while others only to internal links for seo.
noindex
Prevents search engines from adding the page to the index, resulting in it not showing in search results.
noimageindex
Prevents search engines from adding images to the index, although images can still be indexed if they are linked to directly from elsewhere.
none
A shortcut for nofollow, noindex.
nosnippet
Prevents search engines from displaying video or text snippets of the page in search results, and also prevents a cached copy to be shown. Snippets may however still display an image thumbnail, unless noimageindex is added.
notranslate
Prevents search engines from displaying translations of the page in search results.
noyaca
Prevents Yandex Directory page descriptions from being shown in search results snippets.
noydir
Prevents Yahoo from using the Yahoo directory description as the snippet for the page in search results. This tag is deprecated, but might still be found occasionally.
Unavailable_after
Instruct search engines not to show the page in search results after the date/time specified in RFC850 format (e.g., Sunday, 08-Jan-18 16:43:34 UTC).
Robots.txt files
Meta robots tags don’t work the same as instructions in a robots.txt file, and conflicting rules could cause undefined behaviors. Search engines will for example not be able to read meta tags if the page is blocked by the robots.txt.
Avoid defining conflicting values in a meta robots tag, e.g. using both noindex and index parameters. Where parameters are in conflict, the most restrictive interpretation is normally used, i.e., “show” is beaten by “don’t show”.
