Matt Cutts has just put out a posting on how search engines handle the “noindex” metatag. He makes some interesting observations about the results as follows:
- Google doesn’t show the page in any way
- Ask doesn’t show the page in any way
- MSN shows a URL reference and Cached link, but no snippet. Clicking the cached link doesn’t return anything.
- Yahoo! shows a URL reference and Cached link, but no snippet. Clicking on the cached link returns the cached page.
Matt goes on to note that it would be great if all the search engines treated the noindex metatag, well, like a noindex request (translation, “DON’T index it” – my words, not Matt’s). I think that this is an excellent idea.
Danny Sullivan, still of Search Engine Watch, adds his thoughts to the noindex discussion. Danny observes that Google treats the robots.txt file in a way that many webmasters may not expect. In particular, Google may still index pages that are marked as “don’t crawl” in the robots.txt file, may still be indexed by Google.
The one case I have seen where this happens is when pages are marked in robots.txt as “disallow”, yet third party pages link to those pages. In the past, Google has reserved the right to still index these pages.
In fact, Google may still index a page even if it is marked with the noindex metatag. This can happen if it is also listed in the robots.txt file as “disallow”. The reason is that Google treats the robots.txt as being a higher priority than the noindex metatag, and ignores the metatag if the page is also called out in the robots.txt file.
Now it turns out that this is pretty simple to address. All you need to do is not use the robots.txt file, and rely solely on the noindex metatag. In this event, Google will reliably not index the page, in all cases (that I know of).
Danny repeats a call I have seen him make in the past for improved standardization of search engine behavior. More standardization by the search engines sounds like another excellent idea. We all recognize that there are things that the search engines need to keep close to the vest and treat as proprietary. But making life easier on SEOs and search engines by following agreed upon standards will benefit us all.
You can read more about metatags and SEO here.
Eric Enge leads the Digital Marketing practice for Perficient Digital. He designs studies and produces industry-related research to help prove, debunk, or evolve assumptions about digital marketing practices and their value. Eric is a writer, blogger, researcher, teacher, and keynote speaker and panelist at major industry conferences. Partnering with several other experts, Eric served as the lead author of The Art of SEO. Learn More About Eric Enge