Google Had Mentioned Permitting Noindex In Robots.txt

Google’s John Mueller responded to a query on LinkedIn to debate the usage of an unsupported noindex directive on the robots.txt of his personal private web site. He defined the professionals and cons of search engine help for the directive and supplied insights into Google’s inner discussions about supporting it.

John Mueller’s Robots.txt

Mueller’s robots.txt has been a subject of dialog for the previous week due to the overall weirdness of the odd and non-standard directives he used inside it.

It was virtually inevitable that Mueller’s robots.txt was scrutinized and went viral within the search advertising and marketing group.

Noindex Directive

All the pieces that’s in a robots.txt is named a directive. A directive is a request to an online crawler that it’s obligated to obey (if it obeys robots.txt directives).

There are requirements for how you can write a robots.txt directive and something that doesn’t conform to these these requirements is prone to be ignored. A non-standard directive in Mueller’s robots.txt caught the attention of somebody who determined to publish a query about it to John Mueller by way of LinkedIn, to know if Google supported the non-standard directive.

It’s a very good query as a result of it’s straightforward to imagine that if a Googler is utilizing it then possibly Google helps it.

The non-standard directive was noindex. Noindex is part of the meta robots normal however not the robots.txt normal. Mueller had not only one occasion of the noindex directive, he had 5,506 noindex directives.

The search engine marketing specialist who requested the query, Mahek Giri, wrote:

“In John Mueller’s robots.txt file,

there’s an uncommon command:

“noindex:”

This command isn’t a part of the usual robots.txt format,

So do you suppose it’s going to have any affect on how search engine indexes his pages?

John Mueller curious to find out about noindex: in robots.txt”

Why Noindex Directive In Robots.txt Is Unsupported By Google

Google’s John Mueller answered that it was unsupported.

Mueller answered:

“That is an unsupported directive, it doesn’t do something.”

Mueller then went on to elucidate that Google had at one time thought-about supporting the noindex directive from inside the robots.txt as a result of it might present a method for publishers to dam Google from each crawling and indexing content material on the identical time.

Proper now it’s attainable to dam crawling in robots.txt or to dam indexing with the meta robots noindex directive. However you may’t block indexing with the meta robots directive and block crawling within the robots.txt on the identical time as a result of a block on the crawl will stop the crawler from “seeing” the meta robots directive.

Mueller defined why Google determined to not transfer forward with the thought of honoring the noindex directive inside the robots.txt.

He wrote:

“There have been many discussions about whether or not it ought to be supported as a part of the robots.txt normal. The thought behind it was that it might be good to dam each crawling and indexing on the identical time. With robots.txt, you may block crawling, or you may block indexing (with a robots meta tag, when you enable crawling). The concept was that you would have a “noindex” in robots.txt too, and block each.

Sadly, as a result of many individuals copy & paste robots.txt recordsdata with out taking a look at them intimately (few folks look so far as you probably did!), it might be very, very straightforward for somebody to take away essential components of a web site by chance. And so, it was determined that this shouldn’t be a supported director, or part of the robots.txt normal… in all probability over 10 years in the past at this level.”

Why Was That Noindex In Mueller’s Robots.txt

Mueller made clear that it’s unlikely that Google would help that tag and that this was confirmed about ten years in the past. The revelation about these inner discussions is attention-grabbing however it’s additionally deepens the sense of weirdness about Mueller’s robots.txt.

See additionally: 8 Frequent Robots.txt Points And How To Repair Them

Featured Picture by Shutterstock/Kues