.Google's John Mueller answered an inquiry concerning why Google marks web pages that are refused from crawling through robots.txt and why the it is actually safe to ignore the related Look Console documents about those creeps.Robot Visitor Traffic To Query Parameter URLs.The person inquiring the inquiry chronicled that crawlers were creating links to non-existent inquiry criterion URLs (? q= xyz) to web pages along with noindex meta tags that are actually likewise blocked out in robots.txt. What caused the inquiry is that Google.com is actually crawling the hyperlinks to those webpages, getting blocked out through robots.txt (without seeing a noindex robots meta tag) after that obtaining reported in Google Browse Console as "Indexed, though shut out by robots.txt.".The person asked the adhering to inquiry:." Yet right here is actually the major concern: why would Google mark webpages when they can't even view the web content? What is actually the advantage during that?".Google's John Mueller validated that if they can't creep the page they can not find the noindex meta tag. He likewise makes an interesting mention of the web site: hunt operator, recommending to disregard the results because the "common" users won't find those results.He created:." Yes, you're correct: if our company can not creep the webpage, we can't see the noindex. That claimed, if our experts can not creep the web pages, then there is actually not a whole lot for our team to index. Thus while you might observe a few of those web pages along with a targeted web site:- inquiry, the normal customer won't see them, so I would not fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it merely means the Links will definitely wind up being actually crawled (and wind up in the Search Console file for crawled/not catalogued-- neither of these conditions create issues to the remainder of the web site). The essential part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limitations in using the Web site: search progressed hunt operator for analysis reasons. One of those main reasons is given that it is actually not linked to the frequent hunt mark, it is actually a different trait completely.Google.com's John Mueller talked about the web site search driver in 2021:." The brief answer is actually that a web site: inquiry is actually not implied to become comprehensive, neither utilized for diagnostics objectives.A website inquiry is a particular type of hunt that restricts the results to a particular website. It is actually essentially merely words site, a bowel, and afterwards the internet site's domain.This inquiry limits the results to a details internet site. It's certainly not suggested to be a complete collection of all the web pages coming from that site.".2. Noindex tag without utilizing a robots.txt is actually fine for these kinds of situations where a bot is actually linking to non-existent pages that are acquiring found out by Googlebot.3. URLs along with the noindex tag are going to generate a "crawled/not catalogued" item in Browse Console and also those won't possess a negative effect on the remainder of the internet site.Read through the inquiry and answer on LinkedIn:.Why would Google mark webpages when they can not even view the web content?Included Graphic through Shutterstock/Krakenimages. com.