.Google's John Mueller responded to a question concerning why Google indexes webpages that are refused coming from crawling through robots.txt and also why the it's risk-free to ignore the associated Browse Console documents regarding those creeps.Crawler Traffic To Question Criterion URLs.The person inquiring the inquiry recorded that robots were producing links to non-existent concern guideline URLs (? q= xyz) to webpages with noindex meta tags that are actually likewise shut out in robots.txt. What motivated the question is actually that Google is crawling the web links to those pages, acquiring obstructed through robots.txt (without envisioning a noindex robots meta tag) then obtaining shown up in Google Search Console as "Indexed, though obstructed through robots.txt.".The individual asked the observing question:." Yet right here is actually the huge question: why would Google.com index pages when they can not even view the content? What's the benefit because?".Google.com's John Mueller verified that if they can not creep the page they can't observe the noindex meta tag. He likewise creates an exciting reference of the web site: search operator, suggesting to dismiss the end results due to the fact that the "typical" individuals will not see those end results.He wrote:." Yes, you're correct: if our team can not creep the webpage, our experts can not view the noindex. That claimed, if our company can't crawl the pages, after that there's certainly not a whole lot for our company to mark. Therefore while you could observe a few of those webpages along with a targeted site:- query, the ordinary customer will not see them, so I wouldn't fuss over it. Noindex is actually additionally alright (without robots.txt disallow), it simply means the Links will end up being crawled (and end up in the Look Console file for crawled/not listed-- neither of these conditions create concerns to the remainder of the internet site). The important part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limitations being used the Website: search accelerated hunt driver for analysis reasons. Among those reasons is given that it is actually not hooked up to the normal search index, it's a different point entirely.Google's John Mueller talked about the website search driver in 2021:." The short response is actually that an internet site: question is certainly not meant to become full, nor used for diagnostics objectives.A web site question is actually a details type of hunt that restricts the outcomes to a certain internet site. It's primarily merely words website, a bowel, and then the website's domain.This query confines the results to a specific site. It's not meant to become a comprehensive selection of all the webpages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is great for these type of conditions where a bot is actually connecting to non-existent webpages that are actually acquiring found through Googlebot.3. Links along with the noindex tag will generate a "crawled/not recorded" entry in Explore Console and also those will not have a negative effect on the remainder of the internet site.Read the concern and answer on LinkedIn:.Why will Google mark webpages when they can not also find the information?Included Image by Shutterstock/Krakenimages. com.