Case Study on Uncrawled URLs in Google Search

In this case study, Adam identified that Google was reporting on URLs blocked in the robots.txt file in the “Valid with warnings” report and the article talks about how he resolved the problem.

Some takeaways from the post:

1. Google can display blocked URLs in the robots.txt in Search Console
2. Google Search indexing system is still using a policy from 12 years ago
3. Removing blocked URLs from Google’s link graph drops it from being displayed in the Search
4. It can take a long time for Google to fully drop the URLs from Google’s search results.