Blocking 4xx and 5xx Rendering Resources can Reveal Googlebot’s True Server Response

Adam blocked two resources that returned a 404 and 502 Googlebot spent a lot of time crawling each day. Here are some of the things he learned along the way

• HTTP errors bring down average response time
• Blocking client and server errors can “increase” reported avg. response times
• Page load resource higher crawl rate on lower priority sites
• Blocking URLs in robots.txt doesn’t shift crawl budget