Blocking 4xx and 5xx Rendering Resources can Reveal Googlebot’s True Server Response
https://gentofsearch.comAdam blocked two resources that returned a 404 and 502 Googlebot spent a lot of time crawling each day. Here are some of the things he learned along the way
- HTTP errors bring down average response time
- Blocking client and server errors can “increase” reported avg. response times
- Page load resource higher crawl rate on lower priority sites
- Blocking URLs in robots.txt doesn’t shift crawl budget