Saijo George

Curated by Saijo George

Read more

thursday9 Jun 2022

Crawl-delay: Instructions in robots.txt may cause unexpected crawling behaviour from Googlebot

https://twitter.com

This is something you might want to make a note of, when Google comes across Crawl-delay: instructions in robots.txt they ignore it but not it can group lines which may cause issues based on what rules follow that instruction.

SEO


I love tl;dr Marketing because I can get all the latest SEO news and trends in one spot without having to read lengthy articles. I really look forward to the daily emails to see what's new in our industry!

Crawl-delay: Instructions in robots.txt may cause unexpected crawling behaviour from Googlebot 1 - SEO News

Ryan Mews SEO Manager Merkle