Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks


Saijo's tl;dr marketing newsletter has a perfect balance. You are sure to get the best SEO news without being overwhelmed by them.

Samsung Galaxy S21 Provides Google Discover as the Default Feed Now out of the Box 1 - SEO News

Jean-Christophe Chouinard Sr. SEO Specialist Seek