Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks


Since I discovered tldr; marketing, I unsubscribed to other SEO newsletters. Why? It is the only one covering the most important news daily. And on top of this, I love the design of the email and website.

SEO & SEM Specialist - Start Digital

Full Time Perth

Company : Start Digital

Start Digital are a small, innovative web and digital agency looking for the brightest and friendliest minds to help us and our clients grow. We work with a wide range…