Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks


I love tl;dr Marketing because I can get all the latest SEO news and trends in one spot without having to read lengthy articles. I really look forward to the daily emails to see what's new in our industry!

You Can Now Listen to Local News on Google Assistant and Google Podcasts 1 - SEO News

Ryan Mews SEO Manager Merkle