Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks


tl;dr Marketing provides a really succinct overview of the essentials, including the latest updates on social media, SEO, paid media marketing and more. This daily newsletter comes highly recommended if you’re looking to protect your focus while still wanting to keep on top of the latest industry trends and changes.

Googles' reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites 1 - Tips and Tricks News

Francesca Chong Head of Content Mention