Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks


tldrMarketing has really impressed me with the useful information being shared and also the use of emerging technology - web stories. It was already becoming a great source of information for me, and sharing news in new engaging ways has made it my first web stop for SEO news.

Googles' reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites 1 - Tips and Tricks News

Jess Joyce Search Consultant

Prosperity Media SEO Consultant

Full Time Sydney

Company : Prosperity Media

Prosperity Media is a well known SEO Agency based in Sydney. The thing about our company is we are not trying to do everything. We specialise in SEO and mid to large…