Saijo George

Curated by Saijo George

Read more

thursday27 Feb 2020

Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites

https://youtu.be

If some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.

Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.

Tips and Tricks

As an agency director, staying on the pulse of changes in the digital landscape is fundamental to our success. Unlike traditional industries, we can't rely on the methods that worked for us in the past as the pace of change and transformation in digital dictates constant investment in education. Saijo and the tl;dr feed/newsletter is an invaluable source to anyone in digital as it constantly provides value and keeps busy digital talent reliably up-to-date. Make reading it a weekly habit and you'll understand why tl;dr marketing is your secret weapon in digital and why you should consider Saijo a good friend to have in your network.

Senior SEO Specialist / Audience Strategist

Full Time 100% Remote

Company : 10up

10up is a fully distributed, global creative services agency; partnering with some of the biggest brands in the world.  This role at 10up sits on our Audience and Revenue team.…