Googles’ reverse IP check can be easily circumvented by X-Forwarded-For Header Response for Some Sites
https://youtu.beIf some sites are blocking your Googlebot user-agent crawls (Screaming Frog or other crawlers) you might be able to circumvent it by passing off a known Google IP using the X-Forwarded-For Header.
Props to Aarne Salminen for the find, I tried this using Screaming Frog and it does work for some sites. Aarne believes this is a limitation of the AWS platform. Enjoy it while it lasts.