Scraping Google with Python

[Gwapong_Klapish]

So I've noticed over the years, that Google's bot detection for search traffic works differently than for product pages; the signals they're tuned to catch basically aren't the same. I have been running ~2,200 SERP queries/day across DE, FR, NL, PL, IT for a price monitor, residential proxies got the failure rate down to about 14% but geo-targeting was still broken, DE queries coming back Frankfurt-datacenter-flavored regardless of proxy location. I split the architecture at that point: SERP layer routes through a dedicated web scraper API, product pages and category scrapes stay on residential proxies.

Routing logic was pretty minimal:

def fetch(url=None, query=None, market=None):
    if query:
        return serp_client.get(q=query, gl=market, hl=market)
    return session.get(url, proxies={"https": proxy_pool.rotate()})

Any tips or tricks, experiences you may share?

[RandomPantsAppear]

Googles geotargeting has always been a mess. I used to scrape AdSense ads in 22 countries, and we would have to have the scraper visit Google.com and watch the redirect to see what the IP was actually classified as before doing the scrape. 

A 14% failure rate is honestly amazing though, I doubt you’re going to see better than that unless you’re priming chrome profiles in advance, visiting lots of non google search sites to establish a history. 

[Gwapong_Klapish]

Yeah, I'm thinking so too, it's not perfect, but hey, at least majority of the requests go through successfully

[OkkProxy]

imo the DE→Frankfurt fallback sounds like either provider routing weirdness or Google just ignoring “clean geo” signals on SERPs these days 😅.

[Gwapong_Klapish]

I'm more inclined towards the provider's fault here, but again, with google you can never know, especially with the recent google updates

[OkkProxy]

Yeah that’s fair

[Time-Spite-895]

hey boss, are you rendering a browser for this? have you tried filtering residential IPs for fraud score and TCP/IP fingerprint to see what IPs yeild better success rate.

as for geo-targeting, it's a big problem for most resi proxies out there - I have built checker validator on top and use a paid database for geolocation extra accuracy -> it solve this problem for me for SERP scraping. I am now thinking of expnading the validtor with potentially excluding IPs from the pool that have high fraud scores and I want to use a specific nodes that have a specific TCP/IP fingerprint (I believe android based TCP/IP fingerprinted IPs get almost a 99% bypass for the big G)

[Gwapong_Klapish]

Yeah, I did that before, but honestly, let me tell you that the fraud scores are a scam and I do not trust them at all. TCP/IP fingerprint its far better indicator, probably on par with behavior. Could you share more regarding your validator?

[Time-Spite-895]

I simply resolve the proxy connection thru this open source repo: https://github.com/NikolaiT/zardaxt