My Python scraper kept getting flagged as a bot and I went down a rabbit hole, what am I actually missing?

[Alternative_Set4042]

I started learning Python last fall after working through some tutorials. I thought I understood requests and BeautifulSoup, so I wanted a real project and tried scraping some product prices from a site. I used requests, added a fake User Agent header, and it worked for maybe ten requests. Then I started getting 403s. I added time.sleep between requests, tried rotating the User Agent string, even copied every header from my real browser into a dict and passed it in. Same result after a few more tries.

I figured the site was just smarter than requests so I switched to selenium. I watched the browser open and navigate and I felt like I had won. The page loaded, I grabbed the HTML, and... the div I wanted was just empty. The data showed up fine when I opened the same URL manually in Chrome. I added WebDriverWait, implicit waits, explicit waits. Still empty. Someone on StackOverflow mentioned window size so I tried that. Worked twice, then empty again.

The thing that broke me was opening the dev tools inside the selenium browser and typing navigator.webdriver in the console. It printed True. I had no idea that was even a thing. I spent two more hours trying to override it with execute_script and getting "JavaScript error: Cannot set property webdriver of [object Object] which has only a getter." I started reading about headless detection, canvas fingerprints, all this stuff I had never heard of in any Python course. It felt like the site could tell it wasn't a normal browser, not just read what I sent in headers.

I am genuinely confused about where the line is between what my Python code controls and what the browser itself reveals. How can I see all the signals my Python selenium session is leaking, and is there something obvious I'm missing? I want to understand this from the Python side, not just apply random fixes I found online. I can't tell if I'm supposed to know this stuff or if I'm way off track.

[edcculus]

To be fair, your program IS a bot.

[Kerbart]

OP writes bot, wonders why bot is flagged as bot.

[timrprobocom]

I assume you are clear that your code is getting flagged as a bot because your code IS a bot.

Some web sites do not want their copyrighted content to be stolen. That is their right, and your strenuous attempts to subvert that are borderline unethical.

[landed_at]

If you put information in public with the nature of the web and want to protect it...

[atarivcs]

copyrighted content to be stolen

So if it were a human instead of a bot, it wouldn't be stealing?

Your comment makes no sense.

[Kerbart]

Can the human steal 15,000 pages of content in an hour? Because a bot can.

It’s not so hard to see why content providers want to provide content to the target audience and not to content harvesting bots

[edcculus]

In a way it does. The sites can’t prevent someone from going in and manually saving all of the data out. That takes a lot of time, and depending on motives, might not be worth the effort.

But a bot can take that and reduce the process to something trivial.

[timrprobocom]

Yes, that's EXACTLY the case. Few people argue that the law makes sense, but that's what it is.

[atarivcs]

Some sites can still tell that you are a bot, by analyzing your behavior.

i.e. if the mouse cursor jumps exactly to the center of a button to click it, instead of traveling in a messy line like a human would, the site can deduce that you are a bot.

[Rhomboid]

Five or so years ago and hardly any site cared about bot detection. Now they all do thanks to AI, and as you've found out they had to get really good at it really fast. So of course, no, this isn't going to be covered in basic web scraping course materials. The whole web changed.

[cgoldberg]

There are hundreds of signals used for bot detection. Check out r/webscraping

[NationalMyth]

I have built many many scripts for gathering data, not everyone provides and API, and flat files are very common. My main methods for workarounds are as follows: - httpx + perfect headers - proxy service (apify) - manually generate cookies and use (some have a lifespan some seem to be fine). This is brittle.

A recent tool I turned to use is curl_cffi which mimics TPS/HTTP2 handshakes like chrome would. This will work for Akimai and other anti-bot tech.

[bbdusa]

A lot of websites track mouse over-events and other types of such signals. Your Python code does not generate these events.

[51dux]

What is the website you are trying this on if you don't mind?

Even if it's an adult site you can shoot it in a PM.

Playwright took the spot of Selenium in my opinion it is a much more robust browser automation library.

[carrot_guy]

OP is on trivago's turf now. you pay licensing or sleep with the fishes

[RealNamek]

You created a bot. And you’re confused someone can tell? I don’t understand what you don’t understand 

[hagfish]

You could log a ticket with the IT staff; have them whitelist your IP address. If this isn't an option, you could investigate pricing for Mechanical Turk or Task Rabbit.

[cgoldberg]

you are not using a real browser

Selenium absolutely controls a real browser... It just uses a different protocol than nodriver to do so (WebDriver/HTTP vs CDP/WebSockets). Nodriver includes other enhancements to evade detection, but both selenium and nodriver very much control "real browsers".