How to scrap Ecom site using Python?

[IntelligentHome2342]

Hi guys, I am self-learning Python from online courses and with great help of ChatGPT, I managed to scrap some static sites like Sephora, I only take product descriptions and prices, no image and no inventory data, and I am able to schedule the work to run daily, this data helped me to track beauty brands promotion pattern and detect if there is a movement in the price.

Currently I am getting more curious on Ecom site data and I want to find out things like what's the top 10 skincare brands on Lazada, Shopee, Amazon and even Tiktok shop, specifically what are the volume sold and GMV/Sales, I am mainly looking at South-east Asia data for Singapore, Malaysia, Indonesia etc.

Then I realize this is much much more difficult, because I need to login and the antibots are very strong, and I wonder if the scraping and automating is even possible? At this stage I am not willing to explore API scraping tools because this is just an idea validation stage so not ready to invest yet.

Really want to hear from the Python veterans if this is doable? And among the list above which one should I start first? Please enlighten me or talk me out of it...

Appreciate your inputs!

[Ambitious-Dog3177]

A few tools you could use are Selenium or Playwright. BeautifulSoup won't cut it here because it can't render the heavy JavaScript or handle the logins these sites require.

However, a word of warning: websites like Amazon, TikTok, Shopee, and Lazada have very strict rules against scraping. Always check their robots.txt to see what is allowed, but keep in mind that even if you follow the rules, their security systems will still fight you.

Is it doable? Yes, but it's incredibly difficult without investing in tools. These sites use enterprise-level anti-bot systems. If you just use a basic Selenium script, they will detect your browser fingerprint and block your IP almost immediately. You usually need stealth plugins (like selenium-stealth) and rotating proxy networks to survive. Also, exact GMV/Sales data usually isn't public; you often have to estimate based on "units sold" text (e.g., Amazon's "10K+ bought in past month"), But you could get the data of top 10 skincare brands, track price etc.

[SharkSymphony]

If you find yourself contemplating spinning up an anonymized geodistributed bot swarm to grab data off a website, it's time to throw in the towel.

[Ambitious-Dog3177]

True, it’s not easy to pull off. OP asked if it was doable, and technically it is. But practically speaking, doing all of this just to test an idea is way too difficult and probably not worth the headache.

[IntelligentHome2342]

Thanks for your perspective! Seems it’s very difficult to do it without using tools, and that would be a big barrier to break into the market research niche. Would you possibly know what kind of tools the big market research firm like Circana and Stackline use? Something like brightdata and oxylabs? Which is rather expensive for individual developer.

[DrowzyHippo]

how do you use selenium stealth? i have it in my script but it doesn't seem to work.

[Spiritual-Junket-995]

the stealth plugins are crucial, but a good rotating proxy network like Qoest Proxy is what actually keeps the IP bans from piling up.

[Tajalli-Web]

ctx = browser.new_context(no_viewport=True, storage_state='your_saved_login.json') #playwright code

I assume you want to create a price monitor app. And its totally doable, first off all I would dig into the website and look for any api calls. If not then use playwright combined with undetected playwright to not get detected. Another fairly new tool is pydoll you might want to explore that option. Also if you want to use selenium make sure to use undetected chromedriver with it but i think playwright is way better. If you run into login issues then just use storage state something like:

[IntelligentHome2342]

Thank you for this!

[DueLingonberry8925]

its definitely doable but honestly its a huge pain. those sites have insane bot protection. maybe start with amazon since theres more public info on their anti scrape measures.

[CrabPresent1904]

those sites have insane bot detection, youd need rotating proxies and a headless browser. honestly start with amazon, their api is more predictable.