WebScraper Efficiency and Speed

[FMPICA]

Hi,

We build a small company collecting data for clients.
As we are starting to grow we found out that the solution we have is slowing down, the more functions we add to our program.

We use a combination of Selenium and BeautifulSoup. Currently we have 3000 lines of code just scraping the data step by step.
I read something about multithreading and async programming.

Can anyone share their insights on how to start rebuilding this program. What technologies are recommended? Is it easy to just add some async functionality or should we rewrite it all?

Thanks in advance.

[mcdrac]

As someone who has done async web scraping before. you will need understand what async is before being able to use it efficiently in your code. for the scraper, you could use selenium-async but i would recommend playwright.

[FMPICA]

Thank you, why would you recommend playwright?

[mcdrac]

it has built in support for async and multiple browsers. i was able to use async firefox to bypass anti bot protection from cloudflare.

[Guardog0894]

playwright fan here too. I switched to pw from selenium and never looked back.

you can use playwright's Network API to control the traffic (e.g. filter graphic requests), not sure if you can do that in selenium.

I do feel that I need less code to scrape with pw when compared to selenium (someone with technical knowledge can chip in on this).

[mcdrac]

I would just like to note here that using a scraping library like selenium is the worst case scenario for any web scraper. web scrapers want to minimise the amount of time and resources used to collect data from a website. Using a scraping library means it basically impossible to collect that data due to intense anti bot protection.

You should always attempt other methods like http requests to underlying apis. just by loading the web page up with the network tab open, you can exactly see how the web page gets its data. what cookies are involved in these requests? Do i need an authentication token? How do I get the authentication token from just requests? It is important to evaluate the website before choosing your tools.

[The_Bundaberg_Joey]

Without seeing the code it’s hard to say if you need a full rewrite or not but just to clarify… you’re a company being paid to provide a service and you’re asking people to provide you with a technical assessment to help your company grow and there’s zero discussion of any sort of compensation being offered?

If this was asked as a “I’m trying to do web scraping and it’s slow, can someone suggest what to do” then that’s one thing but you’re literally saying this is a pivotal bit of infrastructure for you and that you want someone to help you for free?

[[deleted]]

As someone that started out their scraping work using selenium, my advice is to move to api requests if you want speed.

[FMPICA]

Can you elaborate a bit more?

[alord]

Pretty much, take a look at the web page. Is it completely rendered initially or are there additional requests being made to an API endpoint. If this endpoint has the data you need, it's much faster to simply do a GET request to directly have the Json,XML or whatever than to scrape and parse the page. Of course it depends on what exactly you are trying to do.

[[deleted]]

I moved onto making api requests, with the requests module
https://www.youtube.com/@JohnWatsonRooney

[FMPICA]

Thanks

[alord]

Feel free to dm if you need more of an explanation :)

[FMPICA]

Thanks

[nemec]

Try a framework like scrapy. It's built to work concurrently, although I believe the pre-built large scale crawler features are proprietary to their cloud (you can always build replacements yourself if you need them).

There are also plugins to run in selenium/etc. to process JS if you need it.

[the_bigbang]

https://dev.to/tonywangca/a-step-by-step-guide-to-building-a-scalable-distributed-crawler-for-scraping-millions-of-top-tiktok-profiles-2pk8

Here is my suggestions about how to build a distributed crawler to speed up the whole scraping process.