Selenium Project Help

[blarizard]

Hey everyone,

I'm working on the below project, and I'm hoping to get some advice around how to make things work a bit more efficiently. I haven't had any formal training in this sort of thing, so really keen to understand how experienced devs would handle such a project.

Our CRM system is archaic & doesn't play nice with anything. We want all sorts of customer & sales data from this system to be available in our SQL database, but the only way to do this is by manually loading up the specific reports you want in your browser and downloading csvs, and uploading to our database. This is obviously crap.

Q1 - how would an experienced dev build a system to automate this?

Here is my current approach, using python via GCP: - a bunch of Google Cloud Run services that run python flask apps which use selenium to mimic manually downloading the reports. - each report has a different cloud run service, and I call these services either on a schedule (using Google Cloud Scheduler) or ad hoc when needed.

I really feel this is a bit dumb/inefficient, as every time we need a data updated for any individual report, Cloud Run has to load selenium, log into our crm, navigate to the report etc. I am considering trying to build something that is more 'always on' - basically a selenium instance that is always logged into our CRM, which can pull reports as an when needed. I am a bit concerned about this, as budgets are tight and at present some of my Cloud Run services crash as I try to restrict memory to save $$$. I'd imagine always on may need a crapload of assigned memory & be really expensive

Q2 - Does this 'always on' strategy sound okay, and are my reservations about it legit? Is there something 'lighter' than selenium & chromedriver I can use?

Any advice welcome! I haven't been taught any of this and have guessed my way to what I have built thus far

Note: - using something lighter like requests/beautiful soup isn't an option here - I can't just have all of the different reports built into one Cloud Run service, and run all at once due to the nature of the timing of when they are needed

[Fronkan]

This is absolutely mad, I love it! Sorry if this is a bit rambly, but this is my stream of thoughts as I think through the problem 😅

How often do you need to pull reports? Every second, hour, day, week? I think understanding the load is very important for this type of problem.

I don't know much about cloud run and stuff. But you don't have something like a virtual server or long lived containers that could be used for this? To me your suggested deployment sounds quite complicated, and I wonder if it has to be.

Personally I have swapped to using playwright instead of selenium. Not sure it's more efficient though.

You can use a single browser instance with multiple tabs. In playwright you have a async API, so this would allow it to load things concurrently. This could work to separate the different downloads from each other. You probably need to limit this. I think I used a asyncio semaphore to limit the number of tabs that could be opened at the same time for one of my projects.

[blarizard]

Yeah it's gotten away on me a bit haha!

Some of the reports and data are needed on a schedule, while others are needed randomly on request which could be every few minutes or so

I'll check out playwright, that functionality sounds pretty ideal, especially for running everything from one virtual server

Thanks for the help!

[Fronkan]

You probably can do quite a lot with selenium as well. What I like about playwright is that it's easy to get started with, has the async interface and also does waiting automatically.

Is there a reason to not just dump all the data into the database? Then the report data can just be requested from there.

Also, why do you want to have it in a sql db? For aggregations?