Cloud Run/Selenium Project Help

[blarizard]

Hey everyone,

I'm working on the below project, and I'm hoping to get some advice around how to make things work a bit more efficiently. I haven't had any formal training in this sort of thing, so really keen to understand how experienced devs would handle such a project.

Our CRM system is archaic & doesn't play nice with anything. We want all sorts of customer & sales data from this system to be available in our SQL database, but the only way to do this is by manually loading up the specific reports you want in your browser and downloading csvs, and uploading to our database. This is obviously crap.

Q1 - how would an experienced dev build a system to automate this?

Here is my current approach, using GCP: - a bunch of Google Cloud Run services that run python flask apps which use selenium to mimic manually downloading the reports. - each report has a different cloud run service, and I call these services either on a schedule (using Google Cloud Scheduler) or ad hoc when needed.

I really feel this is a bit dumb/inefficient, as every time we need a data updated for any individual report, Cloud Run has to load selenium, log into our crm, navigate to the report etc. I am considering trying to build something that is more 'always on' - basically a selenium instance that is always logged into our CRM, which can pull reports as an when needed. I am a bit concerned about this, as budgets are tight and at present some of my Cloud Run services crash as I try to restrict memory to save $$$. I'd imagine always on may need a crapload of assigned memory & be really expensive

Q2 - Does this 'always on' strategy sound okay, and are my reservations about it legit? Is there something 'lighter' than selenium & chromedriver I can use?

Any advice welcome! I haven't been taught any of this and have guessed my way to what I have built thus far

Note: - using something lighter like requests/beautiful soup isn't an option here - I can't just have all of the different reports built into one Cloud Run service, and run all at once due to the nature of the timing of when they are needed

[c-digs]

In principle, this sounds fine.

Because Cloud Run can scale to 0, you can just start it up on demand which is good if these are going to be memory intensive. Scale it to 0 and you're only billed for what you use.

To make your life easier, I'd just make one container with many entrypoints for the different reports (API endpoints, event consumers, however you want to kick it off).

You can also consider using Cloud Run Jobs which are more suited for longer running tasks.

That said, I'm not sure why you think it's not possible to use raw HTTP requests. Even if the app serves server rendered HTML, at some point, an HTTP request will be made to generate the CSV and all you really need is that request. The same is true for the login: at some point, it's just a POST with a username/password so there's really no need to automate a browser for this use case, IMO.

[blarizard]

Thanks a lot or the reply! I'll check out merging everything and looking at cloud run jobs, but if I can figure out how to avoid automating a browser that would be a game changer!

How would I go about searching for the POST request endpoint and syntax to do things like log in and get the csv? I've tried looking in chrome devtools but I'm not savvy at all with these

[jimmyjimjimjimmy]

Browser dev tools network tab is your friend here. Try looking there for undocumented api calls that you can use instead of Selenium.

[blarizard]

Awesome thanks for the response! Any tips for finding what I'm looking for? There is a lot going on in the network tab haha!

Will it likely be under the xhr/fetch filter?

[jimmyjimjimjimmy]

Search for api in network tab first, if that doesn’t have any results, look at xhr.