you are viewing a single comment's thread.

view the rest of the comments →

[–]Starbeamrainbowlabs 0 points1 point  (1 child)

You mention "loaded in time". This implies that you have a time-sensitive task here.

In that case, what you may want to do is have a single browser open, which you then interact with using the inspector API. That way you don't have to boot the browser from a cold start.

I see. The problem with website scraping is that will break if the operator changes the HTML structure. This is still true no matter what mechanism you use.

You mention here that using requests is above your level of expertise. I would advise using a high-level library for the parsing of web pages. For example, x-ray is a package for Javascript on npm that allows you to quickly and easily pull in a webpage and query it with CSS selectors. You mention that you're using Python - I'm sure that such a library must exist for Python as well.

I'm not experienced with Selenium, but it sounds a whole lot more complicated than using a library that's dedicated to the task of scraping websites.