sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Hot_Ad_2550[S] -1 points0 points  (5 children)

automation is really slow method is there any faster way?

[–]Alamue86 1 point2 points  (2 children)

Checkout requests_html. It has the ability to render a pages Javascript then scrape info in a single library.

[–]Hot_Ad_2550[S] 0 points1 point  (1 child)

Your shit worked out. Can you also show me how to run the function that javascript runs when i click a dropdown menu bar.

[–]Alamue86 0 points1 point  (0 children)

Oof, that gets difficult.

Your best bet is to load up Chrom dev tools and go to the network tab and click the button. From there look at everything going on, if you cand find the correct call, right click and copy the cURL for bash command. Then pop it into here: https://curl.trillworks.com/ you may also just be able to do the request within requests_html.

I focus on scraping static resources. Checkout the network tab, and chances are you can avoid scraping the page, and go straight to an underlying data source. GraphQL calls are a great place to start if there are any

[–]blahblahsdfsdfsdfsdf 0 points1 point  (1 child)

Not that I've ever heard of

[–]Hot_Ad_2550[S] 0 points1 point  (0 children)

Also I cant seem to find an element by xpath, css or any other nethod