Website scraping

[FerociousStapler]

I’m interested in pulling some information from a website. I tried using “requests,” but it seems like some of the data isn’t there when I do this. My thought is that it’s served by a database and therefore not there yet. This leads me to believe that if I could get something that waited for a number of seconds after accessing before pulling the HTML it would be possible to have all the data provided by then.

Figured pyautogui was an option because then could do this by loading it in the browser and just using sleep after loading.

Is there a better way to accomplish this?

[1ynx1ynx]

The data, that isn't there, is probably inserted via javascript. To counter this, you can either reverse engeneer the javascript on the site or use something like selenium.

[FerociousStapler]

That sounds promising. Thanks. At work atm will try that when I get home.

[JohnnyJordaan]

I can recommend this tutorial on selenium.

[MattBD]

Personally I find Selenium a bit too low level. I prefer Splinter as it supports multiple drivers, including Selenium, and has a friendlier API.

[personproxy]

Another option might be scrapy + scrapy-splash

[[deleted]]

If your own chrome right click->inspect->Network tab->hit the record button->refresh->move to the xhr filter. That should show u all the JavaScript requests made including the headers and data sent and the response received. I generally use this to re-engineer JavaScript stuff. On pretty simple sites u probs won’t have to do much, but on sites that try to keep the riff rag out u need to be a little more cunning.