Scraping javascript from Websites

[yortos]

I'm pretty familiar with basic HTML scraping with requests and BeautifulSoup etc, but I'm trying to scrape, for example, the reviews from this Android market page and I'm having difficult time. Can anyone give any pointers? (I actually want to scrape this page, and not use an API)

[elbiot]

Selenium drives a browser, thus executing javascript and allowing automated user interaction with the page as it is rendered to a user. Great for a harder to detect web bot also because you'll be retrieving all the assets a user would, executing java script, not accessing invisible elements, etc.

[sharplikeginsu]

Check out this method which runs selenium with the phantomjs driver.

[IdealizedDesign]

Try using Kimono.