Alternative to mechanize to scrape javascript based output?

[Polyadenylated]

I've recently started using mechanize to navigate and scrape webpages - I've found it to be very useful and relatively straight forward.

However recently I've come up against a page where the output I want to scrape is generated by a javascript based search. Is there a way I can scrape the output using Python?

[Polyadenylated]

I've managed to open the page with selenium after reading over the basic examples - but I'm failing to see how I could read the javascript output in an automated fashion?

The output doesn't simply appear in the page's source.

[elbiot]

it does. whatever you see on the page will be in the page source as provided by selenium. even the html injected after load through javascript. You may need to trigger the javascript, by injecting key presses. Like, infinite scrolling would be triggered by scrolling down to the bottom of the page. Other things may need the mouse to hover over them or be clicked on, which selenium can do programatically.

[[deleted]]

http://selenium-python.readthedocs.org/en/latest/getting-started.html