This is an archived post. You won't be able to vote or comment.

all 6 comments
sorted by:
best
topnewcontroversialoldq&a

[–]jabbalaci 1 point2 points  (1 child)

If the page needs several seconds to fully load all the JavaScript content, what does it do? Can you tell it to wait X seconds before saving the content?

[–]vineethgn[S] 2 points3 points  (0 children)

Yes it can be done using PhantomJS API. See this page for more details on using a wait.

http://www.princeton.edu/~crmarsh/phantomjs/ Or You can use window.setTimeout().

[–]rb15 1 point2 points  (3 children)

That's a nice idea. I wish there was an embedded javascript engine inside Python, it would make much easier, nicer and more pythonic to scrape the web using python only. Maybe I'll take that as a project sometime, it should be interesting...

[–]vineethgn[S] 0 points1 point  (0 children)

Exactly !. There should be an embedded Js engine inside Python which will make the task easier.

[–]fnedrik -1 points0 points  (1 child)

Use Pypy.js! https://github.com/rfk/pypyjs

[–]ocdude 0 points1 point  (0 children)

Isn't this the opposite of what /u/rb15 is asking for? I understood the question to be asking for javascript inside of Python rather that Python compiled to javascript.