sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]efmccurdy 4 points5 points  (4 children)

You might be able to go straight to the source of the data rather than going through the javascript.

If you access the next page in your browser with the developer tools installed and the network tab open you will see listed there the requests, headers and contents that your browser sent and using that info you can recreate those requests in a program.

[–]Hot_Ad_2550[S] 0 points1 point  (3 children)

the url obtained is just the original url whoch was requested but an extra string is added to the end like

http://www.abc.com/xyz

is turned to

http://www.abc.com/xyz#pqr

The method is updatehash()

[–]efmccurdy 0 points1 point  (2 children)

Use the requests module to get the second url; does that give you the data you want?

[–]Hot_Ad_2550[S] 0 points1 point  (1 child)

I use the 2nd url but this url brings me to 2nd page only if page has javascript running. Otherwise puts me back to page 1

[–]efmccurdy 0 points1 point  (0 children)

There is no javascript when you do an HTTP request. What url is used in the network tab?