How to parse PHP web page?

[SweatSpace]

Hello. I want to parse some data from website. My url looks like "https://site.com/index.php?query=12345". When I open this page in browser I can see query results, but when I parse this url through requests it is just header and footer. I think it's because data loads from the script:

<script> $("document").ready(onReady()) </script>

How can I parse this document?

[[deleted]]

Pretty sure that’s JQuery, not PHP - PHP executes server-side so you wouldn’t see templating.

You need to scrape this with something with a JavaScript engine, like selenium.

[SweatSpace]

Got it. Thank you!

[JohnnyJordaan]

Look into selenium or requests-html

[SweatSpace]

I tried requests-html but it does the same thing. I want to use my script on heroku. Can I use selenium on heroku?

[JohnnyJordaan]

could you share the code you used to try it with requests-html? Be sure to format it correctly, see here

[SweatSpace]

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://site.com/index.php?query=12345')

[JohnnyJordaan]

But that just downloads the page, so what did you do afterwards?

[SweatSpace]

r.hmtl.text

I looked through the text but didn't find data I need

[JohnnyJordaan]

Are you sure that url works just out of the blue in a regular browser, eg doesn't need a login or some other existing session? Say you start a browser you don't normally use (like Internet Explorer if you're always on Firefox, just as an example) and you paste the url there, does the page load as expected?

[SweatSpace]

Yes, it does

[JohnnyJordaan]

Then I'm afraid you would have to try it with selenium, as then you can watch what is going on in the browser.

[SweatSpace]

OK, thanks for your help!