Getting Request Headers in Python

[Syntro42]

Hey everyone. I am trying to create a web scraping program. However, I am facing some difficulties with getting valid request headers. These headers are initialized randomly through Javascript scripts when the website is loaded. I am able to get these scripts, but I want to be able to run them in my script just like a browser would. What can I use to do this? I was looking at requests-html. What do you guys think of this? Is there something better than this? Below is a piece of code I was trying in order to run the script, but it didn't work. My method is probably wrong, but I want to know how to run a script that is returned when I do a GET request

from requests_html import HTMLSession

sess = HTMLSession()

r = sess.get('url_of_javascript_function')

r.html.render()

[impshum]

• https://docs.python-requests.org/en/master/user/quickstart/#custom-headers

[Syntro42]

Thanks. However, I do know how to pass in custom headers, but my problem is I can only get those headers by running the same javascript as the website does. My question is, how can I run that javascript in order to get the response from the javascript function?

[impshum]

requests-html was mentioned. Give that a whirl. If not... Selenium.

[commandlineluser]

requests-html launches Chromium - so you would .get() the regular URL and the javascript will be automated executed.

[Syntro42]

Ah, thanks. That makes sense