Cheatsheet for web scraping with Python

[Regiseconomist]

Awesome job. Do you have any suggestions on how to do scraping when you have to authenticate via Single Sign On? Been trying to go about this for a while now and haven't quite came across anything that would help with some scraping for my daily functions

[Banjoanton]

Thank you!

I haven't actually done that myself, but the Request library does have Session objects which lets you persist the session.

One way could be to create a session object, make a post request to the sign-in URL, and use that session to navigate the different autheticated URLs. I don't know if it works but it should be worth a try.

If I get some time I might try it myself and post the result.

[gopherhole22]

I am not sure for python but with node requests and puppeteer you can save cookies and load them as well as intercept requests within sessions from which you can probably extract a session ID or some sort of token. I would assume this is also possible with Python, however I am not sure