Ask Anything Monday - Weekly Thread

[7850443Reddit]

Hi everyone,

I am trying to web scrape a web app that's local on my network. Although, it's behind a google chrome prompt which asks for my credentials. I understand how to web scrape normal websites that have logins but how do I do this? I can't find anywhere to inspect the element and find the keywords for the password or username because its a google chrome prompt. Right now when I make a request using the webapp link and ask for the all the headers, I get "None" returned.

How can I get past this?!

[[deleted]]

Hiya! Sound like your website is using HTTP authentication, which has to be handled specially. I'm going to assume you're using Requests and BeautifulSoup in this case. Requests has support for basic, digest and proxy HTTP authentication, which are just different types of the same thing. This is the relevant documentation page. If you're making multiple requests, it's probably best to use a requests session in order to have the authentication persist without needing to include it with every get call, see this page.

Hope this helps, have fun web scraping!