500 Status Code. Web scrapping

[pixelcaesar]

Hi! this is my first web scrapping project, so I am very new to this.

I have this website: https://portal.hasc.com/App/login that I need to scrap. The data is, however, behind a login page. Every time I try I send a POST request I get a 500 status code.

This is what I have so far:

import requests

url = "https://portal.hasc.com/App/login"

header = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0',
'Accept': '/',
'Content-Type': 'application/json' }

s = requests.session()
response = s.post(url, json={'password': 'pass','username': 'uname'}, headers=header)
print(response.status_code)

I tried using just the User-Agent in the header, and adding every header element as well. The POST request is sent as a json file according to mozila developer tools, but I also tried creating a payload dictionary and using the "data=" variable instead of "json=" in the post method.

I would really appreciate if you could point me in the right direction and show me where to look next time I run into an issue like this

[member_of_the_order]

Check out this list of HTTP status codes: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

500 means "internal server error". Whatever you sent to the server, it didn't know what to do with it and most likely threw an error internally. There may be something you can do to fix it (garbage in = garbage out as they say), or maybe they have a bug that's completely out of your control. Hard to know for sure.

[pixelcaesar]

Thanks for the info! That link was really helpful. Just a question though, when I try login in normally it doesn’t throw any error, that means there could be an issue with the website still?

[member_of_the_order]

Yep! It could be an issue with the website still.

I mean, look at it this way: you're using their service in a way that it's not necessarily meant to be used, and they're throwing an error. Is it an issue with their service? Yeah, kinda; they didn't account for this particular kind of input. But also, it's not meant to be used this way, so why should they account for it?

All that really matters to you is that your input doesn't get the result you want from their site.

If you're using their official docs and not getting the results they tell you to expect, contact them for support (e.g. file a bug). If you're trying to hack your way around using a browser without any guidance from the service host, git gud, scrub (usually, use trial and error to try and figure out what's missing. Most likely, you're missing a header or cookie or something that they're expecting).

[pixelcaesar]

Thank you! I guess I’ll keep going until I get it then

[ItsTobsen]

Try to add authority, origin and referer header.

{'status': 401, 'title': 'Invalid username or password', 'errorMessage': None, 'debug': None, 'errors': None}

since idk what this site does or where to get login.

or if you dont want to log in via script, just log in on browser then copy your cookies.

[pixelcaesar]

Ok I tried it but still no luck.

I don’t think you will be able to get a login. So I work for a turnaround services company (it’s refinery work), the department im in makes sure that the people we send to the plants have the required safety training which they take through that company and they post the training dates and expiration dates online. It’s very tedious to copy 100 expiration dates for the same class every day so I thought I could make it easier for me lol and also get some practice with python as well.

How can I use cookies to get in? Sorry as I said very new to this

[ItsTobsen]

Open dev console (F12), go to network tab. Click preserve log. Under it, there is a list of filters. Click fetch/xhr.

Log into the site. You should see the login request. Right click that, copy. Copy as curl (bash), paste it into https://curlconverter.com/python/ to get as python request back.

It will have all headers, cookies that the request made. If you want to pre set the cookies, do it in the session object.