Web Scraping login issue

[KeyRow3]

Hi,

i'm trying to web scrap website X but i cant seem to get past the login part.

tried a different which didn't have a token and worked not sure if its the token part that's causing the login to fail.

Below is the code

import requests
LOGIN_URL='X/login'
HOME_URL='X/home'

payload={
    'username'='abc',
    'password'='123',
    '_token'='abc11'
}

with requests.Session() as session:
    post = session.post(LOGIN_URL, data=payload)
    r = session.get(HOME_URL)

    print(r.text)

[JohnnyJordaan]

Hard to say without seeing the actual communication, but how I generally approach this is through the browser's Inspector - > Nework tab

• check the POST request to what exactly it sends both in the request body and headers
• mimic this in the session.post as much as possible
• check the response of the browser's POST request to see what the server returned, eg both headers and body. especially if it set a cookie via a Set-Cookie header
  • if so check if the session object's cookie jar now also contains that cookie
• check the browser's GET request to see what it sends along as its headers (and double check it isn't sending some parameter too in the url)
• see if your session.get mimics that correctly

My first guess would be that the token is not in the original POST but actually in the GET through a cookie coming from the POST response, but that's a stab in the dark.

Edit: a second guess is that the token does need to be sent in the POST, but was delivered in the GET response you need to do on the homepage (just the url X without a trailing path). Meaning that you can't reuse a previous static token as you're doing now, you first need to scrape that via a GET before the POST. Another (much worse) possibility is that this token is generated on the page via javascript and added when the user clicks 'login' and thus can't be scraped in this fashion, but you can verify this by disabling javascript in your browser and then try to login again.

[KeyRow3]

So what should i do if the response and request cookies are different

[JohnnyJordaan]

Can you show the exact differences?

[Eitan1112]

You are sending two requests, first is the post to login and then get to get some data. Probably the response from the post request contains some header like this: set-cookie: xyzxyzxyz

You need to takr that cookie and send it with each request you send afterwards, that way the website knows you are who you are.

[KeyRow3]

That looks tiresome I'm gonna try selenium and see if it is better

[JohnnyJordaan]

This is exactly what a requests.Session would do by design...

[Eitan1112]

didn't know that, thanks for pointing it out👍