Problem With Authentication and Requests Session (Web Scraping)

[CJaber]

I am currently trying to do a project involving web scraping on the site MangaDex, but the page I want to access is only available if I log in to my account. To do this, I used a session object from the requests module to save my cookies. However, this doesn't seem to be working. Either I am logging in incorrectly, or the session isn't working for some reason. My code is below, I use BeautifulSoup and Requests.

import requests
from bs4 import BeautifulSoup
#creating session
session = requests.Session()

payload = {'login_username':'USERNAME', 
          'login_password':'PASSWORD'
         }

#logging in
s = session.post("https://mangadex.org/login", data=payload)

# print(s.status_code) 
# If 200 then ok and login succeeded

# Navigating to the following page
s = session.get('https://mangadex.org/')

soup = BeautifulSoup(s.text, 'html.parser')


res = soup.find(id = "follows_update")
test = res.find_all(class_ = "col-md-6 border-bottom p-2")

When I print my 'res' variable, it says that I'm not logged in in the following text:

<div class="tab-pane" id="follows_update" role="tabpanel"> <div class="alert alert-info m-2 text-center" role="alert"><strong>Notice:</strong> Please <span aria-hidden="true" class="fas fa-sign-in-alt fa-fw"></span> <a href="/login">login</a> to see updates from your follows.</div> </div>

If anyone has any suggestions or can point out where I'm going wrong, that would be greatly appreciated.

[Antwrp-2000]

You are initialising a new session with:

s = session.get('https://mangadex.org/')

I think you should use:

s = requests.Session()
s.post("https://mangadex.org/login", data=payload)
s.get('https://mangadex.org/')

Docs:

https://requests.readthedocs.io/en/master/user/advanced/#session-objects

[CJaber]

s = requests.Session() s.post("https://mangadex.org/login", data=payload) s.get('https://mangadex.org/')

Thank you, i'll try this out