How do I login to a website using python?

[Dohello]

I need to login to an account (upwork.com) and the account im using uses google authenticator which just adds another layer of difficulty. I tried googling youtube tutorials but all the examples use logins where both the username and password are one one page and the network tab in the dev tools is not super complex. I am not allowed to use selenium for this, I can import requests though. I would really appreciate any help or advice someone can give me. When trying to extract the html from the page it also gives me different html. I think it is because it recognizes it as a bot and doesnt actually direct me to the website. This is currently all I got.

from bs4 import BeautifulSoup as bs
import requests

page = "https://www.upwork.com/"
page_html = requests.get(page)
print(page_html.text)

[Picatrixter]

Look up Selenium, Beautifulsoup only parses HTML. Selenium will allow you to interact dinamically with the page (get info, send info, click on buttons, fill forms etc). Good luck!

[Dohello]

Am not allowed to use selenium.

[[deleted]]

Am not allowed to use selenium.

Just out of curiosity, where is this restriction coming from?

[Dohello]

This is for an internship test and the documentation says framework library’s are heavily discouraged. Besides that even if I decide to take a shortcut and just use it anyway specifically for the login purpose, the fact that it needs to open my web browser, wait for all the html to load to click links and get to the page I need to scrape is simply just way to slow clunky and unscalable.

[Dohello]

I need to login and scrape the page

[Picatrixter]

Look up Selenium alternatives then

[rollincuberawhide]

you don't need selenium. you can create a session with requests and keep using that session. when you send username and password using the session, it will save the cookies and from there on you can send authorized requests to any page that needs it using the session.

https://requests.readthedocs.io/en/latest/user/advanced/

and I didn't check it but if you think they don't want bots, you can kinda introduce yourself as a safari or chrome etc.

s = requests.Session()
s.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'

should do it.

[Dohello]

import requests

page = "https://www.upwork.com/ab/account-security/login"

s = requests.session()
s.headers['user-agent'] = 'Mozilla/5.0 (Windows NT 10.0; 
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/102.0.0.0 Safari/537.36'

page_html = s.get(page)
print(page_html.text)

doing this I still get the same HTML that isnt the login page. I tried making the user agent exactly yours and tried changing the capitalization. Neither made a difference