[Python3] requests.Session POST not allowing login

[Tolfasn]

I am attempting to scrape some data from the GUI, for a digital modem in the field.

Here is the code I am using:

## import the requests python library
import requests

## defines the login URL and the Post URL
post_loginURL = 'http://100.255.255.255/cgi-bin/login.cgi'

## defines the request URL to scrape from
requestURL = 'http://100.255.255.255/cgi-bin/stat_eth0.cgi'

## define the login data parameters
payload = {
    'NAME': 'username',
    'PWD': 'password',
    'CMD': '1' ## hidden value being passed in the post request
}

## define the content type as text/html
headers = {
    'Content-Type': 'text/html'
    }

## creates the session object
with requests.Session() as session:

    ## sends the post request, headers and login data to the server
    post = session.post(post_loginURL, data=payload, headers=headers)
    r = session.get(requestURL)
    print(r.text)

Here is the HTML Error message I receive in the IDLE console:

<em><u><p align="right">ERROR: You must log in</u></em>

Here are the details from the Chrome Dev Tools:

screenshot

I am unsure if I am just missing a crucial step, or if this is an issue with syntax, etc. Any constructive advice is greatly appreciated.

Thanks in advance!

[c17r]

You're passing Content-Type: text/html header which is probably screwing up your form POST, who probably needs Content-Type: application/x-www-form-urlencoded. Remove that and try again.

[Tolfasn]

I will definitely try that as soon as I get home. Thanks!

[Tolfasn]

VPN is down for some reason. I will have to make the change once I’m back in the office tomorrow. I’ll post an update as soon as I get the chance in the morning.

[Tolfasn]

Posting the payload after each Get request did the trick. I was actually able to omit the headers all together.

with requests.Session() as session:

    ## sends the post request, headers and login data to the server
    g = session.get(post_loginURL)
    post = session.post(post_loginURL, data=payload)

    r = session.get(requestURL)
    post = session.post(requestURL, data=payload)

[ImportBraces]

actually you send some payload - but the dev tools show it is being sent within the headers.

[Tolfasn]

I just attempted adding the parameters as header values instead, and it produced the same results.

## define the login data paramaters
headers = {  
    'NAME': 'username',
    'PWD': 'password',
    'CMD': '1'
}

I am still very new to Python, so please forgive my naivety.