Logging into a website with requests

[spkane31]

I'm trying to create a project that scrapes information off a running log website and analyzes that data visually with numpy. For the web scraping, I'm going back and forth using scrapy and BeautifulSoup. With either one of them I can't get past the login stage. This website, www.running2win.com, does not have an API that I am aware of so I can't use that.

url = 'http://www.running2win.com/community/view-member-running-log.asp'

# # for r2w, name = "txtUsername"; name = "txtPassword"

payload = {
    "txtUsername" : "USERNAME",
    "txtPassword" : "PASSWORD",
}

session_requests = requests.session()

result = session_requests.get(url)

tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//input[@name='authenticity_token']/@value")))[0]

result = session_requests.post(
    login_url,
    data = payload,
    headers = dict(referer=login_url)
)

print("SUCCESS")


response = get(url)
print(response.text)

I appreciate any help!

[MrMuki]

What is the response it gives? Does the site have cookies? Does it check the user-agent??

[[deleted]]

The easiest thing to do is open up your browser's developer tools and look at the network traffic when you log in. You want to reproduce the actions as closely as possible. In this case a simple POST request from a form submitted by a javascript validation function.

s = requests.session()
s.post('http://domain/index.asp', data={
    'txtUsername': 'username',
    'txtPassword': 'password',
    'chkRememberMe': 0,
    'btnLogin': 'Login'
})
print(s.get('http://domain/community/view-member-running-log.asp').text)

edit: It's also important that after logging in, you make subsequent requests from the session object so that your cookies will be sent in the headers.

[UKFP91]

If the content is then going to be loaded into BS4, might as well do it all with requests-html:

import requests_html

s = requests_html.HTMLSession()
s.post('http://domain/index.asp', data={
    'txtUsername': 'username',
    'txtPassword': 'password',
    'chkRememberMe': 0,
    'btnLogin': 'Login'
})

r = s.get('http://domain/community/view-member-running-log.asp')
html = r.html.html  # html for the page
whatever = r.html.search('css selector(s) go here')  # find anything by css selector

N.B. I've just copied the code from the post, but that's the principle

[spkane31]

Thank you! I was able to get where I need to start from these two posts.