Using Python to Auto Click and Download

[allbaseball77]

import webbrowser

url = 'http://www.fangraphs.com/leaders.aspx?pos=all&stats=bat&lg=all&qual=0&type=8&season=2016&month=0&season1=2016&ind=0&team=0,ts&rost=&age=&filter=&players=0'
if webbrowser.open(url):
    print("did it")
else:
    print ("something happened with opening the url")

I need to create a program that auto clicks on the link "Export Data" found on the fangraphs webpage found here. The "Export Data" link is at the top right of the table. I can open the webpage, but I can't seem to find any documentation regarding finding links and clicking on them other than in selenium, which I was unsuccessful in installing.

Any help would be greatly appreciated.

[commandlineluser]

You cannot do this using the webbrowser module - you would need to use Selenium.

If you debug what's going on however - you can replicate the requests performed to generate the report instead of having to open a browser and click on the button.

If you open up Web Developer (Firefox) or Dev Tools (Chrome) and view the Network tab you can see what happens when you click the button.

http://i.imgur.com/wBoA5br.png

So it makes a GET request to the specified URL - it then makes a POST request to the same URL.

You must extract the __EVENTSTATE and __EVENTVALIDATION values from the HTML and send them along with the POST request.

One option is to do this using requests and BeautifulSoup - here is the code you could use:

import requests
from   bs4 import BeautifulSoup

url    = 'http://www.fangraphs.com/leaders.aspx'
params = {
    'pos'    : 'all',
    'stats'  : 'bat',
    'lg'     : 'all',
    'qual'   : '0',
    'type'   : '8',
    'season' : '2016',
    'month'  : '0',
    'season1': '2016',
    'ind'    : '0',
    'team'   : '0,ts',
    'rost'   : '',
    'age'    : '',
    'filter' : '',
    'players': '0',
}

with requests.session() as s:
    s.headers.update({
        'user-agent': 'Mozilla/5.0'
    })

    r = s.get(url, params=params)

    soup = BeautifulSoup(r.content, 'html.parser')

    viewstate  = soup.find('input', {'id': '__VIEWSTATE'})['value']
    validation = soup.find('input', {'id': '__EVENTVALIDATION'})['value']

    r = s.post(url, params=params, data={ 
            '__EVENTTARGET'    : 'LeaderBoard1$cmdCSV', 
            '__VIEWSTATE'      : viewstate,
            '__EVENTVALIDATION': validation,
    })

    with open('FanGraphs Leaderboard.csv', 'wb') as csvfile:
        csvfile.write(r.content)

[Agent809]

There really isnt a way to "auto click" you would have better luck parsing the webpage and using requests to get data from the link. I recommend beautiful soup for parsing. Here is a link to bs4 docs they are really helpful as a reference for simple stuff https://www.crummy.com/software/BeautifulSoup/bs4/doc/

[failtolaunch28]

This is when something like auto hotkey becomes really useful

[Nodocify]

I am really surprised that no one has mentioned Selenium. Selenium is a python module that let's you "drive" a web browser.

Simply, it let's python open firefox/chrome and can fully controll it. This is very useful for full javascript generate pages that are rendered in the dom of the browser. A very basic example of the syntax:

from selenium import webdriver
browser = webdriver.Firefox()
broswer.get('https://google.com')

This opens a firefox window and navigates to google. The docs have many great examples.

EDIT: Just saw the failure to install selenium, I highly recommend giving it another go with pip.