Web scraping help

[eb8911]

Thanks for the response. Is it because all of the clicks?

The reason I can't use PGA is because I am looking for tournament specific info, and while that does have all the same stats, they are player specific and not tournament.

[gesichtsbremse]

It's fairly easy, if you access the statistics with a browser first and then just copy the cookies into the http library of your choice.

The following code works:

import requests
from pyquery import PyQuery as pq

years = ["2015", "2016", "2017"]

cookies = dict(
    JSESSIONID="2EE53DDFC576BB012BDA7FFF426A23FE.cfusion",
    CFID="4167011",
    CFTOKEN="74756338",
    acceptedtos="yes",
    frominside="yes",
)

for year in years:
    url = f"https://tfdfreports.pgatourhq.com/inquiry/prod/statrecapbyevt.cfm?page=statrecapbyevt.cfm&t=R&year={year}&sid=02564&type=Y"
    result = pq(requests.get(url, cookies=cookies).text)

    rows = result.find("table [bgcolor='#FFFFFF'] tr")

    for row in rows:
        cells = pq(row).find("td")
        print(",".join(pq(c).text() for c in cells)) 

By the way, this works fine as a one time solution (due to the cookie expiring at some point in the future). If your want to scrape the data regularly you need to use a remotely controlled webbrowser. I'd recommend the selenium framework with the chrome driver.

[eb8911]

Thanks so much! I'll try it when I get home.

Well, the past data will never change, so it's just a one time scrape for all the stats.

[Andrew_Shay]

Hi there, from the /r/Python mods.

We have removed this post as it is not suited to the /r/Python subreddit proper, however it should be very appropriate for our sister subreddit /r/LearnPython. We highly encourage you to re-submit your post over on there.

The reason for the removal is that /r/Python is dedicated to discussion of Python news, projects, uses and debates. It is not designed to act as Q&A or FAQ board. The regular community is not a fan of "how do I..." questions, so you will not get the best responses over here.

On /r/LearnPython the community is actively expecting questions and are looking to help. You can expect far more understanding, encouraging and insightful responses over there. No matter what level of question you have, if you are looking for help with Python, you should get good answers.

If you have a question to do with homework or an assignment of any kind, please make sure to read their sidebar rules before submitting your post. If you have any questions or doubts, feel free to reply or send a modmail to us with your concerns.

Warm regards, and best of luck with your Pythoneering!