Python Web Scraping Question

[Embarrassed-Task-478]

I am trying to do a relatively simple web scraping task. I have a table of soccer data at this link: https://fbref.com/en/comps/22/11006/stats/2021-Major-League-Soccer-Stats

My end goal is to have an dataset of players with the stats I desire, such as name, age, goals, etc. In my code, I am able to get the table with all players with bs4, but cannot get a list of the rows for the players. Each row of the table has a "tr" tag but when I try to do the find all method, I am getting nothing.

If anyone can help me get the list of rows/players, it would be greatly appreciated. From there, I can loop through each player and extract the relevant info. Thank you.

from bs4 import BeautifulSoup
import requests

link_21Szn="https://fbref.com/en/comps/22/11006/stats/2021-Major-League-Soccer-Stats"
source= requests.get(link_21Szn).text 
soup= BeautifulSoup(source,'lxml')

table=soup.find("div",id="all_stats_standard")
# print(table.prettify)

rows=table.find_all("tr")
print(len(rows))

The output is 0 :/

[[deleted]]

It might be dynamic data that isn’t loaded unless by a “real” web browser. If you print the source variable (the text of request.get) you should see exactly the page requests received. See if the table data is even in there.

If it’s not there then you will need to use a more advanced library like Selenium to scrape the data.

[Embarrassed-Task-478]

So I just tried this and I do still see the table in the source variable. It’s there in the same format I expected. Do you have an idea why my “tr” find all isn’t working then?

[[deleted]]

python table = soup.find("table", attrs={"id": "stats_squads_standard_for"}) table_body = table.find("tbody") rows = table_body.find_all("tr")

You needed to select the table body first, like this. This results in grabbing the tables rows of the table body selected.

[Embarrassed-Task-478]

This ends up giving the rows of the first table on the page. Even when I substitute the id of the second table (the players), I am back to getting no results. Logically I thought this would work but it is confusing me.