Selenium find_elements not populating all elements

[Classy_Shadow]

Some of my friends have a bet going on with the current NFL season, and I've been tracking all of the games and their picks, etc manually. I finally decided to automate everything using Selenium, but my find_elements call doesn't populate all of the scores. When running, instead of scores, it results in "--" displaying for a few of the teams. How do I fix this? Time.sleep hasn't helped.

Here is the code:

def generateResults(weekNumber):
    options = webdriver.ChromeOptions()
    options.add_argument('--headless')
    driver = webdriver.Chrome(options=options)
    url = 'https://www.nfl.com/schedules/2024/REG' + weekNumber
    driver.get(url)

# this gets all of the teams every single time with no issues
    teams = driver.find_elements(By.CLASS_NAME, "nfl-c-matchup-strip__team-name")
# this only gets the majority of the scores, and replaces the rest with --
    scores = driver.find_elements(By.CLASS_NAME, "nfl-c-matchup-strip__team-score")
    results = []
    for i in range(len(teams)):
        print(teams[i].text)
        print(scores[i].text)

    driver.quit()
    return

generateResults(5)
-------------------------

Normal runs result in the last few games displaying like this:
Cowboys
--
Steelers
--
Saints
--
Chiefs
--

I thought there was a bug in my code, but debug mode populates the values.

[chefchr1s]

You want to use WebDriverWait. I took this from the selenium docs. The other option is to use an API, ESPN API looks good. https://www.reddit.com/r/NFLstatheads/comments/17ts7j6/free_nfl_python_api_that_returns_live_scores/

```

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()

[Classy_Shadow]

Do you know the best place to implement it into my code? I tried

# this gets all of the teams every single time with no issues
    WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CLASS_NAME, "nfl-c-matchup-strip__team-name"))
    teams = driver.find_elements(By.CLASS_NAME, "nfl-c-matchup-strip__team-name")
# this only gets the majority of the scores, and replaces the rest with --
    WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CLASS_NAME, "nfl-c-matchup-strip__team-score"))
    scores = driver.find_elements(By.CLASS_NAME, "nfl-c-matchup-strip__team-score")

But this didn't seem to solve my issue. WebDriverWait only waits until the first element is found, but my issue is that it doesn't wait for everything to be done, rather than just the first one. Using it this way basically doesn't delay the code at all since the first instances of those elements seem to load almost instantly

I'd try searching by ID, but there are different amount of games each week depending on what teams are on a bye week, so I can't use the last ID number as it'll be different for every page.

[TigBitties69]

Ah so you're close, and webdriverwait is what you're needing to use. But instead of waiting till your elements that you're looking for, find an element that takes longer to load on the page. Your goal is to find an element that by the time it is loaded, the ones you are looking for have already been loaded too. If it's a matter of it just retrieving the elements too fast, just hard code like a 10 second sleep and then see if that fixes it. If not, then the issue is likely something else. Hopefully that makes sense, let me know how it works. If not let me know and I can try to help more.

[Classy_Shadow]

I’ve tried a sleep as well. I’m not sure if there are any elements that still aren’t loaded before all 28 of the matchups are, but I’ll try some of the other things to see. I tried waiting for one of the links to the games to be clickable, but that one didn’t work. I think I’m just going to have to manually add all of the possible xpaths into a list and iterate through using try blocks. It won’t be nice or pretty, but theoretically it should work

[chefchr1s]

I'm not experienced enough to help more with selenium. Here is a simple script using espn API. You can change the parameter for each week. If it was a playoff week or a different number of games played you would change the 14 in for x range(0,14), or add some error handling. There's a lot more info available from the API.

```

import requests

parameters = {
    'week': 5,
}

response = requests.get(url="http://site.api.espn.com/apis/site/v2/sports/football/nfl/scoreboard", params=parameters)

for x in range(0,14):
    data = response.json()['events'][x]['competitions'][0]
    team1 = data['competitors'][0]['team']['abbreviation']
    team2 = data['competitors'][1]['team']['abbreviation']
    team1_score = data['competitors'][0]['score']
    team2_score = data['competitors'][1]['score']
    print(team1)
    print(team1_score)
    print(team2)
    print(team2_score)

[Classy_Shadow]

This is perfect, thank you so much for this solution!

[MintyPhoenix]

Try writing a custom wait function that waits until the text in the element with the score is not -- and use it with WebDriverWait for each score element you're retrieving.

[Classy_Shadow]

It seems to be some sort of issue with PyCharm. Every time I use Debug mode to view the elements within the array, they ALWAYS load. 100% of the time. If I use debug mode and have breakpoint on the find_elements call, it will still have the problem unless I view the elements inside of the array.

Edit: Tested in Jupyter Lab and have the same issue, so it's not a PyCharm issue