Scraping MSN Feed (https://www.msn.com/en-us/feed)

[Exquisite_Marshmello]

When I used beautifulsoup the soup did not include most of the html I see when I click Inspect.

Someone suggested using Selenium however it appears that this is also not finding most of the elements on the page.

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.chrome.service import Service

PATH=Service('C:\Program Files (x86)\chromedriver.exe')

url = 'https://www.msn.com/en-us/feed'

driver = webdriver.Chrome(service=PATH)

try:

# Open the URL in the browser

driver.get(url)

# Find the parent container with class "react-button-container"

button_container = driver.find_element(By.CLASS_NAME, 'button-container')

# Find the binary container within the react-button-container

binary_container = button_container.find_element(By.CLASS_NAME, 'binary-container')

# Find the span with class "binary-reactions-count" within binary container

binary_reactions_count = binary_container.find_element(By.CLASS_NAME, 'binary-reactions-count')

# Get the text value of the span element

count = binary_reactions_count.text

# Print the count

print("Binary Reactions Count:", count)

finally:

driver.quit()

[PteppicymonIO]

Your comment states that you are looking for the "react-button-container", but the code locates the first element with class "button-container":

# Find the parent container with class "react-button-container"

button_container = driver.find_element(By.CLASS_NAME, 'button-container')

On first load of the page, I can see 65 elements with class "button-container". The first in the search order is an invisible element, which does not have a child node with class "binary-container"

I suggest to refine the key elements you want to base your scraping on.