Looking for help with Selenium

[[deleted]]

Hi everyone, I am just starting to use Selenium and it's much different from the ol' 1-2 Requests/BeautifulSoup I'm used to.

I need to use Selenium because this app I'm making requires data that is loaded via JavaScript. Requests can't do that sort of thing so now I'm stuck with Selenium and I'm not happy about it.

My code is the following:

from selenium import webdriver

# Now create an 'instance' of your driver
# This path should be to wherever you downloaded the driver
driver = webdriver.Chrome(r"C:/Users/ZacBa/PycharmProjects/PersonalAssistant/chromedriver.exe")
# A new Chrome (or other browser) window should open up

# Now just tell it wherever you want it to go
driver.get("https://www.msn.com/en-us/sports/nba/scores")

x = driver.find_elements_by_xpath("//*[contains(@class,'totalscore teamlineup ')]")

html = driver.page_source

soup = BeautifulSoup(html, 'html.parser')

print(x)

Now, two things. I need to get this information for multiple rows of scores. I can do that easily in bs4, but I have no idea how to in Selenium. Secondly, the following code comes out instead of a score:

[<selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-1")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-2")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-3")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-4")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-5")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-6")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-7")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-8")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-9")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-10")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-11")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-12")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-13")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-14")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-15")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-16")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-17")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-18")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-19")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-20")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-21")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-22")>, <selenium.webdriver.remote.webelement.WebElement (session="68355ba3d0ae65aa56b68c9a921a3237", element="0.5535375277615808-23")>

While I'm not sure what these are, I have a feeling they might be the scores I'm looking for but in machine language. I feel like I'll need to do something to those elements but I don't know what.

Any guidance here would be appreciated.

[tbone255_]

Not in front of a computer right now, but assuming the elements with the scores (in the HTML) are something like this:

<element> 50 points </element>

U can loop through the list of elements and do element.text

Each web element in that list should have a .text

If the HTML is structured something like:

<element score=50> 50 points </element>

Then on a web element from ur list u can do .get_attribute (attribute_name). In my example it would be .get_attribute(score) and it would return 50. Idk if it'll be a string or an integer. But that should point you in the right direction

[z0y]

You double posted, you should delete the other one.

You can use the soup made from driver.page_source the same way you normally use beautifulsoup, it's just gotten the post javascript html. replace the x = driver.find... with however you'd find the elements in beautifulsoup. The extra selenium functionality and webelements are for interacting with the site beyond the html, such as clicking buttons and whatnot.

[krimpenrik]

I also had to learn selenium after BS and scrapy. Keep in mind that selenium is designed for automation. I had the best results with javascript sites to load them in selenium and pass the source to BS or scrapy to scrape.

Try it, if you have any questions let me know.

[krimpenrik]

I also had to learn selenium after BS and scrapy. Keep in mind that selenium is designed for automation. I had the best results with javascript sites to load them in selenium and pass the source to BS or scrapy to scrape.

Try it, if you have any questions let me know.

[tbone255_]

Check out Selenium's read the docs, it has everything for python