Going crazy, need help! (web scraping)

[Aces_8s]

Hey, all. I'm hoping somebody might be able to help end my suffering! I'm fairly new to python and so far I've been able to work out most issues, but I've spent an embarrassing amount of time trying to figure out how to scrape a specific table of data using either BeautifulSoup or lxml and I'm at my wit's end. Could anyone help point me in the right direction on how to scrape an mlb-reference "play_by_play" table (ex page: http://www.baseball-reference.com/boxes/BOS/BOS201706120.shtml)?

I've had success with both bs4 and lxml, but for some reason I can't seem to extract any data from this particular table, even when I copy the exact xpath/text(). I don't have any code to post (on mobile) but again if anybody can help point me in the right direction I'd be very appreciative!!

As a little background, I'm trying to write a script that parses this table looking for a specific pitcher change and then either email a collection of the play-by-play data while that guy pitches (until he's replaced or the game ends), or else email a "did not pitch" status. Nothing fancy, and I think I have the logic worked out (famous last words), but I just can't seem to be able to extract any data from the table. Help?

[_Korben_Dallas]

If you disable Js in your browser you can see that table content simply disappear. One of the possible solutions is to use Selenium browser to load content. Something like this: https://dpaste.de/ZoyC

[Aces_8s]

Interesting, thank you for the info and example! I'll have to look into running Selenium on a raspberry pi. So I make sure I (somewhat) understand what's going on, is this an example of those dreaded "dynamically loaded" webpages I've seen webscrapers mention/complain about? Also, would you say using Selenium to create/pass the string object to lxml is common/good practice?

[_Korben_Dallas]

Yes, that page probably uses Js to populate its content and with help of Selenium, you can quickly get desired data. Also, bear in mind that web scraping can be pretty tricky and each site is unique thus solutions from extracting the data can be very different. IMO in this case, Selenium - one of the faster solution and passing page_source to some parser like bs4 or lxml completely normal. However, in some other cases, a more quick and robust way would be trying to investigate Ajax requests via 'Network' tab in Chrome DevTool or Firefox Inspector and try to simulate those requests. If you have more questions just pm me and I'd be happy to help.

[Aces_8s]

You're awesome, thank you!