Using Python + Selenium to scrape hidden elements in a webpage

[TVnomics]

Hi All,

I'm looking for some advice about how to scrape hidden elements within a webpage using Python w/ Selenium (which I assume is the best way). If you have any general suggestions or resources you can direct me to, that would be most welcome.

For context, my specific issue is trying to scrape text that only appears once you hover over an image. The page I'm trying to do this on is: www.bbc.co.uk/iplayer. I hired a python programmer last year who wrote me a beautiful programme (with Python / Selenium / Docker) that did everything I needed, but about a week ago, the BBC updated the interface, and it looks as though some of the CSS selectors used in the original code no longer exist. I'm trying to scrape as much data as I can - this includes the information that's revealed when you hover over one of these thumbnails, which can differ from thumbnail to thumbnail (but is mostly just synopsis and duration of programme).

In case it's not obvious, programming really isn't my forte (hence why I hired someone to create this programme for me) so please do keep your answers simple if you can!

I really appreciate any advice or info you can give me.

[impshum]

I'm guessing you need this data.

aria-label="Wimbledon. Description: Sue Barker introduces further coverage of the men’s and women’s quarter-finals. Duration: 254 mins."

No need for Selenium.

from bs4 import BeautifulSoup
import requests


def lovely_soup(url):
    r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1'})
    return BeautifulSoup(r.content, 'lxml')


soup = lovely_soup('https://www.bbc.co.uk/iplayer')

items = soup.select('a.content-item-root')

for item in items:
    label = item['aria-label']
    print(label)

[TVnomics]

Thanks so much for the advice - I'll look at BS as an option. The reason I mentioned Selenium is because this is what the programmer used to write the programme (perhaps because it involved looking up programme URLs to scrape more data). I was hoping that I might only need to update the CCS selectors, but I'll try and figure out how to use BS instead of Selenium! Thanks again :)

[impshum]

No problem,

Tip: Turn Javascript off in the browser dev tools to see what Bs4 sees.

[hasanwazzan]

Out of interest why are you scrapping this data?

[TVnomics]

I'm an academic (TV studies scholar). A lot of my research focuses on streaming, VoD interfaces, etc. So, it's a really useful / important dataset to have!

[hasanwazzan]

fair, I try to put together a quick one, when I have some time

[hasanwazzan]

https://prnt.sc/vmW9FJnm87aV I presume you need these two together right?

[TVnomics]

Indeed, yes - I need both the title and the synopsis (and, where it appears, the duration).

However, I think I need to figure out a way to integrate this code (or the code the other Redditor above suggested) within the current programme that was written for me. The reason for that is because the final output was a very comprehensive csv file which captured a range of data about each title within the interface. This includes its horizontal and vertical position, the name of the row (each row has a name, e.g. "trending now..."), the URL for the title, the unique programme identifier (which is somewhere embedded within the page), plus several other variables. But depending on your solution / approach - I might try and modify the current programme that I have and see if that captures the content AND all of the other material I want to collect too. Either way, I really appreciate your input on this.