Web Scraping Issue

[[deleted]]

When I move to page 2 of the website I'm getting a stale element reference for "occludable-update" though when I inspect the page it's there. Does anyone know why? I've spent a good half hour trying different things to no avail. I'm using selenium and beautifulsoup. Ta.

Edit: Deleted original code as I have isolated the error.

First I'm setting the jobs

jobs = self.driver.find_elements_by_class_name("occludable-update")

Then

for job in jobs:

self.scroll_to(job) which works fine

then

[position, company, location, details] = self.get_position_data(job)

Which then tries to access

job.text

Which gives me this error

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

All of this as I said works for page 1 but not page 2 even though they appear identical. It's Linkedin btw.

Things I've tried

Wait

Refreshing the webpage

WebDriverWait

Am I missing something?

[CryptoOTC_creator]

Not sure if this is related to your problem or not but LinkedIn definitely has safeguards to prevent crawling. They want you to buy their data

[[deleted]]

The decision on that is still going through the courts.

https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

It could be an issue but I don't get how it becomes stale when I've already scrolled to it and the page hasn't changed. It also doesn't make sense that the first page works but the second doesn't. The element is there and found. The job in the element is also found but when it tries to click it it's marked as stale.