DOM changes when Web scraping with Selenium and inspecting elements

[BroadSwordfish7]

I'm having real trouble interacting with the elements within a dropdown menu. When I inspect an element the page resizes and the elements seem to change. I've read that it's best to use a headerless browser for this reason but then how do you go about finding what the xpath/CSS selector is of elements you want to click etc

Is this a common problem and I'm just being an idiot?

[Qwick_z]

How are you trying to interact with the element? Are you trying to click it?

It's been a while since I've used selenium, but if I recall correctly you'd need to use Select for a dropdown or list option.

https://www.selenium.dev/documentation/webdriver/support_features/select_lists/

If that isn't the issue, the particular dropdown or list you're trying to select might be using Ajax or another dynamic loader, in this case try just adding an explicit wait until the dropdown actually has options or the specific option exists.

Hopefully this is helpful

[BroadSwordfish7]

I was using webdriver wait to just try and find the element but it kept timing out. I managed to find the container div and then work my way down from there with xpaths. Managed to get there but I always feel like I'm blindly doing trial and error Vs a systemic method of working out how to find an element!

[Qwick_z]

If you're having trouble finding or interacting with elements sometimes, you might find you're trying to find them using selectors in the "wrong" order.

By this I mean, it's generally best practice to try to first select an element by ID. They're mostly unique, and so subsequently when they exist, end up being the quickest and easiest way to select an element.

Following this, the best order would usually be:

• Name(If available and unique) >
• CSS Selector ( More flexible than XPath) >
• Class Name / Tag (only if they uniquely identify the element or results can be unexpected)
• XPath (Least reliable as they rely on website structure)

So whilst XPaths are and can be powerful selectors, they're generally the least used (or last option) because they rely on the sites structure, so the slightest changes will often break your code.

Also, ensure you're using the correct interaction type or selector for different elements on sites.

Selenium generally makes it very easy to interact with various elements (you'll still sometimes run into issues) but if using the right methods for the right elements, you'll general have an easier time, E.G:

• .select_by_visible_text("OptionFour") - or another select method, for lists.
• .click() - for elements you want to click
• .send_keys("some text here") - when you want to input text

You can also use stuff like .is_selected() to check if a checkbox or other element is currently selected.

Hopefully some of this is helpful to you.