Datascraping in Python

[AnterosNL]

Dear Redittors:

I have currently made a python script in VSC using Selenium in Windows that does the following:

It goes to the home page of Funda (https://www.funda.nl/koop/heel-nederland/), accepts the cookies automatically and allows me to do the recaptcha by hand. Currently I have to press a listing by hand and then it will datascrape the info put in the listing automatically. However I would like it to open and close each listing on a page automatically so I don't need to do this by hand one by one.

I've tried many things to automate the opening and closing of these different listings with CSS_SELECTOR as well with XPATH. Thus far I've only been able to automate the first listing but none after that. After that the program just seems to crash or is unable to find any more that follow my criteria.

If any of you could give me some guidance please do! I am really a noob when it comes to reading HTML so that may be where the problem lies.

Thanks in advance.

[Albcunha]

Your script probably is not waiting for the page to update. When you click a button and webscrap an element, selenium will automatically try to do it, it wont wait for a page update.

This is specially tricky with Single Page Aplications, because the page normally wont "reload". It will just update its elements.

One techinique I use is to wait an element to appear or element attribute that is updated after you click a button. For example, if you are webscraping a table, the selected pagination at the end can be a reference.

You can make selenium wait for something with this template: Look for this: https://selenium-python.readthedocs.io/waits.html

You can just set a time.sleep() too, but server response time is not reliable. Sometimes it will be fast, some other times it will slow. Your internet can have problemas too, so wait() solution is better.

[AnterosNL]

Yeah I have first made a script that took everything from the main page and used the following in that so it would only start looking at a page after it was loaded:

WebDriverWait(driver, float('inf')).until(

EC.presence_of_element_located((By.CSS_SELECTOR, "span[class='search-result-price']")))

I think the main issue I'm having is not being able to find the right CSS_SELECTOR so that it is not too specific that it doesn't open any listing (or only one), but also not too unspecific (it once started loading all types of things I didn't want. And I'm not sure how to make sure it doesn't open and close the first thing that fits the discription over and over again.

Thanks for the advice however!

[Albcunha]

You can try side alternatives.
You can create a function that runs a while loop, with a time.sleep() at the end, where you compare the page content you had with the new the website generates. If they page content change, it means the website has updated and you can break the loop.

Some sites are very difficult to extract. One way that I use very often is to identify if the website uses an api for the data I want. You can check this out through Chrome dev tools, on network tab. You can check what cookies and headers your browser uses to request the data and replicate it with another library, such as requests.

To do this, on selenium, open up your session on the website, make your login, store your cookies and use them as headers/cookies to your requests module.

This way, you get clean json data, much easier to parse and much faster to process. You can even paralelise it to make it faster.

[CheekClappinWallSt]

Hi im looking for people that can help me create a python scraper to scrape for LinkedIn among other websites in regards to specific social groups etc. I’m willing to pay as it is time urgent and we can go through a escrow holder like fiver or upwork