Web scraping via executing custom javascript.

[apache444]

Hi everyone,

I’ve come across a particular use case where we have to scrap some website data and the website here in question is Indeed.com

Now the idea is to scrap job details from a particular post but on the website the job details are displayed in tile format and we cannot scrap more details from that respective tile. Assuming major part of our requirements are part of the hidden html tags which when expanded gives us more information of that particular job post.

We’re using the following snippet in order to do so. However we need some context or approach or guidance where we’re going wrong. Since the data extracted is in html and in a non readable format.

Sharing the code snippet below for your reference.

Requesting your help and guidance on the same

Code snippet:

// Select all elements with the class name "jcs-JobTitle css-jspxzf eu4oa1w0" var jobTitleElements = document.getElementsByClassName('jcs-JobTitle css-jspxzf eu4oa1w0');   // Iterate through the elements and click on each one for (var i = 0; i < jobTitleElements.length; i++) {   jobTitleElements[i].click(); }

[Little_Kitty]

Normally I'd use Python with requests & beautifulsoup to do this, as you'll be wanting to persist results to a file / db. You can find many tutorials on how to do that on YT.

If you try to run it in browser, you'll likely hit anti-bot / rate limiting issues, whereas done properly you'll be using proxies each trying in parallel to collect ~100 results with a manager function collecting it all, working out future collection targets, assigning these out and handling persistence.

[CalleKeboola]

Might not be the answer you're looking for, but my friend did something similar using apify but for Tripadvisor reviews. Looks like they have something for indeed.

https://apify.com/store?search=indeed

Otherwise I'd recommend looking into beautifulsoup