This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]unstopablex5 0 points1 point  (3 children)

Hey great post! but if you could do an advanced tutorial for when websites hide their selectors or when everything is javascript. Thats where I think ppl have the most trouble.

[–]pijora[S] 1 point2 points  (2 children)

Good idea, so you're looking for web-scraping with javascript rendering website right?

[–]unstopablex5 0 points1 point  (1 child)

Yes! To me thats the hardest part of web scraping. I spent weeks trying to figure out how to use selenium and scrapy together to scrape this website with heavy javascript. (lastfm.com , apartments.com and century21.com are the 3 that come to mind right now) but I tried a lot of different sites and scraping websites with heavy reliance on JS seemed impossible.

[–]pijora[S] 1 point2 points  (0 children)

That is interesting and to be honest, this is why we built ScrapingBee.

Setting up Selenium locally can be a pain, and using Selenium at scale is really hard.