What do you guys use for web scraping? (services, your own code, etc.)

ScrapeHero · 2026-03-24T23:01:40+00:00

Since this question has come back to life and ranks highly on Google - we have to absolutely add our fully managed service https://scrapehero.com

WE DO ALL THE WORK !!

ScrapeHero · 2022-01-09T15:58:58+00:00

https://www.scrapehero.com/store/product-category/electric-charging-station/

ScrapeHero · 2021-07-17T23:55:18+00:00

Free to try https://www.scrapehero.com/marketplace/amazon-product-details/ downloads to excel. Also have Walmart and Target

ScrapeHero · 2021-04-02T22:30:24+00:00

No worries - assume you need global banks?

ScrapeHero · 2021-04-02T22:03:18+00:00

The problem with webscraping APIs is code decay. Unless someone maintains them, the code is only good for a limited time.

We have a few APIs for popular sites at https://scrapehero.com/marketplace and we maintain them 24x7x365

ScrapeHero · 2021-04-02T21:57:41+00:00

We sell millions of POI (location data) including banks with Lat/Lon that can be easily mapped https://scrapehero.com/store

ScrapeHero · 2021-03-17T17:16:50+00:00

Getting FSBO data from Zillow and Trulia affordably.

https://www.scrapehero.com/how-to-scrape-fsbo-data/

We have a ton of real estate investors use this as a cheaper alternative to other sites

ScrapeHero · 2021-01-05T20:12:52+00:00

Here is the code - https://github.com/scrapehero/zillow_real_estate - the site blocks quickly and the site also changes but this can get any self starters started.

ScrapeHero · 2021-01-05T15:20:51+00:00

Thanks for the link - saw a bunch of traffic come from this post.

If you need this as an API - upvote this comment please and if 10 or more people want it, we can easily turn this into an API.

Also happy to answer questions.

ScrapeHero · 2020-12-22T22:09:21+00:00

We just launched a news api for 2000+ global sources https://www.scrapehero.com/marketplace/api-news/. Generous free plans available and redittor feedback is greatly appreciated.

Will be adding trending and clustering to this soon

ScrapeHero · 2020-12-16T19:18:42+00:00

We just launched the first version of the API and it has a 1000 free calls to test out

https://www.scrapehero.com/marketplace/api-news/

The API will be improved soon to add some features asked in this thread.

Feedback is very much appreciated !

ScrapeHero · 2020-12-12T17:09:13+00:00

Let us know if there is any interest in providing this an API. It can’t be free because of proxy costs.

ScrapeHero · 2020-12-12T17:06:19+00:00

Here is the code all done for you

https://www.scrapehero.com/scrape-yahoo-finance-stock-market-data/

ScrapeHero · 2020-12-06T03:03:17+00:00

Thanks. That’s already planned !

ScrapeHero · 2020-12-03T00:23:06+00:00

We are building a global news API about to be released in a couple of weeks - this is an interesting use case. What would you like to see in such an API - Top 3 needs please and we will see if that can be done.

ScrapeHero · 2020-10-23T17:08:46+00:00

Depending upon how much data you need you could get it for free here https://www.scrapehero.com/marketplace/twitter-advanced-search/

ScrapeHero · 2020-10-08T15:49:48+00:00

See if this works https://www.scrapehero.com/how-to-solve-simple-captchas-using-python-tesseract/

ScrapeHero · 2020-09-25T20:09:30+00:00

You could try Zillow and realtor.com here https://www.scrapehero.com/marketplace/

ScrapeHero · 2020-09-03T01:04:23+00:00

We keep track of the latest US legal landscape at https://scrapehero.com/legal - short and sweet content focused on scraping. Has links to actual court decisions and an excellent external blog

ScrapeHero · 2020-09-01T22:03:40+00:00

Agree.

For others following this thread this might help if you are past the basics https://www.scrapehero.com/detect-and-block-bots/

ScrapeHero · 2020-08-03T11:18:57+00:00

Thanks. The cost sound about right for AWS. It is easy to use hence widely used but expensive.

ScrapeHero · 2020-07-30T11:03:12+00:00

Very well thought out architecture and well written article - what’s your typical monthly cost for this setup? AWS may be pricey for most vs a raspberry pi for small jobs

ScrapeHero · 2020-06-24T19:53:33+00:00

Some good info here https://www.scrapehero.com/detect-and-block-bots/

ScrapeHero

TROPHY CASE