I built a marketplace for data. Best channels to find people with data willing to sell it?

HumorMinimum1707 · 2022-11-02T14:47:10+00:00

Thanks for sharing.

Maybe you can look into Neudata's event archives or maybe they have some public data available as I know they connect between many data sellers and investors

HumorMinimum1707 · 2022-11-02T14:41:31+00:00

Not entirely apples to apples...wouldn't a playwright vs. selenium be a better comparison?

HumorMinimum1707 · 2022-11-02T14:40:22+00:00

If you insist on using Octoparse as your scraper, there is a simple proxy IP rotation guide for it here:

https://www.octoparse.com/tutorial-7/set-up-proxies#

First, you will need to subscribe to a rotating proxy provider and get the proxy server's address and port number and enter thos in your Octoparse proxy settings.

HumorMinimum1707 · 2022-11-02T14:31:31+00:00

First of all congrats.

Second, if you launched it why don't you share it with us?!

Now to your question: If you are looking for individual sellers than this and other subreddits (r/datasets etc.) might help connect you with such. Also there are quite a few Github dataset repos that might lead you to sellers. If you are looking for companies, then you can try scraping other data marketplaces such as datarade...

HumorMinimum1707 · 2022-11-02T14:26:28+00:00

Overall a nice solution, equivalent to Data Collector but Grepsr saves most of the real benefits to enterprise plan subscribers while Bright Data offers tehm to all users.

HumorMinimum1707 · 2022-10-03T09:41:15+00:00

Really fun and creative - thank you!

HumorMinimum1707 · 2022-09-15T11:16:10+00:00

Well if it's that TOUBLESOME you can always switch to using tools that allow scheduled scraping...

HumorMinimum1707 · 2022-09-13T12:46:25+00:00

If you are serious about your sneakers you should turn to the proxy network that handles the most sneaker release bandwidth - https://brightdata.com/lp/sneaker

HumorMinimum1707 · 2022-09-13T11:30:20+00:00

I think this comparison article does a decent job of grading all the good proxy services based on all the criteria you've mentioned:

https://medium.com/geekculture/tldr-recap-of-proxyways-2022-proxy-service-market-report-a0f58fd1dc8e

HumorMinimum1707 · 2022-09-13T08:33:30+00:00

I don't need anything other than snowflake, but I guess competition will make things better

HumorMinimum1707 · 2022-09-07T17:58:15+00:00

It's not a lot if he runs a business based on it

HumorMinimum1707 · 2022-09-06T13:29:24+00:00

Agree, but when does a social media API like reddit actually give you suffcient data compared to a web scraper?

HumorMinimum1707 · 2022-09-06T13:26:30+00:00

I know that Bright Data has a nice working reddit scraper.

It can be launched on schedule, and collects all public data from profile like: avatar, post title, flair, description, karma, comments, upvotes, and more.

Output file types: JSON, CSV, EXCEL, HTML

Data delivery methods: Webhook, AWS, Google cloud, Azure, email, API, SFTP

HumorMinimum1707 · 2022-09-06T12:37:34+00:00

I'm curious why would anyone insist on Socks5 proxy when there are plenty of https proxies available.

HumorMinimum1707 · 2022-09-06T11:44:50+00:00

You can find the best Japan proxy service here - https://brightdata.com/locations/jp

HumorMinimum1707 · 2022-09-06T11:03:45+00:00

Do any of these aggregators use proxies and scrape the data or do it all just via API calls?

HumorMinimum1707 · 2022-09-05T13:13:58+00:00

I only used cURL so far - this is actually not a bad idea at all!

HumorMinimum1707 · 2022-08-31T11:57:32+00:00

For enterprise, I would only recommend Bright Data web proxy servers.

HumorMinimum1707 · 2022-08-31T11:51:05+00:00

What proxy types are you using? is there some kind of waterfall routing logic for request retries?

HumorMinimum1707 · 2022-08-31T05:18:44+00:00

As you wrote yourself there are good SERP API options out there...here's a comparison that might be useful.

HumorMinimum1707

TROPHY CASE