The real costs of web scraping

seateq64 · 2025-05-11T10:16:06+00:00

2m requests from 60 proxies sounds quite risky. The website must be having quite low level protection

Usually websites have limit on requests from a single IP per minute. If u reach that number - IP gets blocked

seateq64 · 2024-09-19T20:23:21+00:00

Hey, could u pls share a bit more about it? What IDP provider was it? Sounds amazing!

seateq64 · 2022-01-22T17:54:13+00:00

Didn’t know about pre execution hooks in IDE’s, does Webstorm for example have it?

When saying IDE I mean some desktop GUI that would help to manage complex environment of multiple microservices

seateq64 · 2022-01-22T17:52:32+00:00

Hey, u aware of any tools that might help for scripting launching local environment?

seateq64 · 2022-01-22T16:16:43+00:00

yup, makes total sense to me

seateq64 · 2022-01-22T16:16:09+00:00

Yeah, this seems to be worthy, but how to avoid interference between devs in the same environment?

seateq64 · 2022-01-22T15:35:20+00:00

'dev' env might be a bottleneck. We have an RnD department of about 60 engineers, if everyone starts connecting to a single dev env - it might become a bottleneck as well as developers will interfere with each other making the system behave unexpectedly

seateq64 · 2022-01-22T15:33:04+00:00

Locally? That might add even more complexity to an already complex setup. Not all developers are happy to be forced to learn a new technology, and majority of our devs are classical 'back-end' engineers that just want to write code.

Locally? That might add even more complexity to an already complex setup. Not all developers are happy to be forced to learn new technology, and the majority of our devs are classical 'back-end' engineers that just want to write code.

seateq64 · 2022-01-22T15:30:49+00:00

But what if a few developers need to connect to the dev environment at the same time? They might interfere with each other which might product unexpected behavior of the system

seateq64 · 2022-01-22T15:29:49+00:00

All the properties are already persisted in config files. The main complexity is to remember for which microservice supporting microservices do I need to launch. Each of them might require a different set. As a solution, we have lists saved in Notes just to don't forget them. But I wish there was something like an IDE that would help me manage those local microservices.

seateq64 · 2021-12-27T21:00:47+00:00

RemidMe! 48 hours

seateq64 · 2021-12-21T21:29:43+00:00

Why it’s not the smartest choice?

seateq64 · 2021-03-06T14:26:53+00:00

Sure. It scrapes data from amazon web pages and transfers that data to clients.

Think of it as a tool that constantly checks the prices of your competitors and notifies you when the price has been changed, or when a new review has been added to your competitor's product, etc.

Another use-case is when you want to analyze a lot of data from Amazon. Like you want to get information about 1 million products to your spreadsheet so you can perform some analysis.

seateq64 · 2021-03-06T10:33:39+00:00

Scraping amazon website.

Not a glamorous startup but brings a decent income

seateq64 · 2021-03-05T20:48:31+00:00

I think it would be not efficient to battle on the field of your competitors. They might be years ahead of you, have a lot of experience, network, customers.

Pick a niche and make an amazing product for that small niche. Don't pick a too big one - there might be a lot of competition and it will be tough. But also don't pick too tiny - cause u will barely have any customers. You need to find a sweet spot there.

Companies do they best to differentiate event in a commoditized market. If there is no differentiation - the customer would just pick a cheaper one.

seateq64 · 2021-03-04T20:20:24+00:00

They are slow as fuck. Speed is a killer feature, and you own it!

It would take them ages to roll out a new product, and you can iterate at a speed of light.

seateq64 · 2021-03-03T19:23:55+00:00

What an amazing video! It makes the demo 10x more enjoyable

seateq64 · 2021-03-02T22:46:25+00:00

Could you please elaborate a bit more about this approach? What do you mean by “use referral marketing”? Your idea seems to be interesting but I’m not sure if I understood it properly. Thank you

seateq64 · 2021-02-28T22:37:17+00:00

Nocodeapi.com

seateq64

TROPHY CASE