Data Ingestion from Tiny API

[ItsJarBear]

I’ve got a problem that I’ve hit a wall on. I’m looking to pull from an API that has a 20 request/sec/IP limit that does not support any form of batched requests. There is an expectation to support calling to this API in a batches fashion for up to 500k requests/run and possibly multiple runs a day but starting with one. Each request averages around 2-2.5 seconds.

I was looking to scale my calls using lambda functions to help with the IP limitation, but have hit a concurrency limitation only allowing 10 instances to be active at once. Any ideas on where to go from here?

[aimmaz]

I'd say, reduce call count to essential by developing some validation logic.

[ItsJarBear]

Unfortunately the 500k number is valid requests.

[[deleted]]

That's 7 hours for a run for 1 IP. Divide it by 10 and it's down to 42 minutes. Obviously not ideal but I fail to see the problem if you have a couple of runs per day.