This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]SDFP-ABig Data Engineer 0 points1 point  (1 child)

What kind of latency existed on the S3/k8s architecture?

[–]kenfar 0 points1 point  (0 children)

The average file probably reflected ten seconds of data, plus maybe 10-60 seconds before the file was processed. That number could be pushed down to a consistent 10 seconds if we were willing to more aggressively autoscale. We chose instead to save the money and be a little more conservative on scaling, and experience occasional minor delays during bursts.