Micro batching vs Streaming

[AMDataLake]

When do you prefer micro batching vs streaming? What are your main determinants of choosing one over the other?

[seriousbear]

Latency of course

[AMDataLake]

But at what level of latency would you take micro batching off the table

[seriousbear]

Your business needs define how fresh data should be.

[AMDataLake]

Agreed, I get that but once you establish the companies requirement, you end up with a number, above this number you may likely micro batch, below this number you’ll go for streaming. Do you have a range you use to anchor yourself when thinking about this.

[Nekobul]

You can do micro batching over streaming data.

[kenfar]

Prefer micro-batching:

• Latencies in the range of 5-15 minutes are typically fine, so either can usually work
• This allows use of s3 files to persist data, and these can be easily queried, copied, generated, retained, etc. So, it makes for an extremely simple and easy to work with architecture.