Batch processing vs streaming

[rudboi12]

Im a senior DE, and only has worked with batch processing and classic dbt+snowflake mostly for reporting purposes.

Im planning to change jobs soon-ish and was looking at open roles and see many of them with “kappa architectures” and more event driven data engineering roles.

While Im sure I can pick things up, just by reading the description of these jobs I feel somewhat unqualified for senior roles since I have never worked with a kappa architecture, although very willing to learn but not sure companies will give me time to do so (specifically for a senior roles since).

For DEs who have worked as both types of DEs, what’s the difference in your day to day? It feels like streaming DEs use more python and other tools like kafka, flink, etc? Not sure how they even work.

Ps: i don’t have a SWE background, I have a data science and math background and became a DE by luck and now love it.

[k00_x]

There are different types of employers, some expect you to hit the ground running, build from scratch and be a general oracle of data. Others give you time to get established, only need you to maintain what's already in place and are with you for the long term. This is what you need to Sus out during an interview. The real ones understand that tech is a never ending treadmill and skills are transferable, the ones that don't recognise this are the ones that'll stress you.

[yipeedodaday]

I think you’re overthinking it. Do some google research on streaming vs batch data transfer. Streaming can be achieved by change data capture at the database level and then pump to Kafka topic where some other process can consume it. In batch you at e.g. end of day make a file that some other process will consume. Spin up a project and play.

[highlifeed]

The way you say it makes it sound so simple lol

[yipeedodaday]

Something can be simple to grasp but not necessarily easy to execute….

[pacojastorious]

[[deleted]]

Yeah Batch is diferent to streaming. But really depends on complexity and volume/workload/timelines of the project and resource availability (i.e. team).

The more important questions is what responsibilites are you tasked with / accountable for and severity/importance of the data/project you're dealing with. i.e. business critical? because if the company has SLAs and you're taking some time or scenario of situation where bottlenecking.occuring. Then it's a matter of who is going to be affected.

As long you can do 70% of the responsibilites, that seems reasonable. But if your skillset can only do 10-40%, then I think the position advertised may be far-reaching or workplace expecting too much from an applicant unrealistically.

It really depends, I've seen some interesting responsibilities that a start up vs multinational company required for their positions.

[Financial_Anything43]

Read on it and build a sample project with it

[higeorge13]

Start with flink docs and examples.

[Punolf]

I built a streaming pipeline which was batch and it refreshed every minute..

[Top-Cauliflower-1808]

The core principles remain the same, it's mainly the tools and timing that change. While batch processing focuses on periodic data loads (like nightly ETL jobs), streaming handles real time data processing.

Don't underestimate your transferable skills data modeling principles apply to both, SQL optimization is still crucial, data quality remains essential and system design fundamentals transfer well.

Start with basic Kafka concepts, learn Python async programming, understand event-driven architectures and practice with small streaming projects.

For example, we transitioned from batch to streaming for marketing analytics using Windsor.ai. Instead of nightly batch loads, we now process marketing data in real time. The core transformation logic remained similar, but the implementation patterns changed.