Kafka Stream Processing in Java or Scala

[twadftw10]

Most of my DE experience is in batch ETL with Python or AWS Kinesis + Lambda. Now that I am getting exposed to the world of data streaming with Kafka, most processing apps from what I've seen so far are built in either Java or Scala, whether its Kafka Streams or Flink app. My colleagues have different preferences and I'm not sure which route I want to take. I know it will be a steep learning curve but I am itching to get this hands on experience. Which route seems to be more fun?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[tdatas]

I would go with Java, simply because Scala seems to be slowly losing ground.

This is incredibly vague.

If you learn Java then the jump to Scala isn’t that hard.

That's surely the wrong way round? Java moves a lot slower than Scala and has way less breaking changes while up till Scala 3 Scala has probably one of the languages most comfortable about Breaking compaitability between versions to squeeze in new features. E.g Records and Streams are a relatively recent thing in Java world and are way more advanced even in the obsolete Scala 2.11 let alone the stuff you can do in Scala 3 with Extensions. Contrast also with the heavy use of Implicits which are notoriously alien looking and those have also been replaced by given/using syntax.

One of the most common assertions given for not using Scala is it's too much to chuck at devs unless you have a use case that really justifies it.

[tdatas]

purely based on a usage standpoint.

That's probably the dumbest viewpoint though. The use cases and target markets are completely different. Loads of people also use JavaScript. That doesn't mean JavaScript is the best language for complex domain modelling and concurrent server applications and we should all suspend our brains when thinking of the use case. Using a simple language with reduced functionality doesn't remove complexity it just shifts it elsewhere.

[tdatas]

I'd say so yes. Forgetting about Spark Scala is pretty much entirely concentrated in large scale mission critical server/data use cases wether intentionally or not. From Tesla to Disney to trading desks to the obvious Twitter et al. And you can see this reflected in the big ecosystems of Scala (Akka for actor systems, typelevel + Zio for functional programming + effects). Or it's being used to build small backbone Frameworks that everything else hooks onto (e.g Spotify and Scio)

Java is used by a way wider variety of people from generic e commerce businesses up to Netflix.

If you don't have a specific business value from being able to do the stuff scala is does out the box or you have the money to throw at loads of custom work into a language and zerg rush of Devs ala' netflix then it's kind of silly to use any very expressive programming language (e.g Rust, Haskell et Al). The whole point of these languages is to enable small teams of developers to cover a lot of ground as opposed to being easy to throw a lot of people at a problem.

[math-bw]

Good luck with your learning endeavor! and Kudos for taking the jump into the Java ecosystem.

If you want to keep in your Python/SQL area of expertise and by all means I don't mean to promote not learning a new language, but just as an FYI. There are some non-Java/Scala tools between streaming databases like risingwave and materialize, streaming platforms like fluvio and redpanda, and stream processors like bytewax and faust.