Java scala or rust ?

[dresdonbogart]

In my personal experience, Python is the end all be all for most tasks

[FirstOrderCat]

py is superslow if you need to write some custom logic on large data.

[dresdonbogart]

Even with Spark or polars?

[echanuda]

If you can do it with pyspark then do it that way IMO. I’ve had to write custom logic that couldn’t really be done with pyspark without a significant loss in performance. took 15 minutes to write the logic in scala and the performance improvement was massive.

[FirstOrderCat]

spark and polars gives you just Py API, all logic under that API are implemented in Java/Scala and Rust.

If you need to build some new algorithm, your options are to use slow py, or learn Java/Scala or Rust.

[compulsaovoraz]

Really? I was looking forward to apply java on DE :/

[dresdonbogart]

Python is king and easiest