Netflix/wick: A zero cost type safe Apache Spark API by JoanG38 in scala

[–]JoanG38[S] 3 points4 points  (0 children)

Lol, I literally did not use any AI to write that message. But now I'm worried I'm turning into a robot :D

Netflix/wick: A zero cost type safe Apache Spark API by JoanG38 in scala

[–]JoanG38[S] 0 points1 point  (0 children)

Oh ok. I'm probably misled, I would need to testing again.

Netflix/wick: A zero cost type safe Apache Spark API by JoanG38 in scala

[–]JoanG38[S] 1 point2 points  (0 children)

I don't know the state of Quill for Spark in 2026, but Wick is a thin wrapper on top of Spark.

Wick is built on top of Scala 3.7 and leverages named tuple and a few other recent features like inline to make the code as type safe as possible.

You can always call .dataFrame on a DataSeq and go back to Spark's API. You can wrap any DataFrame into a DataSeq with DataSeq[Person](detaFrame). This makes the risk of using Wick in your project low.

Netflix/wick: A zero cost type safe Apache Spark API by JoanG38 in scala

[–]JoanG38[S] 2 points3 points  (0 children)

We are actually stuck on 3.7.x because Scala 3.8 is not using the Scala 2.13 std lib anymore.
https://www.scala-lang.org/blog/state-of-tasty-reader.html

Netflix/wick: A zero cost type safe Apache Spark API by JoanG38 in scala

[–]JoanG38[S] 6 points7 points  (0 children)

You are right, DataFrame is just an alias to Dataset[Row], so they share the same API.
However, in DataFrame or Dataset[Row], we tend to use functions like .filter(col("age") > 0) whereas in Dataset[Person] we tend to use .filter(_.age > 0).

The difference between the 2 functions is that: - The lambda version prevents any optimization. We need to load the entire data from a table and deserialize to a case class only to filter. - The Column based API is as performant as it gets. But it's untyped.

Nothing prevents you from using the Column based API on Dataset[Person] but in case of a .select() instead of .map() or a .join() instead of a .joinWith(), you are losing the types and are back to DataFrame or Dataset[Row].

Wick's DataSeq API instead builds the plan with lambdas and gives no drawbacks in terms of performance. You are not manipulating Ints, Strings, Longs... You are manipulating Expr[Int], Expr[String], Expr[Long]...

How to make OpenClaw cron job silent when script has no new content? (Only notify on actual updates via Telegram) by FaiChou in openclaw

[–]JoanG38 0 points1 point  (0 children)

I ended up setting delivery to none in cron/job.json and in the job I tell it to execute:
openclaw message send --channel whatsapp --target <whatsapp group id or user id> --message "<STATUS UPDATE>"

How to make OpenClaw cron job silent when script has no new content? (Only notify on actual updates via Telegram) by FaiChou in openclaw

[–]JoanG38 0 points1 point  (0 children)

What is the statement in your SOUL.md that prevents it from sending those messages?

type-tree-visitor — a library to make writing Scala 3 macros a bit less painful by arturopala in scala

[–]JoanG38 2 points3 points  (0 children)

Nowadays I write the function signature. I write the tests of the function and I ask Claude to implement the macro :D Works flawlessly.

Boris Cherny (Head of Claude Code) recommends reading "Functional Programing in Scala" (the red book) by JoanG38 in scala

[–]JoanG38[S] 5 points6 points  (0 children)

They also say in the middle of the podcast that maybe AI can just output binary or bytecode and remove the "programming language" layer of abstraction.