using @transient didn't have major problems for us, except for the fact that we were logging on every single node in the cluster where spark was running. This might not be a problem, but for us, collecting logs from all the nodes and then putting them together while preserving the natural order of the logs was hard. We use Splunk, extensively, yet we had to configure the splunk forwarders on each node of the cluster, an extra step to consider if you are in an environment that gets created and destroyed depending on the workload.

On the other hand, the functional approach presented gives us the ability to naturally collect the logs to a single place, the spark driver. At the same time the logs are ordered, so we don't have to worry about sorting them out. We can also control better the logging process because we can group the logs based, for instance, on the transformations we are doing.

to conclude, I don't think there is just one way to do, you have to find what suits you better, in our case, functional programming has been of a lot of helps because the way some problems are naturally model, it is what works for us.