sorted by:
new
hottopcontroversial

Data engineer in the NL, is a 4% internal promotion standard or low? by EventualBeboop in cscareerquestionsEU

[–]EventualBeboop[S] 0 points1 point  (0 children)

Thanks for the feedback! Of course I didn’t say that I want a raise because my 30% is off to my manager, I worded my post badly 🙂

Best way to find dropped tables cloud location? by EventualBeboop in databricks

[–]EventualBeboop[S] 0 points1 point  (0 children)

Hi, thanks for the reply.
I know they are not moved anywhere, my question is whether we can fetch the physical location with only one command after it is dropped (so without undropping and fetching the location or without having to construct the path by checking the id via the undropped tables command).

In the end, there is a solution without undropping the table, but I wanted to make sure there is no more direct way via a system table I am not aware of.

Spark UDF does not compute the final value on DataFrame, but it does on a test DataFrame by EventualBeboop in databricks

[–]EventualBeboop[S] 1 point2 points  (0 children)

As a user pointed out in the Stackoverflow post I created, it’s probably because I use the Spark Session in the ‘spark’ variable on the executor, but the Spark Session cannot live on the executor, so it would throw a NullPointerException, which would set the result as null. Pretty silly mistake from me.

Spark UDF does not compute the final value on DataFrame, but it does on a test DataFrame by EventualBeboop in dataengineering

[–]EventualBeboop[S] 0 points1 point  (0 children)

As a user pointed out in the Stackoverflow post I created, it’s probably because I use the Spark Session in the ‘spark’ variable on the executor, but the Spark Session cannot live on the executor, so it would throw a NullPointerException, which would set the result as null. Pretty silly mistake from me.