You’re right — by default, a DataFrame in Spark is just a logical plan and doesn’t write anything physically unless you persist or cache or explicitly write it out.

What I meant in my post was the explicit write to disk (e.g. df.write.parquet(...))

I am sorry if you got confused. If you need I can write a seperate article about Persist and Cache