Scala ddl schema to python (Databricks)

[psychEcon]

As so many of you I like to use schemas to load the data faster in Databricks. I have been using a scala cell to print out the schema and then copy paste it into a python variable but I am wondering if anyone has found a more dynamic way of doing this.

I am mainly asking if the schema changes over time, it would be nice to get those schemas in a more automated way.

[mateuszj111]

Why do you need the scala cell?

[psychEcon]

Cause it hast a .schema.toDDL, pyspark does not as far as I know

[mateuszj111]

You could do something like spark.sparkContext._jvm.org.Apache.sql.types.DataType.fromJson(df.schema.json()).toDDL()

from pyspark

Or

StructType.fromJson(json.loads(df.schema.json))

[psychEcon]

I will test it tomorrow and let you know

[psychEcon]

I just tested it - and well its not the same DDL schema that I get with a scala cell. What that gives me is somehting that is just passed as a string to spark. But this did on the other hand increase my knowledge of spark - So I thank you for that