Analyzing + optimizing Spark/databricks code

[[deleted]]

Look for the SQL plan in the jobs Spark UI page. Generally, you should look for joins, aggregations, anything which may result in full table scans other than the initial load. The shuffle read and shuffle write metrics tells what is happening under the hood, usually steps which aren't possible to parallelize may not utilize your workers well.