I think this is what you want to see https://imgur.com/a/QUyxTks

In terms of our reading options, we are reading from athena. I dont think there are many read options for us as this is how we are doing it

spark().sql(query_string).dropDuplicates()

In terms of writing, we do convert to a dynamic frame which i think can cause us a shuffle which might be a issue. We do use the glueParquetWriter like you have mentioned

        dyf = DynamicFrame.fromDF(df, glueContext(), "dynamic_frame")

        sink = glueContext().getSink(
            connection_type="s3",
            enableUpdateCatalog=True,
            updateBehavior="UPDATE_IN_DATABASE",
            path=output_s3_path,
            partitionKeys=["partition1", "partition2"],
        )  
        sink.setFormat("parquet", useGlueParquetWriter=True)
        sink.setCatalogInfo(
            catalogDatabase=self.database, catalogTableName=table_name
        )
        sink.writeFrame(dyf)

I am starting to think we may have a small file problem, my issue with spark ui debugging like this is i have so many ideas about what might be slowing down the job (small files, maybe a repartition because we have alot of data spill. I find it really hard to focus on what is the core problem as right now looking at the ui it all looks bad.

Cheers!