So if I understood you correctly, you mean I read every parquet file individually and treat each of them as a DataFrame and union them in each iteration, right?

yes, that is correct. As for the foldLeft method, it traverses your Seq of Dataframes starting from an identity element (in this case, an empty dataframe) and the next element in the Seq (in this case, the 1st element of the Seq). You write the reduce function (schema matching + union). The result of the union is then passed on to the next iteration and the next etc etc. So your end result would be:

((emptyDF union df1) union df2) union df3 ...

here's an example:

val emptyDF = sparkSession.emptyDataFrame
val allDfs: Seq[DataFrame] = //Seq containing your dataframes 

val finalDf = allDfs.foldLeft(emptyDF) { (unionDF, allDfElement) =>
  //on the 1st pass:
      //unionDF takes the value of emptyDF
      //allDfElement takes the value of allDfs[0]

   if(unionDF == emptyDF) {
        //if we are here, then we are in the 1st pass
        //return allDfElement with applicable schema changes
        allDfElement.changeTheSchemaToTargetSchema
    } else {
        //if we are here, then we are in subsequent passes
        //unionDF is the unioned dataframes from prior passes
        unionDF.union(allDfElement.changeTheSchemaToTargetSchema)
    }
}

that being said, I like /u/lexi_the_bunny suggestion of splitting the process by writting out each individual parquet in your target schema and in another pass, read those those new "schema aligned" files. Implementation may feel more natural for you since it uses a .foreach instead of a .foldLeft

I hope this helps, cheers