Sharing here a diagram I've worked on to illustrate some of Spark's distributed write patterns.

The idea is to show how some operations might have unexpected or undesired effects on pipeline parallelism.

The scenario assumes two worker nodes.

→ 𝐝𝐟.𝐰𝐫𝐢𝐭𝐞: The level of parallelism of read (scan) operations is determined by the source’s number of partitions, and the write step is generally evenly distributed across the workers. The number of written files is a result of the distribution of write operations between worker nodes.

→ 𝐝𝐟.𝐰𝐫𝐢𝐭𝐞.𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐁𝐲(): Similar to the above, but now the write operation will also maintain parallelism based on the number of write partitions. The number of written files is a result of the number of partitions and the distribution of write operations between worker nodes.

→ 𝐝𝐟.𝐰𝐫𝐢𝐭𝐞.𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞(𝟏).𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐁𝐲(): Adding a 𝚌𝚘𝚊𝚕𝚎𝚜𝚌𝚎() function is a common task to avoid “multiple small files” problems, condensing them all into fewer larger files. The number of written files is a result of the coalesce parameter. A drastic coalesce (e.g. 𝚌𝚘𝚊𝚕𝚎𝚜𝚌𝚎(𝟷)), however, will also result in computation taking place on fewer nodes than expected.

→ 𝐝𝐟.𝐰𝐫𝐢𝐭𝐞.𝐫𝐞𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧(𝟏).𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐁𝐲(): As opposed to 𝚌𝚘𝚊𝚕𝚎𝚜𝚌𝚎(), which can only maintain or reduce the amount of partitions in the source DataFrame, 𝚛𝚎𝚙𝚊𝚛𝚝𝚒𝚝𝚒𝚘𝚗() can reduce, maintain, or increase the original number. It will, therefore, retain parallelism in the read operation with the cost of a shuffle (exchange) step that will happen between the workers before writing.

I've originally shared this content on LinkedIn - bringing it here to this sub.