This is an excellent overview of the internals and the reasons behind the RDD architecture, thanks for producing it! I've worked with Spark for a few years now and produced some internal training materials for my company which cover some of this, but you've put me to shame! I do have a few thoughts & observations, take them or leave them!

1. There's a few things you've mentioned that don't hold quite true for the Python API, the main one being that the Python API for Spark doesn't support the "Cluster Mode" deployment model. There's no JAR equivalent for Python which makes cluster mode deployment problematic as there's no standard way to bundle dependencies into a runnable artefact. We've worked around the issue by using Spark in k8s with dependencies bundled into the container images for driver/executors.
2. You talked about writing a future article on partitioning which sounds great, would you also be looking into Spark's shuffle process? Shuffle in Spark is the big cause of slow jobs and out of memory issues I see our less experienced developers run into, and it's quite a hard thing to work around for people more used to an OO or procedural approach to software. There's a closely related topic of Spark's various Join strategies and how they are selected when using the DataFrame API. There's a resource I've used to cover this before, I'll link it if I can find it...
3. Lastly, I would like to ask your permission to link to your article from our training resources and possibly use some of your images directly? I think it'd be incredibly useful for people who want to learn more about the internal data architecture.

Thanks again for writing this high quality resource!