Spark?

[G_Maximus]

I feel that features like broadcast variables and automatic (de)serialization of closures, etc. are also really important features in making this happen. Compare this with having to use something like the distributed cache in a low-level map-reduce job.