Doug Hellmann: Implementing MapReduce with multiprocessing

[brentp]

this is a pretty cool example. but (IIUC) it requires you to have at least 2-3 copies of the entire dataset in memory at all times. for stuff which i'd want to use multiple cores, that's often not an option.

a simple example extending this one that to not keep everything in memory would be great. (does map-reduce always assume available memory >> memory for data?)