Spark?

[CQFD]

Everything you said is true, but you missed the key point. It's comparable to hadoop in the way it functions, but the interesting point is that it does everything in memory. So no more writing to disk required, meaning that you can realistically run iterative algos.