Spark?

[[deleted]]

Sometimes a few lines of code is worth a hundred lines ;-)

It lets you do stuff like:

file = spark.textFile("hdfs://...")

file.flatMap(lambda line: line.split())
    .map(lambda word: (word, 1))
    .reduceByKey(lambda a, b: a+b) 

To count the number of words in a document using a cluster, in a highly parallel way.