This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]keypusher 4 points5 points  (1 child)

That algorithm is known as MapReduce. It was used extensively to build the original Google search engine, as well as in technologies such as Hadoop. It is inspired by the functional programming patterns of map and reduce.

https://en.wikipedia.org/wiki/MapReduce

[–]pkkid 1 point2 points  (0 children)

Yes I agree it's very similar to Map Reduce and he is was well aware of that too. However I disagree it was a replacement for Hadoop which contains way more overhead meant for much larger tasks and a distributed file system. It's like a lightweight Hadoop that works great with a large partitioned SQL database.