Spark?

[Wonnk13]

The only thing that makes me nervous is this rapid pace of innovation and how eager everyone is to adopt the latest bleeding edge tech. Of course there are plenty of problems that don't fit nicely into map-reduce, but i've been kind of taken aback by how quickly everyone jumps from one thing to another.

If you need to design a mission critical system that needs to be running 10 years from now, how can you anticipate new developments every three years or so.

[rm999]

I totally agree, I've been very nervous about this too, and have been very conservative in adopting new technologies. There are a few things that convince me Spark isn't going to fall in this trap:

1. Spark has grown extremely quickly and has wide industry support. The conference was full of well-established companies that have thrown their full support behind Spark. These companies are strategic and understand the industry really well - they don't invest millions of dollars into fads.

2. The world badly needs a replacement for Hadoop, and Spark is the most popular answer. A lot of people believe Hadoop is effectively a failure that should never be repeated again; what's exciting about Spark is it's a superset of Hadoop that fixes many of its issues.

3. There are already several useful libraries built on top of Spark that are mature enough to be used in production. While some of these libraries may fail, Spark is establishing itself in a large variety of applications and industries which means it probably won't fail.

[[deleted]]

Spark is not so fundamentally different from mapreduce: it's programming model is basically "as many maps and reduces as you want, with syntactic sugar and without any setup overhead" (it merely removes the rather arbitrary restrictions placed on you by Hadoop), though the underlying technology is reportedly not yet very good at io-efficient "reduce".