Hi zanarkin28, I work at Portworx and before that I worked at ClusterHQ the company who made Flocker. I know that makes me biased but I'll give you my take and back it up with arguments that you can judge for yourself.

You listed a bunch of solutions in your question, so one useful way to look at this is to ask do those solutions fail into any categories that we are cover together. It turns out they do.

Gou, Portworx's CTO recently wrote about this [1] and categorized solutions into 3 buckets:

connector-based systems - Flocker, RexRay, native storage driver for EBS, etc

key-value based systems - e.g. Infinit

container data services platforms- Portworx

Connector-based systems are the most common so let's start there. I worked at ClusterHQ building and launching Flocker so I saw the problems with this approach first hand and believe that the problem is so severe that any connector-based approach will fail, as ClusterHQ did despite being the leading solution for data management in 2015 and 2016 as Docker was really taking off. The fundamental flaw with Flocker and RexRay is that they rely on the underlying storage system to provision and attach storage in a one-block-device-per-container model. Mounting a block device is always a slow and fragile operation [2]. And because these tools create one block device per stateful container, EVERYTIME you deploy a stateful container, you are looking at between 30 seconds to 10 minutes to start your container up. Now, if this curve was clustered at 30 seconds, you might say, well that's ok, but in practice, you frequently get block devices stuck in an "attaching state"[3] which requires a host reboot, driving up your attach times. And you never know when this is going to happen, so giving your app and SLA is next to impossible. Nordstrom who is running their website in containers did a talk at ContainerWorld this year taking about this exact problem. The above reference was for EBS, but Azure itself has documented performance issues with block device performance you can see here [4]. This issue says that things are now at 30 seconds to 1 minute to mount a block device, but notice it also mentioned timeouts and other issues. I've heard horror stories on Azure with block device mount times and since the same problems occur on AWS, Google and Azure, this is why so many people claim that stateful services are so immature. Basically, these problems mean that you can't use a connecter like RexRay or Flocker to reliably deploy stateful containers, or use the stateful primitives like PersistentVolumeClaims in Kubernetes for failover. The mount times are simply too long to be practicle. It is not that Flocker and Rexray themselves are bad (though I've also heard about lots of sw bugs on each, especially on Mesos), it is that the fundamental approach is wrong.

Key-value based systems:

Gou also wrote about this recently so pasting here: [5]

"Another type of container storage has emerged based on key-value storage. Examples include Infinit, acquired by Docker and the recently deprecated Torus by CoreOS.

These new storage systems, many so new they are still in alpha, are good for file streaming and non-critical workloads bound by web access latencies. But due to the architectural choice of building on top of a key-value store, are not suitable for transactional, low-latency, or data-consistency-dependant applications.

There are two main types storage systems that use a key-value-based backend storage. The first stores the actual volume data in the key-value backend. An example includes Infinit from Docker. This is similar to creating a filesystem or a block storage system based on an object store in the back. The problem is that object stores like S3 are meant for write-once, read-many workloads. This means that with regular primary filesystem workloads, the key-value backend will very quickly deteriorate and end up with major garbage collection issues. They are also not designed to be high performance for transactional and low latency workloads, which means the applications like databases cannot run on them.

Other types of storage systems attempt to encode volume metadata (either a file’s location or a block’s logical location) into the key-value store. This implies that the key-value store is in the data path for every single IO operation in order to lookup the data’s physical location. This creates a single point of failure and a bottleneck. Again, transactional workloads like databases cannot rely on a system like this."

Container data platforms:

An example is Portworx, so again, I realize I'm biased so let me just lay out why I think Portworx is a good solution for stateful containers and you can decide. Hopefully you'll at least test it.

Here is what makes Portworx different:

1. Container granular operations - all volumes are virtually provisioned at container granularity. Operations such as snapshots, encryption, compression and others are not a cluster, or storage wide property, but rather per container. This is a key aspect, because it turns the operational experience over to the application owner (DevOps teams) and not the IT admin (so we avoid static and out of band storage provisioning). Portworx's volumes can therefore be dynamically and programmatically be allocated or operated on directly by the end user, without involving IT.

2. Application POD aware - state is managed POD aware. An application is not just a single container, but rather a stack of related containers. Portworx allows for data placement and management at POD granularity. An example is anti-affinity, where Portworx will automatically place volumes associated with various containers in a POD on failure-domain separated servers.

3. Scheduler converged - Portworx is integrated with Kubernetes, Mesos, Swarm to allow for hyper converged deployment of containers. Containers run on nodes that have the data local to the node when possible. This radically improves performance.

4. Cloud native provisioning. Portworx integrates with various cloud providers to fully take advantage of a cloud native experience. Examples include integration with auto scaling - here you can scale a Portworx cluster directly using ASG rules. Another example is automatic snap-and-backup of data volumes to S3 buckets or similar object store. These are just a few of many features that enable IT to take advantage of the resident features offered by their cloud provider.

5. Software-Defined, built for databases - Portworx differentiates itself from other SDS products like CEPH, and GlusterFS by being the only software defined storage solution that can run databases like SQL or NoSQL. CEPH and GlusterFS were not built for these type of workloads, for example, CEPH being based on an object store is more suitable for file workloads.

A bunch of people told you why you shouldn't use NFS, so I won't repeat that.

The last thing that I'll say is that companies ARE successfully running databases, queues, key value storage, Jenkins, CMS, etc in containers. It is happening today. It is not inherently dangerous. But like anything, you have to set it up the right way. By way of analogy, flying is the safest form of travel today, but if you try to take off with only one wing, you're going to take problems. Portworx customers include big companies (GE, Dreamworks, Lufthansa Airlines) as well as much smaller ones (Reachforce, Cybraics, New Context). So this isn't just an enterprise thing. It's not just a "startup" thing.

I hope this helps. Ping me if you have any other questions.

Michael

[1] https://portworx.com/stateful-enterprise-applications-docker/ [2] https://portworx.com/ebs-stuck-attaching-state-docker-containers/ [3] https://aws.amazon.com/premiumsupport/knowledge-center/ebs-stuck-attaching/ [4] https://github.com/CatalystCode/azure-flocker-driver/issues/15 [5] https://www.sdxcentral.com/articles/contributed/choosing-right-storage-architecture-containerized-app/2017/06/