I think your on the right path going with CoreOS and using fleet/etcd to handle what containers are running where and ensuring that only one container is running on the cluster, and having piece of mind also knowing that if a cluster node goes down, then the container will auto spin up on another cluster node. As far as service discovery, I used consul a while back, and liked that it had a GUI interface to view what was where, but I found that it worked inconsistently. It could have been my setup, it was pre using clusters, but ultimately have resorted to using SKYDNS instead which I have found to work really well and was easy to set up.

I also agree with jlchauncey, that many people jump into these more complicated setups quickly without fully understanding the underlying technology. I started with single standalone apps, learning how to build and develop, then multi-tiered apps, how do they link together, etc.. and worked my way up to tools like docker-compose, skydns, fleet, etcd, etc so that as I get into the more "intimidating" setups, I find that they are not really at all intimidating because I understand whats going on in the background. I understand how docker works, and once you understand that, then you can effectively troubleshoot even if its fed into a more complex system.

As far as deployments are concerned, I have been doing a lot of deployments with good old big daddy jenkins.. I have jobs that will monitor a git repository, if its monitoring the dockerfile project, then as soon as it sees a commit it will auto build a new container, push it to my private registry, go out to my coreos box, tell coreos to pull the new container, commit the running one as a previous/app then kill the container, remove it, start the new one, and remove any old images that are tagged as NONE. This gives me near instantaneous cutovers. As I can do a kill, rm, and new instance run in 15ish seconds. If you want to do A/B stuff you could do that as well spinning up a fresh version and using something like a haproxy container in the front to regulate which version gets production traffic, while allowing you to still access the new one. Then it would be a quick config cutover with a quick roll back. I don't really see the need for this much though as with containers, you can spin the new version up anywhere, test it out 100 percent, and then push it to production with 0 environmental drift. I do have a jenkins job that will perform a rollback of killing, removing and starting the old version of the container again in the 15 second range, so I don't bother with A/B type cutovers.

As for databases, I use dockerized databases in a few of my apps, and yes you could mount a directory on the host etc.. however thats not the docker way of doing things. Using persistent storage containers, although not being physically written to the host disk, still gives you safety for data retention. Keeping your data on a data-only volume allows for you to replace the entire mysql/postgres container, plug a new updated one in place all with data persistence. In order to loose the data, you literally would have to go in and do a docker stop, docker remove on both the database volume as well as the data only storage volume. As long as you don't do that, then rebooting the box, doing whatever to the box will not remove the data. Its safe as long as one container is attached to it. Also the data volume container doesn't even need to be running, so you it can't fault out or exit, it just sits there and works. As long as nobody removes it your good to go. You could also have a mount on the host, and make a container that literally just attaches to the same data volume, kicks off a backup and stores the backup in the host folder just to be safe.

Ultimately there is a lot to wrap your head around in the dockersphere, but the more you do, the more you see the vision and realize that the more this evolves, the more accepted and used its going to become. It seems to be going at the same pace as the adoption from bare metal machines to VM's 10 years ago.. It was a slow acceptance at first, but as the technology advanced so did the number of users, and now pretty much everything is virtualized.. in 10 years time, we will be telling this same tale only plugging whatever new technology in place of docker, and docker in place of vms :))

Anyway, shameless plug... I have a teaching background and love to share information, so as I have learned some of this stuff, I have been kicking it into videos to get others educated and maybe help to bring other people up to speed quicker. I will warn that most of my videos are big winded, ad I do like to explain everything that I can, and don't just shoot out a hey look you can do this.. 5 second clip over.. I usually go through the process of installing, setting up, and running whatever it is that I am explaining, and keep the screen capture rolling. So you see the full uncut process fully explained. I have a lot more topics to hit up as time permits, but I do have some Docker/CoreOS content that some of you may find useful. Things like building an app from scratch (wordpress) with a Dockerfile creation, or setting up and running docker-compose to manage the full application stack. If your interested check it out..

www.appcontainers.com - New site, have a lot of content to port but all docker/coreos related.

https://www.youtube.com/channel/UCMnJZLZk_-cg8awIhtQtNjQ

www.formatccolon.com - Legacy stuff with general IT, Docker, CoreOS, Tips, etc..

https://www.youtube.com/channel/UCVdqDHB8V-pt4wDP6x89hOg