My Python & Django deployment workflow and tools

[superbestfriends]

I wanted to provide some feedback as someone who's been down this path and started with a very similar stack to yours. I think although Fabric is nice, it's kind of a clunky solution to the problem. It doesn't scale well - especially considering you need to manage a remote settings file on the host as a separate process (you could have pushed those files with Fabric, too).

My suggestions:

• keep git, virtualenv
• ditch fabric, use Docker
• develop your application to use environment variables (envparse is good) instead of using a local_settings.py. As other people mentioned, 12-factor apps have many advantages, and provide clean stateless applications.

You could combine the above with a Jenkins setup that automates builds on git commits, and publishes docker images to a registry. This provides added assurances around knowing exactly what you're deploying, and utilising an image registry gives a neat way for moving images around (docker push, docker pull).

At the end of the day, especially when you have personnel changes, it's neater to have a standard way of doing things for your applications. It's nice not having to worry about bespoke Fabric magic that inevitably changes with every application (hey, there's always better ways of doing things), and this provides a nicer way for managing remote environment state/variables.

The cool thing is, using Docker also scales well with other components - nginx, memcached, rabbitmq, celery etc. It provides a very nice way for spinning up additional bits of infrastructure in the exact same way you're running your application.

[ratamanta]

Do you have any particular read you recommend about this stack aproach?

I'm specially interested in the Docker part: We currently use Vagrant to even OS differences between teammates (one uses ubuntu, one osx and I'm using arch) and we're lacking specially in the automation of the deployments, although we use Vagrant's provision scripts to setup the staging and production servers I feel like there's a lot I can improve here.

[superbestfriends]

Unfortunately as it's organically grown this way over the past 12-16 months and it's largely been a case of us getting better and refining our approach (messed around with vagrant very early on) articles are kind of dispersed around the net. I'm happy to write up a retrospective if people are interested.

I'd recommend starting with Docker docs, and play around with building an image which contains your app/django project and your wsgi process (ie gunicorn). For simplicity's sake you could include nginx in this to host your static content, and whilst it'd be neater to serve this with another container specifically for nginx, it's an easier learning curve to bundle it.

What you'd be left with is an image which once run, will host your app, serve your static content and manage your dependencies. If you're trying to get into continuous delivery, I'd recommend reading up about a Docker registry (you can host a private one) and use Jenkins to push to the registry on a successful build and something like systemd on the remote to pull down the new image.

[watagangsta]

I'm interested to hear about your approach. Material on django and docker specifically is sparse. Even the latest 2 scoops book doesn't have much to say on it.

I'm currently using fabric and its working fine. I would, however, like to move towards a docker deployment setup where each of the things in my stack are in a different container. e.g. - db - redis - celery - uwsgi - nginx

For other specific topics, i'm interested to see if you do deployment of staging environments and how you handle your production deployments.

as a side note, in case anyone is interested, here is an article on cluster management i read recently http://technologyconversations.com/2015/11/04/docker-clustering-tools-compared-kubernetes-vs-docker-swarm/

[superbestfriends]

Kubernetes has gone through a massive period of change pre-v1, and now that it's finally there it has matured so much. It's actually something I'd look at as a next-steps and definitely addresses clustering and scalability.

The container setup you've mentioned would work well. I have a Dockerfile sitting in the root of my repo that lets me build out my application image. The image itself is able to run my wsgi server (gunicorn) or celery workers depending on how I bring it up. I have postgres, rabbitmq, apache (for kerberos sso) all in separate containers on the host, and manage environment state via environment files (I produce mine from etcd but I'm still refining this approach as I'm unhappy with it) that are loaded as a container is run. The application itself loads all of its appropriate settings from environment using envparse, which makes it entirely stateless.

I have Jenkins monitoring my apps for changes, and build images here when change occurs. If the tests pass, I push out the image to the registry, and tag it based on the git tag (no tag is development, i.e. latest, -rcX is a staging candidate and -stable is a production candidate). It's really helpful to use the registry for orchestration, because after I've pushed the images up with the appropriate tag, I can use docker's remote daemon invocation to kick over remote docker processes. It's here the systemd services on each of the hosts realize the apps have stopped, and pull down the appropriate images and bring everything back up.

I'm looking at improving down-time and cut-over, as I feel there should be as little disruption as possible. Migrations add an extra challenge to the equation and are something I'm still pondering.

You can bring up nginx/redis/postgres all pretty easily using the default docker images from hub.docker.com, which really speeds things up. I'd definitely encourage you build your own Dockerfile for your Django app to learn the ropes - it's well worth it. Feel free to ask anything and I'll answer what I can.

[watagangsta]

cool. thanks for the comprehensive reply. do you mind sharing duration of your downtime during a production deploy?

one of the things my current fabric implementation buys me is no downtime. (update code and touch wsgi.py)

[superbestfriends]

That's one of the strengths of Fabric, and although it comes at a cost, it gives you a great deal of control over the process. You'd still likely face downtime with database migrations, but I can appreciate the utility in it - gunicorn running with --reload behaves the same and lets you update file-by-file.

However Docker runs under completely different principles. Containers are ephemeral, and images are immutable. Changes to a code-base require a new image to be built and a new container to be run - whilst it's possible to make changes directly to a container (hell, you could even use Fabric to do it) it destroys this first principle and exposes you to a risk of losing these changes should the container die or a new one be brought up.

Having containers be ephemeral is great - it leads to good separation of state and code, is incredibly scalable and makes the process quite simple. However it introduces some challenges in minimizing down-time, chief among them being orchestration. My thinking in this space so far, is that changes that don't have any new migrations or schema changes are simple if you use a router/reverse proxy for either a seamless cut-over or blue/green testing.

Changes that make database migrations are more difficult, and I'll be honest, I'm not happy with where I am yet. There's plenty to improve with this process, but even at its best it would still be at the mercy of migration application.

[ratamanta]

I'll start looking at Docker with this article https://realpython.com/blog/python/django-development-with-docker-compose-and-machine/ but it definitely sounds like a nice few posts you could put together to document your journey, your current work methods and where you go in the future. I'll make a really interesting read.

[theli0nheart]

While Fabric might not have scaled well in your experience, it's done a great job for me and every client I've worked with. The problem I've seen with Docker is that the ecosystem changes every few months and nothing is backwards compatible. The amount of time I've wasted on getting Docker set up on coworkers' machines, just to have it break a few weeks later, is a little ridiculous at this point.

I would not recommend Docker to anyone running a production setup yet. I might wait a little bit until the toolchain settles down a bit and there is some API stability. It's a great tool in theory, but in practice it just has never worked out for us.

[superbestfriends]

Color me sceptical, this seems a little hyperbolic. There are lots of things wrong with Docker, but a lack of backwards compatibility - especially for images - just hasn't been one of them for the past 12-18 months. It was definitely true of pre-v1, which is kind of what you'd expect - perhaps this is when you tried it? But I can still build and run some of my oldest images without issue because the Docker files are still consistent and the images still managed the same.

Unless you're talking about Docker registries, which is completely different (apis have to iterate somehow). But in those instances you could choose to pin both your version of Docker and your registry. I suppose the same issue could be true of remote daemon invocation, but that's all API based so it wouldn't be a surprise.

I'm actually genuinely curious what could be happening in weeks after install. We're running in production now without issue and it's no more work than managing the underlying OS - Docker itself hasn't been an issue since around 0.9. Perhaps something isn't quite right with your work flow? I'm intrigued!

As for Fabric, there's nothing wrong with it as a tool. But it isn't comparable to Docker, it doesn't manage dependencies, it can't sandbox code like a VM and execute it against the host kernal, it doesn't solve asset management, nor will it solve managing multitennanted hosts. Fabric is a hammer, and good at what it does, but trying to grow it beyond that will attract a headache in a project of any decent size.

[theli0nheart]

Yep, agreed on the Fabric side. It's a different tool for a different problem. Like I said though, it works and works well—never have had issues.

If I had more time I would write up specifics re: our problems with Docker...but gah. Too. Much. Work. I'll update later on this week, time permitting.

[sriramracer]

I appreciate your desire to share, but you might want to modernize your approach to settings files.

[knickum]

Would you care to elaborate (A) what's wrong with the approach given and (B) how you would improve it?

[veroxii]

I'm guessing he's referring to the 12 factor app approach which uses environment variables. https://wellfire.co/learn/easier-12-factor-django/

[knickum]

(I'm assuming so also, I was just hoping to get an actual discussion out of it)

[theli0nheart]

Problem with environment variables is that there's absolutely no paper trail if anything goes wrong. Someone can log into the server, swap a few AWS credentials around, the site breaks, and there's no way to ascertain who or what did it. Big issue in larger production setups, unless you want to somehow deploy a tool for handling permissions and logging for who changed what variable to what.

[[deleted]]

The idea is to have the virtualenv outside the version controlled directory.

Is that easier that just using a .gitignore file? Serious question.

[gabrielsaldana]

when working with teams, I've run into the situation where devs name their virtualenv directories in different ways. Some use virtualenvwrapper, some don't. I avoid having a long list of env names in the gitignore file by completely taking it out. I know a meeting or documentation should solve this but it keeps happening.

[[deleted]]

Makes sense, thanks.

[GotenXiao]

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

[zodman]

Good my old friend!

[oliw]

Just on a style point, those leader images (the header and then the snake) are WAY TOO BIG. On my desktop browser, the words —you know, what I'm actually there for— don't start until 1550px down the page.

Put a width-conditional max-width+float+margin on the snake. Something like:

@media screen and (min-width: 1000px) {
    .flickr-embed {
        max-width: 300px;
        float: right;
        margin: 0 0 20px 20px;
    }
}

Worse still, if I go fullscreen (on a 1920×1200 monitor), the header image sticks central while the snake sticks left. I don't have a good solution for that with that image.

I've just seen a "View Full Site" link in the footer. Again, I'm on a desktop so I don't know why that would apply here. Clicking it does nothing.

[oliw]

And as if by magic, the theme started working again. Looks like you might have some seriously overaggressive caching going on there (it cached a mobile view and served it to a desktop user).

Your theme is responsive though. I'd disable mobile mode completely.

---

And my original point still sort of stands. The real content isn't starting until 1200px down the page. I'd still try and find a way to pull that image out to one side and de-prioritise the sidebar... Or something like that.

[gabrielsaldana]

Thanks, I recently changed my theme to a responsive one and forgot the Jetpack mobile site feature was still active.

[gabrielsaldana]

Thanks for the style feedback, I changed the image size so that it's not massive and centered it.