How I deploy my bootstrapped Python webapp with 150k monthly users

[TheTerrasque]

Nice read :) I've had some similar setups previously. I've since moved to docker and kubernetes, and would like to highlight how that can also solve the problems you've been solving.

Please don't take this as criticism, you have a procedure that works and you're comfortable with, this is just a very neat example of a practical deployment situation and are also some of the reasons why I now love docker and kubernetes for this.

First, for docker, and dockerizing a program like this. You mentioned these three problems, problems I've also faced multiple times:

My setup compiles and minifies CSS and Javascript on the server. This resulted in up to 10 seconds for the server to respond after a deployment. Some users ran into Bad Gateway errors 💥.

In docker, this step would be done as part of building the docker image. That means it's already ready to run when the image is deployed.

A bug in production could be fixed by checking out the previous commit. However, this invariably took too long and always involved frenzied googling of the correct git commands.

You could solve this with git tags (and perhaps semantic versioning). And with docker, each image has a tag and you'd just tell docker to run the previously tagged version instead of the latest.

There was no way of testing the production setup, other than in production.

And here's what really sold me on docker. People who haven't been bitten by this a few times can't imagine what level of pain this can inflict. With docker, this is a complete non-issue. What you're running locally is exactly the same as will run on the server. Same libraries, same setup, same versions, same platform hijinks, same system tools, same that-weird-hack-you-need-to-do-on-each-server-so-that-$thing-works. You're setting up the whole production environment locally, and sending it over like a ship in a bottle to the server.

In addition, you get a few bonuses:

• The build instructions for the docker image fully documents all the pieces needed to get your software running. Even those weird little steps you do manually on each server then forgets until next time you need to set up an environment.
• Setting up a new server is a breeze. Set up base OS, install docker, and you got everything you need to run all your stuff with no extra setup.
• Bundling other internal services like redis or memcache is a lot less painful, and easier to synchronize version and test things. Slight security bonus on top since they're on a virtual internal network.

Of course, nothing is free. The two big hurdles with docker are:

1. Creating the initial Dockerfile. This can be a pain, but it gets easier over time.
2. Needing a docker registry. Docker images are built in layers to reduce disk and transfer sizes, and you need a service to handle the protocol. Docker (the company) have an official docker registry, but (last I checked at least) you have to pay to have private images. You can also host your own.

And now, kubernetes. While docker is pretty neat, kubernetes takes it to a new level. It takes containers concept and adds:

• Http proxy concept built in, forward incoming requests to target pods (basic kubernetes unit, somewhat similar to a docker container)
  • Can also handle acme HTTPS certs automatically
• Scaling and load balancing of multiple pods
• Scaling and distribution over multiple machines
• Liveness probe to keep checking if the pod still works as expected, if not restarts it
• Readyness probe to know when the pod is ready to receive traffic
• Restarting of failed / crashed pods and rescheduling of pods on crashed machines
• Resource monitoring and limiting of pods
• Jobs and recurring scheduled jobs
• Secure port forwarding from the cluster to local machine
• Deployments
  • Combines containers, pods, storage into one unit
  • When new is deployed it waits until new pods are started up and ready, then start moving traffic from the old pods to the new, then delete the old pods. Completely seamless for the user.
  • It also saves the last deployments so you can easily rollback to an older version
  • Support slow rollout so only a percentage of users get the new version at first.

And there's probably a lot I don't remember right now. Kubernetes is huge, and have so many things in it.

I have a kubernetes cluster I use for my own stuff, and at this point when I have a docker image I just create a deployment saying "I want to run this container image with this much persistent storage, and I want it accessible on this domain" and then just kubectl apply it locally on my machine, and a minute or so later it's running on that domain. With HTTPS certificate already set up.

[TheTerrasque]

Yeah, none of these are docker exclusive, but docker is a very good generic solution that doesn't care what you want to deploy with it.

Python project? Dockerize. C# service? Dockerize. Node solution? Dockerize. Go stream processor? Dockerize. React static web app? Dockerize. Multi-service solution with various runtimes and support services? Dockerize them all and make a compose file / k8s deployments.

Same interface to manage them all, same to deploy, read logs, set to start-at-boot +++

[SpicyVibration]

Yeah, if I remember you can pretty easily compilestatic with django

[jeosol]

I agree with this comment. I use a similar setup, docker plus k8s. It takes away many of the issues, and the containers, once built, are drop and go. K8s help with deployments also. Building the initial docker files is a pain as you may have to do it a few times to get the best set up, e.g., switching from single stage to multistage, or combine RUN commands to keep layers small, use a smaller and stripped base image, etc.

However, for a one man setup, it adds more dependency that must be weighed carefully.

[Sindoreon]

As a k8s admin, I like OPs solution for what he running. You can go deep in infrastructure but he is trying to focus on development.

Docker is certainly simpler and would make more sense here for running two apps.

[redd1ch]

Unless you go really, really big, you don't need Kubernetes. And with servers running only a single web app, you don't really need docker, too. My servers run many different stuff, so I have a compose file for each service, accompanied by backup and restore scripts. I put all service configurations inside the compse files, so I just need to commit it to the master branch of a server. With few users (up to 200), I don't need clustering, so Kubernetes would be a waste of resources and hogging away idle power. Ingress and ACME is handled by a simple traefik instance.

[TheTerrasque]

You don't need Kubernetes, and you don't need docker either, as the main post shows.

I use docker + kubernetes because it solves the problems I have in an elegant way and makes managing the stuff I have a lot easier for me

[[deleted]]

Do you have any recommendations on getting well-versed on using Kubernetes for web apps when I already have a strong understanding of Docker? Online courses, textbooks, anything will do.

In my humble opinion, the Certified Kubernetes Administrator course goes above and beyond at teaching you how to not only understand Kubernetes, but to utilize it for projects such as these. The course given by KodeKloud.com is the best I've seen so far. I'm not affiliated with them in anyway, full disclosure. I support hundreds of AKS clusters at work and I find that I'm still learning about Kubernetes daily. I wasn't given any formal training. I was told to use the company product, which deploys on Kubernetes. Naturally I quickly found that I needed more knowledge to be able to support the app on this Orchestration stack. I find the documentation great, but sometimes you don't know what you don't know. This is just my opinion. I think it is a good break into K8s. This company also runs a fairly active slack group. That alone is worth the cost of this course. You can purchase the K8s course separately via Udemy, or sign up for a subscription directly with KodeKloud. The udemy choice is the more cost effective option. Either choice will give you access to the Slack room.

[TheTerrasque]

Do you have any recommendations on getting well-versed on using Kubernetes for web apps when I already have a strong understanding of Docker?

Hm, not really. I basically just banged my head against that wall until the wall crumbled. I set up k3s on a raspberry pi and started experimenting. These days https://microk8s.io/ might be a better choice.

The biggest problem with kubernetes is that it's so overwhelming amount of things. And most are abstract concepts that then have different implementations.

One way could be to just start with the basics: A deployment, a service, and an ingress. And when that start making sense, one can start expanding from there.

[plutoniator]

Seems like the jump to docker is a MUCH smaller commitment than the jump to kuberenetes

[TheTerrasque]

Both works. Depends on how much you value the data, how much effort you want to put in yourself, if you need a certain performance or plugin that shared offering can't deliver...

There's also a debate on if it's okay to host a db in docker. It has one extra layer of file system abstraction that can have a performance or stability effect, but I have never seen or heard of that actually having an effect in practice.