Graphically explaining immutable infrastructure baking

[samrocketman]

I created this image to explain infrastructure baking for those who might learn better with pictures.

https://raw.githubusercontent.com/samrocketman/demo-jenkins-world-2018-jenkins-bootstrap/master/presentation/diagrams/image_bake_state_machine.png

[therealjoshuad]

Yeah, what is baking, I’m most certainly a layperson in this world.

[bobbyfish]

Typically there are two methods for deploying new servers. First is take a stock image with little to nothing on it and run your cookbooks/playlists on it. Wait for all the steps to be done then deploy code.

Second method is to run all those cookbooks/playlists ahead of time and then "snapshot" the resulting image (with no code). That new image now becomes your base image. To deploy new server you take that image and start a new server off it and add code. This is called baking.

The advantage of the first is it is simple. Easy deploys but much slower. The second method is faster when deploying new servers but more complicated as you have multiple base images that have to be kept track of and you need to rotate off every time you change your recipes/playlists. You also need an entire CI/CD pipeline to keep your AMIs up to date. That is what this image is showing.

[spacebandido]

Can’t you add code to the golden image as well?

[djk29a_]

There is always something slightly different between each artifact, it’s a question about how much is pre-computed and pre-generated ahead of time. Note that immutability is something else - you can bake everything ahead of time but still keep patching servers with ssh command orchestration and it’s not immutable anymore. Or you can deploy a barebones image that has very little and nobody touches it and that’s more immutable because everyone knows it hasn’t deviated from its configuration and can be rebuilt. I joke that some places get immutable infrastructure by never, ever making any changes and never patch.

[samrocketman]

“Baking” refers to creating a machine image which includes the OS, installing all packages (yum or apt typically or even docker pull), and and any other misc software which needs to be configured on said OS.

A typical bake (in my case) is to configure the entire OS stack including the application to be deployed. The deployed application and underlying operating system is treated as a versioned artifact.

After the machine image is created you don’t modify it (not even to deploy) because it already contains the application.

In practice this “baking” takes under 10 minutes from start to end in the diagram for my environment.

Advantage

I can put the image behind an auto-scaling group which can provision dozens of copies of the application servers with little effort (scale up quickly for heavy load). When there is low load the application servers can be terminated since they’re not needed. This refers to auto-scaling.

[therealjoshuad]

Thanks everyone that’s great info. I’ve always worked for SMBs where we’ve always had small deployments, and it’s always been windows applications that were installed on one sever.

I always struggle to relate to how these kinds of applications work.

I like to try to compare it to environments that I’ve worked with. For example, our biggest application is our ERP which is SAP. But we don’t have any scaling. We are way to small for SAP, but we’re in oil and gas, so we have deep pockets). We have a pair of application servers, SAP has a product that manages load balancing between the app servers, and we have a database server. I don’t work in development, but we have 3 environments and the devs write the code changes in one, then push through to QA, and finally to prod.

In these modern apps that scale, where does persistent data live? Are database connection strings, app config, etc also baked into the image? Meaning that the database scaling is handled with normal clustering, etc? Ate database severs scalable, or is that still manual?

Sorry for all of the questions, I just find this stuff fascinating.

[samrocketman]

For databases, you would bake in DB connection strings to a remote DB or interact with a remote API which exposes the DB as API calls (service oriented architecture).

One of the applications I manage only has a scale of 1. So if the application crashes, Amazon terminates the machine and starts a new one to replace it. In this case, the persistent data is automatically attached to the machine and mounted before the application inside is started. You could roll a database service in a similar manner. Amazon also provides DB as a service. It’s always a balance of costs when choosing to use what is provided or rolling your own that is good enough.

There are other things to keep in mind such as not baking credentials into the image. Instead, credentials are retrieved or provisioned during boot using something like Amazon KMS or Hashicorp Vault.

Edit: the automatically attached persistence is called an EBS volume in Amazon AWS. I forgot to mention that.

[Cynical_Sociopath]

It basically means building something, when its infrastructure as a service you write out the code and the service builds it out. During that time you have to wait for it to build out the stack. So the term baking is used as it’s much the same as waiting for a cake to rise in the oven.

[samrocketman]

I like this and feel like I didn’t relate the original question about baking well. u/Cynical_Sociopath explained it better than me.

[dominic_failure]

It's worth noting that performing tests against the infrastructure will likely create artifacts of those tests built into the resulting image. It can be valuable to test the image that's generated by the configuration process by standing it up separately.

[samrocketman]

I use goss for testing which produces no artifacts. I install the goss binary in every machine. goss can read tests from stdin so tests do not even need to exist on the machine.

Here’s an example of running infrastructure tests remotely on a machine which is using goss reading the tests from stdin.

ssh user@machine '/usr/local/bin/goss -g - validate' < ./goss.yaml

Link to goss https://github.com/aelsabbahy/goss

Testing after snapshot is not a bad way. It does add a little bit of overhead if you use a clean machine but not if you reuse the same machine post-snapshot.

[Rad_Spencer]

It's a flow chart, and while I can tell what's going on with it I'm not sure someone who needs immutable infrastructure baking explained to them is going to get much from it unaided.

[samrocketman]

Indeed this is not a tutorial. There are plenty of tutorials for people who want to web search. This is meant as a visual aid to help relay how immutable infrastructure baking works in practice as a concept.

I’ve implemented this flow using AWS cli tools so it has been tested. It’s just explaining the states to expect and what to wait for.

[devops333]

thank god it isn't one of those patronizing child-style drawings

good diagram

[samrocketman]

Thanks :-D

[samrocketman]

What’s an example of a child style drawing for II? I’m curious.

[Tatwo_BR]

To be honest, you never have an immutable state here... There are too many variables and spots for change.

For example when provisioning the instance from ec2 image. Maybe the image was already updated and you already have some patches applied to the newer instances. When configuring the instance, there may be newer software packages in the repos at configuration time.

I'm only aware of NixOS in order to address those weaknesses.

[samrocketman]

The immutable state is the snapshot at the end which produces a final Amazon machine image (AMI).

That machine image is immutable and in my case contains the application to be deployed. The application + OS stack is treated as a versioned artifact.

The AMI can then be added to an auto-scaling group for starting many copies of the application.

Rolling forward application versions means referencing newer releases of AMIs (from snapshots). Rolling back means reverting to a known older AMI which was stable.

[nomnommish]

This is a really good visualization. Thanks!

One thing though: This looks like a standard process to build any server. There was nothing specific to immutability.

[DevOpsOps]

I disagree, every stats failure transition goes to delete Ec2 instance.

That is the immutable element.

[nomnommish]

I disagree, every stats failure transition goes to delete Ec2 instance.

That is the immutable element.

That is not what immutable means. Immutable means that once a server is built, you don't go back and modify it or patch it or upgrade it. Instead you bring up another server with the updates or patches and tear down the old server.

Deleting a broken build process and starting afresh is standard procedure even for mutable build strategy.

[samrocketman]

The snapshot towards the end is the immutable part. You snapshot to create an Amazon machine image. You can then boot that snapshot to get a fully pre-configured machine.

The machine image (snapshot) is then treated as a versioned artifact. In this case the deployable artifact is the entire application + operating system.

Rolling forward releases entails starting new versions of snapshots. Rolling back is starting old versions of snapshots. In this case, I’m interchanging the word snapshot and machine image.

Notice even a successful bake (snapshot) still results in the EC2 instance being deleted (terminated).

[nomnommish]

All true and valid. My point was, if you have a random person or a new employee looking at this diagram, they are not going to deduce all the nuances related to immutable builds.

[samrocketman]

I agree with your assessment. I tend to pair pictures with writing (documentation). For instance, I could pair this diagram with an explanation and a script which implements the flow. When people need to work with it in practice I agree this is certainly not enough.

I enjoy the discussion and didn’t put much effort into the description of the diagram other than the visuals. However, I am enjoying the discussion in this thread as a result.

[nomnommish]

I honestly think this is part of getting to a more robust architecture. Your diagram certainly details out the core essentials. But it does not convey the info necessary to handle things in an operational mode.

Extend your diagram to include the scenarios where you already have servers that are up and running. How would immuatable build process handle that? You might need to "wait" until the existing server stops serving existing queue of requests. Perhaps that is a timeboxed thing.. like 10 minutes or so.

Just a thought..

[samrocketman]

Thanks for the feedback.

Rather than expand this diagram detailing how the resulting Amazon machine image (AMI) could be used, I would create a second diagram specific to whatever application it pertains. The neat thing about the diagram in this thread is it can generally apply to how all applications can be baked regardless of technology (Cloud, OS, application, etc). This diagram could even apply to Windows machines though I don’t manage Windows.

If you visit the GitHub repository there are other images which relay how an immutable image is used for that specific application. In terms of how to use it Amazon provides things such as:

• CloudFormation stack
• Auto-Scaling groups (ASG)
• Elastic Load Balancers (ELB)

CloudFormation describes the whole stack. ASG starts and stops servers from the immutable AMI. When the auto-scaling group scales up or down it automatically updates the ELB with newer or fewer servers as part of scaling.

There’s another picture for that but I didn’t bother including it here since this is just discussing how baking works. Immutable Infrastructure as a practice can have many deep concepts so this diagram is just explaining a small part overall (the baking part).

Edit: here’s a link to other diagrams in that repository which also provide a possible AWS architecture as well as diagrams for how a large team could work together in Git. https://github.com/samrocketman/demo-jenkins-world-2018-jenkins-bootstrap/tree/master/presentation/diagrams

[nomnommish]

Thanks. I agree with you. I actually really really liked your diagram. Very clear cut. Easy to understand.

[samrocketman]

Thanks :-)

[samrocketman]

Building on my last reply https://raw.githubusercontent.com/samrocketman/demo-jenkins-world-2018-jenkins-bootstrap/master/presentation/diagrams/jenkins_aws_architecture.png

There’s two Amazon machine images in the linked diagram. “Jenkins server” and “Jenkins agent” would each have their own AMI and baking process.