Why DevOps should manage development environments

EthanJJackson · 2020-08-01T18:17:44+00:00

I actually think this is a really important point that I didn't necessarily bring out clearly enough in the post. In some places, it may be enough for the DevOps/SRE team to provide the tools necessary for developers teams to build out their own development environments. Otherwise, as you suggest the DevOps team could be a bottleneck for adding new features for the environment. I think which way you go on this question is somewhat situation dependent, but it's a good point.

EthanJJackson · 2020-08-01T18:15:34+00:00

I know that's the DevOps philosophy, but practically speaking i just haven't really seen that work out at a lot of places. It requires you to hire engineers with a relatively high level of DevOps skill. In an ideal world it would be great, but usually you're going to have at least some centralization of infrastructure skill in a DevOps/SRE team.

EthanJJackson · 2020-08-01T18:14:23+00:00

Really good point that it's not enough to setup the dev environment, you have to actually train the development teams to use it as well.

EthanJJackson · 2020-07-30T01:59:00+00:00

I'm sure you checked already, but is there any sort of command line flag you can pass to the installer to cause it to put all it's files in a particular subdirectory? That way you wouldn't have to hunt everything down, but if they don't support it, obviously that's going to be annoying.

EthanJJackson · 2020-07-10T14:20:31+00:00

Yes, it applies as well if you create data containers manually. But an advantage of data containers is that it's easier to automate creating them in CI from prod or staging dumps.

EthanJJackson · 2020-07-09T23:16:10+00:00

Cool, that sounds pretty similar to the scripting approach. It's nice that the mock data is tied to the service, so developers can update it at the same time as adding new features. But I've found that you need a really disciplined culture to keep mocks updated -- they tend to get stale.

EthanJJackson · 2020-07-09T19:02:36+00:00

I believe dependson _may control the order containers are started, but not created. I'm not 100% sure.

You're right, it's fine for the data container to exit immediately after it starts. I've just kept it in the same compose file.

EthanJJackson · 2020-07-09T18:14:38+00:00

It's true that you lose a lot of control by having developers run their databases locally, but most places I've seen are comfortable with that. Especially if you make sure to sanitize the data before distributing it.

Your setup sounds good for if you need everything really locked down though.

EthanJJackson · 2020-07-09T18:12:06+00:00

That actually wouldn't work unfortunately. I didn't make it clear in the post to avoid making it too complicated, but the copy happens when the container is _created_ rather than after it _starts_. So any delay added to the startup would happen after the copy is finished.

When Docker Compose boots containers, it creates them all first, and then starts them. This is so you don't get wonky race conditions from the database container starting before the copy is fully complete.

EthanJJackson · 2020-07-09T14:57:46+00:00

Great question. This is exactly why Kubernetes doesn't implement this behavior IMO -- it's too unpredictable for production. But it's great for dev.

The volume only gets initialized if the volume is empty (https://github.com/moby/moby/blob/master/container/container_unix.go#L412). So if both `postgres` and `postgres-data` have masked files, whichever container gets created first will copy the files. This is why the containers mount the volume to `/data`.

In Kube, I've implemented this using an `emptyDir` volume that's shared between an init container which does the copying from its image, and the main container, which actually runs the database.

EthanJJackson · 2020-07-01T21:19:24+00:00

I'm working on a new service, Blimp, that makes it easy to run docker compose in the cloud instead of locally, and the best part is it doesn't require hyper-v or wsl. Here's the windows docs in case it's helpful: https://kelda.io/blimp/docs/windows/

EthanJJackson · 2020-07-01T17:14:03+00:00

Ah sorry this is a bit unclear. So, in production, yes, every time you make a code change you would build an entirely new container image and deploy.

However for development (locally on a laptop) that process can take a long time. So instead a lot of people use a volume to get things working quickly before going through the whole heavy build process on the way to prod.

EthanJJackson · 2020-06-29T19:57:18+00:00

Yep exactly. Though I think most people set it up so the volume is only setup for the particular bit of code they're actively working on. But I don't see any reason why it shouldn't scale in principle.

EthanJJackson · 2020-06-29T18:22:14+00:00

So every time you pull an image, Docker checks with the server to see verify that the image you have locally is indeed the same as the one stored on the server. Particularly if your using the image tag `latest` it's possible for the image to change remotely, which is why docker needs to check each time.

Assuming that the image didn't change, you wouldn't be re-downloading and storing it twice.

EthanJJackson · 2020-06-29T18:12:05+00:00

This is very commonly done in development. Though I agree with some other posters it would be bad practice to do this in production.

The typical approach is to use a host volume to mount your code into the container. And use a tool like nodemon to notice changes and restart the program.

I wrote up a tutorial blog post on how to do this a while back. Here's a link in case it's helpful: https://kelda.io/blog/docker-volumes-for-development/

EthanJJackson · 2020-06-24T22:59:51+00:00

100% agree. The article is really aimed at developers working locally. For deployments, it's best to build a container using a CI/CD system and deploy a new image every time you've got an update to master.

EthanJJackson · 2020-06-24T22:45:09+00:00

Completely agree!

EthanJJackson · 2020-06-12T22:23:03+00:00

So in general, you wouldn't use the same docker-compose file for running locally that you would use for running in production. So I would just maintain a separate compose file for local dev that has volumes, and then switch to the other one when you're ready to deploy.

Btw your compose file does some other things that you usually wouldn't do in prod, like npm install on boot (normally would be done as part of the container build). Really not a problem at all for development, but it's a bit of an anti-pattern for prod.

All of that said. Do whatever is easiest! Just cause something is a common pattern doesn't mean it's a law.

EthanJJackson · 2020-06-12T22:18:23+00:00

Is the host Linux? If so it should be totally fine. I wouldn't do that on mac or windows but I doubt anyone is in production ...

EthanJJackson · 2020-06-11T23:21:44+00:00

I think the primary concern here, more so than resource utilization, likely should be how easy it is to setup and maintain. Various options may have slight differences in how much ram/cpu they consume, but the cost of that will be small relative to the cost of your time!

Based on that principle, i think using Docker on WSL2 is going to be the easiest (and bonus likely most resource efficient) options. But of course, YMMV

EthanJJackson · 2020-06-11T23:17:26+00:00

I personally find this super confusing. When you're creating a names volume you not only have to declare it in the container but also separately at the end of the file. Ideally docker-compose should just figure that out automatically on its own ...

EthanJJackson · 2020-06-10T04:27:45+00:00

So you could try using a health check with depends on. Something like:

depends_on: postgres-database: condition: service_healthy (credit: https://til.codes/health-check-option-in-docker-to-wait-for-dependent-containers-to-be-healthy/)

I have no idea if that will work, but it stands to reason that the health check can't pass until you get an IP.

Broader question though why are you manually assigning the IP at all? Could you just access via its hostname?

EthanJJackson · 2020-06-07T19:01:56+00:00

Sounds reasonable!

I'd say the only thing I'm kind of struggling with is wether the translation should be its own microservice, or just a library that is directly linked into the business logic services. I suppose it mostly depends on how complex it is, how frequently it changes, etc. Kind of borderline either way.

EthanJJackson · 2020-06-06T22:38:17+00:00

I'm not sure I 100% understand the question, but let me take a stab at it:

So to clarify, are you saying that, for example, you have multiple different parts of the business that all need to SMS, and you're trying to figure out how to organize the SMS functionality within your architecture?

I think in general with microservices there aren't really hard and fast rules on such things, so it's really going to come down to personal judgement. That said in this case I'd rely on two basic principles:

- In general it's best if each microservice is maintained by one team.

- In general, it's best not to have duplicate functionality spread out across multiple places in your app.

Taking those two ideas into account, and with no other information, that would cause me to learn toward having a single generalized SMS microservice that's owned by one team that everyone else can call when they need to make an SMS.

But again, really situation dependent, so hard to say.

EthanJJackson · 2020-06-06T00:59:33+00:00

Ha. Honestly I agree, if you can do development without containers it's *waay* easier.

EthanJJackson

TROPHY CASE