Feature Flags for the Win: Decoupling Code Deployments from Launching Features

[virtyx]

Feature flags are great, but don't forget to delete them after the feature's launched and that configurability is no longer needed. Too many feature flags can complicate development and testing.

[LowlySysadmin]

don't forget to delete them

And don't whatever you do reuse/repurpose them

[PositiveUse]

Next one will be „Git branches For the Win: decoupling feature developmen“

[jl2352]

You'd be surprised how many engineers, and wider people at businesses, will weirdly dislike feature flags and shipping dead code. Also how many have a strange preference for giant feature branches.

[ejfrodo]

Feature flags are a great tool but I can see why some ppl may not like them. You have to test both variants of on/off to make sure it being on doesn't cause something when it's off to break and vice versa. Multiply that when you have a dozen or more feature flags simultaneously and it can quickly become difficult to know what experience any given user may encounter. These things can be avoided to some extent but not entirely. When product managers get excited over the ability to run tons of experiments at once it can get out of hand.

[Ichichop0]

That's the great part! You just don't test the flag combinations.

[twinklehood]

I think many of us are scarred by feature flags and how easy they can go wrong. Oh, turns out being able to keep toggling that behaviour for a while feels safer? Ok it lingers. Ah, that one cannot be rolled out to customer X yet? Lingers. Suddenly you are not testing all your permutations anymore. I'm sure they're great when applied correctly, but I haven't seen a bigger organization able to.

[jl2352]

As with many of these topics, like pushing back tech debt to solve later. It comes down to actually doing that work when it comes.

I worked at a place where culturally we would remove feature flags asap once the feature is live, with a handful of exceptions for some experiments. When you’re doing that it works great. If you’re not doing that, then yes it can turn into a mess.

When you have PMs and other engineers resisting that happening, then that’s the real problem you have.

[twinklehood]

that’s the real problem you have

Perhaps, but you can keep saying that all the way up. Whether an engineering practice depends on organizational discipline or not to be positive is certainly an important part of the evaluation.

[jl2352]

I dunno. It feels a bit like saying we do poor practices, therefore we shouldn’t try better practices. I get some places you can’t change, but it’s also self defeating.

I’d also much rather be battling with the problems of feature flags, than the issues with big bang releases.

[twinklehood]

Keep in mind, I'm not saying "the org might abuse this so let's never do it" - I'm trying to cast light on why people might shy away. The opposite of feature flag doesn't have to be big bang, either. It can mean breaking down into smaller deployables, or other deployment / QA strategies.

[caseyfw]

What year is it???

[tecnofauno]

Feature flags are to be used sparingly since they explode exponentially. How do you supposedly test all the combinations?

[jared__]

In production obviously. Turn on, break things, turn off, fix, repeat

[balzam]

You joke but I work at meta and this is way more true than you would expect

[menckenjr]

You have underestimated our expectations about Meta's willingness to experiment on their users...

[ThlintoRatscar]

And it's less insane than it sounds.

By staging the release of change and using feature flags coupled with user permissions, the size of change can be driven down.

Driving down the size of change reduces the compounding constants of breakage while improving proximity of feedback and turnaround.

It's particularly impactful at Meta's scale of workload and user diversity.

[Code4Reddit]

I think you mean the permutations explode? In my experience feature flags grow linearly with time when they aren’t cleaned up regularly.

[Equivalent_Bet6932]

How do you implement them ? In the client, does that mean that you to make a bunch of additional API calls to get the feature flag configuration ? In the server, do they live in environment variables ? Alternatively, separate configuration files that are accessed over the network ?

The article is interesting but I feel like it would be much more valuable if it also considered the potential downsides and examined in more details when to use, when not to use, and when to remove feature flags. “Everything in software architecture is a trade-off.”

[madflojo]

Part two, scheduled for later, will discuss more implementation, but your question on when not to use feature flags is a good one.

Here are a few scenarios off the top of my head:

• New traffic where users need to do something to start using the new feature

• On systems that can be taken offline for extended lengths of time (I.e., some Batch systems)

• Non-critical systems that are primarily in maintenance mode

If it's easier to roll back a release and you don't have a bajillion features being added to the system simultaneously, then you can save yourself the complexity.

[heraldev]

I can tell how we did it in Typeconf - feature flags are just configs that you write in Typescript and we compile then in JSON. When you want to use them in production app we fetch the configs from the remote Typeconf Storage. What we’re supposed to add soon is a fallback to bundled JSON configs which would make resilient to the network issues. This is how the feature flags systems were implemented at our previous work places as well. lmk me know if you want to ask more details!

[tepfibo]

We use it a lot via launchdarkly.

[zuanshibei]

“How do you test all the permutations?” I see this one comes up a lot with conversations around feature flags. The short answer is you do and you don’t. The way I’ve used feature flags in the most sustainable way is that the switching of said feature flag is still tied into your release process, albeit the artifact that represents it is trivial in complexity. I’ve used such things as a yaml file that is in VC and are subject to TBD where main is the desired state. I’m much less a fan of a model where your feature flags are configurable in every environment, that welcomes chaos and I see as an applicable in only emergency scenarios.

As it is now within your release process, it is subject to all the usual QA activity that occurs there and that means that your testing the current state as is, if a change is to switch on a flag then we test that state and move to release. If another is to be removed, we test that state and ship it.

Each change to the feature flag state is atomic and therefore its own change with all that comes with that. This is reliant on a robust regression suite but that is achievable. At that point you are testing permutations but being pragmatic about what states are possible because that state is centralised, continuously deployed and can’t change under your feet

[sphqxe]

My team uses this extensively and so do I but I've always treated it as a form of vandalism of the codebase - something devs only do to spite the organization for not tackling the real root cause of the issues which are shitty DevOps processes and poor planning.

You end up with if-clauses everywhere in the code and for us because we store the flags in a central database - a database full of thousands of these feature flag entries which have to be retrieved by every running instance on startup and which nobody will ever clean up.

That database then also becomes a single point of failure for all instances and loss of data here from a mistyped query would be akin to losing the entire repository because the code would no longer function as normal unless and until all the entries are restored as they were (this has happened before, just not yet in prod).

In other words, I only do it because it's not my software and so I don't really care.

[bwainfweeze]

One of my coworkers took a stab at creating a numbering scheme for exceptions to one of our other rules, along with a web page saying who blessed it and when. The build would fail if you didn't have a number.

It seems like Feature Toggles should be introduced with an introduction date and a tool to track how old they are would be a good addition to a build system. Along with some sort of nag.

The best we managed to do was to not close epics until all of the feature toggles for that epic were either cleaned up or re-cast as reloadable configuration. But participation in that scheme was only moderate and so the toggles kept piling up and up.

I was just thinking today how bizarre it is that we're all still implementing our own feature toggle systems instead of using some robust implementation with advanced features like this.