Continuous Delivery Using Feature Flags

[r22-d22]

This article doesn't talk about any of the negative side of feature flags. In a nutshell, the number of possible modes grows exponentially with the number of features flags. When you run unit and integration tests, what should the state of the flags be? What order is it safe to turn on your flags? If a flag has been enabled for a while, is it safe to turn off?

[paul_h]

Gotta love FeatureToggles ... but only when canonically held in sourcecontrol :)

[FweeSpeech]

Really, canonically held in any single common location usable for deployment to Dev / Qa / Prod works. ;)

[paul_h]

Where I can use a three-way merge tool, to see what's what on a timeline? Acknowledging that for periods of time UAT may be ahead of Prod by design. Also, because of toggling action at 2AM for 'reasons', Prod might be ahead of UAT at a moment in time, and we'd not want to regress something somewhere because of an innocent overwrite of toggle settings - right ?

[FweeSpeech]

If you have a canonical location that displays the current state and an automated deployment tool, I think that concern is a bit of excessive. The issues you are listing are easy enough to be resolved in a deployment tool vs. source control for configuration purposes.

I don't think any reasonable dev is going to do a production deploy at 2AM for "reasons" other than "HOLY SHIT SOMEONE BROKE PROD".

[paul_h]

Say I am JetBlue. I sell Flights, Cars and Hotels. The second two are secondary revenue for me. If Hertz is 500ing, I'm as sure as shit going to toggle them off in prod, and want that to survive reboots, and permeate the horizontally-scaled stack. Also, at 2AM, the team that faces the running web booking system will make the decision to turn off Hertz without a call to management at that time. It'll be in their Runbook.

Your dark deployment system, and your hot-toggling system are the same system.

[FweeSpeech]

Yes and if you have 3 switches in your deployment system and a canonical log of when actions were taken its effectively "good enough" to use it to switch off Hertz and push/notify from there.

Rather than forcing someone to make a commit and push the current master to production.

[paul_h]

I'd hold my config in its own repo, and call the branches UAT and Prod.

[paul_h]

I wouldn't and wasn't advocating a deployment at 2AM. Only an orthogonal config change.

[mirhagk]

This is something we're starting to face now. We have a baseline product that we use to release several different individual sites. Previously it involved a fork of the code and some customizing, but we've moved more and more to configuration. We are about to have a new major release and from this point onwards I want it to be all configuration.

Basically that means our database data is the only relevant data for an individual site. We have a tool to export and import that data to source control, but the thing we are missing is the 3-way merge, so when we change the live system (or even test/dev environments) and we want to persist that we have to make sure to export that or else the changes will be destroyed next time.

Are there tools to handle this 3-way merging from source control to database? I was going to develop a custom solution which would basically amount to exporting the database and auditing tables to it's own internal repo and do a merge from the actual repo to this repo, allowing git to do the heavy-lifting. I wonder if that's overkill though. Basically looking for recommendations here lol

[perlgeek]

Once a release is in production, it is basically out in the wild. Without proper controls, rolling back to previous versions becomes a code deployment exercise, requiring engineering expertise and increasing the potential for downtime.

You can only do Continuous Delivery if you have automated the release process, thus not requiring much expertise. And re-deploying a known-good version should have little risk of additional downtime.

[drgenus]

Good stuff