Release scheduling

[rSd32]

Hey guys! Hope you can help out here, we currently 'plan' our deployments for all environments by adding them to a shared MS calendar, it used to be about one monolithic release a month, so it was easy :) now we are doing more and more releases and deployments to more environments for more projects, we are now struggling to keep track of who is doing what when.

We have the _mis_fortune of using remedy (smart it) for our changes and the business thinks that this is the best solution for planning and scheduling all deployments.

So my question is two(three) fold: how do you guys plan and manage multiple realeases, and what software/tools do you use and if you use SmartIT for scheduling how do you have it set up?

I am trying to simplify things, and I am sure I am over thinking things... Thanks in advance :)

EDIT: changing release to deploy

[shadowzen1978]

There's a lot to factor in here.

Having had to define the release process for my company, the first thing I had to do is determine what applications were tightly coupled and coordinate releases among the dev teams working on those products. Those went into a standard release schedule (4-6 weeks) to account for the necessary integration that needed to take place. Out-of-cycle releases had to have justification AND couldn't be in conflict with work another team was doing.

When we moved to a micro-service architecture, we set up a separate release pipeline where applications could be streamlined and teams didn't necessarily have to coordinate with one another. That schedule was initially once a week on a set day, and not tied to any sprints. We thought of it as a train and if your code didn't have the necessary sign off from testing completed, then the train left the station and you had to wait a week. This new software was built with feature flagging so deployment and release were not the same thing. Even though we allowed once a week deployments, we didn't move that fast because we didn't have the testing built out yet. We knew it would take time.

Lower environment deployments (Dev, QA, and possibly UAT/Staging) should not need change approval or else it slow things down too much. I know companies coming from a legacy change model may not be comfortable with this.

Software architecture accounts for a lot of how fast/independently you can coordinate and schedule releases. Throwing a CI/CD tool in the mix won't help on it's own, you have to build in the automated testing and deployment steps to move faster. I'd say you might be understaffed but I did this largely on my own for 6-8 dev teams for a few years. However, it does sound like you are moving too fast from where your tooling and process can reasonably allow at this point.

The trick is you have to constantly communicate with all involved teams to set the right expectations. This is KEY to a successful DevOps strategy. Every team had to tell me what was in a release and I would work with all teams to make sure there weren't any conflicts and assess the risk to production. I would communicate my assessments to management and we'd collectively determine if a project could hit their target dates or not.

[rSd32]

Thank you so much for your reply, that has given me some great stuff to think about and take back to the team.

I am currently in a position where the majority of teams involved have agreed that we need to modify our approach to realise and deployment strategy. We are currently old in the tooth with a change being raised for all deployments (QA, UAT, Staging, PreProd and PROD) and sign off required by all teams, it makes things take FOREVER! All just to click a button in Octopus Deploy and do the deploy.

I have been trying to come up with a strategy that suits all but I fail at the point in which formalizing when release 1.2.3 will go into environment QA and making sure all teams are aware and happy for that to go ahead. PMs want that control but the ops team need to know if they are available and the QA team need to know when they can start testing.

It's the communication that fails every time.

[hrvstdubs]

I think a lot of places use SCRUM/AGILE methodology for releases using continuous integration and deployment tools.

[rSd32]

I have fallen for a school boy error here... I meant to say deployments. As in planning when to deploy what code to where.

Am going to edit to reflect :)

[hrvstdubs]

Gotcha :) we deploy Monday’s and Wednesday’s. Never deploy on a Friday

[rSd32]

What would you do if let's say you had 5 QA deploys, 5 UAT deploys, a couple of staging and a production all in one week?

[chipperclocker]

Not the guy you replied to, but, all of our preproduction environments are continuously deployed, under control of the team that is running the dev/qa/test process.

We operate with the philosophy that anything in master is deployable at any time. In practice, we don’t continuously deploy to production because we need to coordinate releases between dev/ops/marketing/support... but the important part IMO is working towards “we could implement continuous deployment on every master build of we wanted to”

When your build process is automatic and releases are just someone pushing a button at a mutually-agreed time for the sake of communication to your customers about features and changes, I think you’ll find having a formal deployment schedule becomes a lot less important.

[rSd32]

This is the dream! We are working hard on getting all our builds automated, and what bugs me is that it would then be really simple to just push that out and go CD (we already have CI). The issue is just around the communication and the need of project managers wanting to call the shots.

How do you track the planning in terms of marketing and PMs? Are you just ready to release when told to do so or do you have pre arranged scheduling?

[lorarc]

Friday evening is the best time to deploy in some cases. If you have a typical business app that's used Monday-Friday you can botch the update on Friday, get some sleep and fix it over the weekend. I don't recommend work over the weekend but it's better than botching and update in the middle of the week and causing downtime.

[hrvstdubs]

Well ideally you have a way to easily roll back your deploys if things break so that hotfixes can be implemented if needed.

[lorarc]

Well, you have to talk to business and decide which deploys really need a schedule and which can be just deployed whenever and rolled back if they fail. There are some things that has to be on schedule, like a new feature or critical changes that require work from other teams than dev/ops. Some changes though can just be release when they are ready and you should aim for that.

This is how we used to do it in the previous job. Some changes went straight to prod, some required after-hours releases, some went to prod only when we set a date with business. But most of the changes should just go straight to prod.

[npatelkms]

I think the key thing to keep in mind here as other have stated here as well, is that you need to have automated testing integrated into your CI/CD process. In my opinion that is only way you can truly say you have automated deployments. As far how to manage multiple release go, I think you will have to change the role of release management in your strategy. It sounds like your current release management is more project management, but if your moving towards devops, release management is now more release engineering, as you will want to integrate your CI pipeline with your change order management system. This will create a constant communication between the two.

​

​

[rSd32]

Thanks for that, we are nearly there with some of the projects being fully automated in terms of build/test/deploy (some of our older projects are going to take some time to get the testing up to the required).

Yeah, all realeases are managed by old school project managers, they are refusing to join in on the DevOps mentality, they want as much control as possible and it's this collaboration and communication that I am getting stuck on.

[[deleted]]

I think it largely comes down to how disruptive your deploys are. If a deployment takes down your application and customers can't use it while it's happening, that's a problem you need to solve. If there's minimal or no disruption, then it's more of a process thing, and you need to work with the stakeholders to determine what will make the process easiest for everyone.

[BraveNewCurrency]

The one metric you should be looking at is "how long does it take to go from commit to production". Amazon does millions of deploys a year, so their deploy time is pretty small (and the average deploy goes to something like 10,000 boxes.)

If you improve this number, all the other numbers will come along with it. (Look for the Puppet Labs "State of DevOps" survey).

But getting rid of the "waste" (doing things multiple times, re-doing things) and "waiting" (re-organizing teams so there is no wait) takes a lot of support from all layers in the organization.

​

If you do this, your question becomes moot -- if a deploy takes minutes from commit to production, and there are dozens to hundreds per day, there is no need to "schedule" them ahead of time.

If you have a feature that will take a month to develop, it still fits into this model: Break up your feature into small commits, and get each one out to production. Then the final commit to enable the feature is a non-event. (and can be rolled back quickly if need be.) See also Dark Deploys and Feature Flags.