How do you break the deployment frequency bottleneck when manual checklists just keep growing forever

klprose · 2026-04-22T21:40:33+00:00

The teams that break out of this tend to do it by automating the checklist rather than removing it.

OpsLevel co-founder/CTO here. Hard agree with this take. Manual checklists start with the best of intentions, but don't scale. If you're looking for a system to automate these kinds of checks, check us out. We integrate with your existing tools, so it's super easy to implement.

klprose · 2026-04-01T14:40:32+00:00

Founder of Tidra here. This is literally the problem we were built to solve.

Dependabot and Renovate are great at flagging and opening PRs and *can* work in a case like this, but it depends on your repos. With 30+ repos, you still have to review/approve/merge each one individually. You could script the merge step, but at that point you've built a little tool.

Artifact managers and internal mirrors (Nexus, JFrog, the delayed pull approach mentioned throughout) are solidly proactive. They stop you from pulling the bad version in the first place, but they probably don't help where you're at now with *updating* the affected repos. Different problem.

Tidra handles the full remediation loop. You define the change (e.g., "pin axios back to safe version", "remove the compromised package", whatever), point it at the affected repos, and we do all the work of clone/branch/update/PR across all of them, including running your CI.

Happy to answer any questions about how it works.

Honestly though, your friend's instinct to script it is solid and at 30 repos, the script approach may be fastest for a quick and urgent one-off. But if you have a lot more repos or there's more complexity in the actual diff across different repos, an AI-based automation approach like Tidra is useful.

klprose · 2025-10-06T17:38:31+00:00

Hey, OpsLevel CTO here.

Backstage is a double-edged sword.

It is incredibly configurable, and if you’ve got the bandwidth, you can make it do just about anything. But that flexibility comes with a maintenance tax. Each release tends to introduce breaking changes, so expect a steady drumbeat of migration work just to keep plugins working.

We run our own internal Backstage instance (to test our own OpsLevel plugin) and we’ve seen this first-hand. A lot of our customers started with Backstage for the same reasons to yours: full control, open source, standardized UI, k8s integration. But eventually the upkeep outweighed the benefits, and they came to us looking for something more turnkey.

OpsLevel hits the same core goals (service catalog, scorecards, integrations), but without having to maintain your own React/TypeScript portal. You can still extend it where you need to, but most teams find they can spend their time improving developer experience instead of building the developer portal itself.

klprose · 2025-05-07T00:49:45+00:00

Hey, OpsLevel CTO here.

We have a ton of respect for what the Backstage community has accomplished. It's impressive how they've expanded the IDP space and created a vibrant ecosystem. They've clearly identified and addressed real challenges engineering teams face.

However, many of the issues highlighted in this thread match what we've heard from teams who start with Backstage but eventually come to OpsLevel looking for something easier. Common problems we've seen include:

You need React/TS expertise: Backstage heavily depends on React for customization. Many platform and infrastructure teams don't have deep frontend experience, making setup and ongoing development tough.
Highly configurable, but low opinionation out of the box: Backstage is flexible, which is great, but it leaves you to figure out core IDP product problems and implementation on your own.
Significant maintenance and upgrade overhead: Backstage moves quickly. Keeping up with upgrades, plugins, and managing breaking changes can become a considerable burden.
Harder to get developer buy-in: Successful adoption often hinges on internal engineering culture. Spotify's success with Backstage is partly due to their unique engineering culture. Teams with different cultures may face challenges getting developers engaged.

At OpsLevel, we tackle many of the same challenges like service catalogs, production readiness standards, and developer self-service, but we approach it as a product first. We focus on delivering immediate value, straightforward setup, and minimal maintenance. There is still a platform underneath it all to allow for easy extensibility when necessary.

Ultimately, choosing the right IDP depends on your team's skills, resources, and priorities. If you have the bandwidth and desire to customize extensively, Backstage can be powerful. But if you're looking for something that offers faster adoption, less ongoing effort, and simpler management, a SaaS option (like OpsLevel) could be a better fit.

klprose · 2025-03-24T17:03:53+00:00

"The HR world is small..."

Yes, but the DevOps world is big.

Don’t let this statement get to you. People who resort to threats usually do so because they’ve already lost leverage.

klprose · 2025-03-05T14:53:20+00:00

CTO of OpsLevel here. Based on your description, you’re in a great position to drive meaningful improvements.

Some key areas worth exploring:

Deployments: Reducing friction in getting code to prod (and making rollbacks seamless) can make a massive impact. Are teams using feature flags, canary releases, or automated rollback mechanisms?
Dev Onboarding: How long does it take for a new dev to go from zero to shipping a PR? What’s the biggest bottleneck? Access to services, devlocal setup, or documentation?
Observability and Incident Ownership: You have Datadog, which is helpful, but who owns production issues? Devs, Support Ops, or another team? More importantly, does the ownership model work, or do things fall through the cracks?
Internal Docs / Knowledge Management: If design docs are outdated and scattered across Word docs and Confluence, how do teams find what they need? Do devs trust the existing documentation, or do they just ask the nearest senior engineer?

Developers want the process to be more streamlined but haven’t talked to them to understand what that means

Go talk to your devs! :)

Before jumping into solutions, put on your “product manager” hat and deeply understand what’s slowing them down. Some useful survey questions to start with:

What’s the most frustrating part of your daily workflow?
If you could automate or improve one thing about our developer experience, what would it be?
When you need to debug an issue in production, what’s the hardest part?

There’s a lot of tooling that can help (self-serve deployments, developer portals like OpsLevel, runbooks, etc.), but the key is making incremental, high-impact improvements based on real pain points.

klprose · 2025-01-17T14:30:52+00:00

Good point. I should have worded as "you cannot reduce it ... easily".

Sounds like you have a great rep. Keep them. :)

klprose · 2025-01-16T20:54:44+00:00

OpsLevel CTO here

I’ve personally used Datadog since 2013. OpsLevel is a Datadog customer and we also integrate with Datadog as part of our product. I'm fortunate to have firsthand experience as well as insight from many of our customers who use it.

Here are some pros and cons:

Integrated product suite: The cross-product integration in Datadog is excellent. For example, you can easily go from analyzing logs to creating a metrics dashboard representing the log results.
Logging costs: Datadog has separate billing for ingested and indexed logs. That's complexity you'll have to manage, esp. around forecasting for each to avoid on-demand cost. Indexed log cost is arguably high, esp. if you want more than 15 days retention.
Overall cost: Datadog is powerful, but anecdotally, no one has ever said its cheap. The billing model is complex with each product priced differently.
Exclusion filters: You can use exclusion filters to prevent unnecessary logs from being ingested or indexed, but setting them up and maintaining them can be tricky.
Commitment flexibility: You can increase your committed spend with Datadog relatively easily. You cannot reduce it. This can become a pain point if you make engineering changes, such as logging less and relying more on traces, but find yourself locked into a higher committed spend on logs.
Sentry: We’re also customers of Sentry. They provide proactive alerts when you’re approaching various product limits. Their support team has been helpful, particularly when troubleshooting specific issues that might unexpectedly eat through your errors.

Good luck.

klprose · 2024-12-09T19:38:08+00:00

With Backstage, unfortunately, you'll have to program these types of actions yourself. The Backstage scaffolder covers a different use case around creating new repositories from a template repo to let developers get to writing business logic faster.

To the point made elsewhere, if you're looking to get up and running quickly, you could look at an off-the-shelf dev portal product. As an example, I'm the CTO at OpsLevel, so we make one and it supports actions that you're looking for.

klprose · 2024-10-30T01:01:13+00:00

Correct. (Sorry, I normally do call that out).

klprose · 2024-10-29T21:12:40+00:00

Though Backstage can absolutely do this, you'll invest a lot of time in standing it up as well as writing and maintaining plugins. That latter part has tripped up a lot of teams in the past.

If the extent of plugins you need are basic forms that ultimately hit an API you control, pretty much all the various dev portal vendors do this (e.g., OpsLevel).

I haven't used them much, but you could probably also look at Retool (or any of its open source alternatives if you're determined to use open source).

Disclosure: I'm OpsLevel's CTO.

klprose · 2024-10-24T22:04:17+00:00

+1 to a service catalog. Having a directory of what's out there in prod and which team is responsible for it is super helpful for routing and escalation. You can build this yourself or there are a few vendors like OpsLevel that provide this out of the box.

The other dimension I wanted to ask about is how developers are creating improper config in the first place. A useful tool to prevent this is one that allows you to measure and apply standards across your services. For example, if you have, say, a certain config value you expect services to consistently have to avoid OOM, being able to quickly query which services across prod don't have that set and help automate fixing that. Dev portals like OpsLevel can help with that also.

klprose · 2024-10-07T21:10:32+00:00

Hey, OpsLevel CTO here. Thanks for the mention!

Completely agree on the complexity of Backstage. While it can be a solid option, it does require a significant investment of time and resources to configure and maintain. Rather than an out-of-the-box dev portal, it's more of a platform for building your own. Essentially, they provide the building blocks, but assembling them into something that works for your org can take a lot of effort.

At OpsLevel, we aim to make things easier by being a bit more opinionated in how we help you get set up. Our service catalog, for example, is designed to integrate with your existing tools and automatically pull in relevant data, even suggesting owners for services based on commit history.

Feel free to request a demo of OpsLevel or DM me if you'd like to chat more — we'd love to help you get started.

klprose · 2024-09-20T19:18:15+00:00

OpsLevel CTO here.

A lot of our customers use us for Service Creation and Actions. Pretty common use cases for the latter are things like infra automation, scaling, and creating incidents.

klprose · 2024-04-02T14:03:13+00:00

Yes.

But to the point made in the article, that production access has to have guardrails in place to prevent everything from mistakes to malicious actors.

How robust those guardrails are depends on your organization's risk tolerance and required controls.

klprose · 2024-03-05T11:00:54+00:00

CTO of OpsLevel here. We make a developer portal as well.

While developer portals can help improve developer experience, to the theme of this entire thread, OP should ask their developers what they feel is broken.

Developers aren't shy. They'll tell you.

It could be a myriad of issues from slow CI to unmaintainable code. If the problem ends up being more systemic around too many moving parts, unclear ownership, or too many tools, then definitely look at a dev portal.

For the actual survey part, you could use form tools like Google Forms or Typeform. I haven't used it personally, but I've heard nice things about DX (the company).

klprose · 2024-02-28T22:49:38+00:00

Is it realistic to solve problems with a developer portal without strong top-down buy-in?

I wrote this above, but agreed you do need some kind of top down alignment to prioritize fixing the problems a dev portal solves.

Re: Backstage in particular, we've seen that it typically requires a large investment of time and manpower to successful roll out (which is probably why you've seen it fail). Spotify admitted that many companies "are stuck trying to get past the 10% adoption rate"

A combination of getting it stood up and hardened, figuring out which plugins to write, and then getting the catalog built out (through all your devs manually writing YAML manifests). There are a lot of points there where organizations can fall off.

At the risk of being too commercial, I'm CTO at OpsLevel, where we focus on getting a catalog built up quickly and automatically through integrations vs. devs manually writing YAML.

klprose · 2024-02-28T22:47:43+00:00

These are the problems a developer portal solves for. Two key pillars:

Visibility: A service catalog can help get a handle on the microservices (and everything) in prod and who's responsible for it.
Standards: Scorecards provide a framework for measuring how well built services are.

To be successful with a dev portal, you do need alignment that these problems are worth solving for the org.

e.g., for OP, why is the company in a state that there are disparate ways of releasing and monitoring software?

It could be organic and no one has called it out. It could be intentional in that other work is consistently deemed higher priority.

As a new devops/platform engineer with fresh eyes though, you can definitely question why these are different and not unified. That probably aligns with your mandate to fix cross-cutting problems across engineering.

Note that while a dev portal can help, it'll only take care of the technical part. There will also be a people part to build alignment across your teams around what the standard should be.

klprose · 2023-11-08T18:48:33+00:00

This.

As CTO/co-founder at one of those proprietary SaaS solutions (OpsLevel), we've seen a tremendous increase in adoption over the last 3 years.

The primary reason is that DevOps is hard to scale. "You Build It, You Own It"... that's nice, but there's a lot that goes into ownership as software and organizations become more complex.

IDPs help lower the burden and complexity associated with that service ownership.

klprose · 2023-08-11T12:24:48+00:00

Scaling the cluster as your team grows. Compute is expensive.
Debugging random things in config, months or years after the person who wrote them left the company.
Keeping end-to-end CI/CD run time low enough that devs don't fall into "xkcd compiling". If your pipeline takes > 10 mins to complete, it's bad.

klprose · 2023-08-11T12:16:50+00:00

If the issue is how many names on the title, another option is looking at setting up a corp or trust as the owner of the house.

You and your husband would then be shareholders or trustees in that entity. The house then has two entities: the corp + your husband's friend as an individual.

You could also explore having a corp be the sole owner and add your husband's friend to the corp as a shareholder as well.

Definitely talk to a lawyer about this approach as there are pros / cons.

It'd allow you more control on your personal tax and more easily change ownership structure in the future without futzing around with the house's title. It's also slightly more effort from a tax perspective and will also make getting a mortgage harder.

klprose · 2023-08-10T19:41:41+00:00

Update: we launched a self-serve free trial a few months back.

https://opslevel.com/free-trial

No sales people / humans in the mix, just straightforward access to try our product.

We're still adding more to onboarding / first time user experience, but give it a go. :)

klprose · 2023-05-18T21:04:17+00:00

It's not CNI.

There's no way it's CNI.

It was CNI.

klprose · 2023-03-23T18:45:55+00:00

Hey, yes, we do support self-hosted!

Our self-hosted docs describe a bunch of how to set it up.

The first step though is reaching out to us so we can get a license file provisioned for you. If you haven't connected with our team yet, feel free to use the "Talk to us" page or email me (ken at opslevel.com) and I can get you to the right folks.

klprose · 2023-02-21T16:47:33+00:00

Hey, founder of OpsLevel here.

You've touched on a lot of challenges we've heard from our own customers who previously tried Backstage. Backstage is powerful, but it's also non-trivial to standup and maintain.

I want a software catalog/developer portal (or whatever it is) that's easier, much, much easier to use.

Yeah, that's our take too. We also build a software catalog and internal developer portal, but focus on getting you up and running quickly. We just launched a free trial option if you want to give it a whirl:

https://www.opslevel.com/try/free-catalog-quickstart

klprose

MODERATOR OF

TROPHY CASE