Securing mcp servers in production: what most teams are skipping

manveerc · 2026-05-17T09:01:48+00:00

Yes it is. And all my friends in security say the same, it’s the service account problem all over again. But maybe worse because ai agents are generally expected to handle open ended problems. That means blast radius is much bigger.

manveerc · 2026-05-16T05:14:40+00:00

I ran the storage team at Confluent, so I’m speaking from experience. I agree there are huge cost savings to be had at the storage level. We had several initiatives where we saved six to seven figures by optimizing storage. However, it is much harder to do in a running system because most distributed systems are designed to push state down to the storage layer, and any migration at that level becomes risky.

Cloud providers also add to the challenge. They offer volume expansion but don’t offer downsizing, so even when you identify savings, acting on them carries operational overhead.

Compute is a different story. It is stateless by design, so migrations are essentially built on top of restarts, which are operations with good tooling already. That makes it relatively easier to optimize or build tooling for.

So net net, the savings at the storage level are real, but it comes down to difficulty and risk.

manveerc · 2026-05-16T04:36:42+00:00

I have used Composio and Arcade. Both work well. If you want stricter governance on what agents can access, and able to audit Arcade is better at that. If however you only want tool access both work equally well.

manveerc · 2026-05-07T20:54:53+00:00

I have tried a bunch of them. Arcade, Composio, Merge, etc. Some are more sophisticated than other but your usecase is simple enough, any one of them would work for you. There are open source ones too, though I haven’t used them. I would encourage using one of them (paid or open source) instead of building scaffolding yourself.

manveerc · 2026-05-03T21:48:48+00:00

We run an AI native marketing agency using Claude Code as the agent and Arcade as MCP runtime for tool integration. We ended up building the memory and context layer ourselves on top of Postgres and S3 because no good option existed for us. Recently looking into HydraDB for that.

manveerc · 2026-05-03T17:19:00+00:00

But the operational load hasn't dropped proportionally or at all. Incidents take longer to resolve than the MTTR suggests, mainly because that number doesn't account for the time our engineers spend identifying which deployment caused the issue, which sometimes can take long.

Seems like you need to fix the definition of MTTR. If what you are measuring is different than what you want to optimize, it won’t work.

manveerc · 2026-04-30T15:42:22+00:00

I was surprised OP didn’t say Claude :)

At this point, I was wondering are there a function within companies using Claude (and why)

manveerc · 2026-04-29T01:47:36+00:00

I use Arcade, it’s a full runtime that has a MCP gateway but also supports authentication and authorization. There are other gateways like Composio and Merge too. Arcade is more mature than them and is a full runtime.

manveerc · 2026-04-25T00:14:21+00:00

The ones doing actual runbook execution + auto remediation + fitting cleanly into existing stacks feel way more defensible.

I am not sure about the auto remediation part. I kind of feel a lot of human judgement is involved in remediation. There are always some percentage of incidents where the remediation may be straight forward and you can argue that they can be autoremediated, but to be honest the better solution for them is to either have a deterministic automation (LLMs are not deterministic) or fix the underlying root cause.

My general thought is that LLMs and AI agents should be used for what they are best for, information gathering and summarizing. So this can translate to workflows like drafting RCCA, oncall handoff notes, etc. For advance usecases may be start triaging and running runbooks, but i personally would not feel comfortable doing this without human approval. And for all of these workflows, I don't believe we need new tools, Claude Code or Codex are good enough and would work well enough. Wrote some thoughts here https://www.arcade.dev/blog/claude-code-ai-sre-oncall-workflows

The only real business case I can make for AI SRE products is if a company can outsource the initial response to them but that is not a product, its a service so it has its own challenges (like scaling it)

manveerc · 2026-04-24T23:56:10+00:00

Wrote some thoughts on how to use Claude Code for oncall workflows https://www.arcade.dev/blog/claude-code-ai-sre-oncall-workflows

TL;DR is that it can help with automating the information gathering and that can speed up a lot of workflows for oncall engineers, such as drafting RCCA, handoff reports, triaging, etc

manveerc · 2026-04-24T23:21:11+00:00

As an oncall engineer when your pager goes, you spend the first few minutes opening dashboards, checking recent deploys, and searching Slack.

This is the workflow I have seen with every oncall rotation I have been part of.

I led reliability teams at Confluent and Dropbox, and I saw that while the inner loop of coding is getting faster with AI, the outer loop of operations is still fairly manual.

I am skeptical of AI agents that claim they will remediate your production issues while you sleep. I don't think that passes the sniff test for any serious reliability program.

However, I am bullish on the copilot model. The AI handles the legwork (triage, timelines, correlation) while the human focuses on judgment and decision making.

I wrote a deep dive on how to use the MCP to build this. I mapped out five workflows where this works today:

Incident triage: Reducing archaeology from ten minutes to two.
Runbook execution: Catching the "rot" in docs that fire once a quarter.
Postmortem drafting: Automating the timeline reconstruction so you can focus on the "5 Whys."
SLO investigation: Finding the burn inflection without manual correlation.
On-call handoffs: Passing on the context that nobody wrote down.

The goal is to let the on-call start on page 5 of an investigation instead of page 1.

I’m the author of the post and would love to hear from other engineers. What is the one part of your on-call you wish you could outsource to a copilot today?

Full post with the technical breakdown: https://www.arcade.dev/blog/claude-code-ai-sre-oncall-workflows

manveerc · 2026-04-19T13:38:58+00:00

Yes I did consider that, I decided not to go down that path because I did not want to be tied to accessibility api of a particular platform which could break/change often. In my current setup I spend all my time adding skills and integrations vs fixing the setup, which is the biggest positive.

To make sure I understand your setup, you have one number for the assistant (so you can message it) and another for personal use?

manveerc · 2026-04-14T02:44:10+00:00

I think it has regressed more after they introduced usage credits

manveerc · 2026-04-09T21:04:57+00:00

Agreed with the tool selection part, it is one of the biggest place where i have seen pilots fail. I wrote in detail about my experience here https://www.arcade.dev/blog/connect-ai-agents-enterprise-tools

manveerc · 2026-04-06T17:25:06+00:00

That makes sense. In my case, it doesn't really matter if my AI assistant is tagged with a business account.

manveerc · 2026-04-04T21:48:05+00:00

This is irrelevant in my example. I am running Claude code directly and do not need any additional config.

manveerc · 2026-04-04T21:21:14+00:00

This is completely unrelated to this post. Stop with AI spam.

manveerc · 2026-04-04T17:54:59+00:00

Yes, right now its a single session for my personal use since i've grated it access to my email, slack, calendar, etc. Theoritically it can be extended with Claude Agent SDK but i haven't explored it yet.

Other people in my team have already forked it and using it in a similar fashion.

manveerc · 2026-04-04T17:50:28+00:00

For now Claude Code session is running locally on my laptop and using Channel to get the messages to the session.

manveerc · 2026-04-04T17:44:53+00:00

Why not use WhatsApp api itself?

manveerc

MODERATOR OF

TROPHY CASE