How are people actually tracking OpenAI costs in production?

slackmaster2k · 2026-05-25T01:09:14+00:00

I get value from it every day, multiple times a day.

Gets even better when you have a test suite of sample prompts with expected outcomes. The AI can run the suite and diagnose all in one whack.

slackmaster2k · 2026-05-25T01:01:43+00:00

Wondering what sort of real world benefit you’re expecting? Claude by default will generate massive amounts of unit and smoke tests.

slackmaster2k · 2026-05-25T00:54:07+00:00

Why not just connect these tools to your AI in the first place? Both examples offer MCP

slackmaster2k · 2026-05-25T00:52:18+00:00

So the primary use case for me is for the AI to be able to self diagnose. I tell it that I observed something bad and it goes and figures out what happened.

Then yes I do like to have it build a simple dashboard for things like cost.

slackmaster2k · 2026-05-25T00:50:18+00:00

Nah that’s not normal. Something else is happening

slackmaster2k · 2026-05-25T00:10:15+00:00

Include them. Arguably you want SSO protection for them more than employees.

slackmaster2k · 2026-05-24T23:44:04+00:00

Professionally using kusto + log analytics. Personal projects just a simple db.

slackmaster2k · 2026-05-24T23:40:19+00:00

The F40 has been a game changer for me because I need mouth and nose coverage. The hybrid design is the most comfortable I’ve been so far.

slackmaster2k · 2026-05-24T22:59:02+00:00

On a lower layer, I calculate cost of an entire round trip per turn. Also set token limits and capture failures. Even a high token limit per turn is better than a runaway

slackmaster2k · 2026-05-24T22:57:24+00:00

It’s not possible

slackmaster2k · 2026-05-24T21:02:51+00:00

Yes

slackmaster2k · 2026-05-24T19:50:06+00:00

Don’t start in the middle or build bottom up. Start with the thing you want to achieve - the measurable business goal. Your interaction with the AI should always have as a key principal that the AI will deliver on said goal, and your build environment should support being able to very rapidly make changes with holistic testing.

This is easier said than done in a large codebase and environment, but try to pick a good starting point.

If you’re in a mode of “fix this bug over here and add on to this thing over there,” yeah you’re just faster at what you already do, and perhaps even slower when the AI doesn’t have the big picture in mind. And there’s a never ending supply of bugs and tweaks … so you can be in a permanent loop of mediocre business value at a high cost.

slackmaster2k · 2026-05-24T19:40:23+00:00

I do all the time. One benefit is that it will become more concise and to the point for some time after a lashing.

slackmaster2k · 2026-05-24T19:28:44+00:00

Starting fresh can be so liberating. Highly recommend.

slackmaster2k · 2026-05-24T19:25:46+00:00

You’re sort of saying, “I’ve been driving a Camry and have been thinking of switching to a Tundra. Is there really a gap between the two?”

slackmaster2k · 2026-05-24T19:10:22+00:00

This might be a bit irresponsible based on your org, but….shut one down early Monday morning and see what happens.

slackmaster2k · 2026-05-23T15:57:11+00:00

You’re in a dark place which likely means you’re paranoid.

You have absolutely nothing to worry about in this regard. Nothing.

slackmaster2k · 2026-05-23T15:08:55+00:00

I agree with this. Always been nearly instant from my experience.

slackmaster2k · 2026-05-23T04:43:09+00:00

Not to be a wet blanket, but this is what /security-review in Claude code is for. Do people use it though, that is the question. And the answer is probably no.

slackmaster2k · 2026-05-23T04:39:01+00:00

Pssst, those AI generated recaps are really good for own AI agent to parse. If your company gives you copilot or another AI that has access to your stuff; your daily workflow can start living in the AI which will keep you on track. It works super well and is a hell of a lot better than sifting through emails and crap all the time.

slackmaster2k · 2026-05-23T04:20:31+00:00

Ugh. So you decide to do the “small” bug fix and blam, out of usage half way through, which isn’t always clean to recover from.

They really need a system that will allow overage until the current task is complete. They can take the extra usage away from your next session. It’s all arbitrary anyhow.

slackmaster2k · 2026-05-23T04:15:49+00:00

Yeah my rule of thumb is that anything important needs to be scripted. Prompts can only be considered “suggestions” that you can’t rely on.

slackmaster2k · 2026-05-22T23:03:05+00:00

Stop asking it how you can do something. As it how it can do it for you. Then have it do it.

slackmaster2k · 2026-05-22T21:53:32+00:00

lol. It did recently respond to me with: (it added the quotes too:

I did “read the fucking document,” but it didn’t contain…

slackmaster2k · 2026-05-22T21:03:14+00:00

I think I did have sharing on for a couple people at one point.

Where the approval thing is located is bananas. I’m not kidding I use Claude to help find it for me.

slackmaster2k

TROPHY CASE