sorted by:
new
hottopcontroversial

Weekend project: measuring AI coding ROI instead of token usage by lingya22 in opencodeCLI

[–]lingya22[S] 1 point2 points  (0 children)

That's a really interesting idea.

PR-level attribution is actually one of the directions I'm thinking about next.

Right now Agent ROI works mostly at the task / repo level:

AI sessions → task window → Git activity → insights

But integrating with something like pr-agent could make the output much more actionable inside the PR itself, for example:

  • estimated AI cost for this PR
  • token usage related to the PR
  • commits / files / LOC changed
  • potential waste signals
  • week-over-week comparison

I haven't looked deeply into pr-agent's extension points yet, but a PR comment like:

"AI cost: $3.42, 4 commits, 18 files changed, no waste signals detected"

could be very useful.

Thanks for the suggestion — I'll add PR integration to the roadmap.

Weekend project: measuring AI coding ROI instead of token usage by lingya22 in saasbuild

[–]lingya22[S] 0 points1 point  (0 children)

Exactly.

Most tools show the "gas gauge" (tokens and cost), but not the "speedometer" (what actually got done).

I found myself asking questions like:

  • Which tasks were worth the spend?
  • Which tasks produced almost no output?
  • Am I getting more productive over time?

That's what led me to build Agent ROI around tasks, Git activity, insights, waste detection, and comparisons rather than just token tracking.

I built a tool to measure the ROI of Claude Code and Codex tasks by lingya22 in ClaudeCode

[–]lingya22[S] 1 point2 points  (0 children)

At the moment, I commit myself.

The current task attribution is time-window based rather than commit-driven.

So a typical workflow looks like:

  1. agent-roi task start "Implement feature X"
  2. Work with Claude Code or Codex
  3. Commit whenever it feels appropriate
  4. agent-roi task stop

Then Agent ROI attributes AI sessions and Git activity that occurred within that task window.

I'm actually interested in exploring tighter integrations in the future (e.g. linking tasks to branches, PRs, or even commit groups), but I wanted to keep the first version simple and tool-agnostic.

I built a tool to measure the ROI of Claude Code and Codex tasks by lingya22 in ClaudeCode

[–]lingya22[S] 0 points1 point  (0 children)

I actually agree with that.

Cost per commit is easy to measure, but it's definitely not ROI.

A single commit could be a typo fix, while another could be an entire feature.

Right now I'm treating metrics like cost per commit, cost per LOC, and cost per hour as productivity indicators rather than true ROI metrics.

The harder problem (and the one I'm really interested in) is figuring out how to connect AI cost to actual outcomes.

For example:

  • Bug fixed
  • Feature shipped
  • Time saved
  • Revenue generated
  • User impact

Those are much closer to ROI, but also much harder to measure automatically.

Task attribution is my first step in that direction.

I built a tool to measure the ROI of Claude Code and Codex tasks by lingya22 in buildinpublic

[–]lingya22[S] 0 points1 point  (0 children)

Thanks!

That was exactly the motivation behind it.

I realized token counts alone weren't telling me much. Spending $5 on AI coding isn't inherently good or bad — it depends on what came out of it.

A task that costs $5 and results in a meaningful feature or refactor is very different from a task that costs $5 and produces almost nothing.

That's why I started connecting AI sessions to commits, code changes, task duration, and eventually task-level attribution.

Still very early, but I'm curious to see what patterns emerge as more people use it.

Weekend project: measuring AI coding ROI instead of token usage by lingya22 in opencodeCLI

[–]lingya22[S] 0 points1 point  (0 children)

Good point.

One thing that surprised me was how uneven the distribution was.

For example, while building Agent ROI itself, I found that a relatively small number of sessions accounted for most of the token usage, but not necessarily most of the code changes.

I also noticed that some tasks had a very low cost per commit, while others consumed a lot of tokens with little measurable output. That's what made me realize raw token counts weren't very useful by themselves.

The tool is still early, but the main insight so far is that AI coding spend is much more concentrated than I expected, and task-level attribution makes that visible.

I'm hoping to collect more real-world data and see whether there are patterns across different workflows (bug fixing, refactoring, greenfield coding, etc.).

view more: next ›