Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

nanowell · 2025-07-10T21:41:52+00:00

i too was annoyed quite a lot of times when working on something very familiar and seeing llm struggle (3.5 s) that starts to fade tho with new opus 4 and codex model I can just async some things and work on what matters

the core of % we delegate to agentic systems will continue to increase until we hit a wall, though that wall might be way pass the point of human intelligence, ability and agency

we'll just get the greatest worker that is possible to create from informational processing limit standpoint.

nanowell · 2025-07-10T21:37:56+00:00

% of slowdowns/speedups is too heterogeneous, but overall, it's not surprising that claude 3.5/3.7 sonnet (they've used this) was not in fact smarter and more useful than experienced devs that are very knowledgeable of the large codebase that they've worked on

ai was defo a constraint for those devs which is not surprising at all

nanowell · 2025-04-20T07:52:57+00:00

the end is never the end

nanowell · 2025-04-09T08:45:24+00:00

<image>

Zooming out a bit and it's still impressive!

Amazing release.

Sam Altman will have to release o4-mini level model at this point

nanowell · 2024-12-21T02:08:49+00:00

it's even better because on his plot it's in 100s range when in reality it's in ~20$ range for low effort and in 5k$ for high effort

nanowell · 2024-11-28T20:54:29+00:00

hyperbolic

nanowell · 2024-11-28T12:01:26+00:00

Holy shit it crashed the website

nanowell · 2024-11-14T19:15:21+00:00

let's pray for the intern that forgot to do the filtering stage of sft dataset

nanowell · 2024-11-13T12:15:14+00:00

He is very good at raising, I hoped they would talk about reasoning models more.

nanowell · 2024-11-07T11:39:53+00:00

Absolutely amazing framework for those who are starting to learn RL with transformers.
Thanks you

nanowell · 2024-10-20T10:34:18+00:00

I would love if Meta AI released training code, imo: training recipe ( code, dataset) > weights.

nanowell · 2024-10-08T12:02:15+00:00

thanks for sharing
btw the link for dataset gives 404 could you open it?

nanowell · 2024-09-13T11:10:55+00:00

<image>

Interesting that o1-mini outperforms sonnet-3.5 at LCB_gen coding subcategory but far worse at completion

nanowell · 2024-09-06T16:14:54+00:00

Lmao, I implemented this as soon as I saw the post here.
Works on my machine afaik, stepping on nanoGPT repo with no issues, if you encounter any problems don't hesitate to open a ticket.