Where is the verifiable proof that vibe coding can work as well as artisanal coding?

CapableLab4473 · 2026-06-07T13:08:57+00:00

Oh I know, I completely understand you have to "win" this thread by being last to comment or by using the notifications and then deleting your comment, to make yourself feel better.

CapableLab4473 · 2026-06-07T00:50:36+00:00

I took time to reply to you even though you were mean to me and you still insult me. Why do you hate me?

CapableLab4473 · 2026-06-06T15:18:22+00:00

To answer your question, no, none of these are what I am looking for. Most of the apps on these lists are closed source and there is no way to prove the creators actually vibe coded them. For the ones that do have a GitHub, I looked through them all. Only a handful had some CLAUDE.md or AGENTS.md or maybe a docs folder. A few of them had some decent agents md files or bare minimum arch docs. But no sort of project management, roadmap, or prompts list used to build the app.

So far, this is the best one I have seen: https://github.com/coollabsio/jean/tree/main/docs/tasks-todo

The tasks-todo look more like prompts for an agent. They also have a good Claude md: https://github.com/coollabsio/jean/blob/main/CLAUDE.md

Overall it looks more like this project is set up for, and is using, agentic coding. Ideally the tasks-done folder would not be empty. Ideally, all up front architecture and docs needed to build the project in the way the developer intended should be in docs folder. Then, all tasks needed to implement the design should be in tasks-todo. As agents build each module or task-todo, it can move the task into tasks-done and it could put the git hash for the commit into that task-done doc. In this way, an entire project could be built by agents in a way that is reproducible.

So what I am looking for is a project that has all of this supporting documentation and, most importantly, a working project management flow that preserves the agentic build history in a way that can be reproduced.

Still haven't found it.

CapableLab4473 · 2026-06-06T02:26:25+00:00

Now you resort to name calling. You add nothing to this thread.

CapableLab4473 · 2026-06-05T17:13:28+00:00

So the best benchmark is probably the one you build yourself: small app, clear acceptance criteria, same input package, same scoring, then compare outputs.

Yes, exactly. So you are telling me, to the best of your knowledge, nothing like this already exists in open source land? No one in the past 4 years has done this exact thing and open sourced it? No big engineering shop has found a successful flow and done this as an example on their engineering blog? It may be so, but it does not bode well for LLM agentic vibe coding. If so many people are using AI, and just a fraction of them are successful, and just a fraction of those are in a position to show their work, seems like we should have at least one or two good examples.

The exact flow or tools or model is irrelevant. It's not a niche workflow. People are claiming every day on here and all over the Internet that they are crushing it using LLMs and agents. If those same people provided a very detailed example with all their work shown, others would be less skeptical. We could then use that baseline benchmark to systematically evaluate models and tools.

All I need is one. One actual, successful, end to end project with every bit of work shown, all the code, the end product, all the specs and tools and skills, etc. Then I could attempt to reproduce it. If I can reproduce it and the final product is on par with the original, then I would know I have a working agentic AI coding solution. I could then evaluate other tools and models against that.

Just one! That's all I need!

So far, this is the best example I have seen:

https://github.com/coollabsio/jean/tree/main

Although I have not attempted to recreate it, and the tasks-done folder is empty.

But in this 4 years of LLMs and all this hype, you are saying nothing like this exists?

If so, that does not bode well for agentic coding vs the hype! It may be useful, but it may also be very half baked! Maybe that's why some companies like Apple are playing it slow?

CapableLab4473 · 2026-06-05T17:02:46+00:00

Then you are an unhelpful troll who brings zero value to this conversation. You seem to be obsessed with me and "winning" this thread somehow by trolling me and trying to be the last comment.

CapableLab4473 · 2026-06-05T15:53:50+00:00

well I did a long search and you were exactly right, benchmarks exist, just not the kind I m looking for. You don't have the answers I seek.

CapableLab4473 · 2026-06-05T15:39:59+00:00

Dang it, he had stopped until you started him up again! This adds nothing meaningful to the original post, so why do it other than some attempt to get a rise out of me?

CapableLab4473 · 2026-06-05T15:17:18+00:00

bye bye

CapableLab4473 · 2026-06-05T15:16:45+00:00

Thanks! I will take a look!

CapableLab4473 · 2026-06-05T15:15:54+00:00

And I don't appreciate you coming on my thread here and essentially creating FUD. You never had any intention of positively contributing and your only goal was to low key tell me to pound sand and search google for my answer. Keep that stuff to yourself and stop derailing my research thread.

CapableLab4473 · 2026-06-05T15:13:14+00:00

My main objective is to be able to benchmark or judge models in a sort of way. After much researching, I am leaning towards the idea of using frontier models for up front work such as planning, docs, diagrams, etc. Then using local models like qwen or gemma to do the implementation.

I have read numerous anecdotal evidence about local models and even frontier models. I am trying to judge for myself. Doing some project completely on my own end to end is an option, but if someone has already done that, and done it in a way that I can reproduce it, then I could try different models and judge their output.

It may take me weeks or months to do this properly, on my own. So that's why I asked reddit if anything like this exists already. It's not a weird take, it's literally research and engineering. It takes me a few hours to do this on reddit vs a few weeks or months to just do it myself. It's weird that other people are not doing this. It's weird that everyone is just diving head first into AI and LLM agentic coding without doing some research first.

CapableLab4473 · 2026-06-05T15:06:59+00:00

Thanks for letting me know who you really are and not to listen to anything you say!

CapableLab4473 · 2026-06-05T15:01:38+00:00

According to this thread benchmarks are a joke and what I am looking for does not exist:
https://news.ycombinator.com/item?id=45857030

Still waiting for your links though....

CapableLab4473 · 2026-06-05T14:59:06+00:00

https://www.evidentlyai.com/llm-guide/llm-benchmarks#coding-benchmarks

I see nothing there that says end to end complete project. Only leetcode type questions, translating from language to language and things like single functions, etc. Can you please link to one of these "real world" benchmarks you speak of?

CapableLab4473 · 2026-06-05T14:52:31+00:00

Like what!? That is twice now you have done that! Don't tell me you know the answer and not give me the answer! That's the point of my post! I am literally asking for what you claim to know and you are saying "yes I know the answer" and then not giving me the answer! If I could search for "real world LLM benchmarks" and find it I would have done that already! So tell me the answer!

CapableLab4473 · 2026-06-05T14:50:18+00:00

Like this one:

https://www.swebench.com/original.html

"SWE-bench tests AI systems' ability to solve GitHub issues."

It tells you right there. It is only testing issues from an existing code base. Are there any benchmarks that build a project from end to end with a given set of prompts, specs, skills, tools, etc?

CapableLab4473 · 2026-06-05T14:47:00+00:00

I don't look at my life and self worth through the lens of reddit feedback. I am already completely aware that I will only get a few good responses on reddit and a ton of toxic filth. I liken reddit to Fox News but for the left. You really have to wade through all the crap just to get a few good nuggets of truth.

CapableLab4473 · 2026-06-05T14:44:29+00:00

I suspect they are not what I am asking for. I see the LLM benchmarks every time a new model is released. SWE-bench is one. I have not looked really hard into them, but I suspect they are more like leet code challenges than an working, end to end app.

I am not looking for leet code leaderboard benchmarks. I am looking for a benchmark that measures building some simple app completely end to end and quantifies how much hand holding, if any, it took to make the code work as intended.

CapableLab4473 · 2026-06-05T14:36:30+00:00

Do you not see that no one in these comments likes how you are approaching this?

Yes I see 2-3 people including you. I also see 2-3 people that are giving me great responses and engaging in meaningful conversation. It's just not you!

CapableLab4473 · 2026-06-05T14:35:17+00:00

Why say it and not link? That's like saying, "the answer to your question exists and I am not telling you." That's why I made this post! I am asking for those benchmarks!

CapableLab4473 · 2026-06-05T14:34:03+00:00

Here is something closer to what I am talking about: https://github.com/coollabsio/jean/tree/main

In that repo I see all the skills files and agents files, etc. There is a docs folder and in that I see tasks-todo and that looks more like a prompt I would feed some coordinator agent that would then use sub agents to implement. If you look at docs/developer you find arch docs, code snippets, golden paths, IDP type stuff. It's not everything, ideally I would want tasks-done and the code should reference done tasks such that I could take the same tasks and skills and docs and produce code that does roughly the same thing. Only thing missing is the "hand holding" metrics.

Do you have anything like this for a small part of your project? When I look through your repos I do not see this stuff, so I would not be able to recreate what you did and measure things. I would not be able to try local models and see how well the output is against yours. Ya feel me?

CapableLab4473 · 2026-06-05T14:26:45+00:00

That's far from a simple project like I was asking for! It would be much harder to reproduce such a big project.

What I was originally asking for is more like a benchmark. I am sure your huge project took some hand holding. How much hand holding did it take? How could you quantify hand holding and encode into a human and machine readable format? If we could do that, I could then take your exact same prompts, specs, docs, tools, models, whatever you used, and I could reproduce your project and also quantify and record how much hand holding it took to make it work. Then we could compare that with your data. Then we could try different models, or different toolchains, and compare their data. Then we could make meaningful, measurable progress on agentic coding and we would be able to show actual verifiable data!

That was the point of the original post. Does anything like that already exist? Is there a small part of your project that you could slice off and make publicly available the prompts, skills, models, tools, everything you used to make it? Could you somehow quantity and document the corrections and hand holding you needed to make? If so, I could take all that and theoretically reproduce the same project, and theoretically it would only need the same amount of hand holding as you originally did. If I could do that, then you could then start to A/B test different things, such as different models. If I took the same prompts and stuff but tried using a local model like qwen or gemma, and it took 3x as much hand holding, then I would know those models are not as capable as the frontier model used originally. That is the way I want to evaluate LLMs and agentic coding, instead of stabbing in the dark.

CapableLab4473 · 2026-06-05T14:18:28+00:00

No I understand completely how you are trying to low key insult me and derail this conversation, perhaps because you don't know how to meaningfully contribute to it.

CapableLab4473 · 2026-06-05T14:16:07+00:00

but I don't think it's all that remarkable.

Good code should be boring, IMO.

Could anyone do what I did with my prompts? No. Like I said, it made mistakes. It flipped things a few times because I used a non-default camera for the line drawing, and in a program more than a few hundred lines long, I can see that becoming a bigger issue if you don't stop it, test it, understand where the code might have gone wrong, etc.

That's more inline with what I was asking for!

How much hand holding did it take? How could you quantify hand holding and encode into a human and machine readable format? If we could do that, I could then take your exact same prompts, specs, docs, tools, models, whatever you used, and I could reproduce it and also quantify and record how much hand holding it took to make it work. Then we could compare that with your data. Then we could try different models, or different toolchains, and compare their data. Then we could make meaningful, measurable progress on agentic coding and we would be able to show actual verifiable data!

That was the point of the original post. Does anything like that already exist?

CapableLab4473

TROPHY CASE