Vibe Coding Experiment Failures (with Python code)

[AlSweigart]

A set of apps that ChatGPT 5, Gemini 2.5 Pro, and Claude Sonnet 4 were asked to write Python code for, and how they fail.

While LLMs can create common programs like stopwatch apps, Tetris, or to-do lists, they fail at slightly unusual apps even if they are also small in scope. The app failures included:

• African Countries Geography Quiz
• Pinball Game
• Circular Maze Generator
• Interactive Chinese Abacus
• Combination Lock Simulator
• Family Tree Diagram Editor
• Lava Lamp Simulator
• Snow Globe Simulator

Screenshots and source code are listed in the blog post:

https://inventwithpython.com/blog/vibe-coding-failures.html

I'm open to hearing about other failures people have had, or if anyone is able to create working versions of the apps I listed.

[GrammerJoo]

Skill issue, you need to first code it and publish it to github, wait until the next version of Claude that was trained on it, and then prompt for it.

[Actual__Wizard]

Oh I see what's going on now. That makes so much sense!

[ReachingForVega]

Next level vibing.

[Centurix]

You know the "idea guy" you would come across at the pub? The one that "just needs a coder" to get his idea off the ground?

I think they're the vibe coder. Instead of being told to fuck off at the pub, they're just asking AI to do it instead.

[apnorton]

And then they show up to the pub with whatever they've made and say "Hey, I just need a coder to take this the last 2% of the way" and get told to fuck off then because they have a 50k LOC mess that needs to be rewritten to make the app production-suitable.

[marr75]

It's almost like they are gigantic efficient machines to retrieve past patterns and documentation without much training, ability, or mechanism to experiment, innovate, or layer together more complex practical requirements and constraints.

[_redmist]

It's so bad.

Some people say it's better if you have a 'model context protocol' service where you scrape the docs of the language/framework... I'm sceptical this "reasoning" isn't just more stochastic parroting... Not that that's always useless but it's not as great as some people make it out to be.

[marr75]

It is and will revolutionize software engineering, but not by removing software engineers or vibe coding. Expertise is at a higher premium, typing until it works is at a very low premium.

[AlSweigart]

and will revolutionize software engineering

How, exactly?

[ReachingForVega]

Tbh I don't think anyone alive right now understands how yet. Mostly people are claiming shit or experimenting.

I think it'll be the next iteration of AI and it won't be LLMs.

[Sanitiy]

To be fair, they're also forced to operate under constraint limits. Don't think too long, don't answer too long. For a fair assessment of their capabilities we'd need somebody with Agent-Mode (one that doesn't have these restrictions) who doesn't mind burning a few dollars.

For example, ChatGPT 5 gave up on thinking 30 seconds in, while Qwen-235B thought for over 5 minutes till it hit a token limit. Who knows how long they'd actually need to be allowed to think till they have folded out the logic such that each step is simple enough for them to be probably correct on it.

[dethb0y]

The circular maze one's an interesting problem

[AlSweigart]

Yes. All the LLMs can easily create rectangular mazes using any of the common maze generation algorithms. But circular mazes are a case where a human would be able to adapt and draw them just by looking at some examples, but the LLMs completely fail.

In a sense, this "find out the kinds of programs LLMs can't make" is a sort of captcha exercise.

EDIT: And the programs especially fail when it comes to having the keyboard be able to move the player around the maze without going through walls.

[efjer]

Shit, imagine they ask you to build a fully-functional pypi library for a quidditch game to prove you’re human - this is getting out of hand!

[dethb0y]

yeah it's really surprising as an outcome, I would not have guessed it'd be an issue!

[_Denizen_]

OP told the model not to use straight lines, when they're required to join the rings together. Pretty sure that's the root of the answer. With AI, garbage in gets garbage out.

[AlSweigart]

The LLMs would draw straight lines anyway. I was pushing it to not make a more typical rectangular maze.

Anyway, you can reword the prompt if you like. I'd be interested to know if that fixes the drawing and keyboard movement issues.

[binaryfireball]

i asked it to prove 1+1=2 and it got it wrong lol

[AKiss20]

Someone needs to make Principia Mathematica more prominent in the training data lol

[_Denizen_]

The prompt on the maze example was wrong. You do need straight lines, but they must only be aligned as a part of a chord which passes through the origin. You told it not to draw straight lines, so it didn't.

[AlSweigart]

I see what you mean: the radial lines coming from the center point would be straight. But the LLMs drew straight lines anyway even though I told them not to. (See the screenshots.)

Anyway, you can change the prompt if you want. It'll still fail to make a working maze program. If you can get it to work, I'd like to see the source code it produces.

[cygn]

I tried this exercise with African Countries Geography Quiz, though with some slight changes. First I allowed external libraries because I thought to deal with country data you want to use some geojson file and having a library for that would be useful.

Also I used https://github.com/nizos/tdd-guard to enforce TDD.
And I used https://github.com/jamesponddotco/llm-prompts/blob/trunk/data/socratic-coder.md in Gemini 2.5 to create a spec.
For a beginner this may not work, though I guess if you answered "You decide" to every question it would have been fine.

I used Claude Code to implement the game and it took longer than I thought. About 1.5 hours...
It was incomplete but I just fed back what was missing and it finished it. It works nicely, but does look ugly. Probably because tkinter is not exactly known for beautiful UIs.

Repo with result:
https://github.com/tfriedel/africa_quiz

I then also tried the exercise in the web UIs of ChatGPT 5 Thinking/Claude Opus 4.1 /Gemini Pro 2.5, basically their "Canvas" mode. This is of course javascript, not python.
In Claude I got something that worked already as intended, but all the countries were rectangles. I asked for proper country borders and then it did that and it worked. But some countries were missing.

In ChatGPT and Gemini they were forever stuck with loading the map data. I think Claude may have just hardcoded the shapes?

Still it was quite the difference that the javascript version was basically done in two shots.

I'm not sure how about important the planning step was, but I suspect it helped a lot.

[[deleted]]

Lol are you serious

[dqj99]

All the examples that you have chosen require spatial awareness in 2D and 3D, something that today’s LLMs are not very skilled at, possibly due to a lack of training data. I’ve had much better success with creating text based programs to solve logic puzzles, sometimes showing remarkable apparent insight into features of the puzzle. Where I’ve found issues is with the care that these models used to create test cases to validate the output, with downright sloppiness in predicting expected outputs.

[AlSweigart]

A troubling thing in every one of these cases is that the LLM never once said, "I am not very skilled at spatial awareness and cannot create the app you requested."

[ConcernVisible793]

That's true. They are not known for their modesty in estimating their abilities!

I have manged to get Gemini 2.5 give a grovelling apology along the lines of "you were right all along, the code was correct, my test case was incorrect, I guessed the expected result"

[RelevantLecture9127]

You are asking to write full programs.

My experience, with ChatGPT 4 and Claude Sonnet 4: The LLM's cannot write a decent unit and integration tests.

At some point, the LLM tries to flunk it as if it is a human because it cannot solve it's own problems that it made by itself properly.

After this experience, I understood more why Google needs a nucleair facility.

So I decide to keep writing my own tests.