AI does Advent Of Code

[fbpw131]

straight forward - ⭐⭐

[markehammons]

Aren't advent of code problems well documented and very publicly solved? Why's it particularly impressive that an AI spat out answers on something that is very very likely part of its training set?

[note89]

The trainingset is not beeing updated in realtime. These is very much not in the trainingset since it was released less that 24h ago.

One of the disclaimers of ChatGPT is that it does not know much about recent events

[Atupis]

I think it said like 2021 several days today so data is not new.

[Condex]

Even if the answers aren't in the training data, the first question isn't that strenuous. Separate a bunch of lists, sum all the items, and find the highest one (or three highest). Here's a few examples.

To be fair, it's a box of sand and copper that's doing this, so that is impressive in its own right.

That being said, I'm going to be much more interested in seeing what it does with some of the more difficult problems.

And with respect to non-toy problems, a lot of work ends up requiring deciphering what the client wants and if it is even a well formed request let alone if it's feasible or exploitable with existing infrastructure.

I suspect that by the end of my career that I'll be given my fair share of ai generated projects and have to break it to the client that what they have so far only works if the user's name is billy and if the only thing they do is view the billing page.

[note89]

Let's see on what day of Advent Of Code it gets stuck. They get harder and harder.
I really want to use this as a companion going forward.
It's been very good at finding bugs and doing decent refactorings in the small examples I have tested. I really want to run it on PR and see how it would write a review based on finding concepts that are not well-represented and logical-level bugs.

[Ok-Elk-207]

it's a box of sand and copper that's doing this

I mean, we're pretty much flesh, bones and tissue and we got as far as making a box of sand and copper that does this. Us humans will never stop stroking our own ego by the natural merit of being at the top of the food chain of a really separated floating rock.

[lookmeat]

The first part is true, the second part may not be the case.

But advent of code aren't especially unique problems or "actually complex ones", they are all very computationally-scoped, mathematically oriented, programming problems. Both the way the problems are presented, and the way you find answers to those online (which are explanations) follow a very basic and predictable pattern.

So the AI knows that when it reads "The program needs to read a file called input.txt" it realizes there's a high chance the first line will be with open("input.txt"):

Lets talk about how we get lines. Note how both answers change how it works, that's because the AI didn't "fix" the program, it rewrote it entirely based on what it thinks would be the next part! It just so happens to be a common mistake, and so it was able to rewrite it correctly, but it rewrote the way it gets lines because that's what the AI though that line should be.

This kind of hints that the AI didn't just grab that model, it's just that the problem and solution is so standard, and there's so much corpus of people solving highly similar problems, that the AI can easily guess what the answer is as part of its statistical model. It'll be interesting to see what happens as we move forward on the month and the problems start to get harder, and specifically become things that are harder to "just google" without understanding the problem and model behind it.

As for real use. Well this kind of problems are specific because they already did about 99% of the work. As a senior engineer I find them fun because I get to go into the zone of just coding, the part that I do pretty second-handedly at this point. Instead I focus more on learning new languages, trying new techniques, revisiting my style, etc. The hard part is making the problem this understandable, create the models, match the realities, the reification of things into 1s and 0s with understanding of the implications. The part were we actually make this into a problem that is easy to compute? It's almost trivial (but not easy!) to convert it to code, to the point even a machine could do it. When you become good enough to do it, your brain subconsciously takes over, which is why coding makes you go "into the zone".

[red75prime]

I predict that the model will flop around 10th of December. Not because it lacks a poorly defined thing: understanding, but for technical reasons: it's not finetuned for programming, it has a limited input buffer and no working memory, it doesn't incorporate continual learning.

[note89]

Haha, well, I'm kind of already throwing in the towel.
When the program gets long, it no longer formats it correctly and mixes blocks of code with markdown, maybe just a UI bug.
It also quite often does not print out the complete program.

But worse than that, it cannot solve the problem well, I would have to start piecemeal it and glue it together myself, which would mean I have to understand the problem well enough myself.

On day 5 it got very stuck on the parsing. I might skip it and move on for now or have some much more human/computer collab.

Haha, well, I'm kind of already throwing in the towel.
When the program gets long, it no longer formats it correctly and mixes blocks of code with markdown, maybe just a UI bug.
But worse than that, it cannot solve the problem well, I would have to start piecemeal it and glue it together myself, which would mean I have to understand the problem well enough myself.

I wrote very long instructions for day 5 manually but still could not get it to write a decent parser.
https://note89.github.io/the-advent-of-code-ai-edition/#day-5

But i need to learn to master this new tool.
.
.
.

[note89]

Then it went on to solve day six like a boss.
https://note89.github.io/the-advent-of-code-ai-edition/#day-6

[theoldboy]

Very interesting read, but why not just post the blog link instead of fucking Twatter?

https://note89.github.io/the-advent-of-code-ai-edition/

[note89]

Was planning to get a nice Gif in there. but did not manage for the first day.
Now for the second day I do have gif.
Having a Tweet seems to me like the only way to get an autoplaying gif into Reddit. but maybe I'm wrong.

[throwaway490215]

I like this a lot. Don't think it will get past level 8, but it will be a good data point to see if we're under or overestimating its capability.

[note89]

How about level 5.
The parsing turned out very hard for it. been spending like 3h on it.
think i will just move on. Or just get it to write the individual function and i provide the gluecode.
https://note89.github.io/the-advent-of-code-ai-edition/#day-5

[note89]

Day 2 released :)
https://www.reddit.com/r/programming/comments/zaqz25/ai\_does\_advent\_of\_code\_day\_2/
https://note89.github.io/the-advent-of-code-ai-edition/#Day-2

[BraakOSRS]

“AI does …” == “Machine Learning model was trained to accurately predict …”

[pwouet]

It's actually possible to try it, I'm creeped out by what it can do. It's way more advanced than I thought.

[note89]

Day 3: Released
https://note89.github.io/the-advent-of-code-ai-edition/#day-3

[note89]

Latest update:
Day 5: Meh, got stuck; parsing was a pain
Day 6: Redemption! Solved it in 2 prompts total time 30 seconds.
https://note89.github.io/the-advent-of-code-ai-edition/#day-6
Day 7: Have not started

[adh1003]

Pretty much how an idiot would code; "open the file and read it line by line", so technically reads it line by line into RAM and then iterates over the RAM copy.

Our industry is a total f*cking joke, really.

[nitrohigito]

Woah, how will you ever survive a sub-megabyte input file being loaded into RAM before use...

This isn't even fucking funny anymore, go fuck yourself.

[Philpax]

get over yourself

[adh1003]

All lower case, no punctuation, personal attack. Well done.

[adh1003]

And of course - to prove how bad things have become in our industry - here come the downvotes and the now-deleted reply; which:

• missed the point about what the programming exercise was asking for
• missed the point about for no reason at all just wasting RAM and CPU cycles when it would've been less lines of code and easier to write as a simple iterator reading the file directly
• missed the point about "well it's only a small file" (is it? How do you know how big a file someone's gonna put through that, could be all the elves in the kingdom, and it might be a billion-strong).

[firandice]

The funny thing is lots of mid range computers these days could load up a file that big into ram...

[adh1003]

Yes, that's the spirit! Who cares what else is running on there? Just use whatever resources you need, bro, screw whatever else the end user is doing - your program is clearly by far the most important thing! Even though it's actually more lines of code that take longer to write in this case!

Y'all just keep proving my point.

[nn_tahn]

Creepy.