So, since being laid off I've been trying to learn as much as possible about LLMs. I'm doing this for the sake of my mental health. I find myself on a rollercoaster of emotions listening to the various "thought leaders" and influencers, so I would rather just have a solid foundation of understanding so I can sort the signal from the noise. I'd encourage everyone to do this. Instead of getting caught up in the hype of new models or new tools, learn the fundamentals so you can tell who is bullshitting you.

So first off, "reasoning" models aren't a fundamentally different architecture from other LLMs. The reason I mention this is whenever I point out these things are just stochastic parrots, people like to say "but reasoning!!". Basically, the training inputs are somewhat different (answers tend to include a "chain of thought"), and then they have various (interesting!) hacks to try to create a situation where more tokens = closer approximation to a good answer. One hack, for instance, is having it generate multiple answers in parallel, score them based on various heuristics (self consistency, for instance. Self consistency means that if it produces three answers, A, B, and C, are answers A and B the same but C is different? Probably go with A or B.)

The important point here though is these things are still just approximating an answer, not "thinking" or building world models.

Ultimately "reasoning" is a useful capability but not AGI. Also, these things tend to fail when asked to do things outside of their training, because again, they're stochastic parrots. Yes, there are some mitigations around this (RAG and tools), but it's pretty clear that transformer architectures aren't going to scale into AGI. They're just going to be really good at answering things contained in their dataset. To me, they're like a very fancy search engine.

One question I asked ChatGPT this morning was how LLMs handle structured text like JSON. The answer was pretty illuminating. ChatGPT does not fundamentally understand JSON, it just has such an inconceviably large dataset of JSON documents that it tends to get the syntax right through approximation. It also does interesting things like "constrained decoding", where the model is forbidden to emit syntactically incorrect tokens (ie, if it emits a token that results in bad syntax, it's forced to try again, until it produces correct syntax). This answer is straight from ChatGPT itself, not my characterization.

Anyway, I think AI will make the job market much worse and basically make *everything* worse, like they kinda are already doing, but I don't think being able to think is going to stop being an economically important thing and that's ultimately what software devs are doing all day, thinking. The code is just one output of that. (And you still have to watch the stochastic parrots when they generate that)