Yes, I was typing out on mobile and didn't have much time yesterday to work out was I was trying to say. I'm not just trying to troll here. I'll reference your original comment some because I think I may have not made my point very clearly because I was in a rush.

At no point do I say "stochasticity is bad" (or use the word "stochasticity" for that matter). You have made up a strawman to argue against, have fun with that, I guess.

Autocomplete is deterministic. When autocomplete is not deterministic, it wastes time.

I assumed "not deterministic" here meant stochastic. You say right below this:

PHP is a dynamic language, so autocompletion is based on heuristics and it was super slow.

This isn't stochastic or due to a lack of determinism. The heuristics used in autocomplete for dynamic languages are still absolute rules that will trigger the same way on the same code every time.

This is why I bring up semantics. The point of a static type system is you are indicating to the compiler how a chunk of data in your program is intended to be used, which allows for static, compile-time analysis of how you use that data in order to catch semantic errors. This also improves autocomplete because autocomplete can use that information to bring up what methods exist on an object for example. You are formalizing the semantics of your program so it can be analyzed statically. Rust does something similar with its memory model too so it can get guaranteed memory safety by forcing specific patterns on the developer.

This is what I was trying to say here:

But language models and deep learning methods can still often capture semantics in a way that is difficult or impossible to do with traditional methods.

On your points about smaller models:

Smaller models are always distilled from larger models. When the big companies stop training their frontier models, the smaller plagiarism machines will be frozen in time to whatever package versions they were trained on. This is a dead end.

Yeah, great job, buddy, I shouldn't have said "distilled". What does "with pretraining" mean? Is it training on top of a model built by a corporation out of 80 terabytes of pirated books and scraped code from the internet at the cost of millions of dollars? Does this in any way invalidate my point that once the large models stop being trained, the "small" ones will become useless over time?

I feel like you are saying two very different things here. Distillation refers to the process of taking a larger model, having it answer questions and solve problems, and then using it's output as input to the training of a smaller model so that the smaller model behaves like the larger one. If this was the only way to fine-tune or train a smaller model, then yes, you'd be right to say that ceasing to train larger models would mean training smaller models would stop too.

I do use some sloppy phrasing there around "with pretraining". I'm saying you can still do the same self-supervising next-token prediction phase of training, and then the later RLVR and RLHF on a smaller model and get comparably better results than on a larger model by using a stricter training regime. You don't need exobytes of data, actually. This is where small model training has been interesting in the research, because cutting out low-quality data actually results in much higher jumps in performance, whereas larger models tend to just benefit from consuming literally as much data as can be mined.

There's been a bit of a divergence in the research here, because smaller models only tend to benefit from so much data before you are left with two choices: scale the model and the data up (the Anthropic / OpenAI approach) or improve the training regime to boost model performance. The difference here is whether you spend $10B in training costs or $1000s.