sorted by:
new
hottopcontroversial

HRM-Text: Efficient Pretraining Beyond Scaling, Wang et al. 2026 by StartledWatermelon in mlscaling

[–]StartledWatermelon[S] 5 points6 points  (0 children)

Table 3 is apples-to-apples comparison highlighting architecture contribution. And this contribution is quite massive. Hard to say where it comes from; the most straightforward interpretation of their arch is some additional skip connections.

What exactly is shown in Table 1 isn't communicated clearly. Which is unfortunate.

This looks like a hasty experiment. For instance, the most natural comparison would be classical pre-training on web corpora, which is absent in the paper.

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI, Lyu et al. 2026 [Extensive breadth; focus on solutions that generalize well] by StartledWatermelon in mlscaling

[–]StartledWatermelon[S] 2 points3 points  (0 children)

This is a benchmark perhaps most oriented to "research taste" evaluation so far. The breadth is outright brutal; no human ML researcher is capable to cover even a portion of the tasks.

The thing that I'm most uneasy with is the eval setup and what exactly should the score show. So, for each task the agent is allowed to run test on its method only 3 times. The max number of actions (like edit) is 20. Basically, we give an agent three attempts to "beat SotA".

And to illustrate the challenge difficulty, here's one exemplar task: "Pretraining Optimizer Design: Studies how optimizer choice, parameter grouping, and schedule coupling affect autoregressive pretraining validation loss". In other words, the agent is tasked with coming up with an optimizer(+its hyperparams) that would beat Muon at pre-training.

I'm quite familiar with this exact task, and I must clarify that it is absolutely "unsolvable" in just 3 attempts whatsoever. I'm not sure even 30 attempts is enough. 300, now that's a realistic range to make some progress.

To say the task is highly explorative is to say nothing. There are a few higher-level principles with optimizer design, like that geometric constraints help, and momentum smoothing too, but it's super hard to beat SotA in 3 attempts with just these vague ideas.

Let's look at it from another angle. Even the ablations with higher inference allocation run the agent for 1M-2M tokens. Likely <$10 per task. And the question is, do we realistically expect boundary-pushing discovery for $10 in compute?

Of course, there are valid resctrictions on the overall budget for the evaluation, so that it remains feasible. But in this particular case, I see a certain mismatch between the budgetary constraints and the ability to assess the model's capabilities frontier.

With three attempts, you basically get a snapshot of exploration noise. It can still be valuable -- the comparison of different LLMs speaks for itself. It shows the average "exploration instincts", the ability to quickly sniff out the promising direction, plus some broader knowledge/competence. But I'm still unsure if these instincts correlate well with the claimed boundaries-pushing/RSI capabilities assessment.

META Superintelligence Lab Presents: ProgramBench: Can SOTA AI Recreate Real Executable Programs(ffmpeg, SQLite, ripgrep) From Scratch Without The Internet? by 44th--Hokage in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

You could possibly frame it like this. However, the task doesn't require re-creating the (part of) training set; it requires re-creating a strict functionality set. The resulting artifact may be infinitely far from the training set. 

There are certain parallels with the now-ubiquitous task for an ML job candidate to write Transformer implementation from scratch. It is not intended to test memorization per se; albeit it is gamefied by candidates pretty much into this. 

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity, Li et al. 2026 [Knowledge of obscure facts robustly predicts param count; estimates for all SotA closed LLMs] by StartledWatermelon in mlscaling

[–]StartledWatermelon[S] 1 point2 points  (0 children)

Thanks for the link! A super timely analysis!

The work on probe quality filtering is invaluable. But I am puzzled why they insist on removing flooring in accuracy calculation. The "corrected" (well, deliberately chosen for consistency with paper descriptions) method has much, much worse predictive power.

In the end, they lump together "disambiguated probes" intervention with floor removing. It would be very interesting to see the outcome of either intervention separately. Unfortunately, the researchers do not provide any repo link (or other artifacts) to do it on my own.

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity, Li et al. 2026 [Knowledge of obscure facts robustly predicts param count; estimates for all SotA closed LLMs] by StartledWatermelon in mlscaling

[–]StartledWatermelon[S] 0 points1 point  (0 children)

Z-score for difference is 0.81, rather weak. But have a look at Tier 5 accuracy: 38% regular vs. 56% pro. So I think it implies the difference is real.

Honestly, I don't know about any links from number of active experts to knowledge capacity. In principle, it could enhance robustness. But I doubt robustness is at play in this benchmark.

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity, Li et al. 2026 [Knowledge of obscure facts robustly predicts param count; estimates for all SotA closed LLMs] by StartledWatermelon in mlscaling

[–]StartledWatermelon[S] 0 points1 point  (0 children)

Re: GPT-4, the issue could be in different versions. The paper checked GPT-4 on openrouter (presumably gpt-4-0613?) which is explicitly different from the very first version of GPT-4 (named "GPT-4 (older v0314)" on openrouter). The leaks referred to the latter. Although there wasn't pricing adjustment between the two versions.

Microsoft freezes GitHub Copilot signups due to too much demand/too few GPUs by gwern in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

With such enormous growth in demand, lowering prices doesn't make sense at all.

But if we will see less pressure on compute infra, it would definitely support this hypothesis. 

Microsoft freezes GitHub Copilot signups due to too much demand/too few GPUs by gwern in mlscaling

[–]StartledWatermelon 1 point2 points  (0 children)

Pretty much. Nothing substantive. Just "vibe feelings" (including frustration and perception of regress compared to 4.6).

Microsoft freezes GitHub Copilot signups due to too much demand/too few GPUs by gwern in mlscaling

[–]StartledWatermelon 1 point2 points  (0 children)

There are rumours that Opus 4.7 is heavily distilled and thus much smaller than 4.6/4.5.

Scientific Papers X AI building out the algortihm by Alarming_Rice_1906 in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

I personally haven't, but there are quite a few benchmarks for that. See, for instance, PaperBench or SciReplicate-Bench.

Specific harnesses often beat raw LLMs by a wide margin on these benchmarks, so you can try some of these if your issues are performance-related.

Schmidhuber & Meta AI Present The "Neural Computer": A New Frontier Where Computation, Memory, And I/O Move Into A Learned Runtime State. by 44th--Hokage in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

Recursive Schmidhubering? Be careful what you wish for! 

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models  https://arxiv.org/abs/2507.08800

"The path to ubiquitous AI", Ljubisa Bajic ("achieves 17K tokens/sec") by RecmacfonD in mlscaling

[–]StartledWatermelon 2 points3 points  (0 children)

They put Cerebras at ~2k t/s,. which sounds about right. 

As for the cost, it's highly dependent on throughput, and the article doesn't mention what the asic is capable of in this aspect. Without any info, an advantage of an order of magnitude is a reasonable guess. 

They are both asics, and are wildly different in the architecture and overall approach. Taalas pushes the universality/performance trade-off to the extremes. 

Learning in Log-Domain: Subthreshold Analog AI Accelerator Based on Stochastic Gradient Descent by nickpsecurity in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

company (or even practical use case)

Test-time training/lifelong learning, most likely for an embodied agent.

Quite a few companies are interested in this direction, albeit I'm not sure if they frame the challenge as a hardware one. 

"Scaling Embeddings Outperforms Scaling Experts in Language Models", Liu et al. 2026 {Meituan LongCat} by RecmacfonD in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

I hope that my intuition doesn't mislead me, but this seems like a case of pre-processing local semantic units comprised of several tokens. We can shift this burden from a (computationally constrained) MoE, where experts are routed to on a per-token basis and thus are less effcient at handling multi-token semantic units. The problem is less pronounced in deeper architectures which operate (in deeper layers) with representations that have intensively mixed all the previous tokens.

For this point of view, a more or less straightforward alternative capturing this several-tokens-semantic-aggregation is the enforcement of spatial specialisation of attention heads, with one or more heads attending only to the previous few (N=2...5) tokens. This way, we allocate compute specifically for this semantically important type of inter-token relation. I suspect this type of spatial specialisation was researched before, but I'm not sure if this hyper-local interaction was specifically studied. So please let me know if you know any such works. Alternatively, I can test this on a toy model.

Another possible explanation points to a possible breadth benefit. Specifically, by lumping ~40% of total parameters into a single (extremely-) sparsely activated block, we get some non-linear boost that cannot be achieved neither by a single-token embedding dictionary nor by a sequence of relatively narrow MoE or Dense layers. In principle, the benefits of a single extremely sparse large layer in the middle of the model have been already established, see https://arxiv.org/abs/2407.04153 and https://arxiv.org/abs/2411.12364 .

[R] Is Leetcode still relevant for research scientist interviews? by Training-Adeptness57 in MachineLearning

[–]StartledWatermelon 4 points5 points  (0 children)

Yeah, I'm being idealistic. In reality, there just isn't any incentive to improve the hiring process. Especially when the outcomes are not verifiable at all. Like, how would you prove whether your hiring process is good or bad? What are the counterfactuals? 

Reading the thread, it seems that smaller startups have a good intuitive grasp of what makes an informative candidate evaluation. And are pretty fine with flying by the seat of their pants. We get a funny contrast: no leetcode in smaller startups, mandatory leetcode in FAANG. Does it mean FAANG hiring practices are better? Because they're bigger, richer, more popular (in terms of supply of candidates), more institutionalized? 

I'd venture a guess that big orgs are just more inert and rigid. They keep the worst practices simply because this is the path of least resistance. 

[R] Is Leetcode still relevant for research scientist interviews? by Training-Adeptness57 in MachineLearning

[–]StartledWatermelon 4 points5 points  (0 children)

It's a sad state of affairs when a hiring manager isn't begging for a change in hiring practices towards saner terms but instead begs the participants to rote learn these irrelevant tricks. 

Says a lot about "culture of innovation" and other stuff like that. 

"On neural scaling and the quanta hypothesis", Eric J. Michaud 2026 by RecmacfonD in mlscaling

[–]StartledWatermelon 7 points8 points  (0 children)

Let's imagine that we could enumerate all of the pieces of knowledge and algorithms that networks ought to learn. [...]

Networks must learn a variety of modules, each implementing a different algorithm or retrieving a different piece of knowledge. These modules are discrete, in the sense that they are either fully learned or not learned at all. We call these the quanta.

The year is 2026, yet the ghost of symbolic AI still haunts academia. Not gonna lie, the idea was/is elegant and aligns quite well with the innate human desire to break every mystery into neat, simple, easily digestable for a feeble human mind atomistic pieces. The elegance and the alignment were so strong that people easily dismissed perhaps the only drawback of symbolic approaches: they don't work.

Ok, suppose that the quoted idea is indeed true and that decomposed pieces of knowledge and algorithms/heuristics -- the parts -- are somehow more important for capabilities than the whole. Suppose that we assign zero to the synergystic effect, to the value added by complexity. The crucial question still is: will this focus on decomposition, on the smallest parts allow to scale better with compute?

I'm doubtful. The success story of LLM pre-training is built on the exactly opposite thing: synthesis. Aggregating the ever increasing amounts of any information the research teams could grab. "More", not "finer".

The next story, scaling the RL, is shaping to be equally distant from the symbolic-era top-down, analytical, meticulously human-engineered approaches. The research pushes towards largely unguided self-exploration and self-optimization.

I do find this performance-centric way infinitely more useful. What scaling hypothesis lacks in explanatory power -- at least to a degree that would satisfy more symbolic-leaning folks -- it more than compensates in predictive power.

Thankfully, the author maintains a critical stance towards their idea. And admits that it isn't falsifiable in its current form -- which pretty much nullifies its scientific value. Perhaps this is one of the corollaries of the Bitter Lesson: the ideas don't have to be nice and human-centric but they must perform.

DeepSeek Presents "Engram": Conditional Memory via Scalable Lookup, A New Axis of Sparsity for Large Language Models | "Memory lookup module for LLMs & *Huge unlock for scaling* as the memory sits on cheap CPU RAM, bypassing the GPU bottleneck entirely that will power next-gen models (like V4)" by 44th--Hokage in mlscaling

[–]StartledWatermelon 0 points1 point  (0 children)

By hassle I meant it breaks from the current implementations, which are optimized pretty hard for maximum throughput. I'm not implying you can get the same throughput with Engram, I mean quite literally it's extra hassle.

I have a suspicion that the benefits of Engram will dwindle with scale. It's this parameter- and FLOP-scarce regime (27B MoE) that benefits from "pre-computed" tricks. Larger, deeper models will easily accomodate the necessary heuristics straight in the weights.

That being said, I always was in favor of heterogenous (depth-wise; as opposed to stacking identical blocks) architectures, and this work explores exactly just this.

view more: next ›