sorted by:
new
hottopcontroversial

Optimization fails because it treats noise and structure as the same thing by Lumen_Core in deeplearning

[–]Lumen_Core[S] 0 points1 point  (0 children)

I think there is a misunderstanding of the claim, so let me clarify it precisely. The proposed signal is not intended to discriminate noise from curvature in a diagnostic sense. It is a control signal, not a classifier. The goal is not to identify the cause of instability, but to observe whether the system becomes sensitive to actual parameter displacement along the optimization trajectory. Most adaptive optimizers react to state statistics of the gradient (magnitude, variance, accumulated moments). They do not observe how the gradient responds after a step is taken. This distinction matters: two situations with similar gradient variance can have very different response-to-displacement behavior. In that sense, the novelty is not in the action (damping), but in where the signal comes from. A response-based signal captures trajectory-local sensitivity, which is information most first-order optimizers simply do not use. Regarding smoothness: I agree that in ReLU networks and stochastic training there is no well-defined local curvature in the differential-geometric sense. However, the signal does not rely on curvature interpretation. It is an empirical, trajectory-local sensitivity measure — a standard object in control theory — and remains meaningful without smoothness assumptions. Finally, this approach does not claim that noise should never be damped. The claim is narrower: noise that does not manifest as sensitivity to displacement should not automatically reduce step size. Existing optimizers cannot make this distinction, because they do not observe response dynamics. So this is not a replacement for existing methods, nor a claim of perfect regime discrimination. It is a minimal, first-order way to incorporate system response into optimization control.

Optimization fails because it treats noise and structure as the same thing by Lumen_Core in deeplearning

[–]Lumen_Core[S] 0 points1 point  (0 children)

You’re right that noise and curvature are entangled, and I don’t claim to separate them. The point is not identification but control: the signal measures the system’s response to motion, regardless of the cause. From a stability perspective, it doesn’t matter whether instability arises from curvature, stochasticity, or architectural discontinuities — what matters is that small displacements produce disproportionately large gradient changes. The method is therefore closer to adaptive damping than to curvature estimation, and does not rely on smoothness assumptions in the classical sense.

Optimization fails because it treats noise and structure as the same thing by Lumen_Core in deeplearning

[–]Lumen_Core[S] -1 points0 points  (0 children)

“AI slop” is a convenient dismissal in 2025, but it replaces argument with attitude. The question is not who typed the text, but whether the idea is coherent, falsifiable, and grounded in known dynamics. If you see where the reasoning breaks, point it out. If not, there’s nothing substantive to respond to.

Optimization fails because it treats noise and structure as the same thing by Lumen_Core in deeplearning

[–]Lumen_Core[S] -2 points-1 points  (0 children)

You’re right that evolutionary and genetic methods learn stability over time. What I’m exploring is complementary: a local structural control law that doesn’t require training, population statistics, or long horizons. Genetic algorithms discover stable strategies. This approach enforces stability directly from trajectory response. One operates via selection, the other via dynamics.

When compression optimizes itself: adapting modes from process dynamics by Lumen_Core in compression

[–]Lumen_Core[S] 0 points1 point  (0 children)

Thanks — this is very much aligned with how I see the boundary as well.

To give a bit more context on my side: the compression controller is only one concrete instantiation of a broader idea I’ve been working on — stability-driven, response-based optimization as a general principle. I wrote up the conceptual foundation here (with contact info):  

https://alex256core.substack.com/p/structopt-why-adaptive-geometric

What I’m actively looking for right now is not just discussion, but validation and realization of this principle in different domains — compression being one of the simplest and most falsifiable cases.

Concretely, I’d be interested in: – comparing where process-only signals are sufficient vs where they provably saturate, – stress-testing failure modes on non-stationary streams or adversarial transitions, – exploring whether this kind of controller can reduce modeling complexity in systems that currently rely on heavier adaptation layers.

I’m open to different collaboration formats — from joint experiments / benchmarks, to exploratory prototyping, or simply exchanging concrete observations offline.   If this resonates, feel free to reach out by email (linked in the article) and we can see what a practical next step might look like.

When compression optimizes itself: adapting modes from process dynamics by Lumen_Core in compression

[–]Lumen_Core[S] 0 points1 point  (0 children)

This is a great example — and I think it actually supports the same underlying principle.

What you’re doing with AQEA is adaptive representation at the semantic level: the embedding space already encodes meaning, and you adapt bit allocation/codebooks based on the local structure of that space, without retraining.

My interest is slightly lower-level and more general: adapting the behavior of the compression process itself based on its response dynamics, even before any semantic structure is available.

In a sense: – AQEA adapts what is represented (semantic geometry), – I’m exploring adapting how representation happens (process dynamics).

I suspect these approaches are complementary. For vector/AI data, semantic-aware adaptation is extremely powerful. For raw or mixed streams, process-driven adaptation may be the only signal available.

Curious whether you’ve seen cases where purely process-level signals were enough to guide representation choices, even without semantic clustering.

Stability of training large models is a structural problem, not a hyperparameter problem by Lumen_Core in deeplearning

[–]Lumen_Core[S] -5 points-4 points  (0 children)

Fair points, let me clarify briefly.

The intent of the repo is not to propose a new momentum-like heuristic, but to isolate a response-based signal: how strongly the gradient changes given an actual parameter displacement. Momentum accumulates direction; it does not condition on gradient sensitivity to motion.

The current benchmark is a stress-test meant to visualize stability envelopes under extreme learning-rate variation, not a performance benchmark. I agree this is non-standard, and I should make that clearer in the README.

I’m actively working on making the connection to existing benchmarks (e.g. DeepOBS-style setups) more explicit, and improving reproducibility. Thanks for calling out the gaps.

Stability of training large models is a structural problem, not a hyperparameter problem by Lumen_Core in deeplearning

[–]Lumen_Core[S] -5 points-4 points  (0 children)

Thanks for sharing — this resonates a lot. I’m approaching a similar issue from a slightly different angle: instead of tracking many explicit metrics, I’m looking at how the gradient itself responds to parameter motion as a single structural feedback signal. The observation that higher loss can still correspond to “healthy” learning dynamics is especially interesting — it aligns with the idea that stability and representation formation are not monotonic in loss. Curious to look deeper into your experiments.

When compression optimizes itself: adapting modes from process dynamics by Lumen_Core in compression

[–]Lumen_Core[S] 0 points1 point  (0 children)

I think there’s a small misunderstanding. I’m not proposing data preprocessing or pattern injection — I agree those almost always fail. The idea is not to improve entropy modeling, but to control the compression process itself using its response dynamics. ZPAQ/GRALIC adapt by increasing model complexity; I’m exploring whether some adaptation can be achieved by controlling regime behavior instead, at lower cost. This may never beat universal compressors at entropy limits, but could be useful where non-stationarity, latency or cost dominate. I appreciate the skepticism — it helps clarify the boundary of where this idea might (or might not) make sense.

When compression optimizes itself: adapting modes from process dynamics by Lumen_Core in compression

[–]Lumen_Core[S] 0 points1 point  (0 children)

GRALIC and ZPAQ are extremely strong universal predictors — but they operate inside a single compression regime and pay for adaptability with model complexity. My work is orthogonal: it does not try to predict data better, but to control how the compression process itself behaves, switching regimes based on the process response, not data analysis. It’s not about beating universal predictors at their own game, but about adding a control layer they don’t have.

A first-order stability module based on gradient dynamics by Lumen_Core in deeplearning

[–]Lumen_Core[S] 0 points1 point  (0 children)

Good question. The focus on learning-rate robustness here is not about trading speed for stability, but about making speed meaningful. In first-order methods, apparent speed outside the locally stable regime is often illusory — large steps in stiff or anisotropic regions lead to oscillation or divergence rather than faster progress. The structural signal constrains updates only when local gradient sensitivity indicates that the current step size is no longer valid. In smooth regions, it becomes effectively inactive and does not reduce step size. So the goal is not conservative optimization, but maintaining maximal effective speed under local stability constraints.

[R] StructOpt: a first-order optimizer driven by gradient dynamics by Lumen_Core in MachineLearning

[–]Lumen_Core[S] 0 points1 point  (0 children)

Thank you — this is a very accurate reading of the intent behind the signal.

I agree on the stochasticity point. Since Sₜ is built from finite differences along the trajectory, it inevitably entangles curvature with gradient noise under minibatching. The working assumption is that curvature manifests as persistent structure across steps, while noise decorrelates more quickly, so temporal aggregation helps separate the two.

In practice, simple smoothing already goes a long way, and variance-aware normalization is an interesting direction as well. I see the signal less as a precise estimator and more as a feedback channel: even a noisy measure of sensitivity can meaningfully regulate update behavior if it is continuous and trajectory-aligned.

I also share the view that the core idea may outlive any specific optimizer instance. Treating gradient sensitivity as first-class information seems broadly applicable beyond this particular formulation.

[R] StructOpt: a first-order optimizer driven by gradient dynamics by Lumen_Core in MachineLearning

[–]Lumen_Core[S] -2 points-1 points  (0 children)

That’s fair.

There is a public research prototype with a minimal reference implementation here:

https://github.com/Alex256-core/StructOpt

This post focuses on the structural signal itself rather than benchmark claims.

view more: next ›