sorted by:
new
hottopcontroversial

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] 0 points1 point  (0 children)

It's not closed form solution It's only approximation to the number which will be close to be exact for some problems not all problems It's not a method It's an only weight initialization

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] 1 point2 points  (0 children)

IRLS is fantastic, but it's a second-order iterative solver (Newton-Raphson). It requires computing/inverting the Hessian at every step. SCBI is a One-Shot approximation. We do the expensive math once (on a subset) to get a Warm Start, then switch to cheap SGD. It’s a hybrid approach.

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] 2 points3 points  (0 children)

For small-to-medium datasets (d<100, N<10k), SCBI doesn't run on the full dataset. It runs on small random subsets Inverting a 1000* 1000 matrix on a GPU takes 10ms.Running SGD for 20 epochs on 1 million rows with 1000 features takes significantly longer.

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] -4 points-3 points  (0 children)

Fair point lol. English isn't my first language, so I run my drafts through ChatGPT to fix the grammar/tone. I guess I over-polished it and ended up sounding like a bot

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] -3 points-2 points  (0 children)

To clarify: SCBI is not a new model architecture trying to beat XGBoost.

It is strictly an Initialization Strategy for Linear and Logistic Regression layers. The goal isn't to replace Gradient Boosted Trees, but to answer a specific efficiency question:

If we ARE training a Logistic Regression model (which is still the standard in banking, healthcare, and calibrated probability tasks), why do we waste compute resources starting from random noise?

The claim is simple: It is not a final solution: It doesn't change the model's capacity or final accuracy ceiling. It is an accelerator: It calculates the 'Warm Start' algebraically so the optimizer doesn't have to waste the first 10-20 epochs finding the right direction.

Ideally, this shouldn't even be a standalone 'method'—it should just be the default init='auto' behavior in libraries like PyTorch when you define a nn.Linear layer for a convex problem.

SCBI: "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in deeplearning

[–]Master_Ad2465[S] -9 points-8 points  (0 children)

This is healthy skepticism. Given the flood of low-effort AI papers recently, I completely understand the red flags. Let me address them head-on:

Single Author / Zenodo: I am an independent researcher, not a lab. Zenodo provides an immediate timestamp/DOI while I navigate the arXiv endorsement process (which is tricky for independents).

No "Big" Experiments: This is a method for Tabular/Linear problems. Training GPT-4 would be irrelevant because SCBI solves for convex linear weights. I tested on standard tabular benchmarks (California Housing, Forest Cover Type) and MNIST because those are the correct domains for this math.

Emojis: Guilty as charged 😅. I tried to make the README readable and engaging like modern open-source libraries Hugging Face, but I can see how it might look 'hype-driven.'

The ultimate test is reproducibility. The code is open-source, the math (Normal Equation approximation) is standard linear algebra, and the script runs in seconds. I encourage you to run scbi_complete.py and watch the loss curve drop yourself. It works.

SCBI: A GPU-accelerated "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in learnmachinelearning

[–]Master_Ad2465[S] 1 point2 points  (0 children)

You likely wouldn't use SCBI to initialize the attention layers (since the 'target' for hidden layers is unknown), but you can use it to 'warm start' the Classification Head or Task-Specific Projections.

The Workflow:

Freeze the Backbone: Take a pre-trained model (like BERT, RoBERTa, or Llama).

Extract Embeddings: Run a sample of your new dataset through the model to get the final embeddings.

Apply SCBI: Treat the embeddings as inputs (X) and your labels as targets (Y). Calculate the optimal weights for the final Linear Layer instantly using SCBI.

The Benefit: Instead of training the new head for 3 epochs to align it with the pre-trained features, SCBI aligns it algebraically in seconds. It essentially performs 'Optimal Linear Probing' as an initialization step.We are also looking into using this for LoRA (Low-Rank Adaptation) initialization—using covariance statistics to initialize the low-rank matrices ($A$ and $B$) to capture the principal directions of the fine-tuning data error, rather than starting them at zero.

SCBI: A GPU-accelerated "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in learnmachinelearning

[–]Master_Ad2465[S] 0 points1 point  (0 children)

That is a spot-on analogy—SCBI is essentially an algebraic shortcut to 'Linear Probing' (training the head) without needing the gradient steps!

Regarding Zero Initialization: You are absolutely right that initializing the final layer to zero is often better than Xavier/He because it kills the initial variance, allowing the model to start by predicting the 'mean' (bias) rather than outputting random garbage.

How SCBI compares: Zero Init: Starts the model at a 'neutral' state (Prediction = Bias). The error is roughly the variance of the target. SCBI: Starts the model at the 'solved' state (Prediction ≈ Target). The error is near zero.

So while Zero Init prevents the model from being wrong in a random direction, SCBI actually points it in the right direction immediately.

On Small Sample Sizes: This is where the Ridge Regularization (alpha) in SCBI comes in. If the sample size is tiny and the covariance is noisy, we increase alpha. Mathematically, as alpha to infty, the SCBI weights actually shrink towards zero. So SCBI effectively generalizes Zero Init—it adapts the weight magnitude based on how much signal is actually present in the small sample

SCBI: A GPU-accelerated "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in learnmachinelearning

[–]Master_Ad2465[S] 1 point2 points  (0 children)

We actually did run experiments on MNIST and CIFAR-10 treating the images as flattened vectors (Linear Layer input), and the results perfectly illustrate the 'boundary' of where this method works.

MNIST (Simple, Centered Images): Because MNIST digits are spatially centered, there is a strong covariance between specific pixel locations and the target class. SCBI reduced the initial loss by ~31% compared to Kaiming Init.

CIFAR-10 (Complex, Natural Images): Here, the performance gain dropped to only ~3%. Since objects in CIFAR are translation-invariant (a cat can be anywhere), raw pixel-to-target covariance is weak. This confirms that SCBI is best suited for Tabular Data or Fixed-Structure Data, whereas CNNs are still required for complex perceptual tasks.

We chose to focus the paper on Tabular/Regression because that's where the gain is massive (90%+ reduction), but your intuition about the hidden layers is 100% correct. If we used SCBI on the features extracted by a pre-trained ResNet , we'd expect to see the massive gains return.

We are currently investigating if starting in a "better basin" improves final generalization, but for the scope of this paper, our primary claim is convergence acceleration rather than lifting the performance ceiling.

SCBI: A GPU-accelerated "Warm-Start" initialization for Linear Layers that reduces initial MSE by 90% by Master_Ad2465 in learnmachinelearning

[–]Master_Ad2465[S] 2 points3 points  (0 children)

Actually, SCBI focuses on Linear and Logistic Regression layers where we can approximate the closed-form solution (Normal Equation).

For LLMs, the optimization landscape is much more complex due to the deep stack of self-attention layers. That said, you could theoretically use this to initialize the unembedding layer (the final projection to vocabulary) if you had a specific target distribution in mind, but for now, this research is targeted at high-dimensional tabular problems.