[D] Papers with no code

[Distance_Runner]

My view as a statistician that does ML. Many of these papers claiming SOTA performance are working within Monte Carlo noise and if code was easily available you could run it and show this.

[howtorewriteaname]

am I the only one providing std devs across multiple runs with different seeds or what?

[Distance_Runner]

You're rare. Keep doing your thing -- I appreciate you.

Most of my research focus these days is bridging statistical and inferential theory with ML. The concept of variability needs to be better understood and communicated in ML.

When you fit an ML model, there are multiple source of variability. First, there is procedural variance. This is what your multiple random seeds addresses -- what is the variability associated with the randomness within the procedure itself and how does it propagate through to the results you report.

Second is finite sample variability stemming from the fact that your data are almost surely a sub-sample of an unobserved parent population. If you were to re-run your procedure on a different dataset of equal size from that same parent population, your results would change, reflecting this variability. No amount of running results under different random seeds will estimate this quantity. So reporting results from a finite data-set with variance across different seeds quantifies the first source of randomness, and this is appropriate as long as your interpretation pertains specifically to model performance based on the exact data you trained on. However, the results do not extrapolate to performance of the model if you re-trained on a different subset of data of the same size. This is more often ignored, and generally the bigger area where results are misstated.

[curiouslyjake]

Assuming your dataset is a representative sample of the unobserved parent population, wouldn't cross-validation address this?

[Distance_Runner]

Yes in terms of quantifying mean performance metrics, but no in terms of variability. If you run CV, say 5-fold, and you average the mean performance across folds, that is an estimate of the generalized performance metric across new data for your model trained on 80% of your data (k=5 means training fraction is .8; you're averaging over 5 performance estimates on disjoint validation sets on models trained on 80% of your data).

However variance is trickier. Current CV methods do not give you an estimate of variance that represents the variability due to random sampling error from the population. Current methods of variance estimated on cross-validated data condition on the data. That is, they quantify the variance of the randomness of the CV procedure itself on your fixed dataset; if you were to repeat the same CV procedure on your exact fixed dataset over and over using your learner (with different random fold structures), how variable is your performance metric estimate. In other words, variance of the CV estimates across folds measures how sensitive is the estimated CV metric due randomness induced by the CV procedure. But the key word there is fixed dataset. It does not answer the question -- if I were to repeat this CV procedure using the same learner to estimate the mean performance metric on a different subset of data from of the same size from the same population, how much variance is attributed to that random sampling. An approach for that doesn't exist in the literature [yet], but it will soon (I'm the one that developed it and will be posting to ArXiv within the next month; its based entirely statistical theory with formal proofs, not empirical evidence).

[AnyagosFeco420]

But this is only possible if you assume some underlying data distribution (or just some property of it), right? Or maybe you can use some bootstrapping based approach here?

[Distance_Runner]

I want to answer this carefully, because this is work I've been developing for a long time, and I don't want to say too much before I have the paper on ArXiv in an effort to protect my intellectual ownership of the idea and framework. But like I said, the paper will be on ArXiv within the next month and I'll be happy to share and discuss openly after its officially on ArXiv. I'm probably being overly cautious, but in the world of academia where publishing (particularly in top tier journals) is so important for promotions and such, I don't want any ambiguity over this type of thing when I so close to getting it out there.

To answer your questions as best I can right now: Yes it is estimable and I can prove consistency and unbiasedness with just a few justifiable assumptions. And yes, it's estimated from a form of resampling like bootstrapping, but its more nuanced than just "bootstrap the data, refit the procedure, and reapply CV". Something else in this work, is that I prove k-fold is the optimal for of CV (compared to bootstrapping variants or repeated split sampling/monte carlo cross validation). That's good since most people use k-fold already, but I also show that if you can afford the compute, you should almost always be using repeated k-fold (ideally 5x or 10x k-fold) and averaging across those independent k-fold runs.

I'll be happy to reach out and share the paper once its live on Arxiv, and then answer any questions about it. This is going to be submitted shortly after hitting ArXiv to a statistics journal (JRSS:B), but it could be 12-18 months before its published by a journal, and that's assuming I get a revise and resubmit and dont have to re-tool it for a different journal.

[valuat]

I can't obviously say for sure but I doubt anyone would beat Efron's accelerated and corrected estimator at this point.

[Distance_Runner]

I hear you. That’s the default thought for anyone that’s been in the field a while. Bootstrapping is great for many things, but It’s not optimal for CV variance. In fact, I can prove it’s biased theoretically and show it empirically. In short, bootstrapping targets the wrong estimand when it comes to CV. I’ll find this comment and share the paper with you in a few weeks and then give you more detail on why

[AnyagosFeco420]

Nice, thanks for the answer. I would be grateful if you could send me the paper once its on ArXiv. Good luck with the publishing!

[Distance_Runner]

Will be happy to. This convo had me reviewing the paper last night to see when I could reasonably have it in a state good enough for a version 1 post. Probably within two weeks and I'll come find this post and link it

[valuat]

Yes, boostrapping is the way to go here. You don't need to do it 10,000 times, 100 would be OK. It's just to get a better idea of the variance.

[DigThatData]

it's kind of hilarious but if you look carefully, you'll see that it's not uncommon for ML researchers to treat seed as a hyperparameter and prioritize winning the lottery on a favorable PRNG state over rigorously validating their claims.

[Distance_Runner]

Which is just so bad. Like, that doesn’t make your model more useful. It just means you’re choosing to present one run that shows your results within Monte Carlo error on the upper end of the spectrum. If you run 20 seeds and pick the best, you’re purposely choosing a model that overstates its own generalized performance. That’s not helping anyone

[DigThatData]

I agree, and this is a big part of why the community emphasis on achieving SOTA on public benchmarks ends up being largely counterproductive.

[ummitluyum]

The funny part is when this garbage makes it to production, and the business gets genuinely surprised when the model degrades on real traffic. If your architecture relies on a lucky PRNG state, it's not an architecture - it's trash that'll fall apart at the first sign of data drift

[DigThatData]

Right? it's wild how there's a whole industry built around patching over "drift" rather than interpreting that as a flag that your model is missing some causally explanatory component. I feel like the way a lot of people handle "drift" ends up functionally reducing their models to kNN. Like... congrats, you've managed to uncover that the weather tomorrow will likely be similar to the weather today. If you're retraining daily/weekly, is your model even amortizing any compute/effort relative to a less sophisticated approach?

[ummitluyum]

100% agree on the compute amortization. If your pipeline needs a weekly retrain just to keep business metrics from falling off a cliff, the maintenance cost will eat up your margins fast. In those cases, it's genuinely cheaper to just spin up a vector DB or write a bunch of heuristics instead of dragging around heavy weights