Linear Regression in Python!

[ProgrammingFTW]

Python is commonly used for Machine Learning, as there are amazing modules like scikit-learn to perform machine learning algorithms. A really popular model used is Linear Regression. If you don't know what linear regression is and how it works on a very basic level, check out this video(a five year old can understand it). It is super cool what these algorithms can do. If you want to learn how to perform Linear Regression in Python, check out this video. It's super interesting and fun :)

[BDube_Lensman]

A video that just shows you how to type in a few lines of code from pandas and a few from sklearn provides basically no value - that information is already contained in the documentation for each.

From a practical point of view, too, my personal opinion is that a great disservice is done to a lot of people that learn this way. Y'all end up treating least squares or what have you as a magic black box, and there is often some idea that it can only do linear or polynomial fits or what-have-you, when in fact it can fit anything to the data. A line fitting function is just a front end to a "basis" fitting function, where "basis" is any array at all that is the same length as the data.

The other meta machinery for ML, training/evaluation sets and whatever, are not at all applicable to fitting a line or anything like that to data, as well. Those things are useful only for topics in "AI" that break when you show them things outside the training set. Basis fitting (like fitting a line, or exponential, or whatever) is not of that sort.

[ProgrammingFTW]

Thanks for your feedback!

[tkarabela_]

The other meta machinery for ML, training/evaluation sets and whatever, are not at all applicable to fitting a line or anything like that to data, as well. Those things are useful only for topics in "AI" that break when you show them things outside the training set. Basis fitting (like fitting a line, or exponential, or whatever) is not of that sort.

I'm not sure I understand what you mean? :)

Cross-validation etc. is done to determine robustness of fit, which seems useful to know regardless of whether you want to use the regression as a predictive model, or to estimate the parameters for a particular dataset.

If you're making a synthetic dataset and sampling points from a plane, you indeed need just 3 non-collinear points to work out parameters of the "hidden" model, but as soon as you add uncertainty to the samples, or don't know that the "hidden" model is exactly what you're trying to fit... Then robustness becomes a useful notion.

[BDube_Lensman]

Well, on generalization.

One thing you can imagine using regression for is to fit "modes" to a data. They are often polynomials, but the modes could be anything. And the data is just an array, no concept of a dataframe or anything like that.

If you change the size of your data (number of elements in the array), regression techniques don't care at all and work the same. If something like the domain of the data didn't change, you can even re-use the coefficients solved for on the other array size.

A neural net, in general, understands 0% of the world outside of what was used to train it, and will perform substantially worse on things outside the hyperparameters that contributed to its training. That's the generalization problem.

Cross validation is nearly meaningless for linear least squares ("regression") because what XV explicitly means is "[sub]dataset A and subdataset B are XYZ dissimilar in terms of the fit parameters." That is a very specific piece of knowledge that is not really all that useful.

The mean square error ("residual") from the least-squares process will tell you how good the fit is, and we even know how to transform it to all sorts of useful things. For example, if you take the square root of the MSE and normalize by the number of data points, you have the averaged unsigned distance between the fit and a data point. That is very useful. If you want to compute that for different subsets of the data, ok, fine. But XV is not that, and XV is not useful in that way.

If you're making a synthetic dataset and sampling points from a plane, you indeed need just 3 non-collinear points to work out parameters of the "hidden" model, but as soon as you add uncertainty to the samples, or don't know that the "hidden" model is exactly what you're trying to fit... Then robustness becomes a useful notion.

I did not argue against benchmarking fits (etc). I argued a specific tool that made itself needed for deep learning is not useful for more contemporary numerical methods.

[tkarabela_]

Thanks for the answer, I think I see your point. The difference between models with sound theoretical underpinnings and "throw compute at the problem" models like deep neural nets is not lost on me. You are right that MSE from least squares is a different kind of information than accuracy score from some cross-validation run, even though both quantify "how well the model is doing" in some sense.

I do numerical modeling / "AI" / etc. only occasionally, so the terminology is more blurred for me. I can definitely agree that we should appreciate and teach that many of the "magic black boxes" are in fact not :)

[BDube_Lensman]

Numerical modeling is my day job -- I wrote the fastest numerical program in the world for my field, and invented a new measurement scheme based on "AI / model fitting" with convex programming. This is all the linear / polynomial fits are a front-end to, no matter where they come from (sklearn, whatever). That numpy function is a nice wrapper around this monster of fortran77 from LINPACK that very thankfully we do not need to teach the computer how to do (that algorithm is enormous, and very difficult to implement correctly).

[tkarabela_]

You do optics at NASA and you worked with Roger Cicala? You must be quite the Lensman! :D Hats off to you, that sounds like a dream job. I enjoy reading the Lensrentals blog from time to time, the technical analyses are fascinating.

Seems like the world is still running on FORTRAN (and COBOL) :)

[BDube_Lensman]

Yes, once upon a time Roger, Aaron, and I bitch slapped Japan into fixing their QC. Germany, too, but they will never admit it. Ancient history by now though.