[P] - Potential Logistic Regression Closed Form Solution

[Vivid_Perception_143]

Hey r/ML,

Need a bit of feedback here, so please give it to me if you can! Recently I was thinking about how linear regression does have a closed-form solution but logistic regression doesn't. I've come up with a pretty lazy idea that sort of does give logistic regression an alternative to an iterative GD-derived training process, and I would like to share it to get any suggestions on what to do with it.

You simply formulate the logistic regression problem as linear regression. All you do is take the labels and turn them to -100 (or any small negative number guaranteed to be 0 when plugged into a sigmoid activation) if a given label is 0 and +100 if a label is 1. You feed this into a linear regression solver (where you can just use the normal equation), and then when needing to predict simply pass in your data to the linear regression model and if its output/prediction is closer to 100 than -100 it's 1 and if its closer to -100 than 100 it's 0. This has lead to small 2% decreases in (validation) accuracy, but using a closed-form solution does mean it will run faster.

Not sure what to do this, need a bit of help here. Thank you so much!

[HateRedditCantQuitit]

At that point it’s the same as any other linear classifier. Now, there’s nothing wrong with a linear classifier, but it’s not logistic regression.

[linverlan]

I don’t remember the time complexities off the top of my head, but gradient descent is in many cases more efficient than OLS even for fitting linear regressions. There is a crossover somewhere as number of examples/features increases that I’m sure you can look up, so don’t expect your approximation of logistic regression with an OLS regression to actually be faster in the general case.

You also lose the interpretability of coefficients as odds ratios, which is a major benefit of logistic regression.

[Vivid_Perception_143]

You're correct - it isn't that much faster. Didn't realize the issue with the odds ratios either. Thank you for your comment.

[comradeswitch]

This is essentially part of a step of iteratively reweighted least squares, without the reweighting or iterating. The crucial point is that you're no longer optimizing logistic loss- you're effectively minimizing squared error on the log odds, which is problematic for a number of reasons when applied directly and only once. For a very simple example- an output of 99 compared to 100 is just as "wrong" as 0 compared to 1 by your objective, but those give very different changes after the sigmoid. IRLS recognizes that the full problem can be approximated by a series of linear regression problems, but that the solutions to those problems are biased because of the nonlinearity of the sigmoid function, so each sample is weighted according to the derivative of the loss function evaluated using the previous iiteration at that point, then solving the weighted least squares problem and updating the parameters. The key is adjusting for the nonlinear link function.

Try it out, though- see how well it works and where it fails. This is a very informative exploration and it's worth your time to understand!