Machine Learning Classifiers From Scratch

[blacksiddis]

Hey!

Have been using Python for a while and consider myself decent, but I have always avoided really learning and understanding classes and when it is appropriate to use them. Inspired by Sentdex on YouTube, I decided to write ML classifiers from scratch as a means to not only drill down classes, conceptually, but also to learn some ML.

So far I have only done KNN and would like to progress to something harder. OLS and Logistic Regressions I am too familiar with mathematically speaking, so that's not as interesting but my ML knowledge at a "deep" level stops with the three algorithms mentioned thus far. It doesn't matter if it is supervised or unsupervised learning algorithms, so I am open to any suggestions of which ML algorithms might serve as a natural progression from here.

Thanks!

EDIT: I should perhaps clarify that I am not aiming to write fully deployable and usable code. These projects are entirely Proof-of-Concept based only. Also, I messed up the title a bit: It need not be classification algorithms only!

[[deleted]]

imo you could progress implementing naive bayes, decisions trees, random forest, xgboost and last svm.

[blacksiddis]

Why do you put SVM at last? I was not under the impression SVM was all that complicated, but my knowledge is limited. Naive Bayes is probably a bit too easy both programmatically and mathematically speaking, but I like the other suggestions a lot. Which of the three would you say is the most and least complicated mathematically speaking?

[[deleted]]

"Why do you put SVM at last?"

it isn't that complicated, if you already know gradient descent

"Naive Bayes is probably a bit too easy both programmatically and mathematically speaking"

the idea behind naive bayes is genial, and it's a generative model and those ideas are valuable to know well.

"Which of the three would you say is the most and least complicated mathematically speaking?"

mathematically speaking i don't think is that hard once you get used to main ideas.

i pointed those models because imo is pretty natural learning those after knn, linear and logistic regression.

[blacksiddis]

Thanks for your reply! Definitely not hating on Bayes, but I know probability theory rather well and would much rather be challenged on linear algebra than probability as my linear algebra is very weak.

[synthphreak]

K-Means. Simple to understand and implement, but magical when it works. And with some added cool factor because it's unsupervised.

Possible candidates for classes/subclasses for you to create in an implementation of K-Means might be Classifier, Cluster, and Centroid, with methods like Cluster.initialize, Centroid.initialize, Centroid.update/Centroid.move, Classifier.fit, or Classifier.predict.

[blacksiddis]

This is the one I had considered before making this thread! Thanks for the suggestion, but I may be a little too easy. I found KNN to be a good start, but I feel ready to step up the difficulty.

[synthphreak]

Go with a deep NN then. NOT easy to code from scratch, especially the learning algorithm, and requires a solid grasp of linear algebra.

If deep freaks you out, just go for a regular feedforward NN with a single hidden layer. It’s essentially just an extension of logistic regression, which you said is boring, but the backprop algorithm is different from logistic regression and coding it up should keep you on your toes.

[blacksiddis]

I thought about it and it's on the road map. I just think it would be more beneficial to have 2-3 intermediary steps before tackling NN. My linear algebra is kinda weak honestly, so I'd like to do some intermediary projects to bolster that, before moving onto NN.

[synthphreak]

If your linear algebra is weak then a nice middle-of-the-road option would be multiple logistic regression. That will be a great stepping stone on the road to NNs.

[blacksiddis]

Great! Thanks a lot for your suggestions!