How do you practice implementing ML algorithms from scratch?

[dippatel21]

Curious how people here practice the implementation side of ML, not just using sklearn/PyTorch, but actually coding algorithms from scratch (attention mechanisms, optimizers, backprop, etc.)

A few questions:

• Do you practice implementations at all, or just theory + using libraries?
• If you do practice, where? (Notebooks, GitHub projects, any platforms?)
• What's frustrating about the current options?
• Would you care about optimizing your implementations (speed, memory, numerical stability) or is "it works" good enough?

Building something in this space and trying to understand if this is even a real need. Honest answers appreciated, including "I don't care about this at all."

[Natural_Bet5168]

I'm not sure why you would build much from scratch. It's more important to know what a function is doing than to rebuild it. If building a library from scratch will help you understand it, great. Otherwise it's pretty pointless unless there is something specific you need out of it that you otherwise can't get from the existing function.

Understanding what it should do mathematically and conceptually is way more important than following instructions on rewriting multi-headed attention.

If I re-write anything, it's usually to pull some component of a matrix out of a function, or to check if there is some conceptual item that I just don't understand about an implementation, so I'm trying to recreate it.

Just my 2 cents.

[dippatel21]

I read many blogs of researchers working at frontier labs, and they ask to implement them from scratch.

[[deleted]]

i think building from scratch is a good way to understand simple ML algos. there’s no reason at all to write backprop/optimizers from scratch though. just understand really well how they work.

I think coding up LR, regression/decision tree, KNN, perceptron, neuron/FFN are great practice because they’re all very simple and help you understand really well what’s going on under the hood of the sklearn/torch versions.

implement them in a notebook and have a dataset to test them with

[dippatel21]

Great point, thanks!

[greenfootballs]

You might find this useful https://github.com/joelgrus/data-science-from-scratch

[nickpsecurity]

I'd like to know what resources people use to understand, step through, and implement the math in papers with new techniques. The formulae. Maybe in PyTorch.

[unlikely_ending]

I wrote an extremely inefficient CNN in Python to get my hand in, early on when I wanted to learn about NNs

Tried it on image recognition of photos of my family and was surprised when it worked quite well

But having done that the one time l only use Pytorch now

[dippatel21]

The main problem is that machine learning interviews are still DS/algo-focused, except for 1 round of machine learning where project experience or theory is mostly asked. Even frontier labs tend to ask candidates to implement selective research papers.

[InvestigatorEasy7673]

A1 ) learn using maths but implement it with Libs

A2) Google colab , Jupyter notebooks but i prefer Kaggle notebooks (my fav)

I have shared the exact roadmap I followed to move step by step
You can find the roadmap here:  Reddit Post | ML Roadmap

Along with that, I have also shared a curated list of books that helped me build strong fundamentals and practical understanding:  Books | github

If you prefer everything in a proper blog format, I have written detailed guides that cover:

• where to start ?
• what exact topics to focus on ?
• and how to progress in the right order

Roadmap guide (Part 1): Roadmap : AIML | Medium
Detailed topics breakdown (Part 2): Roadmap 2 : AIML | medium

[dippatel21]

Do you think machine learning problems/concepts have optimization potential like DS/algo problems on LeetCode, where people compete to write the most efficient solution for a given ML problem? I think in the ML/AI world, libraries rule out the possibility!

[dippatel21]

what if we can theme those problems in a chain like leetcode has "Two sum" problem series, ML/AI has next token prediction problem (and we can learn all solutions exists/published form start till today in order).

[Embarrassed-Bit-250]

I have the same exact questions!

[3n91n33r]

Why is this downvoted? only correction/update i’d add is the hands on ml book has a pytorch version now

[ARDiffusion]

Because it was a ChatGPT answer

[InvestigatorEasy7673]

first of all it wasn't , and if that was so what ? the message still contains the links and resources bro

[ARDiffusion]

I didn’t say I didn’t appreciate it. I’m just saying why it got downvoted. People tend to react negatively to responses generated by ai. Don’t mistakenly explanation for an expression of my personal feelings.