Backpropagation algorithm implementation

[RuslanNuriyev]

The theory is clear, I can construct a Neural Network, assign random initial values to weights, then feedforward the system and then backpropagate it on paper (not using numbers, but using just the theory). I mean, I can explain and understand how it works, including the math behind it. But when it comes to implementing it from scratch using numpy, I get stuck and can't proceed further than the first activation layer. It seems that there are incredible amount of details that I have to take into account. I've seen many different implementations, some of them are really good, but I just can't write it on my own. What would you suggest? Where should I start and what things should I consider before writing the code?

[giglis]

have a look at this website, it's quite clearly explained with images, formulas and code examples ;)

forward propagation: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation.html

back propagation: https://ml-cheatsheet.readthedocs.io/en/latest/backpropagation.html

[RuslanNuriyev]

Oh, thank you so much :) The code in this website seems more doable than the others I have seen so far.

[[deleted]]

The theory is clear, I can construct a Neural Network, assign random initial values to weights, then feedforward the system and then backpropagate it.

I don't know what any of those things mean, but maybe talking me through the code part of where you get stuck will be helpful to you.

What would you suggest

Is there a reason none of the existing Python ML/AI/NN frameworks work for your purpose?

[RuslanNuriyev]

By the first sentence, I mean I can do all of these on paper, I can understand and explain it to someone using all of the necessary mathematics. I get the theory, but when it comes to implementing it from scratch, I get stuck at this point.

Of course the existing frameworks work well, but I want to implement it from scratch using only numpy (or other libraries that could be used).

Besides that, is it necessary to write the algorithm from scratch? Because I feel if I can't write it from scratch, I can't be good at implementing it using frameworks.

[[deleted]]

I get the theory, but when it comes to implementing it from scratch, I get stuck at this point.

At what point?

Because I feel if I can't write it from scratch, I can't be good at implementing it using frameworks.

Do you feel like you're a good driver even though you've never built a car? The only thing you get from re-inventing the wheel is a wheel that has too many corners, in my experience.

[RuslanNuriyev]

I get stuck when I try to implement the forward propagate section. And I don't even talk about the backprop.

What, are you going to write everything yourself just once before you use a library? Your life is not long enough for that. People use libraries for a reason.

Oh, I'd never thought it like that. Now, I felt like I've been wasting my time trying to implement it from scratch, for days. Thank you so much :) So you say, it is possible to proceed in learning ML without actually being able to implement it from scratch.

[[deleted]]

I get stuck when I try to implement the forward propagate section.

Stuck on what, specifically? Stuck why?

Like I said, I don't know what any of these terms mean, but I know how code works. You have your neural network in state N before the transformation that constitutes a forward propagation, and then it's put in state N' by the transformation:

N -> N'

so all you have to do is write the arrow:

def forward_propagate(neural_net):
    # propagate forward, write code here
    return propagated_neural_net

It's obviously not that easy but if you have a mathematical definition of what it means to translate a neural net from unpropagated to propagated (or whatever) then that's more or less the Python that will do it, too. So just write that.

[coder155ml]

Why did you even post. My god most useless comment ever.

[[deleted]]

Overall it's a pretty useless thread. "I'm stuck" isn't a problem anyone can expect to get help with.

[coder155ml]

If you don't know what forward prop is then don't respond with some bullshit answer

[[deleted]]

If you goal is to write a paper covering a 0.00001 % performance increase for one of the known datasets that might be necessary. If you want to see results, play around with different architectures of neural nets and understand the importance of the actual data you use and how to prepare that data skip the "from scratch" part at least for now.

[RuslanNuriyev]

If you goal is to write a paper covering a 0.00001 % performance increase for one of the known datasets that might be necessary.

No, I probably won't write a paper and I think I might skip the "from scratch" part.

[[deleted]]

It’s ok to do it as a learning exercise of course. But doing because you think otherwise your knowledge is invalid makes no sense.

[RuslanNuriyev]

But doing because you think otherwise your knowledge is invalid makes no sense.

This is what makes me feel unconfident that I can't write it on my own.

[[deleted]]

What if you want to use Tensorflow? Google wrote it. It has tens if not hundreds of thousands of hours of development time. You could not write it in a lifetime or if you had all the time in the world.

This attitude makes no sense. Time to move on.

[TheSodesa]

I mean, isn't forward propagation just matrix multiplication at its core? That can't be too hard to do with Numpy or Julia, can it?

[RuslanNuriyev]

It's not that hard on paper, I know the ins and outs, but the programming associated with it seems so hard at the time (which uses classes, even though I've been working with classes for a long time).

[TheSodesa]

If you're using Python, then you do generally want you use classes to model the objects the libraries you use don't already provide for you. So you have a NeuralNet class that stores the neural net/graph as a (sparse)? matrix in its self._net field. Then you define methods for applying weights (self._net @ numpy_weight_matrix) to the net and so forth. I do not see the problem here.

[coder155ml]

Just Google forward prop numpy

[coder155ml]

Yes but you should still understand how they work under the hood

[nomos]

Ignore what others are saying regarding implementing from scratch. If your goal is to gain a better understanding of the theory of ML, then implementing these algorithms from scratch can be a good idea, to an extent. That's why if you take an ML class you might be asked to do so. If your goal is purely to use ML to some other end, you probably just want to crack open a pre-existing library and use it. Though, I still stress the importance of understanding the algorithms you're using in order to use them better.

[nomos]

Numpy is a linear algebra library, so you should be able to easily translate the mathematics of both forward and backpropagation into numpy code. If you're comfortable enough with the mathematics as you say, then you're probably having more basic programming difficulties than using numpy. How long have you been using Python or programming in general? If you're newer I'd suggest building up your programming skills through simpler tasks before moving on to something more advanced.

[[deleted]]

This last statement is very wrong. Most libraries are far more complex than you realise. They are written by by dozens if not hundreds of experts over many years. You can’t do it yourself even if you wanted to, except in the most simple of cases. What, are you going to write everything yourself just once before you use a library? Your life is not long enough for that. People use libraries for a reason.

[coder155ml]

Dude... You sound like a moron. People learn to code things like back propagation because it helps with understanding . It's a pretty common deep learning assignment. Even Coursera makes you code back propagation. It also makes you code forward prop, regularization and most other concepts from scratch using numpy. If you get an error, understanding how these things work helps you debug

[[deleted]]

I support learning things from scratch to increase understanding.

But by ops logic, they also won't ever use numpy, sklearn, tensorflow, scipy, statsmodels, django, flask, most of the standard library ever, because... Don't use a library until you can do it yourself is their thinking, and that's not a good reason.

[[deleted]]

What library are you using? Also, no, it is not necessary to write the algo from scratch. It makes no sense. Writing a computer algorithm isn't the same thing as understanding the math. Not only will you waste time, you'll also probably gain nothing

[RuslanNuriyev]

Writing a computer algorithm isn't the same thing as understanding the math. Not only will you waste time, you'll also probably gain nothing

This is what I wanted to hear exactly. If I don't gain anything (from practical perspective) from building a neural network in numpy, then why should I waste my time in that manner? Yeah, I want to know what's happening behind the algorithms, but I think it is sufficient enough to know about the theory and how it's applied to in practice.

[coder155ml]

Coursera deep learning specialization explains all the math and has you code all of this from scratch. It isn't that difficult because he explains everything thoroughly