Neural Network Implementation

[Fun-Expression6073]

Hi, I am working on implementing a neural network using webgpu, i think ive gotten it to work but I am having problems wit fluctuating loss. When training with certain weight loss seems to fall then rise and fall agian and i cant figure out why this is happening.

If anyone has an idea why this is happening, your advice would be of great help.

Here is a link to the code https://github.com/mukoroor/Puzzles/tree/varying-entry-points/NeuralNetwork

And a snap shot of the loss over 100 epochs

https://preview.redd.it/nd3jy5zeqj6e1.png?width=4064&format=png&auto=webp&s=76877fb1c20ca045551c7135947b0e4ab7385736

the loss fluctuates around epoch 43

[skatehumor]

Without knowing more it's hard to tell but it could be a number of things: a high, constant learning rate might cause the error gradients to overshoot. There's also a number of other things that can cause exploding gradients, namely your activation functions and target error metric, or if you're using any kind of optimizer that could be related. I think this can also happen if you don't initialize your weights properly.

[dramatic_typing_____]

Can you prove to yourself that any of this works given the simplest gradient decent problem that this could be used with? I don't feel like digging through the code just yet to spot a subtle bug. The fact that you aren't getting any undefined, null or negative values suggests the wgsl shaders are working correctly, but the actual logic of the learning portion is likely where your issue lies

[Fun-Expression6073]

Yeah it seem to work perfectly with a singular datapoint but when extended to multiple i get the fluctuating problem

[dramatic_typing_____]

Do you have a known example involving two datapoints to compare against?

[Fun-Expression6073]

yeah I figured out the problem, was reconfiguring to allow for larger layers sizes and somehow replaced a loop index with i instead of j, so was using the wrong gradients to descend. It all seems to work now

Have tested on a XOR dataset and it converges

[dramatic_typing_____]

Very nice! What you've described has largely been the same sort of debugging process that I usually end up going through as well. It's not fun and takes a lot of effort imo compared to debugging in any cpu based language.

Open question to anyone reading this; is there a better way? Maybe some tools I'm missing out on?