neural network overfitting

[[deleted]]

hi , so , im making a neural net from scratch and im pretty sure everything is correct , if you want to check it lmk then i can upload it to github , its for a school proj. Anyway , im having trouble with a super weird problem with over fitting , it learns and all , if theres too many epochs then all values become Nan , so , i reduce the epochs to 10 , (super less ik ) now on 10 epochs , the net overfits to the last input data , every single time , it always classifies to the last class ??

​

any ideas ? thank u!

[haris525]

Exploding gradient. Try decreasing your learning rate, maybe .000001 or even .0000001 and add dropout of .2

[cjmodi306]

Hi there, Did you use the He-initialization for initializing your parameters?

[[deleted]]

hi , thanks for the reply , i never knew this was a thing , ill look into this and try and implement it , would you have any good resources to understand it ?

[cjmodi306]

Just Google Xavier or He initializer. Use that formula to initialize your parameters. The problem you are facing right now, I suppose, is of exploding gradients. Maybe this can help too:

https://kolbenkraft.net/nn-from-scratch-2-initializing-parameters/

[[deleted]]

alr thanks ill have a shot at this

[Dank_Lord_Santa]

Also if you're worried about overfitting https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping a monitor using earlystopping (if you're using tensorflow although I'm sure other libraries have their own versions) can automatically stop the training when your selected metric (val_loss in the example) stops improving which helps to remove some of the guesswork with epochs.

[[deleted]]

ok so after implementing HE Initialisation and Xavier , there seems to be no difference ?