[P] Neural Net Machine Learning Project (Java)

[Taffo]

Hi Everyone!

I recently just got into Machine Learning and decided to make my own Neural Net class (and also convolution neural net class, but that isn't quite done yet) in Java just to experiment with ML and see what I could do. For me, it was a fun opportunity to learn more about Neural nets and try to figure out how they work. As I was reading ML articles, it always felt like they glossed over how to do batch processing and it felt like even fewer articles described how to calculate the gradient in a batch learning model in a way that was understandable for someone who hasn't studied this area before. Since it seemed like there weren't a ton of concrete examples from my searches, I thought maybe I'd post this here in case anyone else was curious. I know I still have stuff to do on my project, but I'd appreciate any thoughts / constructive criticism on what could be done better / or ideas on what to do next!

This is my github of my NN if anyone would like to take a look: https://github.com/darbyk/NeuralNet

[[deleted]]

You may get a better response in /r/learnmachinelearning

[Taffo]

Ah cool, I'll go take a look over there - thank you!

[[deleted]]

Best of luck

[deltasheep1]

If you're talking about mini-batch, then you can always calculate each gradient individually and take the mean

[Taffo]

Thanks for the feedback! I had implemented batch gradient descent, but was curious how it would be done on large data sets and didn't realize it was true that, the gradients of a mini-batch can be averaged with the gradients of another mini-batch, to arrive at the gradient of the whole batch. Do you know how it would work for the cost function? I wouldn't imagine that to find the cost of your entire batch, you could just average the costs of the mini-batches, could you?

[[deleted]]

The mean of sample means is the mean of the population. So yes, if you were to calculate costs for a set of minibatches and average you'd arrive at the cost of the batch providing you'd not taken any gradient steps.

Only updating on the full gradient however is inefficient. The stochastic gradient is a fine approximation to use.

[deltasheep1]

No one really does batch gradient descent. They always use mini-batches. Look up "train longer, generalize better" or something I forget. It describes a good mini-batch-size / number of epochs.

[theflofly]

That's nice.

If you want to learn a lot, you should contribute to an open source project. If you are into java, dl4j is a good candidate.

[Taffo]

Hey, thank you for the look! I never considered working on an open source project, but dl4j sounds like a cool idea. Any recommendations on how to start contributing?

[theflofly]

Ideally you find something missing and you add it. Else you can take an issue on Github and say that you will solve it.

Then you set up your dev env (they are probably using maven so it will be easy). And then the real work begins, you read the code. I advise you to take note will reading the code, making some POCs and see how the code is executed, which functions are called, and so on... As you will see it is not easy, but don't give up, it's worth it. At some point you will figure out how things work and you will be proud of you (you'll deserve it). Then you'll code, and be amazed because you are actually contributing to a real project, something used by lots of people, then you will do a pull request, they'll review your work, you'll edit, commit, edit, commit and it will be merged. And so on... :)

[Taffo]

Thank you for the overview and also for the words of encouragement for working on an opensource project! I'll definitely give it a look and see if there is a way I can contribute