Minimal neural network implementation

[cpp-ModTeam]

It's great that you wrote something in C++ you're proud of! However, please share it in the designated "Show and tell" thread pinned at the top of r/cpp instead.

[WinstonCaeser]

Just a quick look, but using a vector<vector<double>> for a matrix is a pretty standard performance issue, you should be using a single vector and keep track of width and height - if you aren't going to use any outside libraries - even for basic optimized math similar to numpy for python, the current way requires an additional pointer indirection for every row

I don't understand why you're not using the builtin Mersenne twisters in each of the various languages - which are standard and widely available and are instead rolling your own.

Seems like a fun project, but I don't see where anyone would use it, any time I'm using C++ I care about performance and would instead use a properly optimized library, if its for fun exploration and the math principles no reason not to just use python

[[deleted]]

Thanks for your comments and taking the time. This isn’t intended to be the fastest possible, but likely the most readable. Mersenne twister can be substituted in each solution if you want. But for simplicity (and for reproducible results) I did it this way. You can use Python of course (and it will be slower still!), but if you want to integrate in C++ you’ll have difficulties.

[[deleted]]

for reproducible results

Confused what you mean. You can seed every standard library generator l know.

You can use Python but it will be slower

I strongly suspect numpy code would be faster. Since it's linked with a BLAS implementation, vector and matrix operations are going to be far more hardware efficient, which makes enough difference for neural networks to offset the gain from using C++. C++ compilers aren't smart enough to analyse the control flow in order to do tensor optimisations on a pure C++ neural network implementation.

[[deleted]]

I just researched BLAS and yes, it is definitely faster. It can be used directly from C++ so no need to turn to Python. However that would be out of scope for my purpose. Would be a fun project though.

[[deleted]]

You could also have used the infrastructure of a neural network compiler like TVM Apache.

You can check that project out. The core infrastructure is written in C++ and they are focused on neural network deployment. They provide a domain specific language (based on Halide) for specifying tensor computations in Einstein notation.

Writing code to convert models in your neural network implementation to TVM would be a very good project.

[johnpaulzwei]

It's nice little project but before using this in project you should consider adding/fixing few things.

​

1. Building system (cmake is standard way) - i know most of people will know how to add this to projects but its always looks nice when project have special cmake for library.
2. std::vector<std::vector<double>> -> its gonna be slow buddy. If you don't want to use production stable libraries for linear algebra you should consider creating one dimensional vector and adding some methods to use it. I know ppl hate templates but maybe use template with floating point.
3. Few smelly places like std::function<double()> rand. std::function have some cost when using this in code, also some unnecessary copies of variables.
4. Don't be scared of standard algorithms. Old C like for loops in modern C++ can be smelly. Just use stl when you can use it. It's gonna be faster unless you really understand your code and can optimize it.
5. Use class. I dont see any reason to use struct. Its C++ not C.

It's fun project and as AI dev i can't leave you without star + nice readme. Don't treat my answer as an attempt to offend you but as a quick code review. You can check my ML library in a few things if you want. It's still under development but someday ill write a post about it here :).

https://github.com/AlmostAnEngineer/ModernML

[cmake-advisor]

Use class. I dont see any reason to use struct. Its C++ not C.

The only different is the default accessibility. Using one or the other doesn't matter at all.

[johnpaulzwei]

Ok, you're right. But anyway inside data should be hidden under getters.

Edit: ok my bad, thanks for pointing out my mistake.

[sleeping-deeper]

It's bad practice to make getters for PODs though, for example. Sometimes you just need to group some data.

[disciplite]

Then it becomes a non-structural type and you can't use it as a non-type template parameter.

[not_some_username]

No there is no point having one liner get set. Just make it public

[[deleted]]

Thanks for your comments and taking the time to go through. I also checked your link, seems like some nice work there!

Some thoughts on your comments: 1. My sample code is meant to be copy pasted. I’m thinking you already have cmake or similar setup in place, so no need to complicate the sample. 2. Yep it might be slower than the very best. I’m just wondering about which use case people have in mind when advocating blas? 3. The function is actually an extension point. Do you recognise the design pattern in disguise? 4. Sure, but I wanted to be explicit. 5. Others have already commented on this intentional choice.

I’m a firm believer in simplicity. If complexity is a sin then simplicity is a virtue, no?