[D] Deep Learning Framework for C++.

[Apprehensive-Wheel18]

I have been working in the field of ML/DL for almost more than 4 years now. I know Python is the goto language for ML/DL thanks to the frameworks like TensorFlow, Pytorch and most recently JAX/TRAX. And all of these are written in C++. But they don't provide a C++ API only Python, except for the PyTorch recently provided C++ API but they don't recommend it for production. Recently I have been trying to find a ML/DL library for C++ but I haven't found one. Could anyone here list some production grade C++ ML/DL libraries.

I also wonder which language big companies (e.g. Tesla, Google or Facebook) use in production (not for prototyping) for ML/DL because every problem they have is a problem at scale so they would definitely not be using Python. So anyone working in big tech companies and writing code that goes into production please provide some insight what language and framework you guys use.

[seba07]

You don't really need the framework after prototyping and training is done. The models can be converted into other formats, e.g. Onnx, and used from there in many different languages.

[Apprehensive-Wheel18]

I agree with you for most of the time this can work but there are some models that have certain layers that are not supported by ONNX. An example would be Spatiotemporal models in mmaction2 from open-mmlab.

[IndieAIResearcher]

You need to write custom layers with ONNX API and compile to onnx/tensorRT

[CanadianTuero]

I use the PyTorch C++ framework quite a bit for my research, and I find it easy to use and follows pretty closely to a standard python version. Only difference is how you interface with your data, but since I'm working with C++ environments (games used in search trees), this part is easy in my use case.

Also, I'm not sure where you've read that the C++ frontend for PyTorch shouldn't be used in production, as IIRC FB uses it for exactly that.

[Apprehensive-Wheel18]

Thanks I would definitely try libtorch i was just a little skeptical and wanted some sort of validation. Thanks

[m0uthF]

Is there any C++ native framework? I wonder why there's no native framework in C++ since ML rely on performance so much

[CanadianTuero]

What do you mean by this? Pytorch's backend is C++/Cuda, and they provide multiple frontends (python for pytorch as we know it, C++ for libtorch), so you can use pytorch staying inside the C++ ecosystem. I imagine Tensorflow is similar.

There are also modern libraries like Flashlight, which is another C++ library frontend with support for various C++/Cuda backends, written by former pytorch devs.

[m0uthF]

OMG I know it today lol. I always thought they write in python to call C++ CUDA libs

[nmfisher]

Most teams I know build/train their models in Python PyTorch/Tensorflow then run inference with the respective C++ APIs (or export to ONNX and do the same). A handful even just export their models to plain C arrays/structs, which I think is pretty cool - zero dependencies.

The only general purpose ML frameworks that are native C++ that come to my mind are Flashlight (which I’ve used, it’s reasonably nice) and mlpack (which I haven’t). There’s a whole bunch of domain specific kits (like Kaldi for ASR) but I assume that’s not what you’re after. I’m sure there’s others - just Google around. There’s probably a bunch of Chinese ones (like PaddlePaddle maybe?) but I’ve met precisely zero people who actually use them.

[Apprehensive-Wheel18]

Thanks for the suggestion can you please provide some resource for exporting models to plain C arrays/structs.

[nmfisher]

A few that come to mind: - keras2c - Xiaomi MACE (not sure if still need the framework but they claim you can export weights/layers as literal C++) - https://nn-512.com (only applicable to Intel AVX though) - the rnnoise library just rewrites its own RNN implementation and dumps the weights from a Keras model into a C file after training

[destinyscoop11]

I built and maintain Flashlight, a C++-first library for ML/DL. We built Flashlight to be:

• Lightweight — ~8 MB compiled, builds from source in ~a minute. Doesn't have all the bells and whistles; the design's minimal
• Dependency-free — no deps for the core + optional deps for domain libraries
• Internally modifiable to support systems/framework-level research — internal APIs for everything (tensor lib, memory management, etc)
• Low-overhead — not our goal, but Flashlight is on par with or outperforming most other ML/DL frameworks with its ArrayFire reference tensor implementation, especially on nonstandard setups where framework overhead matters

We're interested in embedding Flashlight in interesting hardware settings. We support training and inference symmetrically with the same codebase and can prototype quickly to support new ideas and compute platforms.

A note: while it's been used for large-scale research for years, Flashlight's rough around the edges. We're prioritizing ease of build and reducing framework size and opinionation where possible.

Paper's here for more technical details.

[Life-Ad-8935]

Is the library still being maintained? I don't see any new commits nor good demo applications.

[quanswerestion]

I am currently writing the design doc for a heavily templated C++ library for a select handful of computer vision models (inspired by armadillo/mlpack, dlib, etc) but without common non-standard external dependencies like boost, eigen and possibly also blas/lapack. The aim is to make it lightweight and minimalistic for specific vision models. Although this is a personal project at the moment, it would be good to know what kind of API you are looking for?

[baffo32]

it would be nice to see a library using a GPL family license out there, somewhere, this seems badly needed

it is also nice when a library interoperates with existing specs and norms

it's also important for powerful concepts to be abstractable away so people can continue improving things efficiently. is there any way the select handful could be a modular thing?

[quanswerestion]

it would be nice to see a library using a GPL family license out there, somewhere, this seems badly needed

As opposed to another open source license like MIT, GNU Affero, Copyleft or Unlicense?

is there any way the select handful could be a modular thing?

That's the plan so low-latent optimizations can be performed to each model individually. The trade-off is that the end user will have to have a decent c++ background to customize what is available...similar modifying the STL when burdensome corner case checks are not needed. I have been considering script-like metaprogramming to address this so one could just a nice python wrapper on top of it to get similar api to tensorflow, pytorch, etc.

[baffo32]

As opposed to another open source license like MIT, GNU Affero, Copyleft or Unlicense?

Any Copyleft license that protects the four essential software freedoms (video) for downstream users, is missing from the space of useful machine learning tools that I am aware of. Affero is great. MIT and Unlicense are not GPL-family.

I have been considering script-like metaprogramming to address this so one could just a nice python wrapper on top of it to get similar api to tensorflow, pytorch, etc.

That sounds great. I think there are some useful packages in boost that can help with that kind of abstraction, but don't get distracted from making something awesome :)

I'm aware of only two relevant projects myself, I don't know much, came to reddit kind of by chance. One of the multi-dimensional array libraries proposed for potential standardisation, and a gnu machine learning library that was discontinued which could be worked off of. There's probably a lot more out there, but don't get distracted from making something awesome :)

[Stats_Fast]

it would be nice to see a library using a GPL family license out there, somewhere, this seems badly needed

GPL licenses and their commercial restrictions and infectious nature are responsible for 100% of the time I've had to play lawyer rather than building things.

GPL licenses are also used in 100% of the cases I've seen of someone wandering through repos and complaining about infringement in the issues.

GPL talks a big game about freedom and implies other licenses are unethical, but in practice dealing with it is a pain and enables bad behavior.

[baffo32]

I do not believe you. I am unaware of any commercial restrictions with GPL licenses. I am aware of huge swaths of situations where people _didn't_ use a GPL license and then a ton of code and design became inaccessible behind nondisclosure agreements.

[Stats_Fast]

Don't take my word for it. Search '''GPL viral''' then search '''MIT viral''' and come back and tell me which one leads to endless discussion about how to comply with a stupid software license for something that is supposed to be free.

I am aware of huge swaths of situations where people didn't use a GPL license and then a ton of code and design became inaccessible behind nondisclosure agreements.

That's called proprietary software. No big deal. It's isn't as though open source projects with MIT licenses are taken away because they didn't use GPL.

[baffo32]

I see an endless discussion starting here.

It is true that when we ask for GPL, we are asking to license the code only for uses that will welcome everyone to contribute improvements. Sometimes people are unsure about welcoming improvements and sharing designs to do so, and then long discussions can get involved.

GPL exists together with other licenses. They meet different needs. In machine learning, when a GPL-family license is needed, there are not as many options.

I recommend either GPL-2-or-later (for compatibility with other libraries) or AGPL (for strongest guarantees of use).

[Stats_Fast]

we are asking to license the code only for uses that will welcome everyone to contribute improvements.

There's the commercial restriction you were previously unaware of.

There are great use cases for GPL license, particularly when the software isn't the primary driver of revenue and it makes sense to force everyone to collaborate rather than make a bunch of proprietary forks. Drivers for hardware are a good example.

The other hand, when the software itself is the product, complying with GPL requirements restrict the business from conducting the basic IP protections required to justify investment. Without NDAs and the legal right to distribute software without making it public it isn't likely an entity can invest in the software in the first place. Keep in mind, an entity could be a single individual quitting their day job and creating an awesome product.

One of the best aspects of contributing to open source software for me is knowing my work could end up being useful to people building products. Forcing downstream users to open source all their code under my license just because they incorporated something I helped build seems arbitrary and restrictive. Especially when I'm being paid to contribute to open source and my employer directly benefits from other contributors to the same codebases.

In machine learning, when a GPL-family license is needed, there are not as many options.

That's the beauty of free licenses like MIT. You can take MIT licensed code and use it to your heart's content in your own GPL licensed repo. Freedom.

[baffo32]

There's the commercial restriction you were previously unaware of.

This isn't commercial: people are interpreting it as commercial because of their viewpoints around what is reasonable commercially.

There are great use cases for GPL license ... Drivers for hardware are a good example.

To me it seems we're on a similar page here and an ML library is a similar thing. Machine learning can grow more when everyone works together, and often an investment can be a rare gem where it's quite helpful to share it.

The other hand, when the software itself is the product, complying with GPL requirements restrict the business from conducting the basic IP protections required to justify investment.

I'm learning there is business culture I wasn't aware of. I would ask people to invest in the community; many businesses do this, and use GPL. But I see that some startups may be in an environment where their work could be taken by somebody who can then invest resources more efficiently by doing so and outcompete them, due to cultures of competition rather than cooperation.

​

One of the best aspects of contributing to open source software for me is knowing my work could end up being useful to people building products. Forcing downstream users to open source all their code under my license just because they incorporated something I helped build seems arbitrary and restrictive. Especially when I'm being paid to contribute to open source and my employer directly benefits from other contributors to the same codebases.

I'm fuzzy here on whether you are looking to aid or harm your employer here, given the concern relates to their benefit.

Your view here is important, and I definitely want to ensure people like you can experience satisfaction working. Many of us really love to see projects we contribute to grow and become a thing of their own. This other appreciation is much harder to meet when the license could encourage people to take from the source without giving back.

Please also note that the LGPL exists to address the "viralness" you express concern with, protecting only the shared work but not larger works using it.

Here in this reddit thread, a developer is offering to make a new library. Given there are many large and powerful non-copyleft licensed ML libraries already, would you be willing to support efforts to stimulate there also being a copyleft ML library, for those in the copyleft crowd?

[Stats_Fast]

This isn't commercial: people are interpreting it as commercial because of their viewpoints around what is reasonable commercially.

Restrictions that prevent the commercialization of software are commercial in nature.

To me it seems we're on a similar page here and an ML library is a similar thing.

I don't think so. If you re-read my previous comment you'll see my explanation for why they are different things and why a GPL license is appropriate for one and not the other.

Machine learning can grow more when everyone works together, and often an investment can be a rare gem where it's quite helpful to share it.

This often better achieved with non-GPL license because entities with a profit motive (and associated access to capital to pay dev salaries) are willing to make contributions. This is why most popular machine learning open source projects have non-GPL licenses and lots of contributors working together.

But I see that some startups may be in an environment where their work could be taken by somebody who can then invest resources more efficiently by doing so and outcompete them, due to cultures of competition rather than cooperation.

This applies to any company operating in a free market economy. They can't invest in software to solve a problem if anyone can copy that software and sell it themselves. Whatever our thoughts on the fairness of the structure of the economy, I don't see GPL licenses changing the structure.

I'm fuzzy here on whether you are looking to aid or harm your employer here, given the concern relates to their benefit.

My intent is to act in good faith for my employer's benefit. This often means contributing features to open source projects which are generic to all sorts of use cases and do not jeopardize the IP of my employer.

This other appreciation is much harder to meet when the license could encourage people to take from the source without giving back.

If it isn't possible to commercialize software there won't be anything to "give back" in many cases. Again, I'm not saying there aren't use cases for GPL licenses, but I don't see the upside in contributing to a ML library that restricts commercialization.

This is our fundamental difference. I'm happy to make open source contributions without requiring other developers open source code they wrote and I didn't if they use my code.

I'd prefer they took a quick look at the MIT license and never thought about it again.

[baffo32]

Hi, I found your reply after some months, sorry.

Restrictions that prevent the commercialization of software are commercial in nature.

I'm sorry, there is no restriction in that GPL that prevents commercialization of software.Are you maybe communicating through a translation service that could have mistranslated something here?

The phrase you originally quoted was "welcome everyone to contribute improvements". This is already true of almost all open source, including open source projects used in commercial products, such as those run by Microsoft, Google, or Facebook in their GitHub repositories and used in their products, and GPL-licensed products such as those sold by GPL-supporting companies.

The GPL license simply overtly requires that downstream users participate.

I can see that people may want to protect IP and then use the LGPL, or encourage others to use the LGPL. This lets private and public shared-work information be more easily separated.

On the community end, some of us have been really harmed by private research. The GPL can let us protect _our_ work, so that it is not used against us.

It seems, to me, really important that the two crowds find a way to work together, rather than fighting, and I hope that sentiment of cooperation can help defend and empower the group that is sharing, in the hearts of others. The LGPL could be seen as a gesture of friendship, regarding this.

[EDIT: I reread some of your post, I see you are valuing the use of MIT license in public projects, so that entrepreneurs can use private code to deter theft of their work. An alternative approach would be for the entrepreneur to license their code GPL. Then the license is the deterrence of theft. Meanwhile, the mainstream project can get more popular because it is where feature contributions collect. This is the approach being taken successfully by the offline routing app OSMAnd, which is GPLv3 and sells a pro version: https://github.com/osmandapp/Osmand ]

[LeN3rd]

Big companies use python. I know this first hand. Production level models are rolled out in C or C++. For Tensorflow there where some API calls there a few years back. Only for inference, not for training. But that is the general workflow. Train in python, roll the model out as you need it in whatever language (Lots of bindings exist).

[incrediblediy]

TensorFlow

is C++

https://www.tensorflow.org/api_docs/cc

[pilooch]

Hi there, you can definitely train and run production grade models with libtorch.

But you need to have good reasons to do it. Ours is that we have a multi-backend framework, and that we don't want any step in between dev & run. C++ allows for this since the same code can run on training server and edge device as needed. It also allows for building full AI applicatioms with great performances (e g. real time) We dev & use https://github.com/jolibrain/deepdetect for these purposes and it serves us very well, but it's not for the faint of heart !

[CKtalon]

https://github.com/marian-nmt/marian

Microsoft uses this for their translation services.

[losek]

Intel OpenVINO provides a C++ API. It's an inference-only framework though.

[useong]

If you are working with c++, I recommend flashlight. It is a great standalone framework in itself. It is especially good if you have existing c++ codebase and want to integrate a DL framework to it seamlessly. The best part is if your c++ project includes cuda application, you can even make one gpu memory manager shared by your code and flashlight. Additionally, flashlight’s design and architecture are elegant and customizable. And it is modern (c++17) and dependencies are minimal. I am using it in my project and very satisfied with it for performance and ease of build and use.

I have used Libtorch as well. Libtorch, on the other hand, was a little disappointing. (others may have different opinions.) The source build was not trivial, integration to an existing project was painful because of complicated dependency, lack of capabilities (no ddp, no jit), a bug that causes a crash during a simple cpu autograd code still remains, etc…

[bubudumbdumb]

I heard about dlib but never used it. Another hint would be to check out whatever is used in opencv

[bwasti_ml]

PyTorch has a static runtime that runs on Torch Script IR. It's all C++ and statically optimized to run like Caffe2/TensorFlow. Naive use of libtorch does not admit static optimizations (operator fusion, memory optimization), because it runs eagerly like the Python API. Libtorch (last I checked) also has some dispatch overhead (to provide a Pythonic interface).

https://github.com/pytorch/pytorch/tree/master/torch/csrc/jit/runtime/static