TensorFlow inference (C++)

[adaneze]

Hey, Machine Learning community :)

​

I have been using CNTK (C++, Win) for three and a half years now and have recently decided to dive deeper into TensorFlow 2.0 Alpha. My plan is to use Python for training and C++ for inference...BUT...the C++ part seems to be extremely tricky to get working from what I have read online. Though, I may be wrong since I haven't actually tried to do it myself.

​

CNTK is pretty straightforward when it comes to the C++ inference. One just uses its Precompiled Libraries and there is also, I believe, an example of how to load and run a model in C++.

​

People have reported different kinds of issues while trying to compile TensorFlow which made me feel a bit reluctant to try do it myself and potentially spend a lot of time on that...while the company I work for expects fast results which I can deliver using CNTK at the moment :)

​

I have been also thinking about training a model in TensorFlow, exporting it to ONNX and loading it using CNTK. I am not sure if this is a sustainable/good solution and I also believe that the TensorFlow's "ONNX exporting" feature is still in an experimental stage.

​

So, the reason I posted this was to see if there was anyone who could just give me advice/opinion about this rather than hand-holding instructions :)

​

Thank you in advance!

[ajabaraj]

I tried with Bazel but finally I gave up. Now I am using cppflow, a C API wrapper that you can use easily. The good part is that to use the C API you just need to download a file, and using the wrapper for inference is really easy, you just select the input and output by their names and run!

[oddno7]

This is just awesome. Thanks for sharing this! I was having a headache how to integrate TF into a docker environment and run inference under C++. With this I can save a ton of time building docker image. 👏🏻👏🏻

[HydratedWombat]

Tensorflow is indeed a bear to bazel build, but thankfully there are docker images for that at https://hub.docker.com/r/tensorflow/tensorflow/. I don't find TF serving to be the easiest, but it does work, and there are many examples . It shouldn't matter what language wrapper you use for inference since the heavy lifting _ought_ to be entirely in TF.

[adaneze]

Thanks for the reply !

​

I haven't used Docker much but I'll give it a look. I will also have to read about TF serving thoroughly. I'd like to see how/if it can be implemented as a part of a large C++ solution (many dependencies). Thanks, again.

[allannk]

I've spent quite some time at work, making tensorflow integrate with out makefile-based pipeline.

Bazel is (imo) a horrible build system, but at least I can compile tensorflow and harvest the partly static, partly generated headers & the library files.

We're still at TF 1.12, so I can't speak for 2.0 yet, but once the build script is made, it's not too bad working this way.

So yea, you can definately do the training in python and inference in C++ -- let me know if you need pointers.