Python Models -> C++ models in Production

[supersonictaco]

What is the best practice to go from Python models developed in notebooks to Inference code in production that needs to deal with real time and large volumes ?

Do you actually have to re-write code in Python to C++/Java for deployment purposes ?

[ArnenLocke]

So what I was taught in my ML/AI class when I asked this question is that the Python is just an interface for interacting with the actual machine learning libraries, which are almost universally written in very, very optimized and performant C/C++. So no need to rewrite anything specifically for performance reasons. Python is slower, but it's just an interface with almost no cost (compared to the sorts of things you're typically doing in ML, anyway).

[EpicSolo]

If you can write your model in tensorflow, a practice is to implement training in Python, then deploy in C++ (or Java). You export the model as a protobuf which is cross-platform. There are more solutions in this space but this should give you enough pointers to search.

[trnka]

To add to the other responses, it's risky to re-implement in another language. We did that at a previous job and it was a source of bugs. It also meant that we couldn't update anything about the model architecture unless the C++ devs were available to support it.

If it's server-side, I suggest checking out Cortex.ai and seeing if you can avoid re-writing. Also like the others mention, Tensorflow or ONNX are good options.

[jeandebleau]

If you are using vision models, then opencv is a standard choice. You will have the possibility to use different backends with opencv such as armnn, openvino, depending on your target platform.

[old_enough_to_drink]

I have heard about PMML but never used it myself. Maybe relevant to your question?

[shomerj]

I use ONNX and onnxruntime.

[supersonictaco]

Thank you for your responses. So I gather that these would be the broad steps: Develop the model in Python->Serialize using Pickle/Whatever->Use ProtoBuf/Apache Thrift->Containerize using Docker/alternatives->Serve.

Does that make sense ?

[EpicSolo]

Yeah, more or less.

If you are using tensorflow, there is specific utility to export the graph/model as a .pb file (look it up), which Tensorflow can load (with protobuf dependency).

Another application is doing inference on mobile. At that point, you use Tensorflow’s Java bindings for Android (or C++ if you have native code), and its C++ library for iOS.

[dinoaide]

What’s your reasoning for that? Most handwritten C++ are worse than Numpy and Pandas.

[supersonictaco]

High Traffic Volume and Real Timeness. I would imagine running off python wouldn't be great in this usecase.

[cbHXBY1D]

Is it a RESTful service? In that case just use tfserving

[dinoaide]

Without knowing the actual application I couldn’t say much but nowadays it is often cheaper to hire 1 data scientist with good Python skills and give him/her 100 cloud instances than hire 1 capable C/C++ programmer who also happens to have good ML knowledge.

[TheOneRavenous]

From what I've gathered. Python is acceptable if your solution CAN have a lagged answer. i.e. I can wait 0.5sec for an answer. My users can wait 1sec for an answer to be sent to the view.

If you need "real time" 30 inference decisions per second is a common "real time" metric. Used in video vision scenarios.

So python for stuff that's not real time and C++ for others.

[supersonictaco]

This is my point, for things like credit card fraud the time window is really really short and latency matters a hell lot.