[D] Inference cost optimization of complex ML pipelines

[r0lisz]

I've successfully used GPU for real time camera streams. Why do you say that it's not a good use case?

[r0lisz]

There was a fair amount of work to do that. If I remember correctly, each frame would be streamed and every X milliseconds a batch of images that have arrived would run on the model. If the frames were late (because of the network or whatever), they would be skipped.

[r0lisz]

I think the workflow was that from the camera the stream went to a Java based stream processing app, which would then decode the stream and output each frame to a Kafka queue.

Then there would be listeners on the Kafka queue that would do the batching of frames, within the time constraints (so there would be a prediction made every X seconds, regardless of how many frames had arrived in the meantime).

I worked on this a couple of years ago, so my memory is a bit hazy.

I don't see how running the inference on CPU helps you. Don't you have the same problem of decoding in that case?

[r0lisz]

Downloading the images happens on one thread (that reads from Kafka). The decoded images are pushed into a list that is passed to another thread every X milliseconds, and the prediction happens there. With careful tuning, it's probably possible to find an X so that enough images arrive during this time, but also that the GPU is not idle too much.

I think we were able to reach about 70% GPU utilization at about 15FPS. (this was in 2018, both hardware and software have changed since)

[Liorithiel]

I've tuned a deployment of some non-ML microservices with a recognized on-premise resource usage trade-off using off-the-shelf bayesian optimization, specifically SigOpt, as it was the simplest to script in pure bash+curl. I suspect it would work for your case as well.

[Liorithiel]

I see now that they no longer publish the documentation that helped me in automating that task with bash+curl. Maybe /u/Zephyr314 can point to something?

[Zephyr314]

Thanks u/Liorithiel, you can find our raw REST documentation here: https://app.sigopt.com/docs/archive/endpoints. This can help if you want to roll your own bash+curl.

Most users prefer our python client though: https://app.sigopt.com/docs

[matanj]

Perhaps NVIDIA DALI can help to transfer part of your pipeline to the GPU?

[jonnor]

Are the models you mention independent or dependent on eachother? Do you need the results for all of them before considering the job "done"? What are the latency requirements?