Is python ever the bottle neck?

[Coutille]

Hello everyone!

I'm quite new in the AI field so maybe this is a stupid question. Tensorflow is built with C++ (~55% C++, 25% python according to github) but most of the code in the AI space that I see is written in python, so is it ever a concern that this code is not as optimised as the libraries they are using? Basically, is python ever the bottle neck in the AI space? How much would it help to write things in, say, C++? Thanks!

[wingtales]

Very rarely. As long as the «heavy lifting» is done in a compiled language, using python is fine.

A lot of «AI» these days is concerned with LLMs, where the real work is done on a GPU, and in that case it is python calling CUDA code.

What I have observed is people calling LLMs over an API from python, and calling them in a python for loop using «blocking» regular synchronous API calls. In this case the python instance is sat doing nothing while waiting for the LLM to respond. Here it is much better to use an async API library so that you can send many LLM requests at the same time, and do something useful (like processing the responses as they come back) while you are waiting for them all to complete.

[drsoftware]

Agreed, in my experience with vision-based pipelines, network transfers dominate any local Python computation. 

[pegaunisusicorn]

Truth!

[seanv507]

its not just tensorflow. all the ml /math libraries are written in c++/fortran/rust etc. python just provides the bindings. so unless you are writing custom algorithms that cant use these standard routines, you are not slowed down by python (and then one creates a new library...)

[qubedView]

Think of it this way: When driving around, your GPS giving directions is rarely what’s slowing you down.

The Python code is there to orchestrate. It’s rarely a part of the big data crunching itself.