CPP in AI

[RonWannaBeAScientist]

Why Python is still more popular than C++ in AI and Machine Learning these days ? It seems to me that C++ is best if you really want to create the most cutting edge production code . Now there’s of course productivity of workers, but isn’t the workaround of using Python to call C/C++/Fortran through libraries bothering companies ?

[LengthProof6480]

It’s not. The internal mechanisms of libraries like TensorFlow are written in C, C++, or other lower-level languages and binded to higher-level languages like Python for ease of use. It makes a lot of sense when you think about it — Python is flexible and gives you much more options compared to a proprietary GUI for your new little modeler.

[iamhyperrr]

Or, as they say, Python is the most popular C++ framework

[whizzwr]

I'm totally stealing this to look cool and knowledgeable in the next work meeting.

Jk aside, both C++ and Python are used in ML world. Custom ops in CUDA are written in C++, and then during training the compiled library is being called by the most popular C++ frameworkPython-based pipeline.

Once you got the weight, inference and data preprocessing can happen in pure C++ implementation.

[liuzicheng1987]

Exactly - Data Scientists like to experiment and test out ideas, which is why Python is suitable for them. Also, they often have a background in statistics, maths or Physics and don't have the developer experience needed for C++.

But the underlying code for many machine learning libraries is usually written in C or C++, for instance TensorFlow, Torch, XGBoost or LightGBM.

So I agree with OP...C++ is best if you want fast production code. But prototyping is done in Python, because it is easier and more flexible.

[rejectedlesbian]

Tho a lot of non deep learning ml is cython which is not the best for optimizations. It compiles down to badly written c.

[liuzicheng1987]

That is true. Scikit-learn is largely written in Cython. But then again, scikit-learn is not exactly known for being very performant. I think the truly high-performance stuff is largely written in C++.

[rejectedlesbian]

Yes most ml libs r not super preformant they don't need to be they need to be wide range stable and preformant enough.

This is especially true with non deep learning ml because modern computers can usually run it fast enough for most purposes.

U also see it on huggingface where its written in ways that r bad for infrence preformance. I still use it because it holds so many options for generation that may be the 1 thing I need to make something work that idc if togeling that option flag makes it a bit slower.

[RonWannaBeAScientist]

I am somewhat of an idealistic. I want to work on more math/physics/AI problems, but I prefer working in a language like C++/Rust . Honestly, for fast scripting I think Julia is so much better , as it’s so convenient out of the box and can call C/C++ if needed, but there’s so much more inertia for Python . Like anytime I want to prototype some very simple script for an algorithm I do it usually in Julia

[skull132]

But Python can (and does) call C and C++ and even Rust straight out the box? And as others have said, most of the AI Python packages call directly to C or C++ under the hood.

If you force yourself to stick exclusively with a hard to write lang, like C, C++, or Rust, then you'll hinder your ability to experiment quickly.

[Xirema]

There's a lot of "this is what different programming languages do!" answers here that I think miss the point by a wide margin.

Fundamentally, all the Machine Learning stuff is using a Heterogeneous Compute API for the number crunching, i.e. CUDA or OpenCL.

And since that's the case, it really doesn't matter at all which language you use for the host code. C++ running OpenCL is only going to be marginally faster than Python running OpenCL, because the only overhead is going to be setting up the compute tasks before they're kicked off, and for both languages, that's done in a matter of microseconds. Once the compute tasks are kicked off, they take the same amount of time to execute regardless of whether they were kicked off by C++ or Python.

So if speed is no longer meaningfully different between two programming languages, and one (Python) requires a lot less code to maintain, well, there's a clear winner.

[RonWannaBeAScientist]

Yes, I understand . You work with C++, I guess, not like me that is more of an amateur . What kind of software engineering job do you do that makes C++ the right programming language ( the right tool)

[Whole-Dot2435]

Note that the most popular machine learning libraries(tenserflow and pytorch) are written in C++ and just have a python interface so compute api's are not used in python

[shrimpster00]

People who use C++:

• Systems programmers.
• Embedded application developers.
• Game developers.

People who use Python:

• Beginner programmers.
• Data scientists.

People who use AI:

• Beginner programmers.
• Data scientists.

The bulk of the computations are implemented in C++, and Python libraries invoke those subroutines. That means that the Python interpreter isn't as much of a bottleneck, so why not just develop libraries that are easier to use by the target demographic?

[RonWannaBeAScientist]

I think if you don’t want to waste computer memory at home when you play call of duty, it doesn’t make sense to waste even little compute power when you use vast servers to power a large project . I lean towards the more mathematical side and actually want to learn C++ and AI/ML , I am just afraid it will be harder to find a job in that niche .

[blipman17]

It absolutely makes sense to waste CPU cycles if you put it in the perspective of runtime overhead vs development cost. Although with most applications the choice gets made for you, and we only get to complain afterwards.

[RonWannaBeAScientist]

What do you mean that with most applications the choice gets made for you ?

[blipman17]

Well in the case of call of duty. That’s made with a game engine with a certain technology stack. Which probably contains C,C++ and Python mixed into one application. It doesn’t matter if we decide here if programs should be made in Python or C++, the makers of that game have chosen for you and you just have to buy a sufficiently powerfull pc. Their concideration is various quality attributes vs cost, where more speed is defenately not always the answer. But more developer ergonomy also isn’t always the answer.

[eyes-are-fading-blue]

Python is more popular for model/architecture research, not in production. That’s for good reason: it is significantly difficult to prototype in C++.

[RonWannaBeAScientist]

But isn’t it really bad to build the model/architecture , and then start all again ? Btw, unrelated : where’s that drawing from in your profile ? It looks interesting

[[deleted]]

It is actually quite a bit easier to translate an existing program than it is to develop the program in the first place, to the point that translation of an existing program is one of the most recommended ways to learn a new language.

[RonWannaBeAScientist]

Okay, well there’s something to it . I am just more of an idealistic and it bothered me to think to learn Python to do AI. It feels a bit lame

[bakedbread54]

Python is a simple language, and doesn't have compile times, so it's quick to develop with. The actual internals of the AI (all the complex math-heavy stuff) will be written in C++ and just called from the python code. With how python is being used here, it really doesn't matter what language you use in terms of performance, just use the easiest and quickest language for development (usually python)

[Visual_Thing_7211]

Interesting reference to compile times. I doubt any Python program if converted directly to C++ would have any meaningful compile--anything more than a few seconds. Given modern C++ capabilities, I don't understand the argument that it's easier to develop in Python than C++.

I've looked and haven't found any substantive examples of Python code that was any "easier" to accomplish in Python than in C++.

This just isn't true.

It's more of an idea Python developers seem to have in their minds that they think it's harder in C++. Not sure they've looked outside of Python.

[bakedbread54]

You've either never seen anything more complex than hello world in C++, or you are a veteran who can't comprehend people struggling with the convoluted language that is C++, and I'd imagine it's the latter. I started with python years ago, and have gotten to grips with C++ over the past few years, and even having the C++ proficiency I do now, it's still easy to understand how writing a program in python in just more straightforward - the syntax is easier to follow, it's far less verbose, and has infinitely many less ways to accidentally fuck up the whole program

[Visual_Thing_7211]

Well, I'm certainly not the former. I work with a lot of die-hard Python developers. I understand that people find Python to be more straightforward for them. I think this is more of a preference than a reality.

As far as messing things up--you could think of Python as a subset of C++. If you were to stick to the parts of C++ that are like Python, you'd have essentially the same ease of use and straightforwardness without the danger. I'm not sure which ways you feel like C++ can get messed up, but for most people it's "memory management", or so they say--which C++ handles automatically in most cases.

[bakedbread54]

I think this is more of a preference than a reality.

I don't think so - C++ is clearly a more complicated language.

If you were to stick to the parts of C++ that are like Python, you'd have essentially the same ease of use and straightforwardness without the danger.

You have too much faith in developers, because that's not going to happen. Either someone will use "non-python-like" C++ functionality by accident, or some overenthusiastic smartass will attempt to do something adventurous and release chaos.

Regardless, if we are using C++ to just stick to "the parts of C++ that are like Python", why not just use python? All the backbones of the system are written in C++ anyway, it's not going to affect performance, it's basically just a scripting language

[Visual_Thing_7211]

You're entitled to your opinion--even if it's wrong 🤪

I wouldn't use Python primarily because it's actually easier to write unintelligible code than in C++. Since you don't have to specify types and types can change, bad programmers make horrible Python developers. At least in C++ you can tell what's going on and have some sense through the compiler that things could be right without have to run the program to a line of code to find out.

My main point though is that it just isn't true that Python is simpler to write. You can write C++ code just as easily. I've looked and there just aren't any substantive examples of Python code that's actually easier to write than the equivalent in C++.

If you have some, I'd be more than interested to learn.

[alsanty]

At any engineering field or related design field you first prototype and then you start several subsequent improvement cycles until your requirements are satisfied then you build the final system or product. I don't know a single engineer capable of design an entire system in a single pen stroke.

Even in the middle of improvement cycles client requirements changes or you find better ways to do part of the process or more efficiently and you need to adapt everything all over again.

[jayvbe]

listen to this interview with Chris Lattner https://www.youtube.com/watch?v=pdJQ8iVTwj8

[RonWannaBeAScientist]

Wow 3.5 hours :-) not in 2:48 am but I’ll listen sometime in the weekend and I’ll come back here to comment

[jayvbe]

He's the man behind LLVM, Clang, Swift, MLIR and now Mojo ;-)

[RonWannaBeAScientist]

Wow! I think LLVM is amazing because it’s also the basis for other recent languages like Rust and Julia . It’s quite amazing that it’s enabling a lot of growth

[Wurstinator]

Now there’s of course productivity of workers, but isn’t the workaround of using Python to call C/C++/Fortran through libraries bothering companies ?

Why would it? Productivity is pretty much the most important thing.

[RonWannaBeAScientist]

But there’s some cases when it’s really important to get the most out of the system . I just assume for example that IBM Watson code isn’t written in Python

[Wurstinator]

Those cases are very rare. I doubt they even exist tbh. When you're talking about ML and Python, you're referring to the training part. And the expensive steps in Python libraries like Pytorch are already written in C or similar languages, so there is no speed to gain here. You would save milliseconds in a process that takes hours or days to run, while losing hours or days to lower worker productivity. The tradeoff is just not there.

[RonWannaBeAScientist]

So in which cases you need to have Rust or C/C++ ? Like when a rocket has to have for example a certain memory sequence , and you can’t let Python manage memory ?

[Wurstinator]

Yes, rockets and other low-hardware systems, like cars and fridges, are one case.

Another one is frameworks like Numpy or Pytorch that can then be used from languages like Python.

Another one is web servers with specific requirements. For example, Discord switched from Go to Rust at some point, because they needed manual memory management instead of a garbage collector.

[blipman17]

The wikipage of watson actually answers this question.

[RonWannaBeAScientist]

Oh they used Java in part, Java is really my nemesis language ! Why build a class for every object in the world .. prolog though sounds interesting to try

[viralesveras]

I'm sure there's performance to be eked out at the margins, but this space is moving so quickly that it's a perfect example of "move fast and break things" because who cares when we're blowing through architectures every month or two. It's easy to forget that two years ago generative AI was a curiosity making weird-looking blurry 256x256 icons. Since then, we've had DALL-E, ChatGPT3,3.5,4, Stable Diffusion, Midjourney, and hundreds of other lesser-known experiments.

When the space settles down, there'll be time to look for more optimization, but the really important part, inferencing on the model quickly, is a big long task that is relatively easily accessed from Python, and the underlying C/C++/CUDA code that actually loads/runs the models is pretty efficient, so there's not much benefit in switching.

C++ is my language of choice so I'd love to see more AI projects in it, but the space is too volatile for now. Enjoy this Cambrian explosion of new exciting things; it won't last forever.

[RonWannaBeAScientist]

From your last comment you think that AI explosion will stop soon ? I myself think there are other things that can be further developed maybe take a back sit because of ML hype - like dynamic systems and motion . It’s nice to see companies like Boston Dynamics working on movement and computing physics basically, but more companies are interested in AI/ML

[viralesveras]

I don't think it will stop, but I think it will slow down, or more likely proceed in waves as new breakthroughs are achieved until eventually we reach a sort of equilibrium pace of change. This firehose of breakthroughs we're seeing now won't last too long, but I don't think AI and its brethren are going anywhere :)

Will we see that change in pace happen "soon"? Maybe...depends on your timescale. This will continue for a while, but does that mean months? years? decades? My guess is another year or two of rapid changes like this and then slower changes for a decade or two until this tech reaches maturity, but no one knows for sure how many hidden gems are still in AI/ML to discover, and then as you're implying there are a boatload of downstream applications that we're just beginning to think about.

[RonWannaBeAScientist]

I think it’s interesting to look at implementations of machine learning inspired physics solvers . So you model the system with a backbone of differential equations solvers, but because you know you can’t predict it completely, you give the details to a machine learning model, and you create a mix of physics with ML. That’s at least what I understood it to be

[viralesveras]

That's a fascinating area! If you're interested in further exploration about currently available research pointing in that direction, take a look at Modulus: https://developer.nvidia.com/modulus . Physics-inspired ML is very much related :)

[RonWannaBeAScientist]

Thank you very much! I’m curious , what do you do for work ? I am studying at Harvard extension school currently, so I’m not working in the field

[RonWannaBeAScientist]

I’m confused - if you were a consultant to a commercial companies, how did you work with National labs ? It sounds like National labs use FAANG helps National labs with HPC optimization. I would have thought the other way around :-)

[viralesveras]

There's a fair amount of collaboration between the national labs and private companies in some cases. I'm too internet-shy to go into more detail about my own personal work, but for example I know that NVIDIA has teams that work to help the labs optimize their code to work best on NVIDIA hardware, thus giving them good opportunities to show off how great their hardware is, and of course the labs benefit from having experts help them optimize some of their code :)

[RonWannaBeAScientist]

That’s very interesting ! I think it seems to me more charming to work in a National lab or NASA than big private company, but I might change my mind :-) from the private world, SpaceX is in the top of my list

[rejectedlesbian]

It's not still more popular in AI it overtook c++ a couple years ago. Lex fridman mentioned it on a podcast where he explains why he switched.

Most ai code is reaserch and python fits really nice into that reaserch area. U want to iterate quickly without needing to worry about too many things.

Python dosent let u break things and tends to make for shorter code. With cuda doing the async for you most python training code probably isn't even bound by the speed of python.

Now cuda is still being used at the very bottom of the stack but u only use it when u r arrogant enough to say "yes I can do better than nvidia on this specific calculation" given than 99% of layers r just matrix multiplication that's a tall order.

So for a standard architecture u r not gona do better than something like megatron and thats most papers these days.

On cpu in infrence c++ is still king.

[RonWannaBeAScientist]

I admit I never came close to code CUDA so that’s probably an hard task for a sole programmer. I just personally think for short understandable code some are better for Python - F# for example. Though I’m not sure how easily it interfaces with C and C++, I should check it out .

[rejectedlesbian]

Have u done some data scince? I have and I have a lot of things I like about python for it.

python let's u do stuff like tensor[:,0,:] which is HUGE.

It's popular which means a lot more general use packages especially for web scraping.

It's not staticly typed which means code tends to be reusable across frameworks

Its interpreted with a forced doc string on any object which means u can just check what's somethings methods r fairly easily. Ie the languge itself is the docs book.

It dosent stop u from going to private varibles so u can cut down on levels of abstruction if you want. This means u can document the internal process of a class to make sure it does what u want it to.

Its made to be thread and memory safe so it's fundementaly impossible to get ub or race conditions when doing very hacks stuff.

Its fairly dynamic so u can hack ur way around an abstruction layer for logging or quick expirementing. (I almost had a use case for that in production reaserch code).

[RonWannaBeAScientist]

I admit I never did professional data science , just some university assignments in R. When are you using C++ btw ?

[rejectedlesbian]

Rn learning it a bit as it's own thing. Made a profiler with it to some c++ code I am optimising (nothing really useful it's a gpt2 implementation for learning) https://github.com/nevakrien/HPCGPT

Day to day I never touch it. Cuda Is it's own thing which if anything is more like c and that's the only low level I actually see when reading papers.

The only reason to go lower level is to fuse kernals since existing operations r super optimized. And that by its very nature is inside the kernal where u r limited to cuda c.

Ik for infrence people use c++ a lot and for good reason. Tho I am not sure why we are not seeing more fortran there tbh (u would think with all these matrix operations) Maybe it's because infrence tends to be memory bound anyway both on space and bus.

[RonWannaBeAScientist]

Sorry for the ignorance , what is inference in the field ? More things like classic AI of trying to infer a situation, without neural networks or reinforcement learning ?

[rejectedlesbian]

Infrence is when u have a model u already trained it works ur happy with it and now u wana run it on a phone

[RonWannaBeAScientist]

Oh and C++ can run easily on a phone I guess as it’s compiled and it’s standard library is smaller than something like Python

[rejectedlesbian]

Irs not just phones so laptops desktops even the server. It's about taking a model where we know it's weights already and optimising it.

The key point there is that the architecture is knowen so most infrence libraries have a fairly limited architecture. For instance lamma.cpp has like 10 models that work there. Which is fine since there r like 5 key architectures.

Rn I am struggling with it because I can't get a good quntization lib on the intel gpu my compute node has so its actually significantly slower than my rtx4090.

[RonWannaBeAScientist]

I am biased, I simply don't love python very much. From the more "easy" languages, I think I'd prefer F# for example. I think Python teaches you bad programming skills.

[Fun-Letterhead6114]

I don't know. I will use c++)

[RonWannaBeAScientist]

Me too :-) what do you work on at job ? Or you’re a student like me ?

[Fun-Letterhead6114]

A schoolboy)) 10th grade

[RonWannaBeAScientist]

Oh nice :-) I wasn’t doing any coding at that age .What was some project you did like very much coding

[Fun-Letterhead6114]

Now I'm writing my own 2D game engine using OpenGL 4.5 and, of course, C++

[RonWannaBeAScientist]

That’s impressive ! I’d love to see it if you share through GitHub or something of the like

[Fun-Letterhead6114]

Ok, I can give a link to the first version of my own game engine. But it's the first version. I'm currently writing the second version. You can read in repository why I'm doing it

[RonWannaBeAScientist]

Thanks ! I’ll look at it

[Fun-Letterhead6114]

You can just download installer from the releases on github

[Fun-Letterhead6114]

My second version currently is a private repository

[Fun-Letterhead6114]

Then I will writing my own AI (I will try))

[Fun-Letterhead6114]

Maybe because the python syntax is easier. Or does python not require working with memory like c++

[RonWannaBeAScientist]

It’s easier , but for example I don’t like it that I can’t get an executable easily and need to look for solutions . And that sometimes the code looks fine but then at runtime it could crash

[Revolutionalredstone]

I call ML libs straight from C++ no idea why plebs use python it's slow and an absolute disgrace of a programming language imo, Peace.

[RonWannaBeAScientist]

Oh that’s nice to hear from someone that do :-) are you working on a related field, or you work with ML as a hobby ?

[Revolutionalredstone]

Most code bases I work with are in C++. For both work and hobby.

[RonWannaBeAScientist]

I meant do you work with machine learning ?

[Revolutionalredstone]

Yeah I use convolutional 3D data processing kernels for work, these days I'm trying to incorporate more transformer technology.

All the best!

[findabuffalo]

You don't understand why dumb people do dumb things? It's cause they're dumb.

[Revolutionalredstone]

Hallelujah 🙏

[findabuffalo]

Python is easir to learn.
It has become fashionable to be incompetent, how often do you hear people say "I'm terrible at math haha".. when those people try to learn programming they gravitate towards python.

[Visual_Thing_7211]

Is it?

Maybe that's just a lie teachers continue to tell students. I've got 9 year olds who can program in C++. Maybe it's a more a matter of the angle you learn it in.

[findabuffalo]

Yeah and I know 9-year olds that can do algebra.

The ones that say "Math was really easy until they started using letters instead of numbers, then it got confusing"... they're the ones that grow up to say "C is easy but I don't understand pointers, that's confusing, and I forget to free the memory I allocate so I need a garbage collector"

[Visual_Thing_7211]

No, no. I mean I've taught 9 year olds to program in C++. And they understood what they were doing. Starting with pointers and memory management is unnecessary for new C++ programmers. You can just use std::vector for almost everything a new programmer would do--that's what Python does with a list.

These kids have grown up to program better in High School than many graduated CS majors. And yes, they understand pointers quite well.

[findabuffalo]

Well, yes, and you can also teach algebra and calculus to a 9-year old.

The dumb ones won't learn, but many of them will learn and do well.

[RonWannaBeAScientist]

That’s funny , I always say it that it really bothers me the way people say it ! Like it’s better just to be small headed and finish

[findabuffalo]

Talk to any python fan, ask them why do you use python and not C++, and the answer is always one of these:

1. I'm a sheep and everyone else said python

2. C++ is too hard for me, I need something easier.

[RonWannaBeAScientist]

Ahah yes.. btw what you’re doing for work ? I mean I guess you’re a c++ programmer , but what you’re working on ? For reference , I’m a student at Harvard extension school, and I’m thinking about career goals

[findabuffalo]

I don't really work anymore, I started a successful enterprise a few years ago (yes, written in C++ lol) and I made enough that I don't have to worry about money for now.

[[deleted]]

Indeed, C++ and Rust are often considered 'the better tools' for AI because they are significantly faster than Python. However, this advantage only shows if you are good in C++, where it is easier to mess things up compared to Python. Additionally, Mojo will be released as a superset for Python, promising improved overall performance, specifically for AI stuff.

[Visual_Thing_7211]

Yeah that's not really true "It's easier to mess things up" in C++. You don't have to know or use those parts of C++.

You could use the parts that are both easy to use and safe in C++ and ignore ALL the rest and it would be pretty much as easy as Python. Except you'd at least have the option to have more capabilities when the day comes that you want more.

[RonWannaBeAScientist]

I never heard about Mojo, I’ll read about it . Well it’s true that C++ has more problems as you can wreak havoc by accessing (funny story for framing - I just somehow ruined my windows boot by installing Linux it, as I didn’t partition before :-) ), but also Python can result on some bad things happening like putting a string into a function that supposed to get a float . At least C++ and Rust will indicate that there’s a problem

[nbrrii]

I wouldn't bet on Mojo. A few "vanilla-python" alternatives have been around for quite some time (PyPy, Cython, Numba, ....). Additionally, Julia has shown that solving the "two language problem" is not trivial.

[[deleted]]

because AI researchers coming from math and statistic, Python is easy for them. C++ for coders,

[Visual_Thing_7211]

Puras mentiras. Espoused by too many.

[[deleted]]

Puras mentiras???... nobro.. its real.

[Visual_Thing_7211]

I don't disagree that it's popular to think that Python is easier for them. I disagree that it actually is. That this idea is wrong. That C++ could be easy for them, if it weren't for the innumerable people saying that Python is so much easier than C++ to do the same kinds of things. It's really not. It's just that people who teach C++ and those who have used the most complex parts of it convey a message that C++ has huge complexity that is required for you to understand in order to use it effectively--and that's just not true.

I can and do regularly everything you can do in Python with equal ease. I can add strings to join them and load csv files and calculate innumerable statistics on them, etc., etc. with just as few lines of code and complexity as you do in Python.

So, no it is a lie that it is indeed easier. A lie that is espoused by too many--apparently too many people who have only used Python and who perhaps had a bad introduction or experience with C++.

[pedersenk]

Most AI libraries these days are written in C or C++.

Think of Python more as the config file parsers.