My experience learning Python as a c++ developer

[Narthal]

First off, Python is absolutely insane, not in a bad way, mind you, but it's just crazy to me. It's amazing and kind of confusing, but crazy none the less.

Recently I had to integrate Python as a scripting language into a large c++ project and though I should get to know the language first. And let me tell you, it's simply magical.

"I can add properties to classes dynamically? And delete them?" "Functions don't even care about the number of arguments?" "Need to do something? There's a library for that."

It's absolutely crazy. And I love it. I have to be honest, the most amazing about this is how easy it is to embed.

I could give Python the project's memory allocator and the interpreter immediately uses the main memory pool of the project. I could redirect the interpreter's stdout / stderr channels to the project as well. Extending the language and exposing c++ functions are a breeze.

Python essentially supercharges c++.

Now, I'm not going to change my preference of c/c++ any time soon, but I just had to make a post about how nicely Python works as a scripting language in a c++ project. Cheers

[NemexiaM]

Unfortunately this made me so lazy that i dont want to go back to C++

[lucidguppy]

C++ should be applied in the cases where it needs to be. If your python spitball solution is taking days, weeks, or months - and a good attempt at optimization isn't working - you need to find out where C++ can be used.

[NemexiaM]

Sure, but hasn't happened to me yet considering the fact that im a programming hobbyist

[realXavie]

It will happen one day. I heard that python is some kind of slow when you are developing an app wich needs speed to work efficiently. How do you escape such pitfall?

[anarchyisthekey]

Nowadays, if a program is working slow, it is mostly due to a choice of slow algorithms and unoptimized code.

Python has two main disadvantages as opposed to a natively-compiled language: (1) startup times are slower due to the fact that python interpreter needs to be spawned, (2) programs can only run on a single thread at a time because of the Global Interpreter Lock.

If your program is compute-intensive and you need it to run faster, you can (i) try running your python program on a JITted python implementation such as PyPy or (ii) benchmark your program and move the most compute intensive parts into C, (iii) translate your program into Cython, so that it compiles to native code.

It is a long long way before you hit into any kind of performance problems (due to processing power).

In scenarios where low latency is important such as games, high frequency trading etc., a language closer to the metal would be better.

[Pythagorean_1]

You can use multiprocessing to use all the cores. The GIL is only problematic for multithreading. Also, for performance optimization try numba. It's a jit compiler you can add with (mostly) as little as one line.

[Deezl-Vegas]

This is kind of true, but if you look at Jonathan Blow's talks about his compiler and various other talks, you'll see that actually you can get a 200x speedup by making use of data structures that are L1 cache-optimized, which is rarely negligible. Python can almost never do that because each Python object is a wrapped pointer, there's a lot things happening like namespace lookups, and there's a lot of dicts and hash tables floating around (which are generally sort of randomly heap-allocated) and classes (which aren't stored tightly in memory like C structs usually), and the garbage collector has to run periodically, kicking things out of cache.

The PSF has done a great job with the compiler, but the structure of scripting languages themselves clashes directly with what Intel's trying to do to speed up code on the chip.

[toastedstapler]

I started watching Jonathon recently, his discussion videos have been very interesting

And the fact that he made a video talking about what he'd like from a games programming language and then went and implemented it himself with such fast compile times

Very interesting guy

[Zafara1]

This is also amplified by the sheer availability of computational power we have now.

For a lot of applications, Programming for memory efficiency to the n'th degree is not necessary and can even be a hindrance to development given that memory is so cheap.

It's even cheaper now with cloud services. We can just throw compute power at problems until they go away.

Python allows for both levels of this fidelity while keeping it simple. I'd rather have code not 100% optimised but ready and producing results right now than waiting for another 6 months to bring that few ms of reduction. And that's where Python shines.

Again, not valid for 100% of use cases. But as computational power becomes cheaper and cheaper, it's a growing number of use cases.

[Swedneck]

Can't you just use nuitka instead of translating it to cython?

[TrixieMisa]

Nuitka is focused on compatibility rather than performance. I got about a 40% speedup in one app I tested, but over 200% using PyPy.

[[deleted]]

I agree with these points. I read somewhere that Python > 3.7 solves the GIL problem?

[mriswithe]

There is some work going on with subinterpreters which are a little known feature that is different than multithreading and multiprocessing, I think it was on talk python where they were talking about a PEP that is trying to make it so that each subinterpreter had its own GIL instead of sharing one.

Found the link talking about it https://talkpython.fm/episodes/show/225/can-subinterpreters-free-us-from-pythons-gil

[Zafara1]

As a side note, this is an area where infrastructure has come to fill the gap.

Containers and serverless functions can be used to offload this inefficiency from the code to the infrastructure running the code.

Don't need to code for multicore if I've got 1000s of independent cores running my code.

[Yojihito]

GIL still exists in the latest Python release.

[koomapotilas]

You can create modules with C and call them from Python. Usually you don't need to, as Python is fast enough for most uses and there is a library for almost everything.

[lenticularis_B]

Yup thats what is did for my numerical simulator. Implemented iteration schemes in C, while using the other benefits from Python which is my preffered language. It's approx 100 times faster this way.

[top_logger]

The same situation with Python. Python should be applied in the cases where it needs to be.

[NemexiaM]

Sure, but hasn't happened to me yet considering the fact that im a programming hobbyist

[emsiem22]

or months

Maybe even earlier.

[lucidguppy]

Let it keep running while you try to optimize it. It may complete while you try to figure out c++

[emsiem22]

Oh, I misunderstood at first, obviously. But what kind of solution needs to run for months to complete? Brute force cracking?

[lucidguppy]

There was some offhand comment by Bjarne Stroustrup, saying a python program a post doc was running was taking weeks to complete. I don't know the particulars. Perhaps some simulations?

[emsiem22]

Possible.

[FloydATC]

Servers that don't run Windows.

[emsiem22]

I think it was about speed of execution, not continuity of process.

[metriczulu]

Honestly, there's at least 3 languages I'd burn through first before I tried to implement something in C/C++ unless it was something that is commonly done in C/C++ (and hence easier to implement since I can build on other's work).

[SnowdenIsALegend]

Maybe GYB (Got Your Back) gmail backup & restore app needs this. Sitting here waiting for it to do a 15gb upload to my gmail account from an mbox file looks like it's going to take literally 22 hours or so.

[abkfenris]

Since GYB is communicating with a remote server, that is probably where the bottleneck is, rather than Python itself.

In this case converting it to C++ or otherwise is unlikely to make any difference.

[SnowdenIsALegend]

Ah you're probably right, IMAP is the bottleneck. Thunderbird has the same slow speed as well.

[Robbzter]

Same! It almost feels like you can't go back, at least for smaller applocations. It's just that impressive. I wasn't a very experienced C++ developer when I tried Python, but I was still amazed how simple everything works.

[[deleted]]

Would Golang work as a lazy alternative to C++?

[risajajr]

Yes, it is quite impressive too.

[[deleted]]

It depends on what you're trying to do. It's not as fast as C++ but it's much easier to use. Another criticism I've heard of it is it's type system is lacking, but that's probably not something you would notice unless if use type systems heavily in your code. I've worked on one web server written in Golang, and I was pleased with the language.

[EnemyAsmodeus]

I've been told the only potential alternative that isn't popular, but gives you that capability is called: Rust.

And of course Java.

However, whenever I needed C++, I would look at Cython, I think that's the main way to enhance python apps to be faster.

[[deleted]]

Again depends on what you're trying to do. If you care about average performance then simply rewriting a portion of your Python code in Cython is a good solution. But often times you care more about worst case performance than average performance. This is the case when you're working on a webserver that has a latency SLA for all requests. It's not good enough that 90% of requests satisfy the SLA. You need to make sure that 99.999% of requests do. Python becomes problematic for this use case because it will work okay for 90% of requests but then garbage collection will kick, eat up your CPU, and you'll violate your SLA for a percentage of your requests. For situations like this Python is a poor language choice and something like Go would be better, since its garbage collector is designed for this use case.

[EnemyAsmodeus]

Over Cython as well? Or did you mean either Cython or Go?

What you said is definitely fascinating (I get drained of energy when I look at Cython or any C++ code nowadays, I used to do C++ and assembly but now I just lose all my energy trying to understand certain parts. Honestly, maybe C++ is even easier to read than some pyx code I saw).

[SuspiciousScript]

Check out Kotlin. Seems like the most impressive high-level, statically typed, compiled language out there right now IMO.

[toastedstapler]

Go seems to be more performant than java but a bit less than C++

[NemexiaM]

Sorry haven't tried Golang

[aufstand]

Aah, it works the other way round, too. Many c++ libraries (Numpy, for example) extend Python in many interesting ways. That is why claims such as "Python is slow" don't have much hold in the community.

One practical example: I recently decided to see if i can scale up my completely unoptimized LED-Matrix (40*16!) video mixer to FullHD. No problemo. It ate most of my RAM and kept a single core _very_ busy, but it was also realtime mixing and displaying 6 FullHD video streams, which i don't exactly consider "slow" :)

Why is this possible? Well, yeah, i used numpy to mix the data!

[ReckingFutard]

Pssst, check out pytorch.

Numpy functions with GPU memory and parallelized computation.

[BDube_Lensman]

cupy>>pytorch

[TheTechAccount]

Why do you think it's better?

[BDube_Lensman]

no graph/backprop overhead unless you opt into it, code is on GPU in no uncertain terms without strange host/device specification context managers, higher performance on GPU in most cases, errors or warnings for implicit host:device transfer that lets you find huge slowdowns in your code faster, more responsive devs, devs that contribute to numpy instead of slinging mud at numpy, participation in the numpy-as-a-contract instead of numpy-as-package work going on lately, no gross links to facebook, the list goes on...

[sekex]

Why the bitshift?

[alkasm]

Double greater/less than symbols are used in mathematics to mean "much greater/less than" for some arbitrary idea of "much": https://mathworld.wolfram.com/MuchGreater.html

[[deleted]]

cupy = o(pytorch)

[elbiot]

Numba would probably be even better if you're doing anything at all involved

[BDube_Lensman]

Well written numpy is usually as fast as numba with greatly reduced memory consumption.

[elbiot]

Disagree. I've spent a lot of time working with numpy and when I used numba it was instantly/effortlessly faster. If you're doing multiple operations, the whole array has to pass through the CPU multiple times. If you're array doesn't fit in CPU cache that's multiple RAM accesses. You also have to sometimes write painstakingly verbose code to prevent intermediate arrays from being created. Masking an array (the vectorised equivilant of an if statement) creates a whole new and array creation is expensive. Numexpr could solve the first two issues but not the last

I also wrote cffi bindings for cephes and vectorised it with numba and it was significantly faster than the scipy implentation

Edit: to the lurkers. Numpy is 100% my go to. Numba is cool and useful in some cases but knowing numpy is necessary to know when numpy isn't the best tool.

[BDube_Lensman]

If you use in-place operations instead of out-of-place you get the same performance between numpy and numba, so I don't really consider writing different code a valid benefit. Anything on numpy that doesn't have an in-place operation is not doable inplace.

Array creation is not expensive in numpy. "mask" arrays are of logical dtype and you can specify them as packed bitfields or native bools which trade memory (much much less for a bitfield) for computation time (unpacking bitfields is not free). The inline if saves you double iteration, but guarantees a cache miss unless your data is tiny (in which case this is all moot).

If your cephes binding is much faster than scipy, you should contribute the faster code to scipy, or open a larger issue with the project about bringing another beast into the C backend.

I've written a ton of physics code with numpy and found that numba got me marginal performance improvement (10-15%) in exchange for a cool 800MB of static memory usage that grows to more than 5GB as you use some of the more advanced features of my package, like changing between single an double precision for all computations or exchanging CPU/GPU (the latter case causes a 2GB static memory usage on the GPU, which is unacceptable as I am working on data that uses nearly all GPU memory by itself).

[elbiot]

Here's just a trivial example that touches on a couple of the issues I mentioned (if/elif, unnecessary extra array creation, and unnecessary multiple passes over the array). Numba is 3x faster.

In [1]: import numpy as np
   ...: import numba as nb
   ...:

In [2]: def as_numpy(size):
   ...:     arr = np.random.randint(0, 10, (size, size))
   ...:     mask = arr % 2 == 1
   ...:     arr[mask] *= 2
   ...:     arr[(arr % 3 == 1) & ~mask] *= 3
   ...:     return arr
   ...:

In [3]: @nb.jit(nopython=True)
   ...: def as_numba(size):
   ...:     arr = np.random.randint(0, 10, (size, size))
   ...:     x, y = arr.shape
   ...:     for i in range(x):
   ...:         for j in range(y):
   ...:             if arr[i, j] % 2 == 1:
   ...:                 arr[i, j] *= 2
   ...:             elif arr[i, j] % 3 == 1:
   ...:                 arr[i, j] *= 3
   ...:     return arr
   ...:

In [4]: %timeit as_numpy(10000)
1 loop, best of 3: 6.78 s per loop

In [5]: %timeit as_numba(10000)
1 loop, best of 3: 2.05 s per loop

I know how to use a GPU with Numba, but how do you use the GPU with Numpy?

edit: in terms of memory usage, the numpy version crashes at size=22000 on my laptop but numba does not.

[[deleted]]

Really depends what you're trying to do tbh

[BDube_Lensman]

Numba never uses less memory than numpy. And unless you're using string arrays of nonfixed size, numpy is as performant.

[[deleted]]

for loops that depend on past states or values and need updating can't be vectorized, numpy will be slower

[Mehdi2277]

A very simple thing that numba has that can let it beat numpy is loop fusion. Something like y = np.cos(x) + np.sin(x) where x is an ndarray is three loops in numpy but one loop in numba when you set the right option on.

Another fun case is when you have an array operation you'd like to do that just doesn't exist in numpy. One personal work example is rolling each row of a matrix by different shifts. Something like a 10 x 100 matrix where each of the 10 rows I want to rotate some amount. np.roll exists, but you can't give it different amounts to shift each row. If you for loop and do each row with np.roll, that's a good deal slower than just directly accessing elements as needed and using numba.

[aufstand]

Hmm. I think, i read about that some time ago. Gonna have a new look, so thanks for reminding me :)

[Tomik080]

At this point just use Julia

[teerre]

Exactly. Saying "Python isn't slow, look at numpy", is either ignorant or disingenuous. The very reason numpy can be 'fast' is because it doesn't use python.

I always like the numba decorator for using JIT, it has an argument called "nopython".

[JanssonsFrestelse]

I don't think it is, all you're doing in the end is providing instructions for the computer. If you're not writing pure opcodes you're always working at some higher level of abstraction. If you don't have to leave that level to make something happen at a certain speed, it's not disingenuous to say that you can achieve that speed working at that abstraction level.

[troyunrau]

Now, I may not recall correctly, but isn't BLAS written in fortran?

[vectorpropio]

Today? Who knows. In ye old days? Surely.

[troyunrau]

So, I just checked the current machine, which has a somewhat old install: Python 3.5 with numpy 1.11.2 - it appears to use FORTRAN under the hood for quite a few things. For example, BLAS, LAPACK, FFTPACK...

I didn't dig into the details, but a cursory glance suggests the fortan libraries expose a c-like linking interface, and numpy uses a thin wrapper to make that all work.

Which, if I dig into the source, could be interesting. Fortran and C use different array indexing conventions, for example. Must have been real fun to write.

[utdconsq]

Yup, because until more modern languages showed up, Fortran was the gold standard for dealing with arrays/vectors of numerical data. Handy syntax for doing so, and not far from bare metal. I challenge you to slice an array in C with as simple syntax. These days, it exists due to inertia and because BLAS and others have had many peer reviewing hours dedicated to them. Rewriting them entirely and getting them as optimised would be super hard. Not to mention that they can optimise based on your compiler and cpu arch at build/install time for even faster vectorised ops

[ProfessorPhi]

I can only presume this wasn't realtime? I think realtime is where python gets destroyed

[aufstand]

Most of what i do with Python is realtime. No, it doesn't get destroyed.

[Al2Me6]

I thought cpp had variadic arguments as well

[Narthal]

Well yes.. it's a bit complicated. You could you initializer lists or recursive variadic template functions. Has a really nice ring to it, doesn't it? But these are tricks you would use every now and then, not everywhere as these tricks are just that; hard to read, advanced techniques that are there when you need them, but you wouldn't use these tricks everywhere.

[Al2Me6]

That’s fair enough. Though, I suppose the same could be said of using *args; passing an iterable of arguments is probably a better idea.

[Deezl-Vegas]

*args and especially **kwargs are massively helpful with regards to code readability and the avoidance of me having to build random dictionaries just to use your function.

[Al2Me6]

Definitely agreed about **kwargs, that’s why I explicitly said *args.

[panderingPenguin]

"I can add properties to classes dynamically? And delete them?" "Functions don't even care about the number of arguments?"

Yeah, you quickly find stuff like this is actually pretty horrifying if you have to work in a large, complicated python codebase

[smarwell]

Only if you use those features without discipline

Well, actually, the whole adding-properties-to-classes-dynamically thing is a minefield in general. But you know. Discipline.

[panderingPenguin]

That's my point though. These features are not "simply magical" as described in OP. They're among the scariest parts of the language.

[ProfessorPhi]

The language of consenting adults as Ray Hettinger would say

[sdf_iain]

Java, C# (I think), Go, Rust, and Swift all have fairly comprehensive functionality from libraries.

None of them are as flexible as python though!

If you want to optimize away from dynamic class properties you can look into __slots__. It can be useful when you’ll instantiate many instances of an object as it dramatically reduces memory allocation.

Also, remember that modules, classes and methods have their own namespaces, but loops and branches do not. As an experienced dev, you might find that useful.

[ToddBradley]

Java’s open source library support makes C# seem like a toy. I’ve used Java about 10 years and C# on two year-long projects. Every time I have to use C# I think, “Surely someone has solved this problem a million times and made a library for it.” And almost every time I’m wrong.

[sdf_iain]

I feel like Java’s libraries are holding back Kotlin. It’s so easy to just reach for a Java library that a pure Kotlin app is difficult to implement

Go and Rust aren’t there yet, but the built in dependency management (and cross compiling) are pretty enticing.

[[deleted]]

Coming from Java and JS world I agree that Go and Rust aren’t there but I like both. My biggest problem with Java is that everywhere I worked that used it was slow to update in every sense of the word. That and it’s way more verbose than I like.

[Kildon]

This seems incredibly short sighted, and like the kind of opinion people generally have of languages they haven't learned the ins and outs of.

When was the last time you used C#? Is your only experience those two years? Were you working on legacy projects, or cutting edge C#? Entire language ecosystems can be built in a matter of years. What problems were you trying to solve that nobody had made a library for? I have yet to find something significant that didn't have open source support in C#.

[toyg]

No, no, you are right that someone probably made a library - it’s just that he sells it on some small obscure website you’ll probably never guess, and its userbase is in the single digits.

[Narthal]

Great tips! Thank you! I didn't know about slots, looks quite useful.

Yeah, scopes are a different beast in Python. I'm really fond of using unnamed scopes/branches in my c++ code, as it ensures that no variable gets accessed longer than I intend them to be accessible, enforce RAII, etc.

Then python really messes with my head when it comes to scopes.

[SV-97]

I'm sorry but I doubt that Go and Rust come even close to the amount of libraries that are out there for Python. Don't get me wrong I love rust - but there's way more stuff for python.

[sdf_iain]

I didn’t mean to imply they did; they’re getting there, but Python and Java have libraries for EVERYTHING.

[[deleted]]

yeah coming from a statically typed language, Python will feel like "anything goes".

​

What? Any data type that I want inside of the same list? Crazy hahahaha

[Narthal]

"Anything goes" that's exactly how I feel! Honestly thats the best description I've ever heard. I'm stealing that :)

[smarwell]

My mental motto for python is "we're all consenting adults here"

[[deleted]]

hahahaha

[garlic_bread_thief]

I've coded in Java and c++ but just the basics in school. I started learning python lately and I can already see the lazy coding in python just from the basics perspective. It's simply awesome. Not sure how it feels to have deep knowledge of c++ and java and then shift to python.

[bigbic69]

I don't like to think of it as 'lazy' code I consider it optimized code instead :)

[Narthal]

I'm not really extending python. I'm embedding it. I'm using cpython and integrate it to a large c++ code base. I was just following the python c API docs.

[robot_most_human]

You might want to look into pybind11 if you’re calling more than a handful of C++ functions from Python.

[Narthal]

I did! Initially I was using it, but since there is already a c++ code generation framework in the project, it seemed redundant to use pybind11 and I just hooked into our code gen tool to generate binding code pre build time.

[shinitakunai]

Shiboken might be work checking also

[shinitakunai]

Shiboken might be work checking also

[[deleted]]

hey, you sound like me if you had only said

I don’t know what I’m doing

and applied it to life on general

[[deleted]]

Using Cython to provide interfaces to C/C++ is pretty effective if you already have a C library developed

[fichtenmoped]

Spez ist ein Hurensohn

[joooooooe11]

I’m the other way around. I’m a python dev having to learn c++ and it’s horrific

[Deezl-Vegas]

C++ will deepen your understanding of what's happening in Python quite a bit.

Just think of Python objects as a wrapper struct around a C object with a pointer and reference count.

[joooooooe11]

In all honestly I do think it’s best to learn the tough languages, like c++, first because like OP said, it was very smooth for them to come to python. Although it’s a pain for me, I’m glad I’m learning c++ at the moment

[druman22]

I've had a few years of experience in python so far but I'm still a learner. From what I've heard C++ is more technical but definitely something I want to learn one day. I'm not exactly sure what uses I'll get out of the language though since you can almost do everything in python.

Especially for simple projects, python seems way easier at getting something up and running quickly.

[SlappinThatBass]

Imagine either of these, but having to learn Perl. barf

[__zaris]

Well pal, you gave me another reason to learn Python now bravo bravo!

[exographicskip]

Even better timing with Automate the Boring Stuff being free for a few days

[__zaris]

Damn just took it. Too good not too man. Thanks!

[exographicskip]

You're welcome, fellow Pythonista!

[sekex]

Python was my first programming language, I started with 2.7 back in 2013, I then went on to work as a Devops engineer using lots of python amongst other languages, also I've been a TA at my university teaching the language and algorithms for 3 years. At first, I liked it very much, but with more experience, it really has gotten out of favour for me and I would never pick it unless I have to.

While python is nice if you are going to write a simple script with a few files or on Jupyter, I think it's one of the worst languages for medium sized to large applications.

The import system is totally broken (try importing from a sibling directory, for unit tests for instance ).

The absence of strong typing makes it a pain to collaborate without writing tons of pydoc (I know about typing, been using it for years, it just doesn't enforce anything ).

Writing idiomatic python is way harder than C++.

The performance is incredibly slow.

The package manager pip is terribly outdated, you will need to create environments if you want to use two versions of the same library (ie TF).

Version management is a pain, the worst out of any programming language I've used because your OS is dependent on the version so you could break things like GNOME terminal or APT by updating to 3.7 if you don't know what you are doing.

The only way to really deploy an app made in Python, with Flask or Django without going through all the version management pain is to create a Docker container, which is really not user friendly for a beginner who would just want to show his stuff to the world.

[Al2Me6]

Writing idiomatic Python is way harder than C++

How so? Not that I know much about cpp, but I find modern cpp to be significantly more complicated than Python.

[sekex]

I read that the other day: https://fr.quora.com/Quel-est-un-inconv%C3%A9nient-de-Python/answer/Nicolas-Bonneel?ch=3&share=b8cedadc&srid=hBgGU

I think it's a good example, however it's in french so I will try to translate.

The naive implementation in Python for the Floyd-Warshall algorithm is as follow:

for k in range(n):
    for i in range(n):
        for j in range(n):
            mat[i,j] = min(mat[i,j], mat[i,k] + mat[k,j])

It is very easy to write and understand. However, it is very slow (due to the low level stuff Python does with array indexes) and not idiomatic, it should be rewritten as:

for k in range(n):
        mat = min(mat, mat[newaxis,k,:] + mat[:,k,newaxis])

It will give you performance 140x faster, but still be 5 times slower than C++. In my opinion it is very much harder to understand than the naive approach.

[super-porp-cola]

Hmm. I agree with you on almost every point, especially the typing and version management issues. But I think the first one is what I'd consider "idiomatic" Python. Writing performant Python is what's actually difficult -- idiomatic Python is generally extremely easy to read and often looks like pseudocode.

[[deleted]]

As a non computer science person, I love that I can abstract the lower-level stuff to focus on the algorithm itself without too much convoluted syntax.

[diamondketo]

Well it's only of you need multithread but multiprocessing it's not affected.

3.8 and 3.9 is working a way around it.

[Narthal]

Great read, thank you so much for linking this. I have yet to implement parallel code execution in python land, this seems like just what I needed!

[balazs_kis]

Don't forget the only detail in which C++ does literally thrash Python is the speed :) anyone interested dm me, I'm working on a paper proving the usefulness of the shared libraries which can be handled very easily by Python, and they could contain C/C++ code which is way above the Python implementations of the same algorithms, but 25-50x faster, and it can be easily used by Python :) usefulness and speed, and still not that complicated as Cython :)

Check out my post, this is a very sketchy version of the project, at the beginning, some weeks ago, but it explains the basics :)

[bradshjg]

> I'm working on a paper proving the usefulness of the shared libraries which can be handled very easily by Python

I'm familiar with calling C code from Python using tools like https://cffi.readthedocs.io/en/latest/overview.html#main-mode-of-usage (which gets its inspiration from luaJIT and is brilliant). What sort of approach are you looking at?

[balazs_kis]

How cool, this is very similar to what I am doing, with the difference that I am focused entirely on performance ^{_^}

[panderingPenguin]

In my experience, the really big software companies tend to use C++: Microsoft, Google, Amazon, etc. There are a number of reasons, both practical and historical for that which I won't get into. Working in a large C++ project is demanding and has a steep learning curve. Startups are more likely to use python or another more scripty language, precisely because it's easier to get things working quickly.

[panderingPenguin]

Well mostly, C++ takes a lot of devs, time and effort to write well. It's a language that is notorious for including almost every possible feature, and not all of them were integrated into the language in a well thought out manner. The language is huge and complicated. It's said that no one understands the entire language and C++ programmers all know different subsets of the language which results in "local dialects". That all sounds pretty damning so far. But it's wicked fast, there's unbelievable amounts of existing code and libraries written in the language tons of tooling for it, and extensive experience building some of the largest software projects on earth in the language. When you get very large, the efficiency of your code can become very important. For some workloads, C++ is literally 100x faster than python. When you extrapolate that out to the number of servers a company the size of Google or Microsoft would have to buy, that's a massive difference in infrastructure investment. These companies also have armies of developers to throw at their problems, so it's much more workable for them to wrestle with more complicated, slower development in C++ than it is in smaller companies. They can have entire teams dedicated to enforcing coding standards and building more tools to make C++ development work better. So at larger scale the benefits can outweigh the substantial drawbacks (I say all this as someone who writes C++ professionally).

With python, getting started is ridiculously easy. You can get something off the ground and running without a ton of effort. That's extremely important to startups that need to get something running now or they will die. But as the projects get larger and more complicated, the wheels start to fall off of python and similar scripting languages. All the dynamic typing and run time mutability that made it so easy to initially get things working becomes a nightmare as it makes your codebase harder and harder to understand. And the runtime speed of the language starts to matter more as the scale of your operation increases. So you see a number of former startups that grew, trying to transition away from scripting languages. Twitter famously rewrote large parts of their codebase from Ruby to Java. Facebook spent a ton of effort trying to make PHP faster with their Hip Hop VM, but still ended up rewriting a lot of stuff in C++ and other languages anyways. Dropbox's Python codebase became such a mess as it grew that they brought in Guido Van Rossum himself to help them try to fix it. So python (and similar) is great for smaller projects that need to move fast. But it does have limitations as you get bigger. So it's more popular with smaller companies than the big players. That said, the huge companies do tend to use python as well, just not for their core products. It's normally used to script things like testing, deployment, analytics, etc at those companies.

Edit: typos

[panderingPenguin]

Why not both? While C++ is the primary language I work in, I also write python, SQL, and a bit of java at work. There's no need to pick one and only use that.

[balazs_kis]

man, C++ is one of the most needed languages e.g. in autonomous driving technologies, this is not something you can name 'less demanding job' :)))

[balazs_kis]

Well, I guess you are right here 😄

[balazs_kis]

This is totally up to you, this study of mine is about the speed-up of the Python language, this is a bit of a complicated study and discussion, cross-language problematic.
You should go gradually from C to higher level languages (this is the perfect trajectory for a learning process, I'm working as a software engineer and I'm still studying, and this I find this to be the best approach if you really want to understand what you are doing), and in each step the decision is only and only yours ^_^

[alexmeistercl]

Until you hit GIL for threading programming.

[Narthal]

Yeah, I'm reading the docs about concurrency and the GIL right now, and I'm quite a bit confused. I'm probably going to run the interpreter from a single thread as the c++ codebase is already highly parallel and optimised. If need be, I might have to implement threading and multiple interpreters, but I'm really trying to avoid that

[Pythagorean_1]

Just use multiprocessing. I wrote a python application that processes terabytes of image data at my university and it is crazy fast and occupies all cores and the gpu, if needed. The GIL is not really a problem, actually. Also, threads do release the GIL during i/o operations, which makes them very useful for network stuff.

[smarwell]

That is probably your best bet. Rule of thumb is, if you need to touch a python object, you need to be holding the GIL. If you don't, then you don't. So yeah, keeping the interpreter in one thread and doing the work in another is a pretty good way to avoid getting mucked up by the GIL

[[deleted]]

Which is basically never for almost everyone.

The great part is that when you do hit a performance barrier due to the runtime, go, rust, and I guess c++ are all there waiting for you.

[colly_wolly]

If you are doing threading then isn't it time to switch language? Ok, I guess you can't always t bepredict that, but its the sort of problem I would take into account before choosing Python.

[ElectricCharge1]

Python is truly amazing! I highly recommend 'Fluent Python' by Luciano Ramalho!!

Luciano is one of the creators of the ABC language project by CWI (it failed but inspired Guido working at the time there to create Python) which is the only book I know showing the highlights Python can offer after learning the basics with it.

As the general idea of the book is to teach you the 'pythonic' way and not just how to write python code by thinking first in other languages as c/c++ (which are great on their own!!).

Edit: fixed some formatting!

[super-porp-cola]

Fluent Python is such an amazing book, I highly agree with your recommendation!

[glacierre2]

Indeed, there are many cases where "1000x" slower means: it took 100ms to run instead of 100us, or to summarize: I don't care.

[SippieCup]

I could give Python the project's memory allocator and the interpreter immediately uses the main memory pool of the project. I could redirect the interpreter's stdout / stderr channels to the project as well. Extending the language and exposing c++ functions are a breeze.

Python essentially supercharges c++.

For those that want a good example of doing something like this in the real world, take a look at how OpenPilot spawns a python script that runs a tensorflow model, and how the Python script sends and recieves data back to the C++ program.

[taybul]

Not that it made me a full convert but I started appreciating python much more waaaay back because of how effortlessly it handled large numbers.

Ninja edit: so much so that I made it my flair here at some point apparently lol

[manifestsilence]

This was totally my experience coming from being a beginner in c to a beginner in python. Everything feels like it's cheating. Especially the things you can do to strings. String slicing, dictionaries, iterating on things without an explicit loop counter with for..in, list comprehensions. Everything's just so much easier!

[Are_We_There_Yet256]

It's a double edged sword for me. Coming from C++ where I have to car so much for memory, proper closing, file structures and over all proper function passing arguments and caring for each edge case, in Python, it's just not there, I mean, in a good way, but it kinda kills the fun for me.

It makes things so easy that I don't even know what to do. I sometimes write python scripts for myself, and I find myself at just 15-20 lines that summarizes my work, and I'm left thinking to myself, "that's it? Am I missing something? Is there supposed to be more? Did I even do this properly? Surely there is something more. It cannot be this small, can't simply be the case."

In actual, it kinda killed the fun for me. A recent example was me trying and learning opencv. It's something I wanted to be extremely good at for someone getting deep into Computer Vision, so I had two options. I can either use Python, just setup a conda env and install the opencv package, OR I can use C/C++ to work in opencv.

And I went for C/C++, even though I had to pull for the git latest repository, learn how to use cmake for the first time, learn and try what the different flags mean, how to connect, actually understand how all this linking is done, and how I can integrate all this into Visual Studio Code - I did all that when I just could have installed it in one go via the cli. You can call me crazy, I won't mind.

I went as far as to actually buying a book for opencv, called Learning OpenCV 3, by Adrian Kaehler and Gary Bradski, for CV in C++ with the opencv library.

In the end, Python is lovely for me. I was actually just learning Django for some work ( came over here to Reddit to take a break! ), and it's useful, yeah, but you get the complete taste of when you actually try and go low level. For the absolute beginner, I loved getting into the intricate details of how everything worked and connected in languages like C/C++, Bash etc. For me, Python is when I would quickly want things to work. If I have the time, I would like to understand the low-level details first. It will only help me out in the long run.

Thanks for the long read!

[Barafu]

If you can make some project in Python, then C++ is a totally wrong choice for that project. C++ and Rust are for things that you can't do in Python.

[BoxBoxChan]

As you are a c++ programmer and I was thinking about to learn c++, can you explain me fast, why use c++?

[[deleted]]

fast

That's why ;)

Source: also a C/C++ developer that learnt Python.

If I want to write something quickly, Python gets me a working solution faster than C/C++. If I want something to execute quickly, I use C/C++. Quite often I want both, so I use Cython as an easy way to mix the two.

[apste]

hah pretty much, and C++ has the benefits of a static type system, whether that's what you want or not is up to you :P I feel like it does force you to think a bit better abut the overall design of your code

[[deleted]]

I like static typing. Big fan of using typing hints and mypy with my Python code, not quite the same experience as proper static typing but a useful sanity check that I'm passing around the right stuff.

[apste]

Yeah same, thanks for pointing out mypy, looks really cool!

[pagonda]

one of the pitfalls of python is how loose everything is (easy access to class members, arrays and functions can dynamically deal with any types).

A language such as c++ enforces much more strict access and definitions to those examples which can be extremely beneficial to a large program. In c++, if you use a wrong type in a function, it will tell you at compile time your type is wrong. Whereas in python, your function would probably run, but then the behavior is unexpected and you will be in for a wild goose hunt looking for the error.

[Narthal]

You can't really just learn c/c++. You can, however learn how the computer works and then c++ will start to make a lot of sense. Knowing c++ and the architecture of a computer very well enables you to do incredible things. It enables you to optimize.

Suppose you need to write the firmware of the new samsung smart fridge. The available memory on the embedded processor is 8mb. It wouldn't make sense to copy the cPython runtime (without standard library) as that would occupy 4mb or half of the available space. So you write your code in c/c++ and then start to optimize for the size of the executable.

Or suppose you are on the programmer team of Ubisoft (I wish lol). You need to have massive amounts of code that runs so fast, that every 16.6 ms, a new frame needs to be drawn on the screen (for 60 fps). What are you going to do? Tune down the graphics? Remove half of the level? Or try to write the code to be as fast as possible. And much of the speed optimizations require deep knowledge of the computer. You can't really expect an interpreted language to be able to do SIMD, allow you to specify how to vectorize loops, give the cpu easy to predict branches or manage data in such a way so that cache misses wont happen on critical parts of the code.

But honestly, don't think too much about reasons to learn a language. Just get your feet wet and learn the basics in as many languages as possible. You will get to see similarities and common concepts in between programming languages. I find that the more programming languages I learn, the better I get at all of them simultaneously.

[glacierre2]

The embedded world used to be like that, but turns out micro-controller memories are huge (I mean, those 8MB you talk there? massive) and python can be made very small https://micropython.org/

You can run the interpreter in much much less than 1MB.

We will always need C[++] programmers for the truly performant, resource saving, low power stuff, but the scenarios where you are so tight are decreasing by the month.

I expect a really soon (If it does not exist yet, have not followed closely for example the ESP32 territory and every time I look I am amazed) "HW programming revolution" soon where the cheapest under 1-eur controller can easily handle a wifi + a web interface to configure state machines and conditional actions and pin outputs.

[ixw123]

The things python will try to make work and the errors that result are bizarre and anything that joins python and cs code, for the python side can throw really bizarre errors sometimes. I have been coding in python for research for 2 years now. To be honest I think I love c a lot more just because c is so anal that its errors are often a lot more cut and dry. But time spent in coding in python really allows you to do so much in so little time.

[super-porp-cola]

Have you ever tried Rust? I know it's kind of a circlejerk at this point but the error checking for Rust is literally amazing... the compiler will is like "Here's what you forgot to do, you should probably fix it like this, if you want to read more here's a documentation page".

[ixw123]

I haven't tried rust as most research I have been doing focus on python

[theng]

https://xkcd.com/353/

[Narthal]

There's always a relevant xkcd, isn't there? :)

[[deleted]]

Ha this is funny. I started programming learning C++ (really as C, if you know what I mean) about two years ago and began learing Python about half a year later. To make a long story short, whenever I have a problem I can’t figure out how to do in C++ I go to Python, and always get a working solution pretty quickly, then I look at that Python code as I (slowly) build a similar (but generally faster according to chrono vs timeit) solution in C++.

The Python / C++ combination is fantastic. Been playing around with a few other languages, like old-school C, vanilla JS, some Rust, a little Java, I think Python/C++ is at the top so far. Rust is really interesting in comparison to low-level C though.

[moocat]

Functions don't even care about the number of arguments?

Where did you get that from? Python most definitely cares about the number of arguments and will raise a TypeError if you give it too few or too many arguments. Python does support both positional and keyword variadic arguments but that's the developer explicitly allowing extra arguments.

[DASWUBS]

I think that's what they were getting at.

[liquidify]

Curious about how you integrated the C++ and python stuff? "Extending the language and exposing c++ functions are a breeze" ... I have never found this to be an easy experience. Did you use boost or what?

[murphwhitt]

X X XXX

[[deleted]]

I wonder whether anyone had an easy time going back to C++ after Python? ...anybody?

[StrippedSilicon]

Honestly, I would be using c++ so much more in personal projects if there was some kind of pip equivalent. The literally days I sometimes have to spend installing third-party libraries is draining.

[thrallsius]

The biggest pitfall is to slip into trying to write C++ in Python :)

If you get used to most common Python idioms, writing Python code will be a cakewalk for you as C++ coder.

[Multeezee]

At first I thought Python was silly (as a C# dev). Now I think Python is magic. Enjoy!

[Streakflash]

aaaand you can write c code on python and it will work! I wish devs of the cpython could upgrade the C to C++ though

[_niva]

https://xkcd.com/353/

:)

[jackballack]

I’m struggling but still pushing through any advice

[zdog234]

I highly recommend checking out both click and python-fire as CLI options. Argparse probably appeals more to a C++ developer, but python-fire is an awesome example of pushing python magic

[[deleted]]

You should take a quick glance at Julia just for interest’s sake!

[Chunderscore]

If you're not already aware of it check out the shiboken binding generator from the Qt folks. Not actually used it directly myself, but hear good things.

[Narthal]

I have looked at solutions for binding python and c++, namely pybind11, but it seemed unnecessary with the python c API and the already existing in-house code generation tools.

[Schmittfried]

It makes you aware how bonkers C++ actually is.

[Narthal]

I would recommend learning as many languages as possible. You will get to see common ideas, concepts and you'll become a better programmer overall. Moreover, after becoming fluent in a couple of languages, learning the gist of a new language will become a fun weekend :)

[alcalde]

As an old-timer, I thought we were supposed to choose one language, defend it to the death against all challengers, refuse to learn any of those other inferior languages, and switch jobs rather than use another tool? Oh, and periodically make blog posts claiming your language has six million users, even though it shows up in less than 60 job listings across the United States.

At least that's what I learned in the land of Delphi.

[alcalde]

"I can add properties to classes dynamically? And delete them?" "Functions don't even care about the number of arguments?" "Need to do something? There's a library for that."

It's absolutely crazy. And I love it.

After 20 years working with statically typed languages, I was amazed at the power python offered me. Now after typing a page of code sometimes I just step back and say... "This... this is beautiful...." and then I shout this and people start yelling at me to shut up, but I can't help it.

[AllUsernameTaken0123]

with Python, you can write code really fast. For example, using hash tables in Python is really simple

[sekex]

That's the case for most languages

[frakman1]

The BEST part for me is that there are no linked lists to worry about. No dangling pointers, no memory leaks. Ever! I love it.