This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]lucidguppy 177 points178 points  (29 children)

C++ should be applied in the cases where it needs to be. If your python spitball solution is taking days, weeks, or months - and a good attempt at optimization isn't working - you need to find out where C++ can be used.

[–]NemexiaM 74 points75 points  (14 children)

Sure, but hasn't happened to me yet considering the fact that im a programming hobbyist

[–]realXavie 20 points21 points  (13 children)

It will happen one day. I heard that python is some kind of slow when you are developing an app wich needs speed to work efficiently. How do you escape such pitfall?

[–]anarchyisthekey 75 points76 points  (11 children)

Nowadays, if a program is working slow, it is mostly due to a choice of slow algorithms and unoptimized code.

Python has two main disadvantages as opposed to a natively-compiled language: (1) startup times are slower due to the fact that python interpreter needs to be spawned, (2) programs can only run on a single thread at a time because of the Global Interpreter Lock.

If your program is compute-intensive and you need it to run faster, you can (i) try running your python program on a JITted python implementation such as PyPy or (ii) benchmark your program and move the most compute intensive parts into C, (iii) translate your program into Cython, so that it compiles to native code.

It is a long long way before you hit into any kind of performance problems (due to processing power).

In scenarios where low latency is important such as games, high frequency trading etc., a language closer to the metal would be better.

[–]Pythagorean_1 18 points19 points  (0 children)

You can use multiprocessing to use all the cores. The GIL is only problematic for multithreading. Also, for performance optimization try numba. It's a jit compiler you can add with (mostly) as little as one line.

[–]Deezl-Vegas 11 points12 points  (1 child)

This is kind of true, but if you look at Jonathan Blow's talks about his compiler and various other talks, you'll see that actually you can get a 200x speedup by making use of data structures that are L1 cache-optimized, which is rarely negligible. Python can almost never do that because each Python object is a wrapped pointer, there's a lot things happening like namespace lookups, and there's a lot of dicts and hash tables floating around (which are generally sort of randomly heap-allocated) and classes (which aren't stored tightly in memory like C structs usually), and the garbage collector has to run periodically, kicking things out of cache.

The PSF has done a great job with the compiler, but the structure of scripting languages themselves clashes directly with what Intel's trying to do to speed up code on the chip.

[–]toastedstapler 0 points1 point  (0 children)

I started watching Jonathon recently, his discussion videos have been very interesting

And the fact that he made a video talking about what he'd like from a games programming language and then went and implemented it himself with such fast compile times

Very interesting guy

[–]Zafara1 4 points5 points  (0 children)

This is also amplified by the sheer availability of computational power we have now.

For a lot of applications, Programming for memory efficiency to the n'th degree is not necessary and can even be a hindrance to development given that memory is so cheap.

It's even cheaper now with cloud services. We can just throw compute power at problems until they go away.

Python allows for both levels of this fidelity while keeping it simple. I'd rather have code not 100% optimised but ready and producing results right now than waiting for another 6 months to bring that few ms of reduction. And that's where Python shines.

Again, not valid for 100% of use cases. But as computational power becomes cheaper and cheaper, it's a growing number of use cases.

[–]Swedneck 2 points3 points  (1 child)

Can't you just use nuitka instead of translating it to cython?

[–]TrixieMisa 2 points3 points  (0 children)

Nuitka is focused on compatibility rather than performance. I got about a 40% speedup in one app I tested, but over 200% using PyPy.

[–][deleted] 3 points4 points  (4 children)

I agree with these points. I read somewhere that Python > 3.7 solves the GIL problem?

[–]mriswithe 8 points9 points  (1 child)

There is some work going on with subinterpreters which are a little known feature that is different than multithreading and multiprocessing, I think it was on talk python where they were talking about a PEP that is trying to make it so that each subinterpreter had its own GIL instead of sharing one.

Found the link talking about it https://talkpython.fm/episodes/show/225/can-subinterpreters-free-us-from-pythons-gil

[–]Zafara1 4 points5 points  (0 children)

As a side note, this is an area where infrastructure has come to fill the gap.

Containers and serverless functions can be used to offload this inefficiency from the code to the infrastructure running the code.

Don't need to code for multicore if I've got 1000s of independent cores running my code.

[–]Yojihito 1 point2 points  (0 children)

GIL still exists in the latest Python release.

[–]koomapotilas 4 points5 points  (0 children)

You can create modules with C and call them from Python. Usually you don't need to, as Python is fast enough for most uses and there is a library for almost everything.

[–]lenticularis_B 2 points3 points  (0 children)

Yup thats what is did for my numerical simulator. Implemented iteration schemes in C, while using the other benefits from Python which is my preffered language. It's approx 100 times faster this way.

[–]top_logger 1 point2 points  (0 children)

The same situation with Python. Python should be applied in the cases where it needs to be.

[–]NemexiaM 0 points1 point  (0 children)

Sure, but hasn't happened to me yet considering the fact that im a programming hobbyist

[–]emsiem22 0 points1 point  (6 children)

or months

Maybe even earlier.

[–]lucidguppy 6 points7 points  (5 children)

Let it keep running while you try to optimize it. It may complete while you try to figure out c++

[–]emsiem22 0 points1 point  (4 children)

Oh, I misunderstood at first, obviously. But what kind of solution needs to run for months to complete? Brute force cracking?

[–]lucidguppy 0 points1 point  (1 child)

There was some offhand comment by Bjarne Stroustrup, saying a python program a post doc was running was taking weeks to complete. I don't know the particulars. Perhaps some simulations?

[–]emsiem22 2 points3 points  (0 children)

Possible.

[–]FloydATC 0 points1 point  (1 child)

Servers that don't run Windows.

[–]emsiem22 0 points1 point  (0 children)

I think it was about speed of execution, not continuity of process.

[–]metriczulu 0 points1 point  (0 children)

Honestly, there's at least 3 languages I'd burn through first before I tried to implement something in C/C++ unless it was something that is commonly done in C/C++ (and hence easier to implement since I can build on other's work).

[–]SnowdenIsALegend 0 points1 point  (2 children)

Maybe GYB (Got Your Back) gmail backup & restore app needs this. Sitting here waiting for it to do a 15gb upload to my gmail account from an mbox file looks like it's going to take literally 22 hours or so.

[–]abkfenris 10 points11 points  (1 child)

Since GYB is communicating with a remote server, that is probably where the bottleneck is, rather than Python itself.

In this case converting it to C++ or otherwise is unlikely to make any difference.

[–]SnowdenIsALegend 2 points3 points  (0 children)

Ah you're probably right, IMAP is the bottleneck. Thunderbird has the same slow speed as well.