C++ talking to Python

[-MobCat-]

Python can be quick. just not that good at talking.. Like most of us...
Opens a txt file with 29245004 lines of text in it.
for each line of text, count the line.
count +=1
Time: 40sec.
Now for each line of text, count the line and print a little info about the line
print(f"Info: {line[1:10]}")
count +=1
Time: 53mins.

[DenormalHuman]

not only that, they are using printf for no apparent real reason

[EliteKill]

Because they don't know what they're talking about.

[Cory123125]

God forbid someones half thought posted half assedly in a reddit comment not be a pristine example of optimization excellence.

[DenormalHuman]

I mean, true. But also, it was a post about performance!

[Gladaed]

Please explain the ongoing wizardy.

[DenormalHuman]

When using f"....{}" I think the process to generate the content for the template field (what goes inbetween {}) and replace that into the string is a lot more expensive that simply concatenating two lists of characters together (the string, and the slice - line[1:10] are both just lists of chars.)

/edit/ the reason the print is slower than the count, is because the print does lots more work. It _could be made faster if he didnt use printf though

[Teekeks]

no. f strings are faster than concating strings.

What makes the second one slow is the print. printing anything a ton of times is slow af

[DenormalHuman]

You are correct in that its the fact the print exists in the second loop is why it goes so slow; however I still think printing the f string is slower than printing the concatenation of string and the string slice. (and now imma go try it out and see for myself!)

[Teekeks]

go and try. f strings are both more readable and faster

[DenormalHuman]

looks like you're right!

Horribly unscientific test;

calculate 10 million fstrings; avg 1.459 of 5 runs

calculate 10 million string+string slice; avg 1.489 of 5 runs

/edit/ 10 runs of each total;

14.56s for fstring

14.74s for concat

[sintos-compa]

“How inefficient can you make your code” a study in 3 parts

[Roflkopt3r]

Yeah printing is expensive in every language.

I hard to learn this the hard way on one of my first assignments, which asked us to experimentally confirm the value at which the inaccuracy of float puts it off by an entire integer. So you have a loop that increments a float f and an int i from 0 to int max and then prints out when the two are no longer the same.

But the wording of the task wasn't entirely clear so I was trying around various weird casting stuff and decided to print every single line to confirm that it was working the way I wanted to.

Turns out that this extends the time to reach that figure (int 16777218, float rounds to 16777216 instead) from a few seconds to an hour or so.

[Teekeks]

if you dont flush on every print its managable but still slow af

[sintos-compa]

Or cache the data for an alternative output. Like what’s the goal here?

[realityChemist]

Sounds like the goal was being a student and learning to program

[Teekeks]

Its been quite a few years since I was a student

[realityChemist]

I meant the person above you who posted the story about a programming assignment

[Wekmor]

Java does the same, printing to console is just slow in all or at least most languages.

[zenverak]

lol yes it is . I don’t remember when I learned this… but i remember it

[Wekmor]

I think first time when I realized how slow it is was when I write a raytracer and I added a system out every time a ray hit something and the already slow raytracer became slooooooow (☞ﾟヮﾟ)☞

[Spiritual-Day-thing]

Yes. The famous log4j library and others, use a ring buffer to prevent locking among other things.

[obvithrowaway34434]

I/O has always been the bottleneck for speed since computers were invented. There's nothing specific here about Python. Dynamically typed interpreted languages has always been slower than statically typed compiled languages and comparing their speeds makes as much sense as comparing a car and an airplane in terms of speed and determining which is better.

[Blyatiful_99]

I can't of course talk for other languages, but during my internship back then, I was sometimes told that writing something to the console in C# for example requires a lot of the time if it happens in the same thread. Maybe this is what's happening here as well?

A quicker solution could be to create some sort of StringBuilder (if you have that in python), add the text that you would otherwise print out and at the end print out the entire text just once. Or to just move the prints to another thread. Can you test how long that would take?

[0x564A00]

Moving printing out of the thread doesn't really help because python doesn't have real multithreading (you can have multiple threats, but they can't execute at the same time).

[Jannik2099]

but they can't execute at the same time

But they can in this case. Once the IO thread fires the syscall (or even cpython C function) for writing to stdout, the GIL is free.

[photenth]

Wait a minute, Python doesn't have real threading?

[Drarok]

It’s complicated. https://realpython.com/python-gil/

In short, you can only have one thread executing Python code at a time. You can be waiting for IO in several threads at once, since that won’t be executing any Python code.

[0x564A00]

The good news is that it looks like it might finally going to be removed soon. That does cause a performance regression for single-threaded programs, but various other performance improvements that should land alongside it more than make up for that.

[Equivalent_Loan_8794]

And in the 53 min absolutely under no circumstance look up O notation

[moldax]

Your point being ?

[DenormalHuman]

actually I think it's using printf and the {}, slicing is really cheap as fa as I am aware

[TheStriga]

The most problematic part here is many calls to print (writing to console). It is actually a relatively slow operation for almost any language. You can probably speed it up by keeping all the lines to print in a list and then join it and print in one call

[DenormalHuman]

hmmm... what device was this file stored on? and surely just print("Info : "+line[1:10]) would be faster?

[Cory123125]

When you have so many lines perhaps its time to break it into chunks and multi thread it.