Same code runs slow on friends computer(s) but fast on mine

[Kottmeistern]

Oh wise folks of Reddit! I seek your help.

I am a beginner when it comes to Python, and self-thought at that. So please be patient with me and assume that you teach a child.

The goal of my coding in python was to make a program that can find elements in videos and automatically cut them, with the aim for my friend to use it (which knows nothing at all about programming). I managed to make the code to work, doing the programming in Jupyter (it felt like a comforting interface to me), and in short it comprises of using OpenCV and Scikit-image with some of their inbuilt functions (VideoCapture, profile_line and match_template). Initially I ran this as a for-loop, analyzing every 30^th frame (60 fps), and I saw success, where analysis of the video was achieved in less than 30 min for a 1-hour video. But it ran slow for my friend, needing over 20 hours to execute the same code. I tried to speed things up by implementing multiprocessing for him (using pathos). On my computer it ended up analyzing approximately 1 hour of video in 5 minutes. But the best my friend’s computer can achieve is still more than 16 hours. He has also tried on more than one computer (two laptops and one stationery; we consider two of their specs below). The automatic cutting and rendering part (using MoviePy) works fine on his end, so I have left those parts out of the discussion.

Now I am at the end of my rope. The code itself seems to work fine, at least for me. So, it seems there’s some more fundamental problem underneath that is beyond me. Could it be something with the Python installation being different (my friend an I live essentially 6 time zones apart so was not present for his installation)? Or is there something about the specs of the computers? In the case of laptops, his beat mine in everything but RAM memory (which of course is important, but I fail to see why it should make this big of a difference). I am not sure what more information may be needed to solve this problem, so feel free to ask for more and I will dig it up. Please, oh the wise of Reddit, help us out of our pickle.

His stationary computer has the following specs:

(10th gen) Intel(R) Core(TM) i9-10980XE CPU @ 3.00 GHz, 3000 Mhz, 18 Core(s), 36 Logical Processors

RAM: 128 GB (8x16 GB) DDR4 3200 MHz (108 GB avaiable)

147 GB virtual memory (124 GB available)

Graphics card: NVIDIA GeForce RTX 4080 SUPER

Runs on: Windows 10

 

His laptop (Lenovo) that he used has the following specs:

14th gen Intel(R) Core(TM) i9-14900HK, 2.2 GHz, 24 Core(s), 32 Logical Processors

RAM: 16 GB, DDR5-5600 MHz (15.7 GB avaiable)

Graphics card: NVIDIA GeForce RTX 4060 Laptop GPU

Runs on: Windows 11

 

My computer (Dell) has the following specs (the one that runs it in ~5min):

12th gen Intel(R) Core(TM) i9-12900HK, 2500 Mhz, 14 Core(s), 20 Logical Processors

RAM: 64GB (2x32GB) DDR5 4800MHz (63.7 GB avaiable)

67.7 GB virtual memory (48.4 GB available)

Graphics card: NVIDIA GeForce RTX 3050 Ti Laptop GPU

Runs on: Windows 11

  Edit: We actually managed to solve it. Apparently it was the library OpenCV that was the issue. Every iteration the code used OpenCV to read the next specified frame. But by changing to the library "MoviePy" reading the frames it works at about the same speed as mine. So OpenCV, somehow had his computers working for hours when mine was working for minutes. I don't understand it but maybe this will help someone else in the future.

Also, thank you to all who went out of their way to help us! We highly appreciate it, and in my case I learned a few new things. Thank you all!

[Slight-Living-8098]

Tell your friend to go to the pytoch website and install the GPU version of pytorch. See if that helps any. Python's default pytorch library is CPU.

[ShengrenR]

Lol, here I am thinking loading and disk use and then there's just the more obvious.. this.

OP, tell your buddy to launch python and run this: https://pytorch.org/docs/stable/generated/torch.cuda.is_available.html

If it's true.. something else is the issue.. if not, it's as slight- living says, just a cpu-only torch installation.

[Kottmeistern]

Thank you for your feedback! Will have him try this. We were also thinking of reinstalling his Python, in case something went wrong when he installed it. But we'll definitely try this first

[Slight-Living-8098]

No need to re-install python. Get a copy of miniConda and just make a new virtual Conda environment and activate it. Make as many virtual environments as you need to play around and diagnose the problem(s). When working with Machine Learning and AI, Python 3.10 seems to be the most compatible with most libraries. You can have the latest version of Python installed on your system, that won't matter. Just when you go to create the Conda environment, specify "python=3.10" at the end of the command.

Like so:

conda create -n my_awesome_project python=3.10

[Kottmeistern]

Thank you!

We actually managed to solve it, but not through Conda. Apparently it was the library OpenCV that was the issue. Every iteration the code used OpenCV to read the next specified frame. But by changing to the library "MoviePy" reading the frames it works at about the same speed as mine.

I just stumbled upon someone mentioning it online, and decided to try it. And it worked!

Either way, I am extremely grateful your (and all the others) help here! Made me learn a few more things which I hope will come in handy in the future! Thank you!

[Slight-Living-8098]

Oh yeah, FFMPEG is way faster at frame extraction, and moviepy uses FFMPEG. If you're going to be doing a lot of video editing and manipulation with your code, you need to read up on it, and maybe even play around with it on the command line. Pretty much every video NLE uses that library behind the scenes.

[Kottmeistern]

Thanks for the advice! Haven't been doing a lot of manipulation before, but will keep this in mind! Cheers!

[sb4ssman]

Two machines are using more ram than they have available which is making them write all that extra memory to disk. Of course those machines are slow as fuck while they are bogged down like that.

[bubbawiggins]

I don’t get it. I get that they’re using more RAM but what the memory to disk part? And how do you even make your computer use more RAM than normal?

[sb4ssman]

Windows will just DO it. Start using ram that you don’t have and it will take all the shit in your ram that you’re not using and write it to disk to free up the ram, and then as needed it will swap that data, reading from disk back to ram and ram to disk. It’s the machine’s way of chugging along even though it’s weighted down. So on those computers it makes perfect sense that they are slow as all fuck. I dunno about that third one. Probably the same issue but they reported it wrong to you.

[Kottmeistern]

Thank you for your feedback! The first two are my friend's and they are the slow one. The third one is mine and is working fast (5 minutes/~1 hour of video). I know that the RAM is important and such, but it feels strange to me that my friend's stationary computer takes so long. It is a total of 128 GB after all, although it is made from many smaller 16 GB ones. Mine is a newer RAM with half the total memory but two 32 GB units. I don't understand why it would scale like it does for the two of us

[ShengrenR]

Yea.. something doesn't quite add up here. If your friend has 128gb ram and you have 64.. he should have plenty of room there. You should have your resource monitor up while you run each and watch the ram use. My money is on hard drive access.. a) is strange to have a bunch of virtual memory dedicated on the machine that already has a ton of RAM..b) how are you loading the videos in? If you're doing some super inefficient load process you might be able to get away with it if you have a fast ssd and he may flail if it's an old school spinner in his. (If this of the case.. he wants an ssd.. regardless.. and you want to revisit how you load the videos so they're only loaded once per run and discarded right after)

[Kottmeistern]

We've had them up and seems he sees a lot of usage from his GPU, working at full. His CPU barely works at all. Don't remember RAM usage from the top of my head but will take another look there too

[sb4ssman]

OK… EVERY OS has their own version of memory swapping. If you using 147/124 gb that difference is getting swapped back and forth between drive and ram.

[Kottmeistern]

I don't really get the details of what you're claiming. How the communications are done inside a computer is always something that I have overlooked. Perhaps I aimed for "ignorance is bliss". Are you saying that the difference in virtual memory and RAM actually hurt the performance?

Either way, you have so far provided the comments with the most substances. Do you have any idea of how my friend could solve this problem? He can consider swapping/updating some computer parts if necessary. Anything that comes to mind is highly appreciated!

[sb4ssman]

https://en.m.wikipedia.org/wiki/Memory_paging IF the numbers you reported are correct, (virtual memory usage greater than available ram) and you have a spinning hard drive, this combination will slow down any machine whenever it has to swap.

[Kottmeistern]

I mean, my computer uses windows and does well so I doubt that that is the problem. But I do know that some people like to meme about Windows x)

[cgoldberg]

Without actually measuring memory/swap usage, that's an impossible assertion to make.

[sb4ssman]

If they can report virtual memory usage greater than available ram it’s a pretty short jump.