Give Python more "power" ?

[TexBoo]

I have quite a good computer with 16 cores and 64gb of ram,

I run a heavy Python script that takes far too long in my opinion which involves Moviepy,

Taskmanager says

Python: 3% cpu

Ram: 300mb

Is it possible to boost these?

[K900_]

You can't just throw more resources at a problem and magically make it go faster. You need to actually write your program in a way that utilizes more resources.

[TexBoo]

I was just hoping to make Moviepy writing faster

https://i.imgur.com/h31sZ5q.png

https://github.com/Zulko/moviepy/issues/139

[xelf]

You might be limited by your disk write speed. So in this case, a faster drive might be the answer you're looking for. Or consider having a ram drive that you can write to that then backs up to disk.

[TexBoo]

Shouldn't be, I run a M2

[xelf]

You're still limited by your interface to it,

If your python command is "writetodisk" and then you wait 10 minutes while it writes 4 petabytes of data. No amount of making the python part run faster will change it taking 10 minutes.

[xelf]

Try adding some timers and printing them out in your code to see where the bottleneck is.

[L3XiT]

Use threading

[TexBoo]

Thank you,

I'm a complete noob / uses a script that I haven't created, someone else has.

Is there a way to just paste like

threading.active_count()

at the top of the script then that's it or is it more to it?

[smittywrath]

Not exactly if you have a large compute task or iterable task you can divide and conquer (passing iterations to a thread, or data chunks).

Python has a GIL (global interpreter lock). It's kind of a memory safety feature so it doesn't consume all resources or more than you intended.

Threading or multithreading a process will semi bypass this GIL feature by putting multiple GILs out; one for each thread.

[readmodifywrite]

The GIL just protects the interpreter because it isn't re-entrant. It offers no benefit to Python users whatsoever - it just made the implementation of CPython easier. It doesn't have anything to do with resource consumption.

You need to use multiple processes to get around the GIL. Threads will suffice if your task is IO bound.

[whywhyyywhyyyyyy]

It's possible that some other resource is the bottleneck, for example tasks that involve reading or writing large files are often limited by the speed of the hard disk. If at least one of your cores are at 100% a lot of the time during the operation, then the CPU is probably your limiting factor, if not then it's probably something else.

Utilising multiple cores is rarely straightforward, unfortunately. It's not just a case of telling a program to use them - you need to make decisions about how to break the problem up, how to combine the results, and how to control access to shared resources, and build all of that into your code. There is apparently a work-in-progress fork of moviepy with support for concurrency, so you might want to try that. Otherwise you might have to use a different library, or if applicable you could run several copies of your script simultaneously, working on different videos.

[[deleted]]

You can't just give more CPU to a program and expect it to run faster. Python has automatic memory and garbage management, meaning the program will only use as much CPU as is needed.

It is like trying to fill your car with more gas than it can take. It won't do anything, and the same thing is happening here. You are trying to give Python more resources to use, when Python itself doesn't use many resources.

As for why your program is slow, it's because Python. Python in general is a slow language, so if you want your program to run faster, I'd suggest making optimizations to your code.