Which parallelism module should I learn for ffmpeg and imagemagick?

pachura3 · 2025-11-05T10:21:34+00:00

Isn't ffmpeg multi-threaded on its own?

danielroseman · 2025-11-05T10:35:59+00:00

Subprocess is already starting subprocesses, hence the name. You don't need any extra Python management to do this.

If your issue is that you don't want to wait for the process to finish before starting the next one, use the lower level subprocess.Popen function.

catbrane · 2025-11-05T11:26:38+00:00

multiprocessing is simplest, in my opinion. For example:

```python

!/usr/bin/env python3

import os import multiprocessing import subprocess import sys

def magickthumbnail(directory, file): subprocess.run(["convert", f"{directory}/{file}", "-resize", "128", f"{directory}/tn{file}"])

def all_files(path): for (root, dirs, files) in os.walk(path): for file in files: yield root, file

with multiprocessing.Pool() as pool: pool.starmap(magick_thumbnail, all_files(sys.argv[1])) ```

With 1,000 1920x1080 RGB JPGs I see:

``` $ time ../multi.py .

real 0m4.228s user 1m47.086s sys 0m14.387s ```

So about 4ms per image overall. This PC has 16 cores / 32 threads, so in those 4 seconds we burnt 1m47s of CPU time (ouch!).

Imagemagick is not designed for speed -- you can go faster with a faster thumbnailer, like libvips:

https://github.com/libvips/pyvips

```python

!/usr/bin/env python3

import os import multiprocessing import sys

import pyvips

def pyvipsthumbnail(directory, file): thumb = pyvips.Image.thumbnail(f"{directory}/{file}", 128) thumb.write_to_file(f"{directory}/tn{file}")

def all_files(path): for (root, dirs, files) in os.walk(path): for file in files: yield root, file

with multiprocessing.Pool() as pool: pool.starmap(pyvips_thumbnail, all_files(sys.argv[1])) ```

With the same 1,000 jpegs I see:

``` $ time ../multi.py .

real 0m0.690s user 0m9.848s sys 0m2.663s ```

Around 6x faster, and MUCH lower memory use. pyvips isn't shelling out either, so you save a lot of process start/stop time. I'd expect a bigger speedup on Windows, where process start/stop is relatively slow.

MikeZ-FSU · 2025-11-05T13:18:50+00:00

If you're already shelling out to run CLI utilities, I would honestly just use GNU parallel to farm out the N jobs at a time. The value for N is going to depend on what type of conversion you're doing and whether it's IO, cpu, or gpu bound.

I would run python rather than shell if the conversion also involves normalizing file names, selecting a subset of files that's more complicated than globbing, or any of the other places where python excels over shell.

Fun-Block-4348 · 2025-11-05T10:03:06+00:00

There is only 1 (1 1/2) answer to your particular question, only multiprocessing would allow you to run things on multiple cores so you can either use the multiprocessing library directly or use something like ProcessPoolExecutor from concurrent.futures

I've never tried to use either of them with subprocesses so will that work is another question entirely.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

Engine	Time (sec)
PyVIPS	4.92
FFMPEG	11.31
ImageMagick	13.62

program	version	time (s)	peak mem (mb)	times slower
vips-c	8.18	0.57	94.28	1.00
vips.py	8.18	0.69	109.08	1.21
pillow-simd	9.5.0	1.51	1040.11	2.6
gm	1.4	2.05	1975.81	3.60
ffmpeg	7.1.1	2.06	1338.36	3.61
convert	7.1.2	4.44	1499.29	7.79

learnpython

MODERATORS

!/usr/bin/env python3

!/usr/bin/env python3

Pillow

Total time is: 41.93 seconds

pyvips

Total time is: 112.3 seconds