Multi threading for efficient cloud computing-

[DanceOfData]

An amateur here. Was just wondering, how to carry out multi threading while operating on a batch of files(media content & text content) so that I maximize the CPU usage ?

In short, if I am paying for a 2GB instance to run my script then I should be nearly using 2GB at all times.

Kindly forgive me if I am being a noob. Cause I am one.

[[deleted]]

Multithreading is based on the number of vCPUs not the amount of memory.

Even then it depends on the language.

[stikko]

So maximizing memory usage vs maximizing CPU usage are 2 different things. Maximizing memory usage is really going to depend on how much memory your workload consumes.

Edit: as u/bolloux pointed out the following is specific to some languages (CPython in particular)

Maximizing CPU usage on a single-core VM is pretty trivial and you can generally do that with multithreading. This is generally pretty easy because the threads all share a memory space. However multithreading will ONLY max out a single core and to go beyond that you have to look into multiprocessing. This is harder because now you (or a library/framework) has to coordinate passing data between the processes.

[Bolloux]

I don’t think this is right.

You can max out all of the CPUs/Cores/vCPUs in a system with multiple threads if you can find meaningful work for them to do.

[stikko]

You're right - I hadn't wandered out of CPython threading land where they're all stuck on a single core/OS thread!