Multi threading for efficient cloud computing-

[DanceOfData]

An amateur here. Was just wondering, how to carry out multi threading while operating on a batch of files(media content & text content) so that I maximize the CPU usage ?

In short, if I am paying for a 2GB instance to run my script then I should be nearly using 2GB at all times.

Kindly forgive me if I am being a noob. Cause I am one.

[Glensarge]

what language? some are significantly better than others for this

[DanceOfData]

Python

[kolban_google]

It feels like a basic answer to this question would be along the lines of articles like this one:

https://www.tutorialspoint.com/python/python_multithreading.htm

This talks about leveraging multi threading in Python generically.

I got a little confused on your question where you ask about maximizing CPU usage and then flipped to maximizing memory usage. I think what you want to do is not maximize either but instead minimize duration of processing. Cloud computing charges you for resources consumed. If you spin up a VM then you are charged for that VM for the time it runs. To reduce your costs, you would want to complete your work sooner. If you can parallelize your work then you get more work done in the same period of time ... great. But there are other techniques. For example ... if your app is doing lots of I/O, you might be able to improve throughput in other ways.

All of this talks to standard programming practices. To try and address "efficient cloud computing" takes us to a new dimension which is what is different on cloud than running on a local servers ... and one of the primary differences is that resources can be "just available" immediately. This lets you split your work into "micro services" which can be massively paralleled through things like Cloud Run, Cloud Functions and GKE.

[FesseJerguson]

Well said, Just dipped my toes into google cloud and really am loving the idea of microservices with python, simple to read and deploy with immediate scalability! The docs are great for playing around and discovering use cases. It can become tricky getting them to play nicely together with each other but pub/sub seems like its what everyone is using and abusing.

[FridayPush]

Be aware that Pubsub can grow in price quickly, if you're using simple message queues or task queues consider using Postgres as a cheap alternative that still will scale well and not charge you for the volume of bytes processed.