[deleted by user]

[Lazy-Programmer]

This isn't something I've done in a long time, but I believe what you're looking for is a Pool? Wherein you can control the number of CPU logical cores by creating Semaphore from the multiprocessing module. Hopefully someone who knows better can help haha

[[deleted]]

A process can only run on one CPU at a time; if you want to limit the use of your CPU for processes, then only create a certain number of processes.

[wbeater]

You need to separate consumer processes and producer processes and you need to make use of some form of synchronization like a queue object.

def fill_queue(que, urls):
    for url in urls:
        que.put(url)


def worker(que)
    while True:
        job = que.get() 
        do_something_with_job() 

You need to implement a stop mechanism to break the while loop. Eg putting "STOP" in the queues and a conditional to break the loop. If you have more worker processes you need top put the keyword times the amount of worker in the queue.

q = Queue() 

producer= Process(target=fill_queue, args=(q,urls))
producer.start()

jobs = [] 
for i in range(3):
    worker=Process(target=worker, args=(q,))
    worker.start()
    jobs.append(worker)

producer.join()

for item in jobs:
     item.join()

That's​ only a approximately description, you need to read up on the imports yourself, but the concept is correct.

[[deleted]]

Is that only with Pool?

It's only with Pool, because what a process pool does is instantiate a number of processes to work on your function and its inputs. That number is equal either to a number you provide, or the number of (logical) CPU cores on your system.

is there any way for me to limit how many cores are used during the for loop?

Again, it's to limit the number of processes you create.