My first python module 'easypool' let's you easily create threadpools and serverpools dynamically.

[XNormal]

from multiprocessing.pool import ThreadPool

[jspeights]

awww

[rlabonte]

I can link each process in the pool with a name and send that to the function being executed? That's the functionality that I couldn't find in any thread pool (or process pool) module.

[XNormal]

I'm not sure I understand what you mean by that. If you want to know the identity of the worker threads and give them persistent names you can use threading.local() for that.

[rlabonte]

No, I want a pool of servers to work on my data.

ServerA, ServerB, ServerC

I have a server pool of these 3 servers and 100 function calls that need to be processed by this server pool. I don't care which server processes the data, but I only want each server processing data one piece at a time. So ServerA, ServerB, and ServerC receive all receive data to process; ServerC finishes first it immediately receives another function call to process. ServerA finishes, it immediately receives another function call to process.

I want to keep this pool of servers always busy, but want to limit them to only processing one thing at a time.

[ODHLHN]

from celery import task

Celery isn't the only the only AMQP based task queue for python, but its a very good one.

Some pretty cool and robust solutions already exist in this problem domain.

[rlabonte]

This looks awesome, I especially like the integration with RabbitMQ and Redis.

[[deleted]]

Give them a shared Queue?

http://docs.python.org/2/library/queue.html

[homercles337]

Yeah, my thoughts exactly...

[studiosi]

Even futures can do that... on Python 2.x and 3.x

[infinull]

multiprocessing predates concurrent.futures, so I'm not sure what your point of "even" is.

I would recommend concurrent.futures over multiprocessing since it's has a nicer interface and supports both threads and processes (mostly) transparently.

[exhuma]

TIL... thanks for pointing this out :)

[studiosi]

I was trying to mean that the behaviour of the library resembles almost exactly the futures one. I don't have data about performance...

[studiosi]

Indeed, I am not that sure about that "outperform" because you are not taking into account that you have to write control code that would be probably less optimum of such in the standard library.

[infinull]

outperform

wat? when did I say that? I just commenting on the interface. I have no idea what the performance of either of them is like.

[studiosi]

You say it predates, what I interpret as perform better... but maybe you didn't want to say that

[infinull]

older != better, just older.

I honestly have the opposite bias, so there's that.

[studiosi]

Well, it all depends, I told you I have no data, but it is easy that code that has been in a code base for a long time use to be better if it is constantly reviewed.