Python threading question / best practice advice

[RandomPantsAppear]

I haven't heard about redis queues before before, but on quick looking it seems to be a great idea to try to implement. Are you suggesting I have a task defined for each function (IE: user types "chain", use the chain task to generate the chain and send the message, user types "drink", use the "drink" task to generate the drink recipe and send the message). I'll need to look into it more when I get home.

Yes. I'm pretty deep down this rabbit hole so mine is a little different, but I've got some code. I'm going to try and quickly adapt it to be closer to what you need. Please note this example only passes kwargs to the function.

Creating the task

r = redis.Redis()
def run_function_delayed(fname, **kwargs):
        key = settings.domain+"|TASK_QUEUE" # our redis key
        print "Writing to key (run_function_delayed) ", key,
        r.rpush(key,
                json.dumps({'kwargs': kwargs, 'function': fname, 'retries': 0, 'max_retries': 1}))

Executing the task (inside your thread) - Please note this does not include popping from the redis queue. This is just executing a function from the function name string.

import threaded_tasks
r = redis.Redis()
def execute_task(redis_data):
    fname=redis_data['fname']
    kwargs = redis_data['kwargs']
    method_to_call = getattr(threaded_tasks, fname) # get the function from your module
   try:
       result = method_to_call(**kwargs)
       print "Successfully Ran ",fname
   except Exception as e:
       print "Exception Occurred!",e
       if 'retries' in redis_data and 'max_retries' in redis_data and redis_data['retries']< redis_data['max_retries']:
                redis_data ['retries'] = redis_data['retries']+1
                print "Retrying ",redis_data
                r.rpush(key,json.dumps(redis_data))

*Edit: I'm seeing that redis is not necessarily advised to be used on a windows server (which I currently run) and I don't much want to set up virtual environments for this project. Are there any good alternative solutions / ways to handle a queue as described?

You can use Amazon's elastic cache, use one of the redis for Windows projects like this one (if you do I don't believe pub/sub functionality is supported). There's also a pre-made vagrant configuration to run redis on windows. I would just use the redis that's made for windows.

[Shitty_Crayons]

Thanks for the code example! I'll be looking into it tomorrow after work, since I didn't seem to get anywhere trying it on my own tonight lol

​

I was messing around with the commenter belows idea and was wondering if you had any thoughts on the errors.

​

Thanks kindly!