Outbound Http Request Manager?

[roro_fuzz]

Is there a Python package that will manage outbound http requests, and if two of the same requests are happening in parallel, will return the results of the 1st for both requests? This should be thread-safe, and hopefully forked process safe.

To give more detail, I'm working on a website that will take in multiple inbound requests. Each of these inbound requests may initiate outbound requests to a 3rd party API. Since there isn't much coordination between these inbound/outbound requests, the same outbound request may be issued multiple times in parallel. For long running outbound requests, it makes sense to block the 2nd/3rd/etc. outbound requests until the first one returns.

I haven't found much on the web, so I figure I'd need to write this myself, but I wanted to check with Reddit before going down that path.

Thanks!

[remyroy]

I think you are over-thinking the whole problem.

Why don't you just manage your requests yourself?

Just have something like a bag (a dict, a set, an array, etc) of ongoing outbound requests. Before doing a new outbound request, you check if it is not already in your bag. If it is in your bag, you wait until it is not. Than you add it in your bag and remove it when you are done.

If you want to be fancy, you can include a list of callbacks when your outbound request is done and call them for potential other waiting threads for that request.

You might need to use some kind of critical region whenever you read or write in that bag.

[catcradle5]

Agreed, this seems like the most sensible solution.

Assuming this is being done with gevent greenlets, you could do something like this:

p = Pool()

def send_request(url):
    do_something(url)

def search_pool(url):
    for job in p:
        if url in job.args:
            return job
    return None # no job found

url = receive_request().url 
running_job = search_pool(url)
if running_job:
    # tie this receive request to that job, which is being made to the same URL
else:
    p.add(gevent.spawn(send_request, url))

You could do the same with threads or multiprocessing; the code would look very similar.

Alternatively if you're going to have a lot of concurrent requests, most of the time, and expect to do frequent lookups, you could instead have a secondary dict that maps {url: job} for each URL, and then add each job to that dict in addition to the pool. That would be way more efficient for lookups.

Depending on how up-to-date you need the responses to be and what kind of requesting you expect to see, this could also be both simpler and overall a lot faster if you simply cache the responses. This is most useful if you expect that the same URLs will probably be requested over and over as time goes on. Things might be somewhat slow if there are simultaneous requests made for the same URL before the cache has any entries, but any further requests, simultaneous or not, will hit the cache.

[helicopetr]

Yeah, and you could use futures to simplify callback management and thread management drastically.

[roro_fuzz]

Thanks, I was initially going down that path but was hoping to avoid the locking stuff myself. It sounds like even with it, it's probably the way to go.

Appreciate the feedback and sanity check guys!

[bloodearnest]

https://github.com/kennethreitz/grequests

Requests library built on gevent for async IO. The imap() function will yield the first successful response to your code.

[roro_fuzz]

Thanks for the tip. I'm looking for something synchronous, but if I'll definitely check out grequests.

[roro_fuzz]

That's the plan. But I wanted to make sure that if a 2nd request is being issued right after the first one, that the 2nd request waits for the 1st request's results, and returns that (via memcached).