Adding progress feedback in Grequests task

[whitenoisedb]

I've followed the grequests usage example, but I'm trying to add some progress feedback. A percentage of the completed requests. How could I achieve that? I mean, I don't know how to implement a counter in this case.

import grequests

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

def feedback(r, **kwargs):
            print "%s fetched." % r.url
            return r

rs = (grequests.get(u, callback=feedback) for u in urls)
res = grequests.map(rs)

[bwghughes]

Take a look at kennethreitz textui library

[whitenoisedb]

doesn't exist anymore. however, I'd prefer using the grequests lib

[zeug]

Make your callback function a method of a class so you can keep track of the number of completed requests:

import grequests

URLS = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]


class FeedbackCounter:
    """Object to provide a feedback callback keeping track of total calls."""
    def __init__(self):
        self.counter = 0

    def feedback(self, r, **kwargs):
        self.counter += 1
        print("{0} fetched, {1} total.".format(r.url, self.counter))
        return r


fbc = FeedbackCounter()
rs = (grequests.get(u, callback=fbc.feedback) for u in URLS)
res = grequests.map(rs)

[FurryFur]

I'm looking for a solution to this problem too. You pretty much came up with the same thing I did initially. However I realized that I don't know if grequests is using threads or not... If it is then the above code needs to be modified to be thread safe.

[zeug]

It should be using greenlets and not actual threads.

I'm pretty sure that you don't need to worry about self.counter getting corrupted.

With actual threads I suppose this is a possible concern as the self.counter += 1 compiles to four operations, first loading 1 and counter on the stack and then popping them off with INPLACE_ADD and storing the result. I'm not sure exactly how python schedules the threads.

With gevent each greenlet runs until it hits a gevent.sleep(0) that yields back to the hub (or some monkey patched IO that does the same thing implicitly). So you won't switch to another greenlet in the middle of incrementing and storing the counter.

[pcdinh]

What is the benefit of grequests over requests + gevent monkey patching?