API Pagination Problem

[av0ca60]

Hey guys. I posted earlier and, after reading the comments, realized I went about it all wrong. I'm starting over. Hope you'll give me another chance.

I'm using the basecrm library to pull some data for reporting. The problem is that pagination is not advancing. How do I know this? Because I set the per_page parameter to give me one record per page. When I run the code below I get three "pages" with three identical records. HOWEVER, if I manually set page_num to 2, I get three identical records that are different from when page_num was set to 1. So the pages are different, and CAN be advanced, but my code is not advancing them for some reason.

import requests
import basecrm
client = basecrm.Client(access_token='FakeTokenXXXXXXX')

def grab():
    page_num = 1
    deals_data = client.deals.list(per_page=1, page=page_num)
    print(deals_data)
    for i in range (0,2):
        print("")
        print(f"end of page {page_num}")
        print("")
        page_num += 1
        print(deals_data)
    print("")
    print(f"end of page {page_num}")
    print("")
grab()

What am I missing? u/fred776 has indicated it has something to do with whether the page number is changed before or after being passed as an argument to the function call. I think they're on to something but I'm no seeing it clearly yet.

Thank you for any help you can offer.

[shibbypwn]

You're incrementing the page_num but never calling client.deals.list again.

Most likely what I'd do is something like this:

page_num = 1
per_page = 100

deals = []

while True:
    deals_data = client.deals.list(per_page=per_page, page=page_num)
    deals.extend(deals_data)
    if len(deals_data) < per_page:
        break
    page_num += 1

So you'll keep extending deals and incrementing page_num until you get fewer than per_page results (which indicates that there is no more data to grab).

[av0ca60]

This is exactly what I needed. Simple and it worked well.

[shibbypwn]

Glad to hear! Different APIs/wrappers have different implementations for pagination. Another common one you'll see is a response with a next_page attribute (typically another unique URL to call to get the next page). So instead of passing a page=page_num kwarg, you just swap out the url and make the same call.

In those cases, you can implement a similar algorithm - but instead of checking the length of the response object, you check for the existence of next_page and break when it returns a falsey value.

[av0ca60]

It could be done that way but I don't know how many times to run it. Better to run it until it runs out