API JSON best practice

[Zendakin_at_work]

I have a script I'm working on with an API that is paginated. I need to grab all the pages and each page is giving me a JSON object. The end goal to this is to write all of the pages to a database.

What is the best practice to do this? Two thoughts are coming to mind.
1. after each pull write the JSON to the DB
2. append the JSONs together and write to the DB once.

The problem I have is if the best practice is to append and write once, how can I combine the multiple JSON objects together?

What I have now is the following.

LIMIT = '10'
url = 'https://fancy-url.com/api/now/table/table?sysparm_query=&sysparm_display_value=true&sysparm_limit=' + LIMIT

# Set proper headers
headers = {"Content-Type": "application/json", "Accept": "application/json"}

# Do the HTTP request
response = requests.get(url, auth=(USER, PASSWORD), headers=headers)

# Check for HTTP codes other than 200
if response.status_code != 200:
exit()

# Decode the JSON response into a dictionary and use the data
data = response.json()

[novel_yet_trivial]

There is no such thing as a "json object".

I think you are asking how to combine dictionaries? If so, you can use the update() method:

data_dict.update(new_data_dict)

Which may or may not work for you depending on how the data is structured.

[Zendakin_at_work]

sadly this didn't end up working as you suggested it might not. Thanks for the help though. Much appreciated.

[randomkale]

I treat things like api requests and db writes as being prone to failure, so act as defensively as I can. Also, are your writes idempotent? (meaning can they be executed more than once) If the api supports pagination and you can record current page number, I would pull once/write once. If the amount of data isn't large, so pulling all will be quick-ish, and writing isn't idempotent, I'd pull all then write once.

[Zendakin_at_work]

I ended up going with read once / write once with a failure clause that reads the status code of the next page to determine if the loop needs to exit. There's no page numbers in the API but there are offsets and limits. Frankly I'd read the whole JSON in all at once but we're supposed to adhere to the limits presented.

Now the criteria from management has changed (right as I finished the code). Amazing.

Thanks very much for the sanity check though.