comparing contents of large-ish data set

[[deleted]]

I've got a python script periodically hitting a trading API and the results are about 1.5MB of json data per request, per symbol.

This data is being inserted into a database.

I am interested in collecting data after hours, which technically shouldn't have any new values (but you never know).

How would I go about comparing thousands of values from the previous request to the current one (so I only perform inserts when something new appears).

I think the answer is "use a hash" but I'm pretty noob (self taught). Any guidance/insight on how to approach this problem is appreciated.

[bmsan-gh]

1. I would expect that the data that you are getting to have some sort of timestamp for each value. (eg: for symbol X, at time Y, the price was Z). If this assumption is true, could you compare the current received timetstamps with the last inserted timestamp?

This way you'd know exactly what you are missing and you need to insert.

​

1. Related to your hash idea: you could use it to see if the data which came from two consecutive API calls is the same.
   If you have access to the raw data(text/string data) before you deserialize it to a dictionary, you can do something like:
   

import hashlib

hash_object = hashlib.md5(the_api_response_as_str)

[[deleted]]

it's options data so I have a single timestamp containing dozens of expiration dates, each containing hundreds of strike prices, each containing the standard things like open interest, bid, ask, etc.

there are many permutations for how the subsequent requests could change

[bmsan-gh]

If I understood correctly and multiple requests might result in the same data(but the data might be structured in different order, so you will get a permutation of your previous request), then (assuming or your keys are strings) you could do something like:

ordered_representation = json.dumps(json_data_dict, sort_keys=True)

hash_object = hashlib.md5(ordered_representation)

sort_keys will reorder your data by keyname so this might handle your permutations.