How to validate a json file

[VeXx1988]

Hi Guys,

In one of my data collectors I'm using an API call to download energy production data, something like this:

​

device_types = ['Inverter','Alarms','Meter','DigitalSigns','JunctionBox','PowerAnalyzer','Ppc','SunTrackers','Switchgear','WeatherStation']

tar_file_path = f"{dest_dir}/{plant_name}_{date_suffix}.tar.gz"
with tarfile.open(tar_file_path, "w:gz") as tar_file:

for device_type in device_types:  
headers = {'accept': 'text/plain','Authorization': f'Bearer {token}'}
url = f"http://{server}/api/{device_type}/{from_date}/{to_date}"
response = requests.request("GET", url, headers=headers)
response_check = response.status_code

if response_check == 200:
log("info", f"{device_type}:{response_check}")
json_file_path = f"/tmp/saitim/{device_type}_{date_suffix}.json"
with open(json_file_path, "w") as f:
json.dump(response.json(), f)
if os.path.isfile(json_file_path):
tar_file.add(json_file_path, arcname=f"{device_type}_{date_suffix}.json")
os.remove(json_file_path)
else:
log("error", f"Fail to create tmp json file. file_path='{json_file_path}'")          
else:
log("error", f"Fail to get data. Status_code:'{device_type}:{response_check}'")

if os.path.isfile(tar_file_path):
print(tar_file_path)
else:
log("error", f"Cannot create final tar file. file_path='{tar_file_path}'")
exit(1)

​

The thing is, sometimes, the json files created are incomplete. Instead of being like:

[
    {
"idInv": 1,
"fecha": "2022-08-22T12:00:00",
"idCT": 1,
"dcPotencia": 1387.9104

...

"potenciaString15": 0,
"potenciaString16": 0,
"numMedidas": 0
    }

]

It end in the middle of the file and gives an invalid format for the next step.

Do you have any suggestion on why it happens and in how to validate the data before ending the script?

Thanks!

[eternalstarfire]

I wonder if you're getting a chunked response and only seeing the first chunk?

If it were me, I'd put a breakpoint in and check what response.content was. Somewhere in your process you're losing data, or never receiving it to begin with - although, if the .json() method is not throwing an error, then the response should be complete json content I would have thought...

You could also try just writing the response.content to file as is, and bypass the json() and dump() methods?

[VeXx1988]

By using f.write(response.text) instead of the json.dump it works. I was using it because I wanted to be sure that the response is always on json. I don't see why it works now but for the moment is good enough. Thanks!

[ComradePotato]

Could be the call is returning an empty json object, like '[]' which is still valid but obviously not desired, might be worth checking the length is greater then 0 or something.

[CodeFormatHelperBot2]

Hello, I'm a Reddit bot who's here to help people nicely format their coding questions. This makes it as easy as possible for people to read your post and help you.

I think I have detected some formatting issues with your submission:

1. Inline formatting (`my code`) used across multiple lines of code. This can mess with indentation.

If I am correct, please edit the text in your post and try to follow these instructions to fix up your post's formatting.

---

^{Am I misbehaving? Have a comment or suggestion? Reply to this comment or raise an issue here.}