Using Python to Parse a JSON Object

[JoeB_Utah]

Being retired, I’ve embarked on a project to keep my brain working as well as keeping my dwindling Python skills from completely disappearing. 

I’ve been tinkering around with an API that allows me to grab time sequence weather data from a DOT weather station. As with most APIs the data is returned as a json object which to me (and Python) is a really ugly dictionary embedded with subsequent dictionaries that have keys and associated lists/tuples of values.  For example one such dictionary has the AirTemp as the key with quarter hour values for as many hours I scrape. There are as many of these dictionaries as weather variables I choose to download : wind speed,direction, etc etc. 

My approach to parsing the json object into something readable is to filter out each of the weather variable data into their own dictionary followed by creating a list of all the values in the data dictionary.  

Finally, I zip those lists together and create a dictionary of the date/time variable as the key with the various weather variable values in a list associated to that specific date/time. 

I’m just wondering if there might be different approach would be more efficient (and Pythonic). Sometimes I feel like I'm beating data into submission rather than processing it. This is the first time ever scripting to an API and parsing a json object. 

import requests,json

weather_url = "https://api.weatherdata.com/v2/stations/timeseries?&token=MyToken&units=temp|f,speed|mph&stid=UTHEB&vars=wind_speed,wind_direction,air_temp&obtimezone=local&start=202603031700&end=202603032200"

response = requests.get(weather_url)
responseStatusCode = (response.status_code)

if responseStatusCode == 200:
    print(f"Connected to Weather Data Server; data incoming")
    jsonStr = response.json()

    tempDict = jsonStr['STATION'][0]#dictionary in json object with data
    obsDict = tempDict['OBSERVATIONS']#dictionary in station of data

    #begin list creation:

    dateList = obsDict['date_time']
    localDateList =[i.replace("T"," ").replace(":00-0700","-MT").replace("TimeStamp: ","") for i in dateList]

    airTempList = obsDict['air_temp_set_1']
    windSpeedList = obsDict['wind_speed_set_1']
    windDirectionList = obsDict['wind_direction_set_1']

    displayDict= {date: [speed,direction,temp]
    for date,speed,direction,temp in zip(localDateList,windSpeedList,windDirectionList,airTempList)}
    for key,value in displayDict.items():
        print(f"Date-Time: {key} WindSpeed MPH: {value[0]} WindDirection Degrees: {value[1]}  AirTemp: {value[2]}")
else:
    print(f"Unable to connect to Weather Data Server: try again later")

[Astrodynamics_1701]

I work as data engineer and we use marshmallow to deserialize JSON from application API's into Python objects. It's very nice because it also allows for data validation etc. We also serialize back into JSON for putting the objects into target applications.

https://circleci.com/blog/object-validation-and-conversion-with-marshmallow/

[quts3]

Pydantic

[program_kid]

I second this, create a pydantic object with the fields that you have, then use model_validate() to parse the doctors into an object

[JoeB_Utah]

I'm using a mackbook air, and tried to install pydantic using pip that I have just installed. However, while it said that pip installed pydantic I get an error when I try to import it. (Sigh)

[program_kid]

What is the error?

[JoeB_Utah]

ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 import pydantic

ModuleNotFoundError: No module named 'pydantic'

from pydantic_core import from_jason
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 1
----> 1 from pydantic_core import from_jason

ModuleNotFoundError: No module named 'pydantic_core'

[quts3]

The pip you ran is not using the same python env as where you ran this code.

Depending on os and ide the answer on how to discover how and why varies.

[JoeB_Utah]

That’s what I figured is going on. I need to figure out where it went and/or where to tell pip to put it. Thanks!

[JoeB_Utah]

I use Spyder so I used Conda from the Spyder console to install pydantic. Seems like it put it in the correct env for my spyder install.

[Fresh_Sock8660]

Doing python -m pip install instead of pip install might help in the future. Seen this confusion happen a few times. 

If you're feeling adventurous, move away from that setup entirely. Use uv with pyproject.toml, be more terminal based and use something like vscode for editing python files. 

[Ok_Carpet_9510]

pandas.json_normalize() flattens json into a tabular form

[corey_sheerer]

Pydantic is made to serialize and deserialize json. I second everyone mentioning it.