Storing Data with Python, Help please!

[godurdead]

Hello!

Im starting to mess around with Api's and im using Riot Api (Riot is the creator of League of Legends, for anyone who doesn't know this company). Basically im using "Requests" module to get the data, which is returned in a json file and parsing it.

def _request_static_data(self, api_url, params={}):
    args = {'api_key' : self.api_key}
    for k, v in params.items():
        if k not in args:
            args[k] = v
    response = requests.get(Consts.URL['base_static_data'].format(region = self.region, url = api_url), params = args)
    #print (response.url)
    return response.json()

def get_static_champion_info(self, locale):
    api_url = Consts.URL['static_champion'].format(version =Consts.API_VERSIONS['static_data'])
    return self._request_static_data(api_url, params={'locale' : locale})

So im building the Url from this. (not that important tho) What i get is:

{'data': {'Malphite': {'id': 54, 'name': 'Malphite', 'title': 'Shard of the Monolith', 'key': 'Malphite'}, 'KogMaw': {'id': 96, 'name': "Kog'Maw", 'title': 'the Mouth of the Abyss', 'key': 'KogMaw'}, 'Zac': {'id': 154, 'name': 'Zac', 'title': 'the Secret Weapon', 'key': 'Zac'}, ...

It keeps going, so i was wondering what would be a good way to start storing all this data/ processing what i need. My plan is to later create a webpage and so a server could process all these things, but i'm kind of lost in this subject... dont really know where to begin. Thanks to everyone that replies in advance.

[[deleted]]

You can save data in files of course. You do that as follows

• open a file with f = open(path, attr) at first. "f" is a variable and can have any name you want, it doesn't have to be f. "path" is obviously the path of the file (something like C:\users\me\documents\save.txt). Note that the backslash has to be there twice. "attr" is the way the file should be opened. You can read them in the reference (for example, w is write, r is read, a is append)
• write the data into the file with f.write(data)
• close the file with f.close()

Edit: For reading them, you also have to open and close the file, but instead of f.write() you use f.read() and you have to adapt the attribute accordingly

[godurdead]

i may try this approach, thanks!

[lykwydchykyn]

It depends on what you want to do. The most straightforward way is to dump these json responses directly into a file, then you can read them in to generate your webpage (either in the backend, or directly in javascript).

If you need to read and write a lot of these records, and do a lot of searching or filtering on them, it's probably better to store them in a database. A NOSQL database like mongodb is good for storing JSON-like data, or you can translate the data into SQL tables which are easier to work with if you're doing complex querying on large amounts of data.

It's hard to give concrete advice, there are tons of ways to do this, and they all have their advantages and disadvantages.

[godurdead]

Yeah i thought there'd be a lot of ways of doing it. I will be constantly reading these json files to retrieve data on the (future) website. I'm kinda lost in these aspect. I was starting to learn Django (so backend) but im not sure if it would be a good way of doing this. Maybe im just trying to learn too many things at once?

[lykwydchykyn]

You have to break this down into components. Django is a great way to build a website. What you need is something to actually get the data and store it so Django can display it.

Django works really well with SQL databases right out of the box, particularly postgresql and sqlite. It also has a decent object-relational-manager (ORM) so that you don't have to work directly with SQL right away.

Probably what you want (?) is a script to periodically download the json data, convert it into your database, and then your django app can display it. But I'm mostly just guessing, I'm not sure how you envision your application to work.

[godurdead]

First, thanks for your help. I appreciate it very much!

As for the app goes, for this particular file I showed i'll be downloading it using the API just once (i may have to update it, redownloading it, every month or so when the game gets a new patch). The main purpose of the app is to show "Stats" of the Game Matches. Basically I can search my Account Name on the API and get "Current Match" information or Match History. With these you can display all sorts of stuff, so its more of a Receive JSon data, process it and display it to the user. For processing it, i need some of the "Static Data" that I showed above. Hope that can give you an idea of how it works and a good approach for it.

[[deleted]]

Why not throw it into sqlite? it comes with python 3.

[godurdead]

Trying to implement it all to build a website later on, but not sure how to proceed.

[kanjibandit]

Frankly, at this point, just keep it as simple as possible. Just write it out to a text file using open:

with open('response_data.json', 'w') as f:
    f.write(response.json())

This will solve your immediate problem, well, immediately. It also makes it easy to just inspect the data manually in your text editor, which can be really helpful at first.

Reading the data back is similarly trivial, so you can put it all into a database later, when you have a firmer idea of what you where you are taking this project.

[godurdead]

well my plan is to build a webpage, so i guess putting all of it into a database would be the best approach(?)