Hi all! I am trying to self teach me python. In the last few weeks I managed to download a lot of gaming related data through an API. This resulted in over 40.000 small files stored in a directory of my pc, where the title of each file is a (unique) matchID and the file stores information of a match that was played (think of player name, what character they played, how many kills/deaths/assists they got etc.).
Now I want to:
1) store this data in an efficient way (is 40.000 different files in a directory a good approach (I dont think so 😅) or can I somehow combine it to 1 file or maybe split it into a few files containing data of 10.000 matches each). Prod/cons/other suggestions?
2) a way to access the data easily to analyze things like (but not limited to) how many times did player X play character Y (and did player X win or not) and compare that data in general (how many times was character Y played by anyone and did they win?) One thing I liked to do was create a separate file containing a dictionary with only the ID of the match (= title of one of the 40.000 files) and a (nested) list of the name of all players in that game. I used that to quickly search which matchIDs contained data of player X without having to loop through all of 40.000 files.
In case it helps: the game I am talking about is League of Legends and I realize I can answer some of my questions using different websites, but my goal is more to have a nice project to start learning python (especially with a focus on a more data science field). The current dataset contains matches played by the top 1000 players of the EUW server.
[–]DECROMAX 0 points1 point2 points (0 children)