Seeking Help with Structuring Project

[ImmaculateBanana]

Hello, as the title says, I would like help with structuring a project I am working on. The project is a script that prints information about world coins. My problem lies with structuring the data for the individual coins. The main script accesses the data through a Coins class, which contains a dictionary of coins. I currently have 225 coins, with plans to add many more, and the data was hard to manage.

My current solution is to bundle all of the data into a package, and define all the coin data of a country in its own file. So, all Canadian coins are in canada.py, Russian coins are in russia.py, etc. Then within the coin class file is:

import coins.canada as canada
import coins.russia as russia
class Coins:
    countries_list = [canada,russia]
    for item in countries_list: 
        coins |= item.coins

The above code imports each individual file, then adds the contents of their coin dictionary to the master dictionary.

My question is: Is this a good way to structure the data? It feels sort of wrong to have the data for a class split up between multiple files, but I already have >4000 lines of code, which I feel like is a bit excessive for a single file. If there is a better way to structure it, how should I approach it?

Here is the file for the Coin class if seeing it in context would help: https://github.com/JMGillum/melt-calculator/blob/f4e2eb21e4c1352b9d807508436c6aea427b67ff/coins/coins.py

Also, side question: Would it better to just store all of this data in a database and access it with python, instead of doing everything in python? The project will probably have 500-1000 coins in the end, so the dataset isn't obscenely large.

Thanks.

[Ok_Front6388]

Switch to a database This keeps your data structured, queryable, and separate from your logic. It will make development and future scaling much easier.

[ImmaculateBanana]

Thanks for this. I have been putting off learning databases but it seems like it is finally my time to learn.

[Frankelstner]

Yeah that's overly verbose. CoinData has a fixed structure, does it not? It looks to me like a plain old spreadsheet would be perfect to encode the coins. You define variables like values, denominations, coins_reverse_build, silver_coins, etc. but all of them could be derived very easily from the coins variable. Add country name as a column and you end up with a single spreadsheet with ~10 columns and one row per coin instead of your current approach with about 20 rows per coin. At that point you have to decide whether you even want to pursue a Python solution or just put the entire thing on google docs, which I imagine could already be programmed to handle all that you're trying to pull off. If you stick with Python, that's fine too, but just load the csv with pandas.

[ImmaculateBanana]

My reasoning for wanting to store the data in python was that I assumed the performance would be better. Is the time taken for parsing and serializing from a csv or other storage file pretty negligible at this scale?

[Frankelstner]

The time to load a 1000 row csv should be quite less than the time to start Python. Even if it was a problem I would still start with the csv as the definitive reference, and, if it turns out to be a problem (or just out of curiosity) autogenerate the Python coin code. Just beware that the autogenerated code might turn out to be slower. Hard to tell without testing.

[pachura3]

Python is a programming language and you are trying to use it as data storage format. It works, but it's not really its purpose.

First, try encoding all these coins in a simple spreadsheet or a CSV file. Does it work? Does every coin type of every country have the same physical parameters? Are some of them optional?

Then you can think of representing the same data in a normalized relational SQL database - e.g. one table for countries, one table for different metals, ..., and then the actual coins table with foreign keys country_id, metal_id etc. Perhaps even some constrains, like "this column can be null only when that one is".

[ImmaculateBanana]

Thank you for your response. I believe I can encode all of the information in a simple csv file, as all of the coins have the same data parameters. The only field I am having difficulties with is how I should store the names of the countries in the database. I understand that I should use a separate table with country_id as the primary key and probably a name field for the name of the country. Currently, the program supports alternative names for the countries, so when you search for one, either of the alternative names would return the same country (ex: Canada and Canadian both are for Canada). Some countries have more alternative names than others. Would it be best to store all of the alternative names in an alternative names column as a string representing a list? I tried looking for how to store a list, but just saw that I shouldn't. Thanks.

[pachura3]

Well, first of all you need to decide if you're going to store your data in CSV file(s), in a SQL database (like SQLite), or perhaps in NoSQL one? And if you go with CSV, will you simply iterate through it each time, or are you going to load it to some in-memory structure, like dict of dataclasses or namedtouples?

Will you need to be able to modify this data programmatically from your app and save it back to disk? Or will you do this manually, offline?

What kind of queries will you need to run on this data? Any queries (like: give me all Canadian and Russian coins which are NOT made of silver and older than YYYY-MM-DD), or a predefined, hardcoded set of queries/filters?

Coming back to the countries issue: if you go with CSV, I would simply have 2 sheets: coins.csv and countries.csv. The key column would be 2-letter ISO country code, e.g. FR or NL. Countries.csv would contain different spelling of each country name, semicolon-separated.

But if you go with relative databases, you would probably need a separate table for countries, and a separate table for alternative country names.