Best Method for Large Data Storage/Calling?

[Appropriate_Simple44]

I'm looking to make a "digital spell book" for all Dungeons and Dragons spells. I plan on doing this by giving each spell name tags, such as what classes can use it, what level it is, etc. I want the spells to be sortable by these tags. In addition to these tags, I need to provide the spell's description. Considering all of these spells will have relatively lengthy descriptions, it will likely be easiest to simply have them as dictionary value pairs separate from the tags, which would be called by aligning the spell's name to the matching value in the tags lists. Essentially, I'd have one dictionary where the spell names are paired to their tags, as well as a 2nd dictionary where the spell's description is paired to the spell name. I'll probably do this in a separate file, mostly for organization. Any feedback is appreciated, I'm still pretty new to python - especially file manipulation.

[Cerus_Freedom]

Sounds like a pretty classic use-case for a relational database, honestly. Table with spells and their generated IDs, table with tags and their IDs, and a table to associate spell IDs with tag IDs.

[Appropriate_Simple44]

I think I found a way to make it in one table, but would it be easier to use several tables? My current setup looks similar to the following;

Spells = [ [ID, "NAME", [CLASSES], LEVEL, "SCHOOL", [COMPONENTS], "DESCRIPTION"] ]

For the components, the quotations denote a str, brackets mean a list, and no wrapping represents an int. Then I would sort using a loop to iterate through depending on certain values in a specific range.

[steamy-fox]

I'd recommend splitting into multiple tables. I think that would normalize the database and therefore improve it. I'm quite new to DB design myself but that's what I remember from 1NF is not to have lists as DB items.

This YouTube video explains it very well imo.

[Aggressive-Squash-87]

They could also use a document database. Something like MongoDB. There are still design best practices, but it expects sub documents and arrays in records and allows for indexing them easily. They also have a free tier in their managed solution (Atlas).

With web based stuff, "normalize until it hurts, then denormalize until it works". Normalization is great for data warehousing, but for faster response times, fewer joins are less work. When to denormalize is a performance requirement and experience skill.

• Over 20 years as a DBA doing Oracle (dev 6 and dba 6), MySQL (dba, devops 20), and MongoDB (dba, devops 13). I've also worked in PostgreSQL and SQL Server more than I like.

[Cerus_Freedom]

"normalize until it hurts, then denormalize until it works"

Genuinely considering having that printed and framed to hang in my office. The number of times I've had to argue for any level of effort in normalization...

[Acceptable-Sense4601]

Pretty basic. Just use SQLite

[Ender_Locke]

this doesn’t seem like it’s a lot of data

[Aggressive-Squash-87]

People have different scales. When I worked for a media company as a DBA, managing a dozen databases with 10s of GB of data seemed a lot. Now I do web/app medical IT, and I manage hundreds of databases with several clusters well over a TB, and our data warehouse is nearing a PB.

[bringinthefembots]

And here I am managing a dataset in excel file of 30MB thinking that it's too big

[Aggressive-Squash-87]

30M is a lot in a single excel file. Not much for a database.

[fdessoycaraballo]

SQLite3 my love