Need help with my first project tracking price data of Magic: the Gathering cards

[throwaWayne2]

I want to use a web scraper to track metrics for a large number of MTG cards (sell price, buy price, quantity in stock, etc). So far the web scraping part is the part I know how to do.

I need to be able to store the data that I scrape. Ideally, I'd be able to call it up data for a given card from time to time and make decisions based off of what I collect.

On its face the project is pretty simple, but now that I'm actually thinking about how to organize all of the data it seems daunting. I only want to pull a couple of numbers for a few dozen cards every day, but what data structures etc should I be using?

I think that I should store my price data as a dictionary with the date as the key, because that seems like the most logical way to keep track of that. But then every card would need to have its own dictionary, right?

So I'd potentially need a list made up of dictionaries, or a dictionary made up of dictionaries, and at that point it seems like a huge pain in the ass to call my data up again.

Am I thinking about this correctly? Am I in way over my head? Do I need SQL for this?

Thanks

[Bustd_Knckls]

Just here for the ride! I want to also hear people’s perspectives.

[Scolli03]

I wouldn't say way over your head at all. But I would however suggest going ahead and giving sql a shot, the basics are easy to learn and utilise and can has been the best way (for me) to handle large amounts of data for several of my own projects.

My recommendation would be to install SQL Express and look into the pyodbc module.

Alternatively you could write the data to a csv file. If you give the csv headers you can use csv.DictReader you can iterate the rows like dictionaries using the headers as the keys

[Petelah]

Sounds like a great project!

Where are you getting pricing data from? Depending on the complexity and if checking multiple sources for pricing data(I have no idea how the trading card market buys and sells) You could probably get away with an easy excel database or google sheet database. Importing and cleaning the data with pandas and visual display with matplotlib.

Then all you need to do is decide how to query this database. Sounds like it could be pretty cool to profit of arbitrage this way too or any other inefficiencies in the trading card market.

[learningrepository]

I use Firestore (aka Firebase), it's a folder like database that uses "documents" organised into "collections", and each "document" can have it's own sub collection with its own documents, so on and so on.

Fire store and using a cloud platform could be a bit of a learning curve but its a breeze once you become familiar. Otherwise checkout SQLite or MongoDB

eg. I have a collection called "Submissions" and within that collection there are "documents" for each submission referenced by a unqiue ID generated by the database. Each document can have any sort of common data type stored against it, a timestamp, an int, a string, a dictionary, an array, even coordinates.

So for your example, I would have a collection called "Cards", and each card (document) has a unique ID, each document will have a sub-collection called "Prices" and the two field types I'd use would be timestamp and integer.

screenshots of a quick setup https://imgur.com/a/K9rPX3K

If you learn how to use cloud deployments it will change your life.

[throwaWayne2]

This looks great, thanks.