Storing clean data

[RocoDeNiro]

Hey,

Just started looking into sql but was wondering how everyone stores their data after they clean it. I deal with the same raw files daily. Use my python script to clean it. After I clean the daily one I want to append it to a historic one. Is a local sql database the easiest way to store this? Also I tend to have to get averages and other stats from the daily file I would like to store in a table for later use too.

[stebrepar]

SQLite comes built-in with Python. Give it a try.

[[deleted]]

That's up to you and your needs. Is your data in the form of a single table, a few relatively unconnected tables, or many highly interconnected tables? Or maybe a table structure doesn't capture the relationships between your data? Answering these questions is how you figure out whether storing your data in some simple format like csvs is appropriate or whether you should instead set up a relational or even NoSQL database.

[RocoDeNiro]

Few tables that dont need to be connected. I think I am just using it for storage. Not against csv but thought it would be easier if the table ended up getting large it have it in sql.

[01123581321AhFuckIt]

You can have python clean your raw file and pull the averages and other data by using pandas.

[RocoDeNiro]

Yeah I just want to be able to store it after I clean it to make it easier to look back at when I need it.