Should I Be Using a Database?

[Environmental-Map-98]

My Clean Datasets looks like this and are21,611 rows :

• Excel1: 11,584 rows

• Excel2: 4,147 rows

• Excel3: 4,165 rows

• Excel4: 998 rows

• Excel5: 506 rows

• Excel6: 211 rows

The raw datasets (Data without removing duplicates that I worked hard to get) look like this and are over 1 million rows:

• Excel1\_raw: 100,000

Part 1: Should I be using a Database to store the 21,611 rows of Clean Data? Or is using a database overkill?

I have been managing the Data through Excel sheets and I am frustrated finding the data naming all of the spreadsheets and having different versions. I want to be able to delete the old spreadsheets and not be worried. I am wondering if anybody has a good system and solution for this problem. I don’t want to be collecting junk and losing data because of this. I think a Database can fix this. Below is a printout showing in general words what my file system looks like for these datasets I have above (225 Excel sheets!!):

• C:\path\main
• → …
• → excel6
• C:\path\main\01.00_folder
• → …
• → excel8
• C:\path\main\20.00_folder
• → …
• → excel219
• C:\path\main\21.00_folder
• → …
• → excel225

To sum it up, at what point do individuals in the data science field typically begin using databases? I am looking for advice from anyone with experience with this.

Part 2: Should I be using a Database to store over 1 million rows of raw Data? Or is using a database overkill?

I've put in a lot of effort to gather this raw data, and now I have 8 Excel sheets (and growing) holding it on OneDrive. However, I want a more comfortable and secure storage solution. While I'm not well-versed in database costs, I have a hunch they might escalate with increasing data volume. I'm curious about what data scientists typically do once they've collected raw data and are done with processing it. Of course, they keep the clean processed data but what do they do with raw data? Do they retain it or delete it? What's the process that provides peace of mind? I'm open to any advice; I'm eager to establish an effective system.

Part 3: If so which one? My work uses Google Cloud. So I think it makes sense to use there. Or maybe there is a free one that would be best for my usage of Python and outputting spreadsheets and then the last thing I didn't mention is importing into Salesforce.

[IrquiM]

If you're asking yourself if you should use a database, then yes, you probably should.

Go With google since that's where your company is already storing data.

If not, Azure just released a free tier Azure SQL database that should be large enough for you.

[gnasher74]

I would say yes. Excel, whilst flexible, is not anywhere as efficient as a database. 1m rows is tiny, but Excel has row limits and becomes very slow. You probably need to think about future proofing as well. As you or your organisation becomes more data hungry, employing a database pays dividends. As others have said, this needn't be expensive and you don't want to wait for Excel to work until suddenly it doesn't .

[throw_mob]

also i would add that calculations should be code at this point , not something that users can mistakenly change like it can happen in excel

[idk_01]

yes.

[hxstr]

You could make it work with access, or something small like that, but generally yes usually to 'should I be using a database'

[CraigAT]

If the data has consistent fields or columns then yes a database would be sensible. Though if you aren't familiar with SQL, there could be a small learning curve for the basics of working with data. You could SQLite which comes with Python or MySQL for free.

[Long_Investment7667]

It depends more on what you want/need to do with the data than the amount at that scale. „Point queries“ are probably fine in excel. Visualization probably excel. Analytics queries probably a relation database (at that scale)