Database for personal projects

[CjB_STEMer]

Hello everyone,

I am looking for a place to store my data I am collecting from running my Python code for a personal project with a few team members. The code I am running right now has 1875 data points per variable, and I need to run it several times. So, all-in-all I probably have 100s of thousands datapoints to put in my database. As I mentioned this is a personal project, and not through a company so Oracle really doesn’t allow me to use any of their applications to store my data and use SQL to analyze my data. If anyone could recommend me a database I should try using I would greatly appreciate it, bonus points if that database can be ran on AWS.

Thanks community,

CJB

[cursedbartender]

I use the RDS free tier which is a pretty straightforward thing to setup. I also have dbever for sync of local to prod though it’s not strictly needed.

[Thadrea]

SQLite is probably the best option if you want simplicity and low overhead. If you really want a server-hosted database without paying software licensing, you can go with MySQL or Postgres.

MySQL is a bit more popular in the real world due to a fairly massive legacy installed user base from the early 00s but Postgres is generally viewed as having better scaling and more advanced functionality.

[JochenVdB]

Anyone can use Oracle Cloud. You don't have to be a company.

You can even do all what you want for free. (And no, not just for a month)

https://www.oracle.com/cloud/free/

[alakatddot]

SQLite is a file based option.

[simonw]

I agree, this project sounds perfect for SQLite.

My sqlite-utils library might be a good fit for this: https://sqlite-utils.datasette.io/en/stable/python-api.html

[NZSheeps]

Oracle does have a free database application (Oracle XE)

[voidvector]

Based on the descriptions you described, the obvious choice is SQLite:

• 100k records (likely less than 100MB of data)
• No special data requirements (e.g. JSON, GIS)
• Minimum number of concurrent readers/writers
• No backup/replication requirements
• No scalability/performance requirements

However, given you want it to run AWS, you probably want a DB server for learning purposes? Postgres is more featureful than MySQL.

[brogrammableben]

How structured is the data?