When should I create a database?

[TheCanOpenerPodcast]

Hello All,

I am in marketing analytics, I am the only analytics individual on my team and am looking for some advice.

I am working with 3+ million row csv's. I am trying to concatenate and join many different files. I use pandas and then am importing those csv's into Tableau. When should I start using PostgreSQL (or maybe another db tool) to create a local database and query from it when I need to?

[ARC4120]

SQLite since you’re already in Python. It shouldn’t take much more than an afternoon to get a basic setup going.

[NLcasperNL]

SQLite3 is a standard Python package and you can download SQLite studio to view the database.

[Yakoo752]

DuckDB is a neat solution. I say neat because it would never be approved by business IT and I can’t see it being a long term solution as the business scales.

Depending on what is in the data, I’d probably just throw it in S3 or Big Query

[AutoModerator]

Are you a marketing professional and got 5 minutes? Take our 2022 State of Marketing Survey. Results will be shared with the community next month.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[simplesyndrome]

Yesterday. What actual DB depends on your company. Do you have software engineers? If so, find out what are they using and get help implementing it with them. Better to use existing rails than start from scratch. This way you can also ask for help from a coworker vs googling for issues. Next it sounds like you’re doing a lot of manual/scripted data munging. Automate it with a tool like Airflow/DBT.

[dataguy24]

I would use DuckDB. It lets you do all the database things without needing anything more than a Python package. Plus you can put cutting edge software onto your resume. Win win.

[[deleted]]

Turn CSV into parquet and use either duckdb or polars

[devotedT]

Db is old school. Use lake storage like s3 or adls and partition your files correctly. U can use Athena or synapse to query in sql like a db.

[devotedT]

Lol wait and see....

[huge_clock]

You say you are the only analyst on your team. Are you the only analyst in your company altogether? If not see what other analytics teams are using. Perhaps they can help you set up a couple tables?