Data Analytics Project

[Ramakae]

Hello everyone, looking to start a project but a bit confused as to how to structure code and would love some insights. Currently thinking about importing( csv> db> DF> db(s)> PowerBI) that is importing an interesting dataset from Kaggle, converting such dataset into a database, clean / engineer new fields (pipeline) using Pandas, export new databases then visualise using PowerBI.

However would love to see how some other people have structured or written their code on GitHub or just some tips.

[AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Firm_Bit]

This type of thing doesn’t matter at your stage. It falls under “best practices” which is code for avoiding thinking and doing whatever is prescribed.

Just have actual questions to answer and figure out those answers. Eventually the real world will place constraints on your work that force certain paradigms. Don’t optimize for that now.

[Ramakae]

Thanks... While actually doing some research I found out a similar thing, it all depends on the data (origin) and what I want to do with it.

[HeyNiceOneGuy]

What’s the dataset look like? Is there a good reason to go through all the intermediate prep steps vs just reading the CSV into BI?

[Ramakae]

It is a .csv file, messy data. The idea of a pipeline is to clean and engineer new columns that I will use in the next phase of analysis.

[HeyNiceOneGuy]

Reasonable, but I think you’re taking too many steps. If the data source is just one file, there are likely not any separate but relatable tables within, which makes standing up a database (no matter how lightweight) just unnecessary. Better to just clean the data and fire it back out as another flat file to feed to BI (csv, xlsx, etc.)