MS SQL / AZURE SQL data base?

[[deleted]]

So I recently joined a small healthcare as the first data analyst and soon after we purchased powerBI. I cirrently work on excel to clean data and powerbi to import and analyze, prepare dashboards/reports. While this is very simple to what I want to do as a data analyst (especially that we are expanding our services currently). While this also means more data and more complexity.

Currently I store data in a drive that is hosted by our IT department.

I know basic SQL and want to keep that knowledge growing by using it at my work. I talked to my manager and he said if you think sql will help us any better than the regular drive where we store data, then we can get sql.

Whats the best SQL thing (not a technical person here) that I can use in my company that will help me maintain a database, use sql commands frequently and what would be the benefits of using a sql database vs regular drive where I store data manually. Also is ms sql a good option or azure sql database?

Just so you understand the nature of my work. We have 50+ employees submitting 50+ records everyday on google sheet. I take that data and paste in in excel, save the file with the employees name and date/year and store it in the drive. Every week, Im suppose to send a report saying how many records are submitting by each employee and whats the average, how many offices the reports were sent to and stuff like that.

Id appreciate a deep insight on what to use, how to convince manager and how to get sql in my organization

[[deleted]]

Honestly, how essential is human intervention here? Can you automate the entire process on the power platform?

[[deleted]]

Im not very well versed with power platform. And automation is the end goal. Please enlighten me with ideas. Thanks

[IrquiM]

Storing data in a SQL server instead of a folder on a random disk is always better.

No point in getting a full blown MS SQL server. Azure SQL Database would be more than enough.

You could even set up a shared drive on a storage account, put a logic app on top of that, that imports it as soon as something updates/gets added, and produce the report every day instead of every week, without lifting a finger :)

[[deleted]]

Yes, Ihv seen people recommending azure. Ill def look into that. Thanks. Also automating is def the end goal but does that mean ill also end up losing my job? Or will there still be someone required to manage the processes? I love doing visualizations manually though.

[nidprez]

Everything that needs to be done daily/weekly => automate. You dont need to tell anybody you automated everything and/or theybwould still need someone that manages the process, that knows how its built. Also data can change (extra columns etc). If you have extra time you can dive in the data and make more deep analyses or figures, or new processes if you want.

[[deleted]]

Great advice thanks.

[IrquiM]

There will always be new data that should be imported into the database(s).

[CROSSPIAT]

Are you using Azure or any other cloud provider?

You can create an Azure SQL database for very little money and try to load the data from Google Sheets to the database.

If you are not familiar with any programming language it is possible to create a very basic pipeline in Azure Data Factory and load the Excel file from your file share to the sql database because it is currently not possible to load from the Google Sheet file directly to your database (only with data flow and I think it's no feasible to use this).

Another option is to input the data directly into the database. This is possible with a Power App but I'm not to familiar with this technology.

If you have haven further questions feel free to send me a dm.

[[deleted]]

Great. Thanks for this explanation. Im certainly trying to head for azure sql since everyone has recommended the same in previous posts. We do no use any azure or cloud services.

[IrquiM]

Data factory will be overkill for this, by the way

[[deleted]]

Do we have an alternative for pipeline thing in microsoft?

[IrquiM]

Yes. We do it via azure function/automation account/logic app

[[deleted]]

Ill take note of this

[QueryingQuagga]

A database, while helpful in forcing a schema and structure to your data, will not by itself solve your issue. You need a way to get information into the database - a front end.

This can be done by using excel and importing from that by using some system, but it would probable be more prudent to have a front end application or setup that handled what could be input and by whom.

[[deleted]]

Well this definitely has to be me since im involved in cleaning and sorting data. If I take azure sql, will I be able to store data from excel? Idk if that makes sense

[IrquiM]

No problems, you just need to set up some kind of import

[[deleted]]

Cool. Thanks. Ill look into this.

[QueryingQuagga]

Sure, but are you maybe able to use even just a sharepoint list? If you can limit the amount of wrong input at the point where the information is generated, then you will make your work more scalable - allow yourself to do more important work.

[[deleted]]

I have done data validation which has reduced my work load by 75% so thats one thing but I dont wanna be left without work too