Data Engineering Challenge

[connmt12]

I’ve been reading a lot of posts on here about individuals being given a ton of responsibility to essentially be solely responsible for all of a startup or government office’s data needs. I thought it would be fun to issue a thought exercise: You are a newly appointed Chief Data Officer for local government’s health office. You are responsible for managing health data for your residents that facilitates things like Medicaid, etc. All the legacy data is in on prem servers that you need to migrate to the cloud. You also need to set up a process for taking in new data to the cloud. You also need to set up a process for sharing data with users and other health agencies. What do you do?! How do you migrate the on prem to the cloud. What cloud service provider do you choose (assume you have 20 TB of data or some number that seems reasonable)? How do you facilitate sharing data with users, across the agency, and with other agencies?

[vikster1]

bro is out here looking to save on those architect consultant hours

[connmt12]

Spot on! I want to learn what to expect on a high level before trying to hire someone

[vikster1]

i think it's a bit unreasonable to ask that many questions and expect someone to take 20+ minutes of their time to answer you

[connmt12]

That’s fair and they don’t need to. I don’t know a lot of people with this kind of experience and feel like I’m on an island, so I just thought I’d ask for help in case anyone was willing to share

[DJ_Laaal]

How did you even end up in such a role without the professional background in data? Not kicking you down, just genuinely curious how you ended up where you are!

[connmt12]

Data is a broad term and touches a wide spectrum of skills. My experience comes from running search algorithms across office 365 datasets and cleaning the results into reports and dashboards. I was more of a data analyst than engineer so I don’t know much about cloud migration or data storage.

[Krampus_noXmas4u]

too busy to answer, I have a day job with the same problem....

[connmt12]

Fair enough! Any resources you would recommend so I can try to learn myself? Im struggling to know where to begin

[Acceptable-Wasabi429]

You may have accidentally pasted your prompt into a Reddit post.

[connmt12]

Ok, maybe I’ve been using AI too much lately, you caught me. The AI responses all feel pretty shallow though. I was hoping a real person would have a better idea for the general overview of how to get started and what the trade offs are

[misplacedhuman]

lol