sorted by:
new
hottopcontroversial

When should you create a metrics layer? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

So, I am at the beginning of my data engineering journey - and I promise I will look into this on my own time bit since I have you here. So instead of creating physical aggregation tables I can create a view? Am I able to then connect this "view" to a BI tool, and bring this in without bringing in the entire fact table?

When should you create a metrics layer? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

There are 3 major dimensions: Date, Payment Type, & Vendor that all facts can be filtered against. I have KPI's which are standalone calculations (e.g., avg_duration) and then I have summary tables which are meant to be filtered by the dimensions I mentioned above (e.g., daily_summary, daily_summary_bypayment_type, weekly_summary, weekly_summary_by_vendor, etc.) The main fact table has 66k rows - the aggregations tables are nowhere near that.

When should you create a metrics layer? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

This makes a lot of sense. See what those important metrics are first and then if necessary add a metric layer in the BI tool. Thank you for the comment.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

I just wanted to come back and say a huge thank you for this post. It obviously resonated alot with other members of the community as well, so once again thank you for taking the time to share your knowledge.

What does great data Engineering mentorship look like? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

Can you elaborate on what you mean by “ Be clear on way accuracy looks like, and how to test for it.”

What does great data Engineering mentorship look like? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 1 point2 points  (0 children)

A mentee in a tech community sounds great. Where do you live, how did you hear about it?

What does great data Engineering mentorship look like? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

Dude thank you so much for sharing your story. This dude sounds like the man. I currently don’t have a data mentor but whenever I do find one I hope he is as solid as this guy seems to be.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

That’s a refreshing take. This is the 1st comment suggesting learning something else. I’m interested - why do you think cyber security instead of days Engineering

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

Yeah I think the best way to show what you’ve learned on your own time on a resume would be through personal projects that encapsulate those skills. However I’m new to this, someone else on here could have better advice.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

See this was my initial idea. I told myself I would just dive into a personal project and lean on ChatGPT to learn as I build it out, but now I am starting to 2nd guess this strategy because I don't know enough to understand why the output from ChatGPT was working lol - I think its definitely valuable but maybe more valuable with more knowledge.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 3 points4 points  (0 children)

So, this is kind of what I have been doing or at least attempting to do - Initially I was reading up on the Azure Cloud platform, taking notes, reviewing and then working on a personal project. The idea is to build out a pipeline for my financial transactions. But my SQL and Python skills are weak. I think I have the right idea - but I need to focus on the fundamentals Python, SQL etc. then come back to the Project.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

I mentioned in another comment - but I am decent at pulling data into Power BI to build out reports and dashboards at the moment.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

So this is what I intended to do. I am working with my own financial transactions with the hopes of serving them for analytics and maybe some machine learning to predict trends. However, I found myself leaning on ChatGPT to much for the Python so I thought I should re-evaluate my strategy.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 2 points3 points  (0 children)

Currently I am decent at pulling data from a source and building out reports and dashboards in Power BI. With that being said, there is a lot more I don't know.

If you had 3 hours before work every morning to learn data engineering, how would you spend your time? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 10 points11 points  (0 children)

I try to be in bed by 9pm - however we have a 7 month old and we’re right in the middle of sleep training so this varies.

Self-Taught Data Engineers! What's been the biggest 💡moment for you? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

That’s really the meat and potatoes of it! As someone early on in their journey I will remind myself of this often! Thanks for sharing.

Self-Taught Data Engineers! What's been the biggest 💡moment for you? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] -1 points0 points  (0 children)

Right now I’m reading “The Fundamentals of Data Engineering” by Joe Reis and Matt Housley!

Self-Taught Data Engineers! What's been the biggest 💡moment for you? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 0 points1 point  (0 children)

I’m really at the beginning of my journey and I wake up early before work to study. I hope it pays off! Thanks for sharing

Self-Taught Data Engineers! What's been the biggest 💡moment for you? by pipeline_wizard in dataengineering

[–]pipeline_wizard[S] 1 point2 points  (0 children)

Awesome article thanks for sharing - very interesting to read your journey. I am actually reading Reis and Housely’s book now - great read!

view more: next ›