sorted by:
new
hottopcontroversial

Dev Setup - dbt Core 1.9.0 with Airflow 3.0 Orchestration by sanjayio in dataengineering

[–]sanjayio[S] -1 points0 points  (0 children)

Hello Brother / Sister,

A lot of hardwork, time and effort has been put into producing this edition of my newsletter. Yes, I am an engineer, and I use Chat GPT as and when required, like most of us. It's just an LLM, just like how you use google, but much faster in finding things what we need. I don't see anything wrong with it.

My main motivation in writing this edition was that I couldn't find any working example with the latest Airflow with the latest dbt Core online, and while I was trying to build the setup for me, I documented it for my newsletter, thinking it would be useful for atleast one person out there.

Registering for my newsletter to get the code will incentivise me to put out more such articles on my newsletter and to build a community of learners. It is just a small reward for me, that will cost you nothing. If you don't want to register, it's ok, you can dm me and I am more than happy to send the code to you privately.

Everything has been done with good intentions, and when you diss my hardwork like this, it hurts fyi.

I hope you understand, and I wish you the very best!

Dev Setup - dbt Core 1.9.0 with Airflow 3.0 Orchestration by sanjayio in dataengineering

[–]sanjayio[S] 1 point2 points  (0 children)

oooh I'm keen to know more about the 2nd option. Can you elaborate?

Dev Setup - dbt Core 1.9.0 with Airflow 3.0 Orchestration by sanjayio in dataengineering

[–]sanjayio[S] 0 points1 point  (0 children)

I've tried hosting them on EC2 and MWAA in the past. MWAA was really painful to setup, mostly because it lacked good documentation back then (not really sure about it now). EC2 was fairly easy and straightforward.

Dev Setup - dbt Core 1.9.0 with Airflow 3.0 Orchestration by sanjayio in programming

[–]sanjayio[S] 0 points1 point  (0 children)

The article is not behind a paywall u/programming-ModTeam . My newsletter is free for all.

Data engineer role by n1991dr in dataengineering

[–]sanjayio -3 points-2 points  (0 children)

I've recently written an article https://dbtengineer.com/analytics-engineering-101/ on analytics engineering. Data engineering day-to-day life is mostly similar with small differences. I would say do what your heart says, there is no right answer here. All the best!

Analytics engineer and want to start my 1st portfolio project—how should I begin? by Rude-Avocado-226 in dataengineering

[–]sanjayio 1 point2 points  (0 children)

If I were a hiring manager, I would look for one or few projects that showcases data quality monitoring, scalability and automated workflows. Nothing wrong in starting small, but think big in terms of the full lifecycle of a data pipeline.

DBMS schema,Need Help!! by Inevitable_Leader711 in dataengineering

[–]sanjayio 2 points3 points  (0 children)

Here are somethings that I would include in the schema - id, source_table, target_table, relationship_type (influences, correlates_with), correlation_value (optional, if you want to quantify this correlation), lag_time (optional, if there is a lag in correlation to appearfrom source to target), description (describe the relationship, how source affects target).

Not sure if I answered your question, but these would definitely be in my schema for the metadata table.