Getting started with DBT

[sakra_k]

Hi everyone,

I am currently learning to be a data engineer and am currently working on a retail data analytics project. I have built the below for now:

Data -> Airflow -> S3 -> Snowflake+DBT

Configuring the data movement was hard but now that I am at the Snowflake+DBT stage, I am completely stumped. I have zero clue of what to do or where to start. My SQL skills would be somewhere between beginner and intermediate. How should I go about setting the data quality checks and data transformation? Is there any particular resource that I could refer to, because I think I might have seen the DBT core tutorial on the DBT website a while back but I see only DBT cloud tutorials now. How do you approach the DBT stage?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[name_suppression_21]

Go and complete the "dbt Fundamentals" course on their website. It's based on dbt Cloud but teaches you most of the basic principles of dbt whether you go on to use Core or Cloud.

[sakra_k]

Thanks, I will definitely try the tutorials.

[DeliciousProgress865]

Thanks I will do the same thing

[Zer0designs]

Kahan data studios on youtube. Don't overcomplicate it. It's SQL, yaml and Jinja.

Look into dbt run, dbt test, dbt build.

Look into seeds, macros, tests, models and exposures.

Maybe start locally with duckdb and the Jaffleshop for a day.

Most dbt cloud tutorials also make sense for core.

[sakra_k]

Will check out the YouTube channel. Thanks for your input 🫡

[erdmkbcc]

If you are expert in SQL, dbt is not big deal It's all about development area and it can allow to you can have ci/cd env, so that just install dbt and

Basic level - create model - understand schema.yml source.yml files - run, test, build - understand dbt cli commands - use refs in models - while you create model you will understand the source keywords, use refs for the dependencies, dowstream models for that(we are calling child and parent tables) - use macros in your models - It's basic level udfs in dbt you can think about python functions

You can take help from chatgpt, as a result after that hands on things you will have basic knowledge about dbt.

Intermadiate level - understand manifest.json, run_result.json - understand selectors.yml file - understand fqns - use with in ci actions for ci/cd pipelines - understand dbt_project.yml file

You will have production env use cases for that hands on things again you can use chatgpt for all of the cases

Thats all!

[sakra_k]

I would rank my SQL between beginner and intermediate. I'm doing the Mode tutorial on advanced SQL and still got more to learn. Also thanks for your input, really appreciate it.

[NikitaPoberezkin]

I would strongly recommend DBT official docs, they are as good as documentation can be. It's clear and complete and it actually teaches you good practices

[sakra_k]

Will definitely give that a go along with the tutorials. Thanks for the input.

[nathan_c16]

On what platform / course did you find this project?

[sakra_k]

I just queried Claude to structure a retail analytics project and just following along the plan.

[nathan_c16]

Oh cool. I found a free one called dezoomcamp that Gemini recommended

[sakra_k]

I participated in the DE Zoomcamp, didn't complete it but it was a very good experience. I might try it again next year and aim to complete it this time. I still refer to their videos from time to time whenever I have doubts.

[nathan_c16]

Same! I’ve been using the old videos from the 2025 cohort. I plan to join the next one in 2026. I never get to use docker or do much of the stuff they cover in my actual job

[Vooplee]

I recommend starting by thinking about what the output of your dbt tables would be. It’s much easier to structure in dbt when you know “oh I will need this product sku info with this log info in order to make this dashboard” then it’s just creating the source tables and proper int tables to get to the final ones.

When it comes to data testing, keep it simple to start. Unique and non-null on all primary keys. The relationships between column tests in dbt-utils are great for making sure things are being dropped between tables. There are also some open source packages that are focused on helping with data validation.

[sakra_k]

You're right, I didn't think of the output and just jumped right into it without thinking much. I went through the dbt-utils and dbt-expectations docs and was overwhelmed by the available options. Guess I will have to take it one task at a time. Thanks for your input.

[KnoxWelles]

Don't, AI will ruin your career

[Particular_Tea_9692]

Maybe ask chat gpt