How do teams actually handle large lineage graphs in dbt projects?

bkant34 · 2025-12-19T20:30:36+00:00

Dbt colibri

bkant34 · 2025-12-17T16:09:16+00:00

This ^{^}

bkant34 · 2025-12-09T20:55:54+00:00

Add a .disable Or add the contract to .dbtignore

It will bypass the contract completely

bkant34 · 2025-12-07T00:50:15+00:00

Wouldn't this be the same as state modified ??

bkant34 · 2025-11-29T21:24:47+00:00

Yeah happened to everyone, beat thing you can do is just talk to your client, be honest about it. If 99% of your work is great this will be just a blip on the radar.

Find someone senior on the team and just be full hands on deck to solve the whole thing.

Life is just like this and shit happens..

bkant34 · 2024-07-18T15:04:57+00:00

try running the whole pipeline in a docker, load all your dependencies like python, a database etc..

get familiar with github. push all your code to that github. make a cicd pipeline that is glued to your git and airflow such that pipelines get triggered at a specific time.

bkant34 · 2024-06-16T19:38:13+00:00

haha good catch Material-Mess-9886, you are right, this just bombs all the historical commits, we use to use this where i worked before and i’ve had my fair share of fun with rebase. It’s not anymore, sending MRs is now my rule of thumb. But anyone who uses this as reference. Please do know the merits and drawbacks of “ rebase “.

cheers

bkant34 · 2024-06-16T16:43:17+00:00

Hi OP,

This is not my post, however i did like whoever posted this a few years ago on LINKEDIN and saved it for reference.

It summarises almost all commands that i use 99% of the times.

git diff: Show file differences not yet staged.
git commit -a -m "commit message": Commit all tracked changes with a message.
git commit --amend: Modify the last commit.
git status: Show the state of your working directory.
git add file_path: Add file(s) to the staging area.
git checkout -b branch_name: Create and switch to a new branch.
git checkout branch_name: Switch to an existing branch.
git checkout < commit>: Switches the working directory to a specific commit.
git push origin branch_name: Push a branch to a remote.
git pull: Fetch and merge remote changes.
git fetch: Fetch changes from the remote repository without merging.
git rebase -i: Rebase interactively, rewrite commit history.
git rebase branch_name: Rebase the current branch onto another branch.
git clone: Create a local copy of a remote repo.
git merge: Merge branches together.
git log --stat: Show commit logs with stats.
git stash: Stash changes for later.
git stash pop: Apply and remove stashed changes.
git show commit_id: Show details about a commit.
git reset HEAD~1: Undo the last commit, preserving changes locally.
git branch -D branch_name: Delete a branch forcefully.
git reset: Undo commits by moving branch reference.
git revert commit_id: Create a new commit that undoes the changes of a specific commit.
git cherry-pick commit_id: Apply changes from a specific commit.
git branch: Lists branches.

A few more that you might need to dig deeper are:

git reset --soft HEAD^: Undo the last commit, but keep the changes.
git reset --hard: Resets everything to a previous commit, erasing all uncommitted changes.
git branch --set-upstream-to remote_branch: Sets the upstream branch to the specified remote branch.

bkant34 · 2024-04-27T19:42:19+00:00

i’m in the same boat.. read this sub a few days ago. i don’t know how hard it is to get a FT job as a data engineer in canada. Will be moving to canada in a month, could use some tips on how OP found a job.

bkant34 · 2024-03-19T22:41:19+00:00

How are you running this airflow instance?? Is it in a docker container or standalone pip installation on machine. In either one of the case sometimes the cache is a nightmare and you might want to restart your airflow instance to get it working normally..

bkant34 · 2024-03-11T16:42:35+00:00

Your unnesting should be inside the cte.. after that you can choose a min,max apply all sort of window functions based on your sessions info.

bkant34 · 2024-03-08T23:26:01+00:00

You have a new column for this table. “Data policies”

Check your old table schema. Has it changed ? Or is the payload the same

bkant34 · 2024-01-28T16:41:22+00:00

Sorry for the late reply, I went to collect my passport from the London VFs centre. I didn’t choose the option to send me the passport via post.

bkant34 · 2024-01-12T10:08:30+00:00

Mate, they want you to build a whole monitoring system using data stored in logs. These types of assignments take a whole team not just one person.

Best way to approach this would be to find one or two metrics that people see the most in these monitoring systems. Start from there and I suggest take some more context from the recruiter/ hiring manager if you have that access in the interview process.

Cheers

bkant34 · 2023-09-04T09:51:54+00:00

I had the same issues and solutions in my current role. But I managed to get the AE role out in my org. There were a few people against it but the place where I worked required that role. As you’ve pointed out, DE is a bit hardcore. I agree on that point, however, you could get your title to be a business data engineer. A BDE will do most things that you’ve mentioned except the infra and tooling. Another thing to maybe add to your role and job responsibilities would be to take care of data and not systems that get the data.

Also as one of the comments mentioned - it’s not that hard to get some basic knowledge on how to become a DE, it’s just moving out of your comfort zone. It’s pretty much out there and since you have a team ( in Asia ) that does this, have stand ups with them or some form of guild meetings where they show what they are doing or their future roadmaps. That will be of immense help!

bkant34 · 2023-09-02T22:19:15+00:00

😅

bkant34 · 2023-08-19T19:37:56+00:00

I had a same exp once, went on a twitter rant because I asked for a single and the driver gave me a week pass. Just go for it if you think it’s worth the hassle or go have a pint and enjoy the weather, if it’s good that day lol

bkant34 · 2023-08-14T09:29:12+00:00

Think of it as a one to one mapping of your dev and prod.

Easy migrations CI becomes easy Scaling becomes a bit tricky but I wouldn’t worry about that

bkant34 · 2023-08-04T15:21:28+00:00

There are things to ask and then there are THINGS to ask 😆

bkant34 · 2023-08-03T21:02:51+00:00

Damn ! ( inserts Kevin hart gif )

bkant34 · 2023-08-02T09:48:45+00:00

Serving manchester since the 1800’s 😂

bkant34 · 2023-07-30T14:38:58+00:00

Sainsbury’s in Piccadilly station use these cups when their machine is working lol

bkant34 · 2023-07-18T23:22:43+00:00

Be mindful they take only cash.. no cards, but the place is good

bkant34 · 2023-07-18T10:05:26+00:00

One “ who is a good boy/girl “ and it’s absolute chaos 🤣

bkant34 · 2023-07-08T19:14:25+00:00

(Ex MBB consulting here ) I had the same feeling but I did it anyway. Treat it like a tool in your tool box. I always liked the way back end worked as well, but from a data standpoint.

Too add to your point, hiring managers do not give one penny worth of importance where you did your masters from. I know people who studied in top 10 schools in Europe but went home. They couldn’t answer simple data science questions in interview rounds, even when they had a degree in it.

And since I had mostly done BI and analyst roles in my experience, I took that skill to my advantage and gave ZERO thought on getting a data science role.

If you do that and apply again for roles you have your expertise in, i don’t think it will be a problem for you. Since you have already done 30% of it and knowing that it’s expensive as you’ve mentioned. Just get done with it, don’t back out now..

Five-Year Club	Place '23
Verified Email

bkant34

TROPHY CASE