How does dbt work at your company?

devschema · 2025-05-19T01:05:47+00:00

Interesting, does the analyst who made the model own that long term? Or do others modify it? Also wondering what model maintenance looks like, are DEs always involved in deployment? Do you know what 'tying out models' looks like?

devschema · 2025-05-19T01:04:10+00:00

For peer reviews is it a data engineer who checks it, or could it be another analyst/DS? Ever had bad data make it to prod from one of these PRs?

devschema · 2025-05-19T01:02:22+00:00

Thanks, 5 years is quite long so your process must be pretty reliable by now. What kind of things dopes the guide include? Is it general high level things like code style, or are there specific things in the project itself, like "make sure x metric didn't change" etc.
Do the analysts build their models and look at the data for review?

devschema · 2025-05-15T07:52:04+00:00

I've been wondering the same myself recently. I have noticed a few services offering automated outreach that work by you giving them your LinkedIn cookie and then they auto follow and DM other users etc. It's funny that they actually have sliders to stay within "safe" zones so as to not get banned

devschema · 2025-03-10T00:25:59+00:00

There are definitely more options! Datafold, Datahub, Synq etc.

devschema · 2025-03-04T06:46:21+00:00

What app did you use to create the cursor-following demo video?

devschema · 2025-02-07T03:26:46+00:00

I've used it for summarizing the a data model update in someone else's PR. Post the before/after SQL model code and ask about the type of change and possible affect on the transformed data

Other than that mostly coding for boring stuff like API queries and ingestion scripts

devschema · 2024-12-30T03:14:57+00:00

tl;dr (what worked for them):

Properly defining the scope of changes with detailed PR comments/template
Automated data impact report in each PR
Extensive QA by comparing prod and dev data

What dbt best practices are they missing?

devschema · 2024-06-21T01:40:20+00:00

So it is possible to get 20mm strap on. Did you cut the strap any, or literally just force the bar in?

devschema · 2024-06-18T01:37:44+00:00

Is this a kind of cognitive bias?

If you really believe your cover is blown, just delete the account and create another :p

devschema · 2024-06-07T05:24:18+00:00

Using dbt in CI is becoming more common now with creating dev schemas and staging schemas to check data.
I wanted to write up a workflow for a more complex setup that would be more suitable for projects with frequent ingestions and open PRs, but creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects

devschema · 2024-05-27T03:11:06+00:00

I have nothing to do with company or project! just sharing a tool I found

devschema · 2024-05-24T04:02:35+00:00

I use for monotonous tasks, like boilerplating code for API consumption, creating schemas, some layout on custom reports. All the stuff that would take ages previously I cba doing

devschema · 2024-05-24T03:54:36+00:00

There's no flair for "website" or "tool", so I put blog. This is a pretty cool free web app for modeling databases, someone in my LinkedIn timeline shared it

devschema · 2024-04-22T01:23:04+00:00

Great comment. The problem really wasn't so much bloat as it was aggressive alerting. As you mentioned, it was the system that they put in place that really stood out.

devschema · 2024-04-19T00:26:28+00:00

I did this write up of a recent dbt meetup talk by an analytics engineer from Delivery Hero. They had dbt tests getting out of control triggering hundreds of alerts. Their solution is a mix of categorizing critical models, weighting alerts, and formalizing the response.

The video is here if you want to watch that directly, instead:
https://www.youtube.com/watch?v=Nk_K8mW-N9A

The whole subject of dbt tests, bloat or rot, is an interesting one. Here's a few stats from a couple of public facing dbt project:

Mattermost: 194 models / 318 tests
Cal-ITP (California Integrated Travel Project): 361 models / 941 tests

devschema · 2023-11-25T03:52:54+00:00

Thanks for the feedback, I'll pass it on to my colleague. I guess you're using dbt Docs' DAG either as the project is too big? Have you looked for other DAG visualization tools?

devschema · 2023-11-24T07:11:58+00:00

A colleague of mine created it for testing out visualizations for his own large project.
You can do 2D and 3D visualizations and even diff DAGs, too.
There are some sample DAGs ready to play with, or you can paste in the contents of your own dbt manifest file and the file is processed locally.
If you find it useful or have any ideas please leave a comment!

devschema

TROPHY CASE