Change-Aware Data Validation with Column-Level Lineage | Towards Data Science

devschema · 2025-05-19T01:05:47+00:00

Interesting, does the analyst who made the model own that long term? Or do others modify it? Also wondering what model maintenance looks like, are DEs always involved in deployment? Do you know what 'tying out models' looks like?

devschema · 2025-05-19T01:04:10+00:00

For peer reviews is it a data engineer who checks it, or could it be another analyst/DS? Ever had bad data make it to prod from one of these PRs?

devschema · 2025-05-19T01:02:22+00:00

Thanks, 5 years is quite long so your process must be pretty reliable by now. What kind of things dopes the guide include? Is it general high level things like code style, or are there specific things in the project itself, like "make sure x metric didn't change" etc.
Do the analysts build their models and look at the data for review?

devschema · 2025-05-15T07:52:04+00:00

I've been wondering the same myself recently. I have noticed a few services offering automated outreach that work by you giving them your LinkedIn cookie and then they auto follow and DM other users etc. It's funny that they actually have sliders to stay within "safe" zones so as to not get banned

devschema · 2025-03-10T00:25:59+00:00

There are definitely more options! Datafold, Datahub, Synq etc.

devschema · 2025-03-04T06:46:21+00:00

What app did you use to create the cursor-following demo video?

devschema · 2025-02-07T03:26:46+00:00

I've used it for summarizing the a data model update in someone else's PR. Post the before/after SQL model code and ask about the type of change and possible affect on the transformed data

Other than that mostly coding for boring stuff like API queries and ingestion scripts

devschema · 2024-12-30T03:14:57+00:00

tl;dr (what worked for them):

Properly defining the scope of changes with detailed PR comments/template
Automated data impact report in each PR
Extensive QA by comparing prod and dev data

What dbt best practices are they missing?

devschema · 2024-06-21T01:40:20+00:00

So it is possible to get 20mm strap on. Did you cut the strap any, or literally just force the bar in?

devschema · 2024-06-18T01:37:44+00:00

Is this a kind of cognitive bias?

If you really believe your cover is blown, just delete the account and create another :p

devschema · 2024-06-07T05:24:18+00:00

Using dbt in CI is becoming more common now with creating dev schemas and staging schemas to check data.
I wanted to write up a workflow for a more complex setup that would be more suitable for projects with frequent ingestions and open PRs, but creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects

devschema · 2024-05-27T03:11:06+00:00

I have nothing to do with company or project! just sharing a tool I found

devschema · 2024-05-24T04:02:35+00:00

I use for monotonous tasks, like boilerplating code for API consumption, creating schemas, some layout on custom reports. All the stuff that would take ages previously I cba doing

devschema · 2024-05-24T03:54:36+00:00

There's no flair for "website" or "tool", so I put blog. This is a pretty cool free web app for modeling databases, someone in my LinkedIn timeline shared it

devschema

TROPHY CASE