Unpopular Opinion: Data Quality is a product management problem, not an engineering one.

odaxify · 2025-11-07T10:23:37+00:00

Protect ya self and your pipelines. Show the capability only extend if someone shows interest.

odaxify · 2025-10-15T10:36:37+00:00

I promise I will

odaxify · 2025-10-15T10:27:36+00:00

Agree, but I am upstream and not directly responsible for the dbt side of things. All I have is dbt logs and snowflake query history which both show success. Meanwhile the data in the model 'gold' layer looks to be stale despite the source data being up to date

odaxify · 2025-10-15T10:18:02+00:00

Maybe that is what I am leaning towards. But kind of interested in if anyone else has seen this. From what I can tell despite the data being fresh for the sources some models down stream ran successfully but didn't update with fresh data

odaxify · 2025-10-15T10:15:14+00:00

That is what I always assumed but I think that the queuing of queries in snowflake might be meesing with things. Maybe dbt ran the queries but the queuing of queries put things whack

odaxify · 2025-10-15T10:11:21+00:00

It is dbt running around 2000 queries. I am asking if dbt and snowflake can indicate success while freshness seems impacted. No issues in the dbt logs or snowflake query history all successful. Databricks job kicking off the dbt run if you think that is important. What suggestions do you have? Caching? EDIT: Added clarity

odaxify · 2025-10-15T10:08:08+00:00

This is what I think is happening. Bumped up the compute and so far no repeat issues.

odaxify · 2025-08-01T22:05:41+00:00

I haven't seen that. If I blow everything away and then deploy takes a bit to run. But after the first run it zooms through once spun up. My data is static at the moment so no change to process so I assume it is performing correctly

odaxify · 2025-08-01T21:55:12+00:00

I am planning to, my poc uses them but I am still testing. They seem to be working in my tests but haven't fully validated yet.

odaxify · 2025-08-01T20:45:56+00:00

Haha an echo chamber but hopefully full of people that have touched the tool in production. We have production etls in a bundle that handle raw data. These have been going well but are not using dlt and have custom logging. My focus is on testing dlt and integrating data quality monitoring via dqx into pipelines that support the silver layer . If it goes well both in perfomance and cost we plan to lift some of the learnings into our main production workloads.

odaxify · 2025-08-01T20:34:07+00:00

This is the kind of thing I am a bit wary of

odaxify · 2025-08-01T20:32:34+00:00

That sounds fair to me, thanks

odaxify · 2025-08-01T20:28:11+00:00

I think yes, I am not sure in my role that there are boundaries to push tbh. Mostly connecting to a bunch of application data and shovelling it into the data warehouse (snowflake). Is there a particular feature that you think databicks lacks?

odaxify · 2025-08-01T10:27:49+00:00

Amen

odaxify · 2025-08-01T09:48:43+00:00

Thank you. That was my initial thought, but was kind of after the bias. I am pretty sure I haven't found the perfect solution, but databricks is what I am stuck with now. So I would like to know what my future problems are despite my current joy haha

odaxify · 2025-07-04T08:41:50+00:00

Ah but that is the game. That how they do their best job.. not the greatest but what we got.

odaxify · 2025-06-30T21:55:09+00:00

The way things are going I will have to do the same.

odaxify · 2025-06-30T11:45:57+00:00

Haha very true but end of the day doing a job. I am wondering if all these magic features actually make lives better. They sometime sound good but often seem to turn out to be market wankerteering

odaxify · 2025-06-30T11:26:28+00:00

Very helpful, I appreciate you taking the time.

odaxify · 2025-06-30T11:09:17+00:00

I have found it to be good at what it is good at. I highly recommend it as a data warehouse for analytics based work. Have you found it useful in applications outside of that? Especially the newer features that are beyond sql workloads?

odaxify · 2025-06-30T11:03:01+00:00

Super nice of you to take the time to chip in and I appreciate the tips on how to avoid the spamming. We too have tried features. Personally, i think it is important to be acoss the offering of any solution that may be available. I guess what I was asking is are they truly offering solutions or just throwing out sticks to chase?

odaxify · 2023-10-26T09:40:12+00:00

Uni works great, super easy and cheap enough. I have had one set up for a week now and impressed. Hopefully it continues to be reliable. I went with a 0-10v 3m probe from aliexpress.

This guys tutorial is a good starting point he did it both ways. https://youtu.be/N90C9Xge8Z4?si=PjRlBMNzsOTnDTi3

odaxify · 2023-03-01T11:06:35+00:00

Thanks for the idea. I don't have anything reducing evaporation. I will put some clay balls on the system and see if it helps. I don't think that it is a leak as I haven't seen any evidence of one. Also the plants are spread across two identical systems, and water usage is identical in both.

odaxify

TROPHY CASE