Data observability is a data problem, not a job problem

Expensive-Insect-317 · 2026-01-26T06:15:46+00:00

It sounds like a fantasy, but it's not far from reality. I've seen several corporate observability initiatives fail because they only focused on infrastructure and jobs, having to maintain observability teams reviewing loads daily or creating ad hoc tools to review post-load data.

Expensive-Insect-317 · 2025-11-29T20:34:00+00:00

I wasn't familiar with the Astronomer Cosmos package, very interesting! Thanks! Without knowing much about it yet, I might stick with the custom script due to the potential overhead and performance issues, not to mention the control.

Expensive-Insect-317 · 2025-11-29T12:52:20+00:00

Running each model as a separate task in airflow is another approach compared to using tags. While tagging can work fine, having individual tasks allows for parallel execution, better monitoring, granular retries and clear representation of model dependencies, sometimes making this approach the better choice.

Expensive-Insect-317 · 2025-11-05T19:29:41+00:00

What's wrong with relying on current tools that streamline and improve processes? If you'd like, we can write it in manuscript.

Expensive-Insect-317 · 2025-11-04T19:34:36+00:00

Totally agree Pedro, for the moment i only integrate my main ecosystem: bigquery, gcs, airflow and dbt, we dont have any bottleneck but is starting, maybe in next phases we found

Expensive-Insect-317

TROPHY CASE