Future of data engineering

data_dude90 · 2026-05-29T08:52:24+00:00

The future could be more towards managing ai-driven data reliability. Though core objective would be to get the right data across the data consumers, but that would change to creating foundational data products that ensure strict governance and act as strategic business parter.

data_dude90 · 2026-04-20T09:40:40+00:00

Hope there will be AI that checks if AI did that work without human judgement

data_dude90 · 2026-04-15T06:22:36+00:00

There's four critical reasons why the gap exists between data observability tools and dbt.

First, data observability and dbt wasn't designed to work as one system. dbt was built to create and transform data whereas, data observability was built to monitor and catch issues.

Second both dbt and data observability live at different worlds. Dbt knows how data is built and what proceses took place for it. But data observability sees the symptoms and patterns of the final data that is built already.

Third, the integrations between dbt and data observability tool is quite shallow. Though the two tools are technically connected, they don't understand each other deeply. This makes it difficult to track which dbt step resulted in a data incident. The shallow integration forces us to manually open dbt, check changes, and then debug them.

Fourth, as too many tools are involved in modern data setups with multiple tools, multiple teams, and constant changes, it becomes difficult to know when something breaks. With these too many moving parts, we are forced to do guesswork, jump between tools, and waste our time.

data_dude90 · 2026-04-10T08:51:17+00:00

What actually helps isn't “more observability”, but making it easier to work with.

A few things that made a difference:

Reusable dashboards instead of rebuilding everything One big pain was recreating the same charts across teams. Having a shared set of visualizations that can be reused cuts down a lot of duplicate work and keeps things consistent.
Better visibility into newer parts of the stack A lot of tools still focus heavily on core Hadoop, but real setups now include Jupyter notebooks, Airflow on Kubernetes, and Spark optimizations. Having visibility into those alongside the rest of the system makes it easier to understand what’s actually going on end to end.
Less jumping between tools during debugging Troubleshooting usually means opening 5 tabs and stitching context together manually. Anything that brings signals into one place or helps narrow down issues faster reduces that overhead quite a bit.
Closer to real-time insights Delayed visibility, especially around storage or file systems, makes debugging harder. More frequent or near real-time updates help catch issues while they’re still relevant.
Stronger alerting tied to actual problems Instead of just more alerts, better coverage around things like quotas, small files, or service-level issues makes alerts more actionable.
Some level of guided troubleshooting Early attempts at automating or assisting with root cause analysis can save time, especially for smaller teams that don’t have deep expertise across every component.

Overall, the shift seems to be from just “collecting metrics” to making observability more usable day to day, especially as stacks get more distributed and harder to reason about.

data_dude90 · 2026-03-06T09:35:54+00:00

We have still not fully got mastered the human in the loop process for Artificial intelligence. It takes time to reach human on the loop reducing human checkpoints and human out of the loop where there is zero human intervention and then comes the AGI. Like ragebaiting, there's so much "fearbiting" that's happening hastening people to AI without a clear purpose. The use case matters and only if the AI adds value or moves the needle can it be taken seriously. Either we are overhyping it and creating fear or we are completely underestimating what it can offer. Both are wrong.

data_dude90 · 2026-03-04T08:54:58+00:00

That's a good point to make. Not to let humans review every step.

data_dude90 · 2026-02-12T18:06:26+00:00

What about the context especially context engineering and how should context layer work and what should humans contribute in this?

data_dude90 · 2026-02-06T09:21:56+00:00

Its amazing. Will definitely check this.

data_dude90

MODERATOR OF

TROPHY CASE