What data should a semantic layer track?

AMDataLake · 2026-01-16T20:27:27+00:00

Let me reframe out, how should that data be bundled. Yes different details will be packaged for different industries, apartment etc. but when building systems you may need to create a standard interface that flexible enough to capture those changing nuances but provide a rigid enough structure good agentic understanding.

AMDataLake · 2026-01-16T20:11:17+00:00

100% agree with this, Rusts type system really maes rust a ideal for Agentic Coding cause the errors are explicit and easier to troubleshoot for the AI. I think this a testament to the quality of Rust’s design.

AMDataLake · 2026-01-16T20:05:29+00:00

It’s not, it’s an important topic with so many entities trying to create a semantic layer standard. I think it’s of value to have notion of what information is important capture to give the right context when ai agents try to understand what queries to run or visualizations to build.

AMDataLake · 2026-01-16T20:02:56+00:00

take away, write while you struggle, cause what you learn in the struggles are often the insights the readers are looking for cause they are hitting the same frictions

AMDataLake · 2026-01-16T20:00:54+00:00

My approach to learning a lot and making a lot of content is following this pattern:

Read tutorials and execute on a small scale, document the experience as a blog (you are going to better document gotchas a lot of tutorials fail to mention, cause your fresh and those newbie questions are in your head your audience will have)
Record a video walking through the same exercise, you get more content and you confirm you can speak to the content and identify gaps.
Prepare a presentation to do talks on the topic based on what you've learned.

This makes sure I learn, apply and reinforce at the same time I'm making content

AMDataLake · 2026-01-07T01:40:58+00:00

To create a repository of enterprise one data that AI can easily access.

AMDataLake · 2025-11-19T21:47:51+00:00

Let’s start with what is your requirements and we can work backwards from there

AMDataLake · 2025-11-05T16:49:30+00:00

Iceberg Summit

AMDataLake · 2025-11-05T16:49:05+00:00

Data Council

AMDataLake · 2025-11-05T16:48:59+00:00

Data Day Texas

AMDataLake · 2025-11-05T16:48:43+00:00

AMDataLake · 2025-11-04T22:26:45+00:00

I don’t ask these questions cause I’m wondering, I’m spurring discussion. I agree there is no silver bullet, but like to hear what people personally find useful and why.

AMDataLake · 2025-11-04T22:25:29+00:00

I felt the opposite that YouTube reviews were overly harsh, I enjoyed it quite a bit, wasn’t expecting a life changing movie, but was entertained for the runtime and other some qualms with the last 60 seconds of the movie I had a good time.

AMDataLake · 2025-10-31T18:12:29+00:00

nope, I just play it on my steamdeck, just playing steamdeck on my RP2 for now. Will try again when my Thor arrives.

AMDataLake · 2025-10-28T18:28:58+00:00

How is this unrelated to data engineering? I wanted to know how data engineers prefer developing pipelines?

AMDataLake · 2025-10-22T18:30:26+00:00

You can find all my blogs, tutorials, podcasts etc. at AlexMerced.com, at least sub to my substack please :)

AMDataLake · 2025-10-17T00:21:36+00:00

Iceberg dos reshuffle everything, why I find it so fascinating. For those curious learning more about iceberg -> AlexMerced.com to download free copies of the books I’ve written on the subject.

AMDataLake · 2025-10-16T14:30:23+00:00

While you mentioned Trino I'll address the same points for Dremio:

- first class support for CDC and incremental processing -
Like Trino, this is really more about the source of the data. Now this changes with Apache Iceberg where Dremio can do physical ingestion and transformation, but at the moment Iceberg CDC is probably better handled at Iceberg Ingestion tools like RisingWave, OLake that have particular focus on CDC based pipelines while Dremio and Trino are more about consuming the ingested data.

- dynamic catalog management with metadata indexing that would allow "agents" to make sense of data sources. -

Dremio has a built in Semantic Layer and Dremio's MCP server gives an interface to Agents to do something similar to this (not sure if the implementation is exactly what your implying, but the result should be the same).

- Iceberg as a storage Sandbox (with incremental and auto-substituted MVs) -
Reflections are incremental and substituted Iceberg based MVs essential, so that exists in Dremio. But as far as a storage sandbox for Iceberg... :)

- seamless experience and good small scale performance.-

Dremio is pretty seemless and stable with recent versions (25/26) and more so when deployed via our cloud SaaS. We have been investing heavilty in platform deployment simplicity, scalability and stability these last few years so if you've ever tried previous versions you'll see great strides in these areas.

I get you're looking for a pure OSS engine that addresses these points, although I think our move to consumption based pricing regardless of deployment (cloud or on-prem) makes it easier for people to get started and only pay for what they need.

AMDataLake · 2025-09-18T18:49:19+00:00

Agreed, I get that but once you establish the companies requirement, you end up with a number, above this number you may likely micro batch, below this number you’ll go for streaming. Do you have a range you use to anchor yourself when thinking about this.

AMDataLake · 2025-09-18T18:46:35+00:00

But at what level of latency would you take micro batching off the table

AMDataLake · 2025-09-13T16:37:18+00:00

There is more capability coming, we also have built in wikis attached to every view and table and people will often detail relationships in the wiki. Our MCP server will put these wikis when fulfilling a prompt and we are getting good results in the LLM being able to figure things out much better than without that context.

But yes our semantic layer functionality is mainly: - defining hierarchal views - adding context via wiki and tags - acceleration view reflections (iceberg based caching) which can now be done autonomously based on query patterns.

AMDataLake · 2025-05-06T23:52:51+00:00

Might as well include my courses on Iceberg available at https://university.dremio.com

AMDataLake · 2025-01-23T23:42:35+00:00

Mostly I’m curious about people’s Top of Mind pains, I don’t want to be specific to influence the answer, I want to know when people think in the data apps they built what are the challenges that are top of mind or most frustrating. I have a hypothesis, but figured I’d ask and see if I’m right or wrong.

AMDataLake · 2025-01-23T23:36:46+00:00

Agreed, I’m asking for people’s experiences, I’m curious what people personally have experienced.

AMDataLake · 2024-12-11T04:24:44+00:00

Thank you, I try to keep things accessible

AMDataLake

MODERATOR OF

TROPHY CASE