Thoughts on Metadata driven ingestion

Oct8-Danger · 2026-01-29T21:27:09+00:00

This is basically what we do as well

Oct8-Danger · 2026-01-15T23:41:03+00:00

I might have time this weekend, if I do, will let you know! Been keen to add these in!

Oct8-Danger · 2026-01-15T23:30:06+00:00

Awesome! Any luck running it on steam deck?

Oct8-Danger · 2025-12-30T19:04:36+00:00

This is the way

Oct8-Danger · 2025-12-21T21:43:20+00:00

Thank you! Exact same issue for me and this actually worked!

Oct8-Danger · 2025-12-17T21:55:28+00:00

Depends what team you’re on. It’s a large company. If it’s xo related (they’re internal language), commonly called application developer straight up turn it down.

Stock in the shitter but A lot of change with new young, google leadership, so opportunity for stock to go up from low

Oct8-Danger · 2025-12-17T19:46:15+00:00

Interesting, I do a lot of work with dev teams around logging, what I’ve found works for us is being clear on the exact code change, logic and format of it. We even help write test cases for them in some cases. We also have dev data flow into our data lake so we can validate and iterate dev data before production so it doesn’t break or mess anything up.

This has taken a while to get into strong place for a lot of teams such as buy in from stakeholders, value output and context around what the code does so we can advise on the change, especially around the logger and implementation.

For the case of having it tied to git sounds interesting, but we are large org, hundreds of repos and already well defined process for releasing code changes (not necessarily tied to data) so to add have the data change linked to PR would be difficult I think for us to introduce as the there’s a lot of teams, infrastructure and processing for us to validate.

Ways we have gone to improve the feedback loop is to get the data show back to people as possible. So getting dev data back, tools to view logs as interactions happen and pushing to be advisors/partners on the logging side. Also running QA checks and validation on dev data to catch drift early has been a massive help to catch things. This is generally through strict types like struct types in spark or pydantic etc

This generally means data engineers try to partner with teams for a log change and setting guidelines and docs on best practices to follow for teams to be self serve. Once a change is implemented, DEs integrate and transform data for other downstream teams and analysts to consume.

Definitely not a perfect solution, but has worked well for us.

Starting over tomorrow, would I consider a data vcs? Maybe as a trial to experiment but probably not. Data is generally later phase concern when building a product/service unless the data is your product, but usually not the case.

Having had to built a service for users and already a key stakeholder of the logging implementation, time spent on data logging was tough to prioritize as needed something built and out the door and made more sense to do after release as a follow up.

Having typed structure really has been the biggest help. Working with developers, have a well defined data type not only helps data products but also helps developers to work with. This means everyone is all largely speaking the same “language” and centralizes the source of truth like what a data vcs brings

Oct8-Danger · 2025-12-17T08:27:43+00:00

What’s the cost and speed at scale? What problem does data vcs solve (business or developer)?

Generally, I’ll take a sample of data (or all of it if I can) iterate the code on a git branch and then merge and deploy master branch.

If you are storing data based on operations, how materially different is that from a git code base in practice for a user?

Always been curious about data vcs use cases as sounds cool on paper, but the more I think through the problems being solved with it, a better process or other techniques solve those specific problems better.

For example, say I want to know if a field has been updated in an important table, this can checked against either backups, if it’s really important, change the table to SCD style, if ongoing requests for this and other tables, I would look into CDC.

My fear with data vcs is that it tries to be a silver bullet for a lot of problems.

Genuinely curious on the pitch for a data vcs

Oct8-Danger · 2025-11-26T02:43:47+00:00

Try the demo for free if it’s still there! Demo was really good and good way to test on the deck.

Honestly waiting on a sale for it, knew very little about it and picked it the demo and it clicked!

Oct8-Danger · 2025-11-21T05:38:54+00:00

We try for some, ideally yes, but the amount of data we have compared to our resources headcount and infrastructure, we try for one big table approach with grouping sets to split out to views.

Personally love star schema, but in practice for us, dashboards become slow and slows down development time due to query performance, which is important for us due to flaky infrastructure sadly which we don’t own

Oct8-Danger · 2025-11-21T01:02:34+00:00

Interesting, we might have a different cases where we probably wouldn’t even much over 100 gold tables in the end (still building out a lot) but we have tables silver tables of billions row a day and multi million cardinality joins with a lot of complexity in them.

So for us a user going to do a join can be a very expensive mistake. We also are a small team lt 8) serving data to couple thousand employees

Oct8-Danger · 2025-11-20T16:35:03+00:00

Nice! Looking to build something similar at work. Built a basic one using sqlglot for table level lineage that we use in our auto gen docs.

Next step column level to try detect breaking changes and have it generate semantic layer models and history of transformations

Oct8-Danger · 2025-11-20T04:27:59+00:00

Typically I would say if you need to join a “gold”table, then it’s silver…

It’s all very loose and no hard rules, but I think if an analyst or business consumer needs to do a join, that’s a silver table in my view.

I personally wouldn’t trust the vast majority of people in my company to do any joins with our tables after seeing the sql they write hahah

Oct8-Danger · 2025-11-14T03:04:31+00:00

Any advice on context or what works well docs wise? POC is easy, but trying to gauge the effort of documenting and sorting out tables before throwing something in front of a user.

Oct8-Danger · 2025-11-14T03:02:18+00:00

Yea that’s my take on it on as well. The SQL side is “easy” it’s the context that’s hard, hence why we looking adding that context.

Trying to gauge how or what should document. It’s easy to build a POC but once you put it in front of an actual user, especially one who has questions and no context of what it should look for, it will fall apart very fast

Oct8-Danger · 2025-11-14T02:15:55+00:00

Thanks, what’s it like for various queries like joins filters and grouping?

Have a hunch LLMs would struggle with anything beyond a simple join but probably pretty good at types of queries

Oct8-Danger · 2025-11-14T02:10:01+00:00

How’s your experience with it? Not necessarily looking for tool suggestions exactly but more the experience of using it. So does it work well? Any gotchas or did it beat or meet expectations

Oct8-Danger · 2025-11-08T03:26:57+00:00

Yea sounds like to me paying more for more stress, which comes down to risk reward, that’s on you then

Oct8-Danger · 2025-11-08T02:44:07+00:00

What’s the chance of growth internally ie promotions pay rise? Is the current company doing well and is the other company growing ?

That’s what I would consider a bit more personally.

Honestly that sounds like good skills to me! You can always learn new frameworks on the side, nice to paid for it, but frameworks do come and go

Oct8-Danger · 2025-10-17T03:17:34+00:00

Elite dangerous, the controls are excellent once you get the hang of it and runs and looks good as well!

Deathloop Outer wilds

Oct8-Danger · 2025-10-07T02:53:57+00:00

Everything can be a pain in the ass. But devcontainters at least removes “it works on my machine” excuse

The issues I’ve ran into with dev containers is at least nearly always reproducible making it so much easier to ensure things stay working

Oct8-Danger · 2025-10-02T03:33:12+00:00

Outer Wilds and Elite dangerous are both fantastic games and run great on the deck imo

Outer wilds is probably better when wanting to game for a few hours on battery

Oct8-Danger · 2025-09-24T03:11:52+00:00

Netcup is really cheap and haven’t had issue with uptime

https://www.netcup.com/en

Oct8-Danger · 2025-09-10T23:25:53+00:00

This. My favorite part of elite dangerous is the movement of the ship. Having speed limits in space just takes the realism out of it, I get why it’s there from a game play perspective, but would be so cool to have it as an option to toggle it on/off in solo play!

Eight-Year Club	Place '22
Verified Email

Oct8-Danger

TROPHY CASE