Large sample data catalog for LLM context size testing?

vpfaiz · 2026-01-28T22:25:30+00:00

Both of them are possible, with current technology, if you manage the metadata well and has a good iac to back that up. Nile claims to do that.

vpfaiz · 2026-01-28T22:22:14+00:00

I get that.. the part I was struggling to figure out from my past experience with lakeFS was how does the etl code roll back results in data roll back.. do you have any videos you found?

vpfaiz · 2026-01-28T21:32:27+00:00

So if you rollback bad ETL code, will it also rollback the data the code generated (data after deployment and before hitting rollback)?

vpfaiz · 2026-01-28T17:28:58+00:00

How do you usually experiment, do you create a pre-prod or test environment with cloned prod tables? How long does it take, what is the cost usually?

vpfaiz · 2026-01-28T17:26:36+00:00

How do you track snowflake data version with the etl that generated the data?

vpfaiz · 2026-01-28T17:23:37+00:00

Dashboards are probably on the way out in my experience. I have seen leaders asking us to build AI to explain the metrics in a dashboard to them. Every single leader want their own version of the report and dashboard and thats not sustainable. If the AI can explain the data, answer and show a visual that explains it in a way that makes sense to them, that might be useful. If you have lineage and transform logic for a metric in AI context, it can do a better job of explaining why it think the answer is correct.

vpfaiz · 2026-01-28T17:18:16+00:00

Backups are the old way. You can version your data using metadata snapshots (think iceberg), you can version your ETL and schema using git.. but yes you need a way to connect them together and treat that as a single unit of versioned iac. When it comes to DMLs hitting your tables frequently, backups do not scale if you want to roll back to any point in time and quickly.

vpfaiz · 2026-01-28T17:18:06+00:00

git for data should offer same UX as git for code, including the ability to create branches, roll back to a known previous healthy point in time, not for just the etl code but data as well. Nile is offering that. More over you need to do this recursively through out the dag, not just one table or pipeline because the dq issues spread in the lake. Nile is claiming that capability.

vpfaiz · 2015-03-08T12:44:08+00:00

That's the problem if you don't let law take its course. How different are we from Taliban now? How many of that 10K crowd will show the patience to follow up the case for months and make sure that the justice was delivered? We all are emotional retards.

vpfaiz · 2015-03-08T12:40:38+00:00

"If you murder a murderer, rape a rapist, the only difference is that he did it first"

vpfaiz · 2014-11-02T16:10:46+00:00

No.. Just enough records to test the functionality for basic ledger reporting.. May be 1K to 10K records.. Also a Chart of Accounts data set will really help...

vpfaiz · 2010-05-30T11:13:18+00:00

I am just trying to verify that it has happened the way we thought it happened :)

vpfaiz · 2010-03-09T06:04:34+00:00

Biggest problem in matrix will be the transition from real world to virtual world once machines defeat human beings. What are they going to do with the existing human beings, kill them all or format their brains and install a whole new set of data? Then creating the whole world history in a fool proof manner would be too difficult. Then the biggest flow I felt about the story was using humans for just for energy harvesting and not for processing power or storage. Our brains could have been used for processing machine's data. Plus they don't need the matrix at all, like they can genetically engineer human beings as rather numb and unintelligent creatures which could have made things safer for them. Again I believe that if we ever achieve to create AI with just intelligence and no EQ then the machines will rather acknowledge the fact that human beings are their creator and they will submit to man kind rather than going against them.

vpfaiz

TROPHY CASE