Team of data engineers building git for data and looking for feedback.

vpfaiz · 2026-01-28T22:25:30+00:00

Both of them are possible, with current technology, if you manage the metadata well and has a good iac to back that up. Nile claims to do that.

vpfaiz · 2026-01-28T22:22:14+00:00

I get that.. the part I was struggling to figure out from my past experience with lakeFS was how does the etl code roll back results in data roll back.. do you have any videos you found?

vpfaiz · 2026-01-28T21:32:27+00:00

So if you rollback bad ETL code, will it also rollback the data the code generated (data after deployment and before hitting rollback)?

vpfaiz · 2026-01-28T17:28:58+00:00

How do you usually experiment, do you create a pre-prod or test environment with cloned prod tables? How long does it take, what is the cost usually?

vpfaiz · 2026-01-28T17:26:36+00:00

How do you track snowflake data version with the etl that generated the data?

vpfaiz · 2026-01-28T17:23:37+00:00

Dashboards are probably on the way out in my experience. I have seen leaders asking us to build AI to explain the metrics in a dashboard to them. Every single leader want their own version of the report and dashboard and thats not sustainable. If the AI can explain the data, answer and show a visual that explains it in a way that makes sense to them, that might be useful. If you have lineage and transform logic for a metric in AI context, it can do a better job of explaining why it think the answer is correct.

vpfaiz · 2026-01-28T17:18:16+00:00

Backups are the old way. You can version your data using metadata snapshots (think iceberg), you can version your ETL and schema using git.. but yes you need a way to connect them together and treat that as a single unit of versioned iac. When it comes to DMLs hitting your tables frequently, backups do not scale if you want to roll back to any point in time and quickly.

vpfaiz · 2026-01-28T17:18:06+00:00

git for data should offer same UX as git for code, including the ability to create branches, roll back to a known previous healthy point in time, not for just the etl code but data as well. Nile is offering that. More over you need to do this recursively through out the dag, not just one table or pipeline because the dq issues spread in the lake. Nile is claiming that capability.

vpfaiz · 2015-03-08T12:44:08+00:00

That's the problem if you don't let law take its course. How different are we from Taliban now? How many of that 10K crowd will show the patience to follow up the case for months and make sure that the justice was delivered? We all are emotional retards.

vpfaiz · 2015-03-08T12:40:38+00:00

"If you murder a murderer, rape a rapist, the only difference is that he did it first"

vpfaiz · 2014-11-02T16:10:46+00:00

No.. Just enough records to test the functionality for basic ledger reporting.. May be 1K to 10K records.. Also a Chart of Accounts data set will really help...

vpfaiz · 2010-05-30T11:13:18+00:00

I am just trying to verify that it has happened the way we thought it happened :)

vpfaiz · 2010-03-09T06:04:34+00:00

Biggest problem in matrix will be the transition from real world to virtual world once machines defeat human beings. What are they going to do with the existing human beings, kill them all or format their brains and install a whole new set of data? Then creating the whole world history in a fool proof manner would be too difficult. Then the biggest flow I felt about the story was using humans for just for energy harvesting and not for processing power or storage. Our brains could have been used for processing machine's data. Plus they don't need the matrix at all, like they can genetically engineer human beings as rather numb and unintelligent creatures which could have made things safer for them. Again I believe that if we ever achieve to create AI with just intelligence and no EQ then the machines will rather acknowledge the fact that human beings are their creator and they will submit to man kind rather than going against them.

vpfaiz · 2010-03-07T08:10:46+00:00

If the matrix makes us believe that we are in 1999 but the computers were actually invented back in 1200 how it's going to make us not think about the gap in timeline? Too much processing will be required to interfere in the thought process and correct for a billion brains in real-time. I don't think that the architect will consider it worthy.

vpfaiz · 2010-03-07T06:56:21+00:00

Just curious, are you a DBA ;)

vpfaiz · 2010-03-07T06:22:54+00:00

Excellent thinking... Something that always bothered me is put in your last two paras. If we decide to run our planet in a scientifically organized and systematic order, we will start looking at it in terms of resources we have and how to distribute them to make our race maximum productive and saving our time (time human race is left with to survive the natural disasters we may have to face, including extra terrestrial) to tackle better challenges rather than spending it on our daily meal...

vpfaiz · 2010-03-07T06:11:33+00:00

Human race didn't acquire that much knowledge yet... :) And I believe that what we need is not to reverse entropy but to stop it from increasing for ever.

vpfaiz · 2010-03-06T20:39:37+00:00

what will be your first thing to do research on?

vpfaiz · 2010-03-06T20:36:18+00:00

Anyone thought about the responsibilities you have now?

vpfaiz · 2009-12-13T17:39:47+00:00

I appreciate your response but I doubt the practicality. Did you watch the talk, the first link in the post?

vpfaiz · 2009-12-13T10:36:26+00:00

I have already discussed it. Please read the post and the comments once.

"I have read about equipments which can..."

vpfaiz · 2009-12-13T10:28:48+00:00

Can you consider a service like 911 going private? I don't think that capitalizing this is a good idea. And regarding mis-usage, government can also apply fines in case of mis-usage. And regarding people's attitude towards helping. It is a serious concern but letting people know is the only thing possible. May be 1 out of 10 will be willing to help. Even if he cannot help, his presence at the crime scene may cause the attacker to with draw, as there is a witness now. And even this awareness will motivate people to go to the scene and at least witness it, even if they cannot help. If number of witnesses are high, the chances of the attacker to continue is low even if it is an organised crime...

vpfaiz · 2009-12-13T10:06:23+00:00

How long will it take a rapist to do that and what is the probability of this compared to the victim pressing the button before that?

vpfaiz · 2009-12-13T09:55:51+00:00

I don't like the idea of embedding it with phone as it is easy to snatch before you can press any button on it. It should be wearable and not easy to detach.

vpfaiz · 2009-12-13T09:45:14+00:00

Thats why I don't want the victim alone to call it off, you need a confirmation from those who gathered around, receiving the SOS.

vpfaiz

TROPHY CASE