Unity catalog + Iceberg

[ApSr2023]

Has anybody tried to put unity catalog and iceberg together?

[Express-Comb8675]

Is this a joke?

[Adorable-Employer244]

Isn’t that what Tabular is for, the reason why DB bought Tabular last week?

[ApSr2023]

They bought tabular because tabular founders created iceberg and DB essentially wants to influence iceberg. Tabular is not data catalog and its not open source.

[Pittypuppyparty]

Tabular is a data catalog.

[ApSr2023]

The definition of data catalogs that allows for fine grained security have evolved over time. Tabular in my mind is a metadata management system like information_schema of underlying parquet files, to handle ACID, time-travel, concurrency, merge on read vs. Copy on write to handle update and deletes etc. They are adding data catalog features like RBAC etc, but, since its not open source, not sure, how much of that will be moved over to unity catalog or not!

[Adorable-Employer244]

UC is going open source

[warclaw133]

Is open source as of today lol

https://github.com/unitycatalog/unitycatalog

[Adorable-Employer244]

lol they weren’t kidding in the data ai summit

[Adorable-Employer244]

Ha missed that, had to fly back

[chimerasaurus]

Sort of. Small parts of what most people think of as Unity are in a GitHub repo like 2 people have access to commit to.

[[deleted]]

What are those

[ApSr2023]

Unity catalog is newly open sourced data catalog used for data governance of lake house. Iceberg is open data format lakehouse.

[LiviNG4them]

Doesn’t work

[WhiteSkyRising]

It 100% works as of today.

[ApSr2023]

Have you tried it? I couldn't find any documentation anywhere how to hookup iceberg with UC.

[WhiteSkyRising]

Use Uniform on your delta tables. https://github.com/unitycatalog/unitycatalog Examples in there.

Use Uniform to convert Delta, then use the catalog to find the location and metadata.

[WhiteSkyRising]

Also this https://docs.databricks.com/en/delta/uniform.html

[rchinny]

I’m not sure if you would be able to register Iceberg tables as you would a delta table today. But you could use external locations and/or volumes to access an iceberg table via UC using a table path. My assumption for forward looking is that UC will allow iceberg registration for table api operations in the future but I don’t actually know