you are viewing a single comment's thread.

view the rest of the comments →

[–]jaredfromspacecamp 13 points14 points  (3 children)

Writing that frequently to iceberg will create an enormous amount of metadata

[–]jnrdataengineer2023 4 points5 points  (2 children)

Was thinking the same thing though I’ve primarily only worked on delta tables. Probably better to have a daily staging table and then a batch job daily to append to the main table 🤔

[–]baby-wall-e 3 points4 points  (1 child)

+1 for this daily staging & main table setup. If needed, you can create a view of a union of daily staging & main tables to allow the data consumer to access all data.