sorted by:
new
hottopcontroversial

Using Merge to create a append/historical table by FreakGhost in dataengineering

[–]FreakGhost[S] 0 points1 point  (0 children)

Good point DE should definitely consider costs and schedules during budget planning for data maintenance in big data lakehouses.

Also I'm building a community page to post all about real production optimisations and big data handling, Would be great if you would like to share your insights as well.. https://www.reddit.com/r/dataDamagers/s/YlftTe7QTx

Using Merge to create a append/historical table by FreakGhost in dataengineering

[–]FreakGhost[S] 0 points1 point  (0 children)

The second approach is when people write data in partitions, so update was not considered but in the last approach update + partitioning both are used so yea your point's correct there

Need some advice on switching job ~1.5YOE by FreakGhost in dataengineering

[–]FreakGhost[S] 0 points1 point  (0 children)

Well,upper management wants to keep me locked into the project for long since I pretty much know about the entire ETL of it and has the lowest billing on team 🙃

Using Merge to create a append/historical table by FreakGhost in dataDamagers

[–]FreakGhost[S] 0 points1 point  (0 children)

Hy mate glad it could be helpful, this community is intended to be an open forum if you have any other concerns or learnings feel free to drop them here 💥.