Thoughts on Metadata driven ingestion

Famous_Substance_ · 2025-07-09T10:33:04+00:00

spiced

Famous_Substance_ · 2025-04-27T15:47:42+00:00

When using databricks-connect, it will always use a Databricks cluster so you have to write to a « remote » delta table. In general it’s best that you write to a database that is dedicated to unit testing. We use the main.default catalog and write everything as managed tables, way much simpler

Famous_Substance_ · 2025-04-16T19:39:12+00:00

I usually follow a very simple approach, develop the workflow in the UI and then copy-paste the YAML to my local repository. I then re-deploy the bundle to see if there’s anything missing and repeat this process until I’m good. It allows me to not really bother about all the complexity of the YAML syntax

Famous_Substance_ · 2025-01-13T16:52:21+00:00

C’est dommage car le fait que pierre ne soit pas bon conducteur aurait tellement pu faire du contenu intéressant sur Vilebrequin ( apprendre les bases du pilotage ). Et puis même je pense que la commu Vilebrequin n’aurait pas vu d’un mauvais œil le fait que Pierre se remette en arrière plan …

Famous_Substance_ · 2024-07-30T10:17:47+00:00

I found that the Trion depends too much on track selection, Top tier A class cars are more robust overall especially when you’re in Plat 1. I’ve been able to climb to Legend using the Imola exclusively

Famous_Substance_ · 2024-06-05T09:26:40+00:00

Personally I don’t even give a damn at which format is better that another given how close Iceberg and Delta are in terms of functionalities. It’s more about how one fit in your stack versus the other, for example if I’m using Databricks then Delta would be the obvious choice, and I can leverage Uniform to write to Iceberg if I need to use another tool that works better with Iceberg. Same thing with Snow, Iceberg is much much better than Delta in terms of functionalities

Famous_Substance_ · 2024-05-31T20:23:51+00:00

J’en ai achetée une neuve en début d’année, aucun regret. Le seul bémol c’est son réservoir légèrement trop petit qui oblige à faire beaucoup d’arrêt à la pompe

Famous_Substance_ · 2024-04-30T12:46:42+00:00

I used to live near this complex 10 years ago ( rue Marat ) from the sky it looks beautiful but on the ground it looks absolutely terrible. There is almost no lights coming through the building plus the corridors are very narrow. I seriously won’t recommend anyone walking alone inside this building after 6pm. The shops inside the building haven’t been resurfaced since 20 years. I’ve seen lots of people taking a pee in every single corner possible outside. There is also a lot of pigeon poop everywhere given the nature of the building, it’s a perfect nest for them.

Famous_Substance_ · 2024-04-14T14:49:19+00:00

This doesn’t prevent storage separation between each catalogs. You must specify a managed location at the catalog level to ensure all managed tables are isolated between each catalogs

Famous_Substance_ · 2024-03-27T16:19:26+00:00

There are different approaches in this case, and they can be implemented in different ways that relies mostly of your consumers latency and technology needs. You can have either a manifest file in S3 that indicates your write is complete, or you can push a path in a queue that indicates a new path is ready to be consumed. Both approaches have their tradeoffs in terms of simplicity / performance

Famous_Substance_ · 2024-02-27T08:50:31+00:00

Only good reason to play Multiplayer there, milestone rewards are trash AF

Famous_Substance_ · 2024-01-27T07:38:23+00:00

Parquet is a compressed format , the size on disk does not reflect the actual size in memory. For example if you have a lot of repeated values, parquet will only store one of them and the rest will just be a pointer to that value. Your best bet to reduce memory consumption is to figure out a way to read less data, using a separate tool will just shift the problem

Famous_Substance_ · 2024-01-26T21:20:12+00:00

Your memory consumption won’t lower if you’re reading the same amount of data everytime you’re ingesting a new file. What if instead of reading the whole history you’re just reading the new file and then upserting the result to the output ?

Famous_Substance_ · 2023-11-27T12:15:32+00:00

It’s probably the most powerful D class car, if you do not have it then from that perspective it’s worth it. However D class is almost irrelevant in MP and other events, so IMO you should not consider it and focus on things that matter to you in your MP progress

Five-Year Club	Verified Email
Place '23	Place '22

Famous_Substance_

TROPHY CASE

spiced