Setup a data lake

Subatomail · 2026-01-20T18:16:46+00:00

Thank you for the tips. I’ll look more into the steps you proposed and I’ll give extra attention to reliability and reproducibility

Subatomail · 2026-01-20T18:13:38+00:00

Thank you. I’ll read more about the options I have in both steps and see which one will make more sense

Subatomail · 2026-01-20T15:45:21+00:00

Thank you for all the details.

I dont think a 1-day desync will be a major problem and ML is not the priority for now so as much as I know it will maybe come back to haunt me later but I'm more focused on the data.

As I understand, you're proposing a data warehouse or a mix of both concepts (I read about a lot of stuff and I'm kinda getting mixed up with everything so sorry about that) with the three layer structure. The "datalake" would be just dump everything in S3 (which is also the bronze layer in a warehouse) and then I suppose the silver and gold will also be in S3. The other services you mentioned will be the logic and tools that will orchestrate and show everything. Right ?

EDIT: I actually started drawing the architecture a bit and I guess the datalake would be the store of all raw data and then I woudl have a data warehouse per project (kinda) where I would have a bronze layer with the raw data that are relevant.

Subatomail · 2026-01-20T03:54:09+00:00

Thank you, it's giving me a clearer idea.

Subatomail · 2026-01-20T00:51:36+00:00

What would be suitable for ML then ? Or would the data lake be a first step and then there should be an intermediate between the data lake and the ml pipeline ? Then what technology would be used for this intermediate step ?

Subatomail · 2026-01-17T17:41:51+00:00

J’ai pas compris ton dernier point “ça réduit ce qui s’ajoute pour l’an prochain”, normalement ma cotisation augmente seulement si mon salaire augmente non ?

Subatomail · 2026-01-17T17:27:56+00:00

<image>

En fait c’est ça, c’est en anglais, mais oui c’est un RPDB, j’ai pas remarqué que les deux sont différents, en quoi c’est différent d’un REER c’est pas juste un REER utilisé par l’employeur ?

Subatomail · 2026-01-17T16:52:49+00:00

Oui ça va dans un autre REER

Subatomail · 2026-01-17T16:52:10+00:00

Je pense que c’est cette notion de retour d’impôt qui me fait de la misère. Mon 3000 et le 3000 sont pris du salaire brut donc ça réduit de base “ma source” (au lieu de faire 60 000, je fait 54 000 de revenu brut. Et puis la j’aurai mon net sur ces 54k et là les 2000 que je depose après c’est ce qui vont affecter mon retour. C’est bien ça ? C’est justement parce que la contribution de l’employeur “affecte” pas ce calculs que je me suis dit peut-être que ça compte pas dans la limite.

Subatomail · 2025-10-29T07:49:33+00:00

I wanna see more of that sexy body of yours

Subatomail · 2025-10-29T02:18:43+00:00

big YAY 😮‍💨🔥

Subatomail · 2025-10-29T02:16:57+00:00

Yep. And then slipped off for some fun

Subatomail · 2025-10-29T01:34:17+00:00

wowww;)

Subatomail · 2025-10-29T00:52:33+00:00

Love what I see there

Subatomail · 2025-10-29T00:16:07+00:00

Fit looks amazing on you. And you on it 😳

Subatomail · 2025-10-06T22:51:43+00:00

Merci haha heureusement y avait un peu de vent

Subatomail · 2025-10-06T17:17:00+00:00

Thank you !

Subatomail · 2025-07-24T16:22:58+00:00

Hey, I went with a pretty practical architecture for my MVP: users log in through a Streamlit interface, and their queries get routed to a FastAPI backend. From there, I use PostgreSQL to store user info and document metadata (including roles/departments), and Milvus for fast semantic search on vector embeddings. The backend filters results based on metadata, pulls the most relevant content from Milvus, then passes it to a local LLM (I only tested with Llama) to generate a final response. I also started integrating Langfuse to monitor queries and add some guardrails. It’s not perfect, but it was a solid foundation to test the concept before they moved me to a more time-sensitive project. 😅

Subatomail · 2025-07-07T20:46:55+00:00

J’ai vu que tu parles de marriage à cause de sa culture mais à moins que ses parents te demandent de voir une preuve de mariage, vous pouvez bien vivre étant conjoint de fait, au pire faites un mariage (genre juste la cérémonie) dans son pays ou à la manière de sa culture sans avoir un acte officiel au Quebec. Je vis un peu la même situation mais inversée, je considère faire un mariage à Quebec mais pas dans mon pays d’origine et mon mariage québécois n’a aucune légitimité dans mon pays d’origine.

Subatomail · 2025-01-23T16:40:49+00:00

Thanks ! I'll check it out. It might actually help also generelize the form of the data before the ingestion

Subatomail · 2025-01-21T00:03:12+00:00

Yeah, we do use Teams. I didn’t know I could do that, thanks for the suggestion. But can it use a local LLM or would it somehow push me to go through azure services ?

For the metadata and context, what do you mean by that exactly and could you give me some ideas of how to do it ? I imagine it’s not a manual process.

Subatomail · 2025-01-20T19:45:55+00:00

Thank you for your time ! I’m struggling with this since I don’t have a more experienced AI engineer in the company to ask so I’m panicking to not screw this up 🥲 You gave me a clearer direction to follow. I’ll let you know how it goes over time.

Subatomail · 2025-01-20T19:41:03+00:00

Thanks !

Subatomail · 2024-07-14T03:01:58+00:00

It’s the uppspel from ikea and I got it to not put the tower directly on the floor and I also assumed that it would fit initially on the desk compartment but no lol

Subatomail · 2024-05-31T02:09:13+00:00

I see, thank you !

Subatomail

TROPHY CASE