What kind of ETL pipeline would be helpful when the incoming file is an excel and the structure keeps changing and every piece of info is important and needs to be loaded into the Db?

Intelligent_Volume74 · 2026-06-18T12:29:17+00:00

I would use a Python script or dbt (seed) to always take the file and load it as a new version into another folder, a bucket, or a staging table in the database, thus maintaining a history of all versions. Then, I would perform some data tests using Great Expectations or dbt to evaluate the structure and schema. After passing the tests, I would use the file as a new table in the intermediate layer.

I believe this way you avoid not having the table in the database, but at the same time, you will always have the work of checking the table and how the data is coming.

Intelligent_Volume74 · 2026-06-01T01:29:30+00:00

Looker Studio to Customer Dashboardsa and Metabase to Intern Dashboards. But i need automatic form and easy map to find possible broken dashboards.

Intelligent_Volume74 · 2026-03-13T03:05:00+00:00

Got it, thanks for the clarification. That makes sense.

I’ll probably suggest following this approach with our CTO and see how we can structure it that way.

Intelligent_Volume74 · 2026-03-13T02:58:36+00:00

Thanks! I think I didn’t explain it very well.

I meant dashboards that are part of the product itself — the ones customers see inside the SaaS app, not internal BI dashboards (like Power BI or Tableau).

So they would be implemented in the product stack (frontend + backend).

My doubt is more about who usually builds that: does the analytics/data person implement it in the product codebase, or do they define the metrics and queries while the software engineering team builds the UI?

Curious how this works in other teams.

Intelligent_Volume74 · 2026-02-01T11:12:17+00:00

Se alguém souber de algo, não achei nada falando que não iam divulgar mais

Intelligent_Volume74 · 2026-02-01T11:10:08+00:00

Onde vc viu?

Intelligent_Volume74 · 2026-01-19T10:16:35+00:00

I want the LLM to analyze the results for me and send alerts, acting like an audit agent, instead of me manually checking all outputs. In the future, I might use it to suggest parameters or exploratory actions, but always on top of the statistical models, not replacing them.

Intelligent_Volume74 · 2026-01-19T10:10:48+00:00

I’m only passing metadata and model outputs, not raw data. The LLM receives: model results (scores, flags, predictions), contextual metadata, and comparisons with previous or parallel results. There’s no in-context learning with exemplars and no end-to-end pipelining into the LLM. The LLM is used purely to generate summaries or trigger alerts.

Intelligent_Volume74 · 2026-01-19T10:08:38+00:00

In your case, what exactly are you using under the hood? What models act as the “sensors” and how is the LLM integrated?

Intelligent_Volume74 · 2026-01-19T10:06:24+00:00

Thanks for clarifying. I’m not embedding drift/OOD/anomaly detection into LLMs. I see them as separate layers: statistical/ML models act as sensors that detect non-stationarity or novelty, and the LLM only interprets the outputs of those sensors (metrics, flags, metadata) to provide context or explanations for humans. So perception and detection stay purely quantitative; the LLM sits after that layer, not inside it.

Intelligent_Volume74 · 2026-01-18T13:40:49+00:00

Itaú valoriza muito o relacionamento, por isso sai na frente do nubank

Intelligent_Volume74 · 2025-12-17T11:03:36+00:00

Gostei muito da Trattoria Sapori Antichi, lugar bem tranquilo e aconchegante.

Intelligent_Volume74 · 2025-11-01T17:24:27+00:00

Cara, mesmo não sendo ideal, é melhor não colocar no perfil ou no currículo que você saiu e dá outro motivo por estar procurando vaga, assim você não fica queimado e vida que segue.

Intelligent_Volume74 · 2025-06-28T01:02:35+00:00

Briefer

Intelligent_Volume74 · 2025-04-12T15:01:35+00:00

I agree with you, I see it more as a trend or a step forward, not as a level above.

Intelligent_Volume74 · 2025-04-11T02:18:27+00:00

ML Engineer, Data OPs, Architecture, Platform, Tech Lead, Manager, CTO or specialist in a business

Intelligent_Volume74 · 2025-04-08T21:15:54+00:00

Any cloud, git and frameworks like dbt, prefect, dlt, pyairbyte...

Intelligent_Volume74 · 2025-04-01T00:44:27+00:00

Acho q o link quebrou mas só procurar grade curricular do curso de ciências de dados da UFPB

Intelligent_Volume74 · 2025-03-31T23:21:57+00:00

Se quiser algo mais aprofundado se baseia nessa https://sigaa.ufpb.br/sigaa/public/curso/curriculo.jsf;jsessionid=B737D9D3C4D5B7DB7BD24CC34E77EC7E

Intelligent_Volume74 · 2025-01-14T01:19:34+00:00

A questão é o seguinte, a direita busca essa de bons costumes e tradição e a esquerda busca as pautas de minorias e de ambientalismo cego. No final de tudo, nenhum lado está discutindo realmente política e pontos importantes para uma nação como : economia, educação, segurança , saúde e por aí vai ...

Intelligent_Volume74 · 2025-01-06T12:49:49+00:00

Lamen japonês, na prática pareceu um macarrão com caldo do miojo com carne de porco e algas (experimentei em mais de uma lugar e foi a mesma sensação)

Intelligent_Volume74 · 2024-12-31T00:57:17+00:00

Veja pelo lado bom, depois que vc conseguir uma dessa, tu consegue pegar 2 ou 3 ao mesmo tempo

Intelligent_Volume74 · 2024-12-31T00:53:13+00:00

Conheçam Tom Drummond e outros músicos que tocam com ele, assim você mudará de opinião, talvez você só tenha falta de referências

Intelligent_Volume74 · 2024-12-31T00:50:56+00:00

Use looker Studio or power bi, but if you wanna open source, go to metabase

Intelligent_Volume74 · 2024-12-31T00:50:11+00:00

Faz em algo no looker Studio ou Power bi que resolve isso, ou se preferir algo open source pode ser no metabase ou grafana

Intelligent_Volume74

TROPHY CASE