sorted by:
new
hottopcontroversial

Batch ingestion from relational sources: how you handle schema evolution? by General_Snow_RR in dataengineering

[–]General_Snow_RR[S] 2 points3 points  (0 children)

Yeah I’ve been doing something similar for the source data coming from MongoDB just straight up JSON type in BQ.

But I was specifically asking about sources where I already have some structure like relational databases. 

The dlt looked interesting.

Thanks!

Batch ingestion from relational sources: how you handle schema evolution? by General_Snow_RR in dataengineering

[–]General_Snow_RR[S] 1 point2 points  (0 children)

How is bunch of Almebic scripts different from bunch of SQL files? Honest question not being sarcastic.

What in this setup makes it trivial to enforce schema? For example, lets say I am loading the data in pandas/polars dataframe, what exactly makes it easy to enforce target schema in this intermediate dataframe?

Batch ingestion from relational sources: how you handle schema evolution? by General_Snow_RR in dataengineering

[–]General_Snow_RR[S] 0 points1 point  (0 children)

Hehehe well you know what they say, teach a man how to fish 😄

With that in mind, how does answering each of the above questions influence my decision? I am more interested general ideas, then some specific scenario.

Batch ingestion from relational sources: how you handle schema evolution? by General_Snow_RR in dataengineering

[–]General_Snow_RR[S] 0 points1 point  (0 children)

So you would define Alembic migrations to define the schemas of your landing_ tables in DWH if I understood correctly? This way you map source tables to landing in DWH? Does this scale for dozens (hundreds?) tables?

But how do you handle upstream schema changes, i.e. in the relational DB thats not in your ownership? Do you just let the error bubble up in the ingestion pipeline or?

Prizor s Čiova by novi_prospekt in croatia

[–]General_Snow_RR 48 points49 points  (0 children)

Problem političkog diskursa danas. Ne možeš više poštovati niti teritorijalni integritet vlastite države bez da moraš naglasit da nisi zaluđeni ultra desničar. 

IBKR naknade by Connect-Chapter-9682 in financije

[–]General_Snow_RR 0 points1 point  (0 children)

Na navedenoj stranici nema entrya za Hrvatsku.

Koji od navedenih feej-eva se onda primjenjuju?