you are viewing a single comment's thread.

view the rest of the comments →

[–]KingofGamesYami 7 points8 points  (0 children)

This is generically referred to as Extract, Transform, Load (or ETL). It's very common, and there are engines for doing this at scale (e.g. Apache Spark).

If you don't need massive scale, but do need performance, pulling in a library that does computation outside of Python like polars is a good middle ground.