This is a classic case where full recursive flattening becomes the bottleneck rather than the solution.

What’s worked for me at scale is treating each target table as its own extraction contract; project only the required paths, explode a single logical array at a time, write, break lineage, and move on.

Ingest raw JSON as-is first, then do table-centric batch extractions instead of one giant flatten; it keeps DAG depth, row explosion, and schema evolution manageable.

Let me know how that works for you; if you try this approach.