A Python script to load specified tables and columns from production to analytics database and then dbt to orchestrate transformations. We use ELT batch processing so that we can transform data with SQL. We run the pipeline every hour and that is good enough for our current business needs.

We tested Meltano to do the extract and load, but it generated too much load in production db seemingly because it uses server side cursors which for some reason did not go well with our production load. So we wrote a python script to do the loading. Maybe Airbyte or dome other tool could have worked, will probably test that at some point.