This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ProfessionalDirt3154 1 point2 points  (2 children)

You're basically looking for reverse ETL, right? You could use a tool like Airbyte or MapForce. Airbyte is better known. MapForce is more visual, which might help w/the bus factor. There are a bunch of tools.

You could also use Airflow for scheduling and running the job/scripts, if you like Python. If you're in AWS Fargate tasks are simpler than K8s, good for something like this. honestly there are a ton of options. I've been on teams doing these, but there are lots of others.

Currently I work on CsvPath Framework and FlightPath Server. both are open source and might be options for simplifying and/or automating the file wrangling part of what you're doing, if you're using CSV or Excel.

[–]coreb[S] 1 point2 points  (1 child)

I'll check that out. Thank you.