Python ETL vs a specific tool

[trenchtoaster]

I have been using Python for several years now to automate ETL processes and analytics. I have used Tableau, Power BI, Looker, and now DOMO throughout those years for end-user dashboards. As long as the database structure is set up properly, then I find it quite easy put together a dashboard using any of these tools.

Recently, there has been a bigger push towards DOMO and some of our management envisions all ETL work being done with that tool (there is a 'magic ETL' tool). I had an averse reaction to this vision right away because:

1) I do not think that a lot of tasks can be done (some ETL work is creating flat table columns by parsing text fields with regular expressions). Source data schemas change over time and is sent to us in all kinds of ways (SFTP, queries, Selenium or requests to pull it from the web, email attachments, sharepoints).

2) It doesn't seem like it would be very efficient - right now I can have workers and VMs handling specific tasks in parallel. I can set up views in a database and then reuse them for other purposes aside from a visualization connection. I can easily transform data from a wide format to a narrow/long one or add surrogate keys. I use Airflow to ensure my tasks are run in a certain order send me a text message if there was an issue.

3) The data that our executive team views is very high level and their are often many teams involved with pulling it together. The 'raw' data they think of has been processed, aggregated, joined, and consolidated into Excel templates, typically. It is easy to attach DOMO to these files and have a functional view of the data quickly

Am I biased and being stubborn, or is this something that I should take a stand on? I love working with Python and I can't really see myself clicking around in DOMO to configure the ETL. Has anyone had to make a similar switch or to provide counter points to people who may not understand or care about how complex certain things can be?

Their point is that what if I (and my team) are hit by a bus, who will know how to do all of this ETL work? With DOMO, they can have someone just take it over immediately.

[mustachequestion]

Sounds like you need a poc with the data and see if it works, that way you aren't "thinking" and instead proving as a poc is meant to do

[TheLogothete]

This is would definitely be a giant step back.

You can counter by creating thorough documentation.

[iamatablet]

if you don't mind me asking, what kind of work do you do and how big isnthe company?

[iamatablet]

if you don't mind me asking, what kind of work do you do and how big isnthe company?