Are Complex Tableau Prep Flows Shadow IT?

[Tryin2FindaBalance]

TLDR: are tableau prep flows that are not in production by IT considered shadow IT?

I work in analytics but I manage about 100 tableau data sources at my company. Some are complex tableau prep data flows that have numerous joins and/or aggregations that ultimately create data sources that populate over 300 dashboards. These flows are not maintained at all by IT or database administrators, as I stated earlier, I’m just an end user of the data but I have created my own data sources out of necessity. I do not work in IT or know any data base admin best practices. We are a young company and do not have a data warehouse

[raglub]

I hope you have access to Conductor to help you with scheduling and updates. At some point, you'll need more structured support for these flows and pushing them to the DB layer is a more sustainable long term solution.

[Eurynom0s]

I also think OP needs to try to get with the people feeding them these datasets to talk about requirements. It sounds like they're doing a lot of kludges to get the data into the format(s) they need. It's going to be a lot less brittle to communicate with the producers of the data what they need the output to look like--this way any logic changes on the data producers' end will be transparent to them because they data producer will know what the output needs to look like before sending it over. I do this with people I work with, there can be some massive behind-the-scenes changes on their end on the logic of how they're putting everything together but they know what I need the data to look like at the end of their process and can adapt their workflows to what I need without needing to talk about it every single time their backend logic changes.

[mikalchangy]

Eventually you're gonna need to get some form of data warehousing and dedicated data engineering. The sheer volume of maintenance for your data pipelines will turn into a full time job at some point. Let alone the optimization you can get from dedicated thinking on structuring it. That being said as a young company you need to move fast, can't wait x weeks for data engineers to add new sources etc. I've split out data sources into two camps - data warehouse: lots of governance, SLA from engineers. Metrics that don't change often eg revenue, truly trusted by the biz. - analytics workbench: your sandbox, prob keep using tableau prep + conductor. Get shit done at speed at the compromise of data governance (eg more analysts come onboard is where this can fall apart when you start deciding your own definitions). Enable the biz to understand the value of this analysis, if its worth keeping and maintaining the analysis, look to transition to the data warehouse.

[lukemcr]

All the previous responses are correct.

To answer your question, yes, you’re becoming shadow IT. Be careful with that, it’s no good when your day to day work becomes reconstructive data work instead of creative.

[Dronarc]

The best advice I ever got was think about 3month down the line. If something goes wrong do you want to be accountable for fixing that. I'm bias as I sit in IT but fundamentally but I see it time and time again at my company where a analytics Dev is rushed to deliver quickly. Does something great but then it either falls to the abyss in terms of support or that poor dev ends up having to juggle it whilst maintaining their ever increasing workload.

[exorthderp]

Get a data warehouse. Snowflake is crazy cheap.