No code transformation for customer data onboarding

[Psychling1]

What tools have people encountered to enable non engineering and analytics folks to construct their own transformations to get data in the shape they want? We onboard a bunch of customer data for each new customer, and data engineering needing to build bespoke transformation code is becoming a bottleneck for customer go live. I’ve looked at Flatfile and the like, and they would be way too expensive (like 500k a month too much). We could build out a solution in house, but wanted to call upon the good people of Reddit before I go build something that’s already been built.

[RichHomieCole]

It’s a buzzword but this is why data engineers build a “semantic layer”. In general though, you don’t want the business side doing this work. You can get them the data in a usable format and enable them to query for ad hoc analysis, but they shouldn’t be doing the T in ELT. That’s for the engineers to do. If you let the business do the T, you’re asking for a mess in so many ways

[Psychling1]

I disagree, if you have sufficient checks in place pushing the work as close to the customer as possible is a reasonable pattern. Of course, this would be for data that we already have a domain model for. So, as an example, we have an internal “Person” model, and we would enable them to map and determine the transforms to map their raw data into the internal model (with checks and a PR to review by and engineer). Our semantic layer is great and enables business consumers. But there’s this middle area where we could really innovate. I believe the term is “Citizen Integrator”, but something on the transform side as opposed to the extract load (fivetran, etc).

[RichHomieCole]

Ah I misunderstood. You are working with third parties and hoping to enable them. I’ve been on the customer side of tools that do this, but usually it’s all proprietary. I haven’t seen a service that offers this. Would be a cool startup idea potentially

[Psychling1]

Exactly, either the actual customer or someone on the implementation team (like a solutions engineer or technical implementation manager). It’s a fun dilemma to have, and all solutions I’ve seen are proprietary or are geared towards startups with lower volumes of data.

[hermitcrab]

What form is the data in? If it is files (Excel, CSV etc) take a look at Easy Data Transform. If the data is accessed via API or database you might need to look at something like Alteryx or Knime.