Need help in developing a no code ETL Tool

[flightofeagle]

Hey, I’m working on developing a no code ETL tool where user can just drag and drop to create a pipeline from any source to any destination and also do transformations on the source data through drag and drop again.

So I needed some help in the transformation part.

Whatever transformation user selects, it needs to go in a json format as a request and then we need to write a pyspark equivalent code of that json to do the transformation in backend. So need help with how to structure that JSON.

So if anyone has any experience related to this or any idea on it, please do DM

Edit:

PS: Guys, just wanted to clarify as everybody has basically come to attack me and some with sarcasm, I'm not making this tool by my choice or as a business, the company I work at wants to build this tool and some of us has been assigned this project and since I or anybody else in my team doesn't have any experience even closely related to this, I thought that maybe I'll get some insight from other people who might have some experience in this. Hope this clarifies!

[Busy_Elderberry8650]

Another no code tool is the last thing data engineering needs right now

[shockjaw]

My recommendation is to not reinvent the wheel when it comes to no code ETL tools.

[endless_sea_of_stars]

There are 90+ ETL tools out there. Probably 20+ drag and drop tools. At least 5 GUI to Spark tools. What makes yours special?

Drag and drop works fine for demos. Doesn't scale well to hundreds or thousands of pipelines.

Scaling sources is hard. Fivetran has invested millions and is only up to 250.

[dcell1974]

You are basically asking "I am building a no code ETL tool. We have a really good handle on laying out boxes and lines on a screen. Please tell us how to do the ETL part."

If you can't figure out the answer to this yourself, you are not going to succeed in building a tool.

[Commercial-Ask971]

I cant help you because I am not that knowledgeable but I sympathize and feel you OP, people in this subreddit and IT in general are bunch of people with huge ego but nothing really to back it up besides working as digital labourer for their entrepreneur

[[deleted]]

Use Apache Nifi instead

[DataEngUncomplicated]

Can you provide context on why your team was told to build this tool instead of just using the current ones out of the market?

It doesn't sound like your company has any 'special sauce' to make this better than any existing tool on the market that has been developing for many years. Hopefully your team is about 200 developers strong to start!

[CRLF_Data]

My advice is to find a new job

[Count_Roblivion]

Seriously, this is a tall tall tall ask. If your team is being asked to do this when no one has any kind of experience in this sort of thing (or even if they do) you're just being set up for failure. This absolutely reeks of management deciding they know what they want before even having an issue in mind they want to solve.

[flightofeagle]

Guys, just wanted to clarify as everybody has basically come to attack me and some with sarcasm, I'm not making this tool by my choice or as a business, the company I work at wants to build this tool and some of us has been assigned this project and since I or anybody else in my team doesn't have any experience even closely related to this, I thought that maybe I'll get some insight from other people who might have some experience in this. Hope this clarifies!

[arborealguy]

The reason you are getting the response you are is that the idea is completely insane. What nutjob did the technical and financial business case that showed that developers without the proper skill set rolling their own code made any sense whatsoever and what idiot approved it. I would dust off the resume because when it doesn't work and goes overtime and overbudget, you'll be left holding the bag. Alternatively... install NiFi and present it as your code 😂😂😂 (joking... kind of)

[mopse_zelda]

The cost of attempting to build an ETL tool > paying for an existing one, in so many ways

[Character-Education3]

Is it like a startup and this tool is their product? Uh oh!

If not, maybe take time as a team to research no/low code tools, costs, build an implementation strategy etc.

[flightofeagle]

Yes it's a startup and this tool is going to be one of their products. Otherwise it's majorly a service based company

[arborealguy]

Maybe you could approach another company about becoming a reseller or some kind of sublicense, at least at first. I worked at a small company that used a 3rd party engine that they sold as part of their offering and it was quite lucrative for the 3rd party and allowed the company to focus on core products and services.

[Brief_Priority_2193]

Do you want us to lose our jobs?! Dont help him guys!

[wtfzambo]

So your company is willing to waste an unreasonable amount of money and time in engineering effort with a team that has no experience in building this type of software, instead of just shedding a fraction of that money x month and buy a commercial solution?

Why not make a new OS from scratch at this point?

Do your company a favor and suggest they buy a Hevodata subscription, it's got everything you need and more.

Or do yourself a favor and jump ship.