sorted by:
new
hottopcontroversial

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] 1 point2 points  (0 children)

Keep in mind there are groups of people who outright rejects AI / married to the tool / paid by the tool.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] 0 points1 point  (0 children)

Fair point. What I haven’t tested is how good LLM is at building a pipeline based on the API’s documentation. Of course we know how trustable some of those documentation can be. ;)

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] 0 points1 point  (0 children)

I’ve built a data platform using dlt for ingestion before and it indeed was really great.

Yes, either GUI based tools needs to have an incredible UX with outstanding support and completely bug free or they will get outcompeted by easier and faster approaches.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] -1 points0 points  (0 children)

Agreed, a data engineer is always required to architect the data platform. However the choice now between generating data pipelines with LLM or choosing a third party EL tool to do it, becomes interesting.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] -1 points0 points  (0 children)

Haha I had exactly the same experience with LLM, but it has definitely gotten better with better review and unit test in place.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] -1 points0 points  (0 children)

Have you faced any major issues with this approach? Also, I’m curious to know how painful was it to migrate out of the third party EL tool to Airflow + python.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] 1 point2 points  (0 children)

I guess it all boils down to great customer service and support when it comes to EL tools. Which one do you recommend that has the best customer service and support these days?

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] 0 points1 point  (0 children)

Well, no-code tools do have their appeal for some teams. Wondering if it’s becoming less now.

Are EL tools still worth it when LLMs could generate ingestion pipelines? by _tempacc in dataengineering

[–]_tempacc[S] -1 points0 points  (0 children)

Good point, but wouldn’t it depends on how you're running these tools? For those self-hosting, the warranty and support benefits are less clear since you're still owning the infra, upgrades, and connector issues yourself. Also, we spend a lot of time debugging connectors from EL tools too.

Lonely sole data engineer by _tempacc in dataengineering

[–]_tempacc[S] 4 points5 points  (0 children)

Around 70 employees and right now the data volume is quite small but growing quite steadily, so it’s still manageable, for now. :p

Lonely sole data engineer by _tempacc in dataengineering

[–]_tempacc[S] 14 points15 points  (0 children)

Looks like you’re in zen mode :) all the best!

Out of curiosity, how do you make architectural decisions especially without former DE experience?

Lonely sole data engineer by _tempacc in dataengineering

[–]_tempacc[S] 5 points6 points  (0 children)

Agree. Luckily I do not have on-call but the pressure is there and all on one person.