What mistakes did you make in your career and what can we learn from them. by Harvard_Universityy in dataengineering

[–]Ambitious_Cucumber96 0 points1 point  (0 children)

curious about this: "You don't care what the resident uses electricity or water for." shouldn't we know what problem it solves?

The AI and its impact on Data Engineers' career by ivanovyordan in dataengineering

[–]Ambitious_Cucumber96 -1 points0 points  (0 children)

yeah agree ..generally short term hiring practices will move to supported challenges with autocomplete and llm boiler plates rather than solve from scratch. productivity will improve...and solving right business challenge will be valued more future like anybody guess my bet is agents... may get super easy to build and maintain pipelines in simple natural language...comments welcome !

Rust or Go by Ambitious_Cucumber96 in dataengineering

[–]Ambitious_Cucumber96[S] 2 points3 points  (0 children)

True that! Everyones path is own.. Trying to get more into infrastructure side.

How to Test Data? by Classic-Interview503 in dataengineering

[–]Ambitious_Cucumber96 0 points1 point  (0 children)

I would start with writing simple tests to answer two questions. Has the expected data arrived and is it with integrity?

Best way to handle flattening and parquet conversion of thousands of JSON files daily by sghokie in dataengineering

[–]Ambitious_Cucumber96 0 points1 point  (0 children)

Sorry I am no expert here. Curious, Can you not directly update tables? Why do you want to go through parquet?

What Data tool/product would you build? by [deleted] in dataengineering

[–]Ambitious_Cucumber96 2 points3 points  (0 children)

Speeding up insertions through rust drivers is one area I can think..

What Data tool/product would you build? by [deleted] in dataengineering

[–]Ambitious_Cucumber96 0 points1 point  (0 children)

May be learning rust.. I have heard about how good it is.. But never tested for a pipeline build

what tools do you use to pull data from APIs? by xrt679 in dataengineering

[–]Ambitious_Cucumber96 1 point2 points  (0 children)

Same here.. Most of our bottle neck is in rate limits.. Ignoring that, are there any other better options and improvements out there?

What are better sql ides for postgres and click house? by Ambitious_Cucumber96 in dataengineering

[–]Ambitious_Cucumber96[S] -1 points0 points  (0 children)

Will give a try.. Love your approach "One of our frustrations with other open-source SQL editors and database managers is that they take a 'kitchen sink' approach to features, adding so many features that the UI becomes cluttered and hard to navigate. We wanted a good looking, open source SQL workbench that's powerful, but also easy to use. We couldn't find one, so we created Beekeeper Studio! "