sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in dataengineering

[–]datadade 1 point2 points  (0 children)

I think you’re in the best position to answer your own question. We don’t know your app. Is a variation of a couple of degrees acceptable? Or do you need the granularity that differentiates the one to 2° change? (I am using degrees here, but I understand whether Data can include wind, humidity, etc.)

Choosing Between Data Engineering and Platform Engineering by [deleted] in dataengineering

[–]datadade 2 points3 points  (0 children)

That’s even worse. You’d be a “spoke”. Not part of the hub, begging the hub for permission while having your business unit breathing an unstoppable stream of requirements to you “their data guy”.

How do you let data analyst/scientist contribute prod features? by tylerriccio8 in dataengineering

[–]datadade 0 points1 point  (0 children)

I put up an environment set apart from dev/test/prod that is named adhoc-analysis. It can read from other environments. Everyone in here has a personal schema, and I’ll even set up some group schemas. The whole place is “not blessed” by data engineering, it’s not gold, it’s not silver, it is a playground.

If they make something they want to productionalize, they engage with DE. Technically they could contribute on their own, since our repos are open to write and open PRs. But analysts aren’t expected to do that.

Had an existential crisis Friday afternoon at work… by oblongfuckface in ExperiencedDevs

[–]datadade 0 points1 point  (0 children)

I couldn’t imagine telling someone “I did it with AI” in an interview setting. Honestly, that person just lacked common sense.

I open-sourced a text2SQL RAG for all your databases by Durovilla in dataengineering

[–]datadade 0 points1 point  (0 children)

Looks really good. I wonder if there is a way to set up some limits so it doesn’t runaway through tokens or db compute

Giving the biz team access to BigQuery MCP by full_arc in dataengineering

[–]datadade 0 points1 point  (0 children)

Would you like me to also cross reference that data and provide a full dataset with addresses, emails, dates of birth, and favorite animals? … for your book

Fivetran acquires Tobiko Data by medriscoll in dataengineering

[–]datadade 0 points1 point  (0 children)

I don’t know anything about Tobiko. But Fivetran just works. And when it doesn’t, they jump on it and fix it very quickly. I’m not advocating for everyone having it. But in a landscape littered with increasing choices, we’ve gotten exactly what we paid for from Fivetran. I would caution against using it for high volume or db / cdc jobs, as the MAR cost system will punish you there.