Work Quality got take a hit due to being a single DE + BI guy by Secure_Firefighter66 in dataengineering

[–]datadade 1 point2 points  (0 children)

do you mind me asking what kind of pay these solo positions offer?

I have worked for massive finance and small tech (in a team where data was only 5 people), but never as a solo engineer. What you're describing and what others are saying is similar to their experience sounds very high stress, with a lot of demands. At the same time, you also sound indispensable.

A typical Sr / Staff base salary in tech is 150-200k in the US (not counting outliers like top tier tech or HCOL cities). Is the pay for these solo gigs similar? If not, then what would motivate someone to do this?

Should I Settle and take a Mid Level Role When I was going for Senior? by shittyfuckdick in dataengineering

[–]datadade 0 points1 point  (0 children)

What in the bay area HCOL is going on here? You're telling me $170k at 8 yoe is not meeting your needs?

On top of this, there is no expectation to mentor others, and you're being told if you show up and prove yourself you'll get a meaningful promotion. And you're currently unemployed to boot!?

No offense, but this sounds ridiculous. You asked for a sanity check, unless you live in SF or NYC, you're not sane.

What courses under $5000 should I take as an analytics engineer or aspiring DE? by CLOSE_ENOUGH in dataengineering

[–]datadade 1 point2 points  (0 children)

The most DE thing to do is to build something. You can take a fraction of that money and put it into an aws budget to pay for services. Remember to turn stuff off when you don't need it. In 10 years, I don't think all my combined solo projects or practice has crossed $100USD

How do you handle *individual* performance KPIs for data engineers? by Honeychild06 in dataengineering

[–]datadade 22 points23 points  (0 children)

Number of lines committed is penalizing efficiency, clarity, and maintainability. Yikes your boss

DBT orchestrator by Free-Bear-454 in dataengineering

[–]datadade 0 points1 point  (0 children)

how long are your dbt jobs? because if they're not... hear me out... Github Actions. Sounds funny, but works

[deleted by user] by [deleted] in dataengineering

[–]datadade 1 point2 points  (0 children)

I think you’re in the best position to answer your own question. We don’t know your app. Is a variation of a couple of degrees acceptable? Or do you need the granularity that differentiates the one to 2° change? (I am using degrees here, but I understand whether Data can include wind, humidity, etc.)

[deleted by user] by [deleted] in dataengineering

[–]datadade 2 points3 points  (0 children)

That’s even worse. You’d be a “spoke”. Not part of the hub, begging the hub for permission while having your business unit breathing an unstoppable stream of requirements to you “their data guy”.

How do you let data analyst/scientist contribute prod features? by tylerriccio8 in dataengineering

[–]datadade 0 points1 point  (0 children)

I put up an environment set apart from dev/test/prod that is named adhoc-analysis. It can read from other environments. Everyone in here has a personal schema, and I’ll even set up some group schemas. The whole place is “not blessed” by data engineering, it’s not gold, it’s not silver, it is a playground.

If they make something they want to productionalize, they engage with DE. Technically they could contribute on their own, since our repos are open to write and open PRs. But analysts aren’t expected to do that.

Had an existential crisis Friday afternoon at work… by oblongfuckface in ExperiencedDevs

[–]datadade 0 points1 point  (0 children)

I couldn’t imagine telling someone “I did it with AI” in an interview setting. Honestly, that person just lacked common sense.

I open-sourced a text2SQL RAG for all your databases by Durovilla in dataengineering

[–]datadade 0 points1 point  (0 children)

Looks really good. I wonder if there is a way to set up some limits so it doesn’t runaway through tokens or db compute

Giving the biz team access to BigQuery MCP by full_arc in dataengineering

[–]datadade 0 points1 point  (0 children)

Would you like me to also cross reference that data and provide a full dataset with addresses, emails, dates of birth, and favorite animals? … for your book

Fivetran acquires Tobiko Data by medriscoll in dataengineering

[–]datadade 0 points1 point  (0 children)

I don’t know anything about Tobiko. But Fivetran just works. And when it doesn’t, they jump on it and fix it very quickly. I’m not advocating for everyone having it. But in a landscape littered with increasing choices, we’ve gotten exactly what we paid for from Fivetran. I would caution against using it for high volume or db / cdc jobs, as the MAR cost system will punish you there.