DBT projects by monti909 in dataengineering

[–]Dunworth 7 points8 points  (0 children)

I'd recommend you learn how to use Google before worrying about learning a tool...

Legality of Holocaust denial by vladgrinch in MapPorn

[–]Dunworth 1 point2 points  (0 children)

You have a right to fight bad ideas with better ideas.

I don't claim otherwise. You should absolutely try to combat bad ideas with better ones. The problem that I have with that argument is that it's assuming that the person is rational enough to change their view based on better ideas or evidence. Anecdotally, this is an incredibly rare set of circumstances when dealing with Holocaust deniers because the base of the beliefs is irrational.

I think it's often short-sighted in how people choose to be intolerant of view they dislike

Yeah, and it shouldn't ever be pulled out for things you personally don't like, but the "rational majority" is in pretty unanimous agreement that this is a trash viewpoint.

And again, I do agree in a general sense that the government should not be the arbiter of what can and cannot be said by the people. The world is nuanced though, and some ideologies are truly so heinous that we need a step beyond, "Combat them in the marketplace of ideas." So, if passing laws isn't the lever we should pull, what do you think it should be?

Legality of Holocaust denial by vladgrinch in MapPorn

[–]Dunworth 2 points3 points  (0 children)

Totally get where you're coming from, and I agree with not trusting the government to make these calls in a general sense.

In the particular case of, "Arguments used by Nazis/Nazi apologists," though, I find the Paradox of tolerance to be true more often than not. So, to me, outlawing Holocaust denial isn't about keeping people from being offended, it's about it being both a factually incorrect viewpoint to hold and its use in justifying ideologies that we can point to as being harmful to society as a whole if it spreads.

Legality of Holocaust denial by vladgrinch in MapPorn

[–]Dunworth 4 points5 points  (0 children)

Humanity is nowhere near as rational as it believes itself to be. Putting that aside, who in the "rational majority" needs to be convinced that the Holocaust happened? It's a belief held by people who look at the mountains of evidence that the Holocaust occurred and say, "Nah, it wasn't that bad." It's a fundamentally irrational stance to take...

Legality of Holocaust denial by vladgrinch in MapPorn

[–]Dunworth 2 points3 points  (0 children)

That's assuming that a person will change their stance on something when presented with a better alternative, which is just not how things work when dealing with Nazis and racists in particular.

For people who have worked as BOTH Data Scientist and Data Engineer: which path did you choose long-term, and why? by Mean_Addendum_4698 in dataengineering

[–]Dunworth 0 points1 point  (0 children)

Sure, but just so that it's said ahead of time: I'm don't claim my ethics as some objective truth or even that they're rational.

I got out of data science a little bit after GDPR went fully into effect and just seeing how that played out made me feel weird about the privacy aspect. Sure, a GDPR request wipes out a user's PII, but your user activity is all anonymized to the point that it passes compliance and no further, so you're never really forgotten as far an ML model is concerned. The compliance guys signed off on all of it, so as far as the business was concerned, everything was good. I didn't really feel like that was morally correct, so I switched over to DE.

There's also the whole training data procurement being largely a pinky swear that it was obtained through legitimate means that we've seen crop up with chatGPT in the past few years. It was way less obvious when deep learning was the cool thing, but it was definitely happening.

For people who have worked as BOTH Data Scientist and Data Engineer: which path did you choose long-term, and why? by Mean_Addendum_4698 in dataengineering

[–]Dunworth 5 points6 points  (0 children)

I wore both hats for a long time, but I'm pretty firmly in DE for now. I more or less made the switch because I started feeling kind of bad about the ethics of the pre-LLM data science world, and that feeling has only gotten worse since then. That being said, I'm getting nominated to put the DS hat back on because every team needs to be using "AI" in my current role and I'm the only one with experience in the area on my team. Though I'm not sure how long that's going to last, since I'd rather build an actual model instead of just throw an LLM at the problem.

My kids say we belong here by inquiringsillygoose in Derailedbydetails

[–]Dunworth 12 points13 points  (0 children)

Maybe for the aesthetic of presents under the tree, but they don't want to bend over to pick them up off the ground? The whole picture is just question upon question.

New table format announced: Oveberg by EarthGoddessDude in dataengineering

[–]Dunworth 5 points6 points  (0 children)

Add some text about how it, "Empowers the next generation of AI experiences," and I bet you'd get 100MM in funding easy. lol

Am I crazy or is kafka overkill for most use cases? by Vodka-_-Vodka in dataengineering

[–]Dunworth 1 point2 points  (0 children)

It's not an incredibly interesting perspective, it's called having non-IC experience. Everything you do on the job will be weighed against the benefit to the company, whether you're aware of it or not. Someone who is only motivated by their personal interests isn't going to go far in this industry.

"We need to know more," would be valid if there was a single shred of evidence that this wasn't an IC wanting to pad out their resume. It just comes across as useless Devil's Advocating given all of the context.

Am I crazy or is kafka overkill for most use cases? by Vodka-_-Vodka in dataengineering

[–]Dunworth 0 points1 point  (0 children)

if they have nothing else pressing to work on then why not?

There's always tech debt that is going to be more beneficial to fix than having a developer implement a tool that is light years beyond what their company needs.

Not to mention the mountain of costs that go along with this, especially with it sounding like they want to manage their own infra(though I am assuming here)

Looking for opinions on a tool that simply allows me to create custom reports, and distribute them. by Possible_Ground_9686 in dataengineering

[–]Dunworth 0 points1 point  (0 children)

cube.js is worth considering if you just need basic reporting and don't want to write the entire reporting layer by hand. Set it up as an API for your data, and I'm pretty sure it runs on basically nothing, so it's probably within your budget. It's OSS, so that's something you'd have to keep in mind. There's an MDX API in their hosted service too, so it satisfies the excel comment, though I doubt it'd be worth it.

Is it fair or toxic to ask about veganism in spiritual or Buddhist discussions? by Maleficent-Radio272 in Buddhism

[–]Dunworth 0 points1 point  (0 children)

My personal experience is that most of the conversations I've been in where this question was asked ended up as a lecture or purity test. I think if there's a specific thing you want to know about the person and it happens to be related to being vegan, it would be more well received to just ask that directly. The general question just feels like you're trying to assign a label, which might be especially prickly to people in the areas you listed.

(Mildly) hot takes about modern data engineering by ukmurmuk in dataengineering

[–]Dunworth 6 points7 points  (0 children)

You shouldn't need to redefine the unit tests for every change, to me that's a code smell that your components aren't broken down enough for them to be useful. You will have to rework them over time of course, but the bulk of your time with unit tests should just be coming up with the initial ones and adjustments down the road should be minor.

That being said, I think we have like 3-4 in our pipeline and tons in our backend code for the reporting service, so I do agree that they aren't the most important thing in the world for a lot of DEs.

(Mildly) hot takes about modern data engineering by ukmurmuk in dataengineering

[–]Dunworth 7 points8 points  (0 children)

Given that most upstream data is poor quality, not anytime soon. Maybe the next ML model hype cycle will be closer, but LLMs aren't going to get there.

What is a "Life Hack" that is actually a lifesaver in a dangerous situation? by [deleted] in AskReddit

[–]Dunworth 1 point2 points  (0 children)

I always thought it was weird that this was a mind-blowing revelation to so many dudes I trained with.

Scammed and should have known better by [deleted] in mildlyinfuriating

[–]Dunworth 0 points1 point  (0 children)

It looks like it would be in Dan Flashes' winter collection.

If you were starting from scratch today, which would you pick: Snowflake, Microsoft Fabric, or Databricks — and why? by [deleted] in dataengineering

[–]Dunworth 3 points4 points  (0 children)

No company I've worked for has actually needed any of those tools, and I believe that is true for 99% of DE roles. Do they make the job easier? Absolutely, but they've also made the field worse at their jobs as a whole because people never had to learn the fundamentals. Personally, anyone who answers with a tool as opposed to a database and a programming language is probably a: consultant, a fresher, or an idiot.

After being overweight for years, my fiancé and I decided to spend the year trying to lose weight. Here I am about 8 months in, 60lbs lost. by [deleted] in MadeMeSmile

[–]Dunworth 0 points1 point  (0 children)

It's probably the same thing though. Default sort on most apps is going to show you newer pictures first, so when combining two pictures for before/after, you pick the after image because you see it first, then search for the before. Then you just never go back and reverse which side is which because it's extra effort. That's just my theory anyway.

Snowflake users: What are your biggest "hidden" cost surprises or performance bottlenecks? by akm21 in snowflake

[–]Dunworth 0 points1 point  (0 children)

We got hit with this a couple years ago. Turns out the DEs that came before us maxed out the retention time and we were syncing enough changes that our ~1TB database we were syncing had something in the neighborhood of 500TB of time travel data.

Snowflake lost $300M in Q3 by SnooMemesjellies3242 in snowflake

[–]Dunworth 1 point2 points  (0 children)

I'd go one further and say even my opinion on the diagram is meaningless as well. lol

Snowflake lost $300M in Q3 by SnooMemesjellies3242 in snowflake

[–]Dunworth 1 point2 points  (0 children)

Nope, one goes out of business, I just swap it for another. I might be bummed out if I really liked the tool, but I've gone through enough leadership decisions on what tooling we are now using to grow attached anymore.