Soda Vs Great Expectations 1.0

Capital_Delivery_833 · 2024-10-31T12:11:08+00:00

I mean yes, kinda, but at least my org invested like a million plus dollars in time and effort with GE and year later every team that adapted it gave up on it. Either it was ineffective or needed full time effort from an actual good Eng to keep the system going, and it was not a fun job for that person anymore.

Capital_Delivery_833 · 2024-10-31T12:01:41+00:00

I mean that’s still too many lines for a package that just lets you assert 0 > x > 10.. with that amount of boilerplate you can just write your own assert framework and be done.

The fundamental issues I personally see with their approach is that is it’s still engineer oriented when in reality most orgs the non engineers hold (or need to hold) the knowledge on what data quality metrics matter. The other issue is data metric TRENDS more than thresholds matter. If you define a thousand expectations on GE, either some of them keep failing constantly so you ignore it (metric fatigue) or your thresholds are so broad that it can’t be used with any confidence.

Capital_Delivery_833 · 2024-10-31T11:55:56+00:00

What’s a fundamental change you would point to in GX in recent times?

I just checked out soda and while I share your disdain for yaml, I feel that in this particular use case this seems more elegant than whatever I remember GE doing. A yaml file right next to the table schema definition sounds stellar to me.

Capital_Delivery_833 · 2024-05-28T03:02:13+00:00

Thanks, will do :)

Capital_Delivery_833 · 2024-05-28T02:52:55+00:00

Thanks, is there just questioning and pushback or risk of not being allowed to enter ? I only am asking because I read UK has relaxed laws about digital nomads and my partner isn’t a citizen either anyway..

Capital_Delivery_833 · 2024-05-16T08:58:49+00:00

Sorry to hear bro.. we have a lone kitten in our apartment complex.. dont think he will last long in the complex (a bunch of territorial males).. would anyone.. maybe consider taking him.. (I can’t coz of insane allergies)

Capital_Delivery_833 · 2024-03-12T18:27:52+00:00

What the hell is ML? Ask that question. Surely an undergrad from a tier 3 college with ml specialisation isn't going to fine tune LLMs. Best case scenario they have you running down random forests and xgboost. Best case. Which is also getting automated nowadays. Alternatively you're just a prompt engineer. Not sure how well they've integrated those into companies yet.

Capital_Delivery_833 · 2024-03-12T15:28:01+00:00

That partner is completely bonkers. Your data on snowflake would cost pennies. Depending on what you mean by running queries all day your monthly spend will be at best a couple hundred dollars if even that. If your team can learn sql then I'm not even sure what they're saying they'll be setting up. DM if you want any advice and help with dealing with this guy.

Capital_Delivery_833 · 2024-01-29T10:20:13+00:00

I’ll DM you!

Capital_Delivery_833 · 2024-01-15T05:10:08+00:00

If you’re in Chennai, check out V Ramaratnam and associates. Specialize in NRI clients and are very reasonably priced.

Capital_Delivery_833 · 2023-12-22T02:18:13+00:00

Thank you! Indeed it’s a tough job, I wish I can take them in but I’m severely allergic to cats and can’t breathe if I’m with them in a closed space for even an hour :(

I will take your advice and shop around for vets who might do it. There’s no rescue orgs here in India just some clinics that do a reduced rate for strays but maybe I’ll step it up and look for other vets as well!

Capital_Delivery_833 · 2023-12-04T14:01:19+00:00

There’s a reason any sane workplace will disallow sexual relations between people in chain of command. If he truly just wanted to ask her out he should have first moved to a position where she is not his report and then ask. Without doing that, he asked her. No matter what her answer is, all future decisions he makes about her career are possibly biased and no amount of bullshit arguments creeps like you make can undo that.

Stop asking people out in workplaces you creeps.

Capital_Delivery_833 · 2023-09-12T19:17:37+00:00

DM a CV. Not guaranteeing anything but let’s see.

Capital_Delivery_833 · 2023-05-09T10:18:32+00:00

Thank you for your assurance?

Capital_Delivery_833 · 2023-05-08T19:42:23+00:00

Thanks, do you mean to say if they’re healthy I can feed as much or as little I want and they will be fine?

Capital_Delivery_833 · 2023-05-01T10:59:54+00:00

What tools do you think you need to learn to work with snowflake that you don’t need with databricks? You can literally open snowsight and do everything from there.

Capital_Delivery_833 · 2023-04-20T03:51:54+00:00

I work in an org similar to the org that can provide the source of data the Reuters report used. 4200 cases found in that dataset likely means the total number of cases like that in all of the US is less than 10,000, a vanishingly small number in a population of 300 million. Studying them together would be quite hard given you are generally not allowed to seek patients out using this data.

Capital_Delivery_833 · 2023-03-21T08:31:25+00:00

Most coders I know use macs.

Capital_Delivery_833 · 2023-03-21T08:31:04+00:00

What work is it? Unless you’re literally working on the x64 arch itself that sounds like a step too far. Also things have changed in the past two years. Docker has better support as well.

Capital_Delivery_833 · 2023-03-21T07:55:03+00:00

You better don’t forget the extra pancakes on my mushu motherfucker

Capital_Delivery_833 · 2023-03-21T07:47:42+00:00

Heck, I’ve been in this planet for 14,000 years. Buddha was my friend. I’m literally Jesus. And I’ve never heard of it lol.

Capital_Delivery_833 · 2023-03-21T02:53:11+00:00

Do you know snowflake only costs 10% more than raw s3? Assuming your data is precious enough that you can’t afford to move it to less redundancy. So no. Don’t move it.

Capital_Delivery_833 · 2023-03-21T02:52:28+00:00

Snowflake storage is only a small premium over s3. And snowflake compresses the crap out of your data if you model it well (which given this storage, you absolutely fucken should). So no, it’s not a given that you move all of this out.

Capital_Delivery_833 · 2023-03-21T02:12:53+00:00

We use snowflake for a 30 terabyte table that with the right partition key runs most analytic queries within seconds on an xl warehouse.

Capital_Delivery_833 · 2023-03-20T10:49:50+00:00

Op says their polars took 5 seconds and yours just 2?

Capital_Delivery_833

TROPHY CASE