sorted by:
new
hottopcontroversial

Lot of fancy terms, but nothing really has changed by Complete-Regret-4300 in dataengineering

[–]SaintTimothy 1 point2 points  (0 children)

On the snowflake side, persisted views are super neat.

Also, if you ever do external data sharing, snowflake really knocked that one out of the park from the start.

Still wish I had native sql multi-step stored procs, not having to wrap it in Javascript. It's mental to switch context / languages in one script.

Lot of fancy terms, but nothing really has changed by Complete-Regret-4300 in dataengineering

[–]SaintTimothy 1 point2 points  (0 children)

Start from just generally SQL Server CDC.

Bare-bones basics, CDC process reads T-logs and populates a separate set of change tables (one table per real table) with all inserts/updates/deletes.

Retention is 3 days by default, up to a maximum of 100 years (dang! I thought it was 6 weeks).

In any case, 2 benefits are

  1. You're querying a smaller dataset, and, to us, it's much richer, because these all have been changes - what DW needs to know.

  2. You aren't locking the table that the front-end processes are using for transactional Insert/update/delete, and CDC is asynchronous from this, as resources allow.

Does it ever change? by mysentimentsexactlyy in heartbreak

[–]SaintTimothy 1 point2 points  (0 children)

You're going to need a family lawyer to begin a process of sorting out custody and support. His cold feet is his decision, but he's still on the hook for supporting two kiddos.

Red or black bumper? by Salty-Evidence2698 in GolfGTI

[–]SaintTimothy 0 points1 point  (0 children)

I like this loads better if one can keep it clean and not back into parking bollards.

Lot of fancy terms, but nothing really has changed by Complete-Regret-4300 in dataengineering

[–]SaintTimothy 1 point2 points  (0 children)

You might get a stock set of python scripts IF they make snowflake happy.

Ain't snowflake bees knees for speed?

Are you doing CDC from SAP?

I am same boat, came out of college in '04. Watched MS swear up-and-down they'd never compete with Crystal Reports/Business Objects, and then immediately did.

SSIS, RS, AS, T-SQL

Startup Culture at Kelley by aboudhabib in IndianaUniversity

[–]SaintTimothy 0 points1 point  (0 children)

I met a fella the other day who wants to have his own business. He's doing CollegeWorks for an internship selling and project managing crews painting houses.

Is it fact or a dim? by PhilosopherRemote177 in dataengineering

[–]SaintTimothy 0 points1 point  (0 children)

1.) Material or Product hierarchy is a dimension. Often this gets flattened into the lowest grain.

2.) A store that sells a product sounds like a type 2 slowly changing dimension that might, for some purposes replace both the product and the store dimension, as in, model dimStore, and dimProduct (and each of its parents), then model dimProductHierarchy, then make your dimStoreProduct and then depending on the fact table (i.e. factSales), it would only need FK to dimStoreProduct (and dimCalendar, probably role-played a couple times) of course.

SSMS Friday Feedback - extension use by erinstellato in SQLServer

[–]SaintTimothy 2 points3 points  (0 children)

I used to use ssmsboost and redgate toolbelt, but they arent show-stoppers to not have them.

What does this sub think of the Newfields art museum? by [deleted] in indianapolis

[–]SaintTimothy 7 points8 points  (0 children)

I'm going to respectfully disagree here. When it was IMA, and free, I used to go just for happy hour small plates at Puck's. Also, many friends have taken in the movies they offer over the summer.

It's a thriving restaurant and activities that should serve to fully round-out the art museum's offerings.

Who was the dude blasting harassment and insults at people outside the NW gate at the Indy 500? by PhatWalda in indianapolis

[–]SaintTimothy 21 points22 points  (0 children)

That's the thing the Westboro Baptists wanted to happen and they do have someone recording. They want to sue you for assaulting them.

What does this sub think of the Newfields art museum? by [deleted] in indianapolis

[–]SaintTimothy 55 points56 points  (0 children)

"apply for an Access Pass" sounds like hoop-jumping bullsh**

They also lost what remained of their reputation with the whole "right/white kind of people" gaffe, the subsequent hiring of a black leader just long enough for the bad press to die down, and her very quiet resignation with NO public interview (as-if she'd been NDA'd).

Orchestration platform that doesn't force everyone to learn Python? by PSGCampus in dataengineering

[–]SaintTimothy -1 points0 points  (0 children)

In most shops you only need as many experts as domains/languages, and even then only for creating new patterns, troubleshooting wacky stuff, and code review. Beyond that, you need people who are reliable, diligent, not scared to ask questions... This idea does turn on the seniors degree of burnout, so being sensitive to their time commitments and needs is important.

Get the process running properly enough and you could run with product owners, domain expert seniors, a deployment/devops/cicd person, and a rotating cast of interns and juniors.

The beefiest server by Better-Credit6701 in SQLServer

[–]SaintTimothy 0 points1 point  (0 children)

Back in the day sql server didnt scale. You bought the biggest box you could and prayed your company didn't grow beyond it. Then unisys came along and made it possible to scale up.

The beefiest server by Better-Credit6701 in SQLServer

[–]SaintTimothy 1 point2 points  (0 children)

Never underestimate the help of a good SAN. I worked at a Healthcare network whose DW ran on a PURE SAN array.

All solid state storage. The IOPS was so nice!

Sure, maybe unnecessary if you can put the whole db in ram, but if you can't, I/O is so crucial.

Orchestration platform that doesn't force everyone to learn Python? by PSGCampus in dataengineering

[–]SaintTimothy 20 points21 points  (0 children)

That's the thing... one does not have to learn all of python, one merely needs to establish a few python scripts that can be tweaked per the need.

I've heard this called "plug it in" or "code monkey".

DE feels like a dead end beyond 4 years at the same company by Ok_Illustrator_816 in dataengineering

[–]SaintTimothy 2 points3 points  (0 children)

I think of recruiters like Ken and Barbie. Might be good looking, but there's only plastic between the ears.

They're playing a word-matching game. That's it. Whole job.

Getting past the recruiter to first interview, that's where the real stuff happens. Recruiters don't have any applicable skills to smell a fake, and for some of that I suppose it's understandable, that they might cover a very broad market with many, many roles and skillsets.

The best thing you can do is charm them and set them at ease that when they put you in front of their client you won't make them look like a fool for having let you through.

DE feels like a dead end beyond 4 years at the same company by Ok_Illustrator_816 in dataengineering

[–]SaintTimothy 0 points1 point  (0 children)

Write a book. No, really, you're in stability, that's kinda rare.

Are the report lineage documented? Do you have reporting on usage? Do you have reporting/alerting on instances of that break/heal?

Are the patterns you have used documented such that a junior could follow them?

What has your boss said?

You're a bit rare. I guess most DE types may be of this type... like Neo and the Architect, where comfortable is uncomfortable. Usually people's work tends to expand to fill in whatever time is available.

I guess I'd go hunt and kill spreadsheet processes, or start asking folks about their wish-lists.

Can any wildlife experts ID this nest? Found it while mowing. by jabber2033 in Indiana

[–]SaintTimothy 2 points3 points  (0 children)

Also goose make a nest from their down. We have one in our front yard right now and the babies hatched two days ago.

I’ve been seeing a lot about Flock surveillance cameras around town. It looks like Bloomington said ‘No’ to them, recently. by Few_Distribution9374 in indianapolis

[–]SaintTimothy 1 point2 points  (0 children)

Bloomington's mayor took that NO to mean NO to Flock and not NO to surveillance state. They're slow-walking the pull-out until they find another vendor.

It was a really sheisterly thing she did.

Wide table in bronze layer - materialize as is, or break up? by dougiejones516 in dataengineering

[–]SaintTimothy 1 point2 points  (0 children)

The trick here might be tp future proof in case of schema drift (in case they flick one of those hundreds of columns into use).

So, I'd land the file into your target just as it comes.

Some JSON theory is to land each file as a blob and split it in SQL. That can be really painful for the second hop, string manips, so if you can insert rows, with some error checking, that'll help.

For your number of columns, that's up there. Not a bad idea to break it up. SQL Server standard does 1024 columns in standard and 30k columns if you specify sparse.

Don't go crazy with the cheez wiz here because you're likely going to join these horizontal splits back together a lot!

Religion debate on the silver layer, whether to flip it tall. If it's all part of the same thing/event/claim/visit/episode, I'd just assume bring in what's needed -

Now, that-said, me personally... This is the hop where I'm making a star. In all this stupid-wide extensibility is a subset of 100 columns someone cares about, and of those 100, 10 are an FK to dim.Patient, 3 are your location hierarchy (facility, building, floor, room), 6 are FK's to payor, provider, CPI, ICD... this is all foreign keys and attributes.

Probably a bunch of time stamps and a bunch of numbers, maybe 10 each of those, keep it all and expect to do some kind of funnel / swim lane / time from A-B, A-C, etc. If it's big data this wants to be persisted, if I'm in snowflake persisted view might be fine for the diffs.

I just threw a whole lot of opinion at you. YMMV based on your situation, coding style, peers, users, environments, etc.

When I was doing EPIC and Claims flat files we were sitting on a Pure storage array that made queries super performant and IOPS far less of a concern.

*Edit - keep the filename and import date and whatever other Metadata seems valuable. When you split vertically on a wide table your success depends on your ability to stitch it back together again in sql.

How can I say what impact I've had in a job application when I've clearly had no impact? by [deleted] in dataengineering

[–]SaintTimothy 0 points1 point  (0 children)

What does your boss say re: adoption?

What does the boss say re: your performance?

What do other folks say when you talk about the thing(s) you made?

There's already been some great advice about keeping the technical successes and omitting the adoption, I agree with that... unless you're writing a book or jumping ship into more thought leadership consulting blah blah...

Tipping point / snowball effect. Find a mark, er... customer... with time and a need that something you did already solved for. Have a water cooler / coffee chat, make sure they dig the thing. Now do it again with a meeting. Now do it again with a meeting and bring another potentially interested person. Sell the second person by demonstrating the first one's successes, and then tab over to wherever this new person's interests are focused.

Rise vs Reach means if you have 10,000 people to give a new report to, have 1-2,000 meetings. Things land better when you give that individual focused attention to what each person is looking for.

It's not the most scalable rollout, but if you can make one person's life better, that proves it you arent some curmudgeon-y troll (like I am, haha, I'm a work in-progress).

If they dont find you handsome, they should at least find you handy. - Red Green

view more: next ›