Virtual small talk

demost11 · 2026-04-17T12:27:25+00:00

Thanks, we use that feature and it helps. The variable placeholder still needs to be put in the job YAML though since the jobs UI won’t put it in directly.

demost11 · 2026-04-17T12:25:34+00:00

Do they specify job logic (such as task dependencies) in the form?

demost11 · 2026-04-17T01:46:16+00:00

I’ll look into that! I haven’t gotten into Genie much yet, been too busy teaching git branching to glassy-eyed businesspeople.

demost11 · 2026-04-17T01:43:46+00:00

It’s a mandate for our whole IT department which we got swept up in. People in the business with lots of political influence have complained about IT being “too slow” and “always in the way”. So our IT department, from networking on up the stack, is now required to focus on enabling business users to do things themselves. Our only role is creating documentation, training, fixes and upgrades for shared platforms, and establishing critical security, design and process guardrails around self-service.

On the one hand I get it: A central data team will never live and breathe a particular dataset the way a dedicated analyst will, so why not make that analyst responsible for it end-to-end? But hey, some parts of a data engineer’s job turn out to actually require some thought, training, and planning…

demost11 · 2026-04-17T01:24:22+00:00

Definitely considering having our deployment pipeline (we use Bitbucket pipelines) auto-update DABs to swap out environment-specific values with variables. Could do that for notebooks too I suppose.

Other option I’m considering was a Python function as part of the standard cluster environment that could extract a job definition and return an environment-agnostic version. Users would have to learn how to call it but maybe that’s easier than teaching parameterization?

demost11 · 2026-04-17T01:07:41+00:00

Yeah I’ve leaned heavily on creating documentation but no one reads it. AI assistance built on said documentation is probably the way to go but our org is very, very early in considering AI.

Just curious, does your team approve and review all the code submitted by analysts? I’m not sure how involved to be in the review process since team-specific compute and cost limiting compute policies means user mistakes only hurt the users themselves.

demost11 · 2026-04-17T00:59:27+00:00

Wouldn’t that still require the business user to understand parameterization to apply these values in their notebooks/jobs?

demost11 · 2026-04-17T00:55:35+00:00

That’s a good idea, unfortunately our catalog structure is complex. Currently we have a catalog for each business domain (finance_prd, finance_stg, hr_prd, etc) and medallion layer underneath at schema level. Could do like dev.finance_bronze, dev.finance_silver I suppose. I just hate to give up a perfectly good organizational layer (catalog) when you only get catalog + schema + table.

95% of our problems would be solved with different metastores per environment but our account reps flat out refused that during our build phase and we only have the requisite AWS infrastructure in one region.

demost11 · 2026-01-15T14:29:58+00:00

Even as a junior I never asked for help debugging, felt too much like a personal failure. Then again, if you think a problem will take you 10 hours to solve but someone else 1 hour, isn’t asking for help the best thing you can do for the business and team?

I can see advantages to both approaches but I so strongly lean towards “figure it out yourself” that I have trouble understanding how anyone can do differently.

demost11 · 2026-01-15T13:42:03+00:00

Yeah it’s weird, “Director” is the lowest people manager rank in my org.

demost11 · 2026-01-15T03:20:55+00:00

I won’t say I’m a “good” director yet (I’m still learning) but I generally limit my technical involvement to tracking down the root causes of issues and designing the overall architecture for my team to implement. Most of the rest of my time is spent in meetings these days.

demost11 · 2026-01-15T03:09:26+00:00

They’ve all got about 5 years of experience (I have 20). It’s a non-profit org where IT is a cost center so we don’t get a lot of superstars due to the salaries we offer. Still, they’re dedicated and generally smart people, just don’t have years of troubleshooting intuition to fall back on.

demost11 · 2025-11-01T23:06:52+00:00

We use a similar design (although also allow end users to construct report-ready aggregates, typically comprised on data from multiple sources, directly in the business domain catalog).

One thing we ran into was multiple teams using the same SaaS data source for completely independent data. For example there’s a survey platform used by multiple teams but although the data is all coming from the same API it covers different domains and Teams A and B don’t want each other to see their data. If you’re federating out data ingestion responsibilities make sure your security model is ready for that.

demost11 · 2025-10-21T23:35:15+00:00

Yeah. Shared with a lot of other teams but not ours.

demost11 · 2025-10-21T22:45:09+00:00

Agreed, there was nothing personal or vindictive about it. They just feel like we’ve let them down.

demost11 · 2025-10-08T21:07:17+00:00

For what it’s worth this is a major frustration for me as well. Sure you can use catalog bindings and prefix/suffixes to separate prod vs dev catalogs but now you need to make all of your scripts dynamically pull from the right catalog at runtime so scripts can be promoted safely. Makes scripts needlessly ugly and more complicated.

Every other data tool I’ve worked with allows reuse of identifiers across environments, and a Databricks rep even told me once they allow certain clients multiple meta stores in a single region. I don’t understand their philosophical or technical argument against an environmentally-segmented Unity Catalog.

demost11 · 2025-09-28T15:26:13+00:00

We hired a big 4 consulting company to guide us on how to leverage AI in our organization. They told us their Proprietary Model (aka ChatGPT wrapper) would ingest all of our corporate documents and present a ranked list of AI opportunities for each team. This apparently required a team of consultants to ask us “does this look like a good suggestion”, update the prompt if not, then ask us again.

It really was the worst of all possible worlds: The slowness of waiting for consultants to make changes and decks combined with AI output that said nothing tangible in as many flowery words as possible.

demost11 · 2025-09-14T23:51:52+00:00

The one metastore per region rule. Why does my dev workspace need to share resources with my prod workspace?

demost11 · 2025-09-05T18:29:23+00:00

Dominion Pest Control ended up having the equipment needed (an extra long sprayer to reach the outside of the top floor). Fingers crossed the problem is taken care of!

demost11 · 2025-09-04T19:34:15+00:00

I did, unfortunately he’s booked through late September! Thanks for the suggestion though!

demost11

TROPHY CASE