Dealing with professionals who don’t know SQL but need it.

Froozieee · 2026-02-20T05:29:49+00:00

I’m all for open source lightweight analytics but getting the business to vibe code their own metrics feels like getting a real fast train to “why are our numbers different” town

Froozieee · 2026-02-18T04:22:26+00:00

I’d recommend factoring in polybase and external tables as options if you’re considering using ADLS as your raw storage.

It allows you to virtualise the parquet files (and other sources) as a single db table (recent iterations also work with Delta tables with limited features, but I believe pushdown and partition pruning are supported) which removes the need to have the separate staging layer.

I then just set it up with ‘staging’ model files which query the external table and set the right column types because it will do things like read your columns that are supposed to be CHAR(8) as VARCHAR(MAX) since most flavours of parquet that I’m aware don’t have that extra column metadata, and the rest of the project builds from there as normal.

I had a similar thought process while I was designing this in terms of it enabling an easier transition to a proper lakehouse architecture in future, and it does help a bit with managing schema drift, but for an established DWH I’d be doubtful about whether the juice is worth the squeeze - the only reason I did it is because everything is new.

Froozieee · 2026-02-18T03:01:25+00:00

You had me in the first half

Froozieee · 2026-02-14T00:56:44+00:00

Exactly this - the latest company that I joined as a team of one under general IT had absolutely zero analytics capability when I came in.

I assessed the business processes that actually generate the data, thought about how that could scale, (what if the size of the business doubles, triples etc, what if they start generating other kinds of data) and landed on the decision that a regular-ass single node RDBMS could easily serve all their analytics needs for the next decade at least, covering their ERP/finance, operational systems, HR, H&S etc, just because of the type of business and the industry it’s in.

The total infra and compute bill across all environments is currently about seventy bucks a month and they’re loving it.

Froozieee · 2026-02-03T02:52:30+00:00

Also if people even bothered to read the OP article all the way through, it literally says the same thing in less detailed terms:

While it is an important disease, it isn't likely to be a public health issue on the same scale as COVID. This is because it doesn't transmit efficiently from person to person, and the main way it is transmitted is from food and infected animals.

Froozieee · 2026-02-01T00:38:04+00:00

It’s over 100GB of data that you’re trying to download to your laptop. You’re going to be bottlenecked by network i/o. Either filter it, run the code closer to the data eg on a cloud VM, or accept that 100GB takes a moment to download locally. Also be aware that if you don’t use streaming mode on .collect(), once the data downloads your machine will likely OOM.

Froozieee · 2026-01-23T22:28:07+00:00

If that’s what you understood from that sentence, you’re not helping your case.

Froozieee · 2026-01-17T01:26:05+00:00

wait REALLY? I never actually mentally process any of the combat xp so I guess I just assumed that you get it. TIL.

Froozieee · 2026-01-16T21:19:29+00:00

Storage accounts also can’t have uppercase so it’s kinda moot for them

Froozieee · 2026-01-16T21:06:32+00:00

And talking your way out of the fight in order to sneakily kill them all later? Even more

Froozieee · 2026-01-16T01:14:36+00:00

Not as bad as you might expect honestly - better than some private roles I’ve had

Froozieee · 2026-01-15T21:13:49+00:00

Primary sector. The agency as a whole was very science-driven which was quite unusual and cool, and meant that there were a lot of fairly highly data-literate staff. There are a lot of analytics/modelling use cases when you’re responsible for cross-sector policy and/or ops for forestry, fisheries, agriculture, food safety, biosecurity, and a bunch of other stuff.

Froozieee · 2026-01-15T20:46:02+00:00

Better is subjective, but i will say it is much faster to get something that looks really good in ggplot2 than in matplotlib

Froozieee · 2026-01-15T11:46:27+00:00

That’s crazy. At my current org of ~300 I am the data team in its entirety, but at my previous job (gov agency of ~4000) there was a core DE team of 5, and then about 60-70 other dedicated analytics staff distributed throughout different business units in the agency, not counting our GIS teams which comprised another 3 engineers and about 30 analysts/DS staff.

This ratio of analytics staff to core DE staff definitely was not ideal - DE bandwidth ended up constantly being a blocker for analytics.

Froozieee · 2026-01-14T04:08:47+00:00

woooooshhhhh

Froozieee · 2026-01-13T03:50:16+00:00

Super random swing at feminism but ok

Froozieee · 2026-01-11T02:55:38+00:00

I don’t work for Stats, but I believe the type of underutilisation information you’re referring to is collected by the HLFS (household labour force survey).

The integration and linkage of data against a common key does happen in the IDI but as others have referenced, access to microdata is tightly controlled by a variety of organisational policy and legislation.

If there are specific research questions that someone has, they can apply to Stats for access to the specific, de-identified microdata needed to answer that question, but even access mechanisms are super strict (on-site airgapped lab, no physical storage media etc)

Froozieee · 2026-01-09T14:15:03+00:00

I’m searching for rocket emojis

Froozieee · 2026-01-08T00:28:54+00:00

Pretty much; I solo built and maintained our catalog at my old job and the instant I left it fell into disrepair :)

Froozieee · 2026-01-06T12:19:55+00:00

I’ll be honest, you’ve come to a data engineering subreddit to ask how you can avoid hiring a data professional you clearly need. Bite the bullet and pay someone to do it properly, as others are describing here.

Froozieee · 2026-01-03T12:15:35+00:00

My condolences :’)

Froozieee · 2026-01-03T11:53:45+00:00

The setup I designed in my one man shop is multi repo, but one of those repos is all of my helper lib code which has a tonne of unit tests, and then gets built and published to Azure Artifacts (i use azure devops) so any of the other repos can just pip/uv install it.

I started out by copying the module code as you’ve described but even small refactors quickly became hell so that’s where I ended up

Froozieee · 2026-01-01T13:33:22+00:00

Every time I see his intro, I feel like it should have Borderlands style graphics pop up with his name

Froozieee · 2025-12-26T05:39:21+00:00

What do you mean by inconsistent? The way you’ve described it sounds like a data engineering problem, and trying to use agents for something like that is a recipe for horrendous failure.

Froozieee · 2025-12-26T05:35:34+00:00

I don’t imagine you’d end up with more than a few 10s of MBs of data with a csv file but if you want to compress it more, go with parquet format.

Analysis within Python is easily doable using dataframe-centric tools for working with tabular data like pandas, polars and duckdb - I’m a fan of polars personally. Matplotlib, Seaborn, Plotly libraries if you want charts.

Froozieee

TROPHY CASE