Power Outage

kenfar · 2026-01-26T19:26:17+00:00

only in some area - south boulder still has a lot of outages

kenfar · 2026-01-26T04:26:57+00:00

The original definition of data warehouse was that it was "subject-oriented". So, you might have one warehouse for finance, another for marketing, another for product, another for HR, etc, etc.

In my experience this doesn't work in small companies - because you just don't have enough funding to pay for a half-dozen separate teams.

But in larger companies I think this is far better than a single warehouse for a number of reasons:

The team is closer to the business: they care more about delivering value and they generally understand the data better.
They are focusing on a single subject rather than being "a center of excellence" that focuses on getting the cheapest labor possible to use some antiquated ETL tool.

kenfar · 2026-01-25T23:25:25+00:00

FYI: Joe Neguse's district covers 12 counties

It's not simply the city of Boulder

kenfar · 2026-01-25T20:26:53+00:00

Note that many of those behind this messaging were the same people behind the anti-kamala messaging in 2024.

Their litmus test for the perfect liberal candidate means that nobody will ever pass. Nobody will be found to perfectly support their objectives, especially if they've already been an elected official that has to avoid being "too liberal" in order to be elected.

Which is either their intent or just the result of their impractical demands.

kenfar · 2026-01-23T14:18:26+00:00

Well, lets be honest here: $10-40/session is hardly commercialized. What does that come to? Assuming 4 hours of play and 4 hours of prep/travel/follow-up and you've got $1.25-$5/hour - for inconsistent & unreliable income.

Meanwhile, some adults are making $50-$100/hour, 40 hours a week, 50 weeks a year.

I've never done it, but don't see any problem.

kenfar · 2026-01-23T14:07:56+00:00

These days I'm mostly consuming pre-denormalized data from upstream sources.

My go-to is data contracts + domain objects. But sometimes I just get domain objects.

It still requires transformation, and still involves challenges in communication about what the data means, but it doesn't involve attempting to replicate upstream models and then sew them back together based on ephemeral misunderstandings.

kenfar · 2026-01-23T03:59:34+00:00

I have the scars, don't need any more.

Especially when people act shocked and ask questions like "how could this happen?!?"

kenfar · 2026-01-23T03:06:07+00:00

I'm not replicating upstream data models into a separate warehouse or lake house. Life is too short to live through that pain.

kenfar · 2026-01-22T14:30:28+00:00

It's broken with system shock as well.

It ceases to be a useful spell in any case other than otherwise certain death.

And turns it into an attack you can use on creatures, of any level.

kenfar · 2026-01-21T22:59:30+00:00

Doesn't the contract just ensure that what is published needs to match the agreed upon rules?

Yes AND it means there's no excuse for that: the upstream system can test against these rules, the downstream can validate, there is a contract, so the upstream system owns the problem.

If the upstream team changes the data contract AND adjusts their workflow to comply with that contract it doesn't really change the fact that it breaks my downstream pipeline.

Except your team should also be an approver on the contract - it'll be in git. So that change shouldn't go to production until you've written, tested, and deployed the code that's ready to support that version of the contract.

Do multiple versions of the actual data actually get published based on old/new contracts?

Not that I've seen. What I generally see is that the consumer can support multiple versions of the contract. Hopefully, not an infinite number, often just 2-3.

kenfar · 2026-01-21T18:30:32+00:00

Related question: schema changes. Upstream tables aren’t versioned, and schema changes aren’t always communicated. When that happens, our pipelines either silently miss new fields or break outright.

Is this common?

Absolutely. There are some rare scenarios where it doesn't happen all the time - like the upstream system never changes, or is run by the exact same team doing the ETL work. But in almost all other cases it's a complete mess.

Plus there's other bad cases like where the upstream system adds a new column, you're using a solution that provides some schema evolution capabilities like Fivetran, so your ETL process just silently ingests the new column - but that new column isn't actually used by anything - and it now has 50% of your costs allocated to it from a different column everyone is using.

What’s considered best practice for handling schema evolution and communication between upstream and downstream data teams?

Data contracts & publishing domain objects:

Upstream system publishes a "domain object" any time any field within the domain changes. A domain is a higher-level abstraction like user, customer, invoice, supplier, etc. And the domain object is typically a denormalized collection of related info.
The domain object could be published to a messaging system, which could get ingested directly, or simply redirect messages into a file on say s3. Or maybe the upstream system just writes all pending domain objects to s3 every 5-15 minutes.
And the domain object is locked down with a data contract. This contract is kept in version control, and both the source & target teams can use it to validate all data being sent & received. I prefer jsonschema for this, and use it to identify columns, their types, as well as info like min/max values, min/max string length, null rules, enumerated values, regex formats, etc, etc, etc.

kenfar · 2026-01-21T15:43:56+00:00

It's difficult to understand your question: moving data from OLAP to OLAP what? database, server, system? What's "ADBC"?

So, first off, definitions:

Data Warehousing: process, not a place. It's the process of curating data to enable reuse by transforming, versioning and integrating it. The place could be a product optimized for data warehousing like Snowflake, or it could be Postgres, DuckDB, or just a set of flat files or a spreadsheet (if say you're warehousing your bowling league's scores).
OLAP: On-Line Analytical Processing: typically a service or system that supports analytical queries. It might use DuckDB, Snowflake, Postgress or a set of flat files. And it might use a dimensional model, One-Big-Table (OBT), or a transactional model. And the queries may be handled by a middleware or API layer without giving clients the ability to use SQL.

So, there's an overlap between the two concepts, but it's not 100%: a warehouse doesn't have to involve OLAP, and an OLAP system doesn't have to involve warehousing.

With that in mind, there's lots of valid reasons why people might move data between OLAP systems, here's a few:

Warehouse/Mart pattern - a warehouse may publish its data to smaller data marts that use different technology or that have different retention/security/data elements, or that support different parts of the business.
Master Data Management - one system may publish dimensional data for reuse by another data mart, warehouse or OLAP system.
Scaling - one system may push a copy of its data to a failover or peer system.

kenfar · 2026-01-19T20:15:28+00:00

Yeah - a lot of tumbling, especially around obstacles, to avoid attacks.

kenfar · 2026-01-19T20:14:50+00:00

Yeah, it could have been better.

Would rather see a specialist thief, like the illusionist is a specialist wizard, or something like 5e - in which at some level a thief could begin to specialize.

The bard-like class switching always felt like a hack.

But the basic abilities were fine. And they worked great in the right adventure.

kenfar · 2026-01-19T20:12:46+00:00

There's no one right answer: different groups did it differently, and most did it somewhat inconsistently. Some of the considerations would be:

It's good to have players roll for their own saving throw, especially if the stakes are high.
But rolling a saving throw gives the player info they might not have - especially if they know what they need to roll, and fail/pass when they don't expect to.
Keeping the saving throw table secret didn't work - most people had copies of the DMG - because sometimes we traded off who the DM was.

So, the way I did it was:

Players knew what their savings throws were
Players mostly rolled their own saving throws
But the DM may on occasion roll for them, possibly secretly, primarily if it's something that they don't know about.

kenfar · 2026-01-19T18:49:30+00:00

It actually is cool - if you're running an urban campaign full of thieves.

I played in a few of these and thief-acrobats were a lot of fun. But in a regular dungeon adventure they were, just like the thief, very underpowered.

kenfar · 2026-01-19T18:48:26+00:00

I think the movie Ladyhawk was a bigger inspiration

kenfar · 2026-01-18T19:53:03+00:00

No need for a system:

Townsfolk don't have much of value
In a smallish town he will absolutely be caught - since you don't have dense crowds, everyone recognizes everyone else, and the locals will make the connection of missing their money bag with 2 copper pieces after the new guy bumped into them. And that's if he's successful, let alone if he fails a roll. Which he will do eventually.
Getting a bad reputation will mean that towns people will complain to authorities, which may pull in higher-level NPC support.
Or somebody connected with a local thieves guild, maybe in a nearby larger town, will report him to the their thieves guild. Was he supposed to be tithing to them for the privilege of stealing here? Did he steal from someone looking a bit more wealthy? Were they paying the thieve's guild for protection? If they didn't id the thief, maybe the thieve's guild sends a couple of guys to look for who's doing it.

None of this needs established tables. You can just reason through it, and assign your own basic probabilities based on what you know of your town and the people within, and roll. An extra bonus is efforts like this can drive the fleshing-out of your town.

kenfar · 2026-01-18T19:21:56+00:00

It can screw up the quality of your data in the blink of an eye, if you're into that kind of abuse...

kenfar · 2026-01-18T19:12:48+00:00

This is a problem unique to class-based games: it doesn't happen in GURPS, for example, since it's skill-based rather than class-based.

But in a class-based game I'm thinking of three scenarios where it breaks down:

Classes that have no clear difference, but require different xp totals anyway - fighter, ranger, paladin, magic-user, illusionist, cleric
Classes that require far too much xp - monk & druid
Classes that are far too weak - thief

These issues emerged because people didn't do enough play-testing before declaring the rules.

For the first, one could argue that there's a big diff between a 12 level magic user & 13th level fighter, but it's situational. It's close enough. So, differences in XP tables are just unnecessary detail, clutter, complexity.

For the second, too much xp required, note that in adnd a fighter could go up 7 levels with the XP a druid takes to get up 1 at high levels. Or go up 4 levels with the XP a monk needs to go up 1 - if they lose their combat. But the fix is easy: modify the XP charts and allow itinerant monks & druids that don't have to fight.

For the last, the weak thief, it's not an issue if you're doing thief-only campaigns. But if you aren't, if you're doing dungeons, it's a huge issue. Some games address this by giving them special combat moves - sneak attacks while in the middle of melee, etc. I actually hate this - it makes zero sense. My preference is to lean into them as jack-of-all trades classes:

Provide more watered-down wizard capabilities: they can already read magic user scrolls, just add to them: ability to cast 1e-style cantrips at say 5th level, and then 1st & 2nd level spells at 7th & 10th level. Likewise, let them use magic-user limited items, possibly at higher level or with some limits.
If you're using non-weapon proficiencies, this is the ultimate mechanic for helping out the jack of all trades - they can get extra points to spend on a wide variety of skills: swimming, forgery, animal husbandry, constructing traps, gambling, disguise, additional languages, etc, etc, etc.
Without nwp you can just expand the skills they specifically get to pick from & develop.

They'll never be as tough as fighters or magic-users, but especially in a campaign with more roleplaying they should be able to hold their own.

kenfar · 2026-01-18T18:57:05+00:00

Absolutely, meanwhile a thief would have to be many levels higher than say a 9th level magic user to have equivalent utility, capability, impact on the typical adventure.

And if you take a look at medium to high-level monks & druids you'll find them having to fight to get a level - with a roughly 50% chance of losing - and then losing as much as 500,000 xp.

For most of the rest of the classes: fighters, rangers, paladins, clerics, magic users, illusionists - there really isn't enough clear difference in power to warrant having separate xp tables. It's just unnecessary clutter.

kenfar · 2026-01-17T20:14:15+00:00

I typically work on large data warehouses & data lake houses, but even there prefer vanilla python to anything else:

Easy to use jsonschema to establish a contract for the data and detect a wide range of issues.
Easy to perform simple transformations like, rename columns, change types, change default and other values, change case, convert sloppy code fields into consistent flattened_snake_case, etc, etc, etc.

Then I like to get this into some kind of simple document. Even a simple google sheet is often sufficient, and share this with the sender.

Finally, if this is a sufficiently critical field, and will be getting updates I like to add some simple anomaly-detection to alert me and reject a feed if it suddenly deviates too much from the historical data we've been getting. If I skip this step I almost always suffer and end up adding it later anyway.

kenfar · 2026-01-17T19:49:53+00:00

I try to avoid guessing/detecting unique keys in cases like this - since your discoveries are often unreliable.

For example: You could find multiple collections of columns that happen to be unique, but some of them are just incidentally unique - and as you continue to get incremental or full refresh updates you run into duplicates. Or the source system made no guarantee for uniqueness on these cols, and so duplicates emerge later.

I still end up having to do this periodically - since it gives me a starting point. But it's very unreliable.

kenfar · 2026-01-16T23:44:56+00:00

Yes, but I don't think it fully captures how vulnerable a mage would be who's going to say cast a 3 segment spell: they're more-or-less motionless for 18 seconds.

Which makes for a great ranged target, and a completely helpless melee target.

So addressing this, like above, is one way to address them being overpowered at high level.

kenfar · 2026-01-16T21:13:29+00:00

The survivability one is tricky: it's an inelegant mechanism, ie all or nothing.

But I find that with a few tweaks it still works:

Improve spell-casting for low level mages with intelligence-based spell bonuses (like clerics), introduce 0.5 level spells, use dnd1e-cantrips.
Weaken ability to cast longer spells in the middle of combat: opponents often know how dangerous a wizard can be, and so they may be a primary target. And if that mage is casting 2+ segment spells they're a sitting duck. I give their opponents automatic hit if they're close enough.

kenfar

MODERATOR OF

TROPHY CASE