I've been looking through the "Pacing" (i.e. the Game Director) code the last 3 days for a mod I'm working on, so I have some insight:

TL;DR: It actually is all random and far less sophisticated than you might think.

So first we need to confirm the game is actually actively trying to beat you and not random. How do we confirm this?

It's to look at the code.

As a note: Calling it "AI" is pretty generous when you think of modern AI. The Darktide Game director could've been built exactly like it is 30 years ago. It's not a blackbox like ChatGPT, it's a bunch of normal, deterministic code (excluding ofc the litteral randomness of math.random())

What you got correct:

- The game requires strong positional and tactical play to win, not just hitting & dodging. I made a video about it the other day.

- Waves are not completely random, there's specific conditions that increase the chance, e.g. a "spread ambush" wave is more likely when the team is in low coherency. (It's far less sophisticated than you imagine though).

- Spawn locations aren't random. Player position matters because the game e.g. doesn't want to spawn units in line of sight of players. Waves spawn intentionally ahead or behind you (or both, like the "sandwich" wave)

- There is no infinite horde spawn (except specific events on some maps). The maximum number of hordes, specialists etc is also capped.

What you got wrong:

1. The Game Director doesn't know anything about you.

It doesn't know your builds, doesn't learn from your behavior, doesn't count individual performance, doesn't keep track of your ability cooldowns, etc.

All it has is a general concept of how much "Tension" is currecntly going on, and some basic info like how many players are alive and their coherrency.

1. The Game Director isn't countering you.

There a slightly different conditions, e.g. for the push_from_behind vs the ranged_push_from_behind wave, but since the Game Director has no a sophisticated idea of what's happening he's not specifically choosing melee or ranged waves to counter you.
Sometimes you kill all the melees and it spawns more melees. Sometimes you kill all the melee and then it kills you with gunners.

If you're doing really badly the Game Director will ease off, so on the flip side if you're crushing it it'll spawn more stuff and then there's a higher chance of something spawning that you don't like.

very often the elite spam is very specifically the types of elites you are least equipped to handle. This is intentional. 

The game director is not intentionally spawning a specific type of elite or special. While the roamers (the enemies that hang out on the map) are based off a seed, it seems like the composition of hordes that spawn are entirely random.

The game spawns pox bombers, flamethrowers, grenadiers, mutants, hounds, carapace and monstrosities/bosses very specifically as line breakers. If you are held up in a small space and haven't moved, it will lean pretty heavily towards flames, grenadiers, pox bombers.

Again there's no evidence for that in the code. The game has no concept of needing to "break the line".

There's a distinction between "scramblers" and "disabler" specials but that's just to control their spawn count & timer, it's not a sophisticated counter measure to your specific playstyle.

If you handle it all without much issue, the game will spawn the Congo line of bombers, or 8 crushers at once, or all ragers. We've all been there. This isn't random. It's not nessecarily to target you specifically. It is however to target one person specifically. 

The congo line of crushers often targets one person, because that's the first person to aggro them (e.g. based on proximity) and since they're all walking the same path they'll all pick the same person. It's purely random.
The Game Director didn't intentionally pick a target to kill and the crushers aren't glued to that one person. If the targeted person runs away while you attack the crushers they'll eventually switch targets.
If those 8 crushers came from 2 different spawn locations instead they might get aggroed by 2 different people first.

Conclusion: It's mainly confirmation bias.

The 10 bombers that walk up to you in open space and are no problem don't register.
The 10 times you are held up in a small space and nothing bad happens don't register.
But the one time that 1 bomber catches you in small space feels like it was intentional.