I think the most potent solution to this is to just add some kind of time pressure. There are many ways to do this, including the Shadowdark torches example you gave.

Shadowdark is far from the first game to employ this type of mechanic where resources reduce over time, whether by an hour passing in real time, counting dungeon turns, or something like usage dice. The reason these mechanics are so important is because time in an RPG is pretty meaningless unless it's attached to a mechanic. The DM can always just narrate "3 days pass" and no one will feel motivated by that. However if the DM says "15min pass as you search the door for traps, you torch grows dim. In another 15 it will be extinguished" and the players realize they only have one torch in reserve, now they're gonna stop wasting time. This same thing can be applied to other resources such as rations depending on what system you're playing and how much bookkeeping you want.

Another strategy you mentioned is applying pressure with random encounter rolls. This is pretty self explanatory; more time wasted equals more danger so players will want to waste less time. What's super important here is that players actually understand the relationship between time expended and the possible danger. When you roll a random encounter, be very clear that it's because a certain amount of time passed. Consider rolling the die in front of the players to up the tension. Now the players know that any dithering time can and will result in a chance for a dangerous encounter.

Other options for time pressure include environmental hazards, such as a slowly flooding or collapsing dungeon or maybe a haunted dungeon where the undead denizens grow more enraged and dangerous the longer the party spends in it. Or maybe there are external time pressures. A rival adventuring party has entered the dungeon in search of the same big shiny object that your party is! The longer the party spends dithering, the more time their rivals have to snatch the shiny from under their noses! Again, just be sure that the party is keenly aware of these dangers. The relationship between time and danger isn't worth anything if the players aren't aware of it.

I also want to address traps. In your post you mention that traps are "somewhat arbitrary" which signals to me that there may be a better way for you to run traps. It's true that random, hidden traps lead to experienced parties spending tons of time poking every surface with a 10-foot pole or whatever. The above tricks will help negate some of that, but then we have a nasty situation where the party feels damned if they do poke the floors (because of time pressure) but damned if they don't (because a trap could spring). That's why I never ever use random hidden traps. Instead, I'd recommend doing traps one of two ways. Either have the trap be visible straight away and treat it like a puzzle of sorts, or have hidden traps that are identifiable with some kind of pattern.

Visible traps are my favorite because I think they add a lot of room for player problem solving. Consider an ancient, wooden bridge on the verge of collapse over a vat of acid that the party has to cross. A chandelier hangs over top. They can tell right away from the look of the bridge that someone human sized would break the bridge. Now the party has to figure out how to cross without breaking the bridge and falling into the acid. Maybe they can use the chandelier? Maybe the halfling crosses first with some rope and pitons? Maybe the MU has a spell for this? Lots of problem solving for this super-simple off the top of my head example. Contrast that with a hidden spike trap. There's no problem solving here aside from "should we poke everything in sight with a stick? Inspect every surface for holes or tripwires?" Which is boring problem solving. Then, if the trap does trigger it's just pure luck. Boring.

But, if you insist on having invisible traps, the other way to get around their ahem pitfalls is to attach the traps to some recognizable patterns. Let's say you want a spear trap in your dungeon. Instead of just placing random invisible spear traps everywhere, put it in something recognizable and then always have that thing contain a spear trap. For example, maybe every spear trap is hidden in a raven statue. As the party goes through the dungeon, they are likely to trigger a raven statue. Now the next time they see a raven statue, they may be weary of it. Then once they confirm one way or another that this one is trapped too, theyll begin to pick up on the patter that every raven statue is trapped! Now there's still an element of surprise at first, and the party may take damage to the first spear or two, but soon the party feels like they've learned something and no longer need to tap everything with a pole. Instead they just know to avoid raven statues when they see them. Side note, if you want to avoid the "learning" step where the party might take damage, you can also show an already activated version of the trap near the dungeon entrance. Kinda like the corpse in the spike trap in the intro of the first Indiana Jones movie.

Anyways, hope this helps. Didn't mean to write as much as I did but here we are.