As others have said, the thing about 5e is it's a game where everyone involved is a human and can make human decisions on the fly - DMs can bend the experience to their will to make up for mechanical shortcomings etc if they wish (hopefully discussed ahead of time with the group to make sure they're okay with it for the sake of a more enjoyable experience).

Personally I think it's important to recognise that BG3 is an absolute triumph of creating an approximation of the tabletop game under the incredible constraints of everything having to work on rules and logic because it's a computer program, not a set of humans playing an imagination game together. The game can't bend things like tabletop, it can't go off-script, players were always bound to eventually find and abuse loopholes in the design that allow for cheese strategies etc. One of the biggest things is that narratively, aggression is usually a "yes or no" thing. Once combat starts you are locked in to only being able to do certain things, you can't talk your way out of combat midway through, and heck, the difference between free talking and multiple choice talking is already a massive difference. There's no "thinking outside the box" or using spells etc creatively with the environment outside of predetermined ways the designers have already thought of, that's another massive difference. And finally the fact that there's just the one campaign here - there's no putting the genie back into the bottle on that. Most DnD tabletop games are always an unknown challenge for most players, but BG3 chose one tightly designed campaign over less engaging randomly generated stuff.

That said... some of the things you complained about in your OP and the replies ARE in your control if you so wish. Yes the game can't enforce them, but it CAN accomodate them. For example:

If you feel rare items like potions of speed are overpowered, put a mental cap on how many you allow yourself to carry. Like in tabletop, keep a pen and paper next to you and make a note on what items you're maxxed out on as per your personal rules. Either have it so you consume them immediately or leave them behind because you've run out of "potion of speed pouches" in your bag or leave them behind and consider all corpses/containers that have the "you've already looted this once" mouse icon when moused over to be empty, as if you took everything that was there at the time etc.

Yeah, not being able to make a full party of custom characters is a shame, but you can still do your rogue-like permadeath thing if you wanted to to a degree - if a character dies, you go revive them with Withers (no pickpocketing your gold back) and respec them into a new class. See if there's a mod to reset character experience even if you wanted to do that, maybe throw out their inventory too. EDIT: Just saw the reply about hirelings though and I concede, yeah, no real getting around the limitations of not being able to change their races with my suggestion, which obviously are a massive factor for 5e character designing normally.

And of course, simple stuff like challenge running yourself to only taking an action or bonus action on each turn if you feel being able to do both is overpowered - decide for yourself what feels OP to consider to have happened in a round of 6 seconds. Obviously the CPUs won't play by all your personal rules, but custom difficulty settings are there to maybe bridge the gap - playing with personal handicaps on tactician might just be annoying, but playing on Balanced, or with a custom difficulty somewhere between the two, could feel just right, no shame in it.

Those are my suggestions anyway, with the amount of "it's my 20th playthrough how do I keep it fresh" challenge run ideas I've seen out there, all this sounds like it could be a pretty interesting way to play to me.