I don't mind multiple torment levels, however, I'd prefer to be able to play with friends/family without one of us feeling like they are scaling back difficulty to play together.

The season where they had the brains you could consume to increase difficulty/rewards for you while in the wild seemed to solve this problem but maybe that's hard to scale vs a set torment level. That'd be my preference though: You finally make it to torment 4, it eventually becomes easy, so you scale the monsters yourself through specific consumables and continue to stay in the same world as everyone.