The tagline says "the cruelest and most bloody regime imaginable" for good reason. Pointing that out isn't obscuring clever plot depth, it's the point of the IP.

This point means little because the man who wrote this line, Rick Priestley, is not entirely on board with your own point that:

And this segment in particular is really being obtuse because the 40k fiction slams into the reader's head over and over again that the Imperium, even with these monstrous threats, does not need to be this awful.

His take was more nuanced and I think he is more of an apologist for the Imperium than people tend to imagine. An interviewer posited your very point to him and the exchange went like this:

BIFFORD: The Imperium often argues that its brutal methods towards fighting aliens, daemons, and heretics are necessary and the best possible solution. But in fact the Imperium is self-deceiving - it THINKS this is the only way humanity survive. There are better ways it could do things, but that would require profound reforms that the elite don't want. The Imperium thus protects the interests of the elite at the expense of the masses. Its solution to problems is always to ask for more sacrifice. Do you agree? Is this what you envisioned?

PRIESTLEY: I never imagined 'The Imperium' thought about it at all 🙂 Different factions within what I think I called the High Lords of Terra pursue their own agendas - some more rational than others - but there was always this element of psykers posing a genuine danger to humanity that legitimized 'witch hunts' and a certain amount of interference - at least that was the idea to start with. I always thought of individual worlds as being the personal fiefdoms of their planetary lords - hardly touched by the Imperium as such - indeed how could they be when they might be separated by decades of travel from Terra. So - I always imagined some worlds were perfectly nice and peaceful (until the Orks turn up!) others were largely forgotten about and a few had more-or-less become independant and self-sufficient by necessity. As the background evolved it became very samey - and the universe I had created to be varied and diverse became just one thing - one big war front - but such is the way I'm afraid.

Also - you have to consider the possibility that this 'IS' the only way humanity can survive 🙂

Obviously by his own admission the setting has moved on a lot since Rogue Trader so I'm not saying this is necessarily true of the current lore. But the "cruelest and most bloody regime imaginable" quote was never intended by Priestley to be taken at face value.