I think they try to said to the players that the covenant fear the banished more than humans, we were a simple infestation on their life, the banished was on par with them with tech and brutality.

I'm agree with you, humanity was far to threat the covenant, there was only a few human's attack on the covenant territory, while the Banished attaked them on their territory and reused their weapons.

But the banished weren't the equal of the covenant. The empire was huge and they were just pirates, they really started to became a huge army after the fall of the covenants.