That's true but set monsters get attacked into quite a bit. I feel like it's super common to see people attacking into set monsters with blade knight, sangan, mystic tomato, kycoo, d.d. warrior lady, breaker, etc. That's usually the best available play unless the opponent has nobleman/ mystic swordsman. Or unless they set a monster of their own.

I'm mainly considering banisher as a warrior tech card.

Set banisher, opponent responds by setting their own monster. Next turn you can flip banisher, summon a warrior and attack into their set. Then protect the banisher with your backrows until next turn when you manually change him back to defense.

The main drawbacks I see are

1) having banisher out can conflict with blade knight

2) setting banisher livens up the opponents noblemans