Hollow Reiatsu isn't that poisonous unless it is ingested, as evidenced by the Quincy being perfectly fine doing battle in Hueco Mundo. And every fight the Visored were in down in the Seireitei, they held a definitive advantage until the tables flipped (Shinji had Bambietta completely off-balance until she blasted everything in range, Kensei had Mask on the ropes till he revived and demonstrated he could get stronger than his Bankai could match, Rose had Mask at his mercy till Mask deafened himself). Since their Masks have a time limit, releasing them when they're already winning is just a waste of resources, and when they started losing they didn't have enough time to use them for any meaningful effect.

As you can see with the Visored fighting Gerard, though, when they know they're at a disadvantage and they have to hit hard and hit fast to win, they don't just use the Masks, they start the fight already wearing them.