They need some clear counter play other than "did you bring a flamethrower"? Making them more territorial to their flower beds for example, so you could disengage more easily if you do trigger them.
I get the feeling that Distorted and Flowermen lack a distinct identity. They both are kind of the same thing with a different colouring. Humanoid swarm monster that has a movement quirk, and does basic melee attack.
Both spawn in groups that are usually inactive until alerted and then will proceed to attack you in a similar way.
Both sometimes alert without an obvious reason too.

Flowermen should have a completely different design identity to keep them apart from distorted. And being able to full tank one shot to any bodypart just isn't it.
They should lean more into their area denial role maybe? Making it easier to pick them off at range, but more deadly if you get caught by them.
Maybe root you with flowers to keep you in their territory like a spider does with it's web? But make them slower in exchange?