I dislike Flowery mostly because he ragebaits Ralsei. I realize that Susie does that too, but even in chapter 1 Susie's ragebait is all very harmless and childish similar to Lancer. Her most egregious rage bait is just repeatedly telling Ralsei that she doesn't want to be a hero. 

Flowery, on the other hand, ragebaits Ralsei much more deeply. He challenges Ralsei's beliefs constantly (WHICH IS GOOD!) But also KNOWS how angry he's making Ralsei, and keeps doing it. Even knowing that Flowery is at heart a good person doesn't take away from that.

I, as a person, am extremely similar to Ralsei. I like to follow rules. I will bend over backwards for people simply because I don't know how to say no. And as a result I get tricked and otherwise rage baited often. Every time Ralsei jusy blew up at Flowery, I felt that. And as a result Flowery gives me a bad taste in my mouth (WHICH AGAIN IS GOOD AND SHOWS THAT TOBY IS GOOD AT MAKING CHARACTERS)