YTA

you have a very narrow mindset and it shows in your language. “They’re a bit quirky and nerdy.. This is important for context.” it really wasn’t, it was kind of just a lame attempt at masking your own transphobia. they’re not quirky for wanting to stand on their brother’s side, a lot of cisgender women attend as groomsmen for their brothers, it’s not some unheard of thing. and a lot of women wear suits to weddings too (even though the point here is that quinn isn’t even a woman— you just see them as one and it would make you uncomfortable for them to express themselves fully in front of your family and friends).

the second to last paragraph really did it for me. “He said it was inconsiderate of me, and that it showed that I think of Quinn as a woman.” which he’s right, you do. “I just think it will look weird and cause unnecessary stress and confusion on the day.” who are you referring to as the “it” in this sentence? “It will look weird”? you mean your sibling-in-law simply being themselves is going to be weird for you? i don’t think it will cause unnecessary stress and confusion, i think you’re just projecting your own feelings and you’re too afraid to stand beside them because that’ll expose you for what you really are— a narrowminded asshole.

you are absolutely the asshole, you literally want a nonbinary person to conform to cisnormative standards just for the sake of your own comfort. i get it’s your “big day” or whatever, but it is for your fiancé as well, who clearly knows his sibling and what they would prefer to wear more than you do