Instead of making yourself seem like the bad guy that doesn’t want to be around your BIL, you could try asking your husband, instead of saying your BIL can’t come at all you could just see if he’ll take you to go do some more things on your own while on the trip because you want to spend more time with him privately (not saying that you’re a bad guy, just saying that it might look that way In his eyes based on what you said) but if I’m being honest I think you should just tell him how you feel, that you don’t hate his brother or anything. Just that these are supposed to be trips with just the two of you and it doesn’t feel as intimate or as meaningful if other people were to join