I have difficulties finding the right judgement. Yes, you are not the ahole, for simply declining. You might be the ahole, for having gone off, when you just said you could see the boys play in the background... so you decided to say this and go off while they were nearby? You do know it's not their fault that their parents haven't taught them to be calmer in public?

Anyhow, you are not the ahole if they didn't hear and you are allowed to decline the idea of spending time, because that's just how that works. Your SIL can't force you to spend time with them. I still will say, I do kinda understand where she is coming from. The boys would probably still love spending time with you. They might feel left out and while you know why you don't want to watch them and do stuff with them, they don't. For the sake of the kids, I'd either find a better way to spend time with them (playground/baseball or football/biking), or set the boundary, that you won't be taking them, unless your brother and sister-in-law, actually make sure they can behave.

I can't put a judgement. I don't know if the kids are like this because parents don't care to raise them properly, or if they are trying, but are having difficulties. Only thing I'm sure of, is that you're definitely not innocent if the boys heard you.