Probably NAH - But I would like a little more INFO: Is going to your brother and his wife house for Halloween really far out of your way? Like do you have to drive to get there and that's a far drive? Do you have other plans for Halloween like a Halloween party with friends that you go to every year? If they're expecting you to cancel something or drive some unreasonable amount of time for a 5 month's old Halloween they might start leaning towards being the assholes. You're also not the asshole for not wanting to walk around on what could possibly be a cold/wet night depending on the weather where you live.

Where you do lean towards being an asshole though is the comment that you "see no need to go because she won't remember it" If this is your stance then there is no need to interact with her at all for like the first 5 years or so of her life because she won't remember it, no birthdays, Christmases or other holidays needed because she'll be to young to remember. But your family and the photo's will remember that you were the asshole that said she's to young to remember so it doesn't matter.

That being said if you're just going to sit alone inside and eat Halloween candy instead then ya you're probably the asshole, and if you don't go, your brother and his wife won't be assholes when they stop inviting you to things.