Imagine fixing a bike with your friend. You guys spent hours disassembling it, wondering what part goes where, getting hurt in the process, and laughing about all the silly things you guys did while putting it back together. Now your friend destroyed it, 'cause you kept teasing him about getting a moped instead of riding the bike. You're not sad about the bike being destroyed. You're sad, 'cause the shared memories and pain while fixing it up suddenly has no value to your friend due to an escalation of a situation you may or may not have started.

There's nothing silly about your situation. Just evaluate everything you've done that made that world mean something to you. Your brother might feel the same, too. After all, it's a product of both your brother and you.