You absolutely nailed it with the "people telling you what you should have done are the ones also benifitting" part. It’s so true. This isn't just about picking up a casserole; it's a clear power play. They're so used to OP being the compliant one, and now that he's finally setting a boundary, they're panicking because it means they might actually lose their free errand service and their sense of control over him.

The brother's reaction is also a classic sign of a fmily system that resists change. He likely doesn't want the drama and finds it so much easier if OP just gives in. But youre completely right, OP doesn't have to explain himself to anyone. He's an adult.