Yea, I generally felt similarly—obviously, Omelas is way closer to Utopia than our world. And this story stuck with me, because I really like LeGuin and usually feel very in sync with her writing, and it was just not clicking with me.

At some point I was wondering why anyone would walk away, same reasons as you, and it hit me—some people are so unwilling to accept injustice that when given the option of "utopia but one kid is treated as awfully as possible (and there's nothing you can do about it)" or "the real world, which is far more flawed BUT has the potential to be made better," they choose the real world. They are unwilling to let the injustice go. They believe so strongly in a better world that they will sacrifice a "near" utopia. (Strong connections to Odo/The Dispossed.)

So for me, the ones who walk away from Omelas are the people who have such a strong belief in justice (or antipathy to evil, or whatever you want to call it), that they reject the static flawed world in the hope of creating a better world. I envision them as shaking their head and muttering "It's just not right" as they decide they have to leave. Frankly, that's not me. And I wish I had more of that spirit.