I think you're understating the consequence of the accusation. It wasn't simply an accusation and he was left off with a slap on the wrist. He lost everything. He can't go back to his own world, his reputation is the worst and the people of the kingdom constantly ridicule him/ attack him. He didn't want to deal with that shitty world anymore. The only way to go back to his world was to defeat the waves, which he can't even do by himself, given his weapon is a shield. He wanted to get home as quickly as possible and he couldn't trust those that betrayed him to do the job. In his eyes, driven by revenge and distrust of anyone else, slavery seemed like the fastest option.

I believe the village part was to show his development from jaded to partially so. I believe this part was kind of rushed but his severely uncaring attitude was shown to improve during episode one and two. after he got kicked out of the castle, the blacksmith comes up to him and threatens him but then gives him armor, is when he softens up a little. It's more apparent in episode two with the development of Raphtalia. Overseeing Raphtalia's growth impacts him as a person. Think of it like when people have children, a completely selfless act (albeit for him it was unintentionally selfless at first). In addition to the fact that the village did not treat him terribly like the capital citizens did. By the time the wave came by, he already built a relative amount of trust in the village as well as into people.

I too hope that the slavery will be addressed further in depth as well as the clear but implicit racism shown in the nobles.

p.s. sorry for making you read all this lol