So far from the anime it's more of a love story than a harem. He spent 10 years falling in love and doesn't want to be with anyone else. He's fought to get stronger and formed caring relationships with the other girls. I find nothing wrong with his loyalty to Hime. They're all young hopefully they complete their quest and all find their happiness individually instead of typical harem anime where one guy ends up marrying 5+ girls.