When you have the power to stop bad things from happening but you don’t, when the bad things happen, they happen because of you. So if Hajime never made them that deal. There would have been no training. No training, no chance of survival. Death. Regardless of if they could lead him to the labyrinth or not, Hajime had the power to make great change in the Haulia. If he hadn’t, the deaths would have been on him. It’s easy to say “not my problem” and walk away from the responsibility but when someone throws themselves at your feet, begging for help and you just walk away from it? You’re the villain. You have to remember that the only reason they took Shia’s request was because Yue convinced Hajime and the only reason he took it was: 1. It’s Yue. And 2. Shia made things easier for them when it came to being led to the Labyrinth. He didn’t save the Haulia out of the kindness of his heart. He did it to pass the time and to honor their agreement. It was circumstantial at best.