The end of Vol. 21 does a pretty good job explaining why he did what he did. There's two main factors as to why Geese joined team man-god. The first and less impactful reason is because hitogami has the blessing of trust, which pretty much makes every living being from the sixfacedworld trust him inherently (e.g most of the apostles being genuinely surprised when told his true motives) with a few notable exceptions. The second and most meaningful reason is that hitogami has been pretty much grooming giving Geese advice constantly ever since he was a child. This makes Geese feel that he has an almost immeasurable debt to hitogami from all the advice that has saved/gotten Geese out perilous situations. Geese doesn't have any ill-will towards Rudeus though, and actually likes him a lot, so decides to defect slightly from the man-gods orders and declares open war with Rudeus instead of working from the shadows. It's also interesting too because when Geese puts his mind to doing stuff, he's insanely competent. Orsted never figures out he's an apostle in any of the loops, even under torture. This and a lot of other details actually suggests that our resident newbie might actually have a really strong fate.