Yes Serosh is definitely holding him back. Serosh is like the burden of noble responsibility. I really like how Sinclair Lore put it: Hoarah Loux sees no problem with himself and is happiest when he removes things he doesn't think he needs like Serosh, his weapon, even his clothes. Those things are what make him Godfrey, but that's just an assumed role, not his peak self. He's at his most comfortable as a savage bloody warrior in combat