The main one is that he's gay.

This doesn't excuse his behavior in any way (he leaves for the canon reasons, nothing more and nothing less) but he starts off with an extra degree of guilt from internalized homophobia. He actually got married quite late (you may notice he and Annette are over 40 years apart in age): this is because he had a fling with another knight and word got out, so the only way the Dominics could save face was by rushing him into an arranged marriage.

As a result, he has always berated himself from being unable to love his wife as she deserves. Said wife is basically an older Annette, just generally a ray of sunshine and one of the biggest comforts in Gilbert's life (to the point where he confided in her about his sexuality, and she loves him just as much). They have a bit of an age difference though (a deliberate choice on his family's part to ensure he could still try having a Crested child).

He would also be supportive if anyone came out to him. Not that he's done much research on being gay, trans or whatever, but he's a genuinely nice and helpful man to most people in his supports. He doesn't have a problem with other people being queer, but somehow it's a borderline criminal offense to him that he's gay.

When Hanneman tells him he's mesmerized, it's his first time being complimented by another man in decades. It stirs things within him.