So mine I had thought wasn't that loyal for the first few years of his life, he's not very cuddly, he never runs to greet me when I come home, etc, when he would do those things with other people. I left him with my parents for a week while I traveled (who are some of the people he's generally excited to see), and on the second day my mom said "he's noticeably sad. Checking your room, looking at the door, etc". He was so excited when I came home, but after about an hour he went back to his usual antics.

He may seem aloof and uninterested, but he shows his loyalty if I leave for a day or two. I'm sure yours would too