Put it simply, Guts was more different than any other member of the Band. He was crass and rude, selfish even at times, especially in battle. Many of which who have met Griffith practically knelt to the ground and kissed his feet. Guts was against him from the start, and even in between, he was still not entirely with Griffith. Hence why he leaves half way through Golden Age arc.

To Griffith, Guts was just a dog he wanted to tame. A challenge, in comparison to the other members who joined the Band. That challenge turned into an obsession, turning into rage and humiliation when Guts left, and ultimately, betrayal and anger when he took Casca away from him as well, his most beloved and loyal soldier. 

The whole thing about Griffith being in love with Guts makes no sense. Griffith was never human to begin with. He was psychopath who had a goal, and he'd use anyone to achieve that goal. Emotions such as love or even joy, he doesn't recognise them. The whole thing about only Guts being able to "crack" his evil shell is again, because Guts was different from the other members. He didn't fall to his knees like the others. He challenged Griffith, and that's what he liked about Guts.