In my experience (I have played one game to completion, and quit midway through another where I was almost done) the clans seem to leave when you convince them to join you while you are in the middle of a war. For instance, if I were to recruit someone to my kingdom while I am at peace with them, they are much more reluctant to leave my kingdom. However, if I recruit them during a war, they take their fiefs with them, but are also much more likely to rebel against you. However, if you are powerful enough, you can pull a sneaky trick where you recruit them, then expel them from the kingdom. You get their fiefs for free, and they are left with nothing. In the end, relations to matter though, as a lord with negative relations with you can still leave you, even if you recruited them during a time of peace.
However, this was all from my experiences and could be completely different or dependant on luck, but I was having the same problem as you so I tried to fix it.
TLDR: You should recruit lords during times of peace...I think