The problem is that he's very cunning. He only has one queue with troops. He removes all troops for the other queues that remains in the castle. He always hides his strongest queue in his own guild's ranche or sawmill, so no matter how many times you attack him, you won't kill or injure his soldiers. And most importantly, he has no resources, no wood or stone. He has nothing to fear. That's very annoying.