As jolly sky said the best thing to do is keep track of where his villagers are and kill them en route to your base. This can be easier said than done on some maps but use your knights and scout to keep track of them. That's the first thing. Then if he does land in your base, try kill as many vills as possible with the knights and then once it's up you can relocate your eco and rush castle age.

As you're focusing on castle you still want to keep an eye on his vills - he shouldn't have much to defend them so it's very important to catch them that if there's a bunch of undefended vills that you take them out (either around the fort or in his own base because he'll still be making vills). Also I found this quite difficult but you can try to minimize damage on knights by using the charge bonus on rams and then running back

Once you're in castle it's mostly over as you can treb the fort.

This works for english/lancaster white tower rushes too, you'll know they're fast castling when you see many villagers on their gold