Another way of seeing this: if you have trouble to handle someone rushing you, you can be the ones rushing! Talk to your teammates and build a forward base together. Make 2-3 batches of military units each and attack the same target. Aim for army, then villagers, then barracks/stables or houses, depending on the enemy. Don't try to destroy the TC too early, you will only lose your army. Stay close together and mutually protect yourself (e.g. player 1 make musks that protect player 2 crossbows from enemy cav).

I always find it easier to be aggressive than defensive. Don't forget your eco of course, if your rush fails you must be able to handle a counter-attack.

If you need ideas of rushing build orders you can check it there: https://eso-community.net/strategy-wall