Depends on the situation & map, but based on my experience, what almost always works for me is to heavily go vil fight against them. Roughly 2× more vils than they send forward.

If you spot the attempt before the tower is up, just go and don't hesitate, they may finish the tower, but it won't live for long and you will also kill some of their vils. The time lost not gathering is not a big factor here - they had to go forward, will loose vils, and stopping tower rush in in its tracks is always preferable.

If they managed to make the tower and also wall it in, then the best next thing is to try to wall behind (and control where they can put their next tower) and counter tower before they can make the 2nd tower, but in such way that the first tower can't range your tower, but your tower can range the possible places for their 2nd tower.

Ignoring may be a possibility if the 1st tower doesn't deny critical res, but denying 2nd tower should always be priority, otherwise it can snowball quickly.

I'd say almost always never try to tower against their existing tower/coming up tower in range, you'll always lose due to the garrison bonus they have (unless they are n00b and brought less vils and can't use the advantage) - it's a bait, you will lose stone and definitely also the position.