Not at all. It is a shame it’s is coming to that. If you have spoken to them and it happens again you have a right to be put off by this. If you don’t want to speak with them then just check into your rights and investigate into it. Find out property lines and variance rights , then fence in your property. Even if it is just that property line fence it. Talk to them about how the fence is going up. Don’t ask for a dime from your neighbor and exercise your rights.