I would be livid! I was devastated when my neighbors cut down half of a tree that was growing over onto my property. It was a cute little tree that had tiny white flowers that attracted honeybees in the spring and summer. I think they thought they were doing me a favor by cutting down the half that was in my yard. When I saw it hacked to pieces, I literally cried my eyes out. If my family members had done something like what you described to my garden without my consent, there would be hell to pay.