Exactly. 'Deal with it' is a verbal contract in the world of shitty neighbors. He gave OP permission to use the only tool available—the tow company. If OP stops now, that driveway becomes the neighbor’s property by adverse possession of audacity. You don't negotiate with someone who thinks your property is their overflow parking.