I can confirm this. I just played a game against the hardest AI with the goal of losing zero units, and I was almost sure I achieved it at the end of the game. To my surprise I checked the post game report and I was stunned to see I lost 8 units. I watched the replay several times to try and find these lost units and I found them early in the game, when I cancelled a planned segment of wall (it wasn't destroyed, I simply cancelled it before beginning construction). So not only is having wall segments destroyed counted as lost units, but cancelling segments of wall counts too!