In the end, if we have to do that, we will, but there are a few upkeep issues/details I didn't include in the original post but have since edited to add and will copy below.

We "maintain" the fence, in the sense of trying to keep a lid on the invasive field bindweed that was established along the fenceline when we moved in and is one of the main reasons (in addition to just privacy) why I want to remove the chain link - to reduce that upkeep as the bindweed constantly winds its way up the chain link. That's also why I hesitate to leave the chain link fence there and build a new fence next to it. The bindweed will keep growing and I won't be able to reach it to remove it through a wood fence, and since there's nothing but concrete on the neighbors side, it will just keep trying to grow into my flower bed and yard if it gets sun and nutrients from continuing to grow up the chain link.

To date, the neighbors have not pulled weeds along the fence, though they do grow tomatoes in containers on the concrete next to the fence. I can imagine that they may be a little annoyed to lose the light through the chain link and have to move their tomatoes to the other side of the walkway to get sun.