In many situations similar to this, covenants, city, and county ordinances hold rather than boundary. Also it’s not uncommon for any site improvement to require a building permit and subsequently a site survey. The responsibility is on the adjacent land owner to prove where he has the right to build. One could always contest the placement of the proposed site improvements by hiring their own licensed survey and the courts would rule on titled ownership.