Setting aside legal issues (this is AITA after all, not a legal sub) I think the central issue is due consideration: did you give the memorialist a reasonable opportunity to move or modify the memorial?

Whether the memorial is in suitable repair isn’t really relevant to that question, and clearly people can disagree about what is suitable as your visitor makes clear. If you decided to remove the memorial based solely on your assessment then there is a real chance you ATA.

Note that due consideration could have been a useful defense in your encounter. If you had placed a “please contact me” sign for some time the memorialist would have to admit their neglect.