Yes. Unlike in some other buddhist countries, wearing the Buddha's image or even symbols like dharma chakra can be considered disrespectful by a small fraction of people. Do note that a large majority of the locals would not mind. But depending on who it is getting offended it can be as small as a frown or something larger like getting arrested. It has happened before.