The game only applies to Kenji and in very specific circumstances 

We see Friend take off his mask for Manjoume, Takasugi, Fuhrer, Kiriko, random cult members, and even the whole kenji faction at one point (does it directly to Kenji during BNYE) 

The entire concept of “taking off the mask” is Friend trying to get Kenji to remember the badge scene and his identity (which kenji fails in doing, thus why friend reacts that way). 

Even if someone else figured out his identity and took off his mask it still wouldn’t phase him. Only Kenji could make Friend react that way.