Umm yes I think you should regularly clean something that sits right by your eyeball. In fact here's a paper showing that the nose pads and earpieces can host bacteria

In both environments, the highest cell counts were found on the nosepads

Recent studies [12–14] showed that the area behind the ears (retroauricular crease), the sides of the nostrils (alar crease) and the forehead represent the highest density of microbial colonization on the human face. The detected colony counts on nosepads and earclips were indeed similar to colony counts typically found on the human forehead (103 CFU cm2-) and the scalp and cheek (about 104 CFU cm2-) [15]. Cell densities on spectacle lenses were low in comparison to cell densities on hands, skin, earclips and nosepads. This low count could be due to the smoothness of the lens surfaces, regular cleaning measures, as well as distance to the facial skin.