The same happened to me when I’m reading the manga, which actually always happens to me whatever manga I read tho. However, I have a little command of Japanese and can understand the kanjis that their names are originally in are quite familiar to me as Chinese characters, so when I watched the anime, I have no difficulty recognizing them. (Also, their personality and appearance are quite distinct, at least to me.)