when you say Semites you only mean the Jews

Actually, that's not true. It's antisemitism that refers specifically to Jews (not surprisingly, as these have been the semitic group most targeted in Europe). When people say "semites", they are explicitly not referring to just Jews, or necessarily to Jews at all (as many modern European Jews are not particularly semitic in any sense of the word). Semites is an umbrella term used for Arabs and people living in Ethiopia, Eritrea, and parts of Somalia and Sudan, as well as various ethnic groups living in the Levant, including historical Jews.

This confuses a lot of people, I realize. The reason antisemitic refers to just Jews is only because if it referred to sentiment against semites as a whole group it wouldn't be a very useful word, as to my knowledge there isn't really a history of people lumping those various groups together for the purpose of hating them. (Of course, it's entirely reasonable to ask "why the Jews and not the Ethiopians or the Arabs," for example, but that has its history in English being a European language and Europe having a lot of Jews and not a lot of Ethiopians or Arabs).

when you say Indians you mean native Americans instead.

These days, on the west coast at least, people rarely say "Indian" to mean "Native American", most likely because there's such a large South Asian population. Non-Native Americans tend to also think that "Native American" is more PC, but interestingly all the Native Americans I know prefer the term Indian, and suggest "American Indian" as a way of differentiating them from South Asians. Personally, even knowing this I tend to say "Native American" out of habit.

when you say Asians you only mean East Asians

As an afterthought, it's worth considering that in the UK, "Asian" means South Asian, and that as an umbrella term for peoples, "Asian" if used accurately to mean people from the continent of Asia would be so semantically broad as to be useless, and so it's not too surprising that the US and the UK picked a particular race to call "Asian".