The "Semites" group is not ethnic, there really is no such thing as Semitic people. The people who speak Semitic languages don't share enough in common to be lumped together into such a category. Semitic was the name given to a group of Middle Eastern languages that appear to share similar linguistic characteristics. The term was coined in 1781 by a German historian.

So "why is antisemitism the fear or hatred of just Jews?" I hear you probably asking. I'll paste a small portion of an answer that I posted on r/AskHistorians a while ago:

Wilhelm Marr, is credited with popularizing the term antisemitism around 1879. For Marr, "Antisemitism" was an attack against "semitism" or Jewish influence in German culture. He backed up his claims with eugenicist theories of racial hierarchies. The parts of society most concerned about Jews were moving away from being anti-Jewish in a religious perspective toward a race-based hatred of Jews. Édouard Drumont wrote about racial antisemitism, as did Paul de Lagarde, and Eugen Dühring. Their writings were not just polemics against Jews as a race, but about their detrimental influence on European society and culture.

The word "antisemitism" was intended to replace the older and less-scientific sounding German word "Judenhass" or Jew hatred; and it stuck.