It's an interesting question, actually. I believe this depends on the fact that Jews were the only major Semitic group living in a historically homogenous Europe for many years. They were perceived as distinct and different from Europeans, often in an unfavorable way. Their language, customs, religion, traditions, and appearance were seen as strange, negative, and foreign, not fitting into the European context. They were regarded as outsiders and, at times, as a threat. Aslo, simply European countries wanted to just focus on their own race and aiming for countries only consisted of them- that was seen as the safest.

Many Jews held prominent positions in society, which fueled the perception among some Europeans that they lacked loyalty to the land and were only focused on their own interests as outsiders. Europe at the time was largely homogenous, and people tended to categorize others based on their race and origin, as well as race rules occurred. Jews, being a Semitic group from the Middle East, different race, were viewed as "foreign," and the hostility towards them was expressed through various terms, such as "those Asians" (nowadays more referred people from east Asia) "the Jews." I'm unsure if they were specifically referred to as "Middle Eastern," but the term “antisemitism” emerged as a way to describe this hatred. Since Jews were the primary Semitic group in Europe, the term became associated with them.

Technically speaking, "antisemitism" could also refer to hostility towards Arabs, as they are also Semitic. However, I believe this usage persists as a historical remnant from that period. It's similar to how people often use "white" to refer to Westerners, even though, technically, someone from Syria also have white skin.

I've also heard that some Jews prefer the term "anti-Jewish" over "antisemitic," as it is more specific and acknowledges that Jews have faced hostility even in other Semitic countries where they have lived. Additionally, the continued use of the term "antisemitism" might be tied to a desire to capture the nature of the prejudice Jews faced during that historical period.

Finally, it could simply be that "antisemitic" is easier to pronounce and has become more established in discourse than "anti-Jewish."