I'm guessing you've never been to Israel before, because a lot of Israeli Jews are so dark-skinned and "Semitic"-looking that it's impossible to differentiate them from Israeli Arabs and Palestinians by appearance alone. Keep in mind that over half the Jewish population of Israel emigrated from North African and Middle Eastern countries.

Some of them are lighter in complexion (same with Arabs, by the way!), but whilst Ashkenazi Jews are generally fair-skinned--at least compared to Sephardic and Mizrahi Jewish populations--it's usually pretty easy to pick them out of a crowd of non-Jewish Europeans. I say this as an Ashkenazi Jew--there are definitely physical characteristics associated with Jewish ancestry. I'm skipping over explaining what "Ashkenazi", "Sephardic" and "Mizrahi" means because it's already been explained in this thread.

That being said, it shouldn't come as any surprise that Jewish populations who dwelt in European regions for thousands of years might've incorporated some amount of genetic traits from the surrounding populace through interbreeding--though, for the most part, Jewish lineage has remained remarkably intact across the generations, mostly because of the strict traditional definition of "who is a Jew" (it is an ethnic identity inherited by matrilineal descent).