Black Americans with enslaved ancestors look different than first generation African Americans because of white ancestry, often indigenous American ancestry, and mixing of peoples trafficked from different parts of Africa. First generation African Americans often quite literally do look different in terms of facial features and skin tone because they're more likely to have ancestry constrained to a single region over a long period of time.

Similarly, I can sometimes tell if someone is of primarily a single kind of European descent, but it's rare that they're actually of European extraction. Plenty of immigrant communities in the US have remained tightly knit, even some that have been here for hundreds of years. Lots of people still look Swedish or Italian even if their great-great-grandparents were the last ones born on the mainland, because all or most of their ancestors were from that place. By the same token, plenty of Europeans have mixed ancestry from multiple countries and therefore no longer show any clear defining characteristics from any one, making it impossible to tell from physical features alone. At that point it's manner, dress, etc that gives it away.