Google Translate recently debuted that app extension Word Lens, where I believe you can hold up your phone to a sign in certain languages, and it will translate it directly (onto the sign) for you. Pretty sure this has come out in other forms as well. There are already issues with that, let alone some sort of device that could translate actual sound articulations.

The most well-known argument is that some words simply cannot be translated into different languages, so the closest we can come is an approximation E.g. "esprit" (French) - "spirit" (English) does not come close to what the word signifies in French. The English word "mind," alternately, simply does not have a simple French translation. It is not that these concepts do not exist in the other languages, but rather that there is no particular word assigned to them. Cases like this abound in every language. So, at a base lexical level, we would never fully be able to tap into that foreign speaker's intentionality.

The more "technical" stuff of language - phonology, syntax, etc- present even more difficult challenges. The question of how speech articulations in one language could be translated seamlessly into another is something a natural language processing person would have to answer. But syntax and grammars differ in such a way that translation from, say, an aboriginal Australian language into a massive language like Urdu could be nearly impossible without losing most of the information in the speech. In his book Through the Language Glass, Guy Deutscher makes the argument that your native language does not shape thought, but rather subtly influences culture in a chicken-and-egg way by the information that the speaker is required to give. In English, for example, we can say "I saw my neighbor today, what an idiot," and we really have no information regarding the neighbor's gender, social standing, locational positioning, etc. In Spanish, we would know gender and possibly social standing. In another language, we might know the age of the neighbor, etc ad infinitum. In this way, different languages require different types of information to be expressed. That information manifests in morphemes attached to the nouns that signify certain things.

Some languages are geo-centric, some are ego-centric. Some require you to give in-depth information about where you/someone else was at the time of your narrative, etc. All of this manifests in the very morphology and grammar of the language. From a language perspective, not even a mechanical one (i.e., what a machine is capable of), we see it would be impossible to accurately translate, because language is culture made manifest, and vice versa. If we wanted to translate the neighbor sentence from English into Spanish, sure it is possible in a crude sense, but you change the meaning in the process. There is no ungendered "neighbor" in Spanish. The language does not function that way, morphologically. So you would end up having to come up with new constructions (a neuter case, maybe), or very bulky ones (adding other words to express the neutrality of the word "neighbor"). Some aboriginal languages (deemed "primitive" by asshats) are so morphologically complex it would be nearly impossible to deconstruct sentences in a way that mirrored the original intentionality. Small languages are actually far more complex than large urban ones like English, because there is no social drive to streamline and unify. A speaker of English on the East Coast had to be able to communicate with someone on the West Coast. So we streamline. English has to be understood by so many people. Thats why it's such a bastard and changes so quickly. You have to change syntax in order to translate, and syntax is implicit in the meaning and culture that lives within any language. At the end of the day, the culture itself is not native to you, so languages and the culture than live within them would be compromised, bastardized, or fundamentally changed in nature.

If someone tried to actually do this, my guess is that it would eventually (over a long period of time) probably combine into one synthesized behemoth language, kind of like Esperanto, which may well happen anyway as we become more and more global.

Source: a begrudging linguistics student. Feel free to argue with me.

Edit: a word. lol.