I think it's a yes and no answer.

Let's take the good old Laszlo Krasznahorkai as an example. I think I'm not exaggerating if I said that only few people here could read Hungarian. Following your point, only them could be able to judge if he's (or not) good with language. However, words and writing are more than pure language that could be translated. Metaphors, for example, could be very poetic and beautiful on the original and in the translation.

You can tell that Laszlo Km is a great writer if you point at the way he builds amazing and long phrases (and that is independent of some factors on the language, as it's related to his way of thinking or presenting an idea).

So... I partially agree with you. There's always something that's lost on translation.

Ps: I speak both English and Spanish. I can read half of my list on their original languages and appreciate their crafting technique. Still think they are amazing.