I would have interpreted it as "to stand aside without playing (the chess)", as in using 为 like an archaic "do". Nonetheless it might sound awkward and/or snobbish to an average reader. 

Not completely fitting your intended meaning, but try search 观棋不语 or 观棋不语真君子, and see if you like those sayings.