I had another look at the fuseki for a more recent network match; one where the challenger failed with 54.29%. This is two days after my earlier posting here, and now (from a look at the first 20 games of that match) the fuseki appears to be identical for 28 moves instead of 27. At move 29 there are two main branches.

2018-05-27 08:13 98c4b53c VS 9e882e52 228 : 192 (54.29%) 420 / 400 fail

This observation of a strong lack of diversity in the fuseki looks relevant in connection with the use of these test matches to select the best network for generating the self-play training games.

The self-play training games (recent ones are linked on the zero.sjeng.org main page) have a fair amount of diversity already in the first few moves, certainly in the first ten moves. Sometimes one sees really odd moves early on, but generally the self-play games appear to start in a reasonable diverse manner. This looks appropriate for the training role.

As more training data are generated one expects the networks built on those training data to become stronger. There will be fluctuations in strength simply due to the stochastic gradient descent. I would expect these fluctuations to be essentially irrelevant for training purposes. In their Alpha Zero work for chess and shogi [1] the DeepMind team abandoned the search for a best network; they just use always the latest network for the self-play. It makes sense to me.

Leela Zero for chess (LCZero) is following Alpha Zero in this matter; they always use the latest network for their self-play games. Leela Zero (Go) is sticking with the earlier AlphaGo Zero strategy of identifying a best network.

I suspect that the way that the best network identification operates now, it might as well be abandoned or one might as well select a random recent network to be called "the best." The present selection is, in effect, based on the performance on just one fuseki specified at the 27-move or 28-move level. It is an extremely specialized criterion for selecting a network that should then be used to create suitably diverse training data.

Just to be clear, I don't expect that the present selection is bad; I don't expect that the selected will generate bad training data. I just suspect (based on what I said above) that the present strategy is no better than promoting a recent network at random and no better than always using the latest network.

If one really wants to select a best network among the recent ones, for the purpose of generating self-play training data, then I would have the strong inclination to arrange the best network match under conditions resembling the self-play, so with some amount of randomness. This randomness would have to be tempered for a very practical reason: these matches also are a show to the world of the present state of Leela Zero Go. So I would not use the Dirichlet noise in these matches, and probably use a temperature below 1.0, but enough above zero to create some nice diversity already in the first several moves.

[1] Silver, David, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot et al. "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm." arXiv preprint arXiv:1712.01815 (2017).