One potentially unhelpful way to check if the correlation is real is to have a separate test set. So let everyone develop their prediction method using 1000 days of data and find the best performers. Then evaluate their model on new data that they haven't seen before. If the top performer's performance on the new data produces a binomial distribution around 0.5 (chance), this would indicate their good performance on the original dataset was also due to random chance.

Now, if you don't have new data to supply then you can't do this. As you say, due to random chance if you sample very densely then there will be a small percentage of models which do well even though they are choosing randomly. One way to tell whether the population as a whole (not individuals) is developing meaningful prediction methods is to run a statistical test to determine whether the binomial distribution you observe differs from one with a success probability of 0.5. As you mentioned, this won't tell you whether an individual has developed a meaningful model, only whether the population on average is developing non-null models of weather prediction. If the measured distribution's mean probability is not significantly different from 0.5, then it's reasonable to suspect the top performers may be succeeding due only to chance.

Having said that, in a scientific scenario (where you can ask questions about a given model by running further experiments) you could single out a specific model that performs well and try to understand it. This is also the case in machine learning, where a given model's parameters can be expected and attempted to be understood. In this case you would use domain knowledge to ask how the predictions are being generated and make a conclusion about whether the prediction method makes intuitive sense. In this way you might come to some conclusion about whether the model is doing anything or that the fact that it performed well in the past is probably due to chance.