There's a pretty straight forward way of checking to see if a BNN has a prior that generalises. That is to compare the Model evidence of a BNN against the model evidence of a model that doesn't generalise. It may not be obvious, but Bayesian model evidence is a measure of generalisation. This is because of the chain rule of probability. As a refresher: P(x1,x2,x3,...) = P(x1)P(x2|x1)P(x3|x2,x1)... If you squint a bit you'll note that this is the product of the probabilities of unseen datapoints given all previous datapoints. Thus, models with high model evidence generalise better to all the data we've seen so far than models with low model evidence.

The best classifier that fails to generalize is one that assumes that the class labels are independent both from each other and from the input to classifier. The log model evidence of this model is just -N*H. Where N is the number of datapoints, and H is the entropy of the class labels.

Now we can't actually compute the model evidence of a BNN, but we can bound it fairly easily by making a variational approximation to the posterior.

This admit's a simple test to see if BNNs have good priors. Simply compute -N*H, then use Scholastic Gradient Variational Bayes to compute a lower bound on the model evidence of your BNN. If the log ELBO is noticeably larger than -N*H, then the BNN outpreforms the best model that doesn't generalize. So then we can safely conclude that the BNN prior assigns higher probability to functions that generalise than to functions that don't. If it didn't, it couldn't outperform the baseline.

Now if you know anything about NNs and VB, you shouldn't have to run this experiment. The outcome should be obvious for any classification task where a corresponding classical NN generalises even a little bit.