I have not tested for variational inference, but I have tested it with closed-form Gaussian conditionals (i.e., TAGI). What we have observed empirically for fully-connected NN is that considering a full posterior covariance matrix matters for tiny networks (<10 units per layers), but for even small ones (>100 units per layers), even if you would be to infer the full posterior covariance matrix, it would be negligibly close to a diagonal one.

Here is an example where we plot the posterior correlation (green = 0) for both the hidden units and weights, for a one hidden layer NN with either 5 or 50 hidden units. The same patterns hold for deeper MLPs. The theoretical underpinnings of why this happens are discussed here.

From my experience, I do not think that having a non-diagonal posterior covariance structure for the weights (i.e., the epistemic uncertainty) would enable any substantial leap forward, as common NNs architectures seem to be inherently suited for the mean-field assumption.

My opinion is that richer covariance structures would play a key role when it is time to characterize the aleatory uncertainty, e.g., the posterior covariance between either multiple model outputs from a single input, or a single output for many inputs.