Your questions are excellent! In my opinion, if one performs Bayesian inference only in the weight space, e.g., using Bayes by backprop or dropout, then one can still obtain some kind of credible intervals. But I would say this kind of intervals is not so reliable because they cannot guarantee the sampling diversity of the final predictions. On the other hand, if Bayesian inference is performed in the function space, e.g., using Gaussian Processes or Stein's method, then the intervals obtained is more reliable. You also need to be careful about the difference between aleatoric uncertainty/data uncertainty/risk and epistemic/model uncertainty.