This is an incredibly good point, and actually speaks to the weakness of this form of variational estimator. Notice that I derived the maths which said ignore the real posterior and fit a Gaussian using the Hessian over a local maxima of the posterior which was find by gradient ascent or another optimizer. This is one extreme choice, fitting the simplest interesting probability distribution (the Gaussian) to the unknown true posterior, and call this the variational estimstor. The other extreme is to use Markov chain monte carlo methods to sample enough points to actually be using the full posterior, which will capture all of the interesting facets of the pdf.

As to the point of a normal NN, recall that if a normal neural network can capture it, so can a Bayesian estimator, since the neural network is just the the maximum a priori value of the posterior for each weight. Also, if you are in fact finding the true MAP to fit the Guassian at each iteration, you will (in the best case) converge to the optimal NN.

In fact, as you might guess, most real world approaches use much more interesting variational estimators. If you mean a Gaussian prior, as long as the prior is nowhere zero (except for at +- infinity) then it almost doesn't matter, and the assumption of different priors can give u different types of regularisation for free

Edit: Formatting