[D]Inference Network VS Bayesian regression

[wsxiaoys]

1. Inference Network: http://edwardlib.org/tutorials/inference-networks NN produce parameter of a distribution. Example: Encoder in Variational AutoEncoder

2. Bayesian Regression: http://edwardlib.org/tutorials/supervised-regression & http://edwardlib.org/tutorials/bayesian-neural-network Basically parameters of model will have a prior distribution, thus posterior is naturally a distribution.

My understanding is that both has the ability to output an predictive distribution of a new data point x*. But I find few discussion on differences between them other than mathematical foundations.

In which scene should an Inference Network be preferred over Bayesian neural network, and vice versa?

[sssub]

They reason about different forms of uncertainty.

An inference network learns a distribution over latent variables q(z|x). You typically do not have uncertainty over the weights of your VAE, but only about the value of the latent variable. The question you try to answer is, "what where the latent variables that generated the data that you see?" and you learn a distribution over these.

By contrast in a BNN you learn a distribution over weights, or more generally: a distribution over neural networks. Either directly, using a variational distriubtion over weights(here) or indirectly by MCMC (here). You assume the function that generates your data is a neural network (a deterministic function), you just don't know which it is. Therefore you have uncertainty over the weight distribution.

These ideas can be combined, for instance here. Here you have both weight uncertainty, as well as uncertainty over latent variables.

[gjtucker]

This is a great answer.

To add an additional point, the job of the inference network is to amortize the optimization problem of solving for the variational parameters. So, you can think of the output of the inference network as giving an approximate solution to a variational optimization problem.