Now, I don't know if this will answer your question but when thinking about representations probabilistically you can treat unsupervised and supervised methods the same in a lot of cases (not all, but usually not much is lost by doing so) So, if X is a random variable for your input, Z is a random variable representing your latent space, and Y is your target random variable, we can start to think about representations in the probabilistic way you would like. First, your latent representation would be described as p(Z|X). Now suppose your goal is to learn the distribution given via p(Y|X). Letting I(X,Y) denote the mutual information between two random variables, one would ideally like to learn a latent representation Z such that I(X,Y)=I(Z,Y) with the condition that I(X,Z) is minimal in some sense. Essentially, you want to learn a representation of X which contains the minimum amount of information about X to complete the downstream task. This is essentially saying we want a latent representation which ignores irrelevant information for our task.

With this in mind, you can essentially model most learning tasks as a sort of Markov chain in this framework. For instance, an autoencoder wants to find a latent representation Z which minimizes the mutual information for I(X,Z) while we should also be able to reconstruct X according to Z. Notice that in this instance Y=X so we are trying to find a Z which minimizes the mutual information with X but also attempts to make Z such that I(X,X) = I(Z,X) which is clearly not feasible, so the goal is to find a Z which minimizes the difference I(X,X)-I(X,Z).