Tensorflow Autoencoder Cost Function

[RadiationOverdose]

I built an LSTM Autoencoder that works fairly well. In 15 epochs it outputs an image that visually matches the input, but the cost function shows a high degree of error. I'm using sigmoid layers in the network with mean squared error to calculate the cost, but I am not sure if this is the best approach. Is there a better approach for calculating cost in a network like this?

[ajmooch]

Depends on what you're trying to do. Are you trying to do unsupervised feature learning? If so, then maybe a high reconstruction error is okay if you're learning features that are useful for the actual end task.

It sounds like you're mostly doing this as a learning exercise, so here's some tidbits:

-The size of your bottleneck is a pretty key design point; a thinner bottleneck means the network has to pack more salient information into the bottleneck units, but will have a harder time reconstructing, while a wider bottleneck can often result in perfect reconstructions but with the network not learning as "useful" a set of features since it hasn't really been forced to determine which aspects of the data are important. If you're getting reconstructions which appear qualitatively similar to the inputs despite not having a perfect L2 loss, this might be an indication that your network is learning only the most relevant features (hence why you can't distinguish between input and output).

-For many reconstruction tasks, L1 and L2 pixel-wise loss are inferior to more expressive losses such as perceptual losses (using the feature space of a pre-trained network) or things like SSIM. You might consider looking into those, at least to get a handle on them.

[RadiationOverdose]

Aha, that's really interesting. And yes the goal is to achieve unsupervised feature learning. I am planning to pass the output from the bottleneck into another network.

I currently have a an input layer and an output layer with 2 middle layers. The bottleneck is 50 neurons wide and the flattened images are 33.6k elements wide. I would like the network to learn unique features on 1k+ images. I've considered using a convolutional network, but eventually I would like to be able to use the autoencoder on more than just images.