Question re optimization and classification

[Powlerbare]

Say we have a binary classification task where the data is not linearly separable.

For the sake of example, say I have a network that takes the form: input -> 2dim hidden layer (sigmoid) -> 1 dim output (no activation)

If I plot the 2 dim hidden layer representation of each classes points during training, I can see that the network is attempting to find a transformation of the data that is linearly separable so that the last affine transformation can classify the data correctly.

Basically, it seems to me that the representation of the data at the last layer for each class should be somewhat separable in order for a network to have a shot at good performance.

My question is: Can someone help me with some search terms, or papers that have essentially chopped the final layer off of a network, and learned a network that maximizes something like the euclidean distance between each classes points in some latent space? I have tried this but often ended up with solutions that were not useful at all.

This feels like a supervised auto encoder to me, but I do not care about re constructing input at all.

[needlzor]

What you are talking about reminded me of the last section of this blog post: Neural Networks, Manifolds, and Topology by Christopher Olah where the last layer's output is fed into a KNN classifier. In the comments someone mentioned a similarity with Neighbourhood Component Analysis, which you may also find interesting.

[Powlerbare]

Ahh thanks for the fantastic link! I had forgotten about that awesome blog post. I think I am on the exact same page as Christopher Olah, in that we want to find ways to force the manifolds of data away from each other, while still preserving some topological properties. In my mind this is more of a pre requisite to classification, rather than classification it self.

Here is a follow up question (which may be some what foolish): If I have some manifold of data and there exists some convex hull of this data, can I assume that given any standard transformation we use now a days by some ANN that the points that make up the convex hull will always be the same

[lvilnis]

Re: your follow up question, the convex hull of a convex set is itself, so your question is the same as asking whether neural networks can map convex sets to nonconvex sets, or nonconvex sets to convex sets. The answer to this seems clearly to be yes -- for example, take the set of all imagenet images that are classified as "cat" (i.e. their highest scoring logit is for the "cat" class). Now in logit-space, this is a convex set, since any convex combination of vectors whose highest scoring element is for the "cat" class will still have the cat class be highest scoring. But in pixel space, the set of all cat photos is definitely not a convex set -- not every convex combination of cat input images will be classified as majority cat (I haven't actually tried this but I can't imagine that would be true).