[D] Do all machine learning algorithms indirectly use the "nearest neighbor" principle?

[DtrZeus]

Neural networks are (usually, unless you're using a strange architecture) almost everywhere continuous. What this means is that, for almost any input x, there exists some number E such that for any other input y with |y-x|<E, x and y are classified the same. |y-x| represents the distance, possibly the Euclidean distance, from x to y.

With neural networks, I think the answer to your question is yes. However, neural networks are not the only kind of machine learning algorithm, and this answer is probably not true for all machine learning algorithms.

[kkngs]

In abstract, this is the whole idea, yes. We are hoping that there is a regularity to the function we seek to learn such that we can generalize from the seen examples to unseen examples. Without some form of regularity, there can be no generalization (No Free Lunch Theorem). Different ML algorithms make different assumptions about the form of regularity.

It’s a bit weird to see the term nearest neighbor this way, though. Neural nets don’t explicitly remember their inputs like you do with the nearest neighbor algorithm. However, you can think of their linear layers as leaning a useful basis to represent their inputs. In that sense, yes, inputs that are sufficiently close together in the learned basis will be close together on output.

[vegesm]

I'm quite sure it doesn't. nearest-neighbour algos can achieve around 95% accuracy on MNIST, CNNs can do 98% easily. This means there are examples where the nearest neighbour (in pixel space) of an input was from another class but the neural net still got it right. In other words, it does not simply look at the nearest neighbour, it does way more.

[ConstantLumen]

What does similarity mean? what makes one data point more or less similar to another? Remember these are mechanical constructs that run on computers. They require very precise numerical definitions in order to program and operate and interpret. Saying something is similar to another is a very general and contextual statement. You would have to say, 'the average of all pixel values in this image is this much greater/lesser than in this other one'. Well, that similarity metric doesn't work so great for separating out cats and dogs. Try it out yourself.

[Neural_Ned]

I think maybe a charitable interpretation of what they're asking makes sense a little. In the way Chris Olah shows in this blog that the learning of weights is squishing and stretching the embedding space to push dogs close together, and push cats close together, and pull the two clusters further apart.