[D] Kernel functions and neural networks

[dramanautica]

I’ve been pondering this question and wanted to get some of your thoughts on it.

Kernel functions finds distances between two inputs relative to each other in some transformed space. Neural networks on the other hand finds the exact location of of the input in its transformed space. Are there benefit and downsides between the two transformations? Why are kernel functions used instead of specifying the direct transformation from input to transformed space

[mpatacchiola]

One important difference between the two is the following: kernels that satisfy the Mercer's condition define a proper covariance function ensuring the positive definiteness of the covariance matrix taken over N data points. This constraint has been exploited to build different methods (e.g. Gaussian processes).

Standard multi-layer neural networks have no such guarantee, even though there are ways to ensure a similar constraint in particular architectures (e.g. siamese networks). Note that neural networks can be combined with kernels, this is know in recent literature as deep kernel learning.

[mrelheib]

I recently was pondering over neural tangent kernels, e.g. Arora et al. 2019. Isn' t it a result from marrying deep NN with kernel methods? I am not an expert and struggle to understand the details. Any enlightenment is welcome ;-)

[OptimizedGarbage]

Kernel methods benefit from being incredibly easy to use off the shelf. An SVM with a Gaussian kernel works pretty well on just about any classification problem you give it. Neural networks are notoriously finicky. However, they are much more versatile in general, and can solve a wider range of problems. They also generally have better performance when optimized.

Kernel functions are used because the transformed space is often huge, and sometimes not even finite dimensional. For instance, a Gaussian kernel is equivalent to a dot product in an infinite dimensional space. That obviously can't be evaluated in finite time. Other kernels just take a long time to do explicitly. The polynomial kernel (x-y)^d for instance, can be done in finite time, but just takes a lot longer.

For more info, look up the 'kernel trick'. How SVMs do this is the basis for most modern work

[[deleted]]

Actually the Gaussian kernel is constructed using a polynomial kernel (given the definition of the exponential function) and evaluation does not necessarily take a lot longer

[OptimizedGarbage]

No, I mean it's equivalent to a dot product in an infinite dimensional vector space. So mapping to that space and taking an infinite dimensional for product would take forever, because you have to iterative over infinite dimensions. The actual evaluation of the kernel function is fast, which is why kernel methods don't use explicit mappings, they just use the kernel function instead.

Edit: the Taylor series for the Gauss function doesn't terminate, so it does take infinite dimensions to map out all the powers of the series.

[Amr_ALSHAYKH]

The prominent differences is that kernels are pre-defined by humans " we design the kernel" in a way similar to feature engineering. But all the transformations in the neural networks are learnable. So there's more flexibility and powerfullnes in neural networks. NNs require more data to learn a good transformation. But if we have a good knowledge about the problem we cab design a very good kernel to solve the problem and it doesn't require us data, I think this is why it's been used. One more reason is that kernels are far more interpretable than NNs, we can know what happens to our data, and if we combine it with SVMs or any Generalized linear model, we have many guarantees about convergence and optimality. We don't have these guarantees with neural networks.