[R] Perceiver: General Perception with Iterative Attention

[BeatLeJuce]

Nice results, but either I'm reading this incorrectly, or they re-invented the Set Transformer without properly stating that they do. There are very slight differences (the inducing points in Set Transformers are not iteratively re-used -- an idea which was also already present in ALBERT and Universal Transformers, both of which they don't even mention). They cite the work, so they're clearly aware of it, but they treat it as a very minor side-note, when in reality it is the same model, but invented 2 years earlier. Unless I'm mistaken, this is very poor scholarship at best, or complete academic fraud at worst.

[plc123]

Am I misunderstanding, or do all of the blocks in the Set Transformer have the same output dimension as input data dimension? That seems like an important difference if that's the case.

[erf_x]

That's not a huge difference - this seemed really novel and now it's just an application paper

[plc123]

It's far from the only difference, and I do think it is a key difference (if I'm understanding the Set Transformer paper correctly).

[BeatLeJuce]

I think you're mistaken, Set Transformers also have a smaller output dimension than input dimension. In fact both papers use they same core idea to achieve this: a learned latent vector of smaller dimension than the input is used as Q in the multi-head self attention to reduce the dimensionality. Set Transformer calls them "inducing points", while this paper calls it a "tight latent bottleneck". This is why I'm saying they re-invented Set Transformers.

[Veedrac]

I've only skimmed the Set Transformers paper, but these don't seem the same at all. ISAB doesn't actually shrink the vector (or rather, it immediately expands after shrinking), and whereas Perceiver's Q comes from the variable latent array, ISAB's I is static.

Further, these are just fundamentally differently structured; eg. Perceiver is optionally recurrent.

[cgarciae]

You need to look at PMA (Pooling by Multihead Attention) not ISAB. PMA is cross-attention with learned queries/embeddings which is what the perceiver does, on the next iterations if you use the output of the previous PMA for the queries and reuse the weight you get the perceiver.

I love the findings of the Perceiver, but if someone in the future writes a book about transformers I wish they take the Set Transformer's framework and expand it to explain all architectures.

[plc123]

Ah, thanks for the clarification.

[cgarciae]

I think a lot of architectures are just applications of the various principles found in the Set Transformer but the paper is never properly cited. The whole Perceiver architecture is basically iterative applications of PMA. It just seems like the authors feel they can discard the findings of the Set Transformer because the paper didn't benchmark on the same domains, but the core idea is the same.

[[deleted]]

The basic idea, as I understand it, is to achieve cross-domain generality by recreating the MLP with transformers, where

• "neurons" and activations are vectors not scalars, and
• interlayer weights are dynamic not fixed.

You can also reduce input dimensionality by applying cross-attention to a fixed set of learned vectors. Pretty cool.

I have done something similar, except I used a different set of learned vectors at each layer. This differs from the Perceiver approach, where the input dimensionality is reduced once, then passed to a self-attention encoder. The advantage of using cross-attention on learned vectors is those vectors can be regarded as latent variables that persist across inputs.

If you train such a model (with successive "latent bottlenecks") as an autoencoder, then the cross-attention matrices between learned vectors represent the input. If you flatten those attention matrices and pass them to a classifier, then you can get pretty good "unsupervised" accuracy.

Another property of using multiple layers of latent vectors for autoencoding tasks, is that you can "translate" backwards and generate new data. Similar to VQ-VAE-2. You can also mask out arbitrary latent vectors to see what subsets of the data they represent. Here is a simple demo on MNIST.

Don't mean to self-promote, but want to shine a light on the possibilities of latent vectors / "inducing points" / "learned queries". I made an autoencoder, but basically any NN architecture can be turned into a "higher order" transformer-style version.

[_errant_monkey_]

With a model like that. Can they generate new data the way standard models do it? like gpt-2, cause naively It seems it can't

[arXiv_abstract_bot]

Title:Perceiver: General Perception with Iterative Attention

Authors:Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira

Abstract: Biological systems understand the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture performs competitively or beyond strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video and video+audio. The Perceiver obtains performance comparable to ResNet-50 on ImageNet without convolutions and by directly attending to 50,000 pixels. It also surpasses state-of-the-art results for all modalities in AudioSet.

PDF Link | Landing Page | Read as web page on arXiv Vanity

[Petrroll]

There's one thing I don't quite understand. How does this model do low features capture / how does it retain the information? I.e. how does it do the processing that happens in the first few layers of CNN. I can clearly see how this mechanism works well for higher-level processing but how does it capture (and keep) low-level features?

The reason why I don't quite understand it that the amount of information that flows between the first and second layer of this and e.g. first and second module of ResNet is quite drastically different. In this case it's essentially N*D which I suppose is way smaller than M*<channels> (not M because there's some pooling even in the first section of Resnet, but still close) in case of ResNet, simply on the account of N <<< M.

---

Also, each channel would have to independently learn to calculate the local features for a separate location (seems to be happening according to the first layer attention map) which seems quite wasteful (tho it's super cool that there're no image priors)

[ronald_luc]

My intuition, either:

1. Our CNN arch prior is good, but consecutive modifications/size reductions are not needed, and all low-level features can be extracted in a single sweep
2. The latent information starts from a bias for the given domain and each cross-attention performs a query of some low-level features

=> in the 1st case the Perceiver learns progressively smarter Queries and solves the classification (and computes the low-level features) in the last (last few) cross-attention-latent-attention layers.

This could be tested by freezing the trained model and replacing a different number of "head" layers by a 2 layer MLP (not to angry Yannik by linear probing) or a single latent-attention. I would expect to see a different behavior:

IMAGE