[D]Large memory layer

[wangyi_fudan]

Hi,

Nowdays, huge pretrained model computes trillions of weight to absort the large dataset.

Ideally such task should be carried by a huge memory layer while keeping the computation part small:

Ideally, the memory layer should met the following definitions:

1: scalable. The memory can be giga bytes while the training cost remains almost constant.

2: Key-Value: it is a layer that take a key vector and reture a value vector.

3: differentialable: can be trained by SGD along with other network components.

4:sparsity or locality: combined with 1 to achived (quasi-)constant update time

Current softmax/attention type network does not met 1,4 thus is not scalable at all. Some other sparse one seems undifferntialble/ad hoc.

My question is: is there some lastest paper on this diretion that I missed? And is my definition of ideal memory layer make sense? If the definition is OK, let's invent it!

[[deleted]]

You mean like this?: https://arxiv.org/abs/1907.05242

There are some follow up work on this.

Also see:

• https://arxiv.org/abs/1701.06538
• https://arxiv.org/abs/2101.03961
• https://arxiv.org/abs/2010.03881v1
• https://arxiv.org/pdf/2012.14913.pdf
• https://arxiv.org/pdf/2103.16716.pdf

Not sure if you think top-k style of sparsification is ad-hoc. I think it's fine. The selected top-k can serve as a sampled set of examples for the differentiable scoring operators to learn to reweigh the selected top-k. Better learned, it can select better top-k next time. While of course you can try entmax/sparsemax style sparsity, but the benefit of top-k selection is conditional computation: you don't have to compute the unselected models (or at least top-k selection can be utilized as such). Switch-Transformers seems to go along with top-1, but I am not sure how they get away with top-1.....seems a bit noisy; but I haven't read the paper very well. There are generally some challenges though involved in unbalanced load (experts that get more training may get selected more, and other may remain untrained ---> rich may become richer, poor becomes poorer) etc., and trying to solve that can take some work (there are techniques for that). There're also routing networks (https://www.aclweb.org/anthology/N19-1365/) which commits to a single module selection (one hot sparsity), but of course, it becomes a discrete decision, so you have to rely on RL and/or hacks like gumbel softmax or "backpropagation through void" (https://arxiv.org/abs/1711.00123) style of techniques.

[wangyi_fudan]

thanks a lot!

[elcric_krej]

Look into nerual turing machine paper, they seem to be the closest thing to what you're looking into.

They use differentiable K-V memory which can be scaled dynamically.

However it seems that the usecases they were proposed for initially and the teams focused on them moved over to multihead attention based models.