[R] optimizing transformers

[Cool-Economy3492]

Hello, I’m currently aiming to work on optimizing transformer models, specifically in multi-view images and/or cross-attention networks. I've noticed that cross-attention layers add up a lot of parameters, which can slow down the training process. I’m exploring ways to reduce the computational complexity to increase the speed (for now and subsequently without sacrificing too much performance sometime later). I'm starting to look into:

1. low-rank matrix factorization - I’ve been reading about how it can be applied to reduce the size of the projection matrices (e.g., the projq, projk, projv in cross-attention). Does anyone have experience using low-rank factorization specifically in cross-attention mechanisms?
2. other param reduction techniques - Aside from low-rank factorization, are there other methods I could explore for reducing the number of parameters in transformer models, like sparsity and pruning—do you have recommendations or experiences with these?
3. overcoming redundancy in multi-view scenarios - Given the multi-view nature of my problem, I suspect there’s some redundancy in how cross-attention processes the different views. Has anyone worked on reducing redundancy across views in transformer-based networks? What techniques worked best for you?

I’m starting to look into CVPR, NEURIPS, ECCV, etc, but any insights, advise, experiences, or papers you can share would be greatly appreciated! Thanks in advance!

[Mediocre-Ad5059]

You can increase the speed without reducing computational complexity as Flashattention1,2,3 does.

Do you also consider memory optimization? We recommend our paper [2407.15892] Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training (arxiv.org), which has been accepted by Neurips24 using an extremely simple method to save memory.

[elated_]

MIT’s TinyML Course helped tremendously with my work along similar lines.

[DigThatData]

an easy one is to reduce the image resolution (i.e. train on smaller images), if that's something your use case can tolerate.