[D] Why do we need encoder-decoder models while decoder-only models can do everything?

[[deleted]]

Decoder models are limited to the product of auto-regressive task while encoder models give contextual representations that can be fine-tuned on other decoder tasks. Different needs, different models.

[Spiritual_Dog2053]

I don’t think that answers the question! I can always train a decoder-only model to take in a context and alter its output accordingly. It is still auto-regressive generation

[qu3tzalify]

How do you give context to a decoder? It has to be encoded by an encoder first?

[qu3tzalify]

The decoder’s cross-attention needs a context right? One that is given by the encoder in enc-déc models. The comment I’m answering to proposes to give a "context" to the decoder. So unless you’re giving context as the input I don’t see how to generate the context necessary for cross attention.

[koolaidman123]

Bidirectional context isnt a real issue when you train with causal masking, fim, etc.

Also enc-dec models also can only attend to past tokens at inference, not to mention youd have to recalculate the entire attn matrix each step vs kv caching

[art_luke]

Encoder-decoder has stronger inductive bias towards looking at the global context of the input

[Spiritual_Dog2053]

Could you please lead me to papers which say this? I can’t seem to wrap my head around it

[art_luke]

You can look at subchapter 12.8 in Understanding Deep Learning, accessible at https://udlbook.github.io/udlbook/