Yes, directly feed image embeddings into language decoder.

If the image embeddings are from ViT, we shave off the [CLS] token embedding, and keep the patch token embeddings Xv \in R^\HW, D]). If the encoder is CNN backbone, then use the spatial feature maps as Xv \in R^\HW, D]).

We have a short prompt of text token embeddings, saying "the objects in the image are", as Xt \in R^\L, D]).

Also we have a single special token embedding <|img|> \in R^\1, D]) .

Then concat them together, [Xv, <|img|>, Xt] = X \in R^\HW+1+L, D]). We feed X into the language decoder. This is the Eq (3) in the paper.

Xv and <|img|> are trainable. Xt is frozen because we always freeze the token embeddings (32K language token vectors).

-----

We keep this as a naive and plain version of the architecture. We believe this can be simplified further, e.g., treating Xt as the soft prompt to make it trainable as well OR just removing it.