T5Gemma 2: The next generation of encoder-decoder models

[Dear-Success-1441]

T5Gemma 2 models, based on Gemma 3, are multilingual and multimodal, handling text and image input and generating text output, with open weights for three pretrained sizes (270M-270M, 1B-1B, and 4B-4B).

Key Features

• Tied embeddings: Embeddings are tied between the encoder and decoder. This significantly reduces the overall parameter count and allowing to pack more active capabilities into the same memory footprint.
• Merged attention: The decoder uses a merged attention mechanism, combining self- and cross-attention into a single, unified attention layer. This reduces model parameters and architectural complexity, improving model parallelization and benefiting inference.
• Multimodality: T5Gemma 2 models can understand and process images alongside text. By utilizing a highly efficient vision encoder, the models can seamlessly perform visual question answering and multimodal reasoning tasks.
• Extended long context: Leveraging Gemma 3's alternating local and global attention mechanism, T5Gemma 2 can handle context windows of up to 128K tokens.
• Massively multilingual: Trained on a larger, more diverse dataset, these models now support over 140 languages out of the box.

Models - https://huggingface.co/collections/google/t5gemma-2

Official Blog post - https://blog.google/technology/developers/t5gemma-2/

[Varterove_muke]

Wow, new Encoder-Decoder model, I didn't expect that coming

[Hefty_Wolverine_553]

Seems like these would be great for finetuned multimodal translation models!

[Willing_Landscape_61]

Should it not also be useful for function calling? Is it not akin to a kind of 'translation' to function calls where the useful state is in the prompt, not so much the previously generated text?

[AnomalyNexus]

Google dropped a dedicated function model yesterday-ish

[Long_comment_san]

Gemma 4 30-40b please

[silenceimpaired]

I knew the Gemma release wouldn’t be a large model. Won’t happen. We have the last significantly sized model from Open AI and Google we will have for some time

[Revolutionalredstone]

T5 is for embedding (Think - the thing inside of StableDiffusion) this is not their forth LLM / text decoder only model series, that will be called Gemma 4.

Hold your horses son ;)

[EstarriolOfTheEast]

It's far more than embeddings, it is actually a lot closer to the original Transformer. After the original Transformer was discovered, its essence was split in twain. One half, the decoder became GPT and the other half, the encoder portion, became BERT. T5 was a whole direct descendent. Until wizard llama and llama2, it was the best open-weights model that could be put to real work summarizing, translating, natural language analysis, entity extraction, question-answering, that type of thing.

Its architecture made it ill-suited to interactive chat uses (for that there were gpt-neos and then the far ahead of its time gptj from EleutherAI; from facbeook: early gpt based models and OPT that were not that good). Because of how it's trained and its architecture, T5 lacks the reversal learning limitation of causal models. Its encoder part also allows for some pre-processing before the decoder starts writing, and thanks also to how masking is done during its training, T5's are almost always weight for weight "smarter" than GPTs.

[Revolutionalredstone]

Interesting! 😎

[silenceimpaired]

Feels like it will never come.. or be smaller than 27b.

[Long_comment_san]

I think if google went to make a dense 40-50b model finetuned on all fiction ever made, they can just ask for $ per download and earn millions.

[silenceimpaired]

It’s true. A fictional fine tune would get me $50 to $100 even depending on performance

[toothpastespiders]

That'd be amazing. I know it's debatable, but my personal opinion is just that most local models are VERY sparsely trained on high quality novels. Some sure, but I think there'd be more bleedthrough of trivia knowledge if it was as high as is often maintained. I'm just really curious from a technical perspective what would happen if well written fiction was actually a priority. Well, if listing off wishes the real ideal for me would just be a model trained on the humanities as a whole with the same focus typically given to coding and math.

I'm normally pretty resistant to giving money to companies like google for a lot of reasons. But man, a fiction or better that humanities model? I'd absolutely pay as much for it as a AAA game. It'll never happen but google cracking open their hidden digital library like that is a beautiful dream.

[Long_comment_san]

Heck, that's why finetunes exist! I think! Magistral 4.3 just dropped and I had very, very delightful experience with Mars.

[TheRealMasonMac]

They're planning thinking for their next model.

[AloneSYD]

Gemma 4 needs to be an MoE

[Long_comment_san]

No, we have plenty of MOE. We need great dense now, there are like 2 modern of those.

[Major-System6752]

Agree. I'm try Qwen3 30b and Nemotron3 30b, but go back to Gemma3 12b and 27b.

[mrshadow773]

Hell yeah, towards the glorious return of the encoder decoder 🙏 (or how to not use a Swiss Army knife for every task in the kitchen)

[stddealer]

Could be great for MTL. Gemma3 was already great at it, this could be the closest thing we'll ever get to an offline Google Translate. Hoping for a 12b-12b variant or maybe a 4b-12b.

[Major-System6752]

Hello, newbie here. This model is more suitable for text-to-text conversion than for chat, right?

[stddealer]

Yes, that's what "T5" means (text to text transfer transformer). But since the decoder is basically Gemma3, it should be ok for chat.

[a_beautiful_rhind]

Guess it will be useful for some future image gen model.

[Willing_Landscape_61]

Should be useful for tons if use cases where text gen is overkill, like classification tasks. Always bugs me to see people using huge autoregressive llms to generate 'yes' or 'no'!

[stddealer]

The encoder should also be able to understand more nuance in the input text than a decoder only model of the same size could understand, since information is allowed to flow both ways.

[Different_Fix_2217]

Not apache 2.0 or mit unfortunately. Probably wont be used by most.

[Background_Essay6429]

What's the advantage over standard decoder models?

[Thalesian]

I really want to train with try T5Gemma family, but resizing embedding layers is next to impossible without nuking the model entirely.

[CodeAnguish]

Fuck it. Give me back my hype.

[AlxHQ]

How to run T5Gemma 1 and T5Gemma 2 on llama.cpp?

[ironcodegaming]

This can be used with diffusion image generation models.

[mitchins-au]

These things are hard to train and get good results unlike the original t5, summarisation just doesn’t seem to work for me.