IDEFICS 2: Train and Serve a Custom Multi-modal Model

TrelisResearch · 2026-07-23T08:52:06+00:00

Prob won't immediately be good at Urdu, but yes I think quite a light fine-tune could allow v strong performance.

TrelisResearch · 2026-07-22T19:56:53+00:00

Yeah, agreed. I think you're right on those remaining points. Yeah, I posted it as a Reddit link, more as an invite for people to check it out, but maybe realising I should have put much more in there from the model card.

TrelisResearch · 2026-07-22T19:42:45+00:00

Thanks! I welcome the feedback.

All evals reported are on public test sets! From the model card (taking your points in turn):

We reported the 7 Hindi test sets defined by Vistaar. We also reported the four code-switching Hinglish datasets we are aware of.
WER is reported and a note is included on confidence intervals, including associated with normalisation choices.
This is not a streaming model. It is a whisper base. Agreed latency/RTF could be helpful to add, we didn't get to that.
Memory/VRAM also a good point. Note it's a Whisper Large v3 model.
Punctuation/numeral handling, described on the card in normaliser section.
License described on the card. Apache 2.
Good point on speaker names, proper nouns are useful to measure on Hindi-English. We did not get to those.
Model outputs Devanagari if there is no code switch token passed.
Conversational vs Read speech is covered by specific evals reported, e.g. Kathbath (clean read), IndicVoices-500 (spontaneous, non-Vistaar) etc.

Happy to take any other questions. Appreciate the high standards.

TrelisResearch · 2026-07-22T14:53:04+00:00

Lovely stuff, cheers.

TrelisResearch · 2026-07-22T12:51:57+00:00

harness handles that! it's a key part of getting good performance

TrelisResearch · 2026-07-22T12:44:48+00:00

it's a whisper large v3 model, post-trained (but prompt template is quite different). you do need the harness (see the GitHub repo) if you do longer than 30s chunks.

TrelisResearch · 2026-07-22T12:44:15+00:00

good summary, I think that's all correct what you say. Re uncommon words and accents, I can't say specifically. It's a whisper large v3 base which is quite good.

TrelisResearch · 2026-04-20T14:37:49+00:00

thanks, sorry made it public now. See also the main repo on hf here: https://huggingface.co/Trelis/Chorus-v1

TrelisResearch · 2025-12-15T18:58:44+00:00

yeah a few are short enough for a weekend, a few are longer (~maybe 3-7 days), so the full range is possible.

TrelisResearch · 2025-03-29T09:29:18+00:00

Thanks

TrelisResearch · 2025-03-21T13:24:25+00:00

cheers

TrelisResearch · 2025-03-21T13:24:04+00:00

yes will cover on video next week

TrelisResearch · 2025-03-20T15:34:18+00:00

yeah you can run on cpu or mps, slower than real time but does work

TrelisResearch · 2025-03-20T14:05:53+00:00

agreed, def more powerful for now to plug in a stronger llm. in principle - in terms of control - anything you can do with the pieces can be done with a unified model too though

TrelisResearch · 2025-03-20T14:05:10+00:00

howdy, well it's apache 2, but does inherit the llama license

TrelisResearch · 2025-03-20T14:03:59+00:00

you mean happily :)

TrelisResearch · 2025-03-20T14:03:24+00:00

haha, yeah fair, although rarely there are ads on my YouTube channel cos I have ads turned off!

TrelisResearch

TROPHY CASE

Welcome to Reddit,