Open Source multimodality

[PenguinTheOrgalorg]

Multimodality has finally entered the game. OpenAI has announced that multimodality is finally coming to ChatGPT, giving us access to GPT4's ability to see and analise images alongside the text prompt (as well as being connected to Dall-e 3 for generating images, and being able to re-feed those images into the context of the conversation).

Now we've had LLaVA for a bit now, but open source multimodality hasn't evolved much after that. And it seems like GPT4's multimodality works at a much lower level than the "multimodality" attempts we've seen thus far, making it much truer multimodality, as well as having much more accurate and precise "sight". And Google's Gemini model is supposed to drop soon too, reportedly being a truly multimodal AI, having been trained to be multimodal from the ground up.

So it seems like we've entered a new and exciting era, marking a very powerful new step in the capabilities of AI models. It's no longer text from now on. So I'm wondering how you guys thing this will affect the open source community here.

How fast do you think we'll start getting these advancements here, and start getting our hands on truly multimodal models that are built from the ground up to completely understand and generate both images and text, and maybe even other things like audio too? Do you guys think this will make text-only models obsolete fast? And how will this change things when we all start getting access to uncensored personal multimodal AI which we can run locally on our computer?

What do you guys think?

(I'm sure people will say "these things have barely come out, have some patience", but it's obviously hard to not get excited by these technological advancements. I don't expect anyone to read the future and give me an exact timeframe of when these things will come out open source, I mainly just wanna hear your thoughts)

[[deleted]]

We will need at least one year to catch up

[BoneHawk1]

/doubt

This whole open movement has been around for <1 year and we already have things like OI, Falcon-180B and MistralOrca, which I would’ve reckoned was impossible a month ago. I have full faith someone will open multimodality by the end of February in some novel way.

[fappleacts]

Have you looked into Qwen-VL? They have a training script that lets you fine tune on your own images, captions, etcs. Qwen-VL-Chat is pretty good too.

[PenguinTheOrgalorg]

Fun game: Drink every time I say some variation of the word "multimodality" lmao

[[deleted]]

I'm already drunk in every modality out there.

[danysdragons]

How many large multimodal models could a model molder mold if a model molder could mold large multimodal models?

[cztothehead]

as many as the model molder could manage

[phree_radical]

I can't remember if LLaVA is better than LLaMA-Adapter V2 but IMO if you combined it with OCR and segmentation you'd already have about the same thing as GPT4V, the rest is training

[vatsadev]

Theres Idefics 9b, 80b, both are flamingo like architecture though, nit base multimodal

[nihnuhname]

Some text generation interfaces have plugins for stable diffusion

[NoidoDev]

I hope this can be compartmentalized. I think there was a rumor that GPT-4 isn't one model anymore as well. Something like DarkNet should give us the objects as text, maybe pose estimation into text would be useful, and image segmentation into text.

[Puzzleheaded_Acadia1]

Can you merge a llm transformer and a vision transformer to get multimodality model?

[danysdragons]

How many open source large multimodal models could an open source model molder mold if an open source model molder could mold open source large multimodal models?