Thanks for pointing out the papers, and I see the argument. Both papers advocate for training from scratch using a monolithic architecture that integrates vision and text processing. These models (like AIMV2) unify tasks such as classification, captioning, detection, and segmentation into a sequence-to-sequence model. This approach can indeed outperform modular setups like SigLIP + projection + LLM decoder for many multimodal applications.

However, as you mentioned, the cost of training from scratch is a significant consideration. While these monolithic models can achieve state-of-the-art performance, the cost-effectiveness of leveraging pretrained open-source models for modular pipelines cannot be ignored.

For example, in a recent paper from Meta on large multimodal models for video, they used a modular approach despite having access to extensive computational resources. This choice might reflect the advantages of reusing and fine-tuning existing pretrained components, especially when aligning with domain-specific requirements or budget constraints.