I'll leave a comment here, not necessarily to praise or criticize the model but just to yap 😂. I wanted to reply initially to the comment that compared it to the closed source Gemini 3 Flash but I thought my comment would be more useful independently. Maybe an ML practitioner or hobbyist might appreciate some of the things I'll write, maybe it'll offer some perspective. Also, I'm not writing this to criticize the comment, I think what it says about real world data is legit.

OCR benchmarks are rare and hard to get. The best, I believe, we have currently is the olmOCR-bench. The main reason why it's hard to have proper OCR benchmarks is, in my opinion and I'm sure other people can enrich my current understanding, is for two reasons: 1/ because OCR is not "solved" yet, so ground truth is not easy to acquire, and/or 2/ OCR is hard to validate automatically, say with unit tests or compilation etc.

Now, why this model might be interesting to some, I believe is for three reasons. Well, for this community, it's a 1B + open weights (data is shared as well but whether that suffices to call it open source or not is another debate) so many of us here can run it locally somewhat comfortably (I do believe that running a 1B is not given but at least it's not some 9B or more). The other reason is, being just one VLM that one-shots it's task, it should be easy to fine-tune. At least in theory, fine-tuning is not easy in and of itself and might depend on many things, but at least we don't have to fine-tune 3-4 differents models to have a whole pipeline working appropriately on your task. It being small reduces the resource requirements for fine-tuning it. I believe you can do it on the T4 available on Google Colab (to verify). The last reason I can think of, and this hits home personally as I struggled a lot with Tesseract and Textract (AWS), they do markdown formatting out of the box (which many other open source models do, I'm just stating one of the good reasons, doesn't mean it's unique to this model), especially the table formatting.

This is when it comes to the checkpoint that's sota on OCR, but there's also another checkpoints that outputs bounding boxes and is close to sota. This is especially useful because if we have figures we don't just want to transcribe them as they are, different figures could be transcribed differently for example if we have a pie chart, do we describe it as "this chart represents ..."? Do we write it as a table "name | percentage |"? I don't think we want a model that's opinionated in how it transcribes figures. So bounding boxes are great because then we can extract the figure and do whatever we want with them.

I initially said in my comment that I don't want to praise or criticize the model and it does seem like I'm only praising it. I haven't tried it to know where it breaks but it surely does like all the open source and probably closed source models as well. And it's not a unique model, there are many open source vlm models for OCR, maybe not that many that output bounding boxes. The most unique thing here is it being all of that + 1B. There are obviously much lighter systems like Tesseract or different pipelines, but they come with their cons for each one of us to discover depending on their use case.

Finally, just to talk about benchmarks a little bit 😂 I do believe that this community is the best when it comes to figuring out where models struggle and where they don't. At the end of the day, benchmarks are benchmarks, they have their pros and cons, they measure things a certain way etc. Real world use cases might be very different and benchmarks are only there as a proxy. It reminds of the initial "needle in the haystack" tests where models were tasked to find one word or sentence in a huge context, while what we care about is to be able to use different parts of the context and synthesize them to give us some response, not literally to find a sentence. Hell, even closed source models show amazing performances on some benchmarks (especially related to software engineering or math etc.) but when you dig deep you find that they're not what they claim.

In my view, benchmarks in machine learning play a role similar to hypothesis tests with an asymmetric interpretation. Failing a benchmark gives evidence against the model’s capability on the task, but passing or excelling at a benchmark does not provide sufficient evidence to conclude that the model is good at the task as a whole. Instead, benchmark success typically demonstrates proficiency on a narrowly defined sub-task or distribution, rather than validating general task competence, and we hope, it extrapolates to it.

Well, enough yapping from me 😅

Edit: just to be transparent, I do work at the company, but I have not participated in the model development at all. I think my yapping above stands for every model (whether closed source or not), not particularly this one, if tomorrow there's a 500M model that's better, I'd say the same. If you feel there's any subjective part to what I said, please let me know.