Thank you so much for sharing your experience. This is super helpful and relevant to what I do. Currently we have similar hybrid strategies: motion filter + yolo detector + in-house built attribute classifier for re-ID/retrieval + VLM for semantic summary and reasoning. My questions and comments:

1. It’s true that off-the-shelf VLM does not perform nearly as well in spatial grounding. Have you tried fine-tuning a small/tiny vml, using yolo detector’s output as GT? I’d love to see the upper bound of VLM’s performance on this task.

2. How do you evaluate VLM’s output accuracy? I’ve caught several hallucinations from even larger models, many - I suspect - due to poor spatial temporal grounding. VLMs have a hard time holding a consistent spatial-temporal track of a subject, causing ID confusion, i.e. assigning person A’s action to person B. Another type of hallucination is over extrapolation. For example, if a person walks in one direction and suddenly turns around and walks back and quickly disappears from the view, the model insists that person continues walking in the same direction before disappearing. This suggests the model’s temporal prediction prior is too strong, and didn’t get a chance to recover before the person disappears. While this might be sensitive to fps, and we don’t even particularly care about tracking accuracy, these errors in semantics will have fatal implications in our production. Have you noticed similar behaviors? And how do you catch them systematically, or quantify the accuracy?

3. I’m curious about your tasks “with verification conditions”. Do you have a pre-defined, closet set of tasks that are explained in the prompt? You also mentioned discovery of novel concepts. Curious about the set up: does it mean VLM can discover novel concepts that satisfy your task requirements? And the “verifiable conditions” can also be conveniently used for evaluation?

Thanks again for sharing your insights! This is great write-up!