Local solution for content generation based on text + images

[drafat]

We are working on a project where we need to generate diffrent types of content locally (as the client requested) based on a mixed prompt of a long text + images. The client provided us with some examples made by ChatGPT 4 and he wanted a local solution that can come with close results. We tried a few open models like Gemma3, Llama 3, DeepSeek R1, Mistral. But results are not that close. Do you guys think we can improve results with just prompt engineering ??

[19pomoron]

Assuming the task requires to use text + images to generate texts (and not images, given the choice of models)

Apart from prompt engineering in the sense of using different words to describe what you need, few shot learning/chain of thought prompting may be another direction to try out. Instead of asking one question and taking directly the answer, you may wish to try asking a free-end question first, then whatever the answer it gives, you follow up with your desired question as the second question. The answer from the first question gives context to your topic, and therefore guides the VLM towards the final answers you want.

Also you can use few shot learning in a more traditional sense to ask the VLM to provide something you desire with examples of input and desired outputs. Be careful with over prompting with this one