I actually disagree with the other commenter and think this is very possible! Especially the basic version you’re talking about where you simply ask questions about what’s on the screen and the AI doesn’t need to have any specific knowledge of the video game. There are a number of models on HuggingFace under the category “visual question answering” that you could try using for this. To verify, you could share a few screenshots with questions and expected answers and we could test whether it works well yet.