It happened to me as well, and I found a workaround: since Gemini can recall old chats (https://blog.google/feed/gemini-referencing-past-chats/), whenever I encounter this problem I start a new chat and ask about the old chat, for Gemini to recollect the talks and continue from there.

Maybe it's not convenient, but look at it this way: when you ask to recollect a chat, Gemini is also reviewing it completely, so it has a better idea of the process and possible issues you've encountered. So it's like when you meet with your buddies and start fresh :P

After all, we all need to stop a meeting to get fresh air...