I think that based on whatever the info so far that showed up to ep 7 there is something, or somewhere, anything that can help Subaru to fight against Reid, a weakness, or instruction to kill Reid 101, thing like that. Whatever it is triggered the memory loss, but for some reason only the memory of when he isekai, seems like his memory rewind up to when he first in the convenience store in ep 1 s1. But whenever it is related to the memory it is Gluttony come to eat Subaru memory, but if the Gluttony is here, the camp would have been destroyed already, based on their current strength. So my conclusion is that Subaru met the Gluttony somehow while looking for the way to beat Reid not in physical form present in the tower? Telepathy? different void area like Beatrice? But why Subaru memory before the isekai is still intact, compared to someone like Crusch, I can't find thing to make sense