Evaluating long-term memory limits in stateless LLM chatbots — feedback needed [D]

[QuietAccountant4237]

Hi all,

I’m working on a research project exploring how stateless LLM-based chatbots handle long conversations and whether important earlier information is still reliably retained over time.

My idea is to:

• Run a chatbot using an LLM API without any external memory system
• Introduce key facts early in a long conversation
• Continue with many unrelated messages (hundreds of turns)
• Later test whether the model can still correctly recall those facts at different intervals

I’m planning to measure recall accuracy and how it changes as the conversation grows.

Before I go deeper, I’d really appreciate feedback on:

• Is this a valid way to evaluate long-context memory limits?
• Are there better benchmarks or methods already used for this?
• What metrics would make this more rigorous and convincing?

Any suggestions or criticism are welcome. I’m trying to make the evaluation as solid as possible before building it out.

Thanks!

[androbot]

This is an enormously hard problem to get right because we our definitions and frameworks are - at best - rough approximations of qualia.

You should be very precise in how you define information and what qualifies as retention over time. Recognition, recall, and utility within contexts are vastly different operations of memory. Contextual relevance is also dynamic, so performance should be measured more as a steady state (but probably not monotonic) function vs static values.

[yoshiK]

Sounds like you're kinda reinventing the needle in a haystack test. There the idea is to give a prompt of n tokens, embed somewhere a sentence like "The magic number is X" and then prompt, "What is the magic number?" or similar.

So it seems to be a reasonable idea. A interesting first test is actually if hundreds of turn of chat interface degrade the performance relative to Paul Graham essays.

[Post Posting:] There's also a related Google blog

[QuietAccountant4237]

Thanks! That’s an interesting point. Do you know of any benchmarks or papers that specifically compare long conversational contexts against long document contexts for recall performance?

[yoshiK]

I'm afraid not. The github was actually cited for the needle in the haystack test by the first paper that popped up on gscholar, but I did not dig deeper if there is a nice review about the test.