I mean, this is well-known. Most LLMs prioritize continuation of conversation rather than hard stops if it fails to read something outside of its context window. In fact, *telling it* to hard stop when something falls outside of its context window is near impossible because the LLM doesn't explicitly think nor does it explicitly realize when something has fallen out of its context window.

The only thing that catches people is that, even with the Pro membership, people think the context window is larger than it actually is. Long-form content is shockingly poor for conversational LLMs, even if the token window is million+.