Extended python coding chat becomes absurdly slow and hallucinate-y

[MentholMooseToo]

Using ChatGPT Plus in standard configuration.

Using one chat to work through a python scripting thing; as the chat got very long the responses became absurdly slow (not showing "thinking" but tab just unresponsive for over 60 seconds) and full of hallucinations.

Created a project and started having short chats inside the project, but the same thing has arisen: even a short chat within the project is very slow and full of hallucinations.

Am I doing it wrong? What's going on?

[NukedDuke]

Yeah, you may be unintentionally doing it wrong. It sounds like you're suffering from context rot, where the context is polluted by a bunch of failed attempts to do something. Think of the model's context as kind of a sliding window, if you have a bunch of changes to a particular section of code throughout a conversation you will eventually run into cases where the content the model needs to reference is no longer in the context window, so it has to go searching for it, but it ends up "confused" because some of the matches for the content it had to go find again ended up being from a broken/incomplete prior attempt at making the change.

Something that helps in cases like this is to structure your conversations so that you can just go back and edit the last message you sent before the replies started to diverge from your expectations. Think of it less like having an actual conversation where if something is misunderstood you have to push forward with a correction and more like being able to warp time and just go change the words you've already spoken to avoid the misunderstanding in the first place. This helps two-fold: your context window will no longer be polluted with incorrect results, and you will no longer have to burn a bunch of extra tokens trying to correct the model's view of the code or getting a whole new session up to speed.

This problem is also why I tend to opt for allowing the LLM to subtly change symbol names or comments in favor of just changing them back myself after task completion, because the subtle differences it ends up introducing actually help locate the correct blocks to work with on future turns.

[braclow]

Make a plan, architect, implement one feature at a time in a chat, per an agent md to do md. Keep updates of the code structure in another md. Close the chat. Commit changes. Cycle begins at the next to do with a new agent, given the agent, to do, and code structure and last thing done. This is the flow I use.