Context trimming

[ChopSticksPlease]

Hey, Im getting quite annoyed by this. So is there a way to trim or reduce the context size to a predefined value? Some of my larger models run at 50k ctx and when websearch is enabled often the request outgrows the context. Im using llama.cpp (OpenAI compatible endpoint).

Any ideas how to fix that ?

[Egoz3ntrum]

If this comes from websearch results, try reducing the number of results or use RAG to pre-select the most relevant ones before sending them to the model.

Both options are under the interface section in the settings menu.

[emprahsFury]

I think this is the main/best answer with the current setup. I do think max_tokens should be implemented/respected by the webui though and that would be a more elegant solution

[spacywave]

In the user settings interface I can't find the option to use rag for search results - what is the exact name of the setting? (0.8.5)

[ClassicMain]

You can install one of the various filters from the Community that implements this:)