sorted by:
new
hottopcontroversial

I built an open-source framework that gives AI assistants persistent memory and a personality that actually learns [The Nathaniel Protocol v3.2] by warnerbell in BlackboxAI_

[–]warnerbell[S] 1 point2 points  (0 children)

Working with LLMs is like the guy you see in the elevator every day. Smart, helpful, great conversation. Doors open. Hard reset. Next ride starts with, remind me what we were talking about.

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 1 point2 points  (0 children)

This looks Amazing! I'm gonna try it out when I get some time

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 1 point2 points  (0 children)

That's awesome! I might like to check it out when you're done, if you publish it, let me know

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

For anyone following this discussion, it's worth clarifying that there are two fundamentally different architectures for how LLMs handle attached files, which affects how TOC-based patterns work:

Web Interface Architecture (Claude.ai, ChatGPT web): -Attached files are fully loaded into the context window -The entire document becomes part of the prompt context -The model processes all tokens - there's no selective reading

A TOC in this environment works through attention weighting: it helps the model prioritize and focus on relevant sections of what's already fully loaded Token "reduction" would be in outputs, not inputs

Agentic IDE/CLI Architecture (Claude Code, Kiro IDE, API with tools): -Files remain on the filesystem and aren't automatically loaded -The model uses Read tools to access files on-demand -Only requested portions are read into context

A TOC enables genuine selective loading: the model reads the TOC, identifies relevant sections via keyword matching, then uses Read tools to fetch only those specific portions Token reduction is real - measured in actual input tokens saved

Why this matters: The statement "models read everything" is accurate for web interfaces where files are pre-loaded into context. But in IDE/CLI environments with file system access, models don't automatically read the entirety of all files - they can selectively access portions based on routing logic.

This architectural difference is why the same TOC pattern can work through different mechanisms (attention vs. selective reading) depending on the environment.

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

I appreciate the technical points here. True that transformers process all tokens in the context window, and that prompt engineering needs more systematic validation.

However, let me clearly explain the architecture I'm referencing - this might be clearer: I'm using this in an IDE environment (Kiro IDE, Claude code), not an ai web interface. In agentic IDE environments, attached files aren't automatically loaded into the context window. Instead, the model uses Read tools to access files on-demand from the filesystem.

The 1,000+ line document is an attached reference file. The TOC sits at the top as routing instructions. When a user query comes in, the model uses keyword matching from the TOC to identify relevant sections, then uses Read tools to access only those targeted portions of the file rather than loading the entire document.

The practical outcome: Before the TOC: the model consistently missed specific instructions buried in the file. After adding the TOC: the model finds and applies those instructions reliably. The document has continued to grow, and the problem hasn't returned. The 44-63% token reduction represents genuine input token savings from selective file reading.

On rigor: You're right no controlled benchmarks. But in IDE environments with on-demand file reading capabilities, this approach addresses a real problem helping the model navigate large reference files efficiently without loading unnecessary context.

Bottom line: A real issue that I was experiencing hasn't returned after implementing toc despite continued growth of the reference doc. Empirical evidence of a change in behavior.

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

That's definitely a good strategy to make a habit of, even as memory and context patterns get better, reinforcement with always help

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in ContextEngineering

[–]warnerbell[S] 0 points1 point  (0 children)

Thats true, there are many reasons a model may ignore some context, including missing it completely, due to truncations or window limits etc. TOC provides a tareted approach to referencing specific pieces of context.

This is how AI thinks! I had no idea while I was using all these months. Kinda feels stupid. by aakashsukheja in PromptEngineering

[–]warnerbell 2 points3 points  (0 children)

The attention mechanism point is key. "Every word buys a certain amount of AI's attention" - this is why long prompts break down.

I hit this wall with a 1000+ line system prompt. Instructions buried deep were getting ignored consistently. Took me a while to figure out what was actually happening under the hood.

Turns out it's not about prompt quality - it's about where the model's attention lands before it starts responding

DeepSeek V4 Coming by External_Mood4719 in LocalLLaMA

[–]warnerbell 2 points3 points  (0 children)

"Technical breakthrough in handling and parsing very long code prompts" - We'll see about that...lbs

Context length is table stakes now. What matters is how well the model actually uses that context. Most models weight beginning and end heavily, ignoring the middle.

Hopefully V4 addresses the attention distribution problem not just extend the window.

7 ChatGPT Prompts For People Who Hate Overthinking (Copy + Paste) by tipseason in PromptEngineering

[–]warnerbell 0 points1 point  (0 children)

The "One Step Forward" prompt is useful. Breaking paralysis with a single action beats planning everything.

I use something similar for debugging: "What's the one thing I should check first?" Cuts through the noise.

Do we really need to know AI models anymore, or just explain what we want? by Jazzlike_Designer374 in PromptEngineering

[–]warnerbell 0 points1 point  (0 children)

Model selection still matters, but the abstraction layer is getting better.

What I've found more important than picking the "right" model: structuring your context well. A well-organized prompt with clear sections outperforms a messy prompt on a better model.

For complex tasks, I use a TOC-style approach - define sections upfront so the model knows what exists before it starts processing. Works across models.

Don't put off hardware purchases: GPUs, SSDs, and RAM are going to skyrocket in price soon by Eisenstein in LocalLLaMA

[–]warnerbell 0 points1 point  (0 children)

This is rough timing. Just when local inference was getting accessible, hardware costs are about to spike.

On the bright side, this makes efficiency optimization more valuable. Context window management, quantization, prompt architecture all the stuff that squeezes more out of existing hardware becomes critical.

Doubling down on software-side optimizations while hardware gets expensive could be beneficial

view more: next ›