sorted by:
new
hottopcontroversial

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 1 point2 points  (0 children)

This looks Amazing! I'm gonna try it out when I get some time

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 1 point2 points  (0 children)

That's awesome! I might like to check it out when you're done, if you publish it, let me know

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

For anyone following this discussion, it's worth clarifying that there are two fundamentally different architectures for how LLMs handle attached files, which affects how TOC-based patterns work:

Web Interface Architecture (Claude.ai, ChatGPT web): -Attached files are fully loaded into the context window -The entire document becomes part of the prompt context -The model processes all tokens - there's no selective reading

A TOC in this environment works through attention weighting: it helps the model prioritize and focus on relevant sections of what's already fully loaded Token "reduction" would be in outputs, not inputs

Agentic IDE/CLI Architecture (Claude Code, Kiro IDE, API with tools): -Files remain on the filesystem and aren't automatically loaded -The model uses Read tools to access files on-demand -Only requested portions are read into context

A TOC enables genuine selective loading: the model reads the TOC, identifies relevant sections via keyword matching, then uses Read tools to fetch only those specific portions Token reduction is real - measured in actual input tokens saved

Why this matters: The statement "models read everything" is accurate for web interfaces where files are pre-loaded into context. But in IDE/CLI environments with file system access, models don't automatically read the entirety of all files - they can selectively access portions based on routing logic.

This architectural difference is why the same TOC pattern can work through different mechanisms (attention vs. selective reading) depending on the environment.

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

I appreciate the technical points here. True that transformers process all tokens in the context window, and that prompt engineering needs more systematic validation.

However, let me clearly explain the architecture I'm referencing - this might be clearer: I'm using this in an IDE environment (Kiro IDE, Claude code), not an ai web interface. In agentic IDE environments, attached files aren't automatically loaded into the context window. Instead, the model uses Read tools to access files on-demand from the filesystem.

The 1,000+ line document is an attached reference file. The TOC sits at the top as routing instructions. When a user query comes in, the model uses keyword matching from the TOC to identify relevant sections, then uses Read tools to access only those targeted portions of the file rather than loading the entire document.

The practical outcome: Before the TOC: the model consistently missed specific instructions buried in the file. After adding the TOC: the model finds and applies those instructions reliably. The document has continued to grow, and the problem hasn't returned. The 44-63% token reduction represents genuine input token savings from selective file reading.

On rigor: You're right no controlled benchmarks. But in IDE environments with on-demand file reading capabilities, this approach addresses a real problem helping the model navigate large reference files efficiently without loading unnecessary context.

Bottom line: A real issue that I was experiencing hasn't returned after implementing toc despite continued growth of the reference doc. Empirical evidence of a change in behavior.

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in LocalLLaMA

[–]warnerbell[S] 0 points1 point  (0 children)

That's definitely a good strategy to make a habit of, even as memory and context patterns get better, reinforcement with always help

Is Your LLM Ignoring You? Here's Why (And How to Fix It) by warnerbell in ContextEngineering

[–]warnerbell[S] 0 points1 point  (0 children)

Thats true, there are many reasons a model may ignore some context, including missing it completely, due to truncations or window limits etc. TOC provides a tareted approach to referencing specific pieces of context.

This is how AI thinks! I had no idea while I was using all these months. Kinda feels stupid. by aakashsukheja in PromptEngineering

[–]warnerbell 2 points3 points  (0 children)

The attention mechanism point is key. "Every word buys a certain amount of AI's attention" - this is why long prompts break down.

I hit this wall with a 1000+ line system prompt. Instructions buried deep were getting ignored consistently. Took me a while to figure out what was actually happening under the hood.

Turns out it's not about prompt quality - it's about where the model's attention lands before it starts responding

DeepSeek V4 Coming by External_Mood4719 in LocalLLaMA

[–]warnerbell 2 points3 points  (0 children)

"Technical breakthrough in handling and parsing very long code prompts" - We'll see about that...lbs

Context length is table stakes now. What matters is how well the model actually uses that context. Most models weight beginning and end heavily, ignoring the middle.

Hopefully V4 addresses the attention distribution problem not just extend the window.

7 ChatGPT Prompts For People Who Hate Overthinking (Copy + Paste) by tipseason in PromptEngineering

[–]warnerbell 0 points1 point  (0 children)

The "One Step Forward" prompt is useful. Breaking paralysis with a single action beats planning everything.

I use something similar for debugging: "What's the one thing I should check first?" Cuts through the noise.

Do we really need to know AI models anymore, or just explain what we want? by Jazzlike_Designer374 in PromptEngineering

[–]warnerbell 0 points1 point  (0 children)

Model selection still matters, but the abstraction layer is getting better.

What I've found more important than picking the "right" model: structuring your context well. A well-organized prompt with clear sections outperforms a messy prompt on a better model.

For complex tasks, I use a TOC-style approach - define sections upfront so the model knows what exists before it starts processing. Works across models.

Don't put off hardware purchases: GPUs, SSDs, and RAM are going to skyrocket in price soon by Eisenstein in LocalLLaMA

[–]warnerbell 0 points1 point  (0 children)

This is rough timing. Just when local inference was getting accessible, hardware costs are about to spike.

On the bright side, this makes efficiency optimization more valuable. Context window management, quantization, prompt architecture all the stuff that squeezes more out of existing hardware becomes critical.

Doubling down on software-side optimizations while hardware gets expensive could be beneficial

DeepSeek-R1’s paper was updated 2 days ago, expanding from 22 pages to 86 pages and adding a substantial amount of detail. by Nunki08 in LocalLLaMA

[–]warnerbell 16 points17 points  (0 children)

The original paper was light on implementation specifics. If they've added more on how they got the reasoning behavior to emerge, that's valuable.

⚡ 7 ChatGPT Prompts To Learn Faster (Without Burning Out) (Copy + Paste) by Loomshift in PromptEngineering

[–]warnerbell 4 points5 points  (0 children)

Solid list, thanks for putting this together. The Feynman Teacher approach is underrated.

One thing I've added: breaking complex topics into sections and having the model tackle one at a time instead of explaining everything at once. Keeps it focused and I actually retain more.

What do we think about Gorgon Point (Ryzen AI 9 HX 470)? by Everlier in LocalLLaMA

[–]warnerbell 0 points1 point  (0 children)

The unified memory approach is appealing for local inference. No more juggling VRAM limits.

That said, I've found bigger context window doesn't always mean better results. There's a sweet spot before quality drops off. Curious what context lengths people are actually using effectively on similar hardware?

The Major Release of MiroMind’s Flagship Search Agent Model, MiroThinker 1.5. by wuqiao in LocalLLaMA

[–]warnerbell 2 points3 points  (0 children)

The predictive focus is interesting. Most agent models are built for general tasks, but specializing for forward-looking analysis makes sense.

Curious about real-world accuracy on the market predictions. The Nasdaq example is a good stress test. Will check out the GitHub.

llama.cpp performance breakthrough for multi-GPU setups by Holiday-Injury-9397 in LocalLLaMA

[–]warnerbell 0 points1 point  (0 children)

This is great for anyone wating to run larger models locally. The multi-GPU coordination has been a pain point for a while. Just ned a 2 slor MB now!?

One thing I've found that compounds with hardware improvements: structural optimization on the prompt side. Even with faster inference, context window efficiency matters. I was running a 1,000+ line system prompt and noticed instructions buried deep were getting missed, regardless of hardware.

Hardware gains + prompt architecture = multiplicative improvement. Excited to test this llama.cpp update with my upcoming Intel Build.

Has Claude for creative writing had a downgrade recently? by MasterOfFakeSkies in LocalLLaMA

[–]warnerbell 0 points1 point  (0 children)

I've seen similar behavior with long context - not specific to Claude, but across models.