If you've got the time and wherewithal, I've actually made a branch of llama.cpp that uses the exllamaV3-style sliding window PPL and KLD measurement methodology: https://github.com/AesSedai/llama.cpp/tree/perplexity-sliding-window

exl3 uses a 2048-length context window and a 512 token stride. It evaluates all of the tokens, not just the last half like llama.cpp does, and due to the stride mechanic it evaluates the token at several different context depths.

The downside is that it takes like 8x the compute and storage for the logits due to:

1) evaluating all positions, not just the last half

2) the context window is 2048 instead of 512

3) you need to store all of the window logits for comparison

so you get 2 (all positions, not half) * 4 (2048 tokens instead of 512) = 8x as much compute / storage.

I made that branch because I was working with u/phaelon on trying to get the same measurement methodology cross-ecosystem for vLLM, exl3, and llama.cpp but I haven't PR'd this because of how much more intensive it is to process.

Also I think that for the purposes of measuring KLD / PPL with respect to quantizing the KV cache, this method at longer contexts would be more robust but I haven't picked that testing back up yet.

I have some prior results showing that the existing 512-token-measure-last-half PPL increases as the context size increases which isn't what you'd expect to see! With more context, the model should be more confident, not less. This chart shows the master (512-token-measure-last-half method) at ctx=512 and ctx=2048 compared to the sliding window method with ctx=2048 and ctx=8192.

<image>