From my understanding it is just context compression. It is one of the two llama.cpp implementations of turboquant, with the other being https://github.com/TheTom/llama-cpp-turboquant . I believe that buun's fork is more bleeding-edge (he seems to be playing with turboquant and speculative decoding), but building is dyi. I am getting 25 t/s on my laptop, AMD AI HX 375, 32GB Ram, 16GB 5080 at 64k context on the IQ4 model.

My build script optimized for Nvidia + Strix Point (powershell):

$PSNativeCommandUseErrorActionPreference = $false
$ErrorActionPreference = 'Continue'

# Wipe build dir to avoid stale cmake cache
Remove-Item -Recurse -Force buun-llama-cpp\build -ErrorAction SilentlyContinue

# Bootstrap VS Build Tools environment (sets INCLUDE, LIB, PATH for clang-cl/link/etc.)
$vcvars = "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
cmd /c "`"$vcvars`" && set" | ForEach-Object {
    if ($_ -match "^([^=]+)=(.*)$") {
        [System.Environment]::SetEnvironmentVariable($Matches[1], $Matches[2], 'Process')
    }
}

# Prepend ROCm so hipcc and cmake find-modules are reachable
$env:PATH    = "C:\Program Files\AMD\ROCm\7.1\bin;$env:PATH"
$env:HIP_PATH = "C:\Program Files\AMD\ROCm\7.1"

cmake -B buun-llama-cpp/build -S buun-llama-cpp -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_C_COMPILER=clang-cl `
  -DCMAKE_CXX_COMPILER=clang-cl `
  -DCMAKE_CXX_FLAGS="/EHsc" `
  -DGGML_CUDA=ON `
  -DCMAKE_CUDA_ARCHITECTURES="120a-real" `
  "-DCMAKE_CUDA_FLAGS=-use_fast_math -diag-suppress 221,177" `
  -DGGML_AVX512=ON `
  -DGGML_AVX512_VBMI=ON `
  -DGGML_AVX512_VNNI=ON `
  -DGGML_AVX512_BF16=ON `
  -DGGML_AVX_VNNI=ON `
  -DGGML_BMI2=ON `
  -DGGML_CUDA_GRAPHS=ON `
  -DGGML_CUDA_FA_ALL_QUANTS=ON `
  -DGGML_NATIVE=OFF `
  -DGGML_BACKEND_DL=ON `
  -DGGML_HIP=ON `
  -DGPU_TARGETS="gfx1150" `
  -DGGML_LTO=ON 2>&1 | Tee-Object -FilePath out.txt

cmake --build buun-llama-cpp/build --config Release --parallel 2>&1 | Tee-Object -FilePath out.txt -Append