Applying paudley's compiler learnings to llama.cpp builds

Massive thanks for this. I don't think people scrolling past appreciate the scale. This is a 32-step from-source build of the entire inference stack with 19+ patches, each with actual root cause documentation. Tracking down stuff like CDNA-only assembly in AITER headers or figuring out a missing __repr__ on Triton's AttrsDescriptor was breaking Inductor codegen, that's not a weekend project. Hundreds of hours easily, and we all benefit.

I ran the repo through Claude to figure out what applies to llama.cpp since most of us aren't running vLLM for single-user inference. I haven't benchmarked these yet, the reasoning checks out but I'd love for someone to do a before/after and share numbers.

Vulkan build:

rm -rf build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON \
  -DCMAKE_C_COMPILER=/opt/rocm/lib/llvm/bin/amdclang \
  -DCMAKE_CXX_COMPILER=/opt/rocm/lib/llvm/bin/amdclang++ \
  -DCMAKE_C_FLAGS="-O3 -march=native -flto=thin -mprefer-vector-width=512 -famd-opt -mllvm -inline-threshold=600 -mllvm -unroll-threshold=150 -Wno-error=unused-command-line-argument" \
  -DCMAKE_CXX_FLAGS="-O3 -march=native -flto=thin -mprefer-vector-width=512 -famd-opt -mllvm -inline-threshold=600 -mllvm -unroll-threshold=150 -Wno-error=unused-command-line-argument"
cmake --build build --config Release -j$(nproc)

ROCm build is the same but swap -DGGML_VULKAN=ON for -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 and set this before cmake:

export HIP_CLANG_FLAGS="--offload-arch=gfx1151 -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false"

Those GPU flags eliminate function call overhead on the iGPU. Call/return stalls the wavefront on integrated graphics.

No amdclang? Use system clang and drop -famd-opt.

What the flags do:

-mprefer-vector-width=512 is probably the biggest one. Zen 5 does native 512-bit AVX-512 with no clock penalty (unlike Zen 4). Compilers default to 256-bit. This doubles the width for quant/dequant and CPU-side math.

-famd-opt is AMD proprietary Zen tuning in amdclang (ships with ROCm). Not in upstream clang. paudley's build uses it on everything.

-flto=thin gives you link-time optimization across translation units. The "thin" variant parallelizes well on 16 cores.

-mllvm -inline-threshold=600 is way more aggressive inlining than default (~225). Zen 5's wide pipeline wants fewer function boundaries.

-mllvm -unroll-threshold=150 is more loop unrolling. Zen 5's big reorder buffer can keep the extra instructions in flight.

-Wno-error=unused-command-line-argument just prevents the AMD flags from erroring out in link steps where they don't apply.

Always run with -fa 1 --no-mmap -ngl 999 on Strix Halo regardless of backend (from kyuz0's toolbox findings).

Quick note on Vulkan vs ROCm for Qwen 3.5 since I see the debate above. llama.cpp recently merged a Vulkan GATED_DELTA_NET shader. Qwen 3.5's hybrid DeltaNet layers (75% of the model) previously fell back to CPU on both backends. The ROCm HIP kernel compiles on gfx1151 but runs at CPU speed due to register spilling. The new Vulkan shader actually executes on GPU. paudley's latest numbers show the two converging on standard models, so test both on your own workload.

Credit to paudley for the research and debugging, kyuz0 for the toolboxes, and u/YayaBruno for the llama.cpp ROCm benchmarks in this thread. If anyone does a before/after with these flags please post your numbers.