llama.cpp v0.9.0 adds OCP FP8 E4M3 quantisation and a batched tensor copy API
The November release brings an FP8 file type, a more efficient unified KV cache for multi-sequence serving, four new architectures and ggml v0.29.0 with initial multi-node tensor parallelism.
llama.cpp tagged v0.9.0 on 4 November 2026, bundled with ggml v0.29.0. The headline addition is an OCP FP8 E4M3 tensor type in saturating mode, exposed through a new file type, LLAMA_FTYPE_MOSTLY_F8_E4M3, with conversion helpers. The release notes give no speed or memory figures for the new format. A second core change, flash_attn_ext_rows, makes the unified KV cache more efficient when serving several sequences at once and restores prefill performance that had been lost in that mode. A generic batched tensor copy API (ggml_backend_tensor_copy_n and its set and get variants) supersedes the older _2d calls and is implemented for Metal and CUDA.
Four new architectures are supported: Clover-35B-A3B-Preview, a diffusion model run in autoregressive mode, Limite 1B, Metis-1.6 in roughly 2.9 B and 10.2 B sizes with a hybrid Mamba-2 and GQA design, and Mistral Large 4. Audio gains Chatterbox TTS, Nemotron 3.5 Transcribe for offline transcription in 40 languages, and predefined Pocket TTS voices. Gemma 4 E2B and E4B get graph optimisations that the notes describe as large prefill gains, again without numbers.
ggml itself adds an initial multi-process communicator for multi-node tensor parallelism, new SVE kernels on CPU, fused MoE matrix multiplication in CUDA, faster MXFP4 matrix-vector kernels in Metal and wider quantised GEMM coverage in OpenCL and Hexagon. Vendored dependencies move to miniaudio 0.11.26 and cpp-httplib 0.64.0, and the project appears in GitHub’s Octoverse report for a third year.
Why it matters
FP8 E4M3 in llama.cpp closes a gap with vLLM and TensorRT, where FP8 weights have been routine, but the absence of benchmarks in the notes means you should measure perplexity and throughput on your own hardware before switching from Q8_0. The format only pays off on silicon with native FP8 arithmetic, so older GPUs and most Apple parts will see little speed benefit. If you maintain bindings, the batched copy API and the retirement of the _2d calls are the changes to track. Multi-node tensor parallelism is marked initial and should be treated as experimental.