llama.cpp 1.0: local AI finally gets a stable API and an fp8 KV cache
On November 13, llama.cpp and ggml shipped 1.0.0. After three and a half years of nightly builds, the engine that Ollama and LM Studio run on finally has an API contract.
What Hitting 1.0 Actually Means
1.0 brings no new feature: it brings a rule. From now on, any incompatible change to the public C API or the ABI forces a major version bump. The ggml-org maintainers, who joined Hugging Face in February 2026, say they will be stricter about making those breaks visible.
A Number With Rules: Major Version Means Breakage
For anyone who builds llama.cpp into another product, the change is about planning: you stop chasing nightly builds and you know that a 2.0.0 is the formal notice that something broke. Stable here means contract, not free of bugs and not guaranteed on every GPU.
Who Depends on This Without Knowing
Ollama, LM Studio and plenty of in-house clients wrap llama.cpp and its tensor library. If you use local AI tools such as Ollama, these changes reach you when your app updates its internal copy; meanwhile, what you gain is KV cache memory and better MoE handling.
Read also
What Goes Away: 35 Functions and One Option
The cleanup is the breaking part. 1.0 removes 35 functions from the public API already declared obsolete, plus one command-line option that no longer exists.
The llama_set_warmup Case
One of the removed functions is llama_set_warmup. If your integration called it, linking fails at build time. It is not a silent failure, and that is precisely why waiting for a 1.0 to delete made sense.
defrag_thold: The Option That Now Errors Out
The --defrag-thold flag is no longer accepted: passing it makes the binary return an error. If your startup script still carries it, drop it before upgrading.
The Memory Win: fp8 KV Cache
1.0 adds an fp8 KV cache that, in published tests, halves the memory of that cache: from two bytes to one per element. It relies on the FP8 E4M3 tensor type ggml added in 0.29.0, released with llama.cpp 0.9.0 on November 4.
The Math: How Big a KV Cache Really Is
Cache elements per token are 2 × layers × kv_heads × head_dim, because there is a key and a value. With Llama-3-8B style numbers (32 layers, 8 KV heads, dimension 128) that is 65,536 elements per token: 128 KiB per token in f16 and 64 KiB in fp8. At 32,768 tokens the difference is 4 GiB versus 2 GiB. This is arithmetic, not a measurement: repeat it with your own model's numbers.
What You Pay for Half the Memory
There is a precision loss, smaller than with q4 quantization, and fp8 requires backend and GPU support. The practical rule: if the cache fits in f16, there is no reason to touch it; if it does not fit, fp8 buys context in exchange for some quality.
MoE Experts Living in System RAM
The other memory feature is a MoE expert cache kept in system RAM, with a configurable CPU proportion. That is what lets you run mixture-of-experts models that do not fit in VRAM. On a small card it changes which model you can load, not how fast it answers.
llama-server: SSRF, DNS Rebinding and a Speech Endpoint
The server was hardened against SSRF and DNS rebinding with new flags, and it gains a speech endpoint. The idea is simple: a local server that makes outbound requests is also a door, and it should not be usable to reach internal addresses. The usual risk of such protection is over-blocking, so test your real flow after upgrading.
Upgrade Checklist
First, build and see what breaks: if you link against the library, headers and functions change; if you use the binary, downloading the new build is enough. Second, check your llama-server startup line for a leftover --defrag-thold or any retired flag. Third, if you run distributed inference, upgrade the RPC client and server together: the protocol changed in this generation and mixing versions breaks it. And before calling it good, measure: same model, same prompt and same context length, first with the cache in f16 and then in fp8.
Conclusion
1.0 does not improve your model, it improves your ability to plan: now you know when you will be broken and what you gain in memory. The project claims support for more than 150 architecture variants and optimizations for NVIDIA, AMD and Intel, but what changes your server today is fp8 and the expert cache. If you come from local AI, small models that fit a single GPU matter too: take a look at Meta's local model for agents. We keep testing on the blog.

