llama.cpp 0.7.0 moves the server to port 9931 and adds a GPU cache for MoE experts kept in host memory

llama.cpp 0.7.0 moves the port llama-server listens on by default from 8080 to 9931. It also adds a GPU cache for mixture-of-experts (MoE) models whose experts are kept in host memory, which the page puts at up to about 2.2x faster decoding when experts are offloaded , along with new speculative decoding methods and a long list of new models. The release page dates it 15 October, 17:13. If anything on your machine talks to llama-server atlocalhost:8080, check it before you update; if you run a large MoE model on a small card, this is the release to try.

What changed

The server now listens on port 9931 by default instead of 8080. The Docker images are the exception: inside the container, they keep 8080.

A mixture-of-experts model is built from many blocks called experts, and each word passes through only a few of them. When the whole model does not fit in the card's own memory (VRAM), the experts can stay in the computer's main memory instead. 0.7.0 adds a GPU cache for those experts, set with --moe-cache-mib: the experts run on the GPU from an LRU cache — one that keeps what was used most recently and drops the rest — and the page puts the gain at up to 2.2x faster decoding. The cache is shared across several GPUs.

Embedding models — the ones that turn a text into a vector of numbers for search — now skip the logits buffer and the LM head when pooling is enabled. Logits are the scores a model gives every possible next word, which an embedding does not need; pooling merges the vectors of a text's words into one. The page puts the effect at up to 10x less VRAM.

Speculative decoding, where a small model proposes the next words and the main model checks them in one pass, gets two new methods: LiliCorr, which picks candidates with a small draft transformer on top of DFlash , and MTP for GLM5-Next, with an optimized sparse attention path. A new option,--spec-draft-n-ubatch, sizes the draft model's batch separately to save VRAM.

New models arrive: K2 Horizon, in dense and MoVA variants ; Prism Bonsai 2 27B, a ternary model, whose weights take one of three values ; EmbeddingGemma-2, which embeds text, images and audio ; the decision models LiquidAI d1-3B and d1-omni-600M, vLLM decision 2, vLLM d3 and pplx-decider , which the server's /v1/systemone API now serves ; MiniCPM-V 4.7, a multimodal model ; Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B ; and the maion-coder architecture.

Inside the server, the chat layer is rebuilt around common_chat_session, and formatted chat output is now an array of parts instead of a single string. The built-in web UI gains a Models Manager: one table of every served model, with its quantizations, search, filters, sorting and favorites. ggml moves to v0.27.0.

What it was before

The page lists all of this as the changes since v0.6.0. Until this release, llama-server listened on 8080 by default , and its formatted chat output was a single string.

What it means for you

Start with the port. If a script, a browser bookmark, an editor plugin or another app reaches llama-server at localhost:8080, and you start the server without choosing a port, that address finds nothing once you update: the server is now at 9931. Point your clients at 9931, or set the port back to 8080 when you start the server. If you run it from one of the Docker images, the port inside the container stays 8080 , so a setup that reaches the container's port is not affected by this change.

If you run a large MoE model on a modest card, with part of it in main memory, try --moe-cache-mib. The 2.2x is the page's upper bound, for the case where experts are offloaded ; if your model fits entirely in VRAM, the cache has nothing to do. The page does not say how large to make it, so try a few sizes and measure words per second before and after. How much memory a model needs, and why the speed of that memory matters, is explained in our guideWhich Machine Runs a Model Locally.

If you serve an embedding model for search, update and look at how much VRAM it takes now: up to 10x less, when pooling is on. If you use a draft model and VRAM is tight,--spec-draft-n-ubatchis the option to look at.

If your own code reads the server's formatted chat output, note the change from one string to an array of parts. The page lists it among the API changes but does not say which HTTP responses, if any, it reaches; test your code against 0.7.0 before you update a machine it depends on.

The source

The release page on GitHub: llama.cpp v0.7.0.