Run 30B MoE Models on Older GPUs: llama.cpp's Expert Cache Unlocked
For the last couple of weeks, developers have been quietly shipping speedups to llama.cpp that let you run huge Mixture-of-Experts models on GPUs that shouldn't physically be able to hold them. The magic trick: a GPU-resident LRU cache that keeps only the experts you actually use in VRAM, streaming the rest in from system RAM on demand.
The wins are real enough to make older hardware suddenly viable again. On an RTX 4090, a Qwen MoE model jumps from 25 tokens/second to 40 tokens/second with the cache enabled—a 1.6x speedup. On an RTX 5090, the same model goes from 31 to 68 tokens/second. Smaller cards see bigger relief: an RTX 2060 12GB that couldn't run the model at all before now handles it in the low double digits.
How it works
llama.cpp is gaining a GPU-resident LRU cache for Mixture-of-Experts (MoE) weights that normally live in host RAM, so only the experts a decode step actually needs stay on the device. In a Mixture-of-Experts model, every token only activates a small fraction of the total experts—say, 2 out of 128. The old approach was crude: split the layers between CPU and GPU and hope for the best. The cache is smarter. With MoE models only a few experts activate per token, so you can park the bulk of the (cold) expert weights in system RAM and stream them via PCIe, while keeping the always-hot layers and KV cache on the GPU.
The implementation uses a simple slot-based design: you allocate a chunk of VRAM (say, 6–11 GB on a 16 GB card), and the cache maintains an LRU table of which experts are live. On the same rig and settings, a downstream build of leloch's CPU-on-miss cache uses a design: hits run on the GPU, misses run on the CPU. Benchmarks from the pull request show expert routing has strong temporal locality (LRU-64 ~67% hit rate over a mixed workload) even though the long-run expert distribution is near-uniform, so a per-layer LRU cache of expert slices in VRAM removes most of the per-token host-RAM streaming that bounds decode.
What people are actually running
Developers are testing this on Qwen MoE models (Qwen3.8-Flash-Next, Qwen3.6-35B-A3B) and Gemma 4 26B, which are large enough that they spill off consumer GPUs but small enough to manage with the right quantization. A host-offloaded expert-weight LRU cache kept hot experts in VRAM and lifted Qwen MoE decode from about 8 to nearly 19 tokens per second on two RX 6950 XTs. On two Radeon RX 6950 XT 16 GB cards linked over OCuLink from a Ryzen 7 7840U handheld (64 GB system RAM), running a Qwen3.8-Flash-Next UD-Q2_K_XL build with experts forced to CPU and layers on GPU, warm-cache decode rose from 8.2 tokens/s with the cache disabled to 12-13 tok/s at 128 slots (~85% hit rate).
The real appeal is cost: you can now prototype and iterate on large models without renting cloud GPUs. A developer with an older RTX card and fast system RAM can run code generation, chat, or embedding inference at speeds that felt impossible six months ago.
What's actually good
The speedups are consistent and measured. Measured 9.7 -> 37-50 t/s decode on a 24GB 7900 XTX (3.5-4x), depending on workload. The cache works on both NVIDIA (CUDA) and AMD (Vulkan) backends, which is rare—most VRAM tricks are CUDA-only. The llama.cpp MoE expert cache (GPU-resident LRU cache for host-offloaded MoE expert weights) is labelled CUDA-only by every downstream fork. It is not. The upstream implementation is backend-agnostic and runs on Vulkan/RADV, where it produces a measured 1.8x decode speedup on a Radeon RX 9070 XT (gfx1201).
Since the whole thing is in llama.cpp and the code is public, you can tune the cache size to your hardware without paying anyone. It integrates cleanly with existing quantization (Q4_K_M, Q6, etc.) and works with long context (128K+ tokens).
What's not there yet
The cache is single-GPU only in the mainline implementation. Multi-GPU systems have some quirks—the cache currently pins to CUDA device 0, leaving other cards partially idle. There are also edge cases with certain batch sizes or micro-batch settings where hit rates drop sharply. And if your model's experts don't cluster in time (hot experts stay hot), you won't see the speedup; it depends on the model's routing patterns.
How to try it
llama.cpp exposes this directly via --n-cpu-moe N (keep the MoE weights of the first N layers on the CPU) and --n-gpu-layers 999 (everything else on the GPU). With the cache enabled, you add a --moe-cache-mib flag to allocate VRAM for the expert cache. Start with a small allocation (2–6 GB) and increase it as your hardware allows. The sweet spot for most cards is 30–50% of VRAM.
The code landed in PRs #29887 and #27861. Grab a recent build of llama.cpp, a MoE model in GGUF format, and measure your own speedup.