Your Local LLM Is Slow Because of Five Config Flags

Your model fits in memory. You send a prompt and generation falls to a few tokens per second, or the chat dies halfway through. Quantization and layer offload were already set. The runtime still reserved a second buffer at startup: the KV cache, sized for the full context window you asked for, not for the 200-token prompt you typed. On 16GB hardware that buffer is often as large as the weights.

Where the memory goes

llama.cpp puts RAM in two places: the weight tensors and the KV cache.

The weights are static. Llama 3.1 8B at Q4_K_M is about 4.9GB, and that figure does not grow during inference.

The cache grows with sequence length on full-attention layers. Each of those layers stores a key vector and a value vector for every token already in the context, so the next token can attend to those stored vectors instead of recomputing the prompt from scratch. Multiply 2 (keys and values) by layers, KV heads, head dimension, sequence length, and bytes per element:

KV cache = 2 × layers × kv_heads × head_dim × seq_len × bytes_per_element

For Llama 3.1 8B (32 layers, 8 KV heads via grouped-query attention, head dimension 128) at 32K context in FP16, that product is about 4.3GB, close to the 4.9GB weight file.

The 32K and 64K rows are where a 16GB laptop starts swapping. The GGUF filename shows the weight quantization; the startup log prints the cache buffer size.

The table is for that Llama 3.1 8B layout. Hybrid models such as Gemma 4 keep sliding-window layers near a fixed window (often 1024 tokens), so those layers do not grow with --ctx-size the way the table does.

Context size is a memory reservation

--ctx-size in llama.cpp (num_ctx in Ollama) is the token window for a session. The runtime allocates KV cache for that entire window at startup, even if the first prompt is 200 tokens.

--ctx-size 65536 therefore reserves cache for 65,536 tokens immediately. Running an 8B model on a 16GB MacBook, that reservation alone can force swap.

If you omit --ctx-size, llama.cpp default 0 loads the model's advertised window. On a 128K or 256K card that reservation can exceed a 16GB machine before the first token.

Ollama no longer defaults to 2,048 tokens. As of v0.32.5, an unset OLLAMA_CONTEXT_LENGTH means 4,096 below about 23GiB of VRAM, 32,768 from 23GiB, and 262,144 from 47GiB. A 16GB laptop still truncates early. A 24GB Mac defaults to 32K and can over-reserve.

A working budget:

Available for KV cache = total RAM - model size - OS overhead (about 3-4GB)

With a 4.9GB model and Q8_0 cache on 16GB unified memory, about 8-9GB remains for the cache, which holds 32K context. The same RAM with an FP16 cache holds about 16K.

KV cache quantization

By default the cache is FP16 (2 bytes per element). Q8_0 uses 1 byte and Q4_0 uses 0.5 bytes.

Q8_0 frees about 2GB. On the same machine that can be a 14B model instead of a 7B, or a 32K window instead of 16K. Q8_0 is not equal on every architecture: a 2026 token-level comparison put Qwen 3.6 under KL 0.04 at Q8_0 cache, while Gemma 4 26B A4B landed at 0.377.

llama-server --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on

OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Cache quantization requires flash attention. llama.cpp and Ollama now enable flash attention where the backend supports it, so setting the cache type is usually enough. If flash attention did not run, the cache type is ignored.

Q4_0 and missing flash-attn kernels

Q8_0 is a safe default on most dense GQA models; Q4_0 often is not.

A 2026 llama.cpp re-measure on NVIDIA DGX Spark found prompt throughput unchanged by cache quant. Q4_0 decode was about 37% slower than FP16 at around 110K tokens, from dequantizing during attention. Older "92% slower at 64K" figures mixed in failed completions.

The failure that looks like "Q8_0 made my GPU idle" is usually a missing CUDA flash-attn kernel. Attention falls back to the CPU. Gemma 3 hit this in 2025 (head dimension 256). Gemma 4, released 2 April 2026, still hits it: sliding-window layers at head dimension 256 and global layers at 512. llama.cpp still reports FA crashes and silent CPU fallback on quantized KV unless the CUDA build includes GGML_CUDA_FA_ALL_QUANTS. One Gemma 4 12B QAT report went from 96 to 2,361 prompt tokens/s after those kernels were compiled.

If tokens per second collapse and GPU utilization falls, check the log that flash attention ran. Then set the cache back to FP16.

Flash attention avoids the n-squared score matrix

The KV cache holds keys and values. Attention still has to compare the new token against every previous token.

Standard attention builds a score matrix of shape (seq_len × seq_len) per head. At 32K in FP16 that matrix is about 2GB per head per layer if it is written out in full.

Flash attention does not materialize that matrix. It computes attention in tiles that fit in on-chip SRAM, so the extra memory scales as O(n) rather than O(n²), and the kernel moves less data between main memory and SRAM.

On Apple Silicon the kernel is Metal compute shaders. Unified memory means those tiles do not cross a PCIe link between CPU and GPU.

llama.cpp --flash-attn takes on, off, or auto. auto is the default. Ollama enables flash attention where the backend and GPU support it; OLLAMA_FLASH_ATTENTION=1 forces it on and 0 forces it off. Output quality does not change. Read the startup log. Quantized KV does nothing if flash attention stayed off.

Prefill batch size

Inference has two phases. Prefill reads the prompt and can process many input tokens in one forward pass; the limit is compute. Decode emits one new token at a time; the limit is memory bandwidth.

-b / --batch-size in llama.cpp is the logical prefill batch (default 2048). -ub / --ubatch-size is the physical chunk (default 512). A 16,000-token prompt at ubatch 512 is 32 GPU passes; -ub 2048 is 8, if the extra activations fit. That often cuts prompt-eval time. It does not speed decode.

Ollama exposes batch as the Modelfile or API option num_batch, not as a server environment variable in current releases.

If a coding agent resends a growing context on every tool call, wall time is mostly prefill. Raising the physical batch shortens the wait between calls.

Parallel slots share your context budget

On llama-server, --parallel is the number of concurrent requests. --ctx-size is the total KV budget for all slots, not a per-request window. The current default for --parallel is auto (-1). Set --parallel 1 on a laptop anyway.

Two slots at 64K each need --ctx-size 131072 and the RAM to match.

In benchmarks on a laptop GPU, two slots turned a 2-minute request into 3.5 minutes each, because both jobs share the same bandwidth and compute. The flag raises throughput under load rather than making a single chat faster.

A starting command for 16GB-class machines

After you confirm Q8_0 cache is stable on your model:

llama-server \
  -m model-Q4_K_M.gguf \
  -ngl 99 \
  -c 16384 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --parallel 1

For Ollama:

OLLAMA_KV_CACHE_TYPE=q8_0

If tokens per second drop after Q8_0 and the GPU sits idle, attention fell back to the CPU. Set the cache back to FP16. Gemma 4 is the architecture that still hits this on many CUDA builds.

Then raise num_ctx against the KV buffer size printed at startup, not against the model's advertised context length.