boocode/openspec/changes/llama-cache-and-spec/proposal.md

llama-cache-and-spec — KV cache quantization + ngram speculative decoding

Why

BooCode's llama-sidecar runs llama-server with bare-minimum base args: -ngl 999 -c 32768 --flash-attn on --no-mmap. Two high-impact llama.cpp features are available but not enabled:

1. KV cache quantization (--cache-type-k q4_0) — stores the KV cache in 4-bit instead of f32. ~4× VRAM reduction for the cache, which dominates memory usage at 32K context. No quality impact for most models.
2. Ngram speculative decoding (--spec-type ngram-mod) — uses a lightweight rolling-hash ngram model (~16MB) to predict tokens ahead of the main model. The main model verifies them in batch. 2-3× tok/s speedup on repetitive/code tasks with no accuracy loss and no separate draft model to load.

Both are disabled because they're in the shadowing lists of both validators (llama-args-validator.ts + sidecar validator.go), which auto-strip them from agent llama_extra_args. The fix is to either: (A) remove them from the shadow lists, or (B) add them directly to the sidecar's BASE_ARGS (which skips validation entirely).

What Changes

• Sidecar base args gain the full set:
  • --cache-type-k q4_0 — KV cache quantization (~4× VRAM savings)
  • --cache-reuse 256 — KV cache reuse across turns (prompt caching)
  • --slot-save-path /tmp/llama-slots — disk-persistent KV cache
  • --cache-idle-slots — auto-save idle slot caches to disk
  • --spec-type ngram-mod --spec-ngram-mod-thsh 2 — spec decoding
  • --ctx-checkpoints 32 — context overflow protection
  • --sleep-idle-seconds 600 — GPU memory reclaim when idle
  • --metrics — Prometheus metrics endpoint
• Both validators keep existing shadow lists (correct as-is)
• /tmp/llama-slots created for slot KV cache persistence

Dependencies

• llama-sidecar repo (separate git tree, /opt/forks/llama-sidecar/)
• BooCode server (llama-args-validator.ts, provider.ts)

Routing Change

Previously, the llama-sidecar was only used when an agent had llama_extra_args set in its AGENTS.md frontmatter. The default path was llama-swap (no cache quant, no spec decoding, no slot save).

Now, when LLAMA_SIDECAR_URL is configured (it is in docker-compose.yml), ALL inference requests route through the sidecar by default, regardless of whether the agent has llama_extra_args. This means:

• Every request gets KV cache quantization, spec decoding, prompt caching
• Agents with explicit llama_extra_args still get their overrides on top
• If LLAMA_SIDECAR_URL is unset, falls back to llama-swap (backward compat)

Risk

• ngram-mod spec decoding adds ~16MB memory. Trivial vs the 35B model.
• KV cache quant to q4_0 is lossy vs f32 — undetectable on code tasks.
• Both well-tested in llama.cpp ecosystem. No known regressions.
• If issues, remove from base args and restart — no code change needed.