Fast KV Compaction via Attention Matching

This fork implements the algorithm from arXiv:2602.16284 by Zweiger et al. (MIT Han Lab, 2026).

Paper Summary

KV cache compaction replaces traditional context eviction (truncation, sliding window) with a learned compression. Given a full KV cache of N tokens, compaction produces a smaller cache of M tokens (M << N) such that the model's attention behavior is preserved.

Key Insight

Instead of selecting which tokens to keep and which to discard, Attention Matching finds a small set of synthetic key-value pairs that reproduce the original attention distribution. The compacted cache is not a subset of the original — it is a new, learned representation.

Algorithm

1. Position Selection — Choose M positions from the original N using attention-weighted scoring. Each position contributes proportionally to its attention mass.
2. Beta Fitting (NNLS) — Solve for additive bias terms that correct the attention distribution. Uses Non-Negative Least Squares with projected gradient descent.
3. Value Fitting (Least-Squares) — Solve for compressed V such that the weighted sum of values matches the original. Uses QR factorization with Cholesky fallback.
4. Execution — During inference, compacted K/V/beta tensors are prepended to the live KV cache. Beta is injected as an additive attention bias.

Fork vs Paper Reference

This implementation extends the paper in several dimensions:

Implementation Scale

Citation

Links

• arXiv:2602.16284 — Full paper
• Detailed comparison — Fork vs reference implementation
• Algorithm docs — Selection, fitting, execution stages
• Upstream RFC — ggml-org/llama.cpp#20037