KVC single-seed archive (in-loop low-rank KV-cache compression)
Original single-seed checkpoints from the in-loop frozen low-rank KV-cache compression study (master's thesis pivot). Each model was finetuned against a FIXED per-(layer, KV-head) SVD basis inserted in the attention forward pass, so the model's own weights adapt to a frozen compression bottleneck (no trainable compression parameters).
Arms. A0 = plain finetune, evaluated uncompressed (the finetuning floor). A3 = in-loop
compression-aware finetune (the proposed method). The fair comparison is A3 vs A2a (A0's weights +
a fresh post-hoc SVD basis), at matched cache budget. Headline finding: A3 beats A2a by 2-8x PPL
across models and ratios. Keys are truncated pre-RoPE, values directly; rank = head_dim / ratio.
Contents
| Subfolder | Model | Arm | Ratio | Notes |
|---|---|---|---|---|
out_llama/A3 |
Llama-3.2-1B | A3 in-loop | 4x | STE gradient, full-FT (headline) |
out_llama_matrix/A3/r3 |
Llama-3.2-1B | A3 in-loop | 3x | STE, full-FT; bases/ cover r3-r8 |
out_audits/A3_4x_exact |
Llama-3.2-1B | A3 in-loop | 4x | EXACT projection gradient (STE-vs-exact ablation) |
out_llama_lc4096/A3 |
Llama-3.2-1B | A3 in-loop | 4x | trained at seq-4096 (long-context fix) |
out_llama3_3b/{A0,A3} |
Llama-3.2-3B | A0, A3 | 4x | LoRA adapters (merge onto the base) |
out_llama3_8b/{A0,A3} |
Llama-3-8B | A0, A3 | 4x | LoRA adapters (merge onto the base) |
out_downstream/*.jsonl |
-- | -- | -- | downstream MC-accuracy eval results |
bases/*.pt are the frozen SVD bases (base = from the base model, a0/ft = from the finetuned
weights). LoRA folders are adapters: load the matching base model and merge. Full per-checkpoint PPL
and downstream numbers are in the project's RESULTS.md.