Aurov/delta-slotstack-236m

Delta Slot-Stack 236M — a language model built to be read, not to win benchmarks

A 236M-parameter LM whose residual pathway is not an additive stream. Every attention layer writes to its own slot, and that slot is never overwritten. Ten layers, ten slots, plus one for the token embedding.

The point is not accuracy. The point is that after training you can still open the model and see which head put what where — and then change it.

⚠️ Read the Limitations section before drawing conclusions. Most importantly: no same-parameter baseline was trained at this scale.

How this was built. The architecture, the probe-token design, and the experimental questions are the author's. The author does not read or write English and does not write code — the training script, the six analysis scripts, and this model card were all written by Claude (Anthropic) from the author's specifications in Chinese, over roughly one day of back-and-forth.

The author works in a factory and trains on rented GPUs as a hobby. Several results in this card exist because the author pushed back on claims Claude had overstated; the "Things we got wrong" section is not decoration.

中文说明见文末。

---

The one structural difference

Standard transformer — a whiteboard. Layer 1 writes on it, layer 2 writes on the same board, and so on. What layer 3 wrote is physically gone by the end; it has been overwritten and rescaled a dozen times.

This model — a row of boxes. Each layer writes into its own box and the box is sealed. At the end, box 3 still contains exactly what attention layer 3 produced.

Everything below follows from that.

Supporting choices (each is a real trade-off, not free):

• No W_o. Head h permanently occupies dimensions [h·128, (h+1)·128) of its slot. Head mixing is deferred to the downstream gate. Attention is 3d² per layer instead of 4d².
• Per-head normalization. Each head's value vector is normalized within its own 128 dims. Zeroing one head does not rescale the other nine — this is what makes slots separable, and it is the technical content of the whole design.
• Cosine attention scores, √hd · cos(q,k). Since q/k are unit-normalized, the standard 1/√hd becomes a multiply, not a divide.
• Bilinear FFN, no activation function anywhere. Wd(gate ⊙ up), where gate reads the history slots and up reads the newest slot. History selects, newest supplies.
• Embeddings on the unit sphere (hard-projected every optimizer step) with a separate head bias carrying the unigram prior.

x ─┬────────────────────────────────────────────┐ slot0 (raw embedding)
   ├→ att1 ─write→ slot1                        │
   │       gate←[slot0]        up←slot1         │
   │       └─── ffn1 ───┐                       │
   ├────────────────────┴→ att2 ─write→ slot2   │
   │       gate←[slot0,slot1]  up←slot2         │
   │         ⋮                                  │
   └→ … att10 ─write→ slot10                    │
           gate←[slot0..slot9]  up←slot10
           └─── ffn10 → RMSNorm → tied embedding + bias → logits

Model details

Parameters	236,555,904 (172.0M matrices + 64.4M tied embedding)
Shape	d=1280, 10 heads × 128, 10 layers
Residual width	11 × 1280 = 14,080 (slot stack)
Data	FineWeb, 10B tokens (~42 tokens/param)
Final val loss	3.161
Training	4×RTX 4090, ~17h, WSD schedule, lr 4e-4, AdamW
Matrix params	105d² vs 120d² for a same-shape standard transformer (14% fewer)

---

What this model lets you do

All numbers below are measured on the final checkpoint (step 38146) with the scripts in this repo. Every one is reproducible in minutes.

1. Rewrite the answer by patching 3 heads — with zero collateral damage

Minimal pairs differing in exactly one token (answer "cat" vs "dog"). Run the "dog" sequence, but replace three head outputs with their values from the "cat" run:

heads patched	says "cat"	says "dog"	says something else
clean run	0%	100%	0%
+ L8h5	31%	69%	0%
+ L8h7	88%	12%	0%
+ L1h9	100%	0%	0%

The "something else" column staying at 0% is the result. This is targeted rewriting, not damage. (Activation patching itself is standard practice — see Limitations for what is and isn't novel here.)

2. Establish what is not involved

Of 100 head positions, ~90 produce exactly 0.000 effect — not small, zero. A head that doesn't attend to the answer position produces bit-identical output on both sequences, so patching it is a no-op.

Being able to rule heads out is what makes attribution trustworthy in the first place.

3. Recompute only the last FFN — bit-exact

Slot 8 is still sitting there at the end of the forward pass. Change it, and only ffn10 → norm → unembed needs to rerun. Verified: recomputing from clean slots reproduces full-forward logits with max deviation 0.00e+00.

This also cleanly separates direct effect (straight to logits) from indirect (via later attention layers):

head	total effect	direct	direct share
L8h5	0.359	0.364	1.01
L1h9	0.181	0.147	0.81
L3h8	0.011	0.003	0.29

Aggregate direct share is 0.93 (prev) and 0.84 (b128) — depth here builds slots rather than transforming them layer by layer.

4. Zero out "layer i reading layer j" and measure it

Each FFN's gate is k physically separate weight blocks; block j reads slot j. So "is layer 7 actually using layer 3?" is answered by zeroing that block:

consumer\source   x      b1     b2     b3     b4     b5     b6     b7     b8     b9
f2              1.242  0.086
f5              0.069  0.003  0.014  0.008  0.032
f10             0.211  0.033  0.025  0.042  0.046  0.025  0.126  0.056  0.365  0.316

This also falsified a metric we had been logging: the norm-share percentages in the training log (correlation) do not match causal ablation. f8's gate blocks all show 0.003–0.016 individually, yet zeroing them together costs 0.571 — the blocks are mutually redundant, individually removable but collectively necessary.

5. Three end-to-end circuits, four independent methods agreeing

circuit	answer carriers	transport edge
previous-token	L8h5, L8h7, L1h9	f10 ← slot8 (+5.71)
retrieve-128-back	L1h9, L2h8, L8h7	f10 ← slot1 (+6.06)
identity (self)	— (margin 11.75, 0% error)	f10 ← slot0 (+6.55)

The two retrieval circuits share no components except the final FFN. Ablation (necessity), patching (sufficiency), weight zeroing (transport), and direct/indirect decomposition all point at the same heads.

6. Double dissociation between syntax and part-of-speech

The model has real contextual ability: on ambiguous tokens (same token, multiple POS tags — 272 types, 17% of positions) probing accuracy goes 50.3% → 62.0%. Dependency tree depth, which is essentially unavailable from the embedding, goes R² 0.111 → 0.479.

Both peak at slots 3–5 and fall off at the last slot — the same profile reported for BERT-base.

And it localizes to individual heads:

ablated	dependency depth R²	POS (ambiguous)
baseline	0.479	60.5
slot4-head2	0.434 (−0.045)	60.5 (−0.0)
slot3-head3	0.458 (−0.010)	58.4 (−2.4)

One head carries structure, the other carries POS, and removing either leaves the other intact. Downstream propagation checks out too (ablating slot4-head2 also costs slot5 −0.031, while slot3 is untouched).

The part worth pausing on: ablating either head costs only ~0.02 LM loss — the median across all 100 heads. They are functionally specific yet invisible in the aggregate metric. That combination is exactly what interpretability work is usually hunting for.

7. The residual read-out is not additively expressible

An additive residual forces every consumer to read all sources through one shared matrix — formally, the gate must be rank-1 along the block dimension. Projecting the trained model onto that class:

deletion (matched Frobenius energy)	ΔVAL
block-rank-1 (= additive-expressible)	+14.18
random subspace, same energy removed	+3.41
smallest-singular-value truncation	+3.15

4.2× worse than an energy-matched random subspace deletion. Block permutation (each block reads a different slot) costs +9.67, i.e. worse than uniform.

This gap widened over training (3.3× at step 2000 → 4.2× at 38146): the longer it trains, the more it relies on structure an additive stream cannot express.

---

Quickstart

import torch
from delta_d1280_h10_l10 import DeltaLM

# 2.84 GB checkpoint — still carries optimizer state; weights are under "model"
ck = torch.load("delta_d1280_h128_l10_latest.pt", map_location="cpu",
                weights_only=False)
model = DeltaLM()
model.load_state_dict(ck["model"])
model.eval()

# slots stay separable all the way to the output
logits, slots = model.run(idx, keep_slots=True)   # slots[0]=embedding, slots[k]=att_k

# zero layer 5 head 3, or substitute any tensor
logits = model.intervene(idx, [(5, 3, None)])

Deliberately not wrapped in a transformers interface — the architecture doesn't fit the abstraction, and pretending otherwise would add glue code and hide the parts that matter.

Analysis toolkit

script	what it answers
head_test.py	head subspace independence; 10×10 ablation heat map; probe × head
patch_test.py	targeted rewriting; direct vs indirect; greedy joint patching
source_test.py	causal slot-to-slot dependency matrix; gate/up pathway ablation
additive_test.py	additive expressibility, with energy-matched controls
ling_probe.py	POS / dependency-depth probes, per-slot and per-head, with ablation
margin_test.py	float64 probe loss; margin as headroom; error rate in the tail

Each carries its reasoning and its caveats in the docstring, including which controls are required and which metrics turned out to be misleading.

Also included: log_d1280_h10_l10.txt — the complete training log with all 38 diagnostic dumps; and test_log.txt — raw output of every analysis quoted above. Nothing was cherry-picked; the logs are there so you can check.

---

Limitations

Trained probe tokens. Seven special tokens (self, prev, first, dup, b16/b128/b512) are inserted at 0.2% during training. They were designed years ago for probing standard transformers — the original motivation was finding a clean scratchpad inside an additive residual stream — so they are not tailored to this architecture. But they are trained-in, and a model without them would behave differently.

No same-scale baseline. This is the big one. On a 48M controlled run (same params, same data, same steps) the slot stack came out 0.07 nats worse than a standard transformer. At 236M it was not measured. Every "cleaner" claim in this card is an absolute measurement with no comparison — a standard transformer might well produce a similarly sparse effect matrix. Untested.

Depth is a hard ceiling. Gate parameters grow as O(L²) — 52% of matrix parameters at L=10, 71% at L=24, 79% at L=37. At L=24, training becomes unstable: gradient norms spike to 10³ while a parameter-matched 35-layer standard transformer sits at 0.2–0.9, and the loss gap widens during training (+1.38 → +1.70). L=10 is comfortably inside the safe zone; do not assume this design scales in depth.

Gradients are structurally large. Slot 0 feeds every gate directly, so its gradient is a sum over L paths. Hyperparameters cannot be copied from standard transformers, and the mismatch grows with depth.

Head subspaces degrade over training. Mean pairwise overlap of the value subspaces rose from 0.108 (step 4000) to 0.259 (final, L2) against a random baseline of 0.100. "Each head reads its own directions" holds, but weakens.

Not a useful general model. 236M with a 3.16 val loss is not competitive. Use it to study the architecture; don't use it to generate text.

Things we got wrong

Kept deliberately, because the failed hypotheses cost real GPU hours:

• Scaling Wd by 1/√L made everything worse (VAL 6.48 → 6.97, spikes 1704 → 3203). The standard-transformer depth correction addresses forward variance accumulation; this architecture replaces rather than accumulates, so the correction is misapplied.
• Loosening gradient clipping was wrong. The large gradient norms looked like collateral damage from clipping; releasing the threshold triggered spike → damage → more spikes. clip=1.0 is load-bearing.
• A "rotated block rank-1" control was void. It amounts to additive residual with scrambled inputs — strictly worse than additive, so it proves nothing. Additive expressibility and block-rank-1 are the same set; there is no way to hold rank fixed while toggling additivity. Use the random-subspace control instead.
• Norm-share ≠ causal dependency. The gate-dependency percentages in the training log are correlational and do not survive causal ablation.
• Ratio metrics with near-zero denominators rank the worst items first. A "syntactic specificity = POS/identity" score put a head with 22.1% POS accuracy (majority baseline: 15.4%) at the top, purely because its denominator was smaller.

Citation

@misc{delta_slotstack_2026,
  title  = {Delta Slot-Stack: a non-additive residual pathway for interpretable transformers},
  year   = {2026},
  note   = {236M parameter model + interpretability toolkit},
  url    = {https://huggingface.co/Aurov/delta-slotstack-236m}
}

Apache-2.0. Issues and reproductions welcome — especially the missing baseline.

---

中文说明

这是什么。 一个 2.36 亿参数的语言模型，残差通路不是"加性残差流"，而是"槽位栈"： 每个注意力层写进自己专属的槽，写完就锁死，永不覆盖。十层十个槽，加一个放原始词嵌入。

打个比方：标准 Transformer 像一块白板，每层都在同一块板上写，写完是一团混合物， "第三层当时写了什么"物理上已经没了。这个模型像一排格子，每层写一个，到最后 第三个格子里还是第三层写的东西，一个字没变。

它不是用来跑分的。 价值在于：训练完成后，你仍然能打开它，看清哪个头把什么 写在了哪里，然后精确地改掉它。

实测结果（全部来自最终 checkpoint，脚本都在仓库里，几分钟可复现）：

• 只替换 3 个头的输出，模型 100% 改口，且不冒出第三个词（"其他"全程 0%）
• 100 个头位置中约 90 个效应精确为零——不是"很小"，是零。 能确定"谁没参与"，是归因可信的前提
• 只重算最后一个 FFN 就能复现完整前向，逐位一致（最大偏差 0.00e+00）。 槽还原样躺在那儿，前面十层一概不用碰
• 可以把"第 i 层读第 j 层"的那块权重直接清零来测因果—— 标准架构里没有这个可清零的对象
• 双重解离：挖掉槽4第2个头，依存树深度探针 R² 从 0.479 掉到 0.434， 而词性纹丝不动（60.5→60.5）；挖掉槽3第3个头，词性掉 2.4 个点，深度几乎不变。 两个头在语言建模 loss 上都只值 0.02（100 个头的中位数水平）—— 功能专一，却在总成绩上看不见
• 语法能力确实存在：一词多性的词（272 种，占 17% 位置）只看词本身猜对 50.3%， 模型看了上下文能到 62.0%；依存树深度 R² 从 0.111 到 0.479，中层峰值、末层回落， 与 BERT-base 上报告的分层剖面一致

局限请一并看：

• 同规模的标准架构对照没有跑过。 48M 上的受控对照中，本架构比标准 Transformer 差 0.07 nats；236M 上差多少，未测。所有"更干净"的说法都是绝对测量、没有参照系
• 深度是硬天花板。 gate 参数按 O(L²) 增长：L=10 时占矩阵参数 52%， L=24 时 71%，L=37 时 79%。L=24 训练已不稳定，梯度尖峰到 10³， 而同参数量的 35 层标准架构稳在 0.2–0.9
• 梯度天然大一个量级，因为槽0 直插每一个 gate，梯度是 L 条路径求和。 超参不能照抄标准架构，深度越大越不能抄
• 不是好用的通用模型。 236M / loss 3.16 没有竞争力， 请用它研究架构，不要用它生成文本

关于作者与工具。 架构设计（槽位栈、无 Wo、双线性 FFN）和七个探针 token 出自作者本人；作者不懂英文也不写代码，训练脚本、六个分析脚本和这份模型卡 全部由 Claude (Anthropic) 根据作者的中文描述编写，前后约一天。 作者是工厂工人，租用 GPU 做业余研究。

本卡片中好几条结论之所以成立，是因为作者驳回了 Claude 说过头的判断—— "Things we got wrong" 那一节不是装饰。