AlephLM-0 β Technical Companion
Repo: AbstractPhil/alephlm-0 Β· Author: AbstractPhil Β· Status: complete β seed-0 and seed-1 verdicts in, two-seed tie (Section 5)
This document accompanies the week-in-review article and is written to be self-contained: every claim below carries its number, and every number is inherited from the campaign's verified ledger. All weights referenced here β every arm, every seed, including the control and frozen arms β ship to this repository with per-run ledgers.
1. Overview
AlephLM-0 is the first experiment in which the anchored differentiation mechanism β the "aleph": dense signed dispatch through sinh/cosh reads over oriented half-axes, with no argmax, no softmax routing, and no selection event anywhere β serves as the trunk of an encoder rather than as an adapter attached to one. The question it asks: when the routing mechanism is the trunk, does an anchored encoder match a parameter-matched dense encoder under an identical objective?
Two claims were separated at design time and preregistered:
- E1 (this experiment): an anchored encoder trunk, trained as a trunk, matches or beats a parameter-matched dense trunk under the identical objective. E1 was explicitly labeled pre-placement β the routing sits inside the FFN blocks, upstream of any output distribution.
- E2 (the program's bet, next stage): the aleph read pays in a generative output path, where the address parameterizes the token distribution directly. The plan preregistered that an E1 refutation would itself be the placement law's predicted outcome and would leave the program's direction unchanged.
The headline result β a three-way capability tie at .6031 / .6026 / .6033 (seed 0) β is one of the two outcomes the design named in advance, and the design says exactly what it means. Sections 6β8 show what sits underneath the tie: a 3Γ sample-efficiency gap at small budget, a shared training-gauge attractor across eight separately-trained trunks, and an inhibition-majority signed structure in the dispatch.
2. The three arms and exact capacity parity
Three arms, two seeds each, on the identical objective, data, and schedule:
| arm | what it is | what it isolates |
|---|---|---|
| a1 anchored | learned addressing (codebooks train) | the object under test |
| a2 dense | the exact certified dense architecture (ff2048) | capacity baseline + replication anchor |
| a3 frozen-anchor | anchors frozen at random init | routing content vs routing capacity |
Exact capacity parity. Per block, the dense trunk's 2048-wide FFN is replaced by a bank of four experts: one always-on trunk expert (hidden 512) plus three dispatched experts (hidden 512 each) β total hidden width 2048, exactly the dense baseline. Parameter counts: 58,345,764 anchored vs 58,308,864 dense β +0.063%, with the delta accounted exactly (codebooks 18,468 + extra biases 18,432). Any behavioral difference is attributable to routing, since capacity is held fixed.
Zero-init null-path construction. Dispatched experts enter through gates initialized at sigmoid(β3) with zero-initialized output layers, so at initialization the dispatch contributes nothing: gates-off equals the trunk-expert-only path bit-exactly, asserted in fp32 at init and at every checkpoint. The trunk starts as a dense trunk and grows its routed function from silence. One measured consequence was preregistered before training: the routing gradient wakes at step 1 rather than step 0 (dL/dw = sigmoid(g)Β·E(x), and E β‘ 0 at init) β one dead routing step is the price of the bit-exactness guarantee.
Why this construction matters. The toggle law travels with the trunk: because gates-off is exactly the dense path, switching the dispatch off at evaluation time reads the routed experts' contribution causally, on any downstream task, with no retraining and no confound. Every capability number in this document has a toggle row beside it for exactly this reason.
3. Training configuration
The recipe is inherited verbatim from the certified dense line:
| item | value |
|---|---|
| objective | symmetric InfoNCE + MSE against the 5-BERT reference-member consensus target |
| corpus | 31,905,616 rows Γ 4 epochs = 62,312 steps |
| batch | 2048 |
| lr | 6e-4, cosine schedule |
| optimizer | pure Adam, weight decay 0 |
| pooling | CLS (settled by instrument S0e, Section 8) |
| sequence length | 256 |
| seeds | 0 and 1 per arm |
| hardware | rented consumer/datacenter GPUs (RTX 5090-class, then A40-class) |
Wall-clock. Aleph arms take roughly 2Γ the dense step time under inner activation checkpointing (53 vs 115 steps/min). On the seed-0 campaign this put the dense arm at ~10.1 h and the frozen-anchor arm (a3-s0) at ~20.5 h.
Memory. After an activation-checkpointing fix, the anchored arm's worst-case training memory came in leaner than dense (14.77 GB vs 15.47 GB at the probe scale) β the routed trunk carries no memory tax.
Interruption and resume. The campaign's first GPU host failed mid-run and training completed on A40-class rented GPUs. The three arms have three distinct resume histories, recorded here because they matter for audit: the interrupted anchored run resumed from its shipped checkpoint at approximately step 38k; the dense run restarted from a trivial (near-initial) checkpoint; the frozen-anchor run trained fresh end-to-end on the new hardware. The endpoint agreement across these three histories (Section 4) is itself a small robustness datapoint.
4. Endpoint results
Harness certification first. Before any new row was read, the evaluation harness replicated the historical benchmark record to four decimals (distilbert .5883, v2 .6077, v2-B .6031, v2+arms .7287, B+arms .7294, MiniLM .7925), certifying the eval box for the campaign.
Readout note. AlephLM rows are read through their own trained CLS pooling, while all other models in the harness are mean-pooled. The dense control established that the CLS in-domain advantage is capability-neutral out of domain (Section 8, S0e), so this asymmetry is measured rather than assumed away.
The seed-0 verdict:
| arm | train endpoint (cos / erank) | capability (8-task mean) | dispatch OFF | toggle delta |
|---|---|---|---|---|
| a1 anchored (learned) | .8394 / 98.6 | .6031 | .5743 | β.0288 |
| a2 dense (control) | .8418 / 99.2 | .6026 | β | β |
| a3 frozen-anchor (random) | .8392 / 98.8 | .6033 | .5772 | β.0261 |
Capability spread: .0008 β inside single-task noise, and confirmed by the full two-seed spread (Section 5: six runs inside .0040). The anchored trunk gives up nothing against dense at exact capacity parity (the preregistered match bar reads MATCHED). Learned addressing lands even with the frozen-random partition (a1 β a3 = β.0003) even though a1 demonstrably used its freedom: its anchors moved 1.06 radians from init, and it carries more routed function than a3 (toggle β.0288 vs β.0261; dispatched amplitude .101 vs .095).
The toggle rows are the causal content of the verdict. Switching the dispatch off costs 2.6β5.2 capability points across arms and seeds: the routed experts β three quarters of every FFN's width, reached only through signed dispatch β carry real function. In the frozen-anchor trunk a3-s0, the trained operating point reads gates at sigmoid .37β.50 with per-block dispatched amplitude .068β.090. Degradation is graceful: the silenced trunks (.5743 / .5772) still beat raw bert-base (.5289), so the zero-init null path realized its design goal at task level. Learned and frozen addressing also concentrate function differently: with dispatch off, a1 keeps more STS12 (.5200 vs .4948) and gives up more BIOSSES (.4434 vs .4776) β the same total contribution, allocated differently across tasks.
Per-task Spearman β the tie in detail, plus the seed-1 replication (from eval/alephlm0_benchmark.json):
| run | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|
| a1_anchored-s0 | .5731 | .6528 | .4996 | .6014 | .5457 | .7121 | .6761 | .5639 | .6031 |
| a2_dense-s0 | .5731 | .6507 | .4965 | .5962 | .5430 | .7130 | .6798 | .5683 | .6026 |
| a3_random-s0 | .5707 | .6538 | .4966 | .5998 | .5448 | .7103 | .6806 | .5703 | .6033 |
| a1_anchored-s1 | .5681 | .6512 | .4931 | .5996 | .5423 | .7094 | .6745 | .5673 | .6007 |
| a2_dense-s1 | .5740 | .6537 | .4997 | .6038 | .5417 | .7128 | .6789 | .5671 | .6040 |
| a3_random-s1 | .5717 | .6533 | .4946 | .6007 | .5434 | .7128 | .6800 | .5809 | .6047 |
Toggle decomposition β what the dispatched experts carry (ON β OFF per task; positive = function the routing owns):
| run | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean Ξ |
|---|---|---|---|---|---|---|---|---|---|
| a1_anchored-s0 | β.008 | +.030 | β.020 | +.029 | β.010 | +.035 | +.053 | +.121 | +.0288 |
| a1_anchored-s1 | +.005 | +.050 | β.004 | +.064 | +.001 | +.069 | +.085 | +.034 | +.0382 |
| a3_random-s0 | β.006 | +.023 | +.002 | +.026 | β.004 | +.025 | +.050 | +.093 | +.0261 |
| a3_random-s1 | +.026 | +.053 | +.040 | +.026 | +.000 | +.055 | +.083 | +.135 | +.0524 |
The routing reliably owns the hard semantics (SICK-R, STS13/15/16) and the domain task. In three of the four routed runs the largest single carrier is BIOSSES (+.09 to +.14); the anchored seed-1 run is the exception, shifting that weight onto the semantic suites. Total carried function spans .026β.052 and grows from seed 0 to seed 1 in both routed arms.
The preregistered reading: at trunk scale, pre-placement, every partition of identical capacity β learned, frozen, none β reaches the same capability. Routing content here is amplitude, not choice, which is exactly what the program's placement law predicted for this stage, and why E2 proceeds regardless.
5. Seed-1 replication
a1_anchored seed-1 is complete and seed-stable: capability .6007 vs seed-0's .6031, with the training fingerprint reproduced to three decimals; its mimicry R@1 endpoint is .9975 and its toggle reads β.038. a2_dense seed-1 and a3 seed-1 completed 2026-08-07, and both replicate the basin (finals from each run's checkpoints/metrics.json): a2-s1 at retrieval .9975, cos .8420, erank 99.2; a3-s1 at retrieval .9980, cos .8396, erank 98.6.
The complete two-seed verdict β the tie holds:
| arm | seed 0 | seed 1 | seed spread | toggle s0 | toggle s1 |
|---|---|---|---|---|---|
| a1 anchored (learned) | .6031 | .6007 | β.0024 | β.0288 | β.0382 |
| a2 dense (control) | .6026 | .6040 | +.0014 | β | β |
| a3 frozen-anchor (random) | .6033 | .6047 | +.0013 | β.0261 | β.0524 |
Six runs span .6007β.6047 β a band of .0040, inside the seed-noise scale this program has measured on sibling trunks (.003β.005) β so the endpoint verdict is now two-seed: every partition of identical capacity reaches the same capability, at both seeds, at exact parity. Two texture findings ride on the replication. First, both routed arms carry more function at seed 1 than at seed 0 (a1 β.0288 β β.0382; a3 β.0261 β β.0524): how much of the task the routing absorbs is itself seed-modulated, even while the endpoint is not. Second, the frozen-anchor arm posts the highest single capability of the campaign (.6047) while carrying the largest routed function (β.0524) β consistent with the amplitude law: what matters is how much the dispatch contributes, not how its addresses were chosen.
6. Path findings β the tie decomposed
Sample efficiency. A 33-minute small-rig screen (500k rows Γ 2 epochs, paired seed) answered what the 60-hour runs could not see:
| run (small rig, 500k Γ 2 epochs) | mimicry R@1 | cos | erank | 8-task capability |
|---|---|---|---|---|
| a2 dense | .7790 | .5984 | 35.7 | .3530 |
| a1 anchored | .9055 | .6491 | 46.1 | .3833 |
The capability column extends the path finding beyond the training gauge: at small budget the anchored form is also +.0303 ahead on the benchmark scale (with a small-bed toggle of only β.0047 β at 500k rows the routing has barely begun to specialize, consistent with routing function accruing late).
At small budget the anchored trunk converges dramatically faster than dense. At 6 epochs on the same rig the two meet β a1 .9685 / .7332 / 68.0 vs a2 .9645 / .7204 / 62.6 β with the anchored edge persisting at every budget tested. E1's "tie" is endpoint convergence, and endpoint convergence is a weaker statement than equivalence: the aleph accelerates. The full-scale tie and the small-scale gap are the same phenomenon viewed at two exposures.
Basin identity. Eight separately-trained trunks now land on one attractor on the training gauge (cos to target / effective rank):
| trunk | cos | erank |
|---|---|---|
| v2 (original certified dense) | .8410 | 102.9 |
| v2-B (its replication) | .8421 | 103.1 |
| a2-s0 (dense control) | .8418 | 99.2 |
| a2-s1 (dense control, seed replication) | .8420 | 99.2 |
| a3-s0 (frozen-anchor) | .8392 | 98.8 |
| a3-s1 (frozen-anchor, seed replication) | .8396 | 98.6 |
| a1-s0 (learned-anchor) | .8394 | 98.6 |
| a1-s1 (learned-anchor, seed replication) | .8391 | 98.6 |
That is target cos ~.84 and erank ~99β103 across two poolings, two corpus treatments, different cards and allocators, with and without routing, and now across seeds. The dense recipe is now four-times replicated at capability (.6026β.6077: v2, v2-B, a2-s0, a2-s1). The basin attracts: a2 took a single-eval excursion mid-training (erank momentarily 33.2) and snapped back by the next eval. The endpoint is a property of the objective-plus-data stack; the architecture only chooses the path it travels to get there.
7. Sign census β inhibition is the majority mode
Every prior campaign vital was a density gauge (absolute values, norm ratios, squared spectra) β the signed structure of dispatch was invisible to the instruments themselves. The first signed census, run on the finished trunks, examined the frozen-anchor trunk's (a3) K=3 dispatch and found that negative routing weights are the majority mode: fraction-negative .42β.74 per expert, mean weight negative in most cells (one block reads β.07 / β.14 / β.03), |w| ~.12β.19, block-structured. Subtraction is how this trunk works. A companion system's K=64 relays showed a second regime β balanced bipolar, fraction-negative ~.50 β giving two distinct signed regimes within one mechanism family.
The standing assumption that the mechanism's negative-weight capability was still unused was refuted by measurement: it was in use the whole time, invisible to every abs()-based vital. This changes what the tie is β a substantial fraction of the trunk's compute is organized as structured inhibition, detachable by a switch: a qualitative mechanism finding sitting underneath a quantitative tie.
8. Instruments before spend
Two cheap instruments ran before Stage-1 compute was committed.
S0a β the rank-ceiling control. Measured from stored teacher rows before any training: the consensus target's effective rank is 28.1/768, while its reference member (bert-base) sits at 40.7/768 in-domain (39.6 raw) and 40.3 on out-of-domain STS-B β the rank is the encoder's geometry, independent of the corpus. Matched-gauge ratio 1.45Γ, labeled INTERMEDIATE by an in-script operationalization dated before the run (and explicitly flagged as in-script, since it was absent from the preregistered plan). Reading: consensus construction costs about 30% of the member's rank, and the member itself is already low-rank β so the target's ceiling belongs to the reference member, and raising it means changing the reference member, since enlarging the teacher roster leaves the bert-frame cap near 40. This priced the ceiling the trunk trains toward before two GPU-days were spent.
S0e β the pooling settle. The pooling choice (mean vs CLS vs attention readout) was settled at 500k rows with a paired-seed design: all three readout arms share one seed derived independently of the pooling name, so init and batch order are identical and the arms differ only at the readout. A tie band was preregistered before the runs (top two arms within |Ξcos| β€ .003 AND |ΞR@1| β€ .01 β tie β mean, the target-matching pooling).
| readout | cos to target | mimicry R@1 |
|---|---|---|
| mean | .6037 | .7745 |
| cls | .6147 | .8180 |
| attn | .6033 | .7680 |
CLS won both gauges outside the band (Ξcos +.0109 vs band .003; ΞR@1 +.0435 vs band .01) β the mechanical rule fired; no judgment call was exercised. Two notes: (a) CLS beat mean even though the consensus target is a mean-pooled object; (b) the attention readout, initialized at mean, ended statistically at mean β at this scale the readout stays with its incumbent. The dense control later showed the CLS in-domain advantage is capability-neutral out of domain, so pooling does not confound E1 β a confound retired for about an hour of compute, with an auditable decision because the rule preceded the data.
9. Engineering practices relevant to reproduction
Three practices from this campaign are worth adopting anywhere endpoint numbers must be trusted:
- CUDA-refusal guard. Training entry points hard-fail when a CUDA device is unavailable, with no silent CPU fallback. A run that silently degrades to CPU produces checkpoints and logs that look healthy while training under a different numerical and throughput regime; refusing to start is the cheaper failure.
- Canonical batch shapes. Batches are asserted to canonical shapes at the loader boundary rather than trusted downstream. Shape drift between resumes or hardware migrations otherwise changes kernel selection and allocator behavior mid-campaign, contaminating cross-arm comparisons that assume identical execution.
- Streaming checkpoint store. Checkpoints stream to the remote results store as they are produced rather than at run end. When the campaign's first GPU host failed mid-run, the anchored arm resumed from its shipped checkpoint at ~step 38k; the loss horizon of a host failure is bounded by the shipping interval, and the same store doubles as the public weights ledger.
- Environment captured before the rental ends. Library versions, driver, and the container's real CPU/memory quotas were read off the live machine and shipped as
ENVIRONMENT.mdβ including the per-arm record of which code revision each run actually executed, since arms launched hours apart fetch whatever is newest at their own launch instant. That document also names this campaign's one non-arm-gated implementation difference (the anchored seed-1 run consumed its targets through the earlier in-RAM store, the other two through the streaming memmap store) and the console-log truncation caveat, so both are checked facts rather than assumptions. A rented machine takes its provenance with it unless the provenance is written down first. - Bit-exact null assertion. The gates-off = dense-path identity is asserted in fp32 at init and at every checkpoint, so the toggle readout (Section 4) remains a valid causal instrument across the whole run rather than an init-time property that training could quietly erode.
Across the campaign, every decision rule β tie bands, match/beat bars, abort criteria β was preregistered with numbers before the data existed; where a threshold was operationalized in-script rather than in-plan, it is labeled as such in the shipped artifact.
10. Laws this experiment established or sharpened
- Amplitude, not choice (at trunk scale). Routing under conserved signed amplitude apportions rather than selects. E1 is the strongest datapoint yet: three partitions of identical capacity (learned, frozen, none) reach the same capability in a pooled-objective encoder, while the amplitude itself is real, causal, and detachable (toggle β.0288 / β.0261). Design consequence: expert count and addressing sophistication are empirical questions, never free wins, in this placement.
- Endpoint convergence is a weaker statement than equivalence. Two systems tied at the endpoint differed threefold in convergence speed (mimicry .9055 vs .7790 at small budget). Verdicts read only at convergence miss dynamics; sample efficiency is a separately valuable axis, and small-rig screens are the cheap instrument that reads it.
- The placement law (Law 2), pre-placement reading. The address's chain-rule advantage pays only where the composed address directly parameterizes the output distribution. E1 was pre-placement by design and behaved exactly as the law predicts. Independently, an adoption experiment on a frozen 0.5B LLM watched the aleph itself elect the output-distribution placement β opening its pre-head gate ~8Γ while a comparison mechanism closed the same position ~10Γ. The mechanism votes for where E2 puts it: the generative bed, where dispatch feeds the token distribution.
References and attribution
Only sources this document trains on, evaluates against, or inherits methods from are listed.
Models
- BERT (Devlin et al.) β reference frame and tokenizer for the 5-BERT consensus target; bert-base is the raw baseline in Section 4 and the S0a reference member in Section 8.
- DistilBERT (Sanh et al.) β harness-certification row (.5883).
- MiniLM (Wang et al.), via Sentence-BERT (Reimers & Gurevych) β harness-certification row (.7925).
Data
- captionbert-8192-v2-consensus (AbstractPhil) β the 5-BERT reference-member consensus targets all arms train against.
- conceptual-captions-12m-webdataset-berts (AbstractPhil) β the training corpus, built from CC12M (Changpinyo et al.) as distributed in pixparse/cc12m-wds.
- MTEB β source of the eight capability suites (STS12β16, STS-B, SICK-R, BIOSSES) read through the campaign's certified harness.
Methods
- Transformer (Vaswani et al.) β the encoder blocks whose FFNs host the expert banks.
- InfoNCE (van den Oord et al.) β the contrastive half of the training objective.
- Mixture-of-experts (Shazeer et al.) β the routing lineage the aleph departs from: dense signed dispatch with no selection event, in contrast to softmax-gated expert selection.
- LoRA (Hu et al.) β lineage of the zero-init null-path construction: zero-initialized output layers so the dispatch starts as an exact no-op.
- Gradient checkpointing (Chen et al.) β the inner activation checkpointing under which the aleph arms train (Section 3).
Infrastructure
- PyTorch β training and evaluation framework.
- RunPod β the A40-class GPU rental on which the interrupted campaign completed (Section 3).
Related artifacts: alephlm-0 (this experiment: all arm/seed weights and per-run ledgers) Β· captionbert-8192-v2 and captionbert-8192-v2-B (the certified dense line and its replication) Β· alephlm-adopt-0 (the adoption experiment, Section 10) Β· aleph-splat-0 (the K=64 companion system, Section 7) Β· prior installment: geometric-memory-ft4
Provenance: compiled 2026-08-07 by AbstractPhil from the AlephLM-0 campaign record. Every number above is inherited from the campaign's verified key-number ledger; per-run ledgers and all arm/seed weights (including control and frozen arms) ship with this repository.