AbstractPhil
/

alephlm-0

+# AlephLM-0 E1 — execution environment
+The exact stack the seed-1 arms (`a1_anchored-s1`, `a2_dense-s1`, `a3_random-s1`)
+trained on, captured from the live container before it was released. Recorded
+because several of this line's engineering laws are pinned to specific versions
+and to the container's resource envelope — a replication on a drifted image
+would change checkpointing and compile semantics silently.
+## Software
+| component | version |
+|---|---|
+| Python | 3.11.10 (GCC 11.4.0) |
+| PyTorch | 2.4.1+cu124 |
+| CUDA (torch) | 12.4 |
+| cuDNN | 9.1.0 (90100) |
+| Triton | 3.0.0 |
+| transformers | 5.14.1 |
+| tokenizers | 0.22.2 |
+| huggingface_hub | 1.26.0 |
+| datasets | 5.0.1 |
+| safetensors | 0.8.0 |
+| numpy | 1.26.3 |
+| tensorboard | 2.21.0 |
+| torchvision / torchaudio | 0.19.1+cu124 / 2.4.1+cu124 |
+| NVIDIA driver | 570.195.03 |
+| OS / kernel | Ubuntu 22.04.5 LTS / 6.8.0-64-generic |
+Full package set: `pip_freeze.txt` (179 packages) beside this file.
+**Why the torch pin matters.** Three separate behaviors in this line are
+torch-2.4-specific: `torch.utils.checkpoint` needs an explicit import (2.4.1
+does not auto-import the submodule; 2.8+ does); compile-inside-checkpoint hits
+2.4's RNG-fork/fake-tensor collision, so the compile path asserts
+inner-checkpointing off; and 2.4's checkpoint recompute cannot re-enter dynamo,
+a measured structural limit rather than a configuration choice.
+## Hardware and container envelope
+| resource | value |
+|---|---|
+| GPUs | 2 × NVIDIA A40, 46,068 MiB each, PXB peer link |
+| GPU UUIDs | `GPU-b5d01167-717d-2198-1b7f-7ff14788d65a`, `GPU-0b319f75-ee74-46e3-9831-01dc3b8fe681` |
+| Host CPU | Intel Xeon Gold 6342 @ 2.80 GHz, 96 vCPU visible |
+| Host RAM | 540,644,167,680 B (~503 GiB) visible |
+| **Container memory quota** | **110,999,998,464 B (103.38 GiB)** |
+| **Container CPU quota** | **17.85 cores** (cfs quota 1,785,000 / period 100,000) |
+| CPU affinity / NUMA | 0-23, 48-71 / node 0 |
+| Container id | `d2d7b1e1e4e4` |
+| Storage | 200 GB container overlay; 400 GB network volume at the work root |
+**Why the quotas matter.** The visible host figures (96 vCPU, ~503 GiB) are not
+what the container may use — the real envelope is 17.85 cores and 103.38 GiB.
+The streaming memmap target store exists precisely to fit that memory quota: an
+earlier build held the full 49 GB target buffer in RAM and, with transient spike,
+crossed the quota and took a silent cgroup SIGKILL at the final concatenate.
+Any rig sized against that law wants the exact byte value above, not a rounding.
+## Code revision per arm
+The three seed-1 arms each fetched the training module at their own launch
+instant, so they did not all execute the same revision. Launch times come from
+the tensorboard filenames shipped with each run (`events.out.tfevents.<epoch>.<container>.<pid>`):
+| arm | launch (UTC) | revision executed | target store implementation |
+|---|---|---|---|
+| `a1_anchored-s1` | 02:53:28 | `abdd691` | in-RAM concatenate |
+| `a3_random-s1` | 03:36:35 | `e21de50` | streaming memmap cache |
+| `a2_dense-s1` | 20:30:19 | `ae93b6b` | streaming memmap cache (cache hit) |
+`e21de50 → ae93b6b` changes 168 lines, but every hunk on a shared path is gated
+behind the splat arm, which none of these three runs use — so **a2 and a3 are
+behavior-identical**. `abdd691 → e21de50` changes 86 lines located entirely in
+the target store and **not** arm-gated: a1 therefore consumed its training
+targets through a different store implementation than a2 and a3.
+The store change alters how targets are held, not what they contain (the memmap
+byte arithmetic reconciles exactly to the source tensors: 31,905,616 × 768 × 2
+bytes with zero slack), and a1-s1's capability lands within seed noise of
+a1-s0 (.6007 vs .6031, trained under yet another build). The recorded position
+is therefore that the arms are equivalent, stated as a checked fact with its
+one caveat named rather than as an unexamined assumption.
+## Console-log completeness caveat (`a1_anchored-s1`)
+Two orchestrators launched `a1` against the same fixed log path with truncating
+redirection. The duplicate failed its VRAM probe and exited while the real run
+continued writing at its own file offset, so the surviving console log carries a
+490-byte hole and its first 7,195 bytes belong to the failed duplicate. Practical
+consequences: the header line reporting a 14.3 GB allocator cap belongs to the
+**duplicate**, not to the run that produced the shipped weights; and the real
+run's own preamble (its allocator cap, VRAM-probe pass, config echo, resume
+line) plus roughly the first 61% of its console output have no surviving
+witness. The training record itself is unaffected — the tensorboard scalars on
+the hub cover the full span, and the weights, `metrics.json`, and resume state
+are complete. Fixed for future campaigns by giving every launch a distinct log
+path and appending rather than truncating.
+*Captured 2026-08-07 by AbstractPhil before container release.*