Subject-Emu-5259 commited on
Commit
1b03f30
·
verified ·
1 Parent(s): cb3315a

step-32000 refresh: PPL 54.16 (corrected-label cycle), weights/eval/charts/badges + fresh-corpus cross-check

Browse files
README.md CHANGED
@@ -18,7 +18,7 @@ pipeline_tag: text-generation
18
 
19
  <!--
20
  NeuralAI — Nae1
21
- Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259) Last updated: 2026-10-08 (step-28000 refresh)-->
22
 
23
  <img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
24
 
@@ -32,8 +32,8 @@ pipeline_tag: text-generation
32
  <p align="center">
33
  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/🤗%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
34
  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
35
- <img src="https://img.shields.io/badge/held--out%20PPL-146.17-34d399" alt="PPL 146.17">
36
- <img src="https://img.shields.io/badge/steps-28,000-a78bfa" alt="28000 steps">
37
  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
38
  </p>
39
 
@@ -49,9 +49,9 @@ pipeline_tag: text-generation
49
  | **Vocabulary** | 439 tokens (ByteLevel BPE, sliced from a 32,000-target config) |
50
  | **Context** | 2,048 trained · evaluated on `n_ctx=512` |
51
  | **Training data** | 768M tokens — FineWeb CC-MAIN-2013-20, 400k documents, 8 shards |
52
- | **Training schedule** | 28,000 steps — seven cosine cycles (0→4k→8k→12k→16k→20k→24k→28k) · Kaggle T4 |
53
- | **Final training loss** | 22.61 @ step 28,000 (best in cycle 7: 21.88 @ step 26,150; final-step loss is batch-noisy — held-out PPL decides) |
54
- | **Held-out perplexity** | **146.17** (Q4_K_M, 992 docs / 1,027,766 tokens, teacher-forced) |
55
  | **Formats in this repo** | `f32` GGUF (305 MB) · `Q4_K_M` GGUF (45 MB) · config + tokenizer |
56
  | **License** | Apache 2.0 |
57
 
@@ -60,10 +60,10 @@ pipeline_tag: text-generation
60
  ## 📈 Training
61
 
62
  <p align="center">
63
- <img src="neuralai-nae1-training.png" alt="Nae1 training loss step 24000 to 28000" width="92%" />
64
  </p>
65
 
66
- Nae1 is pretrained from random initialization in seven cosine cycles: 0 → 4,000 · 4,000 → 8,000 · 8,000 → 12,000 · 12,000 → 16,000 · 16,000 → 20,000 · 20,000 → 24,000 · 24,000 → 28,000, each resumed from the previous checkpoint. The chart shows cycle seven — loss moves from ~24.1 to a best of **21.88** (step 26,150), finishing at **22.61** at step 28,000 as the learning rate anneals (batch-noisy final step; held-out PPL keeps improving: 148.84 → 147.97 → **146.17**).
67
 
68
  - **Data pipeline**: FineWeb parquet → chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
69
  - **Checkpoints**: every 250 steps; this release ships the final step-28000 checkpoint
@@ -88,9 +88,10 @@ Teacher-forced perplexity on a held-out FineWeb slice (992 documents / 1,027,766
88
  | step-16000 | 151.45 |
89
  | step-20000 | 148.84 |
90
  | step-24000 | 147.97 |
91
- | **step-28000 — this release** | **146.17** |
 
92
 
93
- Raw report: [`eval_step28000_q4_n512.json`](eval_step28000_q4_n512.json) — mean NLL 4.9848 nats/token (prior releases kept alongside: [`eval_step24000_q4_n512.json`](eval_step24000_q4_n512.json), [`eval_step20000_q4_n512.json`](eval_step20000_q4_n512.json), [`eval_step16000_q4_n512.json`](eval_step16000_q4_n512.json), [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json)). Fixed probe ("The quick brown fox", 10 tokens): 174.73 — the 10-token probe is noisy; held-out decides.
94
 
95
  > ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params over 768M tokens, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant.
96
 
@@ -131,7 +132,10 @@ print(out["choices"][0]["text"])
131
  | `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB — for requantization / research |
132
  | `config.json` | Architecture config as trained |
133
  | `tokenizer/` | Native 439-token vocab + merges + manifest |
134
- | `eval_step28000_q4_n512.json` | The exact evaluation report cited above |
 
 
 
135
  | `eval_step24000_q4_n512.json` | Prior release (step-24000), kept for lineage |
136
  | `eval_step20000_q4_n512.json` | Prior release (step-20000), kept for lineage |
137
  | `eval_step16000_q4_n512.json` | Prior release (step-16000), kept for lineage |
@@ -149,7 +153,7 @@ print(out["choices"][0]["text"])
149
 
150
  - **Scale**: 100M parameters pretrained on 768M tokens is a research checkpoint — long-form reasoning, coding, and factual recall are limited.
151
  - **Vocabulary**: a 439-token BPE trained on this corpus; out-of-domain text will tokenize inefficiently.
152
- - **No chat template**: base completion only — it was never instruction-tuned (SFT run in progress).
153
  - **No internet access**: pair with a tool layer if you need live data.
154
 
155
  ---
@@ -175,7 +179,7 @@ NeuralAI was built from resilience, fatherhood, and the belief that personal com
175
  title = {NeuralAI — Nae1},
176
  year = {2026},
177
  url = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
178
- version = {step-28000},
179
  description = {A 100M-parameter language model pretrained from scratch on 768M FineWeb tokens}
180
  }
181
  ```
 
18
 
19
  <!--
20
  NeuralAI — Nae1
21
+ Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259) Last updated: 2026-10-08 (step-32000 refresh)-->
22
 
23
  <img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
24
 
 
32
  <p align="center">
33
  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/🤗%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
34
  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
35
+ <img src="https://img.shields.io/badge/held--out%20PPL-54.16-34d399" alt="PPL 54.16">
36
+ <img src="https://img.shields.io/badge/steps-32,000-a78bfa" alt="32000 steps">
37
  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
38
  </p>
39
 
 
49
  | **Vocabulary** | 439 tokens (ByteLevel BPE, sliced from a 32,000-target config) |
50
  | **Context** | 2,048 trained · evaluated on `n_ctx=512` |
51
  | **Training data** | 768M tokens — FineWeb CC-MAIN-2013-20, 400k documents, 8 shards |
52
+ | **Training schedule** | 32,000 steps — eight cosine cycles (0→4k→…→28k→32k) · Kaggle T4 |
53
+ | **Final training loss** | 17.93 @ step 32,000 (sum over 8 grad-accum micros; cycle 8 is the first trained on corrected next-token labels — resume started at 28.87 under the fixed objective, so pre-fix values are not comparable) |
54
+ | **Held-out perplexity** | **54.16** (Q4_K_M, 992 docs / 1,027,766 tokens, teacher-forced; cross-checked **57.09** on a fresh unseen FineWeb slice vs step-28000's 143.64 — the improvement is real generalization) |
55
  | **Formats in this repo** | `f32` GGUF (305 MB) · `Q4_K_M` GGUF (45 MB) · config + tokenizer |
56
  | **License** | Apache 2.0 |
57
 
 
60
  ## 📈 Training
61
 
62
  <p align="center">
63
+ <img src="neuralai-nae1-training.png" alt="Nae1 training loss step 28000 to 32000" width="92%" />
64
  </p>
65
 
66
+ Nae1 is pretrained from random initialization in eight cosine cycles: 0 → 4,000 · 4,000 → 8,000 · … · 24,000 → 28,000 · 28,000 → 32,000, each resumed from the previous checkpoint. The chart shows cycle eight — the **first cycle trained with corrected next-token labels** (an earlier double-shift bug had models predicting token t+2; the held-out gate was always standard next-token, so historical PPL numbers remain valid). Under the corrected objective the logged loss resumes at 28.87, falls to a best of **15.72** (step 31,500) and finishes at **17.93** as the learning rate anneals to zero. Held-out PPL improves 147.97 → 146.17 → **54.16** — a 63% drop from fixing the training objective; a fresh-corpus cross-check (unseen FineWeb CC-MAIN-2014-10: **57.09** vs step-28000's **143.64**) confirms the jump is genuine generalization, not eval-set leakage.
67
 
68
  - **Data pipeline**: FineWeb parquet → chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
69
  - **Checkpoints**: every 250 steps; this release ships the final step-28000 checkpoint
 
88
  | step-16000 | 151.45 |
89
  | step-20000 | 148.84 |
90
  | step-24000 | 147.97 |
91
+ | step-28000 | 146.17 |
92
+ | **step-32000 — this release** | **54.16** |
93
 
94
+ Raw report: [`eval_step32000_q4_n512.json`](eval_step32000_q4_n512.json) — mean NLL 3.9919 nats/token; fresh-corpus cross-check: [`eval_fresh_step32000_q4_n512.json`](eval_fresh_step32000_q4_n512.json) / [`eval_fresh_step28000_q4_n512.json`](eval_fresh_step28000_q4_n512.json) (prior releases kept alongside: [`eval_step24000_q4_n512.json`](eval_step24000_q4_n512.json), [`eval_step20000_q4_n512.json`](eval_step20000_q4_n512.json), [`eval_step16000_q4_n512.json`](eval_step16000_q4_n512.json), [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json)). Fixed probe ("The quick brown fox", 10 tokens): 174.73 — the 10-token probe is noisy; held-out decides.
95
 
96
  > ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params over 768M tokens, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant.
97
 
 
132
  | `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB — for requantization / research |
133
  | `config.json` | Architecture config as trained |
134
  | `tokenizer/` | Native 439-token vocab + merges + manifest |
135
+ | `eval_step32000_q4_n512.json` | The exact evaluation report cited above |
136
+ | `eval_fresh_step32000_q4_n512.json` | Fresh-corpus cross-check (unseen CC-MAIN-2014-10), this release |
137
+ | `eval_fresh_step28000_q4_n512.json` | Fresh-corpus cross-check, prior release |
138
+ | `eval_step28000_q4_n512.json` | Prior release (step-28000), kept for lineage |
139
  | `eval_step24000_q4_n512.json` | Prior release (step-24000), kept for lineage |
140
  | `eval_step20000_q4_n512.json` | Prior release (step-20000), kept for lineage |
141
  | `eval_step16000_q4_n512.json` | Prior release (step-16000), kept for lineage |
 
153
 
154
  - **Scale**: 100M parameters pretrained on 768M tokens is a research checkpoint — long-form reasoning, coding, and factual recall are limited.
155
  - **Vocabulary**: a 439-token BPE trained on this corpus; out-of-domain text will tokenize inefficiently.
156
+ - **No chat template**: base completion only — it was never instruction-tuned (SFT completed; checkpoint selection in progress).
157
  - **No internet access**: pair with a tool layer if you need live data.
158
 
159
  ---
 
179
  title = {NeuralAI — Nae1},
180
  year = {2026},
181
  url = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
182
+ version = {step-32000},
183
  description = {A 100M-parameter language model pretrained from scratch on 768M FineWeb tokens}
184
  }
185
  ```
eval_fresh_step28000_q4_n512.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "artifacts/nae1_gguf/nae1-llama-Q4_K_M-step28000.gguf",
3
+ "context_size": 512,
4
+ "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
5
+ "probe": {
6
+ "text": "The quick brown fox",
7
+ "tokens": 10,
8
+ "total_nll_nats": 51.63228580836413,
9
+ "perplexity": 174.72766832853299
10
+ },
11
+ "heldout": {
12
+ "documents": 300,
13
+ "tokens": 350788,
14
+ "total_nll_nats": 1742465.3284437277,
15
+ "mean_nll_nats_per_token": 4.967288870895605,
16
+ "perplexity": 143.63694072145765
17
+ }
18
+ }
eval_fresh_step32000_q4_n512.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "artifacts/nae1_gguf/nae1-llama-Q4_K_M-step32000.gguf",
3
+ "context_size": 512,
4
+ "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
5
+ "probe": {
6
+ "text": "The quick brown fox",
7
+ "tokens": 10,
8
+ "total_nll_nats": 34.301195067379204,
9
+ "perplexity": 30.880332937026726
10
+ },
11
+ "heldout": {
12
+ "documents": 300,
13
+ "tokens": 350788,
14
+ "total_nll_nats": 1418830.3483974792,
15
+ "mean_nll_nats_per_token": 4.044694654313942,
16
+ "perplexity": 57.093750042001716
17
+ }
18
+ }
eval_step32000_q4_n512.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/artifacts/nae1_gguf/nae1-llama-Q4_K_M-step32000.gguf",
3
+ "context_size": 512,
4
+ "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
5
+ "probe": {
6
+ "text": "The quick brown fox",
7
+ "tokens": 10,
8
+ "total_nll_nats": 34.301195067379204,
9
+ "perplexity": 30.880332937026726
10
+ },
11
+ "heldout": {
12
+ "documents": 992,
13
+ "tokens": 1027766,
14
+ "total_nll_nats": 4102713.7013922515,
15
+ "mean_nll_nats_per_token": 3.9918752920336456,
16
+ "perplexity": 54.15635317413229
17
+ }
18
+ }
nae1-llama-Q4_K_M.gguf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1e7b4181f919bdf27ef18596ebe76923a2126e07e55b32ae4e86d155c7b4f821
3
  size 45762464
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe1492ec04cc3744a7306e4303dffe25a7fb02f64cb24b7087a8c84ca8807704
3
  size 45762464
nae1-llama-f32.gguf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:60b86a70921906f1dc3d3b79e321ffd299aee6030618f7e038de83040734753a
3
  size 304778816
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2eba24ceefa28a3da26d4d06c79fc478932ada989c3cd889ce1225510ee33465
3
  size 304778816
neuralai-nae1-banner.png CHANGED

Git LFS Details

  • SHA256: 320e34d992deee9e230ccd43c01e66b7b3fa2810ce16430cabb737c3d19555b9
  • Pointer size: 131 Bytes
  • Size of remote file: 328 kB

Git LFS Details

  • SHA256: 78c0f7336ffd550fc7639bffe02eba49caab558f73edd212e2c9ca867f8c9402
  • Pointer size: 131 Bytes
  • Size of remote file: 328 kB
neuralai-nae1-banner.svg CHANGED
neuralai-nae1-comparison.png CHANGED
neuralai-nae1-comparison.svg CHANGED
neuralai-nae1-training.png CHANGED

Git LFS Details

  • SHA256: 19747d45327cf7104fb2fa77054e224f85277789120aec861c3199b1576809c4
  • Pointer size: 131 Bytes
  • Size of remote file: 156 kB

Git LFS Details

  • SHA256: af43c63a25cb3aa2b24ad2007c694adaf6aac56d9aafd3de3d77dd1f4d9c558f
  • Pointer size: 131 Bytes
  • Size of remote file: 126 kB
neuralai-nae1-training.svg CHANGED