Subject-Emu-5259 commited on
Commit
cb3315a
Β·
verified Β·
1 Parent(s): 335c09f

step-28000 refresh: PPL 146.17, weights/eval/charts/badges

Browse files
README.md CHANGED
@@ -18,7 +18,7 @@ pipeline_tag: text-generation
18
 
19
  <!--
20
  NeuralAI β€” Nae1
21
- Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259) Last updated: 2026-10-08 (step-24000 refresh)-->
22
 
23
  <img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
24
 
@@ -32,8 +32,8 @@ pipeline_tag: text-generation
32
  <p align="center">
33
  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/πŸ€—%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
34
  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
35
- <img src="https://img.shields.io/badge/held--out%20PPL-147.97-34d399" alt="PPL 147.97">
36
- <img src="https://img.shields.io/badge/steps-24,000-a78bfa" alt="24000 steps">
37
  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
38
  </p>
39
 
@@ -49,9 +49,9 @@ pipeline_tag: text-generation
49
  | **Vocabulary** | 439 tokens (ByteLevel BPE, sliced from a 32,000-target config) |
50
  | **Context** | 2,048 trained Β· evaluated on `n_ctx=512` |
51
  | **Training data** | 768M tokens β€” FineWeb CC-MAIN-2013-20, 400k documents, 8 shards |
52
- | **Training schedule** | 24,000 steps — six cosine cycles (0→4k→8k→12k→16k→20k→24k) · Kaggle T4 |
53
- | **Final training loss** | 24.17 @ step 24,000 (best in cycle 6: 21.81 @ step 21,250; final-step loss is batch-noisy β€” held-out PPL decides) |
54
- | **Held-out perplexity** | **147.97** (Q4_K_M, 992 docs / 1,027,766 tokens, teacher-forced) |
55
  | **Formats in this repo** | `f32` GGUF (305 MB) Β· `Q4_K_M` GGUF (45 MB) Β· config + tokenizer |
56
  | **License** | Apache 2.0 |
57
 
@@ -60,13 +60,13 @@ pipeline_tag: text-generation
60
  ## πŸ“ˆ Training
61
 
62
  <p align="center">
63
- <img src="neuralai-nae1-training.png" alt="Nae1 training loss step 20000 to 24000" width="92%" />
64
  </p>
65
 
66
- Nae1 is pretrained from random initialization in six cosine cycles: 0 β†’ 4,000 Β· 4,000 β†’ 8,000 Β· 8,000 β†’ 12,000 Β· 12,000 β†’ 16,000 Β· 16,000 β†’ 20,000 Β· 20,000 β†’ 24,000, each resumed from the previous checkpoint. The chart shows cycle six β€” loss drops from ~25.4 to a best of **21.81** (step 21,250), the lowest training loss of the run, finishing at **24.17** at step 24,000 as the learning rate anneals to 1.3e-12 (batch-noisy final step; held-out PPL keeps improving: 151.45 β†’ 148.84 β†’ **147.97**).
67
 
68
  - **Data pipeline**: FineWeb parquet β†’ chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
69
- - **Checkpoints**: every 250 steps; this release ships the final step-24000 checkpoint
70
 
71
  ---
72
 
@@ -87,9 +87,10 @@ Teacher-forced perplexity on a held-out FineWeb slice (992 documents / 1,027,766
87
  | step-12000 | 154.25 |
88
  | step-16000 | 151.45 |
89
  | step-20000 | 148.84 |
90
- | **step-24000 β€” this release** | **147.97** |
 
91
 
92
- Raw report: [`eval_step24000_q4_n512.json`](eval_step24000_q4_n512.json) β€” mean NLL 4.9970 nats/token (prior releases kept alongside: [`eval_step20000_q4_n512.json`](eval_step20000_q4_n512.json), [`eval_step16000_q4_n512.json`](eval_step16000_q4_n512.json), [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json)). Fixed probe ("The quick brown fox", 10 tokens): 171.33 β€” the 10-token probe is noisy; held-out decides.
93
 
94
  > ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params over 768M tokens, expect exploratory, often garbled text β€” this is a research-lineage model, not a chat assistant.
95
 
@@ -130,7 +131,8 @@ print(out["choices"][0]["text"])
130
  | `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB β€” for requantization / research |
131
  | `config.json` | Architecture config as trained |
132
  | `tokenizer/` | Native 439-token vocab + merges + manifest |
133
- | `eval_step24000_q4_n512.json` | The exact evaluation report cited above |
 
134
  | `eval_step20000_q4_n512.json` | Prior release (step-20000), kept for lineage |
135
  | `eval_step16000_q4_n512.json` | Prior release (step-16000), kept for lineage |
136
  | `eval_step8000_q4_n512.json` | Prior release (step-8000), kept for lineage |
@@ -173,7 +175,7 @@ NeuralAI was built from resilience, fatherhood, and the belief that personal com
173
  title = {NeuralAI β€” Nae1},
174
  year = {2026},
175
  url = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
176
- version = {step-24000},
177
  description = {A 100M-parameter language model pretrained from scratch on 768M FineWeb tokens}
178
  }
179
  ```
 
18
 
19
  <!--
20
  NeuralAI β€” Nae1
21
+ Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259) Last updated: 2026-10-08 (step-28000 refresh)-->
22
 
23
  <img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
24
 
 
32
  <p align="center">
33
  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/πŸ€—%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
34
  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
35
+ <img src="https://img.shields.io/badge/held--out%20PPL-146.17-34d399" alt="PPL 146.17">
36
+ <img src="https://img.shields.io/badge/steps-28,000-a78bfa" alt="28000 steps">
37
  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
38
  </p>
39
 
 
49
  | **Vocabulary** | 439 tokens (ByteLevel BPE, sliced from a 32,000-target config) |
50
  | **Context** | 2,048 trained Β· evaluated on `n_ctx=512` |
51
  | **Training data** | 768M tokens β€” FineWeb CC-MAIN-2013-20, 400k documents, 8 shards |
52
+ | **Training schedule** | 28,000 steps — seven cosine cycles (0→4k→8k→12k→16k→20k→24k→28k) · Kaggle T4 |
53
+ | **Final training loss** | 22.61 @ step 28,000 (best in cycle 7: 21.88 @ step 26,150; final-step loss is batch-noisy β€” held-out PPL decides) |
54
+ | **Held-out perplexity** | **146.17** (Q4_K_M, 992 docs / 1,027,766 tokens, teacher-forced) |
55
  | **Formats in this repo** | `f32` GGUF (305 MB) Β· `Q4_K_M` GGUF (45 MB) Β· config + tokenizer |
56
  | **License** | Apache 2.0 |
57
 
 
60
  ## πŸ“ˆ Training
61
 
62
  <p align="center">
63
+ <img src="neuralai-nae1-training.png" alt="Nae1 training loss step 24000 to 28000" width="92%" />
64
  </p>
65
 
66
+ Nae1 is pretrained from random initialization in seven cosine cycles: 0 β†’ 4,000 Β· 4,000 β†’ 8,000 Β· 8,000 β†’ 12,000 Β· 12,000 β†’ 16,000 Β· 16,000 β†’ 20,000 Β· 20,000 β†’ 24,000 Β· 24,000 β†’ 28,000, each resumed from the previous checkpoint. The chart shows cycle seven β€” loss moves from ~24.1 to a best of **21.88** (step 26,150), finishing at **22.61** at step 28,000 as the learning rate anneals (batch-noisy final step; held-out PPL keeps improving: 148.84 β†’ 147.97 β†’ **146.17**).
67
 
68
  - **Data pipeline**: FineWeb parquet β†’ chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
69
+ - **Checkpoints**: every 250 steps; this release ships the final step-28000 checkpoint
70
 
71
  ---
72
 
 
87
  | step-12000 | 154.25 |
88
  | step-16000 | 151.45 |
89
  | step-20000 | 148.84 |
90
+ | step-24000 | 147.97 |
91
+ | **step-28000 β€” this release** | **146.17** |
92
 
93
+ Raw report: [`eval_step28000_q4_n512.json`](eval_step28000_q4_n512.json) β€” mean NLL 4.9848 nats/token (prior releases kept alongside: [`eval_step24000_q4_n512.json`](eval_step24000_q4_n512.json), [`eval_step20000_q4_n512.json`](eval_step20000_q4_n512.json), [`eval_step16000_q4_n512.json`](eval_step16000_q4_n512.json), [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json)). Fixed probe ("The quick brown fox", 10 tokens): 174.73 β€” the 10-token probe is noisy; held-out decides.
94
 
95
  > ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params over 768M tokens, expect exploratory, often garbled text β€” this is a research-lineage model, not a chat assistant.
96
 
 
131
  | `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB β€” for requantization / research |
132
  | `config.json` | Architecture config as trained |
133
  | `tokenizer/` | Native 439-token vocab + merges + manifest |
134
+ | `eval_step28000_q4_n512.json` | The exact evaluation report cited above |
135
+ | `eval_step24000_q4_n512.json` | Prior release (step-24000), kept for lineage |
136
  | `eval_step20000_q4_n512.json` | Prior release (step-20000), kept for lineage |
137
  | `eval_step16000_q4_n512.json` | Prior release (step-16000), kept for lineage |
138
  | `eval_step8000_q4_n512.json` | Prior release (step-8000), kept for lineage |
 
175
  title = {NeuralAI β€” Nae1},
176
  year = {2026},
177
  url = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
178
+ version = {step-28000},
179
  description = {A 100M-parameter language model pretrained from scratch on 768M FineWeb tokens}
180
  }
181
  ```
eval_step28000_q4_n512.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/artifacts/nae1_gguf/nae1-llama-Q4_K_M-step28000.gguf",
3
+ "context_size": 512,
4
+ "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
5
+ "probe": {
6
+ "text": "The quick brown fox",
7
+ "tokens": 10,
8
+ "total_nll_nats": 51.63228580836413,
9
+ "perplexity": 174.72766832853299
10
+ },
11
+ "heldout": {
12
+ "documents": 992,
13
+ "tokens": 1027766,
14
+ "total_nll_nats": 5123168.812462744,
15
+ "mean_nll_nats_per_token": 4.984761913181351,
16
+ "perplexity": 146.1687700255744
17
+ }
18
+ }
nae1-llama-Q4_K_M.gguf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:49006a2a05b822a5568474db9b11533e6d20d5de0723a73e06e64d2af7e0c980
3
  size 45762464
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e7b4181f919bdf27ef18596ebe76923a2126e07e55b32ae4e86d155c7b4f821
3
  size 45762464
nae1-llama-f32.gguf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f2d677d6eadf5beee328f3c7946e8811e31847b89c954f4aa4332a1b505b87a1
3
  size 304778816
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:60b86a70921906f1dc3d3b79e321ffd299aee6030618f7e038de83040734753a
3
  size 304778816
neuralai-nae1-banner.png CHANGED

Git LFS Details

  • SHA256: 8f9ae617b278ad6b3d6bd028b4bb50f8c498e364338f02e078ade07c66f63e2e
  • Pointer size: 131 Bytes
  • Size of remote file: 328 kB

Git LFS Details

  • SHA256: 320e34d992deee9e230ccd43c01e66b7b3fa2810ce16430cabb737c3d19555b9
  • Pointer size: 131 Bytes
  • Size of remote file: 328 kB
neuralai-nae1-banner.svg CHANGED
neuralai-nae1-comparison.png CHANGED
neuralai-nae1-comparison.svg CHANGED
neuralai-nae1-training.png CHANGED

Git LFS Details

  • SHA256: 38ce00556da46312caddb8bd69d3e1ae50686606d8883dc7adaffcb1aba8e16e
  • Pointer size: 131 Bytes
  • Size of remote file: 168 kB

Git LFS Details

  • SHA256: 19747d45327cf7104fb2fa77054e224f85277789120aec861c3199b1576809c4
  • Pointer size: 131 Bytes
  • Size of remote file: 156 kB
neuralai-nae1-training.svg CHANGED