pretrain step_100000 readme (ppl=7.4864104236873255)
Browse files
README.md
CHANGED
|
@@ -5,7 +5,7 @@ library_name: pytorch
|
|
| 5 |
---
|
| 6 |
# ETM-Backbone (attention-free ETM+SSM causal LM)
|
| 7 |
|
| 8 |
-
- **Stage**:
|
| 9 |
- **Arch**: attention-free (dilated causal conv ETM + Mamba-style SSM), no positional embeddings -> context-length unbounded.
|
| 10 |
- **Files**: model.pt (raw PyTorch state_dict), tokenizer.json (HF tokenizers, ChatML specials for SFT).
|
| 11 |
- Custom architecture - load with the repo's ETMSSMLangModel (not AutoModel). vocab/d_model/layers auto-detected from state_dict.
|
|
|
|
| 5 |
---
|
| 6 |
# ETM-Backbone (attention-free ETM+SSM causal LM)
|
| 7 |
|
| 8 |
+
- **Stage**: pretrain | **Checkpoint**: step_100000 | **val_ppl**: 7.4864104236873255
|
| 9 |
- **Arch**: attention-free (dilated causal conv ETM + Mamba-style SSM), no positional embeddings -> context-length unbounded.
|
| 10 |
- **Files**: model.pt (raw PyTorch state_dict), tokenizer.json (HF tokenizers, ChatML specials for SFT).
|
| 11 |
- Custom architecture - load with the repo's ETMSSMLangModel (not AutoModel). vocab/d_model/layers auto-detected from state_dict.
|