| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - small |
| - 30M |
| - slm |
| - micro |
| - dumb |
| --- |
| # Dumb-1.2-Preview-0618 |
|
|
| **——a new dumb model!** |
|
|
| training time: ~1 hour |
|
|
| The architecture is the same as Dumb-1.2-Exp-0616. |
|
|
| <img src="bm1.jpeg" width="100%"><img src="bmgraph.jpeg" width="87%"> |
|
|
| ## Efficiency |
|
|
| <img src="bm2.jpeg" width="87%"> |
|
|
| ## Benchmark (per training step) |
|
|
| | Benchmark | 06-18 1st (6000s) | 06-18 2nd (26000s, easymaxxed) | Dumb-1.2-Preview-0618 (3rd, balanced) | |
| | :--- | :---: | :---: | :---: | |
| | MMLU (acc) | 23.21% | 23.18% | **23.71%** | |
| | HellaSwag (acc_norm) | 26.31% | **26.67%** | 26.43% | |
| | ARC-Easy (acc_norm) | 30.39% | **32.03%** | 30.77% | |
| | PIQA (acc) | 55.98% | **55.55%** | 54.90% | |
| | SciQ (acc_norm) | 35.40% | **38.50%** | 38.00% | |
| | ARC-C (acc_norm) | **22.53%** | 20.90% | 21.76% | |
| | WinoGrande (acc) | 49.09% | **51.54%** | 51.46% | |
| | OpenBookQA (acc_norm) | 24.00% | 24.80% | **25.20%** | |
