yashmarathe commited on
Commit
459a858
Β·
verified Β·
1 Parent(s): 3c79c07

Upload Avey-D MoE 1B (1.01B total, 205M active, trained on FineWeb 1.3B tokens)

Browse files
Files changed (9) hide show
  1. README.md +95 -0
  2. config.json +27 -0
  3. configuration.py +299 -0
  4. model.safetensors +3 -0
  5. modeling_dense.py +413 -0
  6. modeling_moe.py +260 -0
  7. moe.py +500 -0
  8. tokenizer.json +0 -0
  9. tokenizer_config.json +13 -0
README.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - causal-lm
7
+ - mixture-of-experts
8
+ - attention-free
9
+ - avey
10
+ library_name: transformers
11
+ pipeline_tag: text-generation
12
+ ---
13
+
14
+ # Avey-D MoE 1B
15
+
16
+ An attention-free causal language model with Mixture-of-Experts, trained from scratch.
17
+
18
+ ## Model Details
19
+
20
+ | Property | Value |
21
+ |---|---|
22
+ | Architecture | Avey-D MoE (attention-free, interleaved Static/Dynamic layers) |
23
+ | Total Parameters | 1.01B |
24
+ | Active Parameters | 205M (per token) |
25
+ | Hidden Dimension | 640 |
26
+ | Layers | 20 (10 Static MoE + 10 Dynamic dense) |
27
+ | Experts | 32 routed + 1 shared, top-4 gating |
28
+ | Context Length | 2048 (chunk_size=256, k=3 retrieved chunks) |
29
+ | Vocabulary | 50,368 tokens |
30
+ | Training Data | FineWeb 10BT sample (~1.3B tokens seen) |
31
+ | Training Hardware | 1x AMD Instinct MI300X |
32
+ | Training Time | ~4.4 hours |
33
+ | Final Train Loss | 4.17 |
34
+ | Best Val Loss | 4.23 |
35
+ | MFU | 43.6% |
36
+ | Throughput | ~86,500 tok/s |
37
+
38
+ ## Architecture
39
+
40
+ Avey-D replaces self-attention with two types of interleaved layers:
41
+
42
+ - **Static Layers (MoE)**: Learned causal spatial projection for token mixing + MoE Enricher/Fuser (32 routed experts, top-4 gating, 1 shared expert)
43
+ - **Dynamic Layers (Dense)**: Cosine-similarity token mixing + dense Enricher/Fuser
44
+ - **CausalRanker**: Neural compression retrieving k=3 most recent preceding chunks
45
+
46
+ Expert dispatch uses batched `torch.bmm` via `StackedExperts` for efficient GPU utilization.
47
+
48
+ ## Usage
49
+
50
+ ```python
51
+ import torch
52
+ from transformers import AutoTokenizer
53
+
54
+ # Load model (requires the avey_d module)
55
+ from avey_d.modeling_moe import AveyDecoderMoEForCausalLM
56
+
57
+ model = AveyDecoderMoEForCausalLM.from_pretrained("yashmarathe/avey-d-moe-1b")
58
+ tokenizer = AutoTokenizer.from_pretrained("yashmarathe/avey-d-moe-1b")
59
+
60
+ model.eval()
61
+ device = "cuda" if torch.cuda.is_available() else "cpu"
62
+ model.to(device)
63
+
64
+ prompt = "The capital of France is"
65
+ input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)
66
+
67
+ with torch.no_grad():
68
+ output = model.generate(input_ids, max_new_tokens=100, temperature=0.8, top_k=50)
69
+
70
+ print(tokenizer.decode(output[0], skip_special_tokens=True))
71
+ ```
72
+
73
+ ## Training Configuration
74
+
75
+ ```yaml
76
+ model: d_embed=640, n_layers=20, num_experts=32, top_k=4, shared_expert=true
77
+ training: 5000 steps, batch=262144 tok/step, seq_length=2048
78
+ optimizer: AdamW, max_lr=3e-4, cosine schedule, warmup=500 steps
79
+ dataset: HuggingFaceFW/fineweb sample-10BT
80
+ ```
81
+
82
+ ## Limitations
83
+
84
+ This is a small-scale experimental model trained on limited data (~1.3B tokens). It is not intended for production use. The model may generate incoherent, incorrect, or biased text.
85
+
86
+ ## Citation
87
+
88
+ ```bibtex
89
+ @inproceedings{2026aveyb,
90
+ title={Avey-B},
91
+ author={Acharya, Devang and Hammoud, Mohammad},
92
+ booktitle={The Fourteenth International Conference on Learning Representations},
93
+ year={2026}
94
+ }
95
+ ```
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "AveyDecoderMoEForCausalLM"
4
+ ],
5
+ "aux_loss_coeff": 0.01,
6
+ "bos_token_id": null,
7
+ "chunk_size": 256,
8
+ "d_embed": 640,
9
+ "dtype": "float32",
10
+ "eos_token_id": null,
11
+ "eps": 1e-12,
12
+ "expansion_factor": 4,
13
+ "hidden_size": 640,
14
+ "k": 3,
15
+ "max_position_embeddings": 4294967296,
16
+ "model_type": "avey-d-moe",
17
+ "moe_layer_stride": 2,
18
+ "n_layers": 20,
19
+ "num_experts": 32,
20
+ "pad_token_id": null,
21
+ "tie_word_embeddings": true,
22
+ "top_k": 4,
23
+ "transformers_version": "5.2.0",
24
+ "use_shared_expert": true,
25
+ "vocab_size": 50368,
26
+ "z_loss_coeff": 0.001
27
+ }
configuration.py ADDED
@@ -0,0 +1,299 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+
4
+ class AveyDecoderConfig(PretrainedConfig):
5
+ """
6
+ Configuration for the Avey Decoder (causal, autoregressive) model.
7
+
8
+ This is the decoder counterpart to Avey-B (encoder), analogous to the
9
+ GPT vs BERT relationship in the Transformer family.
10
+
11
+ The architecture follows the original Avey design with:
12
+ - Decoupled static/dynamic layers (interleaved S→D pattern)
13
+ - Unidirectional (causal) Ranker that only attends to preceding chunks
14
+ - Neural compression via learned down-projection
15
+ - Causal masking within chunks for autoregressive generation
16
+
17
+ Default config targets ~1B parameters:
18
+ - d_embed=2048, n_layers=32, chunk_size=256, k=3
19
+ - Estimated: ~1.04B parameters
20
+ """
21
+ model_type = "avey-d"
22
+
23
+ def __init__(
24
+ self,
25
+ vocab_size: int = 50368,
26
+ context_len: int = 4_294_967_296, # effectively unlimited
27
+ d_embed: int = 2048,
28
+ n_layers: int = 32,
29
+ chunk_size: int = 256,
30
+ k: int = 3,
31
+ expansion_factor: int = 4,
32
+ eps: float = 1e-12,
33
+ tie_word_embeddings: bool = True,
34
+ # Generation parameters
35
+ bos_token_id: int = None,
36
+ eos_token_id: int = None,
37
+ pad_token_id: int = None,
38
+ **kwargs,
39
+ ):
40
+ self.vocab_size = vocab_size
41
+ self.d_embed = d_embed
42
+ self.n_layers = n_layers
43
+ self.chunk_size = chunk_size
44
+ self.k = k
45
+ self.expansion_factor = expansion_factor
46
+ self.eps = eps
47
+ self.tie_word_embeddings = tie_word_embeddings
48
+
49
+ # For compatibility
50
+ self.max_position_embeddings = context_len
51
+ self.hidden_size = d_embed
52
+
53
+ super().__init__(
54
+ bos_token_id=bos_token_id,
55
+ eos_token_id=eos_token_id,
56
+ pad_token_id=pad_token_id,
57
+ tie_word_embeddings=tie_word_embeddings,
58
+ **kwargs,
59
+ )
60
+ # =========================================================================
61
+ # MoE Configuration
62
+ # =========================================================================
63
+
64
+
65
+ class AveyDecoderMoEConfig(PretrainedConfig):
66
+ """
67
+ Configuration for Avey-D with Mixture of Experts.
68
+
69
+ Extends the dense AveyDecoderConfig with MoE-specific parameters.
70
+ The MoE replaces the Enricher+Fuser in Static layers; the contextualizer
71
+ (spatial_proj) and Dynamic layers remain dense/shared.
72
+ """
73
+
74
+ model_type = "avey-d-moe"
75
+
76
+ def __init__(
77
+ self,
78
+ # ---- Base architecture (same as dense) ----
79
+ vocab_size: int = 50368,
80
+ context_len: int = 4_294_967_296,
81
+ d_embed: int = 2048,
82
+ n_layers: int = 32,
83
+ chunk_size: int = 256,
84
+ k: int = 3,
85
+ expansion_factor: int = 4,
86
+ eps: float = 1e-12,
87
+ tie_word_embeddings: bool = True,
88
+ # ---- MoE-specific ----
89
+ num_experts: int = 32,
90
+ top_k: int = 2,
91
+ use_shared_expert: bool = True,
92
+ moe_layer_stride: int = 2, # MoE every N-th layer (2 = static only)
93
+ aux_loss_coeff: float = 0.01,
94
+ z_loss_coeff: float = 0.001,
95
+ # ---- Generation ----
96
+ bos_token_id: int = None,
97
+ eos_token_id: int = None,
98
+ pad_token_id: int = None,
99
+ **kwargs,
100
+ ):
101
+ self.vocab_size = vocab_size
102
+ self.d_embed = d_embed
103
+ self.n_layers = n_layers
104
+ self.chunk_size = chunk_size
105
+ self.k = k
106
+ self.expansion_factor = expansion_factor
107
+ self.eps = eps
108
+ self.tie_word_embeddings = tie_word_embeddings
109
+
110
+ # MoE
111
+ self.num_experts = num_experts
112
+ self.top_k = top_k
113
+ self.use_shared_expert = use_shared_expert
114
+ self.moe_layer_stride = moe_layer_stride
115
+ self.aux_loss_coeff = aux_loss_coeff
116
+ self.z_loss_coeff = z_loss_coeff
117
+
118
+ # Compatibility
119
+ self.max_position_embeddings = context_len
120
+ self.hidden_size = d_embed
121
+
122
+ super().__init__(
123
+ bos_token_id=bos_token_id,
124
+ eos_token_id=eos_token_id,
125
+ pad_token_id=pad_token_id,
126
+ tie_word_embeddings=tie_word_embeddings,
127
+ **kwargs,
128
+ )
129
+
130
+
131
+ # =========================================================================
132
+ # MoE Model Presets β€” all sizes from 3B to 100B+ total parameters
133
+ # =========================================================================
134
+ #
135
+ # Naming: avey-d-moe-{total_params}
136
+ # Active params β‰ˆ d_embedΒ² Γ— n_layers Γ— (shared + top_k) / num_experts
137
+ #
138
+ # Key design choices:
139
+ # - Fine-grained experts (many small > few large) β€” DeepSeek-V3 finding
140
+ # - 1 shared expert always ON β€” stabilizes training
141
+ # - MoE on static layers only (stride=2) β€” dynamic layers have no
142
+ # learned contextualizer weights to specialize
143
+ # =========================================================================
144
+
145
+ MOE_CONFIGS = {
146
+ # ---- MoE-3B: Target 3B total / 1B active ----
147
+ # Total: ~2.96B | Active: ~1.03B | Compute β‰ˆ dense 1B
148
+ # 16 routed experts + 1 shared, top-4, d=1280, 28 layers
149
+ # Good for: Direct comparison with dense 1B at same active FLOPs
150
+ # VRAM: ~30 GB train | Recommended tokens: 10-50B
151
+ "moe-3b": AveyDecoderMoEConfig(
152
+ vocab_size=50368,
153
+ d_embed=1280,
154
+ n_layers=28,
155
+ chunk_size=256,
156
+ k=3,
157
+ expansion_factor=4,
158
+ num_experts=16,
159
+ top_k=4,
160
+ use_shared_expert=True,
161
+ moe_layer_stride=2,
162
+ aux_loss_coeff=0.01,
163
+ z_loss_coeff=0.001,
164
+ ),
165
+
166
+ # ---- MoE-16B: Mid-scale MoE ----
167
+ # Total: ~16B | Active: ~2.0B | Compute β‰ˆ dense 2B
168
+ # 32 fine-grained routed experts + 1 shared, top-2
169
+ # Good for: Validating MoE dynamics, single MI300X
170
+ # VRAM: ~80 GB train | Recommended tokens: 25-100B
171
+ "moe-16b": AveyDecoderMoEConfig(
172
+ vocab_size=50368,
173
+ d_embed=2048,
174
+ n_layers=32,
175
+ chunk_size=256,
176
+ k=3,
177
+ expansion_factor=4,
178
+ num_experts=32,
179
+ top_k=2,
180
+ use_shared_expert=True,
181
+ moe_layer_stride=2,
182
+ aux_loss_coeff=0.01,
183
+ z_loss_coeff=0.001,
184
+ ),
185
+
186
+ # ---- MoE-31B: Mid-scale MoE ----
187
+ # Total: ~31B | Active: ~3.9B | Compute β‰ˆ dense 4B
188
+ # 64 fine-grained experts, top-6 β†’ rich specialization
189
+ # Good for: Production model, competitive with dense 7B Transformers
190
+ # VRAM: ~150 GB train | 1-2Γ— MI300X | Recommended tokens: 100-500B
191
+ "moe-31b": AveyDecoderMoEConfig(
192
+ vocab_size=50368,
193
+ d_embed=2048,
194
+ n_layers=32,
195
+ chunk_size=256,
196
+ k=3,
197
+ expansion_factor=4,
198
+ num_experts=64,
199
+ top_k=6,
200
+ use_shared_expert=True,
201
+ moe_layer_stride=2,
202
+ aux_loss_coeff=0.01,
203
+ z_loss_coeff=0.001,
204
+ ),
205
+
206
+ # ---- MoE-87B: Large-scale MoE ----
207
+ # Total: ~87B | Active: ~10.7B | Compute β‰ˆ dense 11B
208
+ # 64 experts @ wider d_embed=3072, 40 layers, top-6
209
+ # Good for: Frontier quality, competitive with dense 30B+ Transformers
210
+ # VRAM: ~400 GB train | 4-8Γ— MI300X with expert parallelism
211
+ # Recommended tokens: 200B-1T
212
+ "moe-87b": AveyDecoderMoEConfig(
213
+ vocab_size=50368,
214
+ d_embed=3072,
215
+ n_layers=40,
216
+ chunk_size=256,
217
+ k=3,
218
+ expansion_factor=4,
219
+ num_experts=64,
220
+ top_k=6,
221
+ use_shared_expert=True,
222
+ moe_layer_stride=2,
223
+ aux_loss_coeff=0.01,
224
+ z_loss_coeff=0.001,
225
+ ),
226
+
227
+ # ---- MoE-367B: Frontier MoE ----
228
+ # Total: ~367B | Active: ~28B | Compute β‰ˆ dense 28B
229
+ # 128 experts @ d_embed=4096, 48 layers, top-8
230
+ # Good for: Frontier-scale, matches dense 70B+ quality
231
+ # VRAM: 1.5+ TB train | 32+ MI300X, full expert+data+pipeline parallelism
232
+ # Recommended tokens: 1T+
233
+ "moe-367b": AveyDecoderMoEConfig(
234
+ vocab_size=50368,
235
+ d_embed=4096,
236
+ n_layers=48,
237
+ chunk_size=256,
238
+ k=3,
239
+ expansion_factor=4,
240
+ num_experts=128,
241
+ top_k=8,
242
+ use_shared_expert=True,
243
+ moe_layer_stride=2,
244
+ aux_loss_coeff=0.005,
245
+ z_loss_coeff=0.001,
246
+ ),
247
+ }
248
+
249
+ # =========================================================================
250
+ # MoE Training Presets
251
+ # =========================================================================
252
+
253
+ MOE_TRAINING_PRESETS = {
254
+ "moe-3b": {
255
+ "max_lr": 3e-4,
256
+ "total_batch_tokens": 2**18, # 262K (same as dense 1B baseline)
257
+ "seq_length": 2048,
258
+ "warmup_steps": 500,
259
+ "weight_decay": 0.1,
260
+ "grad_norm_clip": 1.0,
261
+ "tokens_target": "10B-50B",
262
+ },
263
+ "moe-16b": {
264
+ "max_lr": 3e-4,
265
+ "total_batch_tokens": 2**19, # 524K
266
+ "seq_length": 2048,
267
+ "warmup_steps": 2000,
268
+ "weight_decay": 0.1,
269
+ "grad_norm_clip": 1.0,
270
+ "tokens_target": "25B-100B",
271
+ },
272
+ "moe-31b": {
273
+ "max_lr": 2e-4,
274
+ "total_batch_tokens": 2**20, # ~1M
275
+ "seq_length": 2048,
276
+ "warmup_steps": 2000,
277
+ "weight_decay": 0.1,
278
+ "grad_norm_clip": 1.0,
279
+ "tokens_target": "100B-500B",
280
+ },
281
+ "moe-87b": {
282
+ "max_lr": 1.5e-4,
283
+ "total_batch_tokens": 2**21, # ~2M
284
+ "seq_length": 4096,
285
+ "warmup_steps": 3000,
286
+ "weight_decay": 0.1,
287
+ "grad_norm_clip": 1.0,
288
+ "tokens_target": "200B-1T",
289
+ },
290
+ "moe-367b": {
291
+ "max_lr": 8e-5,
292
+ "total_batch_tokens": 2**23, # ~8M
293
+ "seq_length": 8192,
294
+ "warmup_steps": 3000,
295
+ "weight_decay": 0.1,
296
+ "grad_norm_clip": 1.0,
297
+ "tokens_target": "1T+",
298
+ },
299
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0311f4de529761aba70449cf211d8253ce179f7015327326fdadaeca07c3f55
3
+ size 4036986888
modeling_dense.py ADDED
@@ -0,0 +1,413 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Avey-D: Causal (Decoder) variant of the Avey architecture for autoregressive
3
+ text generation.
4
+
5
+ Converts the bidirectional Avey-B encoder into a unidirectional decoder by:
6
+ 1. Applying causal (lower-triangular) masking to the StaticLayer spatial projection
7
+ 2. Applying causal masking to the DynamicLayer cosine-similarity matrix
8
+ 3. Making the Ranker unidirectional (only attends to preceding chunks)
9
+ 4. Adding a CausalLM head for next-token prediction
10
+
11
+ Reference: "Avey-B" (Acharya & Hammoud, ICLR 2026, arXiv:2602.15814)
12
+ """
13
+
14
+ import torch
15
+ import torch.nn.functional as F
16
+ from torch import nn
17
+ from transformers import PreTrainedModel
18
+ from transformers.modeling_outputs import (
19
+ BaseModelOutput,
20
+ CausalLMOutput,
21
+ )
22
+
23
+ from .configuration import AveyDecoderConfig
24
+
25
+
26
+ # ---------------------------------------------------------------------------
27
+ # Causal Static Layer
28
+ # ---------------------------------------------------------------------------
29
+ class CausalStaticLayer(nn.Module):
30
+ """
31
+ Static token-mixing layer with causal masking.
32
+
33
+ The learned spatial_proj weight matrix is masked to be lower-triangular
34
+ at forward time, so position i only mixes with positions <= i.
35
+ """
36
+
37
+ def __init__(self, config: AveyDecoderConfig):
38
+ super().__init__()
39
+ self.norm = nn.RMSNorm(config.d_embed, eps=config.eps)
40
+ self.enricher = nn.Linear(config.d_embed, config.d_embed * config.expansion_factor)
41
+
42
+ proj_size = config.chunk_size
43
+ self.spatial_proj = nn.Parameter(torch.empty(proj_size, proj_size))
44
+ nn.init.xavier_normal_(self.spatial_proj)
45
+
46
+ # Fuser input: half of enricher output (gated) + other half (bypass)
47
+ enricher_out = config.d_embed * config.expansion_factor
48
+ half = enricher_out // 2
49
+ quarter = half // 2 # gated portion after chunk(2)
50
+ fuser_in = quarter + half # gated_output + bypass
51
+ self.fuser = nn.Linear(fuser_in, config.d_embed)
52
+ self.alpha = nn.Parameter(torch.tensor(1.0))
53
+
54
+ # Register causal mask buffer (lower-triangular)
55
+ causal = torch.tril(torch.ones(proj_size, proj_size))
56
+ self.register_buffer("causal_mask", causal, persistent=False)
57
+
58
+ # @torch.compile # Disabled: inductor OOMs on ROCm MI300X at large batch
59
+ def forward(self, x):
60
+ _, T, _ = x.shape
61
+ res = x
62
+ x = self.norm(x)
63
+ x = self.enricher(x)
64
+ x = F.gelu(x)
65
+ x, bypass = x.chunk(2, dim=-1)
66
+
67
+ x, gate = x.chunk(2, dim=-1)
68
+
69
+ # Apply causal masking to spatial projection
70
+ masked_proj = self.spatial_proj[:T, :T] * self.causal_mask[:T, :T]
71
+ x = masked_proj @ x
72
+ x = gate * x
73
+
74
+ x = torch.cat([x, bypass], dim=-1)
75
+ x = self.fuser(x)
76
+ return x + (self.alpha * res)
77
+
78
+
79
+ # ---------------------------------------------------------------------------
80
+ # Causal Dynamic Layer
81
+ # ---------------------------------------------------------------------------
82
+ class CausalDynamicLayer(nn.Module):
83
+ """
84
+ Dynamic (input-dependent) token-mixing layer with causal masking.
85
+
86
+ Computes cosine similarity between all token pairs, then masks the
87
+ similarity matrix to be lower-triangular so each position only
88
+ attends to itself and preceding positions.
89
+ """
90
+
91
+ def __init__(self, config: AveyDecoderConfig):
92
+ super().__init__()
93
+ self.norm = nn.RMSNorm(config.d_embed, eps=config.eps)
94
+ self.enricher = nn.Linear(config.d_embed, config.d_embed * config.expansion_factor)
95
+
96
+ enricher_out = config.d_embed * config.expansion_factor
97
+ half = enricher_out // 2
98
+ quarter = half // 2
99
+ fuser_in = quarter + half
100
+ self.fuser = nn.Linear(fuser_in, config.d_embed)
101
+ self.alpha = nn.Parameter(torch.tensor(1.0))
102
+
103
+ # Register causal mask buffer
104
+ proj_size = config.chunk_size
105
+ causal = torch.tril(torch.ones(proj_size, proj_size))
106
+ self.register_buffer("causal_mask", causal, persistent=False)
107
+
108
+ # @torch.compile # Disabled: inductor OOMs on ROCm MI300X at large batch
109
+ def forward(self, x):
110
+ _, T, _ = x.shape
111
+ res = x
112
+ x = self.norm(x)
113
+ x = self.enricher(x)
114
+ x = F.gelu(x)
115
+ x, bypass = x.chunk(2, dim=-1)
116
+
117
+ x, gate = x.chunk(2, dim=-1)
118
+ x_norm = F.normalize(x, p=2, dim=-1)
119
+ sim_scores = x_norm @ x_norm.mT
120
+
121
+ # Apply causal mask: zero out future positions, then normalize
122
+ sim_scores = sim_scores * self.causal_mask[:T, :T]
123
+ x = F.normalize(sim_scores, p=1, dim=-1) @ x
124
+ x = gate * x
125
+
126
+ x = torch.cat([x, bypass], dim=-1)
127
+ x = self.fuser(x)
128
+ return x + (self.alpha * res)
129
+
130
+
131
+ # ---------------------------------------------------------------------------
132
+ # Causal Ranker (unidirectional - only looks at preceding chunks)
133
+ # ---------------------------------------------------------------------------
134
+ class CausalRanker(nn.Module):
135
+ """
136
+ Neural compression module for long-context processing.
137
+
138
+ Unlike the bidirectional Avey-B Ranker, this only retrieves context
139
+ from *preceding* chunks (causal/unidirectional), making it suitable
140
+ for autoregressive generation.
141
+
142
+ For each chunk i, it:
143
+ 1. Scores all preceding chunks (0..i-1) using MaxSim
144
+ 2. Selects top-k most relevant preceding chunks
145
+ 3. Compresses selected chunks + current chunk via learned down-projection
146
+ 4. Adds compressed context back via residual connection
147
+ """
148
+
149
+ def __init__(self, config: AveyDecoderConfig):
150
+ super().__init__()
151
+ self.chunk_size = config.chunk_size
152
+ self.k = config.k # number of retrieved past chunks (NOT +1)
153
+ self.context_len = self.k * config.chunk_size # past context only
154
+ self.eps = config.eps
155
+ # down_proj compresses ONLY the past context (k * cs tokens β†’ cs tokens)
156
+ # The current chunk is added via residual, never mixed through down_proj.
157
+ # This preserves causality: tokens in the current chunk cannot peek at
158
+ # future positions within the same chunk through the projection.
159
+ self.down_proj = nn.Parameter(
160
+ torch.empty(self.chunk_size, self.context_len)
161
+ )
162
+ nn.init.xavier_normal_(self.down_proj)
163
+
164
+ def preprocess(self, x):
165
+ B, T, E = x.shape
166
+ cs = self.chunk_size
167
+ ctx_len = self.context_len # k * cs
168
+
169
+ padded = False
170
+ orig_T = T
171
+ if T % cs != 0:
172
+ pad_len = cs - (T % cs)
173
+ pad = torch.zeros(B, pad_len, E, device=x.device, dtype=x.dtype)
174
+ x = torch.cat([x, pad], dim=1)
175
+ T += pad_len
176
+ padded = True
177
+
178
+ N = T // cs
179
+ x_chunks = x.view(B, N, cs, E)
180
+
181
+ context_compressed = []
182
+ for i in range(N):
183
+ # CAUSAL: retrieve the k most recent preceding chunks.
184
+ # We use a recency-based strategy (NOT content-based scoring)
185
+ # to avoid causality violations β€” scoring with the current chunk
186
+ # would let future tokens within the chunk influence which past
187
+ # context is retrieved for earlier tokens in the same chunk.
188
+ past_ctx = self._get_recent_past(x_chunks, i) # (B, ≀k*cs, E)
189
+
190
+ # Pad to fixed context length
191
+ cur_len = past_ctx.size(1)
192
+ if cur_len < ctx_len:
193
+ pad2 = torch.zeros(B, ctx_len - cur_len, E, device=x.device, dtype=x.dtype)
194
+ past_ctx = torch.cat([pad2, past_ctx], dim=1)
195
+ else:
196
+ past_ctx = past_ctx[:, -ctx_len:]
197
+
198
+ context_compressed.append(past_ctx)
199
+
200
+ ctx = torch.stack(context_compressed, dim=1) # (B, N, ctx_len, E)
201
+ # Compress past context β†’ (B, N, cs, E), then add current chunk as residual.
202
+ # down_proj: (cs, ctx_len) @ (ctx_len, E) β†’ (cs, E) per chunk.
203
+ # Current chunk passes through UNTOUCHED via residual β€” no future leakage.
204
+ h = (self.down_proj @ ctx) + x_chunks
205
+ h = h.view(B * N, cs, E)
206
+
207
+ state = {"B": B, "N": N, "orig_T": orig_T, "padded": padded}
208
+ return h, state
209
+
210
+ def contract(self, h, st):
211
+ B = st["B"]
212
+ cs = self.chunk_size
213
+ N = st["N"]
214
+ padded = st["padded"]
215
+ orig_T = st["orig_T"]
216
+
217
+ E = h.size(-1)
218
+ final_chunks = h.view(B, N, cs, E)
219
+ out = final_chunks.reshape(B, N * cs, E)
220
+
221
+ if padded:
222
+ out = out[:, :orig_T, :]
223
+
224
+ return out
225
+
226
+ def _get_recent_past(self, x_chunks, chunk_idx):
227
+ """
228
+ Retrieve the k most recent preceding chunks for chunk `chunk_idx`.
229
+
230
+ Uses a simple recency-based strategy: always select the k chunks
231
+ immediately before the current chunk. This is strictly causal β€”
232
+ the retrieval decision does not depend on the current chunk's
233
+ content at all, so no within-chunk information can leak.
234
+
235
+ Args:
236
+ x_chunks: (B, N, cs, E) β€” all chunks.
237
+ chunk_idx: int β€” index of the current chunk.
238
+
239
+ Returns:
240
+ (B, num_selected * cs, E) β€” flattened past context tokens.
241
+ """
242
+ if chunk_idx == 0:
243
+ # First chunk β€” no past context
244
+ B, _, cs, E = x_chunks.shape
245
+ return torch.zeros(B, 0, E, device=x_chunks.device, dtype=x_chunks.dtype)
246
+
247
+ # Take the k most recent preceding chunks
248
+ start = max(0, chunk_idx - self.k)
249
+ past = x_chunks[:, start:chunk_idx] # (B, num_past, cs, E)
250
+
251
+ B, num_past, cs, E = past.shape
252
+ return past.reshape(B, num_past * cs, E)
253
+
254
+
255
+ # ---------------------------------------------------------------------------
256
+ # Base Model
257
+ # ---------------------------------------------------------------------------
258
+ class AveyDecoderPreTrainedModel(PreTrainedModel):
259
+ config_class = AveyDecoderConfig
260
+
261
+ def _init_weights(self, module):
262
+ if isinstance(module, nn.Linear):
263
+ nn.init.xavier_normal_(module.weight)
264
+ if module.bias is not None:
265
+ module.bias.data.zero_()
266
+ elif isinstance(module, nn.Embedding):
267
+ nn.init.xavier_normal_(module.weight)
268
+ if module.padding_idx is not None:
269
+ module.weight.data[module.padding_idx].zero_()
270
+
271
+
272
+ class AveyDecoderModel(AveyDecoderPreTrainedModel):
273
+ """
274
+ Avey Decoder: attention-free causal language model backbone.
275
+
276
+ Architecture:
277
+ - Token embedding (no positional encoding - position info comes from
278
+ the static layer's causal spatial projection)
279
+ - Causal Ranker for long-context neural compression
280
+ - Interleaved CausalStatic → CausalDynamic layers (S→D pattern)
281
+ - Final RMSNorm
282
+ """
283
+
284
+ def __init__(self, config: AveyDecoderConfig):
285
+ super().__init__(config)
286
+ self.chunk_size = config.chunk_size
287
+ self.embed = nn.Embedding(config.vocab_size, config.d_embed)
288
+ self.layers = nn.ModuleList(
289
+ [
290
+ CausalDynamicLayer(config) if (i + 1) % 2 == 0 else CausalStaticLayer(config)
291
+ for i in range(config.n_layers)
292
+ ]
293
+ )
294
+ self.ranker = CausalRanker(config)
295
+ self.final_norm = nn.RMSNorm(config.d_embed, eps=config.eps)
296
+ self.apply(self._init_weights)
297
+
298
+ def _get_hidden(self, input_ids):
299
+ x = self.embed(input_ids)
300
+ x, state = self.ranker.preprocess(x)
301
+ for layer in self.layers:
302
+ x = layer(x)
303
+ x = self.ranker.contract(x, state)
304
+ x = self.final_norm(x)
305
+ return x
306
+
307
+ def forward(self, input_ids, **kwargs):
308
+ x = self._get_hidden(input_ids)
309
+ return BaseModelOutput(last_hidden_state=x)
310
+
311
+
312
+ class AveyDecoderForCausalLM(AveyDecoderModel):
313
+ """
314
+ Avey Decoder with a causal language modeling head.
315
+
316
+ Uses weight-tied output projection (shares weights with the token
317
+ embedding matrix) for parameter efficiency.
318
+
319
+ Loss is computed as standard next-token prediction:
320
+ - Input: [t0, t1, t2, ..., tN-1]
321
+ - Target: [t1, t2, t3, ..., tN]
322
+ """
323
+
324
+ def __init__(self, config: AveyDecoderConfig):
325
+ super().__init__(config)
326
+ # Optional untied head
327
+ if not config.tie_word_embeddings:
328
+ self.lm_head = nn.Linear(config.d_embed, config.vocab_size, bias=False)
329
+ self.apply(self._init_weights)
330
+
331
+ def forward(self, input_ids, labels=None, **kwargs):
332
+ x = self._get_hidden(input_ids)
333
+
334
+ # Compute logits (weight-tied or separate head)
335
+ if self.config.tie_word_embeddings:
336
+ logits = F.linear(x, self.embed.weight)
337
+ else:
338
+ logits = self.lm_head(x)
339
+
340
+ loss = None
341
+ if labels is not None:
342
+ # Labels are expected to be pre-shifted by the dataloader:
343
+ # input_ids = [t0, t1, t2, ..., tN-1]
344
+ # labels = [t1, t2, t3, ..., tN ]
345
+ # So logits[i] should predict labels[i] directly β€” no internal shift.
346
+ loss = F.cross_entropy(
347
+ logits.view(-1, logits.size(-1)),
348
+ labels.view(-1),
349
+ ignore_index=-100,
350
+ )
351
+
352
+ return CausalLMOutput(logits=logits, loss=loss)
353
+
354
+ @torch.no_grad()
355
+ def generate(
356
+ self,
357
+ input_ids,
358
+ max_new_tokens=256,
359
+ temperature=1.0,
360
+ top_k=50,
361
+ top_p=0.9,
362
+ eos_token_id=None,
363
+ ):
364
+ """
365
+ Simple autoregressive generation with top-k/top-p sampling.
366
+
367
+ Args:
368
+ input_ids: (B, T) starting token IDs
369
+ max_new_tokens: maximum number of tokens to generate
370
+ temperature: sampling temperature (1.0 = neutral)
371
+ top_k: keep only top-k logits before sampling
372
+ top_p: nucleus sampling threshold
373
+ eos_token_id: stop generation when this token is produced
374
+
375
+ Returns:
376
+ (B, T + max_new_tokens) tensor of token IDs
377
+ """
378
+ self.eval()
379
+ for _ in range(max_new_tokens):
380
+ # Forward pass on full sequence
381
+ x = self._get_hidden(input_ids)
382
+ logits = F.linear(x, self.embed.weight) if self.config.tie_word_embeddings else self.lm_head(x)
383
+
384
+ # Get logits for the last position
385
+ logits = logits[:, -1, :] / temperature
386
+
387
+ # Top-k filtering
388
+ if top_k is not None and top_k > 0:
389
+ v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
390
+ logits[logits < v[:, [-1]]] = float("-inf")
391
+
392
+ # Top-p (nucleus) filtering
393
+ if top_p is not None and top_p < 1.0:
394
+ sorted_logits, sorted_indices = torch.sort(logits, descending=True)
395
+ cumulative_probs = torch.cumsum(
396
+ F.softmax(sorted_logits, dim=-1), dim=-1
397
+ )
398
+ sorted_indices_to_remove = cumulative_probs > top_p
399
+ sorted_indices_to_remove[:, 1:] = sorted_indices_to_remove[:, :-1].clone()
400
+ sorted_indices_to_remove[:, 0] = False
401
+ indices_to_remove = sorted_indices_to_remove.scatter(
402
+ 1, sorted_indices, sorted_indices_to_remove
403
+ )
404
+ logits[indices_to_remove] = float("-inf")
405
+
406
+ probs = F.softmax(logits, dim=-1)
407
+ next_token = torch.multinomial(probs, num_samples=1)
408
+ input_ids = torch.cat([input_ids, next_token], dim=1)
409
+
410
+ if eos_token_id is not None and (next_token == eos_token_id).all():
411
+ break
412
+
413
+ return input_ids
modeling_moe.py ADDED
@@ -0,0 +1,260 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Avey-D MoE: Mixture-of-Experts variant of the Avey decoder.
3
+
4
+ Architecture:
5
+ - Interleaved CausalStaticMoELayer (odd) β†’ CausalDynamicLayer (even)
6
+ - MoE replaces the Enricher+Fuser in Static layers
7
+ - Dynamic layers remain fully dense (cosine-sim contextualizer has no weights)
8
+ - 1 shared expert + N routed experts per MoE layer (DeepSeek-V3 style)
9
+ - CausalRanker handles long-context compression (same as dense)
10
+
11
+ The contextualizer (spatial_proj) is SHARED across all experts β€” only the
12
+ per-token Enricher+Fuser computation is expert-ified. This means:
13
+ - Cross-token mixing is identical to the dense model
14
+ - Only per-token transformation is specialized per-expert
15
+ - Parameter efficiency: 3-8Γ— total params for same FLOPs as dense
16
+ """
17
+
18
+ import torch
19
+ import torch.nn.functional as F
20
+ from torch import nn
21
+ from transformers import PreTrainedModel
22
+ from transformers.modeling_outputs import BaseModelOutput, CausalLMOutput
23
+
24
+ from .configuration import AveyDecoderMoEConfig
25
+ from .moe import MoEBlock, MoEOutput
26
+ from .modeling_dense import CausalDynamicLayer, CausalRanker
27
+
28
+
29
+ # ---------------------------------------------------------------------------
30
+ # CausalStaticMoELayer β€” Static layer with MoE Enricher+Fuser
31
+ # ---------------------------------------------------------------------------
32
+ class CausalStaticMoELayer(nn.Module):
33
+ """
34
+ Static token-mixing layer where the Enricher+Fuser is replaced by
35
+ a Mixture of Experts block.
36
+
37
+ Flow:
38
+ x β†’ RMSNorm β†’ MoEBlock(enricher+fuser per expert) β†’
39
+ spatial_proj (causal, shared) β†’ gating β†’ fuse β†’ + Ξ±Β·residual
40
+
41
+ The MoE block handles the expand/contract (d β†’ 4d β†’ d) computation.
42
+ The spatial_proj (cross-token mixing) remains shared β€” it's the same
43
+ learned causal weight matrix for all experts.
44
+ """
45
+
46
+ def __init__(self, config: AveyDecoderMoEConfig):
47
+ super().__init__()
48
+ self.norm = nn.RMSNorm(config.d_embed, eps=config.eps)
49
+
50
+ # Spatial projection β€” SHARED across all experts
51
+ proj_size = config.chunk_size
52
+ self.spatial_proj = nn.Parameter(torch.empty(proj_size, proj_size))
53
+ nn.init.xavier_normal_(self.spatial_proj)
54
+
55
+ # Causal mask
56
+ causal = torch.tril(torch.ones(proj_size, proj_size))
57
+ self.register_buffer("causal_mask", causal, persistent=False)
58
+
59
+ # MoE block replaces the Enricher+Fuser
60
+ self.moe = MoEBlock(
61
+ d_embed=config.d_embed,
62
+ num_experts=config.num_experts,
63
+ top_k=config.top_k,
64
+ expansion_factor=config.expansion_factor,
65
+ use_shared_expert=config.use_shared_expert,
66
+ aux_loss_coeff=config.aux_loss_coeff,
67
+ z_loss_coeff=config.z_loss_coeff,
68
+ )
69
+
70
+ self.alpha = nn.Parameter(torch.tensor(1.0))
71
+
72
+ def forward(self, x: torch.Tensor) -> tuple:
73
+ """
74
+ Args:
75
+ x: (B, T, D)
76
+ Returns:
77
+ (output, moe_output) β€” output tensor + MoE metadata for aux loss
78
+ """
79
+ _, T, _ = x.shape
80
+ res = x
81
+ x = self.norm(x)
82
+
83
+ # Apply shared causal spatial projection between expert phase-1 and phase-3
84
+ masked_proj = self.spatial_proj[:T, :T] * self.causal_mask[:T, :T]
85
+ moe_out: MoEOutput = self.moe.forward_static(x, masked_proj)
86
+ x = moe_out.hidden_states
87
+
88
+ out = x + (self.alpha * res)
89
+ return out, moe_out
90
+
91
+
92
+ # ---------------------------------------------------------------------------
93
+ # Wrapper to make DynamicLayer return a compatible tuple
94
+ # ---------------------------------------------------------------------------
95
+ class CausalDynamicLayerWrapper(nn.Module):
96
+ """Wraps CausalDynamicLayer to return (output, None) for uniform interface."""
97
+
98
+ def __init__(self, config: AveyDecoderMoEConfig):
99
+ super().__init__()
100
+ # Reuse the dense dynamic layer β€” no MoE here
101
+ # We need to create a compatible config for the dense layer
102
+ self.layer = CausalDynamicLayer(config)
103
+
104
+ def forward(self, x: torch.Tensor) -> tuple:
105
+ out = self.layer(x)
106
+ return out, None # No MoE output for dynamic layers
107
+
108
+
109
+ # ---------------------------------------------------------------------------
110
+ # Full MoE Model
111
+ # ---------------------------------------------------------------------------
112
+ class AveyDecoderMoEPreTrainedModel(PreTrainedModel):
113
+ config_class = AveyDecoderMoEConfig
114
+
115
+ def _init_weights(self, module):
116
+ if isinstance(module, nn.Linear):
117
+ nn.init.xavier_normal_(module.weight)
118
+ if module.bias is not None:
119
+ module.bias.data.zero_()
120
+ elif isinstance(module, nn.Embedding):
121
+ nn.init.xavier_normal_(module.weight)
122
+ if module.padding_idx is not None:
123
+ module.weight.data[module.padding_idx].zero_()
124
+
125
+
126
+ class AveyDecoderMoEModel(AveyDecoderMoEPreTrainedModel):
127
+ """
128
+ Avey-D MoE backbone: attention-free causal LM with Mixture of Experts.
129
+
130
+ Layer pattern (interleaved, same as dense):
131
+ Layer 0 (Static + MoE) β†’ Layer 1 (Dynamic, dense) β†’
132
+ Layer 2 (Static + MoE) β†’ Layer 3 (Dynamic, dense) β†’ ...
133
+
134
+ The Ranker is identical to the dense model β€” chunk-based neural
135
+ compression with unidirectional (causal) context retrieval.
136
+ """
137
+
138
+ def __init__(self, config: AveyDecoderMoEConfig):
139
+ super().__init__(config)
140
+ self.chunk_size = config.chunk_size
141
+ self.embed = nn.Embedding(config.vocab_size, config.d_embed)
142
+
143
+ # Build interleaved layers: Static+MoE β†’ Dynamic (dense)
144
+ layers = []
145
+ for i in range(config.n_layers):
146
+ if (i + 1) % 2 == 0:
147
+ # Even-indexed in 1-based β†’ Dynamic (dense)
148
+ layers.append(CausalDynamicLayerWrapper(config))
149
+ else:
150
+ # Odd-indexed in 1-based β†’ Static + MoE
151
+ layers.append(CausalStaticMoELayer(config))
152
+ self.layers = nn.ModuleList(layers)
153
+
154
+ self.ranker = CausalRanker(config)
155
+ self.final_norm = nn.RMSNorm(config.d_embed, eps=config.eps)
156
+ self.apply(self._init_weights)
157
+
158
+ def _get_hidden(self, input_ids):
159
+ x = self.embed(input_ids)
160
+ x, state = self.ranker.preprocess(x)
161
+
162
+ total_aux_loss = 0.0
163
+ all_expert_counts = []
164
+
165
+ for layer in self.layers:
166
+ x, moe_out = layer(x)
167
+ if moe_out is not None:
168
+ total_aux_loss = total_aux_loss + moe_out.aux_loss
169
+ all_expert_counts.append(moe_out.expert_counts)
170
+
171
+ x = self.ranker.contract(x, state)
172
+ x = self.final_norm(x)
173
+
174
+ return x, total_aux_loss, all_expert_counts
175
+
176
+ def forward(self, input_ids, **kwargs):
177
+ x, aux_loss, expert_counts = self._get_hidden(input_ids)
178
+ return BaseModelOutput(last_hidden_state=x)
179
+
180
+
181
+ class AveyDecoderMoEForCausalLM(AveyDecoderMoEModel):
182
+ """
183
+ Avey-D MoE with causal language modeling head.
184
+
185
+ The total loss is:
186
+ loss = cross_entropy(logits, labels) + Ξ£ aux_loss_per_moe_layer
187
+
188
+ The auxiliary loss ensures balanced expert utilization.
189
+ """
190
+
191
+ def __init__(self, config: AveyDecoderMoEConfig):
192
+ super().__init__(config)
193
+ if not config.tie_word_embeddings:
194
+ self.lm_head = nn.Linear(config.d_embed, config.vocab_size, bias=False)
195
+ self.apply(self._init_weights)
196
+
197
+ def forward(self, input_ids, labels=None, **kwargs):
198
+ x, aux_loss, expert_counts = self._get_hidden(input_ids)
199
+
200
+ # Compute logits
201
+ if self.config.tie_word_embeddings:
202
+ logits = F.linear(x, self.embed.weight)
203
+ else:
204
+ logits = self.lm_head(x)
205
+
206
+ loss = None
207
+ if labels is not None:
208
+ # Labels pre-shifted by dataloader
209
+ ce_loss = F.cross_entropy(
210
+ logits.view(-1, logits.size(-1)),
211
+ labels.view(-1),
212
+ ignore_index=-100,
213
+ )
214
+ # Total loss = CE + auxiliary MoE balancing loss
215
+ loss = ce_loss + aux_loss
216
+
217
+ return CausalLMOutput(logits=logits, loss=loss)
218
+
219
+ @torch.no_grad()
220
+ def generate(
221
+ self,
222
+ input_ids,
223
+ max_new_tokens=256,
224
+ temperature=1.0,
225
+ top_k=50,
226
+ top_p=0.9,
227
+ eos_token_id=None,
228
+ ):
229
+ """Autoregressive generation with top-k/top-p sampling."""
230
+ self.eval()
231
+ for _ in range(max_new_tokens):
232
+ x, _, _ = self._get_hidden(input_ids)
233
+ if self.config.tie_word_embeddings:
234
+ logits = F.linear(x, self.embed.weight)
235
+ else:
236
+ logits = self.lm_head(x)
237
+
238
+ logits = logits[:, -1, :] / temperature
239
+
240
+ if top_k is not None and top_k > 0:
241
+ v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
242
+ logits[logits < v[:, [-1]]] = float("-inf")
243
+
244
+ if top_p is not None and top_p < 1.0:
245
+ sorted_logits, sorted_indices = torch.sort(logits, descending=True)
246
+ cumprobs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
247
+ remove = cumprobs > top_p
248
+ remove[:, 1:] = remove[:, :-1].clone()
249
+ remove[:, 0] = False
250
+ indices_to_remove = remove.scatter(1, sorted_indices, remove)
251
+ logits[indices_to_remove] = float("-inf")
252
+
253
+ probs = F.softmax(logits, dim=-1)
254
+ next_token = torch.multinomial(probs, num_samples=1)
255
+ input_ids = torch.cat([input_ids, next_token], dim=1)
256
+
257
+ if eos_token_id is not None and (next_token == eos_token_id).all():
258
+ break
259
+
260
+ return input_ids
moe.py ADDED
@@ -0,0 +1,500 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Mixture-of-Experts components for Avey-D.
3
+
4
+ Implements:
5
+ - TopKRouter: Learned gating network with top-k sparse routing
6
+ - Load balancing: auxiliary loss + DeepSeek-V3-style bias-based balancing
7
+ - Router Z-loss: numerical stability for gating logits (ST-MoE)
8
+ - StackedExperts: batched expert weights for single-kernel bmm dispatch
9
+ - Expert dispatch via padded torch.bmm (no Python loops over experts)
10
+
11
+ Design follows DeepSeek-V3 (shared + routed experts, fine-grained, top-k).
12
+ """
13
+
14
+ import torch
15
+ import torch.nn as nn
16
+ import torch.nn.functional as F
17
+ from dataclasses import dataclass
18
+ from typing import Tuple, Optional
19
+
20
+
21
+ @dataclass
22
+ class MoEOutput:
23
+ """Container for MoE layer outputs."""
24
+ hidden_states: torch.Tensor
25
+ router_logits: torch.Tensor # raw logits for aux loss
26
+ aux_loss: torch.Tensor # load-balancing auxiliary loss
27
+ expert_counts: torch.Tensor # num tokens per expert (for monitoring)
28
+
29
+
30
+ def _reshape_tokens(x: torch.Tensor) -> Tuple[int, int, int, torch.Tensor]:
31
+ """Flatten (B, T, D) -> (B*T, D) and return shape metadata."""
32
+ B, T, D = x.shape
33
+ return B, T, D, x.view(B * T, D)
34
+
35
+
36
+ def _unflatten_tokens(x: torch.Tensor, B: int, T: int, D: int) -> torch.Tensor:
37
+ """Restore (B*T, D) -> (B, T, D)."""
38
+ return x.view(B, T, D)
39
+
40
+
41
+ def _apply_spatial_proj(x: torch.Tensor, proj: torch.Tensor) -> torch.Tensor:
42
+ """Apply shared causal spatial projection to (B, T, D) tokens."""
43
+ return proj @ x
44
+
45
+
46
+ class TopKRouter(nn.Module):
47
+ """
48
+ Learned top-k sparse router for Mixture-of-Experts.
49
+ """
50
+
51
+ def __init__(
52
+ self,
53
+ d_embed: int,
54
+ num_experts: int,
55
+ top_k: int = 2,
56
+ aux_loss_coeff: float = 0.01,
57
+ z_loss_coeff: float = 0.001,
58
+ ):
59
+ super().__init__()
60
+ self.num_experts = num_experts
61
+ self.top_k = top_k
62
+ self.aux_loss_coeff = aux_loss_coeff
63
+ self.z_loss_coeff = z_loss_coeff
64
+
65
+ self.gate = nn.Linear(d_embed, num_experts, bias=False)
66
+
67
+ def forward(
68
+ self, x: torch.Tensor
69
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
70
+ B, T, D = x.shape
71
+ x_flat = x.view(B * T, D)
72
+
73
+ logits = self.gate(x_flat) # (B*T, num_experts)
74
+
75
+ z_loss = self.z_loss_coeff * torch.mean(logits.float() ** 2)
76
+ routing_probs = F.softmax(logits.float(), dim=-1)
77
+
78
+ topk_weights, topk_indices = torch.topk(
79
+ routing_probs, self.top_k, dim=-1
80
+ )
81
+ topk_weights = topk_weights / (topk_weights.sum(dim=-1, keepdim=True) + 1e-9)
82
+ topk_weights = topk_weights.to(x.dtype)
83
+
84
+ expert_mask = F.one_hot(topk_indices, self.num_experts).float()
85
+ expert_mask = expert_mask.sum(dim=1)
86
+ tokens_per_expert = expert_mask.sum(dim=0)
87
+ f_i = tokens_per_expert / (B * T)
88
+ P_i = routing_probs.mean(dim=0)
89
+ aux_loss = self.aux_loss_coeff * self.num_experts * (f_i * P_i).sum()
90
+
91
+ total_aux = aux_loss + z_loss
92
+
93
+ return topk_weights, topk_indices, total_aux, tokens_per_expert
94
+
95
+
96
+ # ---------------------------------------------------------------------------
97
+ # ExpertLayer β€” kept for the shared expert (single module, no batching needed)
98
+ # ---------------------------------------------------------------------------
99
+ class ExpertEnricher(nn.Module):
100
+ def __init__(self, d_embed: int, expansion_factor: int = 4):
101
+ super().__init__()
102
+ self.enricher = nn.Linear(d_embed, d_embed * expansion_factor)
103
+
104
+ def forward(self, x: torch.Tensor):
105
+ x = self.enricher(x)
106
+ x = F.gelu(x)
107
+ x, bypass = x.chunk(2, dim=-1)
108
+ x, gate = x.chunk(2, dim=-1)
109
+ return x, gate, bypass
110
+
111
+
112
+ class ExpertFuser(nn.Module):
113
+ def __init__(self, d_embed: int, expansion_factor: int = 4):
114
+ super().__init__()
115
+ enricher_out = d_embed * expansion_factor
116
+ half = enricher_out // 2
117
+ quarter = half // 2
118
+ fuser_in = quarter + half
119
+ self.fuser = nn.Linear(fuser_in, d_embed)
120
+
121
+ def forward(self, x: torch.Tensor, gate: torch.Tensor, bypass: torch.Tensor):
122
+ x = gate * x
123
+ x = torch.cat([x, bypass], dim=-1)
124
+ x = self.fuser(x)
125
+ return x
126
+
127
+
128
+ class ExpertLayer(nn.Module):
129
+ def __init__(self, d_embed: int, expansion_factor: int = 4):
130
+ super().__init__()
131
+ self.enricher_part = ExpertEnricher(d_embed, expansion_factor)
132
+ self.fuser_part = ExpertFuser(d_embed, expansion_factor)
133
+
134
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
135
+ x_q, gate, bypass = self.enricher_part(x)
136
+ return self.fuser_part(x_q, gate, bypass)
137
+
138
+
139
+ # ---------------------------------------------------------------------------
140
+ # StackedExperts β€” all routed expert weights in stacked tensors for torch.bmm
141
+ # ---------------------------------------------------------------------------
142
+ class StackedExperts(nn.Module):
143
+ """
144
+ Stores all routed expert weights as stacked (E, out, in) tensors so that
145
+ expert dispatch can use a single torch.bmm call instead of a Python loop.
146
+ """
147
+
148
+ def __init__(self, num_experts: int, d_embed: int, expansion_factor: int = 4):
149
+ super().__init__()
150
+ self.num_experts = num_experts
151
+ self.d_embed = d_embed
152
+ enricher_out = d_embed * expansion_factor # 4d
153
+
154
+ # Enricher: (E, 4d, d) weight + (E, 4d) bias
155
+ self.enricher_weight = nn.Parameter(torch.empty(num_experts, enricher_out, d_embed))
156
+ self.enricher_bias = nn.Parameter(torch.zeros(num_experts, 1, enricher_out))
157
+
158
+ # Fuser: input = quarter + half = d + 2d = 3d β†’ output = d
159
+ half = enricher_out // 2 # 2d
160
+ quarter = half // 2 # d
161
+ fuser_in = quarter + half # 3d
162
+ self.fuser_weight = nn.Parameter(torch.empty(num_experts, d_embed, fuser_in))
163
+ self.fuser_bias = nn.Parameter(torch.zeros(num_experts, 1, d_embed))
164
+
165
+ # Init
166
+ for w in [self.enricher_weight, self.fuser_weight]:
167
+ # Per-expert xavier: fan_in/fan_out from the last two dims
168
+ for i in range(num_experts):
169
+ nn.init.xavier_normal_(w.data[i])
170
+
171
+ def batched_enricher(self, x_padded: torch.Tensor) -> torch.Tensor:
172
+ """
173
+ Args:
174
+ x_padded: (E, cap, D)
175
+ Returns:
176
+ enriched: (E, cap, 4D) after matmul + bias
177
+ """
178
+ # bmm: (E, cap, D) @ (E, D, 4D) β†’ (E, cap, 4D)
179
+ return torch.bmm(x_padded, self.enricher_weight.transpose(1, 2)) + self.enricher_bias
180
+
181
+ def batched_fuser(self, x_padded: torch.Tensor) -> torch.Tensor:
182
+ """
183
+ Args:
184
+ x_padded: (E, cap, 3D) β€” concatenated [gated_quarter, bypass]
185
+ Returns:
186
+ fused: (E, cap, D)
187
+ """
188
+ # bmm: (E, cap, 3D) @ (E, 3D, D) β†’ (E, cap, D)
189
+ return torch.bmm(x_padded, self.fuser_weight.transpose(1, 2)) + self.fuser_bias
190
+
191
+
192
+ def _build_padded_input(
193
+ x_flat: torch.Tensor,
194
+ expert_indices: torch.Tensor,
195
+ num_experts: int,
196
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, int]:
197
+ """
198
+ Sort token-slot pairs by expert, pad to uniform capacity, build gather/scatter indices.
199
+
200
+ Args:
201
+ x_flat: (N, D)
202
+ expert_indices: (N, top_k)
203
+ num_experts: E
204
+
205
+ Returns:
206
+ x_padded: (E, cap, D) β€” input tokens padded per expert
207
+ token_ids: (total_pairs,) β€” original token index for each pair
208
+ slot_ids: (total_pairs,) β€” original slot index for each pair
209
+ expert_offsets: (total_pairs,) β€” position within expert's padded row
210
+ expert_counts: (E,) β€” actual count per expert (before padding)
211
+ capacity: int
212
+ """
213
+ N, top_k = expert_indices.shape
214
+ D = x_flat.shape[1]
215
+ device = x_flat.device
216
+
217
+ # Flatten (token, slot) pairs
218
+ flat_experts = expert_indices.reshape(-1) # (N*top_k,)
219
+ token_ids = torch.arange(N, device=device).unsqueeze(1).expand(-1, top_k).reshape(-1)
220
+ slot_ids = torch.arange(top_k, device=device).unsqueeze(0).expand(N, -1).reshape(-1)
221
+
222
+ # Sort by expert id
223
+ order = torch.argsort(flat_experts, stable=True)
224
+ sorted_experts = flat_experts[order]
225
+ sorted_tokens = token_ids[order]
226
+ sorted_slots = slot_ids[order]
227
+
228
+ # Count tokens per expert
229
+ expert_counts = torch.zeros(num_experts, dtype=torch.long, device=device)
230
+ expert_counts.scatter_add_(0, sorted_experts.long(), torch.ones_like(sorted_experts, dtype=torch.long))
231
+ capacity = int(expert_counts.max().item())
232
+
233
+ if capacity == 0:
234
+ x_padded = x_flat.new_zeros(num_experts, 1, D)
235
+ return x_padded, sorted_tokens, sorted_slots, torch.zeros_like(sorted_tokens), sorted_experts, expert_counts, 1
236
+
237
+ # Compute within-expert position for each sorted pair
238
+ cumcounts = torch.cumsum(expert_counts, 0) # (E,)
239
+ starts = cumcounts - expert_counts # (E,)
240
+ global_pos = torch.arange(sorted_experts.shape[0], device=device)
241
+ expert_start = starts[sorted_experts.long()]
242
+ within_expert_pos = global_pos - expert_start # position inside expert's group
243
+
244
+ # Build padded input: (E, cap, D)
245
+ x_padded = x_flat.new_zeros(num_experts, capacity, D)
246
+ x_padded[sorted_experts.long(), within_expert_pos] = x_flat[sorted_tokens]
247
+
248
+ return x_padded, sorted_tokens, sorted_slots, within_expert_pos, sorted_experts, expert_counts, capacity
249
+
250
+
251
+ class MoEBlock(nn.Module):
252
+ """
253
+ MoE block using batched torch.bmm dispatch (no Python expert loops).
254
+
255
+ Architecture (DeepSeek-V3 style):
256
+ output = shared_expert(x) + Ξ£_k w_k * routed_expert_k(x)
257
+ """
258
+
259
+ def __init__(
260
+ self,
261
+ d_embed: int,
262
+ num_experts: int = 32,
263
+ top_k: int = 2,
264
+ expansion_factor: int = 4,
265
+ use_shared_expert: bool = True,
266
+ aux_loss_coeff: float = 0.01,
267
+ z_loss_coeff: float = 0.001,
268
+ ):
269
+ super().__init__()
270
+ self.use_shared_expert = use_shared_expert
271
+ self.num_experts = num_experts
272
+ self.d_embed = d_embed
273
+
274
+ if use_shared_expert:
275
+ self.shared_expert = ExpertLayer(d_embed, expansion_factor)
276
+
277
+ # Stacked routed expert weights β€” single bmm replaces loop over E experts
278
+ self.routed_experts = StackedExperts(num_experts, d_embed, expansion_factor)
279
+
280
+ self.router = TopKRouter(
281
+ d_embed=d_embed,
282
+ num_experts=num_experts,
283
+ top_k=top_k,
284
+ aux_loss_coeff=aux_loss_coeff,
285
+ z_loss_coeff=z_loss_coeff,
286
+ )
287
+
288
+ def _batched_dispatch_full(
289
+ self,
290
+ x_flat: torch.Tensor,
291
+ routing_weights: torch.Tensor,
292
+ expert_indices: torch.Tensor,
293
+ ) -> torch.Tensor:
294
+ """
295
+ Full expert dispatch (no spatial proj) via batched bmm.
296
+ Used by the non-static forward path.
297
+ """
298
+ N, D = x_flat.shape
299
+ top_k = expert_indices.shape[1]
300
+ E = self.num_experts
301
+
302
+ x_padded, sorted_tokens, sorted_slots, within_pos, sorted_experts, expert_counts, cap = \
303
+ _build_padded_input(x_flat, expert_indices, E)
304
+
305
+ # Enricher: (E, cap, D) β†’ (E, cap, 4D) β†’ gelu β†’ split
306
+ enriched = self.routed_experts.batched_enricher(x_padded)
307
+ enriched = F.gelu(enriched)
308
+ half_a, bypass_pad = enriched.chunk(2, dim=-1) # each (E, cap, 2D)
309
+ quarter_pad, gate_pad = half_a.chunk(2, dim=-1) # each (E, cap, D)
310
+
311
+ # Fuser: gate * quarter, cat bypass β†’ (E, cap, 3D) β†’ (E, cap, D)
312
+ gated = gate_pad * quarter_pad
313
+ fuser_in = torch.cat([gated, bypass_pad], dim=-1)
314
+ fused = self.routed_experts.batched_fuser(fuser_in) # (E, cap, D)
315
+
316
+ # Scatter back with routing weights
317
+ # Gather per-pair results and weights
318
+ pair_results = fused[sorted_experts.long(), within_pos] # (total_pairs, D)
319
+
320
+ # Reorder routing weights to match sorted order
321
+ flat_weights = routing_weights.reshape(-1) # (N*top_k,)
322
+ order = torch.argsort(expert_indices.reshape(-1), stable=True)
323
+ sorted_weights = flat_weights[order]
324
+
325
+ weighted = pair_results * sorted_weights.unsqueeze(-1)
326
+ output = torch.zeros(N, D, dtype=fused.dtype, device=fused.device)
327
+ output.index_add_(0, sorted_tokens, weighted)
328
+
329
+ return output
330
+
331
+ def forward(self, x: torch.Tensor) -> MoEOutput:
332
+ B, T, D = x.shape
333
+ routing_weights, expert_indices, aux_loss, expert_counts = self.router(x)
334
+
335
+ x_flat = x.view(B * T, D)
336
+ routed_out = self._batched_dispatch_full(x_flat, routing_weights, expert_indices)
337
+ routed_out = routed_out.view(B, T, D)
338
+
339
+ if self.use_shared_expert:
340
+ shared_out = self.shared_expert(x_flat).view(B, T, D)
341
+ combined = shared_out + routed_out
342
+ else:
343
+ combined = routed_out
344
+
345
+ return MoEOutput(
346
+ hidden_states=combined,
347
+ router_logits=self.router.gate(x.view(B * T, D)),
348
+ aux_loss=aux_loss,
349
+ expert_counts=expert_counts,
350
+ )
351
+
352
+ def forward_static(self, x: torch.Tensor, masked_proj: torch.Tensor) -> MoEOutput:
353
+ """
354
+ Static-layer MoE with batched expert dispatch.
355
+
356
+ Flow:
357
+ route β†’ batched enricher (bmm) β†’ scatter to token order β†’
358
+ shared spatial projection β†’ gather back to expert order β†’
359
+ batched fuser (bmm) β†’ weighted scatter β†’ + shared expert
360
+ """
361
+ B, T, D, x_flat = _reshape_tokens(x)
362
+ top_k = self.router.top_k
363
+ E = self.num_experts
364
+
365
+ routing_weights, expert_indices, aux_loss, expert_counts = self.router(x)
366
+
367
+ # --- Build padded expert input ---
368
+ x_padded, sorted_tokens, sorted_slots, within_pos, sorted_experts, e_counts, cap = \
369
+ _build_padded_input(x_flat, expert_indices, E)
370
+
371
+ # --- Phase 1: batched enricher (single bmm) ---
372
+ enriched = self.routed_experts.batched_enricher(x_padded) # (E, cap, 4D)
373
+ enriched = F.gelu(enriched)
374
+ half_a, bypass_pad = enriched.chunk(2, dim=-1) # (E, cap, 2D) each
375
+ quarter_pad, gate_pad = half_a.chunk(2, dim=-1) # (E, cap, D) each
376
+
377
+ # Use enriched dtype for all intermediate buffers (bf16 under autocast)
378
+ buf_dtype = enriched.dtype
379
+ buf_device = enriched.device
380
+
381
+ # --- Scatter quarter to (N, top_k, D) for spatial proj ---
382
+ x_quarter = torch.zeros(B * T, top_k, D, dtype=buf_dtype, device=buf_device)
383
+ x_quarter[sorted_tokens, sorted_slots] = quarter_pad[sorted_experts.long(), within_pos]
384
+
385
+ # --- Shared spatial projection on x_quarter ---
386
+ x_quarter = x_quarter.view(B, T, top_k, D).permute(0, 2, 1, 3).contiguous()
387
+ x_quarter = x_quarter.view(B * top_k, T, D)
388
+ x_after_spatial = _apply_spatial_proj(x_quarter, masked_proj)
389
+ x_after_spatial = x_after_spatial.to(buf_dtype)
390
+ x_after_spatial = x_after_spatial.view(B, top_k, T, D).permute(0, 2, 1, 3).contiguous()
391
+ x_after_spatial = x_after_spatial.view(B * T, top_k, D)
392
+
393
+ # --- Gather back to expert-padded format for fuser bmm ---
394
+ # Skip intermediate (N, top_k) buffers β€” gather directly into (E, cap) format
395
+ x_fuser_pad = torch.zeros(E, cap, D, dtype=buf_dtype, device=buf_device)
396
+ x_fuser_pad[sorted_experts.long(), within_pos] = x_after_spatial[sorted_tokens, sorted_slots]
397
+
398
+ # gate_pad and bypass_pad are already in (E, cap, *) format from enricher β€” reuse directly
399
+
400
+ # --- Phase 3: batched fuser (single bmm) ---
401
+ gated = gate_pad * x_fuser_pad
402
+ fuser_in = torch.cat([gated, bypass_pad], dim=-1) # (E, cap, 3D)
403
+ fused = self.routed_experts.batched_fuser(fuser_in) # (E, cap, D)
404
+
405
+ # --- Weighted scatter to output ---
406
+ pair_results = fused[sorted_experts.long(), within_pos] # (total_pairs, D)
407
+
408
+ flat_weights = routing_weights.reshape(-1)
409
+ order = torch.argsort(expert_indices.reshape(-1), stable=True)
410
+ sorted_weights = flat_weights[order]
411
+
412
+ weighted = pair_results * sorted_weights.unsqueeze(-1)
413
+ routed_out = torch.zeros(B * T, D, dtype=buf_dtype, device=buf_device)
414
+ routed_out.index_add_(0, sorted_tokens, weighted)
415
+ routed_out = _unflatten_tokens(routed_out, B, T, D)
416
+
417
+ # --- Shared expert ---
418
+ if self.use_shared_expert:
419
+ shared_x, shared_gate, shared_bypass = self.shared_expert.enricher_part(x_flat)
420
+ shared_x = _unflatten_tokens(shared_x, B, T, D)
421
+ shared_x = _apply_spatial_proj(shared_x, masked_proj)
422
+ shared_x = shared_x.view(B * T, D)
423
+ shared_out = self.shared_expert.fuser_part(shared_x, shared_gate, shared_bypass)
424
+ shared_out = _unflatten_tokens(shared_out, B, T, D)
425
+ combined = shared_out + routed_out
426
+ else:
427
+ combined = routed_out
428
+
429
+ return MoEOutput(
430
+ hidden_states=combined,
431
+ router_logits=self.router.gate(x_flat),
432
+ aux_loss=aux_loss,
433
+ expert_counts=expert_counts,
434
+ )
435
+
436
+
437
+ # ---------------------------------------------------------------------------
438
+ # Legacy compat: keep old function signatures importable for tests
439
+ # ---------------------------------------------------------------------------
440
+ def _prepare_routing_pairs(expert_indices):
441
+ N, top_k = expert_indices.shape
442
+ flat_experts = expert_indices.reshape(-1)
443
+ token_ids = torch.arange(N, device=expert_indices.device).repeat_interleave(top_k)
444
+ slot_ids = torch.arange(top_k, device=expert_indices.device).repeat(N)
445
+ order = torch.argsort(flat_experts)
446
+ sorted_experts = flat_experts[order]
447
+ sorted_tokens = token_ids[order]
448
+ sorted_slots = slot_ids[order]
449
+ _, counts = torch.unique_consecutive(sorted_experts, return_counts=True)
450
+ return sorted_experts, sorted_tokens, sorted_slots, counts
451
+
452
+
453
+ def sparse_expert_dispatch_phase1(x, experts, expert_indices):
454
+ """Legacy wrapper β€” dispatches via loop (used by tests only)."""
455
+ N, D = x.shape
456
+ top_k = expert_indices.shape[1]
457
+ x_quarter = x.new_zeros(N, top_k, D)
458
+ gate = x.new_zeros(N, top_k, D)
459
+ bypass = x.new_zeros(N, top_k, D * 2)
460
+ sorted_experts, sorted_tokens, sorted_slots, counts = _prepare_routing_pairs(expert_indices)
461
+ start = 0
462
+ for e_id, c in zip(sorted_experts[torch.cumsum(counts, 0) - counts], counts):
463
+ e_idx = int(e_id.item())
464
+ end = start + int(c.item())
465
+ sel_tok = sorted_tokens[start:end]
466
+ sel_slot = sorted_slots[start:end]
467
+ x_q, g, b = experts[e_idx].enricher_part(x[sel_tok])
468
+ if x_q.dtype != x_quarter.dtype:
469
+ x_q, g, b = x_q.to(x_quarter.dtype), g.to(gate.dtype), b.to(bypass.dtype)
470
+ x_quarter[sel_tok, sel_slot] = x_q
471
+ gate[sel_tok, sel_slot] = g
472
+ bypass[sel_tok, sel_slot] = b
473
+ start = end
474
+ return x_quarter, gate, bypass
475
+
476
+
477
+ def sparse_expert_dispatch_phase3(x_after_spatial, gate, bypass, experts, routing_weights, expert_indices):
478
+ """Legacy wrapper β€” dispatches via loop (used by tests only)."""
479
+ N, top_k, D = x_after_spatial.shape
480
+ output = x_after_spatial.new_zeros(N, D)
481
+ flat_weights = routing_weights.reshape(-1)
482
+ sorted_experts, sorted_tokens, sorted_slots, counts = _prepare_routing_pairs(expert_indices)
483
+ order = torch.argsort(expert_indices.reshape(-1))
484
+ sorted_weights = flat_weights[order]
485
+ start = 0
486
+ for e_id, c in zip(sorted_experts[torch.cumsum(counts, 0) - counts], counts):
487
+ e_idx = int(e_id.item())
488
+ end = start + int(c.item())
489
+ sel_tok = sorted_tokens[start:end]
490
+ sel_slot = sorted_slots[start:end]
491
+ sel_w = sorted_weights[start:end]
492
+ expert_out = experts[e_idx].fuser_part(
493
+ x_after_spatial[sel_tok, sel_slot], gate[sel_tok, sel_slot], bypass[sel_tok, sel_slot])
494
+ if expert_out.dtype != output.dtype:
495
+ expert_out = expert_out.to(output.dtype)
496
+ if sel_w.dtype != output.dtype:
497
+ sel_w = sel_w.to(output.dtype)
498
+ output.index_add_(0, sel_tok, sel_w.unsqueeze(-1) * expert_out)
499
+ start = end
500
+ return output
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "errors": "replace",
8
+ "is_local": false,
9
+ "model_max_length": 1000000000000000019884624838656,
10
+ "pad_token": null,
11
+ "tokenizer_class": "GPT2Tokenizer",
12
+ "unk_token": "<|endoftext|>"
13
+ }