vtava/Laya-v49-selective-svd

Laya V4.9 — Selective SVD compression

Compress only 7 pretrained LOCAL follower-layer attention projections using SVD-initialized rank-limited residuals. Every unselected matrix remains identical to the original V4.6 checkpoint. FFNs, layer norms and the 64-token marker-aware decision head are unchanged.

• Selected projections: [{'first': 25, 'second': 26, 'component': 'Wqkv', 'rank': 64}, {'first': 22, 'second': 23, 'component': 'Wqkv', 'rank': 256}, {'first': 25, 'second': 26, 'component': 'Wo', 'rank': 64}, {'first': 22, 'second': 23, 'component': 'Wo', 'rank': 64}, {'first': 7, 'second': 8, 'component': 'Wo', 'rank': 64}, {'first': 16, 'second': 17, 'component': 'Wo', 'rank': 256}, {'first': 19, 'second': 20, 'component': 'Wo', 'rank': 256}]
• Distillation checkpoint: 120
• Measured FP32 checkpoint bytes saved: 35,126,592
• AUDIT teacher agreement: 99.000%
• AUDIT gold accuracy: 84.500%
• AUDIT mean JS: 0.0008069
• GPU median speed ratio vs native V4.6: 0.991×
• All gates passed: True

After downloading this model repository once, import the included modeling_laya_v49.py, then use its load_v49_model(local_dir,device="cuda"). No other model downloads are needed at inference time.

The measured speed/quality only covers the reported Colab GPU and a small held-out audit. Sharing attention weights does not remove attention FLOPs.