Crash with llama.cpp
I'm getting this error with b8121: llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.ffn_gate_inp.weight' has wrong shape; expected 2048, 512, got 2048, 258, 1, 1
Hi @crotron , thanks for the report, and apologies for the confusion — this model was initially listed as llama.cpp compatible, which was incorrect. We've since updated the model
card, but understand this may have led you to download it expecting llama.cpp support.
This model uses layer-adaptive pruning, where each layer retains a different number of experts (e.g., layer 0 has 258 of the original 512 experts). llama.cpp assumes a uniform
expert count across all layers, so it rejects the tensor shape mismatch.
Why layer-adaptive pruning?
We use layer-adaptive allocation because uniform pruning causes severe quality degradation, especially in STEM and math reasoning. At equivalent compression ratios on the 80B
model:
| Method | Size | MMLU | STEM |
|---|---|---|---|
| Original Q4 | 45 GB | 77% | 65% |
| Layer-adaptive 50% | 24.4 GB | 72% | about 63% |
| Uniform 40% | about 27 GB | 64% | about 36% |
Layer-adaptive preserves +8pp MMLU and +27pp STEM over uniform pruning at similar sizes. This is because different layers have very different sensitivity — uniform pruning destroys
math/reasoning circuits in critical layers, causing a sharp quality cliff. Our paper documents this pruning cliff phenomenon across multiple
models.
The trade-off is llama.cpp incompatibility, which is why this model requires moe-stream.
Quick start
Option 1: Pre-built binary (macOS Apple Silicon only, no Rust needed)
Download from moe-stream v0.1.0 Releases:
tar xzf moe-stream-v0.1.0-macos-arm64.tar.gz
./moe-stream path/to/PrunedHub-Qwen3-Coder-Next-50pct-Q4_K_M.gguf 100 \
--prompt "Hello" --stream
Option 2: Build from source
git clone https://github.com/GOBA-AI-Labs/moe-stream.git
cd moe-stream
cargo build --release -p moe-stream-core --features accelerate
cargo run --release -p moe-stream-core --example generate -- \
path/to/model.gguf 100 --prompt "Hello" --stream
Note: moe-stream is currently developed and tested on macOS (Apple Silicon). It may compile on Linux but has not been tested on other platforms. moe-stream automatically selects the
optimal inference mode based on your available memory — if the model fits in GPU memory, it runs fully GPU-resident (e.g., 55 tok/s for smaller models); otherwise it uses SSD streaming
(about 2 tok/s for this 24.4 GB model on 24GB hardware).
llama.cpp-compatible alternatives
If you need llama.cpp support, our GPT-OSS-20B-28x (10.4 GB, MMLU 78%) uses uniform pruning and works directly with llama.cpp at about 55 tok/s.
We're also open to releasing uniform-pruned versions of this model for llama.cpp compatibility — they would be larger at the same quality level, but fully compatible. Let us know if
that would be useful.
We plan to propose experts_per_layer support to llama.cpp in the future to enable broader compatibility.
Hello @TOk-Atsuru ,
I use LM Studio on Windows for inference, which is based on llama.cpp and right now it's my preferred inference option. I'm open to exploring other options, but the tool you're presenting here, the moe-stream sounds like a project meant for expert users. I wouldn't even know how to compile it for Windows to begin with, if that's even possible at all. On the other hand, a version of this model that is smaller than the original, with as much retention of the original quality as possible would be very nice to have since my hardware has some limits that won't let me to enjoy the full version.
Is there any chance for contribution to bring support for what moe-stream does into llama.cpp? This seems like a very pragmatic approach that would bring this special version of the model to the wider masses.