SuperLogic/Qwen3.5-4B-3D-Coder-MTP-GGUF

Qwen3.5-4B-3D-Coder-MTP — the missing draft head, put back

47.9 → 90.7 tok/s on an 8 GB laptop GPU, same prompt, same output.

ccore/Qwen3.5-4B-3D-Coder writes complete Three.js scenes from a paragraph of description. It also declares one MTP layer in its config and then does not ship it, so every runtime decodes it one token at a time. This repository contains exactly the same weights with that draft head restored.

Nothing else was touched. Every Q8_0 tensor of the main model is byte-identical to the original file; the only additions are the 15 MTP tensors and three metadata values.

Sample output

Two scenes written by this model from one paragraph of description each — no reference code, nothing modelled by hand:

_{Each is a single self-contained HTML file: geometry, materials, lighting, camera and the animation loop all come from the model. The bicycle needed three runtime fixes after generation (a wrong CDN path, a render loop that ran once, and a camera parked inside the frame) — all mechanical errors, no geometry was edited by hand. Details in what this model is good and bad at.}

---

The problem

The published checkpoint is internally inconsistent:

	tensors	*.mtp.*	config.text_config.mtp_num_hidden_layers	GGUF nextn_predict_layers
Qwen/Qwen3.5-4B (base)	738	15	1	1
ccore/Qwen3.5-4B-3D-Coder	723	0	1	0
this repo	738	15	1	1

The config claims an MTP layer; the safetensors do not contain one. Fifteen tensors were simply not copied during export. Because the GGUF carries nextn_predict_layers = 0, any runtime that looks for a draft path finds nothing:

• llama-server --spec-type draft-mtp-adaptive has nothing to draft from;
• LM Studio either hides the MTP toggle or accepts it and does nothing;
• decoding falls back to the plain single-token rate.

The original model card's advice — "LM Studio: disable MTP" — is about this missing head, not about the architecture.

The fix

The 15 tensors are taken from the base model Qwen/Qwen3.5-4B, whose MTP head was never trained away by the fine-tune; it was only left out of the export. They are fetched with HTTP range requests (241 MB instead of the whole 8 GB base checkpoint), renamed with the same mapping convert_hf_to_gguf.py uses, and appended to the original GGUF:

• qwen35.block_count 32 → 33
• qwen35.nextn_predict_layers 0 → 1
• qwen35.attention.recurrent_layers gains one trailing false

No requantisation. Two details are worth knowing if you repeat this:

1. Norms must be F32, matrices F16. Storing the new layer as BF16 crashes the CPU-side RMSNorm on the first request: binary_op: unsupported types: dst: f32, src0: f32, src1: bf16.
2. Qwen3.5 stores RMSNorm weights as w-1. convert_hf_to_gguf.py adds the 1 back; a raw byte copy does not. Without it the head loads, runs, and accepts 0 of 1198 tokens — a silently useless draft path. Symptom: nextn.enorm mean is negative (should be ≈ 0.4).

---

Benchmarks

RTX 4060 Laptop, 8 GB, exclusive use of the GPU, -fa on, 64K context, q8_0 KV cache, one prompt repeated with 1200-token outputs, first round discarded as warm-up.

configuration	tok/s	acceptance	tokens/step
original GGUF, no speculation	47.94	—	1.00
this file, --spec-draft-n-max 2	68.44	0.86	2.72
this file, --spec-draft-n-max 4	90.66	0.71	3.60
this file, --spec-draft-n-max 6	85.07	0.62	3.80
n-max 4 + --spec-type ngram-mod,draft-mtp-adaptive	89.48	0.69	3.60
n-max 4 + LLAMA_MTP_DRAFT_UBATCH=64	89.42	—	—

n-max 4 is the sweet spot. ngram and the larger draft ubatch both land inside the noise, so neither is recommended.

Peak VRAM with the vision projector loaded: 6769 / 8188 MiB at 64K context.

Why speculation pays off here. At 47.9 t/s the 4.48 GB of weights are being read at about 215 GB/s against a 256 GB/s peak — decoding is purely bandwidth-bound and the GPU's compute units sit idle. Verifying 4 drafted tokens therefore costs almost the same as verifying 1, and 3.6 tokens per step turns into roughly 2× throughput. This is also why the numbers above only reproduce under the same conditions; see the next section.

---

Running it in LM Studio

MTP is a runtime switch, not a property of the file. The same .gguf runs at 47.9 t/s or 90.7 t/s depending solely on whether the runtime is told to use the draft head. In LM Studio that means three settings, all in the model's load panel:

setting	value	why
Speculative decoding → Draft MTP	on	off by default. Without it the head is never called and you get the base rate
Max concurrent predictions	1	each extra slot allocates its own compute buffers; at 4 slots the model no longer fits and the drafter's gains are lost
Draft probability (min continue probability)	0	the 0.75 default rejects a large share of correct drafts; 0 accepts them all and lets the target model verify
Context length	≤ 16384 on 8 GB	see below
GPU offload	max	must actually be resident

Verified in LM Studio: with these settings the speed-up is clearly visible, and it collapses back to the base rate when Draft MTP is switched off — the cheapest way to confirm the switch took effect.

Command-line equivalent:

lms load <model-key> -c 16384 --gpu max --parallel 1 \
  --speculative-draft-mtp \
  --speculative-draft-max-tokens 4 \
  --speculative-draft-min-continue-probability 0

The trap: an over-committed GPU kills speculation silently

This is the failure mode worth knowing about, because nothing reports it.

On Windows, LM Studio defaults to Strict GPU VRAM cap: OFF, so it will load with Num Offload Layers: max even when its own estimate exceeds free VRAM — and logs nothing beyond the estimate. The driver then pages the overflow into system memory. Everything still works, just slowly, and speculative decoding stops helping entirely: the bottleneck moves from 256 GB/s of VRAM bandwidth to roughly 25 GB/s of PCIe, so skipping forward passes saves nothing, while the draft model adds a forward pass of its own.

Two things cause this without any error message:

• another process holding VRAM — a second llama-server, a game, a browser;
• context length sized right at the edge of the budget.

Check before blaming the model:

nvidia-smi --query-gpu=memory.used,memory.free --format=csv
nvidia-smi --query-compute-apps=pid,process_name --format=csv

If free VRAM is under ~1 GB with the model loaded, you are paging. Drop context to 16384, close the other process, and reload — the model must be reloaded for weights to come back into VRAM.

Running it with llama-server

llama-server \
  -m Qwen3.5-4B-3D-Coder-MTP-Q8_0.gguf \
  --mmproj mmproj-Qwen3.5-4B-BF16.gguf \
  -c 65536 -b 4096 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 \
  --spec-type draft-mtp-adaptive --spec-draft-n-max 4 --spec-draft-sampling probabilistic \
  --temp 0.7 --top-p 0.95 --top-k 20 --repeat-penalty 1.0

--spec-draft-sampling probabilistic matters: with rejection sampling the output distribution is exactly the target model's, so speculation changes speed only, never content.

Bounding the thinking

Scenes take 8–11k output tokens and the model likes to think for a long time first. Its chat template exposes only enable_thinking — there is no thinking_budget and no reasoning_effort, so a prompt asking it to "think briefly" is advisory at best. Use the sampler instead:

--reasoning-budget 2048 \
--reasoning-budget-message "(thinking budget reached - stop reasoning and write the final answer now)"

--reasoning-budget force-emits the end-of-think tag at the budget and hands control to the answer, and the message gives the model a chance to wrap up rather than being cut mid-thought. Do not set it low: below roughly 500 tokens models tend to reason their way to the right conclusion and then answer with a wrong one, which is worse than not thinking at all.

---

Files

file	size	notes
Qwen3.5-4B-3D-Coder-MTP-Q8_0.gguf	4.72 GB	main weights + MTP head
mmproj-Qwen3.5-4B-BF16.gguf	0.68 GB	vision projector, unchanged from upstream
assets/*.png	—	the two sample renders shown above

The projector is required for image input; drop it and the model still works as a text-only coder.

What this model is good and bad at

It writes complete, self-contained Three.js scenes from a paragraph of description — geometry, materials, lighting, camera, animation loop. On three upstream demo prompts: ~8–11k output tokens per scene, 90–110 s each at the speed above.

It is not one-shot reliable. Across three generated scenes we hit four runtime bugs that pass every syntax check:

• a duplicate const in the same scope, which kills the whole ES module → white page;
• Euler.prototype.multiplyScalar — does not exist;
• a float used as an array index, yielding undefined and then TypeError: reading 'x' inside the animation loop;
• a camera that was never positioned, so it sat inside the geometry.

All four are wrong-API or wrong-type mistakes, not structural ones. Budget for a check-and-repair loop: syntax check → headless render → feed the console error back.

Credits and licence

Fine-tune by ccore; base model by Qwen. Both Apache-2.0, and this derivative keeps that licence. This repository only re-adds weight tensors that were published in the base model, plus the conversion notes needed to reproduce it.