kingjones777/KAT-Coder-V2.5-Dev-Abliterated-MTP-ROCmFP4-GGUF

KAT-Coder-V2.5-Dev-Abliterated-MTP — ROCmFP4 for AMD Strix Halo (gfx1151)

The model here is not our work. The abliterated MTP checkpoint is jakeroxs's (KAT-Coder-V2.5-Dev-35B-A3B-MTP-ABLITERATED-GGUF); the base model is Kwaipilot's (KAT-Coder-V2.5-Dev). This repository adds only the ROCmFP4/ROCmFPX quantisation ladder, the dual-backend build, and the measurements. Go star their repos.

Four quantisations of the abliterated KAT-Coder-V2.5-Dev 35B-A3B MoE, each carrying a working Multi-Token-Prediction block, built for AMD Ryzen AI Max+ 395 / Radeon 8060S. Every file runs on BOTH the HIP (ROCm) and Vulkan backends from a single binary — the backend is a runtime -dev flag, not a rebuild.

Lead result: MTP is worth +6.5 % on Vulkan and +8.9 % on ROCm here, and the backend crossover flips by tier — Vulkan takes the 4-bit tiers, ROCm takes both Q6 tiers and runs them in 7.2 GiB less GTT.

✅ This model answers cleanly out of the box. 6/6 non-empty content at every budget from 400 tokens up, on all four workloads. That is worth stating because its architectural sibling does not — see the note at the end.

---

Which file should I use?

File	ftype	Size	BPW	Bytes
Q4_0_ROCMFP4_COHERENT	102	18.93 GiB	4.65	20,327,319,392
Q4_0_ROCMFP4_STRIX_LEAN	106	17.88 GiB	4.39	19,199,444,832
Q6_0_ROCMFPX_AGENT	114	30.89 GiB	7.58	33,169,606,496
Q6_0_ROCMFPX_LEAN	116	26.94 GiB	6.61	28,931,524,448

STRIX_LEAN (106) on Vulkan is the pick for coding work — fastest on the code workload (80.76 tok/s) and the smallest file. If your work is prompt-heavy, the same file on -dev ROCm0 prefills at 478 tok/s.

⛔ ftype 105 (STRIX) is deliberately not published — it leaves output.weight and the *_shexp tensors at 4 bit.

---

Quick start — the backend is one flag

# Vulkan — best decode (chat, agentic loops)
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev Vulkan0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

# HIP / ROCm — best prefill (long prompts, cold starts, RAG)
#   SAME FILE. Only -dev changes.
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev ROCm0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

These MTP flags are for single-shot use. For multi-turn / agentic work drop them entirely — see Multi-turn agentic work below, where speculation costs 31 % end-to-end. For single-shot, --spec-draft-p-min 0.0 is not optional if you want the speed. The default is 0.75 and it throttles the draft.

⛔ You need a ROCmFPX build with Vulkan enabled — stock llama.cpp will NOT load these files

ROCmFP4/ROCmFPX use ggml tensor types 100–119. Upstream llama.cpp's type table stops at 43, so a stock build rejects these files outright, on any backend:

gguf_init_from_reader: tensor '...' has invalid ggml type 101. should be in [0, 43)

Build one with both backends — the exact configuration behind every number on this card:

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
cmake -B build-hipvk -G "Unix Makefiles" \
  -DCMAKE_MAKE_PROGRAM=/usr/bin/gmake \
  -DCMAKE_C_COMPILER=/usr/bin/cc -DCMAKE_CXX_COMPILER=/usr/bin/c++ \
  -DCMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang \
  -DROCM_PATH=/opt/rocm-7.2.4 -DCMAKE_PREFIX_PATH=/opt/rocm-7.2.4 \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DAMDGPU_TARGETS=gfx1151 \
  -DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc \
  -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-hipvk -j12
build-hipvk/bin/llama-server --list-devices   # must list ROCm0 AND Vulkan0

Build traps, all hit for real: no Ninja on this box — use Unix Makefiles + gmake, and pass -DCMAKE_C_COMPILER/-DCMAKE_CXX_COMPILER explicitly or the HIP language probe dies with CMAKE_C_COMPILER not set, after EnableLanguage; the HIP compiler is at /opt/rocm-7.2.4/lib/llvm/bin/clang, not /opt/rocm/llvm/bin/clang++; glslc is not auto-detected.

---

Measured performance

AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB unified, GTT 131072 MiB, ROCm 7.2.4, Mesa 25.2.8 (RADV), kernel 6.17.6, Ubuntu 24.04.4, governor performance, box idle — 0 co-resident seats. -c 4096 -fa on -fit off -np 1, temperature 0, cache_prompt:false, unique nonce per request, 400 tokens, 1 warm-up discarded, median of 12.

All four tiers × both backends (MTP on, p_min 0.0)

Tier	Backend	code	prose	reason	json	prefill	acc len	GTT
COHERENT	Vulkan0	75.69	67.85	76.65	82.08	246	3.02	19.1 GiB
COHERENT	ROCm0	76.00	65.20	68.03	72.39	405	3.12	20.1 GiB
STRIX_LEAN	Vulkan0	80.76	68.13	74.76	77.96	260	3.01	18.6 GiB
STRIX_LEAN	ROCm0	79.47	65.58	72.13	70.66	478	3.12	19.1 GiB
Q6_AGENT	Vulkan0	56.85	51.70	57.02	57.27	229	3.01	34.8 GiB
Q6_AGENT	ROCm0	59.14	51.27	55.33	59.28	219	3.13	31.9 GiB
Q6_LEAN	Vulkan0	59.08	52.47	56.68	59.09	228	3.09	35.3 GiB
Q6_LEAN	ROCm0	64.89	54.22	58.73	62.41	152	3.20	28.1 GiB

⚠️ The prefill figures in the table above were measured on ~30-token prompts and understate real performance badly. On realistic long prompts (1,800–7,600 tokens) this model prefills at 745–813 tok/s. Short-prompt pp is not a meaningful number — quote long-prompt prefill.

What MTP is worth — STRIX_LEAN, single-shot, speculation off vs on

Tier	Backend	nodraft	MTP	gain	prefill
STRIX_LEAN	Vulkan0	71.69	76.36	+6.5%	294
STRIX_LEAN	ROCm0	65.53	71.39	+8.9%	542

With enable_thinking: false, 1600 tokens

Backend	code	prose	reason	json	prefill	non-empty
Vulkan0	83.73	71.80	79.20	78.29	219	24/24
ROCm0	81.45	68.79	70.98	70.21	410	24/24

The findings that matter

1. The backend crossover flips by tier. Vulkan wins decode on the 4-bit tiers; on both Q6 tiers ROCm wins outright — Q6_LEAN 64.89 vs 59.08 — and uses 7.2 GiB less GTT (28.1 vs 35.3). ROCm keeps the prefill crown everywhere (478 vs 260 on STRIX_LEAN, ~+84 %).
2. MTP pays on both backends (+6.5 % / +8.9 %), acceptance 3.01–3.20 of 4 on every arm.
3. Q6 buys footprint, not speed. Q6_AGENT (30.89 GiB) runs slower than STRIX_LEAN (17.88 GiB) on every workload. Choose Q6 for precision, not throughput.

⚠️ These rankings are model-specific and we say so. Measure them on your own model.

---

Silent-truncation gate — 480 generations, 0 truncated

On this hardware a Vulkan failure can be silent: 1 token, empty content, clean exit, /health still 200. A benchmark reporting only a median cannot see it. We count generations that actually reach n_predict:

Scope	Generations	Truncated	tg=1000000 sentinel
All four tiers × both backends × 4 workloads	480	0	0

Speculative decoding works on Vulkan on this model, across 240 Vulkan generations.

---

Content adequacy — clean at every budget

Non-empty content, 6 reps per cell, thinking at default:

max_tokens	code	prose	reason	json
400	6/6	6/6	6/6	6/6
800	6/6	6/6	6/6	6/6
1200	6/6	6/6	6/6	6/6
1600	6/6	6/6	6/6	6/6

We measured this because the architectural sibling we built alongside it — an abliterated Agents-A1 on the same qwen35moe 35B-A3B graph, same fork, same box — returns empty content on most non-code prompts at every budget up to 1600 unless enable_thinking:false is set. This model has no such problem. Same architecture, opposite behaviour: do not assume a serving parameter transfers between models on this graph.

---

---

⛔ Multi-turn agentic work — turn MTP OFF

This is the most important correction on this card. The speculative-decoding flags that make single-shot generation fast break the prompt cache on multi-turn work. The server reports:

prompt cache cold fallback: reason=spec-boundary-mismatch

Every turn then re-prefills the entire accumulated conversation. In a real agentic run we watched per-turn prefill climb to 9.5 s at 7,124 tokens while decode sat at 92.9 tok/s — the model was barely getting to generate.

Measured on an 8-turn growing conversation (each turn appends the previous answer, 300 tokens per turn, enable_thinking:false), reporting end-to-end effective throughput — total tokens generated over total wall time, which is what a user actually experiences:

Backend	Speculation	-cram	End-to-end tok/s	Prefill % of wall	Cache misses
Vulkan0	MTP	512	48.61	36 %	10
Vulkan0	MTP	4096	38.75	34 %	10
Vulkan0	MTP + --spec-mtp-strict-qwen	4096	57.72	33 %	10
Vulkan0	none	4096	63.62	9 %	0
ROCm0	none	4096	58.92	6 %	0
ROCm0	MTP	4096	57.78	26 %	10

Every arm with MTP enabled took 10 cache misses. Every arm without took 0. Dropping speculation costs ~19 % of single-shot decode and returns +31 % end-to-end, because prefill falls from 36 % of wall time to 9 %.

--spec-mtp-strict-qwen recovers much of the loss but does not eliminate the misses — it is not a way to keep both.

Serving config for agentic / multi-turn

llama-server -m <this-file>.gguf \
  -dev Vulkan0 -ngl 999 -fa on --no-mmap -fit off -c 131072 \
  -ctk q8_0 -ctv q8_0 -np 1 --jinja -cram 4096 \
  --host 0.0.0.0 --port 8080

Size -cram deliberately. It defaults to 8192 MiB per server and scales with context. A 65-token prompt costs ~63.6 MiB of cache state here, so 512 MiB holds about eight short prompts — an agentic conversation evicts itself immediately. 4096 MiB held an 8-turn session with zero misses. On a box running several models, N servers × 8 GiB of default cache will oversubscribe it.

Deployed and re-verified on the seat afterwards: 63.51 tok/s end-to-end, 0 cache misses, prefill 10 % of wall.

Use MTP only for single-shot / batch

The --spec-type draft-mtp numbers elsewhere on this card are single-shot measurements — one request, no conversation history. They are real, and they are the right choice for batch generation, summarisation, or any one-shot call. They do not describe multi-turn behaviour.

---

Head protection — verified, not assumed

Left to their defaults several ftypes emit a 4-bit output.weight, a documented quality-collapse pattern. Every file here was quantised with an explicit head flag and then read back out of the finished file by exact tensor name — never substring-matched, because attn_output matches output and gives a false pass.

Tier	Head flag passed	output.weight	token_embd.weight
COHERENT (102)	--output-tensor-type q6_K	Q6_K ✅	Q6_K
STRIX_LEAN (106)	--output-tensor-type q6_K	Q6_K ✅	Q5_K
Q6_AGENT (114)	--output-tensor-type q8_0 --token-embedding-type q8_0	Q8_0 ✅	Q8_0
Q6_LEAN (116)	(none — default already protected)	Q6_0_ROCMFPX ✅	Q6_0_ROCMFPX

On 106 token_embd is deliberately not overridden — Q5_K embeddings are the point of the LEAN tier; only the head needs forcing.

---

Source integrity

The source checkpoint was gated before quantisation, not assumed:

arch                 qwen35moe
tensors              753   (blk.0-39 + blk.40 MTP)
block_count          41
nextn_predict_layers 1
blk.40 tensors       20
output.weight        BF16   (unquantised source)
bytes                71,066,994,848

Quantised from BF16 only — never from an existing Q4_K_M or ROCmFP4.

---

Reproduction block

binary          llama-server  sha256 d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7
                llama-quantize sha256 1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
source          ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e  (-DGGML_HIP=ON -DGGML_VULKAN=ON)
hardware        AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151), 128 GB unified
gtt_total       131072 MiB
rocm            7.2.4          vulkan  Mesa 25.2.8 (RADV GFX1151), API 1.4.318
kernel          6.17.6-061706-generic      os  Ubuntu 24.04.4 LTS
governor        performance
co-resident     0 seats (box idle for the whole measurement)
env             HSA_OVERRIDE_GFX_VERSION=11.5.1
serve argv      -ngl 999 -fa on -fit off -np 1 -c 4096 --jinja
                --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0
                -dev {ROCm0|Vulkan0}
quantise argv   llama-quantize <headflag> <BF16> <OUT> <FTYPE> 16
protocol        temperature 0, cache_prompt false, unique nonce, 400 tok,
                1 warm-up discarded, median of 12

SHA256

3a4e59d1f7dfb8257ba81604f5739fa556de02f18887205f136f8fa427c8cd4f  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_COHERENT.gguf
21278799c27afa5894c7626fdf6b3717b0a30fa7f8fa4b63aa831bb8e5904d01  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf
a016cd522b15666b719bc087b03f07234cc140fc663449d37f1e9dc59b4cedae  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_AGENT.gguf
f92370a5b2b513b940bf46fb53c9e26898e5d85c91f84c8d85ff3bfe8bc4dfbf  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_LEAN.gguf