sdwilsh/Ling-3.0-flash-GGUF

Ling-3.0-flash: Atomic Dynamic Q4_K_S and Q5_K_M, rebuilt for upstream llama.cpp

A GGUF quantization of inclusionAI/Ling-3.0-flash that uses AtomicChat's "Atomic Dynamic" (AD) bit-placement recipe and AtomicChat's published importance matrix, but is built from an upstream-format source so that it loads on stock, unmodified llama.cpp.

	AD-Q4_K_S	AD-Q5_K_M
Folder	Ling-3.0-flash-AD-Q4_K_S/	Ling-3.0-flash-AD-Q5_K_M/
Shards	2	3
Unsharded size	75,981,701,568 bytes (70.76 GiB)	91,576,648,128 bytes (85.29 GiB)
Bits per weight	4.77	5.75
Architecture	bailingmoe3, 938 tensors, 43 blocks	bailingmoe3, 938 tensors, 43 blocks
Base ftype	Q4_K_S, with 8 per-tensor rules	Q5_K_M, with 8 per-tensor rules
Rule file	tensor-types-q4ks.txt	tensor-types-q5km.txt
Loads on	stock llama.cpp v0.5.0 (build b11146) and later, with no fork required	same

---

Credit where it is due

The recipe in these files is not mine. It is AtomicChat's.

The bit-placement design (routers and expert bias kept at F32, attention and KDA gates at Q8_0, output at F16, token_embd at Q8_0, extra bits for the edge MoE blocks, and only the routed experts squeezed) is AtomicChat's "Atomic Dynamic" line, published at:

→ AtomicChat/Ling-3.0-flash-GGUF

The importance matrix used to quantize these files is also theirs, and used unmodified. If you find this quant useful, the design credit belongs to them, and their repo is the one to follow.

Why these files exist

AtomicChat's own AD- files were converted about two weeks (11 to 13 days) before upstream llama.cpp gained bailingmoe3 support. Their card says so plainly ("the files need a TurboQuant build"). That was entirely correct when published, as their fork shipped BailingMoeV3 around 2026-08-04, and upstream's support PR did not close until 2026-08-17.

The practical consequence today is that their files do not load on an upstream build. If you try, you will see this:

check_tensor_dims: tensor 'blk.0.ssm_f_a.weight' not found

This is a structural rejection by the upstream bailingmoe3 loader, and it is backend-independent. If you have hit that error, this repo is what you want.

This repo is the same bit placement, rebuilt against an upstream-compatible source, for people who want the AD bit placement without running a fork. It was checked tensor by tensor against AtomicChat's published AD-Q4_K_S and AD-Q5_K_M files: all 917 tensors they carry have the same type and shape here (see "How the match was checked"). Nothing here is a criticism of AtomicChat's work. It is a repackaging of it, for a llama.cpp that did not exist yet when they shipped.

---

Provenance

Everything below is recorded so the file can be rebuilt bit-for-bit.

Base model

inclusionAI/Ling-3.0-flash, MIT licence. 124B total parameters, 5.1B active. Hybrid linear attention: 35 KDA blocks interleaved 5:1 with 7 gated MLA blocks, over a 512-expert MoE.

Source weights

Repo	bartowski/Ling-3.0-flash-GGUF
Revision	7e1a1243566501d52af98e9aa7a7a61986279c6d
Directory	Ling-3.0-flash-bf16/
Shards	7
Bytes	255,091,083,264 (237.57 GiB)

Already bf16 GGUF in upstream format, so no HF→GGUF conversion step was needed, and this is the whole point: the conversion is where AtomicChat's files and upstream diverge.

Importance matrix

Repo	AtomicChat/Ling-3.0-flash-GGUF
Revision	253738fe190c15f329001f263f355fc1562bbe7c
Path	imatrix/Ling-3.0-flash.imatrix.gguf
Bytes	487,748,480 (465.15 MiB)
sha256	7d3c0ebe9eb235cc08e0b7c91886f5c422772ea95eebbf0f53b0974a1c040991

Quantizer

Image	ghcr.io/ggml-org/llama.cpp:full-v0.5.0
Digest	sha256:386ead2afc762e8ca7d2e65d6cd8b031d9a64ffcf989bce815c1614d7ef2afeb
Build	b11146, commit 7fe450e19
Wall clock	AD-Q4_K_S 4,759.0 s (79.3 min); AD-Q5_K_M 4,691.3 s (78.2 min); 6 threads each, two builds running side by side on one node with 12 threads (6 cores, SMT)

---

Reproducing these files

1. The per-tensor rule files

Both are included in this repo. The order of the rules is load-bearing: llama.cpp uses std::regex_search and the first match wins, and any manual match bypasses the built-in k-quant mixture for that tensor entirely. The placement was read from the tensor tables of AtomicChat's own GGUF files (HTTP Range reads of the headers, revision 253738fe190c15f329001f263f355fc1562bbe7c); AtomicChat publish no recipe file.

tensor-types-q4ks.txt (base Q4_K_S):

^token_embd\.weight$=q8_0
^blk\.(2|3|39|40|41)\.ffn_(down|gate|up)_exps\.weight$=q5_k
^blk\.\d+\.ffn_(down|gate|up)_exps\.weight$=q4_k
^blk\.\d+\.attn_(q|k|v|output|gate|k_b|v_b|kv_a_mqa)\.weight$=q8_0
^blk\.\d+\.ssm_(f_a|g_a|beta)\.weight$=q8_0
^blk\.\d+\.ffn_(down|gate|up)_shexp\.weight$=q8_0
^blk\.\d+\.ffn_(down|gate|up)\.weight$=q8_0
^blk\.\d+\.nextn\.eh_proj\.weight$=q8_0

tensor-types-q5km.txt (base Q5_K_M) differs only in the two expert rules, which move up one step:

^token_embd\.weight$=q8_0
^blk\.(2|3|39|40|41)\.ffn_(down|gate|up)_exps\.weight$=q6_k
^blk\.\d+\.ffn_(down|gate|up)_exps\.weight$=q5_k
(the remaining five rules are identical)

In both files, edge blocks 2, 3, 39, 40 and 41 get one step more than the other routed-expert blocks, and down, gate and up get the same type. Norms, routers (ffn_gate_inp), exp_probs_b and the ssm_* tensors (small, norm or conv tensors that llama.cpp never quantizes) come out as F32 with no rule, and output.weight is F16 by the flag below.

sha256 of the rule files: tensor-types-q4ks.txt d854d6f53dd34d0e96060e4cd083604840474ad04dbfdc55c5eacb7b2c6d4711, tensor-types-q5km.txt a0d82ffdd269bcd000b15bb34d330ac3ba165c9b5467aac7d0a59305d7129434.

2. The quantize invocation

Verbatim, as run (Q4_K_S shown; for the other file use tensor-types-q5km.txt, an output name with AD-Q5_K_M, and Q5_K_M):

llama-quantize \
  --imatrix            Ling-3.0-flash.imatrix.gguf \
  --output-tensor-type f16 \
  --tensor-type-file   tensor-types-q4ks.txt \
  --max-buffer-size    2048 \
  Ling-3.0-flash-bf16-00001-of-00007.gguf \
  Ling-3.0-flash-AD-Q4_K_S-upstream.gguf \
  Q4_K_S \
  6

6 is the thread count. token_embd.weight is set to Q8_0 by the first rule (the --token-embedding-type q8_0 flag is equivalent).

3. The split

llama-gguf-split --split --split-max-size 45G \
  Ling-3.0-flash-AD-Q4_K_S-upstream.gguf \
  Ling-3.0-flash-AD-Q4_K_S

llama-gguf-split --split --split-max-size 32G \
  Ling-3.0-flash-AD-Q5_K_M-upstream.gguf \
  Ling-3.0-flash-AD-Q5_K_M

--split-max-size units are decimal (45G = 45,000,000,000 bytes). This keeps every shard under Hugging Face's 50 GB per-file limit.

How the match was checked

The tensor tables (name, type, shape) of the shards in this repo were read with the same HTTP Range method as AtomicChat's, and compared one tensor at a time. Result for both files: all 917 tensors that AtomicChat publish are present here with the same type and the same shape (0 type mismatches, 0 shape mismatches). The only name difference is that their fork calls the KDA tensors ssm_f.weight/ssm_g.weight, where upstream says ssm_f_a/ssm_g_a; those were compared as the same tensor. The 21 extra tensors in these files are the NextN block (below).

---

Composition

AD-Q4_K_S (measured from the tensor table):

type	tensors	GiB	share
Q4_K	108	56.95	80.5%
Q5_K	15	9.67	13.7%
Q8_0	424	3.18	4.5%
F16	1	0.75	1.1%
F32	390	0.21	0.3%

AD-Q5_K_M (measured from the tensor table):

type	tensors	GiB	share
Q5_K	108	69.61	81.6%
Q6_K	15	11.54	13.5%
Q8_0	424	3.18	3.7%
F16	1	0.75	0.9%
F32	390	0.21	0.2%

That table looks lopsided until you notice 424 Q8_0 tensors occupy about 3 GiB. All the attention projections, the KDA gates, every shared expert and token_embd together are only about 3% of the weights, so keeping them at 8 bits is nearly free. The routed experts are 97.12% of the parameters (measured: 123.816B of 127.486B; AtomicChat's card states 97.1%), and they are the only thing this recipe actually squeezes.

For scale against neighbouring builds of the same model:

build	bytes	GiB	tensors	vs AD-Q4_K_S here
AD-Q4_K_S here	75,981,701,568	70.76	938	n/a
AD-Q5_K_M here	91,576,648,128	85.29	938	+20.5%
AtomicChat AD-Q4_K_S	74,223,552,640	69.13	917 (no NextN)	-2.31%
AtomicChat AD-Q5_K_M	89,441,011,840	83.30	917 (no NextN)	+17.7%
bartowski Q4_K_S	75,049,688,768	69.90	938	-1.23%
bartowski IQ4_XS	68,739,201,216	64.02	938	-9.53%

Each file here is about 2.4% larger than AtomicChat's file of the same name. The whole difference is the NextN block (21 tensors, about 1.76 GB in AD-Q4_K_S and about 2.14 GB in AD-Q5_K_M), which their converter drops and the upstream source keeps. On the 917 tensors both files carry, the types and shapes are identical.

---

Files

Ling-3.0-flash-AD-Q4_K_S/

file	bytes	GiB	tensors	sha256
Ling-3.0-flash-AD-Q4_K_S-00001-of-00002.gguf	44,797,227,616	41.72	573	2be134529a49adde75f144164c2e1a57533b39f0bdf7beab5337461b30ae3c9f
Ling-3.0-flash-AD-Q4_K_S-00002-of-00002.gguf	31,184,474,144	29.04	365	ab85ee0989690904c47d7098a140524ed46653614c912dae0490c04c273a763d

Ling-3.0-flash-AD-Q5_K_M/

file	bytes	GiB	tensors	sha256
Ling-3.0-flash-AD-Q5_K_M-00001-of-00003.gguf	31,660,549,472	29.49	338	b16ca636ddbdc4d9c43b429e4989efbeb67cfc34567f995895278f8a5f580e23
Ling-3.0-flash-AD-Q5_K_M-00002-of-00003.gguf	31,523,973,888	29.36	323	e851bf4907c987a5999adaa0f9d28ba6f28701f8404e3becdb545755cff05e04
Ling-3.0-flash-AD-Q5_K_M-00003-of-00003.gguf	28,392,125,088	26.44	277	63916691d10600f2986f7872e6b9cd4a777c220533bb728b29391194eb23c384

The shards of AD-Q4_K_S sum to 192 bytes more than the unsharded file, and those of AD-Q5_K_M to 320 bytes more, because gguf-split adds its split metadata and a header to every shard. Their tensor names together are exactly the original 938, in the original order, with no duplicates.

The unsharded files, for reference: AD-Q4_K_S is 75,981,701,568 bytes with sha256 5210843db2e5b15784d6379d6c9aed80b7db99fb7b43f4ef7723778dc99a2d8c; AD-Q5_K_M is 91,576,648,128 bytes with sha256 0f05755aa0141ba9b174d30e02e8a007d2c53b738d60453c8e762782ca9da047. They are not uploaded here, because both exceed the Hugging Face 50 GB per-file limit. You can recombine the shards with llama-gguf-split --merge if you need to verify them, or just point llama-server at shard 1; it follows the split automatically.

---

Usage

llama-server \
  --model Ling-3.0-flash-AD-Q5_K_M-00001-of-00003.gguf \
  --ctx-size 131072 \
  --jinja

Use the first shard of whichever file you downloaded (...-AD-Q4_K_S-00001-of-00002.gguf or ...-AD-Q5_K_M-00001-of-00003.gguf). Pass only the first shard. llama.cpp reads split.count from its metadata and opens the rest itself.

Requires llama.cpp v0.5.0 / b11146 or newer. v0.5.0 is also the first release with the dedicated Ling 3.0 (Bailing V3) chat parser (upstream #28682, merged 2026-09-19), so tool calling works correctly there.

---

Performance

Measured with llama-bench (build b11146, Vulkan/RADV) on an AMD Ryzen AI Max+ 395 (Strix Halo, Radeon 8060S, gfx1151, 128 GB unified memory). Both files ran on the same machine, one at a time, with nothing else on the GPU:

llama-bench -m <shard 1> -p 2048 -n 128 -d 0,32768,65536,131072 -r 2 -fa 1 -ngl 999 -ctk <kv> -ctv <kv>

Prefill is pp2048 and decode is tg128, both in tokens/s, at the given depth of context already in the cache. Each value is the mean of 2 repetitions.

KV cache f16:

depth	AD-Q4_K_S prefill	AD-Q4_K_S decode	AD-Q5_K_M prefill	AD-Q5_K_M decode
0	499.6	34.11	450.7	32.95
32,768	279.3	28.28	264.5	27.44
65,536	194.1	24.18	186.4	23.54
131,072	120.8	18.77	118.3	18.41

KV cache q8_0:

depth	AD-Q4_K_S prefill	AD-Q4_K_S decode	AD-Q5_K_M prefill	AD-Q5_K_M decode
0	482.2	33.99	436.9	32.80
32,768	167.3	25.00	161.8	24.36
65,536	74.5	19.64	75.0	19.25
131,072	40.4	13.66	40.2	13.48

Peak GPU memory (GTT) during each run, at 133,120 tokens of context: AD-Q4_K_S 70.2 GiB (q8_0) and 70.6 GiB (f16); AD-Q5_K_M 84.4 GiB and 84.8 GiB.

On this hardware AD-Q5_K_M costs 1 to 4% of decode speed and up to 10% of prefill for about 14 GiB more memory. A q8_0 KV cache saves little memory here and is much slower at long context, so use f16.

What is not measured

There is no quality benchmark for these files. They have not been evaluated against bartowski's plain IQ4_XS, and they have not been evaluated against AtomicChat's own AD- files. Those cannot run on the stock build at all, which is why this repo exists and also why a head-to-head is not available.

Nothing here should be read as a measured quality win over any other quant of this model. AtomicChat publish a KL table for their AD line, but their controls are their own stock builds, so that result does not transfer to these files either.

What is verified is that both files load on stock llama.cpp (a CPU-only smoke test on build b11146), and that their tensor types and shapes match AtomicChat's own files. AD-Q4_K_S answered both smoke prompts correctly. AD-Q5_K_M hit the 120-token limit mid-answer on the first (391 appeared only in reasoning_content) and answered the second (a Fibonacci function) correctly. AtomicChat's KL numbers for AD-Q5_K_M versus AD-Q4_K_S were measured on their fork and do not transfer to these files until measured here.

The NextN/MTP block

These files carry the NextN / multi-token-prediction block (blk.42, 21 tensors, about 1.76 GB in AD-Q4_K_S and about 2.14 GB in AD-Q5_K_M), because the bartowski source does. AtomicChat's own files do not: they have 917 tensors, and their card says the converter drops this block ("MTP / speculative decoding is not wired up"). The block's expert tensors are quantized with the same rule as the other routed experts (Q4_K in AD-Q4_K_S, Q5_K in AD-Q5_K_M). llama.cpp ignores the block unless you pass --spec-type draft-mtp, and logs 21 unused tensor warnings on load. Those warnings are harmless. Nothing is wrong with the file.

If you are tempted to turn MTP on: on the hardware above, a controlled single-variable test measured decode −49% at 57.7k and −64% at 115.7k of context versus MTP off, despite healthy draft acceptance rates (0.55 and 0.50). That is a property of this llama.cpp version on this hardware, not a property of the quant, and it may well not reproduce on a discrete GPU. But it is the reason MTP is left off where these files are deployed.

---

Licence

MIT, inherited from inclusionAI/Ling-3.0-flash.