laya-pdpa-onnx-v4-32k

ONNX export of Suarify/laya-pdpa-my-v4 — a Malaysian PDPA classifier for call transcripts. Ten typed questions per call: eight yes/no classes, one ordered severity, one choice for who disclosed.

Exported with torch.onnx.export(dynamo=True), opset 17, external data.

Variants

file	size	accuracy	1 question, 4 vCPU
onnx/model.onnx	602 MB	85.9%	1622 ms
onnx/model_q4f16.onnx	192 MB	85.9% (same total, 46 different answers — see below)	1759 ms
onnx/model_fp16.onnx	302 MB	does not load in this build	—

Accuracy is over all 960 held-out decisions (96 rows × 10 questions), by argmax, against a majority-class baseline of 72.8%.

Parity of the fp32 export against the PyTorch checkpoint is 1.07e-05 max absolute logit difference across batches 1, 3, 8 and 16.

q4f16 scores the same total and does not give the same answers

fp32 and q4f16 both score 825/960. That identical total is a coincidence of cancellation, not agreement: the two builds disagree on 46 of 960 decisions, with up to 10.15 absolute logit difference, and the flips happen to break 23 each way.

The disagreements are not spread evenly. They land on the two questions this model exists for:

question	decisions that flip under 4-bit
pdpa_actor	25 of 96 (26%)
pdpa_severity	10 of 96 (10%)
pdpa_health	5 of 96
pdpa_bank_card	3 of 96
pdpa_mykad, pdpa_phone, pdpa_email	1 each

Both multi-class heads read a 3- or 4-way argmax over close logits, so 4-bit weights move the decision far more often than they do on a binary yes/no. Aggregate accuracy hides this completely.

So choose on what you need. For a queue-ranking or actor-attribution decision you intend to act on, use onnx/model.onnx (602 MB) — the 192 MB build agrees with it on only 74% of actor calls. For the eight binary classes, q4f16 is within one decision of fp32 on all but pdpa_bank_card and is the obvious choice in a browser.

An earlier version of this card said 4-bit "flipped zero decisions". That was read off the matching accuracy totals before the per-decision comparison was run, and it was wrong.

The fp16 file in this build fails to load with a type error. The cause is now known and fixed in the exporter, so a future export will ship a working one: five nodes in this graph end up with genuinely mixed inputs, and convert_float_to_float16 has to be told to leave them alone so it inserts the casts around them. It converges to 302 MB that loads and whose argmax matches fp32. For a browser, use model_q4f16.onnx; for severity or actor decisions you act on, read the next section first.

Pruned vocabulary

The base checkpoint carries a 256,000-token multilingual vocabulary, which made the embedding table 61% of its weights. MatMulNBitsQuantizer skips Gather, so quantisation never touched it and the full-vocab q4f16 build still weighed 880 MB.

This export keeps 32,000 tokens, chosen on the train and validation splits only, and slices the embedding to match: 256,000 → 32,000 rows, 322M → 150M parameters.

embedding          fp32      98 MB    fp16     49 MB
everything else    fp32     501 MB    4-bit    70 MB

The keep-set is script-aware. Ids below 32,000 hold 28,564 Latin tokens but only 571 CJK and 27 Tamil, so keeping the lowest ids would leave Chinese and Tamil encoding as raw bytes. Every single-character CJK and Tamil token is kept, and byte fallback is retained, so no input is ever unencodable.

The tokenizer in this repo is not interchangeable with the base one. Token ids differ. Use tokenizer.json from here or every answer is read from wrong rows.

Inputs

input_ids       int64  [batch, seq]
attention_mask  int64  [batch, seq]
marker_pos      int64  [batch, options]   index of each [MASK] slot
marker_mask     bool   [batch, options]
qtype           int64  [batch]            0=choice 1=score 2=noul
->  logits      float  [batch, options]

This is not a standard sequence classifier and transformers.js cannot load it. The sequence must be built as laya.common.build_sequence does:

[CLS] <type> question: <ins> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]

with marker_pos carrying each [MASK] index. Special tokens are mmBERT's <bos>/<eos>/<mask>/<pad> = 2/1/4/0, not [CLS]/[SEP]. Option order is label order — get it wrong and every answer is silently relabelled, not an error. A working single-file browser port is in the source repository.

Reading the ordered score question: take argmax over the option logits. Do not truncate a continuous expectation — int(2.84) is 2, and that one mistake reported severity at 22.9% for four versions when it scores 75.0%.

Calibration

temperature (choice, score, noul) = [1.0323784351348877, 1.0251051187515259, 1.1000169515609741]

All three fitted inside laya's [0.5, 5] clamp, so these confidences are genuinely calibrated rather than saturated. Earlier checkpoints shipped [10.0, 10.0, 7.916], which is clamped on load with a warning — accuracy was unaffected, because temperature does not move argmax, but the probabilities could not be used.

Limitations

All training data is synthetic; no real call has ever been labelled. Disclosure prevalence in the corpus is far above production, so precision here is optimistic by roughly an order of magnitude.

The gain over a majority-class predictor is about fourteen points, not eighty-six. Three of the eight binary classes — pdpa_email, pdpa_mykad, pdpa_credential_otp — have confidence intervals containing their own baseline, and pdpa_bank_card is below its. A page of regexes beats this model on all three. What it is actually for is pdpa_severity (+34.4 over baseline) and pdpa_actor (+29.1), which no pattern can express: ranking a queue, and telling an agent soliciting an NRIC from a caller volunteering one unprompted.

See the base model card for the per-question table, and for two published figures that turned out to be wrong in the evaluation rather than in the model.