ONNX export of CohereLabs/cohere-transcribe-03-2026 (nemo-conformer-aed)

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+encoder-model.int8.onnx.data filter=lfs diff=lfs merge=lfs -text
+encoder-model.onnx.data filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,107 @@
+---
+license: apache-2.0
+language: [ar, de, el, en, es, fr, it, ja, ko, nl, pl, pt, vi, zh]
+tags: [automatic-speech-recognition, onnx, onnx-asr, nemo-conformer-aed, cohere]
+base_model: CohereLabs/cohere-transcribe-03-2026
+pipeline_tag: automatic-speech-recognition
+library_name: onnx-asr
+---
+
+# Cohere Transcribe 2B — ONNX (nemo-conformer-aed)
+
+ONNX export of [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
+for [onnx-asr](https://github.com/istupakov/onnx-asr).
+
+**Stock onnx-asr loads this model. No patches and no new model family are needed.**
+
+Despite the "2B ASR" framing, `cohere_asr` is not a speech-LLM. It is an attention
+encoder-decoder built on a NeMo FastConformer: `CohereAsrConfig` declares
+`sub_configs = {"encoder_config": ParakeetEncoderConfig}`, and its decoder prompt is
+token-for-token the NVIDIA Canary prompt. It therefore drops straight into the existing
+`nemo-conformer-aed` family that already serves Canary.
+
+## Usage
+
+```python
+import onnx_asr
+
+model = onnx_asr.load_model("nemo-conformer-aed", "path/to/this/repo")   # or quantization="int8"
+print(model.recognize("audio_16khz.wav", language="en"))
+```
+
+`language` accepts any of the 14 supported codes: `ar de el en es fr it ja ko nl pl pt vi zh`.
+`pnc=True|False` toggles punctuation and capitalisation.
+
+## Graph contract
+
+Two graphs, drop-in compatible with `istupakov/canary-1b-v2-onnx`:
+
+| graph | inputs | outputs |
+|---|---|---|
+| `encoder-model.onnx` | `audio_signal` [B,128,T] f32, `length` [B] i64 | `encoder_embeddings` [B,T/8,1024] f32, `encoder_mask` [B,T/8] i64 |
+| `decoder-model.onnx` | `input_ids` [B,C] i64, `encoder_embeddings` [B,E,1024] f32, `encoder_mask` [B,E] i64, `decoder_mems` [9,B,P,1024] f32 | `logits` [B,C,16384] f32 (log-softmax), `decoder_hidden_states` [9,B,P+C,1024] f32 |
+
+Two details make the zero-patch fit work:
+
+- The 1280 -> 1024 `decoder.proj` linear is **folded into the encoder output**, so the decoder
+  graph sees 1024-dim memories exactly as the contract requires.
+- The HF `CohereAsrDecoder` uses a standard transformers KV cache. The export re-expresses it as
+  NeMo-style `decoder_mems` (per-layer pre-layernorm hidden states, `num_layers + 1 = 9` entries),
+  which is what the onnx-asr AED decode loop drives.
+
+No feature extractor is baked into the graph. `CohereAsrFeatureExtractor` turned out to be the
+standard NeMo log-mel front end — dither 1e-5, preemphasis 0.97, n_fft 512 / win 400 / hop 160,
+symmetric Hann, 128 slaney mels, `log(x + 2**-24)`, per-feature normalisation — so onnx-asr's
+built-in `nemo128` preprocessor already matches it, down to the frame-count formula.
+
+## Files
+
+| file | size |
+|---|---|
+| `encoder-model.onnx` + `.data` | 7.59 GB |
+| `decoder-model.onnx` | 676 MB |
+| `encoder-model.int8.onnx` + `.data` | 1.91 GB |
+| `decoder-model.int8.onnx` | 170 MB |
+
+fp32 total 8.3 GB, int8 total 2.1 GB.
+
+## Accuracy
+
+Four FLEURS clips (2 en, 2 pt), greedy decoding, against native `transformers` on the same clips.
+
+| build | token-identical to native |
+|---|---|
+| fp32 | **4 / 4** |
+| int8 | 2 / 4 |
+
+fp32 is exact. int8 dynamic quantisation costs a little accuracy — observed drift is a spurious
+comma and a mis-spelled rare proper noun. Session-swapping shows both graphs contribute (encoder
+int8 alone: 3/4 drift; decoder int8 alone: 2/4 drift), so there is no single subgraph to exclude.
+Use fp32 when accuracy matters and int8 when size matters.
+
+Note: onnx-asr's built-in detokeniser drops the space before an opening bracket or quote
+(`sugar(especially`). That is upstream onnx-asr behaviour shared with Canary, not an export
+defect — the token ids are identical.
+
+## Speed
+
+AMD Ryzen 5 7600 (6 cores / 12 threads), CPU execution provider, `OMP_NUM_THREADS=6`, `nice -n 10`.
+
+| clip | duration | fp32 RTF | int8 RTF |
+|---|---|---|---|
+| en_1 | 6.5 s | 0.178 | 0.186 |
+| en_2 | 16.4 s | 0.355 | 0.186 |
+| pt_1 | 11.8 s | 0.348 | 0.214 |
+| pt_2 | 14.6 s | 0.301 | 0.186 |
+| **mean** | | **0.30** | **0.19** |
+
+## Scope
+
+Single-clip transcription only. The source processor splits audio longer than 35 s at
+low-energy boundaries and stitches the pieces back together; that chunking is not part of this
+export. Feed clips under about 30 s, or segment them yourself.
+
+## Attribution
+
+Source model and weights: **Cohere Labs**, [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026), Apache-2.0.
+This repository contains only an ONNX conversion; the license is inherited unchanged.

REPORT.md

@@ -0,0 +1,199 @@
+# cohere-transcribe-03-2026 -> ONNX export report
+
+Date: 2026-08-03. Machine: `192.168.1.200` (AMD Ryzen 5 7600, 6C/12T, 124 GB RAM).
+Staging: `/media/hdd16/onnx-asr-exports/cohere-transcribe-2b/`.
+Venv: `/home/miro/cohere-export-venv` (python 3.12, transformers 5.14.1, torch 2.13.0+cpu,
+onnx 1.22, onnxruntime 1.23, numpy pinned `<2.5` because numba/librosa reject 2.5).
+
+## 1. Architecture found
+
+Not a speech-LLM. `cohere_asr` is a plain attention encoder-decoder (AED), and specifically a
+NeMo Canary derivative retrained by Cohere Labs.
+
+| part | detail |
+|---|---|
+| encoder | `ParakeetEncoderConfig` (NeMo FastConformer). 48 layers, d_model 1280, 8 heads, FFN 5120, SiLU, conv kernel 9, subsampling factor 8, 128 mel bins. ~1.9 B params. |
+| projection | one `nn.Linear(1280, 1024)` at `model.decoder.proj`, applied to encoder states before cross-attention |
+| decoder | 8 layers, d_model 1024, FFN 4096, ReLU, pre-LN, learned `pos_emb` + `embedding_layernorm`, self-attn + cross-attn, untied `proj_out`, vocab 16384. ~0.17 B params |
+| front end | `CohereAsrFeatureExtractor` — standard NeMo log-mel |
+| tokenizer | SentencePiece, 16384 tokens |
+| languages | ar de el en es fr it ja ko nl pl pt vi zh |
+
+Evidence of the Canary lineage:
+
+- `config.json` is a verbatim NeMo `EncDecMultiTaskModel` config (`preprocessor`, `encoder`,
+  `transf_decoder`, `head`, `prompt_format`, `prompt_defaults`).
+- The special-token id block is identical to `istupakov/canary-1b-v2-onnx/vocab.txt`:
+  0 `<unk>`, 1 `<|nospeech|>`, 2 `<pad>`, 3 `<|endoftext|>`, 4 `<|startoftranscript|>`,
+  5 `<|pnc|>`, 6 `<|nopnc|>`, 7 `<|startofcontext|>`, 8 `<|itn|>`, 9 `<|noitn|>`,
+  10 `<|timestamp|>`, 11 `<|notimestamp|>`, 12 `<|diarize|>`, 13 `<|nodiarize|>`.
+- Vocab size 16384, same as Canary.
+- `processing_cohere_asr.get_decoder_prompt_ids("en")` returns
+  `[13764, 7, 4, 16, 62, 62, 5, 9, 11, 13]`, where 13764 is `_`. That is token-for-token
+  `NemoConformerAED._transcribe_input` in onnx-asr.
+
+## 2. Runtime changes
+
+**None.** `onnx-asr` `main` (both our fork and upstream) already registers `nemo-conformer-aed`
+and the `nemo128` preprocessor. No branch was created and no repo file was touched.
+
+The feature extractor did not need baking into the graph. `CohereAsrFeatureExtractor` matches
+`NemoPreprocessorNumpy` / `nemo128.onnx` on every constant that matters:
+
+| | Cohere FE | onnx-asr nemo |
+|---|---|---|
+| preemphasis | 0.97, padding masked | 0.97, padding masked |
+| n_fft / win / hop | 512 / 400 / 160 | 512 / 400 / 160 |
+| window | `torch.hann_window(400, periodic=False)` | `np.hanning(400)` (same symmetric window) |
+| centering | `center=True`, `pad_mode="constant"` | explicit zero pad of `n_fft//2` |
+| mels | librosa slaney, 128, fmin 0, fmax 8000 | `melscale_fbanks(..., "slaney", "slaney")`, 128 |
+| log guard | `2**-24` | `2**-24` |
+| normalisation | per-feature, `n-1` variance, `+1e-5` | per-feature, `n-1` variance, `+1e-5` |
+| frame count | `(L + 2*(n_fft//2) - n_fft) // hop` | `L // hop` (algebraically identical) |
+
+Only difference: Cohere applies a seeded dither of 1e-5 that onnx-asr omits. Empirically it
+changes nothing — fp32 output is token-identical on every clip tested.
+
+## 3. Graph contract
+
+Matches `istupakov/canary-1b-v2-onnx` exactly, so the stock decode loop drives it unmodified.
+
+```
+encoder-model.onnx
+  in : audio_signal [batch, 128, seq_len] f32
+       length       [batch] i64
+  out: encoder_embeddings [batch, encoded_len, 1024] f32
+       encoder_mask       [batch, encoded_len] i64
+
+decoder-model.onnx
+  in : input_ids          [batch, input_len] i64
+       encoder_embeddings [batch, encoded_len, 1024] f32
+       encoder_mask       [batch, encoded_len] i64
+       decoder_mems       [9, batch, mems_len, 1024] f32
+  out: logits                 [batch, input_len, 16384] f32, log-softmax
+       decoder_hidden_states  [9, batch, mems_len + input_len, 1024] f32
+```
+
+Two export decisions:
+
+1. `model.decoder.proj` (1280 -> 1024) is folded into the encoder wrapper's output. The contract
+   fixes the memory width at 1024; Cohere's encoder is 1280 wide. Folding is the only way to keep
+   the decoder graph unchanged.
+2. HF's `EncoderDecoderCache` is re-expressed as NeMo `decoder_mems`: element *i* holds the
+   pre-layernorm hidden states entering layer *i*, element 0 is the embedding output, and there
+   are `num_layers + 1 = 9` of them (Canary has 10 because it has 9 layers; the runtime reads the
+   count from the graph, so this is fine). Because LayerNorm is per-position,
+   `LN(concat(past, new)) == concat(LN(past), LN(new))`, so caching pre-LN states is exact.
+
+Cross-attention K/V are recomputed from `encoder_embeddings` each step, as in the Canary export.
+
+`config.json` shipped alongside is just `{"model_type": "nemo-conformer-aed", "features_size": 128}`.
+`vocab.txt` is written in onnx-asr's `"<token> <id>"` form with raw SentencePiece `_`; the loader
+maps `_` to a space, which is what makes `self._tokens[" "]` resolve to 13764.
+
+## 4. Parity
+
+Clips: 4 FLEURS clips copied from the qwen3-asr export (`en_1`, `en_2`, `pt_1`, `pt_2`).
+Both English and Portuguese are in the model's 14 supported languages.
+
+**Stage 1 — torch wrappers vs native `model.generate`** (proves the mems rewrite before ONNX
+enters the picture): 4 / 4 character-exact.
+
+**Stage 2 — ONNX fp32 through stock onnx-asr vs native, compared at the token-id level:**
+
+| clip | tokens | identical |
+|---|---|---|
+| en_1 | 18 | yes |
+| en_2 | 69 | yes |
+| pt_1 | 41 | yes |
+| pt_2 | 54 | yes |
+
+**4 / 4 token-identical.** Target met.
+
+The rendered strings differ from native in one respect only: onnx-asr's detokeniser regex
+`\A\s|\s\B|(\s)\b` drops a space before an opening bracket or quote, giving `sugar(especially`
+and `como"squimbans"`. Token ids are identical, so this is upstream onnx-asr behaviour shared
+with Canary, not an export defect.
+
+**Stage 3 — int8:** 2 / 4 token-identical. Drift seen:
+
+- `en_1`: a spurious comma (`miles away, you are`)
+- `pt_2`: `"squimbans"` -> `"squimãs"`
+
+Session-swap isolation:
+
+| build | clips differing from ONNX fp32 |
+|---|---|
+| encoder int8 + decoder fp32 | 3 / 4 |
+| encoder fp32 + decoder int8 | 2 / 4 |
+| both int8 | 2 / 4 (errors partially cancel) |
+
+Both graphs contribute; there is no single offending subgraph to exclude from quantisation (and
+no in-graph feature extractor here, since the FE stayed external). int8 is shipped with the
+accuracy caveat documented rather than suppressed.
+
+## 5. Speed
+
+AMD Ryzen 5 7600 (6C/12T), CPUExecutionProvider, `OMP_NUM_THREADS=6`, `nice -n 10`, one warm-up
+pass discarded.
+
+| clip | duration | fp32 RTF | int8 RTF |
+|---|---|---|---|
+| en_1 | 6.54 s | 0.178 | 0.186 |
+| en_2 | 16.38 s | 0.355 | 0.186 |
+| pt_1 | 11.82 s | 0.348 | 0.214 |
+| pt_2 | 14.58 s | 0.301 | 0.186 |
+| mean | | 0.30 | 0.19 |
+
+int8 is roughly 1.6x faster on the longer clips. On the shortest clip the two are within noise.
+
+## 6. Sizes
+
+| artifact | bytes |
+|---|---|
+| `encoder-model.onnx` | 1 794 752 |
+| `encoder-model.onnx.data` | 7 586 112 512 |
+| `decoder-model.onnx` | 676 100 265 |
+| `encoder-model.int8.onnx` | 3 239 179 |
+| `encoder-model.int8.onnx.data` | 1 902 071 040 |
+| `decoder-model.int8.onnx` | 169 764 577 |
+| `vocab.txt` | 207 437 |
+
+fp32 8.26 GB, int8 2.08 GB.
+
+## 7. Deviations and known limits
+
+- **Long audio is out of scope.** `CohereAsrFeatureExtractor` splits anything over 35 s at
+  low-energy boundaries (5 s search window) and `CohereAsrProcessor.decode` restitches the
+  pieces. onnx-asr has no equivalent hook, so this export handles single clips only. Documented
+  in the model card.
+- **int8 is not exact** — see section 4. fp32 is the accuracy build.
+- **Detokeniser spacing** around opening brackets/quotes, inherited from onnx-asr.
+- **Beam search not exported.** The source `config.json` asks for `beam_size: 1`, i.e. greedy, so
+  the onnx-asr greedy loop is faithful to the shipped decoding config.
+- **Upstream packaging bug spotted in passing:** the `onnx-asr` 0.12.0 wheel on PyPI is not
+  importable. Its `loader.py` does `from onnx_asr.models.speech_llm import SpeechLlm` and
+  `from onnx_asr.models.wav2vec2 import Wav2Vec2Ctc`, but neither module is in the wheel's
+  RECORD. Testing here was done against our fork's `main` instead. Worth reporting upstream.
+- **HF token vs HF_HOME:** with `HF_HOME=/media/hdd16/amalia-hf-cache` the token at
+  `~/.cache/huggingface/token` is not found. Remote shells need
+  `export HF_TOKEN=$(cat ~/.cache/huggingface/token)` as well.
+- The source repo is **gated** (`gated: "auto"`). A human had to accept the terms as `Jarbas`
+  before anything could be downloaded.
+
+## 8. Scripts
+
+Kept in `/home/miro/` on `.200`:
+
+- `export_cohere_asr.py` — the two graph wrappers and the export driver
+- `verify_torch.py` — stage-1 torch equivalence
+- `verify_onnx_cohere.py` — text + RTF through stock onnx-asr
+- `tokencmp.py` — token-id parity, native vs ONNX
+- `mixcmp.py` — session-swap isolation
+- `consolidate.py` — collapse the encoder's loose external tensors into `encoder-model.onnx.data`
+- `quantize.py` — dynamic int8
+
+## 9. Published
+
+`OpenVoiceOS/cohere-transcribe-2b-onnx`, Apache-2.0, attribution to Cohere Labs.
+Added to collection `OpenVoiceOS/stt-asr-onnx-699321e8732462509c642fbe`.

config.json

@@ -0,0 +1,4 @@
+{
+    "model_type": "nemo-conformer-aed",
+    "features_size": 128
+}

decoder-model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27f798f1df6aa7c64cdf9a79b1151a3745b8aac0a9da4e42186e073358733618
+size 169764577

decoder-model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be826a79ec81c90217b3742de9f915aa9f77dea808b5705814c6cf1da510402f
+size 676100265

encoder-model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679e2423ede014219ad0ce555d863a9e3cd36fc08ce9f1fb761958a9417de8a
+size 3239179

encoder-model.int8.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13da58e80b5989b0e3e39a02b0b216c62a7d61677660122cbff81d95dbc8c26b
+size 1902071040

encoder-model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26aafa10446f753498667c20089acf8679c8526dbf3beb2d6793d2b016af7e39
+size 1794752

encoder-model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5cde4b754acb8071c286804b8cb1e48d193bdd99d61073b200b39e3107d3a6e
+size 7586112512