--- license: mit language: - ar - az - bg - bn - bs - ca - cs - da - de - el - en - eo - es - et - fa - fi - fr - he - hi - hr - hu - hy - id - it - ja - ka - kk - ko - ku - lt - lv - mk - mn - mr - ms - my - nb - nl - pl - pt - ro - ru - sk - sl - sq - sr - sv - sw - ta - th - tr - uk - ur - vi - zh - af - be - cy - eu - ga - gl - is - kn - ml - ne - nn - si - te - tl - uz - gu - pa - km - la - lb - mt - so - am - ky pipeline_tag: token-classification tags: - punctuation - punctuation-restoration - truecasing - capitalization - asr - speech-recognition-postprocessing - multilingual - ensemble - modernbert - xlm-roberta base_model: - jhu-clsp/mmBERT-base - FacebookAI/xlm-roberta-large base_model_relation: finetune datasets: - wikimedia/wikipedia - Helsinki-NLP/europarl - HPLT/HPLT2.0_cleaned - facebook/voxpopuli - CohereLabs/aya_dataset metrics: - f1 model-index: - name: Dewpoint results: - task: type: token-classification name: Punctuation restoration dataset: name: IWSLT 2017 TED, official tst2010-2015 (9 languages) type: IWSLT/iwslt2017 metrics: - type: f1 value: 0.7559 name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages - type: f1 value: 0.6557 name: Comma F1 - type: f1 value: 0.7923 name: Period F1 - type: f1 value: 0.8197 name: Question F1 - task: type: token-classification name: Punctuation restoration dataset: name: WIT3 TED, same 90 talks (57 language tracks) type: wit3 metrics: - type: f1 value: 0.7303 name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages - type: f1 value: 0.6482 name: Comma F1 - type: f1 value: 0.7794 name: Period F1 - type: f1 value: 0.7632 name: Question F1 - task: type: token-classification name: Punctuation restoration dataset: name: IWSLT TED through TTS and Whisper ASR (33 languages) type: iwslt-ted-asr metrics: - type: f1 value: 0.6332 name: Macro-F1 (COMMA, PERIOD, QUESTION), mean over languages - type: f1 value: 0.5656 name: Comma F1 - type: f1 value: 0.7102 name: Period F1 - type: f1 value: 0.6238 name: Question F1 --- ![Dewpoint: punctuation and capitals for the spoken word](assets/banner.jpg) # Dewpoint **Punctuation and truecasing for speech-recognition output, in 79 languages.** Speech recognisers hand back a stream of lowercase words with no punctuation, and Dewpoint restores it: it adds the commas, full stops and question marks, and the capitals. Open weights, MIT licence, free for everyone. ```text so i said meet at three thirty tuesday what do you think the iphone was announced by steve jobs in san francisco → So I said meet at three thirty Tuesday. What do you think? The iPhone was announced by Steve Jobs in San Francisco. ich weiß nicht ob wir das schaffen aber wir sollten es versuchen was meinst du → Ich weiß nicht, ob wir das schaffen, aber wir sollten es versuchen. Was meinst du? πού είναι ο σταθμός ξέρεις να μου πεις → Πού είναι ο σταθμός; Ξέρεις να μου πεις; ``` These are unedited outputs of this checkpoint. Dewpoint is a **tagger, not a generator**. For each word it decides which mark follows (none, `,` `.` `?` `!`) and how the word is cased (lower, Capitalised, UPPER). It never adds, drops, reorders or rewrites a word, so it cannot invent content. ## Results On the IWSLT TED benchmark, Dewpoint scores higher than every other multilingual punctuation model we tested, under all three test conditions. On the two text test sets it is behind no other model in any language: **0 of 9** on the official set and **0 of 57** on the extended one. Under ASR noise it trails in **7 of 33**, 5 of them to raw Whisper. ![Macro-F1 on three benchmarks](assets/benchmarks.jpg) | system | params | official, 9 lang | WIT3, 57 lang | ASR, 33 lang | |---|---|---|---|---| | **Dewpoint** | 869M | **0.7559** | **0.7303** | **0.6332** | | Dewpoint, mmBERT-base half alone | 307M | 0.7392 | 0.7098 | 0.6197 | | Dewpoint, XLM-R-large half alone | 561M | 0.7443 | 0.7241 | 0.6208 | | [oliverguhr/fullstop-punctuation-multilang-large](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large) | 559M | 0.6438 | 0.5984 | 0.5301 | | [kredor/punctuate-all](https://huggingface.co/kredor/punctuate-all) | 278M | 0.6344 | 0.5950 | 0.5333 | | [oliverguhr/fullstop-punctuation-multilingual-sonar-base](https://huggingface.co/oliverguhr/fullstop-punctuation-multilingual-sonar-base) | 278M | 0.6381 | 0.5653 | 0.5205 | | [1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase](https://huggingface.co/1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase) | 278M | 0.6204 | 0.5710 | 0.5346 | | raw [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) punctuation | 809M | — | — | 0.4436 | Macro-F1 over COMMA, PERIOD and QUESTION, averaged over languages. Every system runs through the same harness and gets the same dev-set threshold calibration. **Head to head on each model's own languages.** The averages above include languages the other models were never trained on, which pulls their means down. Scored only on the languages each model lists on its own model card, the margins are smaller: | other model | languages it supports | official | WIT3 | ASR | |---|---|---|---|---| | [oliverguhr/fullstop-punctuation-multilang-large](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large) | 4 | 0.755 vs **0.791** (4) | 0.758 vs **0.793** (5) | 0.741 vs **0.772** (4) | | [oliverguhr/fullstop-punctuation-multilingual-sonar-base](https://huggingface.co/oliverguhr/fullstop-punctuation-multilingual-sonar-base) | 5 | 0.734 vs **0.786** (5) | 0.752 vs **0.797** (6) | 0.702 vs **0.753** (5) | | [kredor/punctuate-all](https://huggingface.co/kredor/punctuate-all) | 12 | 0.726 vs **0.786** (5) | 0.733 vs **0.797** (14) | 0.695 vs **0.755** (9) | | [1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase](https://huggingface.co/1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase) | 47 | 0.620 vs **0.756** (9) | 0.598 vs **0.744** (37) | 0.540 vs **0.636** (27) | Dewpoint is ahead in every one of these languages except Bengali on Whisper ASR output against xlm-r truecase. Each cell is that model's mean macro-F1 vs Dewpoint's on the same languages, with the number of languages in brackets. This is the fairer comparison. --- ## Quick start ```bash pip install torch transformers safetensors huggingface_hub numpy # GPU or CPU # or, with no torch at all: pip install onnxruntime tokenizers huggingface_hub numpy ``` ```python import os, sys from huggingface_hub import hf_hub_download sys.path.insert(0, os.path.dirname(hf_hub_download("valkayuh/dewpoint", "dewpoint.py"))) from dewpoint import Punctuator p = Punctuator.from_pretrained("valkayuh/dewpoint") # torch if installed, else ONNX p.restore("what time is it in tokyo right now", lang="en") # 'What time is it in Tokyo right now?' ``` `from_pretrained` downloads only what the chosen backend needs. The torch backend fetches the safetensors (3.5 GB, or 1.2 GB with `members=["mmbert-base"]`), the ONNX backend fetches the ONNX graphs, and neither fetches the other. **Always pass the language.** It selects the calibrated decision thresholds, turns casing off for scripts that have none, and picks the right marks to write: `؟` for Arabic, `;` for the Greek question mark, `。` and `，` for Chinese, `।` for Hindi. **Single-model mode.** Pass `members=["mmbert-base"]` to download and run only the 307M-parameter mmBERT-base half. It uses its own calibration, runs in about half the GPU time, and still beats every other model we tested on every benchmark: ```python p = Punctuator.from_pretrained("valkayuh/dewpoint", members=["mmbert-base"]) ``` **Streaming.** For live captions, each word is released only once it has right-hand context and a stable label, so marks don't flicker: ```python s = p.stream(lang="en", lag=6) for chunk in asr_chunks: # partial transcripts as they arrive print(s.push(chunk), end="", flush=True) print(s.finish()) ``` **Labels and probabilities.** `p.predict(words, lang)` returns per-word labels plus the averaged posteriors (`punct_probs`, `case_probs`), in case you want confidence scores or your own decision rule. **Command line.** ```bash echo "i called the office this morning but nobody answered did you get my message" | python dewpoint.py --lang en # I called the office this morning, but nobody answered. Did you get my message? ``` ## Formats and integration | you want | use | |---|---| | Python with a GPU | `Punctuator(..., backend="torch")`: bf16 on CUDA, fp32 on CPU | | Python without torch, e.g. a slim CPU server | `Punctuator(..., backend="onnx")`: needs only `onnxruntime`, `tokenizers`, `numpy` | | a local HTTP API | `python serve.py` gives `POST /restore {"text", "lang"}` (standard library only) | | a hosted API | deploy this repo as a [Hugging Face Inference Endpoint](https://huggingface.co/inference-endpoints); `handler.py` serves `{"inputs", "parameters": {"lang"}}` | | the shell | `python dewpoint.py --lang de "…"` or pipe text on stdin | | C#, Java, JavaScript, C++, mobile | the ONNX graphs in `onnx/`, with the pre- and post-processing below | **ONNX graphs.** There is one per member, in `onnx/mmbert-base/` and `onnx/xlm-roberta-large/` (the latter keeps its weights in `model.onnx.data` beside the graph). Both are fp32, opset 17, with dynamic batch and sequence axes: - inputs: `input_ids`, `attention_mask`, int64 `[batch, seq]`, from that member's `tokenizer.json` with the words passed pre-tokenised - outputs: `punct_logits` `[batch, seq, 5]` and `case_logits` `[batch, seq, 3]` Around the graph, `dewpoint.py` does four things, and a port needs the same four: 1. split the input into overlapping windows of at most 508 subwords, keeping each window's centre (`plan_windows`) 2. read each word's label from its first subword 3. average the two members' softmax posteriors 4. add the calibrated per-class bias from `ensemble_config.json` for the language, then take the argmax **Verified.** Run over the whole official test set on CPU, the ONNX backend scores **0.7558** macro-F1 against the published 0.7559; the largest difference in any scored language is 0.0011, the gap between fp32 ONNX and the bf16 GPU run the published numbers come from. Its tokenisation matches transformers exactly on 323 documents in 57 languages. ### Speed Measured on a 584-word TED passage, after warm-up, with the code in this repo: | hardware | Dewpoint (869M) | mmBERT half (307M) | |---|---|---| | RTX 5080, torch bf16 | 45 ms (13k words/s) | 23 ms (25k words/s) | | Ryzen 7 5800X3D, torch fp32 | 2.18 s | 632 ms (1k words/s) | | Ryzen 7 5800X3D, ONNX Runtime fp32 | 2.13 s | 606 ms (1k words/s) | Input of any length is handled with overlapping 512-token windows. Only the centre of each window commits its predictions, because words near a window edge lack context and score measurably worse. --- ## Model Dewpoint is an **ensemble of two dual-head token classifiers**. Both members read the same word list and produce one posterior per word, and those posteriors are averaged. | member | encoder | architecture | parameters | of which embeddings | |---|---|---|---|---| | `mmbert-base/` | [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) | ModernBERT | 307.5M | 196.6M | | `xlm-roberta-large/` | [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) | XLM-RoBERTa | 561.0M | 256.5M | **Each member** is an encoder, then a residual shared trunk (Linear, GELU, LayerNorm), then two linear heads: - **punctuation**, 5 classes: `O` `COMMA` `PERIOD` `QUESTION` `EXCLAM` - **case**, 3 classes: `LOWER` (*meeting*), `CAP` for a capitalised first letter (*Tuesday*, *Tokyo*, a sentence's first word), and `UPPER` for a word written entirely in capitals (*NASA*, *BBC*, *DNA*). Mixed-case words such as *iPhone* are handled by a word list, described below. There is one decision per word, not per subword: the label sits on each word's first subword. Punctuation and case are learned together because they are correlated: a sentence end means a capital comes next. **Combining the members.** Both members label the same words, so their outputs line up without any tokenizer reconciliation. The average is taken in **probability space**, not log space. With a geometric mean, one confident but wrong member could veto a class; the arithmetic mean degrades gracefully instead. **Calibration.** Class weighting stops the model predicting "no punctuation" everywhere, but it buys recall at the cost of precision, so a plain argmax is not the F1-optimal rule. Instead, a per-class additive bias on the log-posterior is fitted on IWSLT dev2010, which is held out of every test table. Whether to add per-language biases on top, and how far to trust them, is chosen by cross-validation *inside* dev2010: its talks are split in half five times, and each strategy is fitted on one half and scored on the other. An unshrunk bias per language, fitted on eight talks, overfits: it was never the winning strategy, and for Dewpoint it scored worst. Dewpoint uses per-language biases pulled three quarters of the way back to the pooled one. The ensemble is calibrated as a system in its own right, and every other model in the tables went through the identical procedure, each choosing its own strategy. **Post-processing**, in `restore()` only (`predict()` returns the raw decision): - a capital after every sentence-final mark - a gazetteer of 4,896 mixed-case forms such as *iPhone*, *McDonald* and *GHz* - a closing mark if the text would otherwise end on none --- ## Evaluation - **Three test conditions:** - the IWSLT 2017 official tst2010–2015 sets, with 9 scorable languages - the WIT3 releases of the *same 90 talks* in 57 language tracks, so only the language changes - an ASR condition: TED text spoken by TTS, transcribed by Whisper large-v3-turbo, and aligned back to the reference, in 33 languages - **Metric.** Macro-F1 over COMMA, PERIOD and QUESTION, averaged over languages. EXCLAM is folded into PERIOD, because no other model emits it. Accuracy is never reported: about 87% of words carry no mark, so a model that predicts nothing is "87% accurate". - **Exclusions.** Japanese and Thai are excluded, and named here. Their TED references carry almost no punctuation (0.38% and 0.12% of tokens). - **Reference audit.** Two errors in the reference's label maps were fixed for every system alike. Armenian writes its full stop as an ASCII colon; fixing that doubled *every* system's Armenian score. And the Burmese comma U+104A had been mapped to a full stop. - **No TED data was used for training.** TED is the test set. ### Every language on the extended set ![Per-language macro-F1 on 57 WIT3 language tracks](assets/per_language_wit3.jpg) ### Under ASR noise ![Per-language gap to the best rival under ASR](assets/asr_condition_gap.jpg) 5 of the 7 are to raw Whisper (English, Spanish, French, Italian, Portuguese). Whisper punctuates from the audio itself, so it can use pauses and intonation that a text-only tagger never sees. The rest: Bengali (−0.044 to fullstop-sonar), Albanian (−0.025 to xlm-r truecase). Languages marked * in the chart have under 3,000 scored words, so their gaps are noisy. ### Per class, and truecasing ![Per-class F1](assets/per_class.jpg) Mean per-class F1 against the strongest other model: | test set | system | COMMA | PERIOD | QUESTION | |---|---|---|---|---| | IWSLT 2017 official (9 languages) | Dewpoint | **0.6557** | **0.7923** | **0.8197** | | IWSLT 2017 official (9 languages) | fullstop-large | 0.5334 | 0.7206 | 0.6772 | | WIT3, same 90 talks (57 language tracks) | Dewpoint | **0.6482** | **0.7794** | **0.7632** | | WIT3, same 90 talks (57 language tracks) | fullstop-large | 0.5067 | 0.6988 | 0.5896 | | Whisper ASR output (33 languages) | Dewpoint | **0.5656** | **0.7102** | **0.6238** | | Whisper ASR output (33 languages) | fullstop-large | 0.4489 | 0.6556 | 0.4858 | Truecasing F1 against the only other model that restores case, on cased scripts: | test set | cased languages | Dewpoint CAP | UPPER | xlm-r truecase CAP | UPPER | languages behind | |---|---|---|---|---|---|---| | IWSLT 2017 official (9 languages) | 6 | **0.8772** | **0.8748** | 0.7708 | 0.7888 | 0 | | WIT3, same 90 talks (57 language tracks) | 43 | **0.8533** | **0.8599** | 0.7173 | 0.8005 | 0 | | Whisper ASR output (33 languages) | 25 | **0.8007** | **0.7584** | 0.7136 | 0.7083 | 0 | **Reading the truecasing table.** `CAP` scores words whose first letter should be a capital: names, places, and the first word of every sentence. `UPPER` scores words written entirely in capitals, which in practice means acronyms and initialisms such as *NASA*, *BBC* or *DNA*. These are much rarer, so they get their own score and are not swamped by ordinary capitals. A single capital letter such as *I* counts as `CAP`. The truecasing comparison covers cased scripts only. On caseless scripts (Arabic, Chinese, Korean, Hindi and others) the case head is masked by design, so Latin loanwords inside them are never capitalised. `xlm-roberta_punctuation_fullstop_truecase` does capitalise them, so it wins on those words by construction. The native EXCLAM class scores 0.11 F1 on the official set and 0.07 on WIT3. No other model emits it at all. ### Size ![Parameters against macro-F1](assets/size_vs_score.jpg) ### Where the gain comes from Every other model in these tables is an XLM-RoBERTa model, so a win could come from the architecture or from the training recipe. To separate the two, we trained **XLM-R-large, the encoder `fullstop-punctuation-multilang-large` uses, on this corpus with this recipe**. ![Method against architecture](assets/method_vs_architecture.jpg) The recipe is worth **+0.08 to +0.13** macro-F1. Swapping the encoder is worth −0.004 to +0.016. The corpus, the orthography fixes and the decision rule matter roughly an order of magnitude more than the encoder. Neither encoder dominated the other, and averaging exploits exactly that: members that do not make the same errors. ### Per-language tables
IWSLT 2017 official, 9 languages | language | Dewpoint | mmBERT half alone | best other model | which | gap | |---|---|---|---|---|---| | Arabic (`ar`) | 0.6538 | 0.6229 | 0.5586 | fullstop-large | +0.095 | | Chinese (`zh`) | 0.6551 | 0.6546 | 0.5502 | xlm-r truecase | +0.105 | | Dutch (`nl`) | 0.7686 | 0.7587 | 0.7304 | fullstop-sonar | +0.038 | | English (`en`) | 0.8033 | 0.7935 | 0.7430 | fullstop-large | +0.060 | | French (`fr`) | 0.7956 | 0.7841 | 0.7581 | fullstop-large | +0.038 | | German (`de`) | 0.8496 | 0.8402 | 0.8338 | fullstop-large | +0.016 | | Italian (`it`) | 0.7145 | 0.6886 | 0.6838 | fullstop-large | +0.031 | | Korean (`ko`) | 0.8010 | 0.7825 | 0.6553 | xlm-r truecase | +0.146 | | Romanian (`ro`) | 0.7618 | 0.7274 | 0.6860 | fullstop-large | +0.076 |
WIT3 extended set, 57 language tracks | language | Dewpoint | mmBERT half alone | best other model | which | gap | |---|---|---|---|---|---| | Albanian (`sq`) | 0.7152 | 0.6828 | 0.6036 | fullstop-large | +0.112 | | Arabic (`ar`) | 0.6538 | 0.6225 | 0.5585 | fullstop-large | +0.095 | | Armenian (`hy`) | 0.7902 | 0.7709 | 0.4924 | kredor | +0.298 | | Azerbaijani (`az`) | 0.7286 | 0.7105 | 0.5832 | fullstop-sonar | +0.145 | | Bengali (`bn`) | 0.6940 | 0.6497 | 0.5584 | fullstop-large | +0.136 | | Bosnian (`bs`) | 0.7753 | 0.7540 | 0.6324 | fullstop-large | +0.143 | | Bulgarian (`bg`) | 0.8089 | 0.7906 | 0.7491 | kredor | +0.060 | | Burmese (`my`) | 0.4427 | 0.3569 | 0.3553 | xlm-r truecase | +0.087 | | Catalan (`ca`) | 0.7189 | 0.6880 | 0.6339 | fullstop-large | +0.085 | | Chinese (`zh`) | 0.6543 | 0.6349 | 0.4509 | xlm-r truecase | +0.203 | | Chinese (Simplified) (`zh-cn`) | 0.6486 | 0.6483 | 0.5439 | xlm-r truecase | +0.105 | | Chinese (Traditional) (`zh-tw`) | 0.5941 | 0.5960 | 0.5136 | xlm-r truecase | +0.081 | | Croatian (`hr`) | 0.7712 | 0.7540 | 0.6942 | fullstop-large | +0.077 | | Czech (`cs`) | 0.8120 | 0.7930 | 0.7461 | kredor | +0.066 | | Danish (`da`) | 0.7661 | 0.7498 | 0.6975 | fullstop-large | +0.069 | | Dutch (`nl`) | 0.8171 | 0.8060 | 0.7819 | fullstop-sonar | +0.035 | | English (`en`) | 0.8036 | 0.7937 | 0.7431 | fullstop-large | +0.061 | | Esperanto (`eo`) | 0.7663 | 0.7297 | 0.6385 | fullstop-sonar | +0.128 | | Estonian (`et`) | 0.8146 | 0.7884 | 0.7178 | fullstop-large | +0.097 | | Finnish (`fi`) | 0.8207 | 0.7996 | 0.7283 | kredor | +0.092 | | French (`fr`) | 0.7961 | 0.7842 | 0.7588 | fullstop-large | +0.037 | | French (Canada) (`fr-ca`) | 0.7778 | 0.7604 | 0.7445 | fullstop-sonar | +0.033 | | Georgian (`ka`) | 0.7142 | 0.6583 | 0.6179 | fullstop-large | +0.096 | | German (`de`) | 0.8472 | 0.8392 | 0.8302 | fullstop-large | +0.017 | | Greek (`el`) | 0.7331 | 0.7071 | 0.6951 | fullstop-large | +0.038 | | Hebrew (`he`) | 0.7398 | 0.7030 | 0.6506 | fullstop-large | +0.089 | | Hindi (`hi`) | 0.5855 | 0.5600 | 0.4925 | fullstop-large | +0.093 | | Hungarian (`hu`) | 0.7903 | 0.7637 | 0.6771 | fullstop-large | +0.113 | | Indonesian (`id`) | 0.7350 | 0.7174 | 0.6401 | fullstop-large | +0.095 | | Italian (`it`) | 0.7412 | 0.7211 | 0.7148 | fullstop-large | +0.026 | | Kazakh (`kk`) | 0.7657 | 0.7556 | 0.6508 | xlm-r truecase | +0.115 | | Korean (`ko`) | 0.8008 | 0.7825 | 0.6564 | xlm-r truecase | +0.144 | | Kurdish (`ku`) | 0.1114 | 0.2843 | 0.0298 | xlm-r truecase | +0.082 | | Latvian (`lv`) | 0.8174 | 0.7928 | 0.7117 | kredor | +0.106 | | Lithuanian (`lt`) | 0.7935 | 0.7725 | 0.6945 | kredor | +0.099 | | Macedonian (`mk`) | 0.7479 | 0.7190 | 0.6378 | fullstop-large | +0.110 | | Malay (`ms`) | 0.7748 | 0.7426 | 0.6387 | fullstop-large | +0.136 | | Marathi (`mr`) | 0.6624 | 0.6120 | 0.5224 | fullstop-large | +0.140 | | Mongolian (`mn`) | 0.7962 | 0.7447 | 0.6349 | xlm-r truecase | +0.161 | | Norwegian (`nb`) | 0.7704 | 0.7531 | 0.7399 | fullstop-large | +0.031 | | Persian (`fa`) | 0.6895 | 0.6700 | 0.5903 | fullstop-large | +0.099 | | Polish (`pl`) | 0.8181 | 0.7999 | 0.7696 | kredor | +0.049 | | Portuguese (`pt`) | 0.7725 | 0.7555 | 0.7095 | kredor | +0.063 | | Portuguese (Brazil) (`pt-br`) | 0.7550 | 0.7354 | 0.6701 | kredor | +0.085 | | Romanian (`ro`) | 0.7623 | 0.7320 | 0.6975 | fullstop-large | +0.065 | | Russian (`ru`) | 0.8354 | 0.8218 | 0.7312 | kredor | +0.104 | | Serbian (`sr`) | 0.7783 | 0.7588 | 0.6984 | fullstop-large | +0.080 | | Slovak (`sk`) | 0.8050 | 0.7787 | 0.7440 | kredor | +0.061 | | Slovenian (`sl`) | 0.8386 | 0.8211 | 0.8100 | kredor | +0.029 | | Spanish (`es`) | 0.7582 | 0.7463 | 0.7070 | fullstop-large | +0.051 | | Swahili (`sw`) | 0.6403 | 0.5944 | 0.5180 | fullstop-sonar | +0.122 | | Swedish (`sv`) | 0.7822 | 0.7611 | 0.7432 | fullstop-large | +0.039 | | Tamil (`ta`) | 0.5091 | 0.4862 | 0.4771 | xlm-r truecase | +0.032 | | Turkish (`tr`) | 0.7290 | 0.7082 | 0.5883 | xlm-r truecase | +0.141 | | Ukrainian (`uk`) | 0.8149 | 0.7951 | 0.7241 | kredor | +0.091 | | Urdu (`ur`) | 0.5309 | 0.5186 | 0.5042 | fullstop-large | +0.027 | | Vietnamese (`vi`) | 0.7105 | 0.6830 | 0.5800 | fullstop-sonar | +0.131 |
ASR condition, 33 languages | language | Dewpoint | mmBERT half alone | best other model | which | gap | |---|---|---|---|---|---| | Albanian (`sq`) | 0.3580 | 0.4005 | 0.3830 | xlm-r truecase | −0.025 | | Arabic (`ar`) | 0.5309 | 0.5020 | 0.4840 | fullstop-large | +0.047 | | Bengali (`bn`) | 0.0730 | 0.1404 | 0.1170 | fullstop-sonar | −0.044 | | Bulgarian (`bg`) | 0.7247 | 0.6928 | 0.6776 | kredor | +0.047 | | Catalan (`ca`) | 0.7112 | 0.6698 | 0.6025 | fullstop-sonar | +0.109 | | Chinese (`zh`) | 0.7528 | 0.7620 | 0.7312 | xlm-r truecase | +0.022 | | Dutch (`nl`) | 0.6765 | 0.6661 | 0.6115 | kredor | +0.065 | | English (`en`) | 0.7837 | 0.7869 | 0.9382 | raw Whisper | −0.155 | | Finnish (`fi`) | 0.7233 | 0.7053 | 0.6593 | kredor | +0.064 | | French (`fr`) | 0.7702 | 0.7705 | 0.8240 | raw Whisper | −0.054 | | German (`de`) | 0.8329 | 0.8204 | 0.8140 | fullstop-large | +0.019 | | Greek (`el`) | 0.6437 | 0.5979 | 0.5943 | kredor | +0.049 | | Hebrew (`he`) | 0.6332 | 0.5870 | 0.5265 | fullstop-sonar | +0.107 | | Hindi (`hi`) | 0.6238 | 0.5805 | 0.5509 | fullstop-sonar | +0.073 | | Hungarian (`hu`) | 0.7210 | 0.7041 | 0.6368 | fullstop-large | +0.084 | | Indonesian (`id`) | 0.6723 | 0.6582 | 0.6249 | fullstop-large | +0.047 | | Italian (`it`) | 0.7025 | 0.6750 | 0.7745 | raw Whisper | −0.072 | | Kazakh (`kk`) | 0.4953 | 0.4933 | 0.3990 | raw Whisper | +0.096 | | Latvian (`lv`) | 0.7218 | 0.7039 | 0.6108 | kredor | +0.111 | | Malay (`ms`) | 0.6743 | 0.6516 | 0.5907 | fullstop-large | +0.084 | | Marathi (`mr`) | 0.2584 | 0.2685 | 0.2000 | xlm-r truecase | +0.058 | | Persian (`fa`) | 0.5209 | 0.4990 | 0.4512 | xlm-r truecase | +0.070 | | Polish (`pl`) | 0.7608 | 0.7462 | 0.7409 | kredor | +0.020 | | Portuguese (`pt`) | 0.7616 | 0.7535 | 0.8522 | raw Whisper | −0.091 | | Romanian (`ro`) | 0.6462 | 0.6148 | 0.5553 | xlm-r truecase | +0.091 | | Russian (`ru`) | 0.7627 | 0.7647 | 0.6927 | kredor | +0.070 | | Spanish (`es`) | 0.7866 | 0.7844 | 0.8386 | raw Whisper | −0.052 | | Swahili (`sw`) | 0.4165 | 0.3931 | 0.3596 | kredor | +0.057 | | Swedish (`sv`) | 0.6646 | 0.6514 | 0.6435 | fullstop-large | +0.021 | | Tamil (`ta`) | 0.4052 | 0.3825 | 0.2485 | fullstop-sonar | +0.157 | | Turkish (`tr`) | 0.6670 | 0.6613 | 0.5448 | xlm-r truecase | +0.122 | | Ukrainian (`uk`) | 0.7310 | 0.6955 | 0.6545 | xlm-r truecase | +0.076 | | Vietnamese (`vi`) | 0.6895 | 0.6684 | 0.5873 | xlm-r truecase | +0.102 |
All per-language, per-class scores for every system are in [`results/scores.json`](results/scores.json). --- ## Training ### Text: 79 languages, no TED | source | licence | role | |---|---|---| | [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | CC-BY-SA-4.0 | volume, and most of the proper nouns truecasing needs | | [Europarl](https://huggingface.co/datasets/Helsinki-NLP/europarl) | no known restrictions | spoken-style parliamentary prose | | [Tatoeba](https://tatoeba.org/en/downloads) | CC-BY-2.0 FR | questions: 10–25% of its sentences are questions | | [HPLT 2.0 cleaned](https://huggingface.co/datasets/HPLT/HPLT2.0_cleaned) | CC0-1.0 | fills six languages Wikipedia leaves thin | | [VoxPopuli transcripts](https://github.com/facebookresearch/voxpopuli) | CC0-1.0 | verbatim spoken transcripts, 16 languages | | [Aya dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) | Apache-2.0 | question-dense human prose for low-resource languages | The corpus is temperature-sampled so that the largest languages don't swamp the rest. Labels come free: strip the punctuation and case from clean prose, and what was stripped is exactly what the model must predict. **Synthetic ASR noise** is applied to about two thirds of training examples: filler words, dropped short words, character substitutions, merged and split words, numbers written the way they're spoken, and truncated utterances. ### Speech: round-trip ASR data A slice of the training text was spoken aloud by TTS, transcribed by Whisper large-v3-turbo, and aligned back to the original. That gives inputs with genuine recognition errors but ground-truth labels. The corpus has 0.96M unique words across 23 languages, voiced by three permissively licensed engines. Oversampled to match the per-language dose of the original recipe, it makes up 6.8% of the final fine-tuning stage. | engine | licence | languages (utterances kept) | |---|---|---| | [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) | Apache-2.0 | en (5,998), es (5,077), fr (3,996), hi (3,879), it (3,996), pt (3,996), zh (2,992) | | [Piper voices](https://huggingface.co/rhasspy/piper-voices) | CC0 or CC-BY-4.0, per voice | bg (2,180), de (2,198), el (2,124), fa (1,057), fi (1,196), hu (1,184), kk (244), lv (1,067), nl (2,168), pl (2,189), ro (1,194), ru (1,177), sq (662), sv (2,183), uk (1,150) | | [Chatterbox Multilingual](https://huggingface.co/ResembleAI/chatterbox) | MIT | ar (606) | Every Piper voice's licence was read from its model card by code, and any voice whose card said non-commercial, share-alike, "see URL" or nothing at all was refused. The voice that produced a language's ASR *test* audio is never used for its training audio. French is the exception, because Kokoro has only one French voice. No audio was kept or is distributed. ### Procedure Each member was trained in stages. Every stage starts from the weights the previous stage ended with, and each later stage makes smaller, more targeted changes: 1. **Learn the task.** The encoder starts from its public pretrained weights, knowing language but nothing about punctuation, and learns to predict marks and capitals on a large clean-text corpus. 2. **Rebalance toward speech.** Training continues on a resample of the corpus that gives more weight to spoken-style sources (parliamentary transcripts, Tatoeba, Aya) and mixes short passages in with long ones, closer to what ASR produces. 3. **Harden against recognition errors.** A short final stage on a 55M-word mix that includes the round-trip ASR data above, at a low learning rate, so the model learns to punctuate text containing Whisper's mistakes without forgetting the rest. | member | stage | starts from | data | steps | peak learning rate | |---|---|---|---|---|---| | mmBERT-base | learn the task | `jhu-clsp/mmBERT-base` | 1.02B-word corpus, long passages | 34,000 | 4e-5 | | mmBERT-base | rebalance | previous stage | 735M-word resample | 13,000 | 1.8e-5 | | mmBERT-base | rebalance, continued | previous stage | 735M-word resample | 3,500 | 8e-6 | | mmBERT-base | harden | previous stage | 55M-word mix with round-trip data | 4,000 | 9e-6 | | XLM-R-large | learn + rebalance | `FacebookAI/xlm-roberta-large` | 735M-word resample | 17,000 | 8e-6 | | XLM-R-large | harden | previous stage | 55M-word mix with round-trip data | 4,000 | 3e-6 | How to read the table: - **Steps.** A step is one update of the weights, computed from a batch of 64 text windows of up to 512 subword tokens each, roughly 30,000 tokens. The first mmBERT stage, for example, saw about a billion tokens. - **Peak learning rate.** This is the largest step size in that stage's schedule. It falls from stage to stage because later stages refine rather than relearn. - **XLM-R-large has fewer stages** because it went straight to the rebalanced corpus. It was first trained as a controlled comparison against mmBERT-base, and later became the second member. Settings shared by all stages: - AdamW, with a learning rate that warms up and then decays along a cosine curve - a higher learning rate for the two heads than for the encoder - class-weighted cross-entropy on both heads, so the rare marks are not drowned out by the many words that take none - label smoothing 0.02 - for XLM-R-large only, a frozen embedding matrix Everything was trained on a single RTX 5080 with 16 GB. ### What did not work | attempt | result | |---|---| | supervised contrastive loss (reported +1.9 to +3.2 F1 on this task) | +0.001 / +0.001 / −0.002; the rarest class got worse | | joint punctuation–case decoding | cut head contradictions from 9.0% to 0.27% but lowered F1 at every setting | | mDeBERTa-v3-base as a third encoder | lost on all three sets | | distilling the ensemble into one mmBERT-base | in-domain validation +0.015, every held-out set flat | | choosing case *given* the decided punctuation | lowered case F1 at every coupling strength | The distillation result is why Dewpoint is an ensemble. Its robustness comes from the two members disagreeing on each input. A single model trained to mimic their average has no disagreement left to exploit. Two of these experiments were pre-registered: their success criteria were written down before the run. --- ## Limitations - **Two forward passes.** The ensemble has 2.8× the parameters of its mmBERT half, for +0.013 to +0.020 macro-F1. If latency matters more than that margin, use `members=["mmbert-base"]`. - **Text only.** Raw Whisper still wins in English, Spanish, French, Italian, Portuguese under ASR conditions, most likely because it hears pauses and intonation. Adding a tagger after ASR only helps if your ASR doesn't already punctuate. - **English is not state of the art against English-only systems.** On English tst2011, Dewpoint scores 0.800 macro-F1 (COMMA 0.715, PERIOD 0.823, QUESTION 0.860) without having seen any TED text. The strongest published English-only systems report about 0.85 overall F1 on IWSLT2011; they were trained in-domain on TED and scored on a differently preprocessed version of the set. - **Weak languages.** Kurdish (0.11), Burmese (0.44), Tamil (0.51), Urdu (0.53) are still low in absolute terms, even where Dewpoint leads the other models. In Kurdish (0.11 against 0.28) the ensemble is below its own mmBERT half, so averaging in probability space does not help everywhere. - **Japanese and Thai are unmeasured.** Their TED references are nearly unpunctuated, so no trustworthy number exists. The model is trained on both. - **Very short inputs.** With only a few words there is little context, and the case head can guess a sentence boundary that the punctuation head didn't place (`hello how are you` → `Hello, How are You?`). On full passages it is rare: 0.4% of words on the official test set. Pass whole utterances, not fragments. - **Caseless-script loanwords** are never capitalised (see truecasing above). - **EXCLAM is weak**: 0.11 F1 on the official set and 0.07 on WIT3, and exclamation marks are rare there too. - **Only five marks.** Colons, semicolons, dashes, quotes and parentheses are not restored, and French typographic spacing before `?` and `!` is not added. - **Domain.** The benchmark is TED, which is prepared monologue. Conversational, overlapping, multi-speaker speech is harder, and isn't measured here. --- ## Licence and provenance **Weights and code: MIT.** Both base encoders are MIT-licensed. Every TTS voice used for training audio is MIT, Apache-2.0, CC0 or CC-BY-4.0. A few things are worth knowing before commercial use: - **Wikipedia is CC-BY-SA-4.0.** Whether share-alike terms reach model weights trained on such text is legally unsettled. Many encoders trained on Wikipedia, BERT among them, are released under permissive licences, and Dewpoint follows that practice. - **Tatoeba is CC-BY-2.0 FR.** Sentences © Tatoeba contributors, https://tatoeba.org. - **Some Piper voices are CC-BY-4.0**, which asks for attribution to the voice datasets. They are listed in [`results/rt2_voices.json`](results/rt2_voices.json). - **Evaluation data.** IWSLT/WIT3 TED transcripts are CC-BY-NC-ND-4.0. They were used for measurement only and are not included here. This is a description, not legal advice. --- ## Files | path | contents | |---|---| | `dewpoint.py` | self-contained inference: `Punctuator` (torch or ONNX backend), streaming, command line | | `onnx/` | both members as fp32 ONNX graphs | | `serve.py` | a standard-library HTTP server | | `handler.py` | the Hugging Face Inference Endpoints handler | | `requirements.txt`, `requirements-onnx.txt` | dependencies for each backend | | `ensemble_config.json` | member list, weights, the ensemble's calibrated biases | | `mmbert-base/` | weights (safetensors), encoder config, tokenizer, its own calibration | | `xlm-roberta-large/` | the same, for the XLM-R-large member | | `gazetteer.json` | mixed-case surface forms, per language | | `results/` | per-language scores for every system, and round-trip voice provenance | | `assets/` | the images on this page | The inference file reproduces the published official-set score exactly, per language, against the evaluation harness it was measured with. ## Citation ```bibtex @misc{dewpoint2026, title = {Dewpoint: multilingual punctuation restoration and truecasing for speech}, author = {valkayuh}, year = {2026}, url = {https://huggingface.co/valkayuh/dewpoint} } ``` Please also cite the encoders Dewpoint builds on: [mmBERT](https://huggingface.co/jhu-clsp/mmBERT-base) and [XLM-RoBERTa](https://arxiv.org/abs/1911.02116).