Whisper Small Malayalam v2
openai/whisper-small fine-tuned for Malayalam speech recognition on six Malayalam speech corpora. It replaces v1.
On a held-out test set with no sentences shared with training, v2 scores 31.2% WER and 4.0% CER, against 43.5% / 7.3% for v1. It is the same size and speed as v1.
Try it: Malayalam voice assistant, a demo that runs this model with whisper.cpp on a free CPU, together with a small reply model and a Malayalam voice. Speak a question and watch each stage's timing.
Run it on CPU: two options, both needing one setting changed so Malayalam speech longer than about 8 seconds isn't cut off. CTranslate2 / faster-whisper int8 is the fastest measured in a like-for-like comparison (1.9Γ faster than Transformers, 32.7% WER / 4.2% CER). The included whisper.cpp Q8_0 build scores 31.4% / 4.1% at a quarter of the size.
More accurate when paired with IndicConformer: running this model and AI4Bharat's IndicConformer side by side, and keeping whichever transcript a Malayalam character language model rates more plausible, gives 28.5% WER / 3.35% CER with the included 10 MB selector (as low as 26.4% / 3.2% with larger LMs, beam search and n-gram rescoring). That is a significant gain over either model alone, with no retraining; see Pairing with IndicConformer.
Compared with other Malayalam models: on the same leakage-free clips, v2 is the most accurate fine-tuned Whisper tested, and statistically tied with AI4Bharat's IndicConformer (a CTC model that is much faster on CPU); see Comparison with other models. The benchmark code and per-clip outputs are public: malayalam-asr-eval.
How it was built: the model was developed and evaluated in public, in three posts: the leakage audit, re-testing v1 on clean data and v2 and a bug in my own evaluation.
Results
Both models were scored with the same notebook, on the same 231 clips, with the same settings: FP32, greedy decoding, CPU, transformers 4.46.3, up to 444 new tokens per clip (no clip hit the limit).
| Leakage-free test set | v1 | v2 |
|---|---|---|
| WER, normalised (95% bootstrap CI) | 43.5% (39.9β47.0) | 31.2% (27.9β34.6) |
| CER, normalised | 7.3% | 4.0% |
| Substitutions / deletions / insertions | 33.3% / 5.1% / 5.1% | 23.0% / 4.0% / 4.1% |
| Utterances fully correct | 15.2% | 27.7% |
| Empty outputs / repetition loops | 0 / 0 | 0 / 0 |
| Source | Clips | v1 WER / CER | v2 WER / CER |
|---|---|---|---|
| IndicTTS | 50 | 33.1% / 4.6% | 26.7% / 2.6% |
| OpenSLR 63 | 32 | 41.0% / 6.1% | 28.9% / 3.7% |
| Shrutilipi | 50 | 48.2% / 9.4% | 31.4% / 4.7% |
| IMaSC | 44 | 39.5% / 3.9% | 31.8% / 2.2% |
| CommonVoice | 50 | 56.0% / 13.2% | 38.0% / 7.8% |
| SMC | 5 | 35.0% / 3.9% | 25.0% / 3.1% |
SMC has only 5 clean test rows, so its score is not meaningful on its own.
Normalisation: Unicode NFC, old-style chillu (consonant + virama + ZWJ) converted to atomic chillu, zero-width joiners removed, punctuation stripped. CER is computed with spaces removed.
How to read these numbers
- The test set is leakage-free at the sentence level. It is the 2,684 rows of the corpus
testsplit whose transcript does not appear, exactly or as a near-duplicate, intrain. Up to 50 clips per source were sampled, so the headline weights sources roughly equally. - Speakers may still overlap. The filter removes repeated sentences, not repeated voices. The same news readers or recording sessions can appear in both splits, so accuracy on entirely new speakers and microphones may be lower. This affects v1 and v2 equally.
- CER is the better guide for Malayalam. Words are long and agglutinative, so one wrong letter makes a whole word wrong. At 4.0% CER, about 96 of every 100 characters are correct.
- On rows that do appear in training, v2 scores 12.5% WER (60 rows). That gap from 31.2% is memorisation, which is why the clean set is the number to trust.
- CommonVoice is the weakest source: crowdsourced phone recordings with the most varied voices and microphones.
Comparison with other models
Scored with malayalam-asr-eval on the same 231 leakage-free clips, with the same normalisation and scoring as above. Kaggle CPU (Intel Xeon @ 2.20 GHz), pinned to 2 physical cores, batch 1, greedy decoding, October 2026. Full tables and every transcript: leaderboard.
| Model | Params | WER (95% CI) | CER | WER on unseen sourcesΒΉ | Speed vs v2Β² |
|---|---|---|---|---|---|
| ai4bharat/indic-conformer-600m-multilingual (CTC) | 600M | 30.1% (27.2β33.3) | 4.0% | 31.3% | 13.8Γ |
| whisper-small-malayalam-v2 | 244M | 31.2% (27.9β34.6) | 4.0% | 31.2% | 1.0Γ |
| v2, CTranslate2 int8 | 244M | 32.7% (29.6β36.1) | 4.2% | 32.7% | 1.9Γ |
| vrclc/Whisper-medium-Malayalam | 769M | 39.7% (33.1β47.3) | 10.1% | 52.9% | 0.35Γ |
| facebook/mms-1b-all (Malayalam adapter) | 965M | 52.5% (49.0β56.1) | 9.2% | 51.5% | 2.2Γ |
| kavyamanohar/whisper-small-malayalam | 244M | 87.0% (82.4β93.7) | 29.2% | 88.8% | 0.84Γ |
ΒΉ The test set is only filtered against this model's training corpus. Other models may have been trained on some of the same public corpora, so this column drops the clips from any corpus listed in that model's own training data (for example, vrclc's card lists IMaSC, OpenSLR 63 and SMC, where it scores 16.1%, 8.4% and 0.0% WER, against 62β70% on Shrutilipi and CommonVoice). IndicConformer's training data isn't published; its Shrutilipi rows were dropped as a precaution. Β² v2's real-time factor divided by the model's, both measured on the same machine in the same session. Kaggle's "same" CPU varied by about 20% between sessions, so relative speed is more reliable than absolute numbers.
- Among Whisper models, v2 is the most accurate tested, including the 3Γ larger vrclc medium.
- IndicConformer matches v2's accuracy and is about 14Γ faster on CPU (7Γ faster than real time on 2 cores), because CTC decoding doesn't generate tokens one at a time. If you need real-time Malayalam ASR on a small CPU, it is the better choice today.
- These rankings are for read and broadcast speech; see Limitations.
Pairing with IndicConformer (higher accuracy)
Whisper v2 and IndicConformer-600M (CTC) are statistically tied on their own but make different mistakes: on the 231 clips, picking the better transcript per clip with hindsight would score 23.9% WER. A selector with no tuned parameters gets much of that gain. Normalise both transcripts, score each with a character n-gram LM, and keep the one with the higher average log-probability per character.
Same 231 leakage-free clips, normalisation and scoring as above. ΞWER is against v2 greedy, with a paired bootstrap 95% CI. All weights were tuned on a separate 90-clip leakage-free dev set, never on these clips.
| System | WER (95% CI) | CER | ΞWER [95% CI] |
|---|---|---|---|
| v2 greedy (FP32) | 31.2 (27.9β34.6) | 4.04 | β |
| v2 beam 5 | 31.4 (28.3β34.9) | 3.96 | +0.3 [β1.0, +1.5] |
| v2 beam 5 + word 4-gram N-best rescoring | 30.4 (27.2β34.1) | 3.96 | β0.8 [β2.5, +0.9] |
| IndicConformer CTC alone | 30.1 (27.2β33.3) | 4.00 | β1.0 [β3.6, +1.7] |
v2 greedy + IndicConformer, LM picks (5-gram, selector/) |
28.5 (25.3β31.7) | 3.35 | β2.7 [β4.6, β0.7] |
| v2 greedy + IndicConformer, LM picks (two 8-gram LMs) | 27.6 (24.5β30.8) | 3.32 | β3.5 [β5.4, β1.6] |
| v2 beam 5 + rescoring + IndicConformer, LM picks (8-gram) | 26.4 (23.3β29.7) | 3.16 | β4.8 [β6.9, β2.5] |
| v2 CTranslate2 int8 + IndicConformer, LM picks (5-gram) | 29.1 (26.0β32.4) | 3.45 | β3.5 vs int8 alone [β5.6, β1.5] |
- Without Shrutilipi (in case IndicConformer was trained on it) the best system still scores 26.6% against 31.1%, so the gain is not from data overlap.
- The LMs are KenLM character n-grams trained on this model's training transcripts (any sentence that also appears in the test or dev sets was removed) plus 1.25M Malayalam Wikipedia sentences.
selector/ml_char5_pruned.arpa.gzis the pruned 5-gram (10 MB).selector/asr_select.pyreads it in pure Python, with scores matching KenLM, so no compiled package is needed. Two unpruned 8-gram LMs (~680 MB) score about 1 point better. The LM file is derived from Wikipedia text (CC BY-SA 4.0) and the corpus transcripts (CC BY 4.0). - What didn't help: beam search alone; n-gram N-best rescoring (β3 points on dev, not significant on test); loudness normalisation and silence trimming. Selecting with Whisper's own probability was worse than the LM-only rule (29.9%), because it favours Whisper's own outputs.
- The Malayalam voice assistant Space uses the int8 + IndicConformer + 5-gram setup.
# pip install huggingface_hub onnxruntime transformers torch (+ the Whisper engine of your choice)
import importlib.util, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel
repo = "sajilck/whisper-small-malayalam-v2"
spec = importlib.util.spec_from_file_location("asr_select", hf_hub_download(repo, "selector/asr_select.py"))
asr_select = importlib.util.module_from_spec(spec); spec.loader.exec_module(asr_select)
lm = asr_select.ArpaLM(hf_hub_download(repo, "selector/ml_char5_pruned.arpa.gz")) # ~1 s, ~0.3 GB RAM
conformer = AutoModel.from_pretrained("ai4bharat/indic-conformer-600m-multilingual", trust_remote_code=True) # gated
def transcribe(wav): # 16 kHz mono float32 numpy array
w = whisper_transcribe(wav) # this model, any engine (see Usage / CTranslate2 below)
c = conformer(torch.from_numpy(wav).unsqueeze(0), "ml", "ctc")
c = c[0] if isinstance(c, (list, tuple)) else str(c)
return [w, c][asr_select.select(lm, [w, c])]
What changed from v1
| v1 | v2 | |
|---|---|---|
| Learning rate | 1e-5 | 1e-4 (linear decay, 500 warmup steps) |
| Training steps | 3,500 | 10,000 (about 2.3 epochs) |
| Training rows | ~86,900 | 69,437 after cleaning |
| Repeated sentences | kept | at most 3 copies of any sentence |
| Over-long labels | kept | removed (β₯ 448 tokens) |
- Learning rate. The Vividh-ASR work suggested 1e-5 was far too low. Short 300-step runs at 1e-4 and 2e-4 were both stable and scored within noise of each other (56.3% vs 57.8% WER on the in-training eval subset), so the full run used the lower, safer rate.
- Data cleaning. IMaSC and SMC read the same fixed sentences many times. Capping each sentence at 3 copies removed 16,771 rows. 703 rows with transcripts too long for Whisper's 448-token decoder were also removed.
- Evaluation from the start. Training was evaluated every 1,000 steps on a fixed, source-stratified subset of the clean set (379 clips). WER fell from 53.9% at step 1,000 to 31.4% at step 9,000 and stayed there; the final checkpoint is published.
Usage
from transformers import pipeline
pipe = pipeline(
"automatic-speech-recognition",
model="sajilck/whisper-small-malayalam-v2",
generate_kwargs={"language": "malayalam", "task": "transcribe", "max_new_tokens": 444},
)
print(pipe("your_audio.wav")["text"])
For audio longer than 30 seconds, add chunk_length_s=30, stride_length_s=5.
Set max_new_tokens high. Whisper's tokenizer splits Malayalam into about 25 tokens per second of speech. A limit of 225 tokens cuts off anything longer than about 9 seconds. That mistake affected an earlier evaluation of v1; see its model card.
CPU speed (Transformers, no GPU)
Measured on a Kaggle CPU (Intel Xeon @ 2.20 GHz, 4 vCPU / 2 physical cores), FP32, greedy, batch size 1, 231 clips averaging 4.8 s:
| Metric | Result |
|---|---|
| Real-time factor (all clips) | 2.02 (slower than real time) |
| Latency per clip, p50 / p90 / max | 9.1 s / 14.6 s / 23.6 s |
| Decoding speed | ~14.5 tokens/s |
| Peak RAM | ~4.7 GB* |
* Peak RAM was measured in this standalone notebook and is not comparable with the RAM column in the CTranslate2 table, which comes from the benchmark framework's own process (2.4 GB for the same FP32 model). Compare RAM only within one table.
Almost all of the time is the decoder generating tokens one at a time, so CPU speed is limited by transcript length in tokens. On the same CPU type, the real-time factor ranged from 1.9 to 2.3 across Kaggle sessions. For CPU deployment, use CTranslate2 int8 or whisper.cpp below.
CTranslate2 / faster-whisper int8 (CPU)
faster-whisper runs an int8 CTranslate2 conversion of this model (~255 MB). Measured in the same session as Transformers FP32, on the same 231 clips (2 physical cores):
| Engine | WER (95% CI) | CER | Real-time factor | Latency p50 / p90 | Peak RAM |
|---|---|---|---|---|---|
| Transformers FP32 | 31.2% (27.9β34.6) | 4.0% | 2.28 | 10.2 s / 17.2 s | 2.4 GB |
| CTranslate2 int8 | 32.7% (29.6β36.1) | 4.2% | 1.22 | 5.6 s / 7.9 s | β€ 2.6 GBβ |
β Overstated: in this run the FP32 β int8 conversion happened inside the process being measured, so its peak includes the FP32 weights (967 MB on disk, against 255 MB for the int8 model). The benchmark adapter has since been changed to convert in a separate process; this figure will be replaced with a re-measured one.
int8 makes decoding about 1.9Γ faster for 1.5 WER points. It is still slightly slower than real time on 2 cores.
Important: CTranslate2 generates at most half of max_length for Whisper. faster-whisper's default max_length is 448, so every transcript silently stops at 224 tokens, about 8 seconds of Malayalam. With the defaults this model scored 38.0% WER / 11.2% CER on the same clips; 31 transcripts ended at exactly 224 tokens. Double the limit as below.
# pip install faster-whisper transformers
from ctranslate2.converters import TransformersConverter
from transformers import WhisperFeatureExtractor, WhisperTokenizerFast
from faster_whisper import WhisperModel
repo, out = "sajilck/whisper-small-malayalam-v2", "whisper-small-ml-v2-ct2-int8"
# One-time conversion. faster-whisper reads tokenizer.json and preprocessor_config.json
# from this folder; without them it silently uses the stock OpenAI tokenizer.
TransformersConverter(repo).convert(out, quantization="int8", force=True)
WhisperTokenizerFast.from_pretrained(repo).save_pretrained(out)
WhisperFeatureExtractor.from_pretrained(repo).save_pretrained(out)
model = WhisperModel(out, device="cpu", compute_type="int8", cpu_threads=2)
model.max_length = 888 # CTranslate2 emits at most max_length // 2 = 444 tokens
segments, _ = model.transcribe(
"your_audio.wav", language="ml", beam_size=1, temperature=0.0,
without_timestamps=True, condition_on_previous_text=False,
max_new_tokens=884, # 888 minus the 4 prompt tokens
)
print(" ".join(s.text.strip() for s in segments))
On recent ctranslate2 versions with transformers older than 4.56, the converter fails with an unexpected dtype argument; upgrading transformers for the conversion step fixes it. The benchmark's adapter handles both. On a GPU, use compute_type="int8_float16" or "float16".
whisper.cpp β CPU inference (no GPU)
GGML builds for whisper.cpp are in the ggml/ folder. They run on any laptop or small server without a GPU.
| Variant | File | Size | Notes |
|---|---|---|---|
| F16 | ggml/ggml-model-f16.bin | ~488 MB | Full precision |
| Q8_0 | ggml/ggml-model-q8_0.bin | ~264 MB | Recommended; used by the demo Space |
Both were converted from this model with whisper.cpp's convert-h5-to-ggml.py.
Accuracy and speed on the same 231 leakage-free clips as the results above, greedy decoding, patched build (see below). The two rows were run on different machines, so accuracy is comparable but speed is not:
| Engine | WER, normalised (95% CI) | CER | Real-time factor | Size | CPU used |
|---|---|---|---|---|---|
| Transformers FP32 | 31.2% (27.9β34.6) | 4.0% | 2.02 | 967 MB | Kaggle Intel Xeon @ 2.20 GHz, 2 cores / 4 vCPU |
| whisper.cpp Q8_0 | 31.4% (28.3β34.7) | 4.1% | 1.61 | 264 MB | Kaggle AMD EPYC 7B12, 2 cores / 4 threads, build compiled natively for that CPU |
Q8_0 quantisation costs about 0.2 WER points, at a quarter of the size. Do not read 1.61 against 2.02 as a measured speed-up: the CPU and the build differ between the rows, and part of the gain may come from the native build. A same-session comparison of Transformers, CTranslate2 and whisper.cpp is planned. For a like-for-like speed comparison, see the CTranslate2 table above. whisper.cpp Q8_0 is still slower than real time on 2 CPU cores.
Important: raise whisper.cpp's token limit for Malayalam. Stock whisper.cpp decodes at most 220 tokens per 30-second window (n_max = whisper_n_text_ctx(ctx)/2 - 4 in src/whisper.cpp). Malayalam needs about 27 Whisper tokens per second, so speech longer than about 8 seconds is cut off without warning. 32 of the 231 test clips are that long. On the same clips (same machine for all three rows):
| whisper.cpp Q8_0 build | WER | CER |
|---|---|---|
| Stock, timestamps off | 37.1% | 11.5% |
| Stock, timestamps on | 40.7% | 12.6% |
| Token limit raised to 440 | 31.4% | 4.1% |
Turning timestamps on does not help: this model was fine-tuned without timestamps and never emits them, so whisper.cpp cannot continue past the limit. The fix is a one-line change before building (step 1 below). It is the same problem as the 225-token setting described in Usage, inside a different engine.
Quick start
# 1. Build whisper.cpp, raising the per-window token limit from 220 to 440 (needed for Malayalam)
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
sed -i 's|n_max = whisper_n_text_ctx(ctx)/2 - 4;|n_max = whisper_n_text_ctx(ctx) - 8;|' src/whisper.cpp
grep -n "n_max = whisper_n_text_ctx(ctx) - 8;" src/whisper.cpp # must print one line
cmake -B build && cmake --build build --config Release
# 2. Download the Q8_0 model
wget https://huggingface.co/sajilck/whisper-small-malayalam-v2/resolve/main/ggml/ggml-model-q8_0.bin
# 3. Transcribe Malayalam audio (16 kHz mono WAV)
./build/bin/whisper-cli -m ggml-model-q8_0.bin -l ml -f your_audio.wav
Convert other audio to the right format with:
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
From Python, pywhispercpp loads the same file: Model("ggml-model-q8_0.bin", language="ml", no_timestamps=True, single_segment=True). The PyPI wheel has the 220-token limit built in. To lift it, build pywhispercpp from source with the same one-line change in its whisper.cpp/src/whisper.cpp submodule; the scores above used pywhispercpp v1.5.1 built this way.
Training details
| Parameter | Value |
|---|---|
| Base model | openai/whisper-small (244M parameters) |
| Training data | sajilck/malayalam-asr-corpus train split, cleaned to 69,437 rows |
| Steps / batch size | 10,000 / 16 |
| Learning rate | 1e-4, linear decay, 500 warmup steps |
| Precision | fp16, gradient checkpointing |
| Hardware | one free-tier Kaggle GPU, single session |
| Framework | transformers 4.46.3, Seq2SeqTrainer |
Training data sources: IMaSC, SMC Malayalam Speech Corpus, IndicTTS Malayalam, OpenSLR 63, CommonVoice 25 Malayalam and Shrutilipi (All India Radio news). Shrutilipi is the largest. See the v1 card for links and the leakage audit of the corpus. The leakage-free test list (own_corpus_test_clean.csv) and the exact 231-clip sample are in malayalam-asr-eval.
Limitations
- Read and broadcast speech only. Training data is read, scripted, crowdsourced or radio-news speech. Expect lower accuracy on spontaneous conversation, dialects and code-mixed MalayalamβEnglish.
- Thin test coverage for some sources. Only 44 IMaSC, 32 OpenSLR 63 and 5 SMC test rows survive leakage filtering, so their scores have wide error bars.
- Possible speaker overlap between train and test (see above).
- Spelling and word-boundary variation. Many remaining word errors are valid alternative spellings or compound words split differently from the reference.
- Not real-time on 2 CPU cores. Real-time factor is about 2 with Transformers, about 1.2 with CTranslate2 int8 (same session as Transformers), and 1.6 with whisper.cpp Q8_0 (measured on a different CPU, so not directly comparable). Whisper's tokenizer needs about 27 tokens per second of Malayalam speech, and each token is a decoding step. A CTC model such as IndicConformer is real-time on the same hardware (see Comparison).
- Default output limits cut off Malayalam. Stock whisper.cpp stops at 220 tokens per window and faster-whisper at 224, so speech longer than about 8 seconds is cut off unless the limit is raised (see CTranslate2 and whisper.cpp).
Citation
@misc{sajilck2026whispermalayalamv2,
author = {Sajil C.K.},
title = {Whisper Small Malayalam v2},
year = {2026},
publisher = {HuggingFace},
url = {https://huggingface.co/sajilck/whisper-small-malayalam-v2}
}
License
Apache 2.0, consistent with the base Whisper model. The training corpora keep their own licenses; check each source dataset before commercial use.
- Downloads last month
- 191
Model tree for sajilck/whisper-small-malayalam-v2
Base model
openai/whisper-smallDataset used to train sajilck/whisper-small-malayalam-v2
Spaces using sajilck/whisper-small-malayalam-v2 2
Evaluation results
- WER (normalised) on malayalam-asr-corpus test, leakage-filtered (231-clip source-balanced sample)test set self-reported31.160
- CER (normalised) on malayalam-asr-corpus test, leakage-filtered (231-clip source-balanced sample)test set self-reported4.040