You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Whisper Small Malayalam v2

openai/whisper-small fine-tuned for Malayalam speech recognition on six Malayalam speech corpora. It replaces v1.

On a held-out test set with no sentences shared with training, v2 scores 31.2% WER and 4.0% CER, against 43.5% / 7.3% for v1. It is the same size and speed as v1.

Try it: Malayalam voice assistant, a demo that runs this model with whisper.cpp on a free CPU, together with a small reply model and a Malayalam voice. Speak a question and watch each stage's timing.

Run it on CPU: two options, both needing one setting changed so Malayalam speech longer than about 8 seconds isn't cut off. CTranslate2 / faster-whisper int8 is the fastest measured in a like-for-like comparison (1.9Γ— faster than Transformers, 32.7% WER / 4.2% CER). The included whisper.cpp Q8_0 build scores 31.4% / 4.1% at a quarter of the size.

More accurate when paired with IndicConformer: running this model and AI4Bharat's IndicConformer side by side, and keeping whichever transcript a Malayalam character language model rates more plausible, gives 28.5% WER / 3.35% CER with the included 10 MB selector (as low as 26.4% / 3.2% with larger LMs, beam search and n-gram rescoring). That is a significant gain over either model alone, with no retraining; see Pairing with IndicConformer.

Compared with other Malayalam models: on the same leakage-free clips, v2 is the most accurate fine-tuned Whisper tested, and statistically tied with AI4Bharat's IndicConformer (a CTC model that is much faster on CPU); see Comparison with other models. The benchmark code and per-clip outputs are public: malayalam-asr-eval.

How it was built: the model was developed and evaluated in public, in three posts: the leakage audit, re-testing v1 on clean data and v2 and a bug in my own evaluation.

Results

Both models were scored with the same notebook, on the same 231 clips, with the same settings: FP32, greedy decoding, CPU, transformers 4.46.3, up to 444 new tokens per clip (no clip hit the limit).

Leakage-free test set v1 v2
WER, normalised (95% bootstrap CI) 43.5% (39.9–47.0) 31.2% (27.9–34.6)
CER, normalised 7.3% 4.0%
Substitutions / deletions / insertions 33.3% / 5.1% / 5.1% 23.0% / 4.0% / 4.1%
Utterances fully correct 15.2% 27.7%
Empty outputs / repetition loops 0 / 0 0 / 0
Source Clips v1 WER / CER v2 WER / CER
IndicTTS 50 33.1% / 4.6% 26.7% / 2.6%
OpenSLR 63 32 41.0% / 6.1% 28.9% / 3.7%
Shrutilipi 50 48.2% / 9.4% 31.4% / 4.7%
IMaSC 44 39.5% / 3.9% 31.8% / 2.2%
CommonVoice 50 56.0% / 13.2% 38.0% / 7.8%
SMC 5 35.0% / 3.9% 25.0% / 3.1%

SMC has only 5 clean test rows, so its score is not meaningful on its own.

Normalisation: Unicode NFC, old-style chillu (consonant + virama + ZWJ) converted to atomic chillu, zero-width joiners removed, punctuation stripped. CER is computed with spaces removed.

How to read these numbers

  • The test set is leakage-free at the sentence level. It is the 2,684 rows of the corpus test split whose transcript does not appear, exactly or as a near-duplicate, in train. Up to 50 clips per source were sampled, so the headline weights sources roughly equally.
  • Speakers may still overlap. The filter removes repeated sentences, not repeated voices. The same news readers or recording sessions can appear in both splits, so accuracy on entirely new speakers and microphones may be lower. This affects v1 and v2 equally.
  • CER is the better guide for Malayalam. Words are long and agglutinative, so one wrong letter makes a whole word wrong. At 4.0% CER, about 96 of every 100 characters are correct.
  • On rows that do appear in training, v2 scores 12.5% WER (60 rows). That gap from 31.2% is memorisation, which is why the clean set is the number to trust.
  • CommonVoice is the weakest source: crowdsourced phone recordings with the most varied voices and microphones.

Comparison with other models

Scored with malayalam-asr-eval on the same 231 leakage-free clips, with the same normalisation and scoring as above. Kaggle CPU (Intel Xeon @ 2.20 GHz), pinned to 2 physical cores, batch 1, greedy decoding, October 2026. Full tables and every transcript: leaderboard.

Model Params WER (95% CI) CER WER on unseen sourcesΒΉ Speed vs v2Β²
ai4bharat/indic-conformer-600m-multilingual (CTC) 600M 30.1% (27.2–33.3) 4.0% 31.3% 13.8Γ—
whisper-small-malayalam-v2 244M 31.2% (27.9–34.6) 4.0% 31.2% 1.0Γ—
v2, CTranslate2 int8 244M 32.7% (29.6–36.1) 4.2% 32.7% 1.9Γ—
vrclc/Whisper-medium-Malayalam 769M 39.7% (33.1–47.3) 10.1% 52.9% 0.35Γ—
facebook/mms-1b-all (Malayalam adapter) 965M 52.5% (49.0–56.1) 9.2% 51.5% 2.2Γ—
kavyamanohar/whisper-small-malayalam 244M 87.0% (82.4–93.7) 29.2% 88.8% 0.84Γ—

ΒΉ The test set is only filtered against this model's training corpus. Other models may have been trained on some of the same public corpora, so this column drops the clips from any corpus listed in that model's own training data (for example, vrclc's card lists IMaSC, OpenSLR 63 and SMC, where it scores 16.1%, 8.4% and 0.0% WER, against 62–70% on Shrutilipi and CommonVoice). IndicConformer's training data isn't published; its Shrutilipi rows were dropped as a precaution. Β² v2's real-time factor divided by the model's, both measured on the same machine in the same session. Kaggle's "same" CPU varied by about 20% between sessions, so relative speed is more reliable than absolute numbers.

  • Among Whisper models, v2 is the most accurate tested, including the 3Γ— larger vrclc medium.
  • IndicConformer matches v2's accuracy and is about 14Γ— faster on CPU (7Γ— faster than real time on 2 cores), because CTC decoding doesn't generate tokens one at a time. If you need real-time Malayalam ASR on a small CPU, it is the better choice today.
  • These rankings are for read and broadcast speech; see Limitations.

Pairing with IndicConformer (higher accuracy)

Whisper v2 and IndicConformer-600M (CTC) are statistically tied on their own but make different mistakes: on the 231 clips, picking the better transcript per clip with hindsight would score 23.9% WER. A selector with no tuned parameters gets much of that gain. Normalise both transcripts, score each with a character n-gram LM, and keep the one with the higher average log-probability per character.

Same 231 leakage-free clips, normalisation and scoring as above. Ξ”WER is against v2 greedy, with a paired bootstrap 95% CI. All weights were tuned on a separate 90-clip leakage-free dev set, never on these clips.

System WER (95% CI) CER Ξ”WER [95% CI]
v2 greedy (FP32) 31.2 (27.9–34.6) 4.04 –
v2 beam 5 31.4 (28.3–34.9) 3.96 +0.3 [βˆ’1.0, +1.5]
v2 beam 5 + word 4-gram N-best rescoring 30.4 (27.2–34.1) 3.96 βˆ’0.8 [βˆ’2.5, +0.9]
IndicConformer CTC alone 30.1 (27.2–33.3) 4.00 βˆ’1.0 [βˆ’3.6, +1.7]
v2 greedy + IndicConformer, LM picks (5-gram, selector/) 28.5 (25.3–31.7) 3.35 βˆ’2.7 [βˆ’4.6, βˆ’0.7]
v2 greedy + IndicConformer, LM picks (two 8-gram LMs) 27.6 (24.5–30.8) 3.32 βˆ’3.5 [βˆ’5.4, βˆ’1.6]
v2 beam 5 + rescoring + IndicConformer, LM picks (8-gram) 26.4 (23.3–29.7) 3.16 βˆ’4.8 [βˆ’6.9, βˆ’2.5]
v2 CTranslate2 int8 + IndicConformer, LM picks (5-gram) 29.1 (26.0–32.4) 3.45 βˆ’3.5 vs int8 alone [βˆ’5.6, βˆ’1.5]
  • Without Shrutilipi (in case IndicConformer was trained on it) the best system still scores 26.6% against 31.1%, so the gain is not from data overlap.
  • The LMs are KenLM character n-grams trained on this model's training transcripts (any sentence that also appears in the test or dev sets was removed) plus 1.25M Malayalam Wikipedia sentences. selector/ml_char5_pruned.arpa.gz is the pruned 5-gram (10 MB). selector/asr_select.py reads it in pure Python, with scores matching KenLM, so no compiled package is needed. Two unpruned 8-gram LMs (~680 MB) score about 1 point better. The LM file is derived from Wikipedia text (CC BY-SA 4.0) and the corpus transcripts (CC BY 4.0).
  • What didn't help: beam search alone; n-gram N-best rescoring (βˆ’3 points on dev, not significant on test); loudness normalisation and silence trimming. Selecting with Whisper's own probability was worse than the LM-only rule (29.9%), because it favours Whisper's own outputs.
  • The Malayalam voice assistant Space uses the int8 + IndicConformer + 5-gram setup.
# pip install huggingface_hub onnxruntime transformers torch  (+ the Whisper engine of your choice)
import importlib.util, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel

repo = "sajilck/whisper-small-malayalam-v2"
spec = importlib.util.spec_from_file_location("asr_select", hf_hub_download(repo, "selector/asr_select.py"))
asr_select = importlib.util.module_from_spec(spec); spec.loader.exec_module(asr_select)
lm = asr_select.ArpaLM(hf_hub_download(repo, "selector/ml_char5_pruned.arpa.gz"))   # ~1 s, ~0.3 GB RAM

conformer = AutoModel.from_pretrained("ai4bharat/indic-conformer-600m-multilingual", trust_remote_code=True)  # gated

def transcribe(wav):                      # 16 kHz mono float32 numpy array
    w = whisper_transcribe(wav)           # this model, any engine (see Usage / CTranslate2 below)
    c = conformer(torch.from_numpy(wav).unsqueeze(0), "ml", "ctc")
    c = c[0] if isinstance(c, (list, tuple)) else str(c)
    return [w, c][asr_select.select(lm, [w, c])]

What changed from v1

v1 v2
Learning rate 1e-5 1e-4 (linear decay, 500 warmup steps)
Training steps 3,500 10,000 (about 2.3 epochs)
Training rows ~86,900 69,437 after cleaning
Repeated sentences kept at most 3 copies of any sentence
Over-long labels kept removed (β‰₯ 448 tokens)
  • Learning rate. The Vividh-ASR work suggested 1e-5 was far too low. Short 300-step runs at 1e-4 and 2e-4 were both stable and scored within noise of each other (56.3% vs 57.8% WER on the in-training eval subset), so the full run used the lower, safer rate.
  • Data cleaning. IMaSC and SMC read the same fixed sentences many times. Capping each sentence at 3 copies removed 16,771 rows. 703 rows with transcripts too long for Whisper's 448-token decoder were also removed.
  • Evaluation from the start. Training was evaluated every 1,000 steps on a fixed, source-stratified subset of the clean set (379 clips). WER fell from 53.9% at step 1,000 to 31.4% at step 9,000 and stayed there; the final checkpoint is published.

Usage

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="sajilck/whisper-small-malayalam-v2",
    generate_kwargs={"language": "malayalam", "task": "transcribe", "max_new_tokens": 444},
)
print(pipe("your_audio.wav")["text"])

For audio longer than 30 seconds, add chunk_length_s=30, stride_length_s=5.

Set max_new_tokens high. Whisper's tokenizer splits Malayalam into about 25 tokens per second of speech. A limit of 225 tokens cuts off anything longer than about 9 seconds. That mistake affected an earlier evaluation of v1; see its model card.

CPU speed (Transformers, no GPU)

Measured on a Kaggle CPU (Intel Xeon @ 2.20 GHz, 4 vCPU / 2 physical cores), FP32, greedy, batch size 1, 231 clips averaging 4.8 s:

Metric Result
Real-time factor (all clips) 2.02 (slower than real time)
Latency per clip, p50 / p90 / max 9.1 s / 14.6 s / 23.6 s
Decoding speed ~14.5 tokens/s
Peak RAM ~4.7 GB*

* Peak RAM was measured in this standalone notebook and is not comparable with the RAM column in the CTranslate2 table, which comes from the benchmark framework's own process (2.4 GB for the same FP32 model). Compare RAM only within one table.

Almost all of the time is the decoder generating tokens one at a time, so CPU speed is limited by transcript length in tokens. On the same CPU type, the real-time factor ranged from 1.9 to 2.3 across Kaggle sessions. For CPU deployment, use CTranslate2 int8 or whisper.cpp below.

CTranslate2 / faster-whisper int8 (CPU)

faster-whisper runs an int8 CTranslate2 conversion of this model (~255 MB). Measured in the same session as Transformers FP32, on the same 231 clips (2 physical cores):

Engine WER (95% CI) CER Real-time factor Latency p50 / p90 Peak RAM
Transformers FP32 31.2% (27.9–34.6) 4.0% 2.28 10.2 s / 17.2 s 2.4 GB
CTranslate2 int8 32.7% (29.6–36.1) 4.2% 1.22 5.6 s / 7.9 s ≀ 2.6 GB†

† Overstated: in this run the FP32 β†’ int8 conversion happened inside the process being measured, so its peak includes the FP32 weights (967 MB on disk, against 255 MB for the int8 model). The benchmark adapter has since been changed to convert in a separate process; this figure will be replaced with a re-measured one.

int8 makes decoding about 1.9Γ— faster for 1.5 WER points. It is still slightly slower than real time on 2 cores.

Important: CTranslate2 generates at most half of max_length for Whisper. faster-whisper's default max_length is 448, so every transcript silently stops at 224 tokens, about 8 seconds of Malayalam. With the defaults this model scored 38.0% WER / 11.2% CER on the same clips; 31 transcripts ended at exactly 224 tokens. Double the limit as below.

# pip install faster-whisper transformers
from ctranslate2.converters import TransformersConverter
from transformers import WhisperFeatureExtractor, WhisperTokenizerFast
from faster_whisper import WhisperModel

repo, out = "sajilck/whisper-small-malayalam-v2", "whisper-small-ml-v2-ct2-int8"

# One-time conversion. faster-whisper reads tokenizer.json and preprocessor_config.json
# from this folder; without them it silently uses the stock OpenAI tokenizer.
TransformersConverter(repo).convert(out, quantization="int8", force=True)
WhisperTokenizerFast.from_pretrained(repo).save_pretrained(out)
WhisperFeatureExtractor.from_pretrained(repo).save_pretrained(out)

model = WhisperModel(out, device="cpu", compute_type="int8", cpu_threads=2)
model.max_length = 888                 # CTranslate2 emits at most max_length // 2 = 444 tokens

segments, _ = model.transcribe(
    "your_audio.wav", language="ml", beam_size=1, temperature=0.0,
    without_timestamps=True, condition_on_previous_text=False,
    max_new_tokens=884,                # 888 minus the 4 prompt tokens
)
print(" ".join(s.text.strip() for s in segments))

On recent ctranslate2 versions with transformers older than 4.56, the converter fails with an unexpected dtype argument; upgrading transformers for the conversion step fixes it. The benchmark's adapter handles both. On a GPU, use compute_type="int8_float16" or "float16".

whisper.cpp β€” CPU inference (no GPU)

GGML builds for whisper.cpp are in the ggml/ folder. They run on any laptop or small server without a GPU.

Variant File Size Notes
F16 ggml/ggml-model-f16.bin ~488 MB Full precision
Q8_0 ggml/ggml-model-q8_0.bin ~264 MB Recommended; used by the demo Space

Both were converted from this model with whisper.cpp's convert-h5-to-ggml.py.

Accuracy and speed on the same 231 leakage-free clips as the results above, greedy decoding, patched build (see below). The two rows were run on different machines, so accuracy is comparable but speed is not:

Engine WER, normalised (95% CI) CER Real-time factor Size CPU used
Transformers FP32 31.2% (27.9–34.6) 4.0% 2.02 967 MB Kaggle Intel Xeon @ 2.20 GHz, 2 cores / 4 vCPU
whisper.cpp Q8_0 31.4% (28.3–34.7) 4.1% 1.61 264 MB Kaggle AMD EPYC 7B12, 2 cores / 4 threads, build compiled natively for that CPU

Q8_0 quantisation costs about 0.2 WER points, at a quarter of the size. Do not read 1.61 against 2.02 as a measured speed-up: the CPU and the build differ between the rows, and part of the gain may come from the native build. A same-session comparison of Transformers, CTranslate2 and whisper.cpp is planned. For a like-for-like speed comparison, see the CTranslate2 table above. whisper.cpp Q8_0 is still slower than real time on 2 CPU cores.

Important: raise whisper.cpp's token limit for Malayalam. Stock whisper.cpp decodes at most 220 tokens per 30-second window (n_max = whisper_n_text_ctx(ctx)/2 - 4 in src/whisper.cpp). Malayalam needs about 27 Whisper tokens per second, so speech longer than about 8 seconds is cut off without warning. 32 of the 231 test clips are that long. On the same clips (same machine for all three rows):

whisper.cpp Q8_0 build WER CER
Stock, timestamps off 37.1% 11.5%
Stock, timestamps on 40.7% 12.6%
Token limit raised to 440 31.4% 4.1%

Turning timestamps on does not help: this model was fine-tuned without timestamps and never emits them, so whisper.cpp cannot continue past the limit. The fix is a one-line change before building (step 1 below). It is the same problem as the 225-token setting described in Usage, inside a different engine.

Quick start

# 1. Build whisper.cpp, raising the per-window token limit from 220 to 440 (needed for Malayalam)
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
sed -i 's|n_max = whisper_n_text_ctx(ctx)/2 - 4;|n_max = whisper_n_text_ctx(ctx) - 8;|' src/whisper.cpp
grep -n "n_max = whisper_n_text_ctx(ctx) - 8;" src/whisper.cpp   # must print one line
cmake -B build && cmake --build build --config Release

# 2. Download the Q8_0 model
wget https://huggingface.co/sajilck/whisper-small-malayalam-v2/resolve/main/ggml/ggml-model-q8_0.bin

# 3. Transcribe Malayalam audio (16 kHz mono WAV)
./build/bin/whisper-cli -m ggml-model-q8_0.bin -l ml -f your_audio.wav

Convert other audio to the right format with:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

From Python, pywhispercpp loads the same file: Model("ggml-model-q8_0.bin", language="ml", no_timestamps=True, single_segment=True). The PyPI wheel has the 220-token limit built in. To lift it, build pywhispercpp from source with the same one-line change in its whisper.cpp/src/whisper.cpp submodule; the scores above used pywhispercpp v1.5.1 built this way.

Training details

Parameter Value
Base model openai/whisper-small (244M parameters)
Training data sajilck/malayalam-asr-corpus train split, cleaned to 69,437 rows
Steps / batch size 10,000 / 16
Learning rate 1e-4, linear decay, 500 warmup steps
Precision fp16, gradient checkpointing
Hardware one free-tier Kaggle GPU, single session
Framework transformers 4.46.3, Seq2SeqTrainer

Training data sources: IMaSC, SMC Malayalam Speech Corpus, IndicTTS Malayalam, OpenSLR 63, CommonVoice 25 Malayalam and Shrutilipi (All India Radio news). Shrutilipi is the largest. See the v1 card for links and the leakage audit of the corpus. The leakage-free test list (own_corpus_test_clean.csv) and the exact 231-clip sample are in malayalam-asr-eval.

Limitations

  • Read and broadcast speech only. Training data is read, scripted, crowdsourced or radio-news speech. Expect lower accuracy on spontaneous conversation, dialects and code-mixed Malayalam–English.
  • Thin test coverage for some sources. Only 44 IMaSC, 32 OpenSLR 63 and 5 SMC test rows survive leakage filtering, so their scores have wide error bars.
  • Possible speaker overlap between train and test (see above).
  • Spelling and word-boundary variation. Many remaining word errors are valid alternative spellings or compound words split differently from the reference.
  • Not real-time on 2 CPU cores. Real-time factor is about 2 with Transformers, about 1.2 with CTranslate2 int8 (same session as Transformers), and 1.6 with whisper.cpp Q8_0 (measured on a different CPU, so not directly comparable). Whisper's tokenizer needs about 27 tokens per second of Malayalam speech, and each token is a decoding step. A CTC model such as IndicConformer is real-time on the same hardware (see Comparison).
  • Default output limits cut off Malayalam. Stock whisper.cpp stops at 220 tokens per window and faster-whisper at 224, so speech longer than about 8 seconds is cut off unless the limit is raised (see CTranslate2 and whisper.cpp).

Citation

@misc{sajilck2026whispermalayalamv2,
  author = {Sajil C.K.},
  title = {Whisper Small Malayalam v2},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/sajilck/whisper-small-malayalam-v2}
}

License

Apache 2.0, consistent with the base Whisper model. The training corpora keep their own licenses; check each source dataset before commercial use.

Downloads last month
191
Safetensors
Model size
0.2B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for sajilck/whisper-small-malayalam-v2

Finetuned
(3,860)
this model

Dataset used to train sajilck/whisper-small-malayalam-v2

Spaces using sajilck/whisper-small-malayalam-v2 2

Evaluation results

  • WER (normalised) on malayalam-asr-corpus test, leakage-filtered (231-clip source-balanced sample)
    test set self-reported
    31.160
  • CER (normalised) on malayalam-asr-corpus test, leakage-filtered (231-clip source-balanced sample)
    test set self-reported
    4.040