ΰ΄šΰ΅†ΰ΄΅ΰ΄Ώ Mozhi Chevi

Mozhi Chevi

Chevi (ΰ΄šΰ΅†ΰ΄΅ΰ΄Ώ) means ear. This is a single 1.94 MB ONNX model that listens to Malayalam speech and tells you how it is spoken:

  • style tags: gender, pitch, speaking rate, expressiveness, recording quality, emotion
  • a caption built from those tags, in the same style as the dataset's prompt-TTS captions
  • measurements: Median F0, F0 variability, Syllables / s, PESQ, STOI, SI-SDR, where PESQ / STOI / SI-SDR are estimated without a clean reference
  • a 128-d style embedding: clips with similar delivery land close together

It takes raw 16 kHz audio (the spectrogram is part of the graph), runs in about 5 ms for 3 s of audio on one CPU thread, and was trained on a CPU in 26 minutes.

Take Demo here - https://huggingface.co/spaces/nuhmanpk/mozhi-chevi-demo

What it hears

Clip Chevi says Dataset annotation
imasc_f6a35ad0109541 A male speaker with a very high-pitch voice reads in a calm, neutral tone, speaking slightly slowly. The delivery is monotone, and the recording is clean. In a calm, neutral tone, a male speaker with a very high-pitch voice reads slightly slowly, sounding slightly expressive and animated. Recording quality: slightly noisy.
imasc_7ff8c733a85d3f A female speaker with a very high-pitch voice reads in a calm, neutral tone, speaking fast. The delivery is very expressive and animated, and the recording is noisy. A female speaker with a very high-pitch voice reads in a cheerful tone, speaking fast. The delivery is expressive and animated, and the recording is slightly clean.
imasc_f800c369e0dd73 A male speaker with a high-pitch voice reads in a calm, neutral tone, speaking at a moderate speed. The delivery is monotone, and the recording is very clean. In a calm, neutral tone, a male speaker with a high-pitch voice reads at a moderate speed, sounding very monotone. Recording quality: slightly noisy.
imasc_7a36e882ea1d32 A male speaker with a slightly high-pitch voice reads in a calm, neutral tone, speaking at a moderate speed. The delivery is monotone, and the recording is clean. A male speaker speaks slightly fast in a calm, neutral tone. His slightly high-pitch voice sounds monotone; the recording is clean.

Results

Test split of nuhmanpk/mozhi-malayalam-speech, 767 clips. Two baselines keep this honest: overall majority always answers the most common label, and per-voice majority knows who is speaking and answers that speaker's usual label. Beating the per-voice baseline means the model hears the clip, not just the voice.

Tag Accuracy Macro F1 Within one step Per-voice majority Overall majority
Gender 100.0% 1.000 - 100.0% 49.0%
Pitch 100.0% 1.000 100.0% 100.0% 67.9%
Speaking rate 45.1% 0.441 83.2% 29.9% 14.1%
Expressiveness 39.4% 0.387 78.1% 25.6% 19.0%
Recording quality 34.8% 0.346 69.9% 31.3% 13.3%
Emotion 73.0% 0.472 - 69.2% 65.0%
Measurement MAE Pearson r r within a voice MAE, per-voice mean MAE, overall mean
Median F0 6.452 Hz 0.993 0.495 11.126 58.639
F0 variability 3.480 Hz 0.867 0.448 4.539 9.198
Syllables / s 0.266 syl/s 0.895 0.614 0.442 0.605
PESQ 0.178 0.818 0.409 0.197 0.345
STOI 0.004 0.866 0.193 0.005 0.010
SI-SDR 2.040 dB 0.693 0.367 2.482 3.239

Tags

Measures

Voices it has never heard

A second copy was trained with 3 voices removed (mlf_04310, mlm_03993, WnrwjCDN1PgNuCK4IrstUpDLJB93) and tested on 172 of their clips. This is the number that matters for new speakers or TTS output.

Accuracy (unseen voices) Overall majority
Gender 97.7% 32.6%
Pitch 0.0% 0.0%
Speaking rate 81.4% 32.6%
Expressiveness 30.2% 27.3%
Recording quality 55.8% 55.8%
Emotion 44.8% 65.1%
MAE (unseen voices) Pearson r MAE, overall mean
Median F0 12.539 0.952 52.548
F0 variability 7.099 0.865 22.645
Syllables / s 0.579 0.925 1.539
PESQ 0.308 0.519 0.722
STOI 0.014 0.577 0.015
SI-SDR 2.954 0.537 3.138

Confusion

Embedding

Use it

Everything needed to read the outputs (label lists, units, caption template) is stored inside the .onnx file as metadata under the key mozhi_chevi.

Python

import json, numpy as np, onnxruntime as ort, soundfile as sf, librosa
from huggingface_hub import hf_hub_download

sess = ort.InferenceSession(hf_hub_download("nuhmanpk/mozhi-chevi", "mozhi_chevi.onnx"))
spec = json.loads(sess.get_modelmeta().custom_metadata_map["mozhi_chevi"])

audio, sr = sf.read("clip.wav", dtype="float32", always_2d=True)
audio = librosa.resample(audio.mean(1), orig_sr=sr, target_sr=16000)
out = dict(zip([o.name for o in sess.get_outputs()], sess.run(None, {"audio": audio[None]})))

tags = {k: v[int(out[k][0].argmax())] for k, v in spec["tags"].items()}
measures = {k: float(out[k][0]) for k in spec["measures"]}
print(tags)
print(measures)   # f0_median Hz, syl_per_sec, pesq, stoi, si_sdr dB ...

Browser (onnxruntime-web), live from a microphone or a file

import * as ort from "onnxruntime-web";
const s = await ort.InferenceSession.create("https://huggingface.co/nuhmanpk/mozhi-chevi/resolve/main/mozhi_chevi.onnx");
// samples: Float32Array, mono 16 kHz (decode + resample with an OfflineAudioContext)
const out = await s.run({ audio: new ort.Tensor("float32", samples, [1, samples.length]) });
const argmax = a => a.indexOf(Math.max(...a));
const TAGS = {"gender": ["male", "female", "unknown"], "pitch": ["very low-pitch", "low-pitch", "slightly low-pitch", "slightly high-pitch", "moderate pitch", "high-pitch", "very high-pitch"], "speaking_rate": ["very slowly", "slowly", "slightly slowly", "at a moderate speed", "slightly fast", "fast", "very fast"], "expressiveness": ["very monotone", "monotone", "slightly expressive and animated", "expressive and animated", "very expressive and animated"], "recording_quality": ["very noisy", "noisy", "slightly noisy", "balanced in clarity", "slightly clean", "clean", "very clean"], "emotion_label": ["neutral", "happy", "sad", "disgusted", "surprised", "angry"]};
const tags = Object.fromEntries(Object.entries(TAGS).map(([k, v]) => [k, v[argmax([...out[k].data])]]));
const pesq = out.pesq?.data[0], rate = out.syl_per_sec?.data[0];

Ideas

  • Judge a TTS model without references. Run Chevi on synthesised speech: estimated PESQ/STOI/SI-SDR, pace and pitch tell you whether a checkpoint got better, and whether the speed control really changes syllables per second.
  • Caption new recordings for prompt-based TTS training, in the same vocabulary as this dataset.
  • Filter data: drop clips Chevi rates as noisy or low-PESQ before training anything.
  • Live feedback in a browser page: a reading-practice meter that shows pace, pitch range and recording quality as someone speaks.
  • Search by delivery: nearest neighbours in the style embedding find clips spoken the same way.

How it was made

  • Data: 17444 train / 754 validation / 767 test clips (26.2 h, 124 speakers, sources imasc, openslr63, smc_msc), using the dataset's own splits. Training clips are capped per speaker.
  • Targets: the dataset's per-clip annotations. Tags rarer than 30 training clips are ignored; ordered tags (pitch, rate, expressiveness, quality) are sorted by their underlying measurement.
  • Model: fixed-DFT log-mel front end (80 bands) β†’ dilated TDNN with residual blocks β†’ mean+std pooling β†’ 128-d style vector β†’ one softmax per tag + one regression layer. 0.30 M parameters.
  • Training: CPU only (Intel(R) Xeon(R) CPU @ 2.20GHz, 2 threads), 20 epochs Γ— 150 steps, batch 64, random 2 to 5 s crops, time masking, AdamW + one-cycle LR (0.003), class-balanced cross-entropy + Smooth-L1. Best epoch by validation: 20.
  • ONNX check: outputs match PyTorch to a relative difference of 3.43e-06.

Training

Limits

  • Chevi imitates the dataset's automatic annotations. Where those are noisy (emotion especially, which is mostly neutral), so is Chevi.
  • Quality numbers are estimates learned from studio read speech; treat them as relative scores, not lab-grade PESQ or STOI.
  • Tags such as pitch partly follow the speaker; the unseen-voice table above shows how far it carries to new voices.
  • Built for Malayalam read speech at 16 kHz. Other languages, singing, or heavy music will be outside what it has heard.

License and citation

Trained on nuhmanpk/mozhi-malayalam-speech (licenses in the training data: CC-BY-SA-4.0); this model follows the same terms. The IMaSC recordings come from:

@article{gopinath2022imasc,
  title   = {IMaSC -- ICFOSS Malayalam Speech Corpus},
  author  = {Gopinath, Deepa P and D K, Thennal and Nair, Vrinda V and K S, Swaraj and G, Sachin},
  journal = {arXiv preprint arXiv:2211.12796},
  year    = {2022}
}

Card generated 2026-10-07.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train nuhmanpk/mozhi-chevi

Space using nuhmanpk/mozhi-chevi 1

Paper for nuhmanpk/mozhi-chevi

Evaluation results