MeloTTS (Korean + English) for Sentis (com.sky.sentis.melotts)

Unity Sentis (com.unity.ai.inference 2.6.1) port of MyShell MeloTTS for Korean (MeloTTS-Korean) and English (MeloTTS-English). Each language ships two fp32 graphs: the language BERT (feature extractor) and the VITS synthesizer. The stochastic duration predictor's noise is baked into a fixed pool, so the same input always gives the same audio.

using MeloTTS;
using Unity.InferenceEngine;

using var tts = new MeloTts("ko", backend: BackendType.GPUCompute);   // loads StreamingAssets/MeloTTS
var phrases = MeloPhrases.Load(Path.Combine(MeloTts.DefaultModelDirectory, MeloTts.PhrasesFile("ko")));
tts.Warmup(phrases[0].Sentences[0]);

float[] pcm = tts.Synthesize(phrases[0].Sentences);                   // 44.1 kHz mono

Install

Add the package by git URL, then run Tools ▸ MeloTTS ▸ Install Models to StreamingAssets (or call MeloTTS.Editor.MeloBaker.Bake()). Model~ is hidden from the asset pipeline and not included in player builds, so the .sentis files are copied to Assets/StreamingAssets/MeloTTS. Both languages are about 1.1 GB together; delete the language you don't need from StreamingAssets.

https://huggingface.co/Sky-Kim/com.sky.sentis.melotts

Text frontend (offline)

The graphs take the output of MeloTTS's own text frontend (g2pkk + jamo for Korean, g2p_en for English): phone ids, tone ids and language ids with blanks interspersed, plus BERT word-piece ids and word2ph. There is no in-Unity G2P. Model~/phrases_<lang>.json holds precomputed inputs for a test corpus (already split into sentences); generate your own with Tools~/make_phrases.py <lang>.

Layout

Path	What
Model~/bert_ko.sentis	kykim/bert-kor-base, embeddings + first 10 layers (= hidden_states[-3])
Model~/bert_en.sentis	bert-base-uncased, embeddings + first 10 layers
Model~/melo_ko.sentis, melo_en.sentis	VITS synthesizer (text encoder, stochastic + deterministic duration, flow, HiFi-GAN decoder)
Model~/phrases_ko.json, phrases_en.json	precomputed frontend output for the benchmark corpus
Model~/fixtures/<lang>/	PyTorch / ONNX Runtime reference tensors used by the tests
Runtime/	MeloTts (runner), MeloInput, MeloPhrases
Editor/MeloBaker.cs	copies Model~ into StreamingAssets
Tests/Editor/	parity per graph (CPU, GPUCompute) and end-to-end synthesis, both languages
Tools~/	export_melo.py <lang> (graphs + fixture), make_phrases.py <lang>, melo_common.py

Graph contract

bert_<lang>: input_ids, attention_mask (ones), token_type_ids (zeros), int [1,N] → hidden [1,N,768].

Host step (MeloTts.ExpandWord2Ph): ja_bert[0,:,t] = hidden[0,i,:] repeated word2ph[i] times, Σ word2ph = T.

melo_<lang>:

Input	Shape	Notes
x, tones, lang_ids	int [1,T]	blanks already interspersed
sid	int [1]	KR 0; EN 0 EN-US, 1 EN-BR, 2 EN_INDIA, 3 EN-AU, 4 EN-Default
ja_bert	float [1,768,T]	from the BERT graph
scales	float [4]	[noise_scale 0.6, length_scale 1.0, noise_scale_w 0.8, sdp_ratio 0.2]

Output audio [1,1,S] at 44.1 kHz. Limits from the baked noise pool: T ≤ 8192 and ≤ 8192 frames (~95 s) per call. The 1024-d Chinese bert input of MeloTTS is always zero for KR/EN, so it is built inside the graph.

Export notes

• remove_weight_norm on all 128 weight-normed modules.
• torch.randn / randn_like are patched to slice one fixed N(0,1) pool [1,192,8192] (seed 1234) in both the reference forward and the export, so the noise is baked, not zeroed.
• The masked-index rational-quadratic spline in the stochastic duration predictor exported as NonZero/GatherND/ScatterND (NonZero is CPU-only in Sentis). It is rewritten as a dense clamp + Where; bit-identical in torch.

Verification (Sentis 2.6.1, Unity 6000.5.5f1, Apple Silicon)

Edge	KR	EN
PyTorch vs ONNX Runtime (synthesizer, fixture)	1.2e-5	8.8e-6
BERT ONNX vs HF hidden_states[-3]	7.9e-6	1.6e-5
End to end ORT (BERT → expansion → synthesizer) vs torch	4.1e-5	3.0e-5
Reference vs Sentis CPU (audio)	3.7e-6 (SNR 114 dB)	6.1e-6 (SNR 101 dB)
Reference vs Sentis GPUCompute (audio)	1.8e-5 (SNR 112 dB)	3.7e-6 (SNR 104 dB)
Back-transcription (Whisper small)	CER 3.8 %	WER 5.3 %
UTMOS	3.19	2.75

Benchmark (Apple M5 Max, standalone player, warm median, BERT included)

	KR RTF	KR TTFA	EN RTF	EN TTFA
GPUCompute	0.023	82 ms	0.022	66 ms
CPU	0.072	239 ms	0.070	200 ms

The synthesizer has about 9k small ops and a data-dependent output length (one GPU→CPU sync per call), so GPU gains are smaller than for convolution-heavy vocoders. Measure on your own target device.

License

MeloTTS code and weights © MyShell.ai, MIT. bert-base-uncased © Google, Apache-2.0. kykim/bert-kor-base declares no license on its model card (the upstream LMkor project is reported as Apache-2.0); confirm before commercial use. See LICENSE.md.