Automatic Speech Recognition
Transformers
Vietnamese
vietnamese
whisper
speech-to-text
rain1024 Claude Opus 4.6 commited on
Commit
5763d9e
·
1 Parent(s): 4a30ce6

Initial commit: ASR-1 Vietnamese speech recognition model

Browse files

Vietnamese ASR based on fine-tuned Whisper with training, evaluation,
and transcription scripts. Supports Common Voice and VIVOS datasets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (10) hide show
  1. .gitattributes +1 -33
  2. .gitignore +59 -0
  3. CLAUDE.md +66 -0
  4. README.md +133 -0
  5. pyproject.toml +47 -0
  6. src/__init__.py +0 -0
  7. src/data.py +96 -0
  8. src/evaluate.py +123 -0
  9. src/train.py +239 -0
  10. src/transcribe.py +110 -0
.gitattributes CHANGED
@@ -1,35 +1,3 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
  *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  *.pt filter=lfs diff=lfs merge=lfs -text
2
+ *.bin filter=lfs diff=lfs merge=lfs -text
 
3
  *.safetensors filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
.gitignore ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Environment
2
+ .env
3
+ .env.*
4
+
5
+ # Python
6
+ __pycache__/
7
+ *.py[cod]
8
+ *$py.class
9
+ *.so
10
+ .Python
11
+ build/
12
+ develop-eggs/
13
+ dist/
14
+ downloads/
15
+ eggs/
16
+ .eggs/
17
+ lib/
18
+ lib64/
19
+ parts/
20
+ sdist/
21
+ var/
22
+ wheels/
23
+ *.egg-info/
24
+ .installed.cfg
25
+ *.egg
26
+
27
+ # Virtual environments
28
+ .venv/
29
+ venv/
30
+ ENV/
31
+ env/
32
+
33
+ # IDE
34
+ .idea/
35
+ .vscode/
36
+ *.swp
37
+ *.swo
38
+ *~
39
+
40
+ # Data (large files)
41
+ data/
42
+ tmp/
43
+
44
+ # Logs
45
+ *.log
46
+ wandb/
47
+
48
+ # Jupyter
49
+ .ipynb_checkpoints/
50
+
51
+ # OS
52
+ .DS_Store
53
+ Thumbs.db
54
+
55
+ # Audio cache
56
+ *.wav
57
+ *.mp3
58
+ *.flac
59
+ !tests/fixtures/*.wav
CLAUDE.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CLAUDE.md
2
+
3
+ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
+
5
+ ## Project Overview
6
+
7
+ ASR-1 is a Vietnamese automatic speech recognition model based on fine-tuned Whisper. Part of the UnderTheSea NLP ecosystem, it provides speech-to-text capability for Vietnamese audio.
8
+
9
+ ## Build & Development Commands
10
+
11
+ ### Setup
12
+ ```bash
13
+ uv sync # Install dependencies
14
+ uv sync --extra dev # Include pytest and wandb
15
+ uv sync --extra cloud # Include runpod for cloud training
16
+ ```
17
+
18
+ ### Training
19
+ ```bash
20
+ uv run src/train.py # Default training
21
+ uv run src/train.py --base-model openai/whisper-large-v3 # Specify base model
22
+ uv run src/train.py --dataset vivos # Train on VIVOS
23
+ uv run src/train.py --wandb --wandb-project asr-1 # With W&B logging
24
+ ```
25
+
26
+ ### Evaluation
27
+ ```bash
28
+ uv run src/evaluate.py --model models/asr-1 # Evaluate on test set
29
+ uv run src/evaluate.py --model models/asr-1 --dataset vivos # Evaluate on VIVOS
30
+ ```
31
+
32
+ ### Running Tests
33
+ ```bash
34
+ pytest tests/
35
+ ```
36
+
37
+ ## Architecture
38
+
39
+ **Pipeline:**
40
+ ```
41
+ Audio (WAV/MP3) → Resample to 16kHz
42
+ → Log-Mel Spectrogram (80 mel bins, 30s chunks)
43
+ → Whisper Encoder (transformer)
44
+ → Whisper Decoder (transformer, autoregressive)
45
+ → Vietnamese Text
46
+ ```
47
+
48
+ **Key Design Decisions:**
49
+ - Fine-tune Whisper large-v3 for Vietnamese-specific improvements
50
+ - Use LoRA for parameter-efficient fine-tuning
51
+ - Evaluate on both Common Voice and VIVOS benchmarks
52
+ - Compatible with underthesea API
53
+
54
+ ## Key Files
55
+
56
+ - `src/train.py` - Training script (fine-tune Whisper)
57
+ - `src/evaluate.py` - Evaluation script (WER/CER metrics)
58
+ - `src/transcribe.py` - Inference/transcription script
59
+ - `src/data.py` - Dataset loading utilities
60
+ - `TECHNICAL_REPORT.md` - Detailed methodology and benchmark results
61
+
62
+ ## RunPod
63
+
64
+ - **API keys**: `~/.env` (RUNPOD_API_KEY, WANDB_API_KEY, HF_TOKEN)
65
+ - **GPU requirements**: Fine-tuning Whisper large-v3 requires ~24GB VRAM
66
+ - RTX 4090 (24G) or A100 (40/80G) recommended
README.md ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - vi
5
+ tags:
6
+ - automatic-speech-recognition
7
+ - vietnamese
8
+ - whisper
9
+ - speech-to-text
10
+ library_name: transformers
11
+ pipeline_tag: automatic-speech-recognition
12
+ metrics:
13
+ - wer
14
+ - cer
15
+ datasets:
16
+ - mozilla-foundation/common_voice_17_0
17
+ - VIVOS
18
+ ---
19
+
20
+ # ASR-1
21
+
22
+ ASR-1 is a Vietnamese automatic speech recognition model developed by UnderTheSea NLP.
23
+
24
+ ## Model Description
25
+
26
+ - **Model Type:** Fine-tuned Whisper for Vietnamese ASR
27
+ - **Base Model:** openai/whisper-large-v3
28
+ - **Language:** Vietnamese
29
+ - **License:** Apache 2.0
30
+ - **Task:** Automatic Speech Recognition (Speech-to-Text)
31
+
32
+ ## Installation
33
+
34
+ ```bash
35
+ pip install transformers torch torchaudio datasets
36
+ ```
37
+
38
+ ## Usage
39
+
40
+ ```python
41
+ from transformers import WhisperProcessor, WhisperForConditionalGeneration
42
+ from huggingface_hub import snapshot_download
43
+ import torchaudio
44
+
45
+ # Download model
46
+ model_path = snapshot_download('undertheseanlp/asr-1')
47
+
48
+ # Load model and processor
49
+ processor = WhisperProcessor.from_pretrained(model_path)
50
+ model = WhisperForConditionalGeneration.from_pretrained(model_path)
51
+
52
+ # Transcribe audio
53
+ waveform, sample_rate = torchaudio.load("audio.wav")
54
+ if sample_rate != 16000:
55
+ waveform = torchaudio.transforms.Resample(sample_rate, 16000)(waveform)
56
+
57
+ input_features = processor(
58
+ waveform.squeeze().numpy(),
59
+ sampling_rate=16000,
60
+ return_tensors="pt"
61
+ ).input_features
62
+
63
+ predicted_ids = model.generate(input_features, language="vi", task="transcribe")
64
+ transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
65
+ print(transcription)
66
+ ```
67
+
68
+ ## API (compatible with underthesea)
69
+
70
+ ```python
71
+ from asr import AsrTranscriber, transcribe
72
+
73
+ # Quick transcription
74
+ text = transcribe("audio.wav")
75
+ print(text)
76
+
77
+ # With model instance
78
+ transcriber = AsrTranscriber.load("models/asr-1")
79
+ result = transcriber.transcribe("audio.wav")
80
+ print(result.text)
81
+ print(result.confidence)
82
+ ```
83
+
84
+ ## Training
85
+
86
+ ```bash
87
+ uv run src/train.py
88
+ uv run src/train.py --base-model openai/whisper-large-v3 --dataset common_voice
89
+ uv run src/train.py --wandb --wandb-project asr-1
90
+ ```
91
+
92
+ ## Evaluation
93
+
94
+ ```bash
95
+ uv run src/evaluate.py --model models/asr-1
96
+ uv run src/evaluate.py --model models/asr-1 --dataset vivos
97
+ ```
98
+
99
+ ## Datasets
100
+
101
+ | Dataset | Split | Hours | Samples |
102
+ |---------|-------|-------|---------|
103
+ | Common Voice 17.0 (vi) | train | ~30h | ~25,000 |
104
+ | Common Voice 17.0 (vi) | test | ~5h | ~5,000 |
105
+ | VIVOS | train | 15h | 11,660 |
106
+ | VIVOS | test | 0.6h | 760 |
107
+
108
+ ## Metrics
109
+
110
+ - **WER (Word Error Rate)**: Lower is better
111
+ - **CER (Character Error Rate)**: Lower is better
112
+
113
+ ## References
114
+
115
+ - [Whisper: Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
116
+ - [Common Voice Dataset](https://commonvoice.mozilla.org/)
117
+ - [VIVOS: Vietnamese Speech Corpus](https://ailab.hcmus.edu.vn/vivos)
118
+ - [Underthesea NLP Toolkit](https://github.com/undertheseanlp/underthesea)
119
+
120
+ ## Citation
121
+
122
+ ```bibtex
123
+ @article{radford2022whisper,
124
+ title={Robust Speech Recognition via Large-Scale Weak Supervision},
125
+ author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
126
+ journal={arXiv preprint arXiv:2212.04356},
127
+ year={2022}
128
+ }
129
+ ```
130
+
131
+ ## Technical Report
132
+
133
+ See [TECHNICAL_REPORT.md](TECHNICAL_REPORT.md) for detailed methodology and evaluation.
pyproject.toml ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [project]
2
+ name = "asr"
3
+ version = "0.1.0"
4
+ description = "Vietnamese Automatic Speech Recognition - Fine-tuned Whisper for underthesea"
5
+ readme = "README.md"
6
+ requires-python = ">=3.10"
7
+ license = "Apache-2.0"
8
+ authors = [
9
+ {name = "UnderTheSea NLP", email = "undertheseanlp@gmail.com"}
10
+ ]
11
+ keywords = ["vietnamese", "nlp", "asr", "speech-recognition", "whisper"]
12
+ dependencies = [
13
+ "torch>=2.0.0",
14
+ "torchaudio>=2.0.0",
15
+ "transformers>=4.36.0",
16
+ "datasets>=2.14.0",
17
+ "click>=8.0.0",
18
+ "tqdm>=4.60.0",
19
+ "python-dotenv>=1.0.0",
20
+ "huggingface_hub>=0.20.0",
21
+ "librosa>=0.10.0",
22
+ "jiwer>=3.0.0",
23
+ "soundfile>=0.12.0",
24
+ ]
25
+
26
+ [project.optional-dependencies]
27
+ dev = [
28
+ "pytest>=7.0.0",
29
+ "wandb>=0.15.0",
30
+ ]
31
+ cloud = [
32
+ "runpod>=1.6.0",
33
+ ]
34
+ peft = [
35
+ "peft>=0.7.0",
36
+ ]
37
+
38
+ [project.urls]
39
+ Homepage = "https://huggingface.co/undertheseanlp/asr-1"
40
+ Repository = "https://github.com/undertheseanlp/asr"
41
+
42
+ [build-system]
43
+ requires = ["hatchling"]
44
+ build-backend = "hatchling.build"
45
+
46
+ [tool.hatch.build.targets.wheel]
47
+ packages = ["src"]
src/__init__.py ADDED
File without changes
src/data.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # requires-python = ">=3.10"
3
+ # dependencies = [
4
+ # "datasets>=2.14.0",
5
+ # "torchaudio>=2.0.0",
6
+ # "transformers>=4.36.0",
7
+ # ]
8
+ # ///
9
+ """
10
+ Dataset loading utilities for Vietnamese ASR.
11
+
12
+ Supports:
13
+ - Common Voice 17.0 (Vietnamese)
14
+ - VIVOS
15
+ """
16
+
17
+ from pathlib import Path
18
+ from typing import Optional
19
+
20
+ from datasets import load_dataset, Audio
21
+
22
+
23
+ def load_common_voice(
24
+ split: str = "train",
25
+ streaming: bool = False,
26
+ cache_dir: Optional[str] = None,
27
+ ) -> "Dataset":
28
+ """Load Common Voice 17.0 Vietnamese dataset.
29
+
30
+ Args:
31
+ split: Dataset split ("train", "validation", "test")
32
+ streaming: Whether to use streaming mode
33
+ cache_dir: Custom cache directory
34
+
35
+ Returns:
36
+ HuggingFace Dataset with audio column cast to 16kHz
37
+ """
38
+ ds = load_dataset(
39
+ "mozilla-foundation/common_voice_17_0",
40
+ "vi",
41
+ split=split,
42
+ streaming=streaming,
43
+ cache_dir=cache_dir,
44
+ trust_remote_code=True,
45
+ )
46
+ ds = ds.cast_column("audio", Audio(sampling_rate=16000))
47
+ # Keep only relevant columns
48
+ ds = ds.remove_columns(
49
+ [c for c in ds.column_names
50
+ if c not in ("audio", "sentence")]
51
+ )
52
+ return ds
53
+
54
+
55
+ def load_vivos(
56
+ split: str = "train",
57
+ cache_dir: Optional[str] = None,
58
+ ) -> "Dataset":
59
+ """Load VIVOS Vietnamese speech dataset.
60
+
61
+ Args:
62
+ split: Dataset split ("train", "test")
63
+ cache_dir: Custom cache directory
64
+
65
+ Returns:
66
+ HuggingFace Dataset with audio column cast to 16kHz
67
+ """
68
+ ds = load_dataset(
69
+ "vivos",
70
+ split=split,
71
+ cache_dir=cache_dir,
72
+ trust_remote_code=True,
73
+ )
74
+ ds = ds.cast_column("audio", Audio(sampling_rate=16000))
75
+ ds = ds.rename_column("sentence", "sentence")
76
+ return ds
77
+
78
+
79
+ def prepare_dataset(batch, processor):
80
+ """Prepare a batch for Whisper training.
81
+
82
+ Args:
83
+ batch: Dataset batch with 'audio' and 'sentence' columns
84
+ processor: WhisperProcessor instance
85
+
86
+ Returns:
87
+ Batch with input_features and labels
88
+ """
89
+ audio = batch["audio"]
90
+ batch["input_features"] = processor.feature_extractor(
91
+ audio["array"],
92
+ sampling_rate=audio["sampling_rate"],
93
+ ).input_features[0]
94
+
95
+ batch["labels"] = processor.tokenizer(batch["sentence"]).input_ids
96
+ return batch
src/evaluate.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # requires-python = ">=3.10"
3
+ # dependencies = [
4
+ # "torch>=2.0.0",
5
+ # "torchaudio>=2.0.0",
6
+ # "transformers>=4.36.0",
7
+ # "datasets>=2.14.0",
8
+ # "click>=8.0.0",
9
+ # "tqdm>=4.60.0",
10
+ # "jiwer>=3.0.0",
11
+ # ]
12
+ # ///
13
+ """
14
+ Evaluation script for ASR-1 Vietnamese Speech Recognition.
15
+
16
+ Computes WER and CER on test datasets.
17
+
18
+ Usage:
19
+ uv run src/evaluate.py --model models/asr-1
20
+ uv run src/evaluate.py --model models/asr-1 --dataset vivos
21
+ uv run src/evaluate.py --model openai/whisper-large-v3 # baseline
22
+ """
23
+
24
+ import sys
25
+ from pathlib import Path
26
+
27
+ from dotenv import load_dotenv
28
+ load_dotenv()
29
+
30
+ import torch
31
+ import click
32
+ from tqdm import tqdm
33
+ from transformers import WhisperProcessor, WhisperForConditionalGeneration
34
+ from jiwer import wer, cer
35
+
36
+ sys.path.insert(0, str(Path(__file__).parent.parent))
37
+ from src.data import load_common_voice, load_vivos
38
+
39
+
40
+ @click.command()
41
+ @click.option('--model', '-m', required=True, help='Model path or HuggingFace model ID')
42
+ @click.option('--dataset', type=click.Choice(['common_voice', 'vivos', 'both']), default='common_voice',
43
+ help='Evaluation dataset')
44
+ @click.option('--batch-size', default=8, type=int, help='Batch size')
45
+ @click.option('--cache-dir', default=None, help='Dataset cache directory')
46
+ @click.option('--num-samples', default=0, type=int, help='Number of samples to evaluate (0=all)')
47
+ def evaluate_model(model, dataset, batch_size, cache_dir, num_samples):
48
+ """Evaluate ASR-1 model on Vietnamese speech benchmarks."""
49
+
50
+ device = "cuda" if torch.cuda.is_available() else "cpu"
51
+ click.echo(f"Using device: {device}")
52
+
53
+ click.echo("=" * 60)
54
+ click.echo("ASR-1: Evaluation")
55
+ click.echo("=" * 60)
56
+
57
+ # Load model
58
+ click.echo(f"\nLoading model: {model}")
59
+ processor = WhisperProcessor.from_pretrained(model)
60
+ asr_model = WhisperForConditionalGeneration.from_pretrained(model).to(device)
61
+ asr_model.eval()
62
+
63
+ # Determine datasets to evaluate
64
+ datasets_to_eval = []
65
+ if dataset in ("common_voice", "both"):
66
+ datasets_to_eval.append(("Common Voice (test)", load_common_voice("test", cache_dir=cache_dir)))
67
+ if dataset in ("vivos", "both"):
68
+ datasets_to_eval.append(("VIVOS (test)", load_vivos("test", cache_dir=cache_dir)))
69
+
70
+ for ds_name, ds in datasets_to_eval:
71
+ if num_samples > 0:
72
+ ds = ds.select(range(min(num_samples, len(ds))))
73
+
74
+ click.echo(f"\nEvaluating on {ds_name} ({len(ds)} samples)...")
75
+
76
+ all_predictions = []
77
+ all_references = []
78
+
79
+ for i in tqdm(range(0, len(ds), batch_size)):
80
+ batch = ds[i:i + batch_size]
81
+ audios = batch["audio"]
82
+
83
+ # Extract features
84
+ input_features = processor.feature_extractor(
85
+ [a["array"] for a in audios],
86
+ sampling_rate=16000,
87
+ return_tensors="pt",
88
+ ).input_features.to(device)
89
+
90
+ # Generate
91
+ with torch.no_grad():
92
+ predicted_ids = asr_model.generate(
93
+ input_features,
94
+ language="vi",
95
+ task="transcribe",
96
+ )
97
+
98
+ # Decode
99
+ predictions = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
100
+ references = batch["sentence"]
101
+
102
+ all_predictions.extend(predictions)
103
+ all_references.extend(references)
104
+
105
+ # Compute metrics
106
+ wer_score = 100 * wer(all_references, all_predictions)
107
+ cer_score = 100 * cer(all_references, all_predictions)
108
+
109
+ click.echo(f"\n{ds_name} Results:")
110
+ click.echo(f" WER: {wer_score:.2f}%")
111
+ click.echo(f" CER: {cer_score:.2f}%")
112
+ click.echo(f" Samples: {len(all_references)}")
113
+
114
+ # Show some examples
115
+ click.echo(f"\nExamples:")
116
+ for j in range(min(5, len(all_predictions))):
117
+ click.echo(f" REF: {all_references[j]}")
118
+ click.echo(f" HYP: {all_predictions[j]}")
119
+ click.echo()
120
+
121
+
122
+ if __name__ == '__main__':
123
+ evaluate_model()
src/train.py ADDED
@@ -0,0 +1,239 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # requires-python = ">=3.10"
3
+ # dependencies = [
4
+ # "torch>=2.0.0",
5
+ # "torchaudio>=2.0.0",
6
+ # "transformers>=4.36.0",
7
+ # "datasets>=2.14.0",
8
+ # "click>=8.0.0",
9
+ # "tqdm>=4.60.0",
10
+ # "wandb>=0.15.0",
11
+ # "python-dotenv>=1.0.0",
12
+ # "jiwer>=3.0.0",
13
+ # "huggingface_hub>=0.20.0",
14
+ # ]
15
+ # ///
16
+ """
17
+ Training script for ASR-1 Vietnamese Speech Recognition.
18
+
19
+ Fine-tunes OpenAI Whisper on Vietnamese speech datasets.
20
+
21
+ Usage:
22
+ uv run src/train.py
23
+ uv run src/train.py --base-model openai/whisper-large-v3
24
+ uv run src/train.py --dataset vivos
25
+ uv run src/train.py --wandb --wandb-project asr-1
26
+ """
27
+
28
+ import sys
29
+ from pathlib import Path
30
+ from dataclasses import dataclass
31
+ from typing import Any, Dict, List, Union
32
+
33
+ from dotenv import load_dotenv
34
+ load_dotenv()
35
+
36
+ import torch
37
+ import click
38
+ from transformers import (
39
+ WhisperProcessor,
40
+ WhisperForConditionalGeneration,
41
+ Seq2SeqTrainingArguments,
42
+ Seq2SeqTrainer,
43
+ )
44
+ from datasets import Audio
45
+ import evaluate
46
+
47
+ sys.path.insert(0, str(Path(__file__).parent.parent))
48
+ from src.data import load_common_voice, load_vivos, prepare_dataset
49
+
50
+
51
+ @dataclass
52
+ class DataCollatorSpeechSeq2SeqWithPadding:
53
+ """Data collator for Whisper speech-to-text training."""
54
+
55
+ processor: Any
56
+
57
+ def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
58
+ # Split inputs and labels
59
+ input_features = [{"input_features": f["input_features"]} for f in features]
60
+ batch = self.processor.feature_extractor.pad(input_features, return_tensors="pt")
61
+
62
+ # Pad labels
63
+ label_features = [{"input_ids": f["labels"]} for f in features]
64
+ labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt")
65
+
66
+ # Replace padding with -100 for loss computation
67
+ labels = labels_batch["input_ids"].masked_fill(
68
+ labels_batch.attention_mask.ne(1), -100
69
+ )
70
+
71
+ # Remove BOS token if present (Whisper adds it during generation)
72
+ if (labels[:, 0] == self.processor.tokenizer.bos_token_id).all().cpu().item():
73
+ labels = labels[:, 1:]
74
+
75
+ batch["labels"] = labels
76
+ return batch
77
+
78
+
79
+ def compute_metrics(pred, processor, wer_metric, cer_metric):
80
+ """Compute WER and CER metrics."""
81
+ pred_ids = pred.predictions
82
+ label_ids = pred.label_ids
83
+
84
+ # Replace -100 with pad token
85
+ label_ids[label_ids == -100] = processor.tokenizer.pad_token_id
86
+
87
+ pred_str = processor.tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
88
+ label_str = processor.tokenizer.batch_decode(label_ids, skip_special_tokens=True)
89
+
90
+ wer = 100 * wer_metric.compute(predictions=pred_str, references=label_str)
91
+ cer = 100 * cer_metric.compute(predictions=pred_str, references=label_str)
92
+
93
+ return {"wer": wer, "cer": cer}
94
+
95
+
96
+ @click.command()
97
+ @click.option('--base-model', default='openai/whisper-large-v3', help='Base Whisper model')
98
+ @click.option('--dataset', type=click.Choice(['common_voice', 'vivos', 'both']), default='common_voice',
99
+ help='Training dataset')
100
+ @click.option('--output', '-o', default='models/asr-1', help='Output directory')
101
+ @click.option('--epochs', default=3, type=int, help='Number of training epochs')
102
+ @click.option('--batch-size', default=8, type=int, help='Per-device batch size')
103
+ @click.option('--grad-accum', default=2, type=int, help='Gradient accumulation steps')
104
+ @click.option('--lr', default=1e-5, type=float, help='Learning rate')
105
+ @click.option('--warmup-steps', default=500, type=int, help='Warmup steps')
106
+ @click.option('--max-steps', default=-1, type=int, help='Max training steps (-1 for epoch-based)')
107
+ @click.option('--fp16/--no-fp16', default=True, help='Use mixed precision')
108
+ @click.option('--wandb', 'use_wandb', is_flag=True, help='Enable W&B logging')
109
+ @click.option('--wandb-project', default='asr-1', help='W&B project name')
110
+ @click.option('--push-to-hub', is_flag=True, help='Push model to HuggingFace Hub')
111
+ @click.option('--hub-model-id', default='undertheseanlp/asr-1', help='HuggingFace Hub model ID')
112
+ @click.option('--eval-steps', default=500, type=int, help='Evaluate every N steps')
113
+ @click.option('--save-steps', default=500, type=int, help='Save checkpoint every N steps')
114
+ @click.option('--cache-dir', default=None, help='Dataset cache directory')
115
+ def train(base_model, dataset, output, epochs, batch_size, grad_accum, lr,
116
+ warmup_steps, max_steps, fp16, use_wandb, wandb_project, push_to_hub,
117
+ hub_model_id, eval_steps, save_steps, cache_dir):
118
+ """Train ASR-1 Vietnamese Speech Recognition model."""
119
+
120
+ device = "cuda" if torch.cuda.is_available() else "cpu"
121
+ click.echo(f"Using device: {device}")
122
+
123
+ click.echo("=" * 60)
124
+ click.echo("ASR-1: Vietnamese Automatic Speech Recognition")
125
+ click.echo("=" * 60)
126
+
127
+ # Load processor and model
128
+ click.echo(f"\nLoading base model: {base_model}")
129
+ processor = WhisperProcessor.from_pretrained(base_model)
130
+ model = WhisperForConditionalGeneration.from_pretrained(base_model)
131
+
132
+ # Force Vietnamese language and transcription task
133
+ model.generation_config.language = "vi"
134
+ model.generation_config.task = "transcribe"
135
+ model.generation_config.forced_decoder_ids = None
136
+
137
+ n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
138
+ click.echo(f" Parameters: {n_params:,}")
139
+
140
+ # Load datasets
141
+ click.echo(f"\nLoading dataset: {dataset}")
142
+ if dataset == "common_voice":
143
+ train_ds = load_common_voice("train", cache_dir=cache_dir)
144
+ eval_ds = load_common_voice("validation", cache_dir=cache_dir)
145
+ elif dataset == "vivos":
146
+ train_ds = load_vivos("train", cache_dir=cache_dir)
147
+ eval_ds = load_vivos("test", cache_dir=cache_dir)
148
+ else: # both
149
+ from datasets import concatenate_datasets
150
+ cv_train = load_common_voice("train", cache_dir=cache_dir)
151
+ vivos_train = load_vivos("train", cache_dir=cache_dir)
152
+ train_ds = concatenate_datasets([cv_train, vivos_train])
153
+ eval_ds = load_common_voice("validation", cache_dir=cache_dir)
154
+
155
+ click.echo(f" Train: {len(train_ds)} samples")
156
+ click.echo(f" Eval: {len(eval_ds)} samples")
157
+
158
+ # Prepare datasets
159
+ click.echo("\nPreparing datasets...")
160
+ train_ds = train_ds.map(
161
+ lambda batch: prepare_dataset(batch, processor),
162
+ remove_columns=train_ds.column_names,
163
+ )
164
+ eval_ds = eval_ds.map(
165
+ lambda batch: prepare_dataset(batch, processor),
166
+ remove_columns=eval_ds.column_names,
167
+ )
168
+
169
+ # Data collator
170
+ data_collator = DataCollatorSpeechSeq2SeqWithPadding(processor=processor)
171
+
172
+ # Metrics
173
+ wer_metric = evaluate.load("wer")
174
+ cer_metric = evaluate.load("cer")
175
+
176
+ # Training arguments
177
+ training_args = Seq2SeqTrainingArguments(
178
+ output_dir=output,
179
+ per_device_train_batch_size=batch_size,
180
+ per_device_eval_batch_size=batch_size,
181
+ gradient_accumulation_steps=grad_accum,
182
+ learning_rate=lr,
183
+ warmup_steps=warmup_steps,
184
+ max_steps=max_steps,
185
+ num_train_epochs=epochs,
186
+ fp16=fp16 and torch.cuda.is_available(),
187
+ eval_strategy="steps",
188
+ eval_steps=eval_steps,
189
+ save_strategy="steps",
190
+ save_steps=save_steps,
191
+ logging_steps=25,
192
+ load_best_model_at_end=True,
193
+ metric_for_best_model="wer",
194
+ greater_is_better=False,
195
+ predict_with_generate=True,
196
+ generation_max_length=225,
197
+ report_to="wandb" if use_wandb else "none",
198
+ push_to_hub=push_to_hub,
199
+ hub_model_id=hub_model_id if push_to_hub else None,
200
+ save_total_limit=3,
201
+ dataloader_num_workers=4,
202
+ remove_unused_columns=False,
203
+ )
204
+
205
+ # Trainer
206
+ trainer = Seq2SeqTrainer(
207
+ model=model,
208
+ args=training_args,
209
+ train_dataset=train_ds,
210
+ eval_dataset=eval_ds,
211
+ data_collator=data_collator,
212
+ processing_class=processor.feature_extractor,
213
+ compute_metrics=lambda pred: compute_metrics(pred, processor, wer_metric, cer_metric),
214
+ )
215
+
216
+ # Train
217
+ click.echo(f"\nTraining for {epochs} epochs...")
218
+ trainer.train()
219
+
220
+ # Save best model
221
+ click.echo(f"\nSaving model to {output}")
222
+ trainer.save_model(output)
223
+ processor.save_pretrained(output)
224
+
225
+ # Final evaluation
226
+ click.echo("\nFinal evaluation...")
227
+ metrics = trainer.evaluate()
228
+ click.echo(f" WER: {metrics['eval_wer']:.2f}%")
229
+ click.echo(f" CER: {metrics['eval_cer']:.2f}%")
230
+
231
+ click.echo(f"\nModel saved to: {output}")
232
+
233
+ if push_to_hub:
234
+ click.echo(f"Pushing to HuggingFace Hub: {hub_model_id}")
235
+ trainer.push_to_hub()
236
+
237
+
238
+ if __name__ == '__main__':
239
+ train()
src/transcribe.py ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # requires-python = ">=3.10"
3
+ # dependencies = [
4
+ # "torch>=2.0.0",
5
+ # "torchaudio>=2.0.0",
6
+ # "transformers>=4.36.0",
7
+ # "click>=8.0.0",
8
+ # ]
9
+ # ///
10
+ """
11
+ Transcription script for ASR-1.
12
+
13
+ Usage:
14
+ uv run src/transcribe.py --model models/asr-1 --audio audio.wav
15
+ uv run src/transcribe.py --model models/asr-1 --audio-dir ./recordings/
16
+ """
17
+
18
+ import sys
19
+ from pathlib import Path
20
+
21
+ from dotenv import load_dotenv
22
+ load_dotenv()
23
+
24
+ import torch
25
+ import torchaudio
26
+ import click
27
+ from transformers import WhisperProcessor, WhisperForConditionalGeneration
28
+
29
+
30
+ def load_audio(audio_path: str, target_sr: int = 16000):
31
+ """Load and resample audio to target sample rate."""
32
+ waveform, sample_rate = torchaudio.load(audio_path)
33
+ if sample_rate != target_sr:
34
+ waveform = torchaudio.transforms.Resample(sample_rate, target_sr)(waveform)
35
+ # Convert to mono if stereo
36
+ if waveform.shape[0] > 1:
37
+ waveform = waveform.mean(dim=0, keepdim=True)
38
+ return waveform.squeeze().numpy()
39
+
40
+
41
+ @click.command()
42
+ @click.option('--model', '-m', default='models/asr-1', help='Model path or HuggingFace model ID')
43
+ @click.option('--audio', '-a', default=None, help='Path to audio file')
44
+ @click.option('--audio-dir', '-d', default=None, help='Directory of audio files')
45
+ @click.option('--output', '-o', default=None, help='Output file for transcriptions')
46
+ def transcribe(model, audio, audio_dir, output):
47
+ """Transcribe Vietnamese audio to text."""
48
+
49
+ device = "cuda" if torch.cuda.is_available() else "cpu"
50
+
51
+ # Load model
52
+ click.echo(f"Loading model: {model}")
53
+ processor = WhisperProcessor.from_pretrained(model)
54
+ asr_model = WhisperForConditionalGeneration.from_pretrained(model).to(device)
55
+ asr_model.eval()
56
+
57
+ # Collect audio files
58
+ audio_files = []
59
+ if audio:
60
+ audio_files.append(Path(audio))
61
+ elif audio_dir:
62
+ audio_dir = Path(audio_dir)
63
+ audio_files = sorted(
64
+ p for p in audio_dir.iterdir()
65
+ if p.suffix.lower() in ('.wav', '.mp3', '.flac', '.ogg', '.m4a')
66
+ )
67
+ else:
68
+ click.echo("Error: Provide --audio or --audio-dir")
69
+ sys.exit(1)
70
+
71
+ click.echo(f"Transcribing {len(audio_files)} file(s)...\n")
72
+
73
+ results = []
74
+ for audio_path in audio_files:
75
+ # Load audio
76
+ waveform = load_audio(str(audio_path))
77
+
78
+ # Extract features
79
+ input_features = processor.feature_extractor(
80
+ waveform,
81
+ sampling_rate=16000,
82
+ return_tensors="pt",
83
+ ).input_features.to(device)
84
+
85
+ # Generate
86
+ with torch.no_grad():
87
+ predicted_ids = asr_model.generate(
88
+ input_features,
89
+ language="vi",
90
+ task="transcribe",
91
+ )
92
+
93
+ # Decode
94
+ transcription = processor.tokenizer.batch_decode(
95
+ predicted_ids, skip_special_tokens=True
96
+ )[0]
97
+
98
+ click.echo(f"{audio_path.name}: {transcription}")
99
+ results.append((str(audio_path), transcription))
100
+
101
+ # Save output
102
+ if output:
103
+ with open(output, "w", encoding="utf-8") as f:
104
+ for path, text in results:
105
+ f.write(f"{path}\t{text}\n")
106
+ click.echo(f"\nTranscriptions saved to: {output}")
107
+
108
+
109
+ if __name__ == '__main__':
110
+ transcribe()