marcoyang/spear-xlarge-speech-audio-v2

SPEAR XLarge v2 (speech + general audio)

SPEAR XLarge v2 is the flagship open-source SPEAR encoder for unified speech and general-audio representation learning. This is the ICML 2026 accepted version of SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations.

This model is the XLarge v2 release, aligned with the model used in the ICML 2026 paper. Compared with the earlier XLarge v1, v2 is enhanced for complex acoustic scenes through token mixing, improving robustness for overlapped speech, noisy audio, and real-world sound mixtures while keeping SPEAR's unified speech-and-audio design.

SPEAR XLarge v2 uses a Zipformer backbone with about 600M parameters, consisting of 13 Zipformer stacks. It produces 1280-dimensional frame-level representations at approximately 50 Hz from 16 kHz waveforms.

🚀 Why Use This Model?

SPEAR XLarge v2 is designed as a general-purpose acoustic foundation model: one encoder for speech understanding, audio-event understanding, and downstream fine-tuning.

• Unified speech + audio representations: trained to model both spoken language and general acoustic events in a single SSL encoder.
• ICML 2026 model version: this is the v2 XLarge model corresponding to the accepted ICML 2026 work.
• Stronger complex-scene robustness: v2 adds token mixing to improve learning under overlapped/noisy speech and mixed audio scenes.
• State-of-the-art representation quality: SPEAR achieves SOTA results on both SUPERB and HEAR, covering speech and general-audio representation evaluation.
• Strong fine-tuning performance: the model performs very well when fine-tuned for automatic speech recognition (ASR) and audio tagging (AT).

🎧 Pre-training Data

The model was pre-trained on 197k hours of mixed English speech and general-audio data: 184k hours of speech and 13k hours of general audio.

Speech data:

Dataset	Duration
Libriheavy	~50k hours
Gigaspeech	~10k hours
VoxPopuli (en)	~24k hours
yodas-granary	~100k hours

General-audio data:

Dataset	Duration
AudioSet	~5k hours
Freesound	~2.8k hours
Music4all	~1k hours
VGGSound	~0.5k hours
MTG-Jamendo	~3.8k hours

🏆 Downstream Performance

SPEAR XLarge v2 delivers state-of-the-art results on both major representation-learning benchmarks: SUPERB for speech representation evaluation and HEAR for general-audio representation evaluation. This makes it a strong default encoder when a project needs one model that works well across speech, environmental audio, music, and mixed acoustic scenes.

SPEAR XLarge v2 is pre-trained only with unlabeled audio. For supervised tasks such as ASR or AT, fine-tune it with labeled downstream data.

When fine-tuned on LibriSpeech for ASR, the XLarge model achieves:

Fine-tuning data	test-clean WER	test-other WER
LS960	1.6	2.9

When fine-tuned on AudioSet for audio tagging, the XLarge model achieves:

Fine-tuning data	mAP
AudioSet Balanced	39.4
AudioSet Full	50.0

⚡ Usage

The model expects mono or batched 16 kHz waveform input. The example below extracts the top-layer representation and all intermediate hidden states.

from transformers import AutoModel
import torch

model = AutoModel.from_pretrained(
    "marcoyang/spear-xlarge-speech-audio-v2",
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()

# Dummy 10-second waveform at 16 kHz.
audio = torch.randn(1, 160000, device=device)
audio_len = torch.tensor([audio.shape[1]], device=device)

with torch.no_grad():
    outputs = model(audio, audio_len)

encoder_out = outputs["encoder_out"]       # (batch, frames, 1280)
encoder_out_lens = outputs["encoder_out_lens"]
hidden_states = outputs["hidden_states"]   # list of intermediate states

print(encoder_out.shape)
print(encoder_out_lens)
print(len(hidden_states))                  # 13 layers
print(hidden_states[-1].shape)

Important: SPEAR models are pre-trained on 16 kHz audio. Resample your input to 16 kHz before passing it to the model.

📄 Paper

SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

Authors: Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland

Abstract: Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.

📜 License

Apache-2.0