Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

WaveCut commited on Sep 12

Commit

71490a1

verified ·

1 Parent(s): 6d79051

Keep only baseline and main fast audio demos

Browse files

Files changed (16) hide show

.gitattributes +1 -0
README.md +1 -1
SHA256SUMS +2 -14
examples/{runtime-v10/int8-ordered-cfg1.wav → fast-int8.wav} +0 -0
examples/runtime-v10/int8-ordered-cfg1.5.wav +0 -3
examples/runtime-v10/int8-subwarp-cfg1.5.wav +0 -3
examples/runtime-v10/int8-subwarp-cfg1.wav +0 -3
examples/runtime-v3/fast-rms.wav +0 -3
examples/runtime-v4/fast-rms.wav +0 -3
examples/runtime-v5/fast-rms.wav +0 -3
examples/runtime-v6/fast-rms.wav +0 -3
examples/runtime-v7/fast-rms-cfg1.5.wav +0 -3
examples/runtime-v8/int8-kv-cfg1.5.wav +0 -3
examples/runtime-v8/int8-kv-cfg1.wav +0 -3
examples/runtime-v9/int8-kv-cfg1.5.wav +0 -3
examples/runtime-v9/int8-kv-cfg1.wav +0 -3

.gitattributes CHANGED Viewed

@@ -64,3 +64,4 @@ examples/runtime-v10/int8-subwarp-cfg1.5.wav filter=lfs diff=lfs merge=lfs -text
 examples/runtime-v10/int8-subwarp-cfg1.wav filter=lfs diff=lfs merge=lfs -text
 src/runtime/kernels/int8-ordered/torch210-cxx11-cu128-x86_64-linux/_int8_ordered_attention_cuda_29ba9ae.abi3.so filter=lfs diff=lfs merge=lfs -text
 src/runtime/kernels/int8-subwarp/torch210-cxx11-cu128-x86_64-linux/_int8_subwarp_attention_cuda_92f98d4.abi3.so filter=lfs diff=lfs merge=lfs -text

 examples/runtime-v10/int8-subwarp-cfg1.wav filter=lfs diff=lfs merge=lfs -text
 src/runtime/kernels/int8-ordered/torch210-cxx11-cu128-x86_64-linux/_int8_ordered_attention_cuda_29ba9ae.abi3.so filter=lfs diff=lfs merge=lfs -text
 src/runtime/kernels/int8-subwarp/torch210-cxx11-cu128-x86_64-linux/_int8_subwarp_attention_cuda_92f98d4.abi3.so filter=lfs diff=lfs merge=lfs -text
+examples/fast-int8.wav filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -66,7 +66,7 @@ Rental rates: $0.72/h for PRO 4500 and $0.28/h for 4000 Ada, observed September
 ## Audio and limitations
-Download [WAV examples](examples/runtime-v10). Experimental RMS fusion, INT8 KV and attention reductions can change the arrangement at the same seed. Spectral similarity is a diagnostic, not proof of perceptual equivalence. Russian was tested on this prompt; upstream advertises Chinese and English.
 Code and kernels are under `src/`; metrics under `evaluation/`; prompts and audio under `examples/`. [SHA256SUMS](SHA256SUMS) and [runtime-lock.json](src/runtime-lock.json) pin the shipped artifacts.

 ## Audio and limitations
+WAV demos: [original BF16](examples/original-ru.wav), [W4A4 reference](examples/orbitquant-ru.wav), [fast INT8 KV](examples/fast-int8.wav). Experimental RMS fusion, INT8 KV and attention reductions can change the arrangement at the same seed. Spectral similarity is a diagnostic, not proof of perceptual equivalence. Russian was tested on this prompt; upstream advertises Chinese and English.
 Code and kernels are under `src/`; metrics under `evaluation/`; prompts and audio under `examples/`. [SHA256SUMS](SHA256SUMS) and [runtime-lock.json](src/runtime-lock.json) pin the shipped artifacts.

SHA256SUMS CHANGED Viewed

@@ -1,5 +1,5 @@
 060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0  LICENSE
-0104517a9b1ce11c421650ae34afc5146dda3b234f1b5b958ce7a0a56bef2c35  README.md
 14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3  THIRD_PARTY_NOTICES.md
 e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56  config.json
 5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d  evaluation/ar-fused-fixed.json
@@ -631,22 +631,10 @@ a8a166b2e235fbc927eb3dc017e47e04255a6015461376743c4843b79a629445  evaluation/run
 77c897a1fdf3e8a94bbfb828d5e5a766d0f036521da10a0c06fbe45021bedc89  evaluation/runtime-v9/v9-full/output.log
 acac9c9098eaffec853d3fe5b10cb23acfab49d239e5dc153042c741c04d8682  evaluation/runtime-v9/v9-full/status.json
 00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df  evaluation/vae-chunk-benchmark.json
 6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a  examples/orbitquant-ru.wav
 4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6  examples/original-ru.wav
 a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0  examples/prompts/default-ru.json
-1d4e9cf0eb5a6b71fd27faf82da465c274ff162573f7ba24e895e1b4aa79293c  examples/runtime-v10/int8-ordered-cfg1.5.wav
-7e66bfe39c6a34e6aeae47cd7b2e1d0a58995fecbe1302579bda53cc8a06ffcf  examples/runtime-v10/int8-ordered-cfg1.wav
-f4a59733814697406704a6dad96418abee0ac5d386a4fe1352b1b72ae0d06c9a  examples/runtime-v10/int8-subwarp-cfg1.5.wav
-03c1d1c79b6611c34fafbc2ac0d7f460a2b9bfbc847abf96adf5c0639a37ec83  examples/runtime-v10/int8-subwarp-cfg1.wav
-30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e  examples/runtime-v3/fast-rms.wav
-d314b4b9876095782a7b0aabf7b0de6c96b4d2f203809e14c2318c166c569dec  examples/runtime-v4/fast-rms.wav
-e0693fca03f50e48d597a3e22607df6b383a28de44be5d59e243807e97b98e44  examples/runtime-v5/fast-rms.wav
-78b6257e68ef5f003ebce4cd582b610d94cbbb1c4322384bdb2b60a7e47157d0  examples/runtime-v6/fast-rms.wav
-1cfad8b2c95630da7630dfb87723bea1175f4b6fbcf2916e993bb606bce4aabe  examples/runtime-v7/fast-rms-cfg1.5.wav
-8c88e3aad65b435efb880c24962c8d455b76b2c486d45cff97dea1e2c3444844  examples/runtime-v8/int8-kv-cfg1.5.wav
-7ec97d940b7b5d3065a0fc09ce5ae73d37e09b12749e444afd5e5c142b9fd7bb  examples/runtime-v8/int8-kv-cfg1.wav
-212b2abc44b8ce5f51a1cbb2b75f56fea047f7706675044ee2b2ee0ae2f584df  examples/runtime-v9/int8-kv-cfg1.5.wav
-2831fb6e362b393ae13a8b2aa1346da39835f8c68a45c0afc2b3ef665870517e  examples/runtime-v9/int8-kv-cfg1.wav
 ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170  generation_config.json
 da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc  licenses/SnakeBeta-NVIDIA-MIT.txt
 a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101  licenses/stable-audio-tools-MIT.txt

 060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0  LICENSE
+d8b6905c3c2fb2e74c364e1844577fba3af08370d7d01663fb950f75e06e0dae  README.md
 14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3  THIRD_PARTY_NOTICES.md
 e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56  config.json
 5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d  evaluation/ar-fused-fixed.json
 77c897a1fdf3e8a94bbfb828d5e5a766d0f036521da10a0c06fbe45021bedc89  evaluation/runtime-v9/v9-full/output.log
 acac9c9098eaffec853d3fe5b10cb23acfab49d239e5dc153042c741c04d8682  evaluation/runtime-v9/v9-full/status.json
 00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df  evaluation/vae-chunk-benchmark.json
+7e66bfe39c6a34e6aeae47cd7b2e1d0a58995fecbe1302579bda53cc8a06ffcf  examples/fast-int8.wav
 6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a  examples/orbitquant-ru.wav
 4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6  examples/original-ru.wav
 a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0  examples/prompts/default-ru.json
 ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170  generation_config.json
 da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc  licenses/SnakeBeta-NVIDIA-MIT.txt
 a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101  licenses/stable-audio-tools-MIT.txt

examples/{runtime-v10/int8-ordered-cfg1.wav → fast-int8.wav} RENAMED Viewed

File without changes

examples/runtime-v10/int8-ordered-cfg1.5.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1d4e9cf0eb5a6b71fd27faf82da465c274ff162573f7ba24e895e1b4aa79293c
-size 63328856

examples/runtime-v10/int8-subwarp-cfg1.5.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f4a59733814697406704a6dad96418abee0ac5d386a4fe1352b1b72ae0d06c9a
-size 69503576

examples/runtime-v10/int8-subwarp-cfg1.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:03c1d1c79b6611c34fafbc2ac0d7f460a2b9bfbc847abf96adf5c0639a37ec83
-size 69795416

examples/runtime-v3/fast-rms.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e
-size 70455896

examples/runtime-v4/fast-rms.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d314b4b9876095782a7b0aabf7b0de6c96b4d2f203809e14c2318c166c569dec
-size 70455896

examples/runtime-v5/fast-rms.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e0693fca03f50e48d597a3e22607df6b383a28de44be5d59e243807e97b98e44
-size 70455896

examples/runtime-v6/fast-rms.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:78b6257e68ef5f003ebce4cd582b610d94cbbb1c4322384bdb2b60a7e47157d0
-size 70455896

examples/runtime-v7/fast-rms-cfg1.5.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1cfad8b2c95630da7630dfb87723bea1175f4b6fbcf2916e993bb606bce4aabe
-size 70640216

examples/runtime-v8/int8-kv-cfg1.5.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8c88e3aad65b435efb880c24962c8d455b76b2c486d45cff97dea1e2c3444844
-size 63328856

examples/runtime-v8/int8-kv-cfg1.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7ec97d940b7b5d3065a0fc09ce5ae73d37e09b12749e444afd5e5c142b9fd7bb
-size 63912536

examples/runtime-v9/int8-kv-cfg1.5.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:212b2abc44b8ce5f51a1cbb2b75f56fea047f7706675044ee2b2ee0ae2f584df
-size 63328856

examples/runtime-v9/int8-kv-cfg1.wav DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2831fb6e362b393ae13a8b2aa1346da39835f8c68a45c0afc2b3ef665870517e
-size 63912536