Upload folder using huggingface_hub

README.md

@@ -0,0 +1,91 @@
+---
+license: apache-2.0
+base_model: Wan-AI/Wan2.1-T2V-14B
+tags:
+- lip-sync
+- talking-head
+- video-to-video
+- video-generation
+- diffusion
+- autoregressive
+- distillation
+- wan2.1
+---
+
+# Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
+
+[Paper (arXiv:2606.11180)](https://arxiv.org/abs/2606.11180) ·
+[Project page](https://cvlab-kaist.github.io/LipForcing/) ·
+[Code (GitHub)](https://github.com/cvlab-kaist/LipForcing)
+
+<p align="center"><img src="https://cvlab-kaist.github.io/LipForcing/assets/main_arch.png" width="95%" alt="Lip Forcing architecture"></p>
+
+**Lip Forcing** is, to our knowledge, the first autoregressive diffusion method for
+video-to-video (V2V) lip synchronization. It distills a high-fidelity bidirectional 14B
+audio-conditioned teacher (*OmniAvatar-LS*, our lip-sync finetune of OmniAvatar) into causal
+few-step students that generate each chunk in just **two denoising steps with no
+inference-time CFG** - enabling real-time, streaming lip-sync on a reference video with
+sub-millisecond time-to-first-frame.
+
+## Files in this repository
+
+| File | What | For |
+|------|------|-----|
+| `lipforcing_14b.pth` | 14B student - merged, self-contained (base Wan + OmniAvatar-LS adapter + Self-Forcing student baked into one file) | inference |
+| `teacher/omniavatar_ls_14b.pt` | OmniAvatar-LS 14B V2V adapter (frozen distillation teacher + student init) | training |
+| *1.3B student* | *coming soon (same repo)* | |
+
+## Quickstart
+
+Install the code from [GitHub](https://github.com/cvlab-kaist/LipForcing):
+
+```bash
+git clone https://github.com/cvlab-kaist/LipForcing.git && cd LipForcing
+conda create -y -n lipforcing python=3.12 && conda activate lipforcing
+pip install -e .
+```
+
+Download this checkpoint plus the small external encoders/decoders (commands for every
+component are in the repo README's *Weights* section), then:
+
+```bash
+python scripts/inference/inference_streaming.py \
+  --ckpt_path weights/lipforcing_14b.pth \
+  --vae_path weights/Wan2.1-T2V-14B/Wan2.1_VAE.pth \
+  --wav2vec_path weights/wav2vec2-base-960h \
+  --mask_path weights/mask.png \
+  --taehv_ckpt weights/taew2_1.pth \
+  --text_encoder_path weights/Wan2.1-T2V-14B/models_t5_umt5-xxl-enc-bf16.pth \
+  --video_path ref.mp4 --audio_path speech.wav --output_path out.mp4
+```
+
+Streaming inference encodes, denoises, decodes, and composites each AR chunk on the fly -
+first frames arrive before the clip finishes, GPU memory stays constant for any clip length
+(**~37 GB** peak at 14B with precomputed text embeddings; ~50 GB with runtime T5 encoding),
+and face detection + 512×512 alignment + paste-back run automatically, so any talking-head
+video works as input.
+
+Training (two stages: Diffusion-Forcing init, then Self-Forcing DMD distillation with the
+SyncNet reward) is documented in the repo's
+[README](https://github.com/cvlab-kaist/LipForcing#training) and
+[DATA.md](https://github.com/cvlab-kaist/LipForcing/blob/master/DATA.md).
+
+## Acknowledgements
+
+Builds on [NVIDIA FastGen](https://github.com/NVlabs/FastGen),
+[Self Forcing](https://github.com/guandeh17/Self-Forcing),
+[Reward-Forcing](https://github.com/JaydenLyh/Reward-Forcing),
+[OmniAvatar](https://github.com/Omni-Avatar/OmniAvatar), and
+[Wan2.1](https://github.com/Wan-Video/Wan2.1). Third-party licenses ship in the
+[code repository](https://github.com/cvlab-kaist/LipForcing/tree/master/licenses).
+
+## Citation
+
+```bibtex
+@article{cho2026lipforcing,
+  title   = {Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization},
+  author  = {Cho, Paul Hyunbin and Jang, Jinhyuk and Lee, SeokYoung and Lee, Joungbin and Jin, Siyoon and Shin, Heeseong and Yi, Jung and Park, Yunjin and Park, Chulmin and Kim, Seungryong},
+  journal = {arXiv preprint arXiv:2606.11180},
+  year    = {2026},
+}
+```

lipforcing_14b.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea9f111f374a208a80b6604e2c698639f03ad666bb7cda72c727a93cd43e4307
+size 28588528943

teacher/omniavatar_ls_14b.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7e2d22592fa20d5622e82fdd2c3cc9fab9ca5c2fb080306ccd877d9e6b9fa45
+size 1238964875