Image-Text-to-Text
Transformers
Safetensors
English
Chinese
llava_onevision2
multimodal
vision-language
video-text-to-text
llava
llava-onevision-2
qwen3
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForImageTextToText model = AutoModelForImageTextToText.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct
Update README.md
Browse files
README.md
CHANGED
|
@@ -167,6 +167,47 @@ backend is usually a better choice.
|
|
| 167 |
- Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`).
|
| 168 |
- Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends.
|
| 169 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
## License
|
| 171 |
|
| 172 |
Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).
|
|
|
|
| 167 |
- Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`).
|
| 168 |
- Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends.
|
| 169 |
|
| 170 |
+
## Citation
|
| 171 |
+
|
| 172 |
+
If you find **LLaVA-OneVision-2** useful in your research, please cite the following paper:
|
| 173 |
+
|
| 174 |
+
```bibtex
|
| 175 |
+
@misc{an2026llavaonevision2nextgenerationperceptualintelligence,
|
| 176 |
+
title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
|
| 177 |
+
author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
|
| 178 |
+
year = {2026},
|
| 179 |
+
eprint = {2605.25979},
|
| 180 |
+
archivePrefix = {arXiv},
|
| 181 |
+
primaryClass = {cs.CV},
|
| 182 |
+
url = {https://arxiv.org/abs/2605.25979}
|
| 183 |
+
}
|
| 184 |
+
```
|
| 185 |
+
|
| 186 |
+
```bibtex
|
| 187 |
+
@misc{an2025llavaonevision15fullyopenframework,
|
| 188 |
+
title = {LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
|
| 189 |
+
author = {Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Didi Zhu and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
|
| 190 |
+
year = {2025},
|
| 191 |
+
eprint = {2509.23661},
|
| 192 |
+
archivePrefix = {arXiv},
|
| 193 |
+
primaryClass = {cs.CV},
|
| 194 |
+
url = {https://arxiv.org/abs/2509.23661}
|
| 195 |
+
}
|
| 196 |
+
```
|
| 197 |
+
|
| 198 |
+
```bibtex
|
| 199 |
+
@misc{tang2026onevisionencodercodecalignedsparsityfoundational,
|
| 200 |
+
title = {OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence},
|
| 201 |
+
author = {Feilong Tang and Xiang An and Yunyao Yan and Yin Xie and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Chunyuan Li and Shikun Feng and Changrui Chen and Huajie Tan and Ming Hu and Manyuan Zhang and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
|
| 202 |
+
year = {2026},
|
| 203 |
+
eprint = {2602.08683},
|
| 204 |
+
archivePrefix = {arXiv},
|
| 205 |
+
primaryClass = {cs.CV},
|
| 206 |
+
url = {https://arxiv.org/abs/2602.08683}
|
| 207 |
+
}
|
| 208 |
+
```
|
| 209 |
+
|
| 210 |
+
|
| 211 |
## License
|
| 212 |
|
| 213 |
Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).
|