Update README.md

README.md

@@ -1,3 +1,36 @@
 ---
 license: apache-2.0
 ---
+
+# PVIT model
+
+This is the model weights of paper: [Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models](https://arxiv.org/abs/2308.13437).
+
+## Model description
+
+Position-enhanced Visual Instruction Tuning (PVIT) extends the MLLM by incorporating an additional region-level vision encoder to facilitate support for region-based inputs. Specifically, we adopt the vision encoder from RegionCLIP and utilize it to extract region-level features by taking images and regions as inputs. As an additional source of information, the incorporation of region-level features in this way has a minimal impact on the original MLLM. Furthermore, since the features provided by RegionCLIP are themselves already aligned to the language at a fine-grained level, the overhead of aligning it to the MLLM will be relatively small. Following [LLaVA](https://github.com/haotian-liu/LLaVA), we design a two-stage training strategy for PVIT that first pre-training a linear projection to align the region features to the LLM word embedding, followed by end-to-end fine-tuning to follow complex fine-grained instructions.
+
+For more details, please refer to our [paper](https://arxiv.org/abs/2308.13437) and [github repo](https://github.com/THUNLP-MT/PVIT).
+
+## How to use
+
+Users have to apply it on top of the original LLaMA weights to get actual PVIT weights. See [here](https://github.com/THUNLP-MT/PVIT#pvit-weights) for instructions.
+
+## Intended use
+
+Primary intended uses: The primary use of PVIT is research on large multimodal models and chatbots.
+
+Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
+
+## BibTeX entry and citation info
+
+```bibtex
+@misc{chen2023positionenhanced,
+      title={Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models}, 
+      author={Chi Chen and Ruoyu Qin and Fuwen Luo and Xiaoyue Mi and Peng Li and Maosong Sun and Yang Liu},
+      year={2023},
+      eprint={2308.13437},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```