Instructions to use mistralai/Mistral-7B-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-v0.1")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-v0.1" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-v0.1

SGLang

How to use mistralai/Mistral-7B-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-v0.1 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-v0.1
```

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

#49

by Jenad1kr - opened Oct 8, 2023

Discussion

Jenad1kr

Oct 8, 2023

I get this error when I try to use gradio to lanuch, both in spaces as well as my local.

raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

any idea what could be wrong?

ArthurZ

Oct 8, 2023

Could you share the full traceback?

Jenad1kr

Oct 8, 2023

@ArthurZ ,
here it is. I created a new space.
===== Application Startup at 2023-10-08 21:21:11 =====

/home/user/app/app.py:3: GradioDeprecationWarning: gr.Interface.load() will be deprecated. Use gr.load() instead.
gr.Interface.load("models/mistralai/Mistral-7B-v0.1").launch()
Fetching model from: https://huggingface.co/mistralai/Mistral-7B-v0.1
Running on local URL: http://0.0.0.0:7860

To create a public link, set share=True in launch().
Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/gradio/queueing.py", line 406, in call_prediction
output = await route_utils.call_process_api(
File "/home/user/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1554, in process_api
result = await self.call_function(
File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1192, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/user/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 659, in wrapper
response = f(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/gradio/external.py", line 415, in query_huggingface_api
errors_json = response.json()
File "/home/user/.local/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

piperod91

Oct 21, 2023

Same issue

dwipper

Oct 30, 2023

•

edited Oct 30, 2023

@ArthurZ Same Issue. I did the quick-create to host a model in a new space. I generated this Gradio code:

import gradio as gr
import os
import requests

HUGGING_FACE_TOKEN=os.environ.get('HUGGING_FACE_TOKEN', None)

gr.load(name="models/meta-llama/Llama-2-7b-chat-hf",hf_token=HUGGING_FACE_TOKEN).launch()

juliaah

Nov 5, 2023

Did you find a solution to this?

dwipper

Nov 5, 2023

Unfortunately no. If you search Spaces for "Llama-2-7b-chat-hf" numerous people have created Spaces the same way, and they have errors.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment