Instructions to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

Use Docker

docker model run hf.co/EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

SGLang

How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with Ollama:
```
ollama run hf.co/EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with Docker Model Runner:
```
docker model run hf.co/EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
```

Lemonade

How to use EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Small-3.1-24B-Instruct-2503-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

-----------------------------------------------
- Model Details and Specifications: -
-----------------------------------------------

Mistral Small 3.1 24B Instruct 2503 GGUF (Ollama & Llama.cpp)

This release contains:
Llama.cpp and Ollama compatible GGUF converted and Quantized model files (Compatible with both Ollama, and Llama.cpp)
(More information and updates just landed! More on the way (^.^) - Thank you for taking the time to view this release!)

Quantized GGUF version of:

mistralai/Mistral-Small-3.1-24B-Instruct-2503
(by MistralAI)

Original Model Link:

mistralai/Mistral-Small-3.1-24B-Instruct-2503

---------------------------------------------------
- Conversion and GGUF Quantization: -
---------------------------------------------------

Software used to convert Safetensors to GGUF:

llama.cpp, release verson: (b7540)

Software used to create Quantized GGUF Files:

llama.cpp, release verson: (b7540)

Specific GitHub Commit Point:

b7540

Converted to GGUF and Quantized by:

EnlistedGhost

-------------------------------
---- Updates & News ----
-------------------------------

Model Updates (as of: January 25th, 2026)

Uploaded: Remaining GGUF Converted and Quantized model files
Updated: ModelCard
(this page)
Planned Updates: Yes, more updates coming! Thank you whoever you are for your patience (^.^)

--------------------------------------
---- How to run this Model ----
--------------------------------------

Compatible Software (Required to use this Model)
You can run this model by using either Ollama (or) Llama.cpp
(Below are instruction on running these GGUF files with Ollama)

How to run this Model using Ollama
You can run this model by using the "ollama run" command.
Simply copy & paste one of the commands from the list below into
your console, terminal or power-shell window.

Quant Type	File Size	Command
QX_X	0.00 GB	(Currently Uploading Files, Check again very soon!)

Vision Projector (Files)
mmproj (Vision Projector) Files

Quant Type	File Size	Download Link
Q8_0	465 MB
F16	870 MB
F32	1.74 GB

---------------------------
---- Original Info ----
---------------------------

(Crossposted from the link in the above section: "Model Details"):

Building upon Mistral Small 3 (2501), Mistral Small 3.1 (2503) adds state-of-the-art vision understanding and enhances long context capabilities up to 128k tokens without compromising text performance. With 24 billion parameters, this model achieves top-tier capabilities in both text and vision tasks.
This model is an instruction-finetuned version of: Mistral-Small-3.1-24B-Base-2503.

Mistral Small 3.1 can be deployed locally and is exceptionally "knowledge-dense," fitting within a single RTX 4090 or a 32GB RAM MacBook once quantized.

It is ideal for:

Fast-response conversational agents.
Low-latency function calling.
Subject matter experts via fine-tuning.
Local inference for hobbyists and organizations handling sensitive data.
Programming and math reasoning.
Long document understanding.
Visual understanding.

For enterprises requiring specialized capabilities (increased context, specific modalities, domain-specific knowledge, etc.), we will release commercial models beyond what Mistral AI contributes to the community.

Learn more about Mistral Small 3.1 in our blog post.

Key Features

Vision: Vision capabilities enable the model to analyze images and provide insights based on visual content in addition to text.
Multilingual: Supports dozens of languages, including English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, Farsi.
Agent-Centric: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Advanced Reasoning: State-of-the-art conversational and reasoning capabilities.
Apache 2.0 License: Open license allowing usage and modification for both commercial and non-commercial purposes.
Context Window: A 128k context window.
System Prompt: Maintains strong adherence and support for system prompts.
Tokenizer: Utilizes a Tekken tokenizer with a 131k vocabulary size.

Benchmark Results

When available, we report numbers previously published by other model providers, otherwise we re-evaluate them using our own evaluation harness.

Pretrain Evals

Model	MMLU (5-shot)	MMLU Pro (5-shot CoT)	TriviaQA	GPQA Main (5-shot CoT)	MMMU
Small 3.1 24B Base	81.01%	56.03%	80.50%	37.50%	59.27%
Gemma 3 27B PT	78.60%	52.20%	81.30%	24.30%	56.10%

Instruction Evals

Text

Model	MMLU	MMLU Pro (5-shot CoT)	MATH	GPQA Main (5-shot CoT)	GPQA Diamond (5-shot CoT )	MBPP	HumanEval	SimpleQA (TotalAcc)
Small 3.1 24B Instruct	80.62%	66.76%	69.30%	44.42%	45.96%	74.71%	88.41%	10.43%
Gemma 3 27B IT	76.90%	67.50%	89.00%	36.83%	42.40%	74.40%	87.80%	10.00%
GPT4o Mini	82.00%	61.70%	70.20%	40.20%	39.39%	84.82%	87.20%	9.50%
Claude 3.5 Haiku	77.60%	65.00%	69.20%	37.05%	41.60%	85.60%	88.10%	8.02%
Cohere Aya-Vision 32B	72.14%	47.16%	41.98%	34.38%	33.84%	70.43%	62.20%	7.65%

Vision

Model	MMMU	MMMU PRO	Mathvista	ChartQA	DocVQA	AI2D	MM MT Bench
Small 3.1 24B Instruct	64.00%	49.25%	68.91%	86.24%	94.08%	93.72%	7.3
Gemma 3 27B IT	64.90%	48.38%	67.60%	76.00%	86.60%	84.50%	7
GPT4o Mini	59.40%	37.60%	56.70%	76.80%	86.70%	88.10%	6.6
Claude 3.5 Haiku	60.50%	45.03%	61.60%	87.20%	90.00%	92.10%	6.5
Cohere Aya-Vision 32B	48.20%	31.50%	50.10%	63.04%	72.40%	82.57%	4.1

Multilingual Evals

Model	Average	European	East Asian	Middle Eastern
Small 3.1 24B Instruct	71.18%	75.30%	69.17%	69.08%
Gemma 3 27B IT	70.19%	74.14%	65.65%	70.76%
GPT4o Mini	70.36%	74.21%	65.96%	70.90%
Claude 3.5 Haiku	70.16%	73.45%	67.05%	70.00%
Cohere Aya-Vision 32B	62.15%	64.70%	57.61%	64.12%

Long Context Evals

Model	LongBench v2	RULER 32K	RULER 128K
Small 3.1 24B Instruct	37.18%	93.96%	81.20%
Gemma 3 27B IT	34.59%	91.10%	66.00%
GPT4o Mini	29.30%	90.20%	65.8%
Claude 3.5 Haiku	35.19%	92.60%	91.90%

Downloads last month: 855

GGUF

Model size

24B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF

Base model

mistralai/Mistral-Small-3.1-24B-Base-2503

Finetuned

mistralai/Mistral-Small-3.1-24B-Instruct-2503

Quantized

(63)

this model

EnlistedGhost
/

Mistral-Small-3.1-24B-Instruct-2503-GGUF

-----------------------------------------------
- Model Details and Specifications: -
-----------------------------------------------

Mistral Small 3.1 24B Instruct 2503 GGUF (Ollama & Llama.cpp)

---------------------------------------------------
- Conversion and GGUF Quantization: -
---------------------------------------------------

-------------------------------
---- Updates & News ----
-------------------------------

--------------------------------------
---- How to run this Model ----
--------------------------------------

---------------------------
---- Original Info ----
---------------------------

Key Features

Benchmark Results

Pretrain Evals

Instruction Evals

Text

Vision

Multilingual Evals

Long Context Evals

Model tree for EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF

Dataset used to train EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF

----------------------------------------------- - Model Details and Specifications: ------------------------------------------------

Mistral Small 3.1 24B Instruct 2503 GGUF (Ollama & Llama.cpp)

--------------------------------------------------- - Conversion and GGUF Quantization: ----------------------------------------------------

------------------------------- ---- Updates & News ---- -------------------------------

-------------------------------------- ---- How to run this Model ---- --------------------------------------

--------------------------- ---- Original Info ---- ---------------------------

Key Features

Benchmark Results

Pretrain Evals

Instruction Evals

Text

Vision

Multilingual Evals

Long Context Evals

Model tree for EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF

Dataset used to train EnlistedGhost/Mistral-Small-3.1-24B-Instruct-2503-GGUF

-----------------------------------------------
- Model Details and Specifications: -
-----------------------------------------------

---------------------------------------------------
- Conversion and GGUF Quantization: -
---------------------------------------------------

-------------------------------
---- Updates & News ----
-------------------------------

--------------------------------------
---- How to run this Model ----
--------------------------------------

---------------------------
---- Original Info ----
---------------------------