Instructions to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
# Run inference directly in the terminal:
llama cli -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
# Run inference directly in the terminal:
llama cli -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Use Docker

docker model run hf.co/ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

LM Studio
Jan

vLLM

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Ollama
How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with Ollama:
```
ollama run hf.co/ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
```
Unsloth Desktop

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with Docker Model Runner:
```
docker model run hf.co/ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K
```

Lemonade

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Run and chat with the model

lemonade run user.Qwen3.8-4B-Distill-GGUF-Q2_K-Q2_K

List all available models

lemonade list

Hermes Agent

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ZichenAI/Qwen3.8-4B-Distill-GGUF-Q2_K:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-4B — GGUF (Q2_K)

Q2_K imatrix quantization of empero-ai/Qwen3.8-4B — a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture — for llama.cpp, Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes.

This repo hosts the Q2_K quant only. For higher-quality quantizations (Q4_K_M and up), see empero-ai/Qwen3.8-4B-Distill-GGUF. Capability details and full benchmark results live on the main model card.

Qwen3.5-class models are hybrids: three Gated DeltaNet layers for every full-attention layer. A recent llama.cpp build with Qwen3.5 / Gated DeltaNet support is required — older builds will fail to load the architecture.

Q2_K is an aggressive 2-bit quantization. Expect a noticeable quality drop compared to Q4_K_M and above, especially on long reasoning chains. Use this file when memory is the hard constraint.

Files

File	Quant	Size	Notes
`Qwen3.8-4B-Q2_K.gguf`	Q2_K (imatrix)	2.044 GB	Importance-matrix weighted Q2_K, fits in ~2.5 GB VRAM at modest context.

Size is exact decimal GB from the uploaded file (1 GB = 1,000,000,000 bytes).

Quantization details

Method: Q2_K with importance matrix (imatrix) weighting
Calibration: imatrix.dat computed from a diverse English calibration dataset (~4,900 chunks)
Source weights: Qwen3.8-4B-BF16.gguf (full-precision reference)

Usage

llama.cpp

llama-cli -m Qwen3.8-4B-Q2_K.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Use the built-in chat template (-cnv). The model is a reasoning model: every answer opens with a <think> block, so allow a generous -n and strip the <think>...</think> span for end users.

Ollama / LM Studio / Jan / KoboldCpp

Download the GGUF and load it directly; the chat template is embedded in the file. Recommended sampling: temperature=0.6, top_p=0.95, top_k=20.

Model architecture (from GGUF metadata)

Property	Value
Architecture	`qwen35` (hybrid: Gated DeltaNet + full attention every 4 layers)
Parameters	~4B
Layers	33
Embedding size	2560
Attention heads (KV)	16 (4)
Context length	262,144 tokens

Provenance & licensing

Quantization of empero-ai/Qwen3.8-4B, a distillation of Qwen3.8 2.4T A95B into Qwen/Qwen3.5-4B.

This repo's weights are distributed under a custom license (see LICENSE): free for personal / non-commercial use; commercial use requires a paid license — contact novaweb6868@outlook.com. The underlying source weights remain available under their original Apache-2.0 terms from the upstream repositories above.