---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-35B-A3B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- vllm
- w8a8
- int8
- llm-compressor
- compressed-tensors
- qwen3_5_moe
- gptq
---

<h1 align: center; style="display: flex; align-items: center; gap: 10px; margin: 0;">
  Qwen3.5-35B-A3B-quantized.w8a8
</h1>

## Model Overview
- **Model Architecture:** Qwen/Qwen3.5-35B-A3B
  - **Input:** Text, Image
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** Int8
  - **Activation quantization:** Int8

This model is a quantized version of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B).
It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

### Model Optimizations

This model was obtained by quantizing the weights and activations of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) to Int8 data type, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Only the weights and activations of the linear operators are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, token embeddings and MTP layers are kept in original precision.

## Deployment

### Use with vLLM

This model can be deployed efficiently using [vLLM](https://github.com/vllm-project/vllm).

1. **Text-Only**: Skip the vision encoder to free up memory for additional KV cache:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --language-model-only
```

2. **Multimodal (Text + Image)**: Serve with full vision support:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3
```

Image input is supported by the upstream architecture but was not evaluated for this release.

3. **Tool Call**: Enable tool use support:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

Tool calling was not evaluated for this release.

4. **Multi-Token Prediction (MTP)**: For speculative decoding:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'
```

Send requests to the server:

```python
from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)
```

The checkpoint retains native MTP weights in BF16. This speculative-decoding configuration was not tested with the quantized checkpoint.

## Creation

This model was created by applying [LLM Compressor](https://github.com/vllm-project/llm-compressor) with w8a8 integer quantization using the GPTQModifier calibrating on the ultrachat dataset, as presented in the code snippet below.

<details>

```python
from pathlib import Path

from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.5-35B-A3B"
DATASET = "ultrachat"
SPLIT = "train_sft[:512]"
save_dir = Path("gptq-ultrachat")
save_dir.parent.mkdir(parents=True, exist_ok=True)

with load_context(Qwen3_5MoeForConditionalGeneration):
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
        MODEL_ID,
        device_map="auto_offload",
    )
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = GPTQModifier(
    targets="Linear",
    scheme="W8A8",
    ignore=[
        "lm_head",
        "re:.*mlp.gate$",
        "re:.*mlp.shared_expert_gate.*",
        "re:.*norm.*",
        "re:.*embed_tokens.*",
        "re:.*visual.*",
        "re:.*conv1d.*",
    ],
)

oneshot(
    model=model,
    processor=processor,
    dataset=DATASET,
    splits=SPLIT,
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
    moe_calibrate_all_experts=True,
)

model.save_pretrained(str(save_dir), save_compressed=True)
processor.save_pretrained(str(save_dir))
print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")
```

</details>

## Evaluation

The BF16 reference and this GPTQ W8A8 checkpoint were served with [vLLM](https://github.com/vllm-project/vllm) in `--language-model-only` mode. Accuracy scores are means over seeds 1234, 2345, and 3456. Recovery is the quantized score divided by the BF16 score.

### Accuracy

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>Qwen/Qwen3.5-35B-A3B (BF16)</th>
      <th>RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8</th>
      <th>Recovery</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="4"><b>Reasoning</b></td>
      <td>GSM8K Platinum, strict match</td>
      <td>95.0648%</td>
      <td>94.8167%</td>
      <td>99.74%</td>
    </tr>
    <tr>
      <td>Math500</td>
      <td>84.8667%</td>
      <td>84.9333%</td>
      <td>100.08%</td>
    </tr>
    <tr>
      <td>GPQA Diamond</td>
      <td>84.1751%</td>
      <td>83.6700%</td>
      <td>99.40%</td>
    </tr>
    <tr>
      <td>MMLU COT Llama</td>
      <td>89.1112%</td>
      <td>88.7552%</td>
      <td>99.60%</td>
    </tr>
    <tr>
      <td rowspan="2"><b>Instruction Following</b></td>
      <td>IFEval prompt-level strict</td>
      <td>90.5730%</td>
      <td>90.3882%</td>
      <td>99.80%</td>
    </tr>
    <tr>
      <td>IFEval instruction-level strict</td>
      <td>93.1655%</td>
      <td>92.8457%</td>
      <td>99.66%</td>
    </tr>
  </tbody>
</table>

### WikiText-2 perplexity

Perplexity was measured on the full 62-document test set from the registered `wikitext` task. Lower is better.

| Metric | Qwen/Qwen3.5-35B-A3B (BF16) | RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 | Change |
|---|---:|---:|---:|
| Word perplexity | 7.46346 | 7.50737 | +0.5883% |
| Byte perplexity | 1.45628 | 1.45788 | +0.1100% |
| Bits per byte | 0.542285 | 0.543868 | +0.2921% |

### Reproduction

The accuracy evaluations used the evaluation frameworks [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [Lighteval](https://github.com/huggingface/lighteval). GSM8K Platinum and IFEval used their registered 0-shot prompts; Math500 and GPQA Diamond used the registered Lighteval tasks. MMLU used `mmlu_cot_llama` with 5-shot examples formatted as multi-turn chat and a maximum of 8,192 generated tokens. All accuracy tasks used temperature 1.0, top-p 0.95, top-k 20, and presence penalty 1.5.

The checkpoint was served with vLLM `0.30.1rc1.dev674+g87954c5b0`, tensor parallel size 2, the Qwen 3 reasoning parser, and text-only mode.

<details>
<summary>Evaluation commands and settings</summary>

#### vLLM server

```bash
vllm serve "$MODEL_ID" \
  --served-model-name "$SERVED_MODEL" \
  --port "$PORT" \
  --tensor-parallel-size 2 \
  --max-model-len 69632 \
  --reasoning-parser qwen3 \
  --language-model-only
```

The same serving and evaluation settings were used for the BF16 reference by replacing the model ID with `Qwen/Qwen3.5-35B-A3B`.


#### IFEval strict (`lm_eval`, 0-shot)

```bash
lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path "./eval/ifeval_seed${SEED}.json" \
  --seed "$SEED" \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"
```


#### MMLU COT Llama (`lm_eval`, 5-shot)

Restart vLLM with a 20,480-token context for this task, then run:

```bash
for SEED in 1234 2345 3456; do
  lm_eval --model local-chat-completions \
    --tasks mmlu_cot_llama \
    --model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
    --num_fewshot 5 \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \
    --seed "$SEED" \
    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"
done
```

#### Math500 and GPQA Diamond (`lighteval`, 0-shot)

```bash
mkdir -p ./eval
for SEED in 1234 2345 3456; do
  CONFIG="./eval/lighteval_seed${SEED}.yaml"
  cat > "$CONFIG" <<EOF
model_parameters:
  provider: hosted_vllm
  model_name: hosted_vllm/${SERVED_MODEL}
  base_url: ${API_BASE}/v1
  api_key: ""
  timeout: 3600
  concurrent_requests: 64
  generation_parameters:
    max_new_tokens: 65536
    presence_penalty: 1.5
    seed: ${SEED}
    temperature: 1.0
    top_k: 20
    top_p: 0.95
EOF

  lighteval endpoint litellm "$CONFIG" 'math_500|0' \
    --output-dir "./eval/math500_seed${SEED}" --save-details
  lighteval endpoint litellm "$CONFIG" 'gpqa:diamond|0' \
    --output-dir "./eval/gpqa_diamond_seed${SEED}" --save-details
done
```

#### WikiText-2 perplexity

The PPL task used lm-eval's `local-completions` model against the vLLM `/v1/completions` endpoint, with the Qwen3.5 tokenizer, `max_length=20480`, `num_concurrent=64`, `max_retries=3`, and `timeout=3600`. It used a separate server configured for a 20,480-token context:

```bash
lm_eval --model local-completions \
  --tasks wikitext \
  --model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \
  --num_fewshot 0 \
  --seed 1234 \
  --output_path ./eval/wikitext2_ppl/results
```

</details>