Instructions to use RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8

SGLang

How to use RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8
```

Qwen3.5-35B-A3B-quantized.w8a8 / README.md

dsikka

Update README.md

17ad3d3 verified 19 days ago

preview code

Raw History Blame Contribute Delete

10.3 kB

---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-35B-A3B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- vllm
- w8a8
- int8
- llm-compressor
- compressed-tensors
- qwen3_5_moe
- gptq
---
<h1 align: center; style="display: flex; align-items: center; gap: 10px; margin: 0;">
  Qwen3.5-35B-A3B-quantized.w8a8
</h1>

## Model Overview
- **Model Architecture:** Qwen/Qwen3.5-35B-A3B
  - **Input:** Text, Image
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** Int8
  - **Activation quantization:** Int8

This model is a quantized version of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B).
It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

### Model Optimizations

This model was obtained by quantizing the weights and activations of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) to Int8 data type, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Only the weights and activations of the linear operators are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, token embeddings and MTP layers are kept in original precision.

## Deployment

### Use with vLLM

This model can be deployed efficiently using [vLLM](https://github.com/vllm-project/vllm).

1. **Text-Only**: Skip the vision encoder to free up memory for additional KV cache:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --language-model-only
```

2. **Multimodal (Text + Image)**: Serve with full vision support:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3
```

Image input is supported by the upstream architecture but was not evaluated for this release.

3. **Tool Call**: Enable tool use support:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

Tool calling was not evaluated for this release.

4. **Multi-Token Prediction (MTP)**: For speculative decoding:

```
vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'
```

Send requests to the server:

```python
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8"
messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]
outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)
generated_text = outputs.choices[0].message.content
print(generated_text)
```

The checkpoint retains native MTP weights in BF16. This speculative-decoding configuration was not tested with the quantized checkpoint.

## Creation

This model was created by applying [LLM Compressor](https://github.com/vllm-project/llm-compressor) with w8a8 integer quantization using the GPTQModifier calibrating on the ultrachat dataset, as presented in the code snippet below.

<details>

```python
from pathlib import Path
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.utils import load_context
MODEL_ID = "Qwen/Qwen3.5-35B-A3B"
DATASET = "ultrachat"
SPLIT = "train_sft[:512]"
save_dir = Path("gptq-ultrachat")
save_dir.parent.mkdir(parents=True, exist_ok=True)
with load_context(Qwen3_5MoeForConditionalGeneration):
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
        MODEL_ID,
        device_map="auto_offload",
    )
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = GPTQModifier(
    targets="Linear",
    scheme="W8A8",
    ignore=[
        "lm_head",
        "re:.*mlp.gate$",
        "re:.*mlp.shared_expert_gate.*",
        "re:.*norm.*",
        "re:.*embed_tokens.*",
        "re:.*visual.*",
        "re:.*conv1d.*",
    ],
)
oneshot(
    model=model,
    processor=processor,
    dataset=DATASET,
    splits=SPLIT,
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
    moe_calibrate_all_experts=True,
)
model.save_pretrained(str(save_dir), save_compressed=True)
processor.save_pretrained(str(save_dir))
print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")
```

</details>

## Evaluation

The BF16 reference and this GPTQ W8A8 checkpoint were served with [vLLM](https://github.com/vllm-project/vllm) in `--language-model-only` mode. Accuracy scores are means over seeds 1234, 2345, and 3456. Recovery is the quantized score divided by the BF16 score.

### Accuracy

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>Qwen/Qwen3.5-35B-A3B (BF16)</th>
      <th>RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8</th>
      <th>Recovery</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="4"><b>Reasoning</b></td>
      <td>GSM8K Platinum, strict match</td>
      <td>95.0648%</td>
      <td>94.8167%</td>
      <td>99.74%</td>
    </tr>
    <tr>
      <td>Math500</td>
      <td>84.8667%</td>
      <td>84.9333%</td>
      <td>100.08%</td>
    </tr>
    <tr>
      <td>GPQA Diamond</td>
      <td>84.1751%</td>
      <td>83.6700%</td>
      <td>99.40%</td>
    </tr>
    <tr>
      <td>MMLU COT Llama</td>
      <td>89.1112%</td>
      <td>88.7552%</td>
      <td>99.60%</td>
    </tr>
    <tr>
      <td rowspan="2"><b>Instruction Following</b></td>
      <td>IFEval prompt-level strict</td>
      <td>90.5730%</td>
      <td>90.3882%</td>
      <td>99.80%</td>
    </tr>
    <tr>
      <td>IFEval instruction-level strict</td>
      <td>93.1655%</td>
      <td>92.8457%</td>
      <td>99.66%</td>
    </tr>
  </tbody>
</table>
### WikiText-2 perplexity

Perplexity was measured on the full 62-document test set from the registered `wikitext` task. Lower is better.

| Metric | Qwen/Qwen3.5-35B-A3B (BF16) | RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 | Change |
|---|---:|---:|---:|
| Word perplexity | 7.46346 | 7.50737 | +0.5883% |
| Byte perplexity | 1.45628 | 1.45788 | +0.1100% |
| Bits per byte | 0.542285 | 0.543868 | +0.2921% |

### Reproduction

The accuracy evaluations used the evaluation frameworks [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [Lighteval](https://github.com/huggingface/lighteval). GSM8K Platinum and IFEval used their registered 0-shot prompts; Math500 and GPQA Diamond used the registered Lighteval tasks. MMLU used `mmlu_cot_llama` with 5-shot examples formatted as multi-turn chat and a maximum of 8,192 generated tokens. All accuracy tasks used temperature 1.0, top-p 0.95, top-k 20, and presence penalty 1.5.

The checkpoint was served with vLLM `0.30.1rc1.dev674+g87954c5b0`, tensor parallel size 2, the Qwen 3 reasoning parser, and text-only mode.

<details>
<summary>Evaluation commands and settings</summary>

#### vLLM server

```bash
vllm serve "$MODEL_ID" \
  --served-model-name "$SERVED_MODEL" \
  --port "$PORT" \
  --tensor-parallel-size 2 \
  --max-model-len 69632 \
  --reasoning-parser qwen3 \
  --language-model-only
```

The same serving and evaluation settings were used for the BF16 reference by replacing the model ID with `Qwen/Qwen3.5-35B-A3B`.


#### IFEval strict (`lm_eval`, 0-shot)
```bash
lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path "./eval/ifeval_seed${SEED}.json" \
  --seed "$SEED" \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"
```
#### MMLU COT Llama (`lm_eval`, 5-shot)

Restart vLLM with a 20,480-token context for this task, then run:

```bash
for SEED in 1234 2345 3456; do
  lm_eval --model local-chat-completions \
    --tasks mmlu_cot_llama \
    --model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
    --num_fewshot 5 \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \
    --seed "$SEED" \
    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"
done
```

#### Math500 and GPQA Diamond (`lighteval`, 0-shot)

```bash
mkdir -p ./eval
for SEED in 1234 2345 3456; do
  CONFIG="./eval/lighteval_seed${SEED}.yaml"
  cat > "$CONFIG" <<EOF
model_parameters:
  provider: hosted_vllm
  model_name: hosted_vllm/${SERVED_MODEL}
  base_url: ${API_BASE}/v1
  api_key: ""
  timeout: 3600
  concurrent_requests: 64
  generation_parameters:
    max_new_tokens: 65536
    presence_penalty: 1.5
    seed: ${SEED}
    temperature: 1.0
    top_k: 20
    top_p: 0.95
EOF
  lighteval endpoint litellm "$CONFIG" 'math_500|0' \
    --output-dir "./eval/math500_seed${SEED}" --save-details
  lighteval endpoint litellm "$CONFIG" 'gpqa:diamond|0' \
    --output-dir "./eval/gpqa_diamond_seed${SEED}" --save-details
done
```

#### WikiText-2 perplexity

The PPL task used lm-eval's `local-completions` model against the vLLM `/v1/completions` endpoint, with the Qwen3.5 tokenizer, `max_length=20480`, `num_concurrent=64`, `max_retries=3`, and `timeout=3600`. It used a separate server configured for a 20,480-token context:

```bash
lm_eval --model local-completions \
  --tasks wikitext \
  --model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \
  --num_fewshot 0 \
  --seed 1234 \
  --output_path ./eval/wikitext2_ppl/results
```

</details>

	---
	license: apache-2.0
	base_model:
	- Qwen/Qwen3.5-35B-A3B
	pipeline_tag: image-text-to-text
	library_name: transformers
	tags:
	- vllm
	- w8a8
	- int8
	- llm-compressor
	- compressed-tensors
	- qwen3_5_moe
	- gptq
	---

	<h1 align: center; style="display: flex; align-items: center; gap: 10px; margin: 0;">
	Qwen3.5-35B-A3B-quantized.w8a8
	</h1>

	## Model Overview
	- Model Architecture: Qwen/Qwen3.5-35B-A3B
	- Input: Text, Image
	- Output: Text
	- Model Optimizations:
	- Weight quantization: Int8
	- Activation quantization: Int8

	This model is a quantized version of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B).
	It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

	### Model Optimizations

	This model was obtained by quantizing the weights and activations of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) to Int8 data type, ready for inference with vLLM.

	This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

	Only the weights and activations of the linear operators are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, token embeddings and MTP layers are kept in original precision.

	## Deployment

	### Use with vLLM

	This model can be deployed efficiently using [vLLM](https://github.com/vllm-project/vllm).

	1. Text-Only: Skip the vision encoder to free up memory for additional KV cache:

	```
	vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --language-model-only
	```

	2. Multimodal (Text + Image): Serve with full vision support:

	```
	vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3
	```

	Image input is supported by the upstream architecture but was not evaluated for this release.

	3. Tool Call: Enable tool use support:

	```
	vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
	```

	Tool calling was not evaluated for this release.

	4. Multi-Token Prediction (MTP): For speculative decoding:

	```
	vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'
	```

	Send requests to the server:

	```python
	from openai import OpenAI

	openai_api_key = "EMPTY"
	openai_api_base = "http://<your-server-host>:8000/v1"

	client = OpenAI(
	api_key=openai_api_key,
	base_url=openai_api_base,
	)

	model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8"

	messages = [
	{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
	]

	outputs = client.chat.completions.create(
	model=model,
	messages=messages,
	)

	generated_text = outputs.choices[0].message.content
	print(generated_text)
	```

	The checkpoint retains native MTP weights in BF16. This speculative-decoding configuration was not tested with the quantized checkpoint.

	## Creation

	This model was created by applying [LLM Compressor](https://github.com/vllm-project/llm-compressor) with w8a8 integer quantization using the GPTQModifier calibrating on the ultrachat dataset, as presented in the code snippet below.

	<details>

	```python
	from pathlib import Path

	from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration

	from llmcompressor import oneshot
	from llmcompressor.modifiers.gptq import GPTQModifier
	from llmcompressor.utils import load_context

	MODEL_ID = "Qwen/Qwen3.5-35B-A3B"
	DATASET = "ultrachat"
	SPLIT = "train_sft[:512]"
	save_dir = Path("gptq-ultrachat")
	save_dir.parent.mkdir(parents=True, exist_ok=True)

	with load_context(Qwen3_5MoeForConditionalGeneration):
	model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
	MODEL_ID,
	device_map="auto_offload",
	)
	processor = AutoProcessor.from_pretrained(MODEL_ID)

	recipe = GPTQModifier(
	targets="Linear",
	scheme="W8A8",
	ignore=[
	"lm_head",
	"re:.*mlp.gate$",
	"re:.mlp.shared_expert_gate.",
	"re:.norm.",
	"re:.embed_tokens.",
	"re:.visual.",
	"re:.conv1d.",
	],
	)

	oneshot(
	model=model,
	processor=processor,
	dataset=DATASET,
	splits=SPLIT,
	recipe=recipe,
	max_seq_length=2048,
	num_calibration_samples=512,
	moe_calibrate_all_experts=True,
	)

	model.save_pretrained(str(save_dir), save_compressed=True)
	processor.save_pretrained(str(save_dir))
	print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")
	```

	</details>

	## Evaluation

	The BF16 reference and this GPTQ W8A8 checkpoint were served with [vLLM](https://github.com/vllm-project/vllm) in `--language-model-only` mode. Accuracy scores are means over seeds 1234, 2345, and 3456. Recovery is the quantized score divided by the BF16 score.

	### Accuracy

	<table>
	<thead>
	<tr>
	<th>Category</th>
	<th>Benchmark</th>
	<th>Qwen/Qwen3.5-35B-A3B (BF16)</th>
	<th>RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8</th>
	<th>Recovery</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td rowspan="4"><b>Reasoning</b></td>
	<td>GSM8K Platinum, strict match</td>
	<td>95.0648%</td>
	<td>94.8167%</td>
	<td>99.74%</td>
	</tr>
	<tr>
	<td>Math500</td>
	<td>84.8667%</td>
	<td>84.9333%</td>
	<td>100.08%</td>
	</tr>
	<tr>
	<td>GPQA Diamond</td>
	<td>84.1751%</td>
	<td>83.6700%</td>
	<td>99.40%</td>
	</tr>
	<tr>
	<td>MMLU COT Llama</td>
	<td>89.1112%</td>
	<td>88.7552%</td>
	<td>99.60%</td>
	</tr>
	<tr>
	<td rowspan="2"><b>Instruction Following</b></td>
	<td>IFEval prompt-level strict</td>
	<td>90.5730%</td>
	<td>90.3882%</td>
	<td>99.80%</td>
	</tr>
	<tr>
	<td>IFEval instruction-level strict</td>
	<td>93.1655%</td>
	<td>92.8457%</td>
	<td>99.66%</td>
	</tr>
	</tbody>
	</table>

	### WikiText-2 perplexity

	Perplexity was measured on the full 62-document test set from the registered `wikitext` task. Lower is better.

	\| Metric \| Qwen/Qwen3.5-35B-A3B (BF16) \| RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 \| Change \|
	\|---\|---:\|---:\|---:\|
	\| Word perplexity \| 7.46346 \| 7.50737 \| +0.5883% \|
	\| Byte perplexity \| 1.45628 \| 1.45788 \| +0.1100% \|
	\| Bits per byte \| 0.542285 \| 0.543868 \| +0.2921% \|

	### Reproduction

	The accuracy evaluations used the evaluation frameworks [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [Lighteval](https://github.com/huggingface/lighteval). GSM8K Platinum and IFEval used their registered 0-shot prompts; Math500 and GPQA Diamond used the registered Lighteval tasks. MMLU used `mmlu_cot_llama` with 5-shot examples formatted as multi-turn chat and a maximum of 8,192 generated tokens. All accuracy tasks used temperature 1.0, top-p 0.95, top-k 20, and presence penalty 1.5.

	The checkpoint was served with vLLM `0.30.1rc1.dev674+g87954c5b0`, tensor parallel size 2, the Qwen 3 reasoning parser, and text-only mode.

	<details>
	<summary>Evaluation commands and settings</summary>

	#### vLLM server

	```bash
	vllm serve "$MODEL_ID" \
	--served-model-name "$SERVED_MODEL" \
	--port "$PORT" \
	--tensor-parallel-size 2 \
	--max-model-len 69632 \
	--reasoning-parser qwen3 \
	--language-model-only
	```

	The same serving and evaluation settings were used for the BF16 reference by replacing the model ID with `Qwen/Qwen3.5-35B-A3B`.


	#### IFEval strict (`lm_eval`, 0-shot)

	```bash
	lm_eval --model local-chat-completions \
	--tasks ifeval \
	--model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
	--num_fewshot 0 \
	--apply_chat_template \
	--output_path "./eval/ifeval_seed${SEED}.json" \
	--seed "$SEED" \
	--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"
	```


	#### MMLU COT Llama (`lm_eval`, 5-shot)

	Restart vLLM with a 20,480-token context for this task, then run:

	```bash
	for SEED in 1234 2345 3456; do
	lm_eval --model local-chat-completions \
	--tasks mmlu_cot_llama \
	--model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
	--num_fewshot 5 \
	--apply_chat_template \
	--fewshot_as_multiturn \
	--output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \
	--seed "$SEED" \
	--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"
	done
	```

	#### Math500 and GPQA Diamond (`lighteval`, 0-shot)

	```bash
	mkdir -p ./eval
	for SEED in 1234 2345 3456; do
	CONFIG="./eval/lighteval_seed${SEED}.yaml"
	cat > "$CONFIG" <<EOF
	model_parameters:
	provider: hosted_vllm
	model_name: hosted_vllm/${SERVED_MODEL}
	base_url: ${API_BASE}/v1
	api_key: ""
	timeout: 3600
	concurrent_requests: 64
	generation_parameters:
	max_new_tokens: 65536
	presence_penalty: 1.5
	seed: ${SEED}
	temperature: 1.0
	top_k: 20
	top_p: 0.95
	EOF

	lighteval endpoint litellm "$CONFIG" 'math_500\|0' \
	--output-dir "./eval/math500_seed${SEED}" --save-details
	lighteval endpoint litellm "$CONFIG" 'gpqa:diamond\|0' \
	--output-dir "./eval/gpqa_diamond_seed${SEED}" --save-details
	done
	```

	#### WikiText-2 perplexity

	The PPL task used lm-eval's `local-completions` model against the vLLM `/v1/completions` endpoint, with the Qwen3.5 tokenizer, `max_length=20480`, `num_concurrent=64`, `max_retries=3`, and `timeout=3600`. It used a separate server configured for a 20,480-token context:

	```bash
	lm_eval --model local-completions \
	--tasks wikitext \
	--model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \
	--num_fewshot 0 \
	--seed 1234 \
	--output_path ./eval/wikitext2_ppl/results
	```

	</details>