Release Finch-4B-KTO: model weights + README + assets

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/intro_teaser.png filter=lfs diff=lfs merge=lfs -text
+assets/results_kto.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,204 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model:
+- minnesotanlp/Finch-4B
+datasets:
+- minnesotanlp/Finch-Collection
+tags:
+- evolution-fine-tuning
+- evolutionary-search
+- discovery
+- preference-learning
+- kto
+- mutation-operator
+- mid-training
+---
+
+<div align="center">
+
+  
+  <h2 style="font-weight: bold; font-size: 1.8em; margin-top: 11px;">
+    Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks
+  </h2>
+  <p style="font-size: 1.05em; margin-top: 4px;">
+    <em>A mid-training "practice phase" that teaches small open-source LLMs <strong>how to evolve solutions</strong>.</em>
+  </p>
+
+<p align="center">
+  <a href="https://open-galapagos.github.io/evolution_finetuning/"><img src="https://img.shields.io/badge/Website-Project%20Page-1E90FF.svg?logo=googlechrome&logoColor=white" alt="Website"/></a>
+  <a href="https://github.com/Open-Galapagos/evolution_finetuning"><img src="https://img.shields.io/badge/GitHub-Code-181717.svg?logo=github" alt="GitHub"/></a>
+  <a href="https://huggingface.co/datasets/minnesotanlp/Finch-Collection"><img src="https://img.shields.io/badge/HuggingFace-Dataset-FFD21E.svg?logo=huggingface" alt="Dataset"/></a>
+  <a href="https://huggingface.co/minnesotanlp/Finch-2B"><img src="https://img.shields.io/badge/-Finch--2B-FFD21E?logo=huggingface&logoColor=black" alt="Finch-2B"/></a>
+  <a href="https://huggingface.co/minnesotanlp/Finch-4B"><img src="https://img.shields.io/badge/-Finch--4B-FFD21E?logo=huggingface&logoColor=black" alt="Finch-4B"/></a>
+  <a href="https://huggingface.co/minnesotanlp/Finch-8B"><img src="https://img.shields.io/badge/-Finch--8B-FFD21E?logo=huggingface&logoColor=black" alt="Finch-8B"/></a>
+  <a href="https://huggingface.co/minnesotanlp/Finch-9B"><img src="https://img.shields.io/badge/-Finch--9B-FFD21E?logo=huggingface&logoColor=black" alt="Finch-9B"/></a>
+  <a href="https://huggingface.co/minnesotanlp/Finch-8B-KTO"><img src="https://img.shields.io/badge/-Finch--8B--KTO-FFD21E?logo=huggingface&logoColor=black" alt="Finch-8B-KTO"/></a>
+  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="Apache 2.0"/></a>
+</p>
+
+</div>
+
+**Finch-4B-KTO** extends [**`Finch-4B`**](https://huggingface.co/minnesotanlp/Finch-4B) with a second, preference-learning stage ([**KTO**](https://arxiv.org/abs/2402.01306)). On top of evolution fine-tuning — which teaches the model *how to evolve a solution* as a **mutation operator** — KTO adds the ability to **self-judge** which candidate solutions are promising and which fall short, further sharpening cross-task discovery. Lineage: **Qwen3.5-4B → Finch-4B (EFT + SFT) → Finch-4B-KTO**.
+
+## TL;DR
+
+Evolution Fine-Tuning (EFT) turns evolutionary search **trajectories** into supervision, moving discovery behavior from the *scaffold* into the *model*. **KTO** then trains the model on `improved` vs. `regressed` transitions jointly, so it internalizes a sense of solution quality — improving every measured discovery metric over Finch-4B.
+
+<div align="center">
+  <img src="./assets/intro_teaser.png" alt="EFT as mid-training" width="960"/>
+</div>
+
+- (Left) EFT acts as mid-training, boosting Finch's discovery on the Erdős minimum-overlap problem under both test-time search and test-time learning.
+- (Right) On NP-hard competitive programming, Finch composes strategies learned across diverse domains, while the base model relies on a single repetitive strategy.
+
+## Finch family
+
+| Model | Base | Params | Training | 🤗 Hugging Face |
+|---|---|---:|---|:---:|
+| `Finch-2B` | Qwen3.5-2B | 2B | EFT | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-2B) |
+| `Finch-4B` | Qwen3.5-4B | 4B | EFT | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-4B) |
+| `Finch-8B` | Qwen3-8B | 8B | EFT | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-8B) |
+| `Finch-9B` | Qwen3.5-9B | 9B | EFT | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-9B) |
+| **`Finch-4B-KTO`** ← *this model* | **Qwen3.5-4B** | **4B** | **EFT + KTO** | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-4B-KTO) |
+| `Finch-8B-KTO` | Qwen3-8B | 8B | EFT + KTO | [![Open on Hugging Face](https://img.shields.io/badge/-Open-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/minnesotanlp/Finch-8B-KTO) |
+
+## How to Use Finch
+
+1. **Execute OpenEvolve scaffold with Finch**
+
+Finch is a **mutation operator for evolutionary search**, most effective driven by a scaffold such as **OpenEvolve** (`T = 100`, temperature `0.7`, top-`p` `0.95`, up to `30K` tokens).
+You can also use other scaffolds in the [SkyDiscover](https://github.com/skydiscover-ai/skydiscover) framework, but we do not guarantee performance, as our model is trained on OpenEvolve's trajectories — one of this work's limitations.
+
+2. **Calling Finch directly**
+
+You can also call Finch directly:
+
+**System prompt** (task-level instruction from the OpenEvolve scaffold):
+```
+You are an expert mathematician specializing in circle packing problems and computational geometry.
+Your task is to improve a constructor function that directly produces a specific arrangement of
+26 circles in a unit square, maximizing the sum of their radii.
+The AlphaEvolve paper achieved a sum of 2.635 for n=26.
+
+Key geometric insights:
+- Circle packings often follow hexagonal patterns in the densest regions
+- Maximum density for infinite circle packing is pi/(2*sqrt(3)) ≈ 0.9069
+- Edge effects make square container packing harder than infinite packing
+- Similar radius circles often form regular patterns, while varied radii allow better space utilization
+```
+
+**User prompt** (evolutionary state — current program + evaluator feedback + evolutionary history):
+```
+# Current Program Information
+- Fitness: 0.3642 (sum_radii: 0.9598)
+- Focus areas: Fitness unchanged at 0.3642. Consider simplifying — code length exceeds 500 characters.
+
+# Program Evolution History
+## Previous Attempts
+
+### Attempt 1
+- Changes: Replace concentric ring placement with hexagonal lattice (5-6-5-6-5 row pattern)
+- Metrics: sum_radii: 0.9598, validity: 1.0 — Improvement in all metrics
+
+# Current Program
+
+# EVOLVE-BLOCK-START
+import numpy as np
+
+def construct_packing():
+    n = 26
+    centers = np.zeros((n, 2))
+    centers[0] = [0.5, 0.5]                              # center circle
+    for i in range(8):                                    # inner ring
+        angle = 2 * np.pi * i / 8
+        centers[i+1] = [0.5 + 0.3*np.cos(angle), 0.5 + 0.3*np.sin(angle)]
+    for i in range(16):                                   # outer ring
+        angle = 2 * np.pi * i / 16
+        centers[i+9] = [0.5 + 0.7*np.cos(angle), 0.5 + 0.7*np.sin(angle)]
+    centers = np.clip(centers, 0.01, 0.99)
+    radii = compute_max_radii(centers)
+    return centers, radii, np.sum(radii)
+# EVOLVE-BLOCK-END
+```
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "minnesotanlp/Finch-4B-KTO"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
+
+# Given an evolutionary state — task instruction + parent program + evolutionary history
+# + evaluator feedback — Finch proposes an improved candidate program.
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},  # provided by your evolutionary scaffold
+    {"role": "user", "content": USER_PROMPT},      # parent program + feedback + history
+]
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt"
+).to(model.device)
+
+out = model.generate(inputs, max_new_tokens=30000, do_sample=True, temperature=0.7, top_p=0.95)
+print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+## Training
+
+Two stages:
+
+- **Stage 1 — EFT (SFT).** [`Finch-4B`](https://huggingface.co/minnesotanlp/Finch-4B): full SFT of **Qwen3.5-4B** on `improved` transitions from the [Finch Collection](https://huggingface.co/datasets/minnesotanlp/Finch-Collection) (355 training tasks; one run/task → **30,445** examples; 900 for validation) with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) — 1 epoch, global batch size 128, LR 1e-5, on 8× NVIDIA H200 140GB.
+- **Stage 2 — KTO.** Preference learning ([KTO](https://arxiv.org/abs/2402.01306)) on `improved` (desirable) and `regressed` (undesirable) transitions jointly, maximizing the contrastive signal that guides the model toward self-judging which solutions are promising and which fall short.
+- **Teacher (data).** Trajectories generated by **Qwen3.5-397B-A17B** inside the **OpenEvolve** scaffold.
+
+## Results
+
+- Finch outperforms its same-size base model by +10.2% on 22 held-out tasks across 5 domains, with improvements of up to +290% on individual tasks.
+- Larger models benefit more, and Finch-4B matches a model roughly 2× larger on the Erdős task.
+
+
+<div align="center">
+  <img src="./assets/results_main.png" alt="main results" width="900"/>
+</div>
+
+- On competitive programming (FrontierCS), Finch-9B averages 46.01 vs base Qwen3.5-9B's 32.46; on CALICO's P263 (UC Berkeley's official open-ended contest) it scores 86.10 vs 55.09 
+
+
+<div align="center">
+  <img src="./assets/results_frontiercs.png" alt="frontiercs results" width="760"/>
+</div>
+
+- With preference learning (KTO), Finch-8B surpasses the best human score on AC1 and AC2, while its competitive programming score improves from 24.56 → 37.30.
+- Finch-8B matches SOTA on two circle-packing tasks and improves the Erdős task by +3.2%.
+
+<div align="center">
+  <img src="./assets/results_kto_ttrl.png" alt="frontiercs results" width="760"/>
+</div>
+
+## Limitations
+
+Trajectories are collected and evaluated only with **OpenEvolve**; behavior under different scaffolds is not guaranteed.
+
+## License
+
+The **Finch Collection** is released under the [**CC-BY 4.0 License**](https://creativecommons.org/licenses/by/4.0/) and is recommended for **non-commercial academic research**. The accompanying **code** and **Finch model weights** are released under the [**Apache 2.0 License**](https://www.apache.org/licenses/LICENSE-2.0).
+
+## Acknowledgements
+
+This research was supported by the "Advanced GPU Utilization Support Program" funded by the Government of the Republic of Korea (Ministry of Science and ICT). We are grateful to the SkyDiscover team for their valuable feedback on the dataset construction process, the use of the SkyDiscover framework, and the overall direction of this research — in particular, [Shu Liu](https://shulynnliu.com/), [Shubham Agarwal](https://skejriwal44.github.io/), and [Mert Cemri](https://people.eecs.berkeley.edu/~mert_cemri/) for their insightful comments and discussions. We also thank the OpenEvolve team, especially Ritik Vijayvergiya and [Asankhaya Sharma](https://asankhaya.github.io/), for their guidance on using the OpenEvolve framework and for their thoughtful comments on this work. We further thank the authors of ALE-Bench, especially [Yuki Imajuku](https://imajuku.tech/), and the AtCoder team for authorizing the public release of the evolutionary search trajectories derived from their CC BY-ND 4.0-licensed dataset. Finally, we thank [Byung-Kwan Lee](https://byungkwanlee.github.io/ByungKwanLee-CV/) for valuable feedback during the early stages of this project.
+
+## Citation
+
+```bibtex
+@misc{lee2026evolutionfinetuning,
+      title        = {Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks},
+      author       = {Young-Jun Lee and Seungone Kim and Minki Kang and Alistair Cheong and Zerui Chen and Seungho Han and Taehee Jung and Dongyeop Kang},
+      year         = {2026},
+      note         = {Preprint},
+      url          = {https://open-galapagos.github.io/evolution_finetuning/},
+}
+```

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,113 @@
+{
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "dtype": "bfloat16",
+  "eos_token_id": 248046,
+  "hidden_size": 2560,
+  "image_token_id": 248056,
+  "model_type": "qwen3_5",
+  "pad_token_id": 248044,
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": null,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "initializer_range": 0.02,
+    "intermediate_size": 9216,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 32,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "mlp_only_layers": [],
+    "model_type": "qwen3_5_text",
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 4,
+    "pad_token_id": null,
+    "partial_rotary_factor": 0.25,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "tie_word_embeddings": true,
+    "use_cache": false,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.2.0",
+  "use_cache": false,
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 24,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1024,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "model_type": "qwen3_5",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 2560,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "transformers_version": "5.2.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc4ef99daf78da81982347168ff7de42ec7505945cb3aec2ed7f72f7821e8340
+size 10350019328

processor_config.json

@@ -0,0 +1,63 @@
+{
+  "image_processor": {
+    "data_format": "channels_first",
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessorFast",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "data_format": "channels_first",
+    "default_to_square": true,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4
+size 19989343

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}