Ternary Bonsai 2 2B · PT-BR NPC

A ternary (weights in {−1, 0, +1}) 2B model fine-tuned to play non-player characters in Brazilian Portuguese for a language-learning game: a baker, a waiter, a beach vendor talking with a student of Portuguese. It stays in character, remembers what the student said earlier in the conversation and answers in at most two short sentences.

Created using Bonsai by Prism ML.

The production file is 319 MB and runs in about 430 MB of RAM (2,048-token context), small enough for a phone.

Where it comes from

Step Model Who
Base Qwen/Qwen3.5-2B (24 layers: 6 × (3 Gated DeltaNet + 1 Gated Attention), vocabulary 248,320, Apache 2.0) Qwen Team
Ternary prism-ml/Ternary-Bonsai-2-2B (ternary weights, FP16 scale per group of 128, fixed Walsh–Hadamard rotation, Apache 2.0; PrismML does not publish its training method) PrismML
This model NPC fine-tune by distillation from a Qwen3.5-4B teacher, then vocabulary cut to 24,000 tokens this repo

Files

Path What
bonsai2-2b-ptbr-npc-24k-PTQ1_0.gguf production: vocabulary cut to 24,000 tokens, 319 MB, ~430 MB RAM
bonsai2-2b-ptbr-npc-PTQ1_0.gguf same weights, full 248,320-token vocabulary, 429 MB, ~620 MB RAM
mlx/ fine-tuned weights in the layout of prism-ml/Ternary-Bonsai-2-2B-mlx-2bit (full vocabulary); load them as that repo's PACK-RUNTIME.md describes
LICENSE, NOTICE.txt Apache 2.0 and PrismML's notice
eval/ the 60 evaluation conversations of the production file and the verdicts
code/ training, export, vocabulary cut, evaluation and judge scripts (comments and notes in Portuguese)

Usage (llama.cpp)

PTQ1_0 is PrismML's ternary GGUF type. Tested with PrismML-Eng/llama.cpp at commit 6447906.

llama-server -m bonsai2-2b-ptbr-npc-24k-PTQ1_0.gguf -c 2048 -np 1 \
  --cache-ram 0 -ctxcp 0 \
  --reasoning-budget 0 --chat-template-kwargs '{"enable_thinking":false}' \
  --temp 0 --repeat-penalty 1.15 --repeat-last-n 256 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2

The sampling flags matter: with plain greedy decoding the model repeats itself in 23 % of conversations; with repeat penalty + DRY it drops to 7 %. --cache-ram 0 -ctxcp 0 keeps RAM flat over long conversations (the prompt cache and the hybrid-model context checkpoints otherwise grow every turn; ~100 MB per turn measured on the 4B).

System prompt used in training and evaluation:

Você é Seu Jorge, padeiro simpático de uma padaria em Copacabana. Fique sempre no personagem, nunca diga que é uma IA. Responda em português do Brasil, em até 2 frases curtas.

Training

  1. Load check. The PrismML weights loaded in PyTorch were compared with PrismML's MLX reference: mean KL divergence ~8e-5, same top-1 token 98–100 %.
  2. Teacher. Qwen3.5-4B with LoRA (r 64, α 128), 3,000 steps, LR 1e-4, on the same conversations (1 h 53 on one A100).
  3. Student, quantization-aware. Each ternary weight gets a full-precision latent copy and its group scale; the forward pass rounds to {−1, 0, +1} and the gradient passes straight through. The Hadamard rotation stays fixed.
  4. Distillation. Loss = 0.5 × cross-entropy + 0.5 × forward KL to the teacher, on the NPC's tokens only. 6,000 steps × 8 conversations (~1.3 epochs), LR 2e-5, 3 h 49 on one A100. Mean loss per 1,000 steps went from 4.02 to 0.79.
  5. Export to GGUF PTQ1_0 (code/gguf_export.py).
  6. Vocabulary cut to 24,000 tokens (code/corte_vocab.py, code/corte_gguf.py): the 256 bytes, every token of the training data with its BPE merge closure, and the most frequent tokens of 20,000 Portuguese Wikipedia articles. 298 of 300 test chat texts tokenize identically, texts get 1.2 % more tokens, and 95 % of the evaluation replies are identical to the full-vocabulary model's.

Data (not released): 37,509 Portuguese conversations — 13,277 dialogues from the game's activities, 20,000 Portuguese role-play conversations, 3,200 service dialogues generated from the NPC character cards, and 1,032 distilled memory conversations.

Evaluation

30 ten-turn conversations played by a scripted student, each run twice: with the plain system prompt above and with a full character sheet. The student introduces herself, says where she is from, orders something, asks whether the NPC is an AI, and later asks the NPC to recall those facts. Measured on a Mac with the production flags above.

base Ternary Bonsai 2 2B this model, full vocabulary this model, 24k
remembers the student's facts (plain / sheet) 0.34 / 0.57 0.98 / 1.00 0.98 / 1.00
remembers the city (plain / sheet) 0.00 / 0.33 0.93 / 1.00 0.93 / 1.00
says its own name — 1.00 1.00
breaks character 27 % 0 % 0 %
repeats itself 100 % 7 % / 0 % 7 % / 0 %
at most 2 sentences — 100 % 100 %
Portuguese only — 100 % 100 %
RAM (MB) — 617 429
judge score 1–5 (plain / sheet) 1.84 / 2.41 3.67 / 3.46–3.53 3.72 / 3.49

The base model was measured with plain greedy decoding (with PrismML's recommended sampling it reaches 0.49 / 0.59 memory and 20–23 % out of character). The judge is Qwen3.6-27B (Q8_0) with a role-play rubric. For comparison, the same recipe on the 4B ternary model scores 3.81 / 3.61 but needs 1,208 MB of RAM.

Limitations

  • Portuguese NPC dialogue only; replies are deliberately short. General knowledge and reasoning were not measured and are expected to be below the base model.
  • The judge is an LLM from the same family as the teacher, which may favour its style. No human rating yet.
  • With a long character sheet in the system prompt the judge score is lower (3.49) than without it (3.72).
  • Without the anti-repetition sampling above, the model repeats itself.

References

  • Qwen Team (2026). Qwen3.5. https://huggingface.co/Qwen/Qwen3.5-2B
  • PrismML (2026). Ternary Bonsai 2: small ternary vision-language models. https://github.com/PrismML-Eng/Bonsai-demo
  • Ma et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764
  • Liu et al. (2025). ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS 2025. arXiv:2502.02631
  • Bengio, Léonard & Courville (2013). Estimating or Propagating Gradients Through Stochastic Neurons. arXiv:1308.3432
  • Hinton, Vinyals & Dean (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531
  • Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685
  • Ushio, Zhou & Camacho-Collados (2023). An Efficient Multilingual Language Model Compression through Vocabulary Trimming. Findings of EMNLP 2023. arXiv:2305.15020
  • Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685
Downloads last month
438
GGUF
Model size
1B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

1-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for marcosremar2/bonsai2-2b-ptbr-npc

Finetuned
Qwen/Qwen3.5-2B
Finetuned
(1)
this model

Papers for marcosremar2/bonsai2-2b-ptbr-npc