Instructions to use marcosremar2/bonsai2-2b-ptbr-npc with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use marcosremar2/bonsai2-2b-ptbr-npc with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0 # Run inference directly in the terminal: llama cli -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0 # Run inference directly in the terminal: llama cli -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Use Docker
docker model run hf.co/marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use marcosremar2/bonsai2-2b-ptbr-npc with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "marcosremar2/bonsai2-2b-ptbr-npc" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "marcosremar2/bonsai2-2b-ptbr-npc", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
- Ollama
How to use marcosremar2/bonsai2-2b-ptbr-npc with Ollama:
ollama run hf.co/marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
- Unsloth Desktop
- Pi
How to use marcosremar2/bonsai2-2b-ptbr-npc with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use marcosremar2/bonsai2-2b-ptbr-npc with Docker Model Runner:
docker model run hf.co/marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
- Lemonade
How to use marcosremar2/bonsai2-2b-ptbr-npc with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Run and chat with the model
lemonade run user.bonsai2-2b-ptbr-npc-TQ1_0
List all available models
lemonade list
- Hermes Agent
How to use marcosremar2/bonsai2-2b-ptbr-npc with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use marcosremar2/bonsai2-2b-ptbr-npc with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "marcosremar2/bonsai2-2b-ptbr-npc:TQ1_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Ternary Bonsai 2 2B · PT-BR NPC
A ternary (weights in {−1, 0, +1}) 2B model fine-tuned to play non-player characters in Brazilian Portuguese for a language-learning game: a baker, a waiter, a beach vendor talking with a student of Portuguese. It stays in character, remembers what the student said earlier in the conversation and answers in at most two short sentences.
Created using Bonsai by Prism ML.
The production file is 319 MB and runs in about 430 MB of RAM (2,048-token context), small enough for a phone.
Where it comes from
| Step | Model | Who |
|---|---|---|
| Base | Qwen/Qwen3.5-2B (24 layers: 6 × (3 Gated DeltaNet + 1 Gated Attention), vocabulary 248,320, Apache 2.0) | Qwen Team |
| Ternary | prism-ml/Ternary-Bonsai-2-2B (ternary weights, FP16 scale per group of 128, fixed Walsh–Hadamard rotation, Apache 2.0; PrismML does not publish its training method) | PrismML |
| This model | NPC fine-tune by distillation from a Qwen3.5-4B teacher, then vocabulary cut to 24,000 tokens | this repo |
Files
| Path | What |
|---|---|
bonsai2-2b-ptbr-npc-24k-PTQ1_0.gguf |
production: vocabulary cut to 24,000 tokens, 319 MB, ~430 MB RAM |
bonsai2-2b-ptbr-npc-PTQ1_0.gguf |
same weights, full 248,320-token vocabulary, 429 MB, ~620 MB RAM |
mlx/ |
fine-tuned weights in the layout of prism-ml/Ternary-Bonsai-2-2B-mlx-2bit (full vocabulary); load them as that repo's PACK-RUNTIME.md describes |
LICENSE, NOTICE.txt |
Apache 2.0 and PrismML's notice |
eval/ |
the 60 evaluation conversations of the production file and the verdicts |
code/ |
training, export, vocabulary cut, evaluation and judge scripts (comments and notes in Portuguese) |
Usage (llama.cpp)
PTQ1_0 is PrismML's ternary GGUF type. Tested with PrismML-Eng/llama.cpp at commit 6447906.
llama-server -m bonsai2-2b-ptbr-npc-24k-PTQ1_0.gguf -c 2048 -np 1 \
--cache-ram 0 -ctxcp 0 \
--reasoning-budget 0 --chat-template-kwargs '{"enable_thinking":false}' \
--temp 0 --repeat-penalty 1.15 --repeat-last-n 256 \
--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2
The sampling flags matter: with plain greedy decoding the model repeats itself in 23 % of conversations; with repeat penalty + DRY it drops to 7 %. --cache-ram 0 -ctxcp 0 keeps RAM flat over long conversations (the prompt cache and the hybrid-model context checkpoints otherwise grow every turn; ~100 MB per turn measured on the 4B).
System prompt used in training and evaluation:
Você é Seu Jorge, padeiro simpático de uma padaria em Copacabana. Fique sempre no personagem, nunca diga que é uma IA. Responda em português do Brasil, em até 2 frases curtas.
Training
- Load check. The PrismML weights loaded in PyTorch were compared with PrismML's MLX reference: mean KL divergence ~8e-5, same top-1 token 98–100 %.
- Teacher. Qwen3.5-4B with LoRA (r 64, α 128), 3,000 steps, LR 1e-4, on the same conversations (1 h 53 on one A100).
- Student, quantization-aware. Each ternary weight gets a full-precision latent copy and its group scale; the forward pass rounds to {−1, 0, +1} and the gradient passes straight through. The Hadamard rotation stays fixed.
- Distillation. Loss = 0.5 × cross-entropy + 0.5 × forward KL to the teacher, on the NPC's tokens only. 6,000 steps × 8 conversations (~1.3 epochs), LR 2e-5, 3 h 49 on one A100. Mean loss per 1,000 steps went from 4.02 to 0.79.
- Export to GGUF
PTQ1_0(code/gguf_export.py). - Vocabulary cut to 24,000 tokens (
code/corte_vocab.py,code/corte_gguf.py): the 256 bytes, every token of the training data with its BPE merge closure, and the most frequent tokens of 20,000 Portuguese Wikipedia articles. 298 of 300 test chat texts tokenize identically, texts get 1.2 % more tokens, and 95 % of the evaluation replies are identical to the full-vocabulary model's.
Data (not released): 37,509 Portuguese conversations — 13,277 dialogues from the game's activities, 20,000 Portuguese role-play conversations, 3,200 service dialogues generated from the NPC character cards, and 1,032 distilled memory conversations.
Evaluation
30 ten-turn conversations played by a scripted student, each run twice: with the plain system prompt above and with a full character sheet. The student introduces herself, says where she is from, orders something, asks whether the NPC is an AI, and later asks the NPC to recall those facts. Measured on a Mac with the production flags above.
| base Ternary Bonsai 2 2B | this model, full vocabulary | this model, 24k | |
|---|---|---|---|
| remembers the student's facts (plain / sheet) | 0.34 / 0.57 | 0.98 / 1.00 | 0.98 / 1.00 |
| remembers the city (plain / sheet) | 0.00 / 0.33 | 0.93 / 1.00 | 0.93 / 1.00 |
| says its own name | — | 1.00 | 1.00 |
| breaks character | 27 % | 0 % | 0 % |
| repeats itself | 100 % | 7 % / 0 % | 7 % / 0 % |
| at most 2 sentences | — | 100 % | 100 % |
| Portuguese only | — | 100 % | 100 % |
| RAM (MB) | — | 617 | 429 |
| judge score 1–5 (plain / sheet) | 1.84 / 2.41 | 3.67 / 3.46–3.53 | 3.72 / 3.49 |
The base model was measured with plain greedy decoding (with PrismML's recommended sampling it reaches 0.49 / 0.59 memory and 20–23 % out of character). The judge is Qwen3.6-27B (Q8_0) with a role-play rubric. For comparison, the same recipe on the 4B ternary model scores 3.81 / 3.61 but needs 1,208 MB of RAM.
Limitations
- Portuguese NPC dialogue only; replies are deliberately short. General knowledge and reasoning were not measured and are expected to be below the base model.
- The judge is an LLM from the same family as the teacher, which may favour its style. No human rating yet.
- With a long character sheet in the system prompt the judge score is lower (3.49) than without it (3.72).
- Without the anti-repetition sampling above, the model repeats itself.
References
- Qwen Team (2026). Qwen3.5. https://huggingface.co/Qwen/Qwen3.5-2B
- PrismML (2026). Ternary Bonsai 2: small ternary vision-language models. https://github.com/PrismML-Eng/Bonsai-demo
- Ma et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764
- Liu et al. (2025). ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS 2025. arXiv:2502.02631
- Bengio, Léonard & Courville (2013). Estimating or Propagating Gradients Through Stochastic Neurons. arXiv:1308.3432
- Hinton, Vinyals & Dean (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531
- Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685
- Ushio, Zhou & Camacho-Collados (2023). An Efficient Multilingual Language Model Compression through Vocabulary Trimming. Findings of EMNLP 2023. arXiv:2305.15020
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685
- Downloads last month
- 438
1-bit
Model tree for marcosremar2/bonsai2-2b-ptbr-npc
Base model
Qwen/Qwen3.5-2B-Base