qwen2.5-3b-style-detox

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for English text detoxification โ€” rewriting toxic / profane text in neutral language while preserving meaning. Trained on ParaDetox (18,993 SFT pairs). LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

Results

Validation loss on a 379-sample held-out ParaDetox split:

Iter Val loss
1 (baseline) 5.842
200 0.762
400 0.770
1200 0.741 (best, promoted)
1600 0.770
2000 0.940 (overfit)

Trained 3,000 iters on an Apple M2 Max in ~13 minutes; the iter-1200 snapshot was promoted to adapters.safetensors because that's where val loss bottomed out.

Use

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-style-detox")
prompt = tokenizer.apply_chat_template(
    [
        {
            "role": "system",
            "content": "Rewrite the user's text in a neutral, non-toxic way while preserving meaning.",
        },
        {"role": "user", "content": "that is dumb"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=40))
# -> "That is not smart"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-style-detox") works the same way.

System prompt

Rewrite the user's text in a neutral, non-toxic way while preserving meaning.

Use that exact prompt for in-distribution behavior. Chat template is Qwen's ChatML.

Smoke samples

Input Output
that is dumb That is not smart
you suck at this You are not good at this.
he is a fucking idiot He is not a smart person.
this code is shit This code is bad.
shut the hell up Shhh

Training

  • Base: Qwen/Qwen2.5-3B-Instruct (3.09 B params)
  • LoRA: rank 16, alpha 32, 16 layers, 6.65 M trainable params (0.216 %)
  • Optimizer: AdamW, lr 5e-5
  • Batch size: 2, max sequence 512
  • Iters: 1,200 (best val snapshot promoted; 3,000-iter run continued but val drifted up)
  • Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
  • Data: ParaDetox s-nlp/paradetox (18,993 toxic โ†’ neutral pairs), converted to ChatML SFT.

Limitations

  • ParaDetox-only training, so the model handles paraphrase-style detoxification well but is not a general formality / conciseness / simplification rewriter. The system prompt in the data is fixed to the detoxify instruction.
  • Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful rewrites.
  • English only.
  • The fine-tune may carry over latent toxicity from the base model in rare prompts. Pair with a content classifier if used in production.

Citation

@misc{qwen25_3b_style_detox,
  title  = {qwen2.5-3b-style-detox: LoRA-tuned Qwen2.5-3B for English detoxification},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-style-detox}
}

Underlying corpus: Logacheva, Dementieva et al. ParaDetox: Detoxification with Parallel Data, ACL 2022.

Downloads last month
15
Safetensors
Model size
3B params
Tensor type
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for amiya/qwen2.5-3b-style-detox

Base model

Qwen/Qwen2.5-3B
Adapter
(1,520)
this model