zhongweixie/inplace-ttt-models

In-Place TTT - Model Repository

This repository is prepared for In-Place Test-Time Training model checkpoints.

🎯 Purpose

This repository will host:

• Trained model checkpoints with In-Place TTT enabled
• Converted HuggingFace-compatible models
• Model configuration files
• Evaluation results

⚠️ Model Checkpoints Status

Important Notice: The trained model checkpoints are currently not available for download.

What Happened

The model checkpoints were trained and successfully evaluated in May 2026, but were subsequently removed from storage to free up disk space. Based on our investigation:

• ✅ Training completed successfully: May 15, 2026
• ✅ Models were evaluated on multiple benchmarks with excellent results
• ❌ Checkpoints deleted after: May 17, 2026

Trained Models (Evaluation Results Available)

These models were successfully trained and evaluated, but weights are no longer available:

1. qwen3-4b-prolong-32k-ttt (global_step_9000)

   • Base: Qwen3-4B
   • Training: ProLong 32k pretokenized dataset with In-Place TTT
   • Status: ❌ Checkpoint deleted

2. qwen3-4b-prolong-32k-baseline (global_step_9537)

   • Base: Qwen3-4B
   • Training: ProLong 32k pretokenized dataset (standard training)
   • Status: ❌ Checkpoint deleted

What IS Available ✅

Even though the model weights are not available, we provide comprehensive research artifacts:

Complete Evaluation Results

• Benchmark scores: MMLU, HellaSwag, PIQA, ARC, WinoGrande, etc.
• RULER long-context evaluations: 4k, 8k, 16k, 32k, 64k
• Detailed metrics in JSON format
• TTT vs Baseline comparison data
• Available at: results dataset

Full Training History

• WandB training logs (29 complete runs)
• Training curves, loss trajectories
• Learning rate schedules
• All hyperparameters and configurations

Complete Implementation

• Full source code for In-Place TTT
• Training scripts and configurations
• Evaluation pipelines and tools
• Reproducible setup instructions
• Available at: GitHub repository

Detailed Investigation Report

• See: CHECKPOINT_INVESTIGATION.md in the GitHub repository
• Timeline of events
• Evidence from logs and configurations

🔄 Reproducing the Training

You can reproduce the training using our complete code and configurations:

# Clone the repository
git clone https://github.com/zhongweixie/inplace_ttt.git
cd inplace_ttt

# Train TTT model
sbatch slurm_qwen3_prolong_pretok_32k_ttt.sh

# Train baseline model
sbatch slurm_qwen3_prolong_pretok_32k_baseline.sh

Training Requirements:

• Hardware: 8x H100 GPUs (or equivalent)
• Time: ~48 hours per model
• Storage: ~10GB per checkpoint

🔗 Related Resources

• Code Repository: https://github.com/zhongweixie/inplace_ttt
• Dataset Repository: https://huggingface.co/datasets/zhongweixie/inplace-ttt-results
• Paper: In-Place Test-Time Training (ICLR 2026 Oral)

🚀 Usage (When Checkpoints Are Available)

If you train your own models, you can use them with:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model with In-Place TTT
model = AutoModelForCausalLM.from_pretrained(
    "path/to/your/checkpoint",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "path/to/your/checkpoint"
)

# Inference with test-time training enabled
outputs = model.generate(
    inputs,
    max_length=128000,  # Long context support
    use_cache=True
)

📊 Model Features

• Drop-in TTT: No architectural side modules required
• LM-aligned updates: Optimized for autoregressive language modeling
• Chunk-wise processing: Efficient for long contexts
• HuggingFace compatible: Standard model loading and inference

📄 License

Apache 2.0 License

🎓 Citation

@inproceedings{feng2026inplace,
  title     = {In-Place Test-Time Training},
  author    = {Feng, Guhao and Luo, Shengjie and Hua, Kai and Zhang, Ge and Huang, Wenhao and He, Di and Cai, Tianle},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  note      = {Oral Presentation},
  url       = {https://arxiv.org/abs/2604.06169}
}

📞 Contact

For questions or issues, please open an issue in the GitHub repository.

---

Status: Checkpoints not available - see status section above
Evaluation Results: ✅ Available in dataset repository
Source Code: ✅ Available in GitHub
Generated: 2026-09-04
Project: In-Place Test-Time Training
Organization: ByteDance Seed Team