jonathanhe123/SemCoT-Sheared-LLaMA-1.3B-gsm8k

SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens

🚀 Overview

SemCoT is a novel framework designed to accelerate Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs). Instead of generating long, verbose textual explanations, SemCoT encodes reasoning steps within hidden representations (implicit reasoning). This approach significantly speeds up inference while maintaining high performance by ensuring semantic alignment between implicit tokens and ground-truth reasoning.

This specific checkpoint is fine-tuned from Sheared-LLaMA-1.3B using the GSM8K dataset.

• Paper: SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens
• Code: Official GitHub Repository

🎯 Key Features

• 🗣️ Semantic Alignment: Utilizes a contrastively trained sentence transformer to ensure that implicit reasoning remains semantically consistent with human-readable CoT explanations.
• ⚡ Efficiency Optimization: Introduces a lightweight implicit reasoning generator, fine-tuned via knowledge distillation, to reduce token generation time and enhance inference speed.
• 🧩 Joint Optimization: SemCoT is the first approach that enhances CoT efficiency by jointly optimizing token-level generation speed and preserving semantic alignment with ground-truth reasoning.

💻 Usage

Since this model requires custom classes to handle implicit reasoning tokens, please refer to the official GitHub repository for instructions on how to load and use the model.

# Example setup from the repo
git clone https://github.com/YinhanHe123/SemCoT.git
cd SemCoT
pip install -r requirements.txt

Citation

If you find this work useful, please cite the paper:

@inproceedings{he2025semcot,
  title={SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens},
  author={He, Yinhan and Zheng, Wendy and Zhu, Yaochen and Zheng, Zaiyi and Su, Lin and Vasudevan, Sriram and Guo, Qi and Hong, Liangjie and Li, Jundong},
  booktitle={39th Conference on Neural Information Processing Systems (NeurIPS 2025)},
  year={2025}
}