File size: 3,452 Bytes
7981ede 7465501 ff46b59 7981ede 773efb1 8613202 773efb1 290e0ef 773efb1 76b6ccf 773efb1 290e0ef 773efb1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | ---
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
---
<div align="center">
<img src="https://huggingface.co/bharatgenai/Param-1-2.9B-Instruct/resolve/main/BharatGen%20Logo%20(1).png" width="60%" alt="BharatGen" />
</div>
<hr>
<div align="center">
<a href="https://arxiv.org/abs/2507.13390" target="_blank" style="margin: 4px;">
<img alt="Paper" src="https://img.shields.io/badge/%20Paper-arxiv-0033ad?style=flat&logo=arxiv&logoColor=white" />
</a>
<a href="https://huggingface.co/bharatgenai/Param-1-2.9B-Instruct/blob/main/LICENSE" target="_blank" style="margin: 4px;">
<img alt="License" src="https://img.shields.io/badge/License-yellow.svg" />
</a>
</div>
# Param-1
**BharatGen** introduces **Param-1**, a bilingual language model pretrained from scratch on English and Hindi. With 2.9 billion parameters, it serves as a powerful foundational model for text completion.
**Param-1** outperforms leading models like **LLaMA-3.2B**, **Gemma-2B**, **Granite-2B**, and **Granite-3B** on various standard benchmarks.
This early release is equipped with inference support via **NVIDIA NeMo**.
---
## 🚀 Model Inference
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# Load tokenizer and model
model_name = "bharatgenai/Param-1"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
device_map="auto"
)
prompt = "Your prompt here."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# --- Generate output ---
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=300,
do_sample=True,
top_k=50,
top_p=0.95,
temperature=0.6,
eos_token_id=tokenizer.eos_token_id,
use_cache=False
)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print("Generated Text:\n", generated_text)
```
---
## 📊 Benchmarks
| Task | **Param-1 (PT)** |
|------|----------------------------|
| ARC Challenge | 53.6 (few) |
| ARC Easy | 74.2 (few) |
| HellaSwag | 73.8 (few) |
| HellaSwag Hi | 43.1 (few) |
| MMLU En | 46.2 (few) |
| MMLU Hi | 34.6 (few) |
| TriviaQA | 42.8 |
| TruthfulQA - Gen (BLEU) | 37.3 |
| TruthfulQA - MC1 Acc | 28.4 |
| TruthfulQA - MC2 Acc | 42.9 |
| PIQA | 79.2 |
| SuperGLUE - WiC | 50.6 |
| SuperGLUE - WSC | 52.9 |
| SuperGLUE - boolq | 72.6 |
| SuperGLUE - rte | 66.8 |
> **Notes:**
> - **PT**: Pre-Trained
> - **en-hi**: English-Hindi
> - Pre-trained on **5 Trillion tokens**
---
## 🧠 Model Architecture
- Hidden size: 2048
- Intermediate size: 7168
- Number of attention heads: 16
- Number of hidden layers: 32
- Number of key-value heads: 8
- Maximum position embeddings: 2048
- Activation function: **SiLU**
- Positional embeddings: **Rotary (RoPE)** with `rope_theta=10000.0`
- Attention: **Grouped-query attention**
- Precision: **bf16-mixed**
---
## 🏗️ Training Details
- **Training Infrastructure**: Yotta’s Shakti Cloud
- **Hardware**: NVIDIA H100 – 512 GPUs
- **Framework**: NVIDIA NeMo
--- |