README.md

---
base_model: unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
- qlora
- peft
- sql
- text-to-sql
- code-generation
datasets:
- DanielRegaladoCardoso/text-to-sql-mix-v2
model-index:
- name: sql-generator-qwen25-coder-7b-lora
  results:
  - task:
      type: text-generation
      name: Text-to-SQL Generation
    dataset:
      name: text-to-sql-mix-v2
      type: DanielRegaladoCardoso/text-to-sql-mix-v2
    metrics:
    - type: train_loss
      value: 0.2658
      name: Final training loss
---

# SQL Generator — Qwen2.5-Coder-7B (QLoRA)

Fine-tuned **Qwen2.5-Coder-7B-Instruct** for **text-to-SQL** generation. Given a SQL schema and a natural-language question, the model produces a syntactically correct SQL query.

Trained as part of the [SQL Agent LLMOps](https://github.com/DanielRegaladoUMiami/sql-agent-llmops) project — a multi-model SQL agent with deployment on HuggingFace Spaces.

## Model details

| | |
|---|---|
| **Base model** | [unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit) |
| **Architecture** | Qwen2 (7.6B params, 4-bit quantized base) |
| **Fine-tuning method** | QLoRA via [Unsloth](https://github.com/unslothai/unsloth) + TRL |
| **LoRA rank** | 16 |
| **LoRA alpha** | 32 |
| **Target modules** | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
| **Trainable params** | ~70 M (~0.9% of base) |
| **Language** | English |
| **License** | Apache 2.0 |

## Training data

[`DanielRegaladoCardoso/text-to-sql-mix-v2`](https://huggingface.co/datasets/DanielRegaladoCardoso/text-to-sql-mix-v2) — a curated mix of 5 public text-to-SQL datasets:

- `b-mc2/sql-create-context`
- `gretelai/synthetic_text_to_sql`
- `knowrohit07/know_sql`
- `NumbersStation/NSText2SQL`
- `Clinton/Text-to-sql-v1`

**Final training set**: 672,949 examples (after filtering sequences > 1024 tokens — kept 93.1% of original 723,097 rows). With sequence packing, this compressed to 154,462 effective sequences of length 1024.

## Training configuration

| Hyperparameter | Value |
|---|---|
| Hardware | 1× NVIDIA L40S (48 GB) |
| Epochs | 1 |
| Batch size (per device) | 16 |
| Gradient accumulation | 1 |
| Effective batch size | 16 |
| Max sequence length | 1024 |
| Learning rate | 1e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 0.03 |
| Optimizer | adamw_8bit |
| Precision | bf16 |
| Sequence packing | Enabled |
| Total steps | 9,654 |
| Wall-clock time | 13.5 hours |
| Final training loss | **0.2658** |

## Prompt format

The model expects a chat-style prompt with a system message defining the SQL-expert role and a user message containing the schema and question:

```
<|im_start|>system
You are a SQL expert. Given a SQL schema and a natural-language question, generate a correct SQL query answering the question. Return only the SQL.
<|im_end|>
<|im_start|>user
### Schema
CREATE TABLE players (id INT, name VARCHAR, hometown VARCHAR);

### Question
List all players from Tampa, Florida.
<|im_end|>
<|im_start|>assistant
```

## Usage

### Option A — Load merged 16-bit model (recommended)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora"

model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(REPO)

messages = [
    {"role": "system", "content": "You are a SQL expert. Given a SQL schema and a natural-language question, generate a correct SQL query answering the question. Return only the SQL."},
    {"role": "user", "content": "### Schema\nCREATE TABLE players (id INT, name VARCHAR, hometown VARCHAR);\n\n### Question\nList all players from Tampa, Florida."},
]

input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(input_ids, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))
# → SELECT * FROM players WHERE hometown = 'Tampa, Florida'
```

### Option B — Load LoRA adapter on top of base model

Useful if you want to keep the base model in 4-bit (lower VRAM footprint).

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora")
tokenizer = AutoTokenizer.from_pretrained("DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora")
```

### Option C — Inference with Unsloth (fastest)

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora",
    max_seq_length=1024,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)
```

## Files

| File | Size | Purpose |
|---|---|---|
| `adapter_model.safetensors` | 161 MB | LoRA adapter weights |
| `adapter_config.json` | 1 KB | LoRA configuration |
| `model-0000{1..4}-of-00004.safetensors` | 15.2 GB | Merged 16-bit model |
| `tokenizer.json` + `tokenizer_config.json` | 11 MB | Tokenizer |
| `chat_template.jinja` | 4 KB | Qwen chat template |

## Limitations

- **English only** — training data is English; performance on other languages is not validated.
- **Sequence length cap** — examples requiring > 1024 tokens (large schemas, complex multi-CTE queries) were filtered out during training. The model may underperform on inputs above this length.
- **No execution validation** — the model is trained to produce syntactically correct SQL, but generated queries are not guaranteed to execute or return correct results without manual review. Always sanity-check against your real database.
- **Single dialect bias** — training data mixes multiple SQL dialects (SQLite, ANSI, MySQL); the model may produce queries that lean toward one dialect over another.

## Citation

If you use this model, please cite the [SQL Agent LLMOps project](https://github.com/DanielRegaladoUMiami/sql-agent-llmops).

```bibtex
@misc{regalado2026sqlagent,
  author       = {Daniel Regalado Cardoso},
  title        = {SQL Generator: Qwen2.5-Coder-7B fine-tuned for text-to-SQL},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora}},
}
```

## Acknowledgments

- [Unsloth](https://github.com/unslothai/unsloth) — 2× faster QLoRA training
- [TRL](https://github.com/huggingface/trl) — SFTTrainer
- Qwen team — Qwen2.5-Coder-7B base model
- All authors of the source datasets cited above

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)