--- base_model: unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit library_name: transformers pipeline_tag: text-generation license: apache-2.0 language: - en tags: - text-generation-inference - transformers - unsloth - qwen2 - qlora - peft - sql - text-to-sql - code-generation datasets: - DanielRegaladoCardoso/text-to-sql-mix-v2 model-index: - name: sql-generator-qwen25-coder-7b-lora results: - task: type: text-generation name: Text-to-SQL Generation dataset: name: text-to-sql-mix-v2 type: DanielRegaladoCardoso/text-to-sql-mix-v2 metrics: - type: train_loss value: 0.2658 name: Final training loss --- # SQL Generator — Qwen2.5-Coder-7B (QLoRA) Fine-tuned **Qwen2.5-Coder-7B-Instruct** for **text-to-SQL** generation. Given a SQL schema and a natural-language question, the model produces a syntactically correct SQL query. Trained as part of the [SQL Agent LLMOps](https://github.com/DanielRegaladoUMiami/sql-agent-llmops) project — a multi-model SQL agent with deployment on HuggingFace Spaces. ## Model details | | | |---|---| | **Base model** | [unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit) | | **Architecture** | Qwen2 (7.6B params, 4-bit quantized base) | | **Fine-tuning method** | QLoRA via [Unsloth](https://github.com/unslothai/unsloth) + TRL | | **LoRA rank** | 16 | | **LoRA alpha** | 32 | | **Target modules** | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` | | **Trainable params** | ~70 M (~0.9% of base) | | **Language** | English | | **License** | Apache 2.0 | ## Training data [`DanielRegaladoCardoso/text-to-sql-mix-v2`](https://huggingface.co/datasets/DanielRegaladoCardoso/text-to-sql-mix-v2) — a curated mix of 5 public text-to-SQL datasets: - `b-mc2/sql-create-context` - `gretelai/synthetic_text_to_sql` - `knowrohit07/know_sql` - `NumbersStation/NSText2SQL` - `Clinton/Text-to-sql-v1` **Final training set**: 672,949 examples (after filtering sequences > 1024 tokens — kept 93.1% of original 723,097 rows). With sequence packing, this compressed to 154,462 effective sequences of length 1024. ## Training configuration | Hyperparameter | Value | |---|---| | Hardware | 1× NVIDIA L40S (48 GB) | | Epochs | 1 | | Batch size (per device) | 16 | | Gradient accumulation | 1 | | Effective batch size | 16 | | Max sequence length | 1024 | | Learning rate | 1e-4 | | LR scheduler | Cosine | | Warmup ratio | 0.03 | | Optimizer | adamw_8bit | | Precision | bf16 | | Sequence packing | Enabled | | Total steps | 9,654 | | Wall-clock time | 13.5 hours | | Final training loss | **0.2658** | ## Prompt format The model expects a chat-style prompt with a system message defining the SQL-expert role and a user message containing the schema and question: ``` <|im_start|>system You are a SQL expert. Given a SQL schema and a natural-language question, generate a correct SQL query answering the question. Return only the SQL. <|im_end|> <|im_start|>user ### Schema CREATE TABLE players (id INT, name VARCHAR, hometown VARCHAR); ### Question List all players from Tampa, Florida. <|im_end|> <|im_start|>assistant ``` ## Usage ### Option A — Load merged 16-bit model (recommended) ```python from transformers import AutoModelForCausalLM, AutoTokenizer REPO = "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora" model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="auto", device_map="auto") tokenizer = AutoTokenizer.from_pretrained(REPO) messages = [ {"role": "system", "content": "You are a SQL expert. Given a SQL schema and a natural-language question, generate a correct SQL query answering the question. Return only the SQL."}, {"role": "user", "content": "### Schema\nCREATE TABLE players (id INT, name VARCHAR, hometown VARCHAR);\n\n### Question\nList all players from Tampa, Florida."}, ] input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(input_ids, max_new_tokens=256, do_sample=False) print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True)) # → SELECT * FROM players WHERE hometown = 'Tampa, Florida' ``` ### Option B — Load LoRA adapter on top of base model Useful if you want to keep the base model in 4-bit (lower VRAM footprint). ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer import torch base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-Coder-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto", ) model = PeftModel.from_pretrained(base, "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora") tokenizer = AutoTokenizer.from_pretrained("DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora") ``` ### Option C — Inference with Unsloth (fastest) ```python from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( "DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora", max_seq_length=1024, load_in_4bit=True, ) FastLanguageModel.for_inference(model) ``` ## Files | File | Size | Purpose | |---|---|---| | `adapter_model.safetensors` | 161 MB | LoRA adapter weights | | `adapter_config.json` | 1 KB | LoRA configuration | | `model-0000{1..4}-of-00004.safetensors` | 15.2 GB | Merged 16-bit model | | `tokenizer.json` + `tokenizer_config.json` | 11 MB | Tokenizer | | `chat_template.jinja` | 4 KB | Qwen chat template | ## Limitations - **English only** — training data is English; performance on other languages is not validated. - **Sequence length cap** — examples requiring > 1024 tokens (large schemas, complex multi-CTE queries) were filtered out during training. The model may underperform on inputs above this length. - **No execution validation** — the model is trained to produce syntactically correct SQL, but generated queries are not guaranteed to execute or return correct results without manual review. Always sanity-check against your real database. - **Single dialect bias** — training data mixes multiple SQL dialects (SQLite, ANSI, MySQL); the model may produce queries that lean toward one dialect over another. ## Citation If you use this model, please cite the [SQL Agent LLMOps project](https://github.com/DanielRegaladoUMiami/sql-agent-llmops). ```bibtex @misc{regalado2026sqlagent, author = {Daniel Regalado Cardoso}, title = {SQL Generator: Qwen2.5-Coder-7B fine-tuned for text-to-SQL}, year = {2026}, howpublished = {\url{https://huggingface.co/DanielRegaladoCardoso/sql-generator-qwen25-coder-7b-lora}}, } ``` ## Acknowledgments - [Unsloth](https://github.com/unslothai/unsloth) — 2× faster QLoRA training - [TRL](https://github.com/huggingface/trl) — SFTTrainer - Qwen team — Qwen2.5-Coder-7B base model - All authors of the source datasets cited above [](https://github.com/unslothai/unsloth)