tries to improve the generation paramaters

README_SMOLLM3_FEATURES.md

@@ -0,0 +1,227 @@
+# SmolLM3 Features Implementation
+
+This document describes the SmolLM3 features implemented in the Petite Elle L'Aime 3 chat interface.
+
+## 🎯 SmolLM3 Features
+
+### 1. Thinking Mode
+
+SmolLM3 supports extended thinking mode with reasoning traces. The implementation includes:
+
+- **Automatic thinking flags**: System prompts automatically get `/think` or `/no_think` flags
+- **Manual control**: Users can manually add thinking flags to system prompts
+- **UI toggle**: Checkbox to enable/disable thinking mode
+- **Response cleaning**: Thinking tags are properly cleaned from responses
+
+#### Usage Examples:
+
+```python
+# With thinking enabled (default)
+system_prompt = "Tu es TonicIA, un assistant francophone rigoureux et bienveillant./think"
+
+# With thinking disabled
+system_prompt = "Tu es TonicIA, un assistant francophone rigoureux et bienveillant./no_think"
+
+# Manual control in UI
+enable_thinking = True  # or False
+```
+
+### 2. Tool Calling
+
+SmolLM3 supports both XML and Python tool calling formats:
+
+#### XML Tools (Default)
+```json
+[
+  {
+    "name": "get_weather",
+    "description": "Get the weather in a city",
+    "parameters": {
+      "type": "object",
+      "properties": {
+        "city": {
+          "type": "string",
+          "description": "The city to get the weather for"
+        }
+      }
+    }
+  }
+]
+```
+
+#### Python Tools
+```python
+# Tools are called as Python functions in <code> tags
+# Example: <code>get_weather(city="Paris")</code>
+```
+
+### 3. Generation Parameters
+
+Following SmolLM3 recommendations:
+
+- **Temperature**: 0.6 (recommended default)
+- **Top-p**: 0.95 (recommended default)
+- **Repetition Penalty**: 1.1 (recommended default)
+- **Max tokens**: 2048 (configurable up to 32,768)
+- **Context length**: Up to 65,536 tokens (extensible to 128k/256k with YaRN)
+
+### 4. Long Context Processing
+
+The model supports:
+- **Base context**: 65,536 tokens
+- **Extended context**: Up to 256k tokens with YaRN scaling
+- **YaRN configuration**: Available for longer inputs
+
+```json
+{
+  "rope_scaling": {
+    "factor": 2.0,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  }
+}
+```
+
+## 🔧 Implementation Details
+
+### Chat Template Integration
+
+The app uses SmolLM3's chat template with proper thinking and tool calling:
+
+```python
+def create_prompt(system_message, user_message, enable_thinking=True, tools=None, use_xml_tools=True):
+    formatted_messages = []
+    
+    # Handle thinking flags
+    if system_message and system_message.strip():
+        has_think_flag = "/think" in system_message
+        has_no_think_flag = "/no_think" in system_message
+        
+        if not enable_thinking and not has_no_think_flag:
+            system_message += "/no_think"
+        elif enable_thinking and not has_think_flag and not has_no_think_flag:
+            system_message += "/think"
+        formatted_messages.append({"role": "system", "content": system_message})
+    
+    formatted_messages.append({"role": "user", "content": user_message})
+    
+    # Apply chat template with SmolLM3 features
+    template_kwargs = {
+        "tokenize": False,
+        "add_generation_prompt": True,
+        "enable_thinking": enable_thinking
+    }
+    
+    # Add tool calling
+    if tools and len(tools) > 0:
+        if use_xml_tools:
+            template_kwargs["xml_tools"] = tools
+        else:
+            template_kwargs["python_tools"] = tools
+    
+    return tokenizer.apply_chat_template(formatted_messages, **template_kwargs)
+```
+
+### Tool Call Detection
+
+The app detects and formats tool calls in responses:
+
+```python
+# Handle tool calls if present
+if parsed_tools and ("<tool_call>" in assistant_response or "<code>" in assistant_response):
+    if "<tool_call>" in assistant_response:
+        tool_call_match = re.search(r'<tool_call>(.*?)</tool_call>', assistant_response, re.DOTALL)
+        if tool_call_match:
+            tool_call = tool_call_match.group(1)
+            assistant_response += f"\n\n🔧 Tool Call Detected: {tool_call}\n\nNote: This is a simulated tool call."
+    elif "<code>" in assistant_response:
+        code_match = re.search(r'<code>(.*?)</code>', assistant_response, re.DOTALL)
+        if code_match:
+            code_call = code_match.group(1)
+            assistant_response += f"\n\n🐍 Python Tool Call: {code_call}\n\nNote: This is a simulated Python tool call."
+```
+
+## 🎮 UI Features
+
+### Advanced Settings Panel
+
+- **Temperature slider**: 0.01 to 1.0 (default: 0.6)
+- **Top-p slider**: 0.1 to 1.0 (default: 0.95)
+- **Repetition Penalty slider**: 1.0 to 2.0 (default: 1.1)
+- **Max length slider**: 10 to 32,768 tokens (default: 2048)
+- **Thinking mode checkbox**: Enable/disable reasoning traces
+- **Tool calling checkbox**: Enable/disable function calling
+- **XML vs Python tools**: Choose tool calling format
+- **Tool definition editor**: JSON editor for custom tools
+
+### Default Tool Set
+
+The app includes two default tools for demonstration:
+
+1. **get_weather**: Get weather information for a city
+2. **calculate**: Perform mathematical calculations
+
+## 🚀 Usage Examples
+
+### Basic Chat with Thinking
+```python
+system_prompt = "Tu es TonicIA, un assistant francophone rigoureux et bienveillant./think"
+user_message = "Explique-moi la gravité en termes simples."
+```
+
+### Chat with Tool Calling
+```python
+tools = [
+    {
+        "name": "get_weather",
+        "description": "Get the weather in a city",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "city": {"type": "string", "description": "The city name"}
+            }
+        }
+    }
+]
+
+user_message = "Quel temps fait-il à Paris aujourd'hui?"
+```
+
+### Agentic Usage
+```python
+# The model can call tools automatically based on user requests
+# Example: "Calculate 15 * 23" will trigger the calculate tool
+# Example: "What's the weather in London?" will trigger the get_weather tool
+```
+
+## 📋 Requirements
+
+- **Transformers**: v4.53.0+ (required for SmolLM3 support)
+- **PyTorch**: Latest version
+- **Gradio**: For the web interface
+- **Hugging Face Spaces**: For deployment
+
+## 🔄 Migration from Previous Version
+
+The updated app includes:
+
+1. **SmolLM3-compatible generation parameters**
+2. **Thinking mode with proper flag handling**
+3. **Tool calling support (XML and Python)**
+4. **Extended context support**
+5. **Improved response cleaning**
+
+## 🎯 Best Practices
+
+1. **Use recommended parameters**: temperature=0.6, top_p=0.95, repetition_penalty=1.1
+2. **Enable thinking for complex reasoning tasks**
+3. **Use tool calling for structured tasks**
+4. **Keep context within limits**: 65k tokens base, 256k with YaRN
+5. **Test tool definitions before deployment**
+6. **Adjust repetition penalty**: Use 1.0-1.2 for creative tasks, 1.1-1.3 for factual responses
+
+## 🔗 References
+
+- [SmolLM3 Model Card](https://huggingface.co/HuggingFaceTB/SmolLM3-3B)
+- [SmolLM3 Documentation](https://huggingface.co/docs/transformers/model_doc/smol-lm-3)
+- [Tool Calling Guide](https://huggingface.co/docs/transformers/chat_templating#tool-use) 

README_TORCHAO.md

@@ -1,46 +1,39 @@
-# TorchAO Quantization Implementation
+# Pre-Quantized Model Implementation
 
-This project now uses **TorchAO** for proper quantization and inference. TorchAO is PyTorch's architecture optimization library that provides high-performance quantization techniques.
+This project uses a **pre-quantized int4 model** for efficient deployment. The model is already quantized and stored in the `int4` subfolder, so we don't need to apply additional quantization during loading.
 
 ## Key Changes Made
 
-### 1. Proper TorchAO Configuration
+### 1. Loading Pre-Quantized Model
 
-The app now uses the correct TorchAO quantization configurations:
+The app now correctly loads the pre-quantized model without trying to re-quantize it:
 
 ```python
-from transformers import TorchAoConfig
-from torchao.quantization import Int4WeightOnlyConfig, Int8WeightOnlyConfig
-from torchao.dtypes import Int4CPULayout
-
-def get_quantization_config():
-    if DEVICE == "cuda":
-        # For CUDA, use Int8WeightOnlyConfig for better performance
-        quant_config = Int8WeightOnlyConfig(group_size=128)
-    else:
-        # For CPU, use Int4WeightOnlyConfig with CPU layout
-        quant_config = Int4WeightOnlyConfig(group_size=128, layout=Int4CPULayout())
+def load_model():
+    """Load the pre-quantized model and tokenizer"""
+    global model, tokenizer
     
-    return TorchAoConfig(quant_type=quant_config)
-```
-
-### 2. Correct Model Loading
-
-The model is now loaded with proper TorchAO quantization:
-
-```python
-quantization_config = get_quantization_config()
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    quantization_config=quantization_config,
-    device_map="auto" if device == "cuda" else "cpu",
-    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
-    trust_remote_code=True,
-    low_cpu_mem_usage=True,
-)
+    try:
+        # Load tokenizer from int4 subfolder
+        tokenizer = AutoTokenizer.from_pretrained(MAIN_MODEL_ID, subfolder="int4")
+        
+        # Load pre-quantized model without additional quantization config
+        model_kwargs = {
+            "device_map": "auto" if DEVICE == "cuda" else "cpu",
+            "torch_dtype": torch.float32,  # Use float32 for compatibility
+            "trust_remote_code": True,
+            "low_cpu_mem_usage": True,
+        }
+        
+        model = AutoModelForCausalLM.from_pretrained(MAIN_MODEL_ID, subfolder="int4", **model_kwargs)
+        
+        return True
+    except Exception as e:
+        logger.error(f"Error loading model: {e}")
+        return False
 ```
 
-### 3. Proper Inference with Cache Implementation
+### 2. Proper Inference with Cache Implementation
 
 The most important fix is using `cache_implementation="static"` for generation:
 
@@ -54,108 +47,92 @@ output_ids = model.generate(
     attention_mask=inputs['attention_mask'],
     pad_token_id=tokenizer.eos_token_id,
     eos_token_id=tokenizer.eos_token_id,
-    cache_implementation="static"  # CRITICAL for TorchAO quantized models
+    cache_implementation="static"  # CRITICAL for quantized models
 )
 ```
 
-## TorchAO Quantization Types
+## Why This Approach Works
 
-### For CUDA (GPU)
-- **Int8WeightOnlyConfig**: Best performance for most use cases
-- **Int8DynamicActivationInt8WeightConfig**: For more aggressive quantization
-- **GemliteUIntXWeightOnlyConfig**: Optimized for H100/A100 GPUs
+### Avoiding Quantization Conflicts
+
+The warning you saw:
+```
+You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.
+```
 
-### For CPU
-- **Int4WeightOnlyConfig with Int4CPULayout**: Optimized for CPU deployment
-- **Int8WeightOnlyConfig**: Alternative for better compatibility
+This happens because:
+1. Your model in the `int4` subfolder is already quantized
+2. When you try to apply TorchAO quantization to an already quantized model, it conflicts
+3. The solution is to load the pre-quantized model directly without additional quantization
 
-### For Sparsity (Advanced)
-- **Int4WeightOnlyConfig with MarlinSparseLayout**: For 2:4 sparsity
+### Benefits of Pre-Quantized Models
+
+1. **No Quantization Overhead**: The model is already optimized
+2. **Consistent Performance**: No runtime quantization variations
+3. **Memory Efficient**: Already compressed for deployment
+4. **Faster Loading**: No quantization step during loading
 
 ## Testing the Implementation
 
-Run the test script to verify TorchAO quantization is working:
+Run the test script to verify the pre-quantized model works:
 
 ```bash
-python test_torchao_inference.py
+python test_pre_quantized_model.py
 ```
 
 This will test:
-- Model loading with TorchAO quantization
+- Loading the pre-quantized model without conflicts
 - Text generation with proper cache implementation
-- Memory usage optimization
+- Verification of quantization status
 
 ## Performance Benefits
 
-1. **Memory Reduction**: Up to 50% memory reduction with Int4 quantization
-2. **Faster Inference**: Optimized kernels for quantized operations
-3. **Better Compatibility**: Works with torch.compile for additional speedup
-4. **Device Optimization**: Different configs for CUDA vs CPU
+1. **Memory Reduction**: Pre-quantized models use ~50% less memory
+2. **Faster Loading**: No quantization step during model loading
+3. **Consistent Performance**: No quantization variations between runs
+4. **Optimized Kernels**: Pre-quantized models use optimized inference kernels
 
 ## Common Issues and Solutions
 
+### Issue: Quantization config warning
+**Solution**: Don't apply additional quantization to pre-quantized models
+
 ### Issue: Model outputs incorrect or garbled text
 **Solution**: Ensure `cache_implementation="static"` is used in generation
 
 ### Issue: Memory errors during loading
-**Solution**: Use appropriate quantization config for your device (Int4 for CPU, Int8 for CUDA)
+**Solution**: Use `low_cpu_mem_usage=True` and appropriate device mapping
 
 ### Issue: Slow inference
 **Solution**: 
 1. Use `cache_implementation="static"`
 2. Consider using `torch.compile` for additional speedup
-3. Use appropriate group_size (128 is usually optimal)
-
-## Advanced Configuration
+3. Monitor memory usage
 
-### Per-Module Quantization
-
-You can quantize different layers with different configs:
-
-```python
-from torchao.quantization import ModuleFqnToConfig
+## Model Structure
 
-# Skip quantization for certain layers
-config = ModuleFqnToConfig({
-    "_default": Int4WeightOnlyConfig(group_size=128),
-    "model.layers.0.self_attn.q_proj": None  # Skip this layer
-})
+Your model repository should have this structure:
 ```
-
-### Autoquantization
-
-For automatic quantization selection:
-
-```python
-quantization_config = TorchAoConfig("autoquant", min_sqnr=None)
-# After loading, call:
-model.finalize_autoquant()
-```
-
-## Requirements
-
-Make sure you have the latest TorchAO version:
-
-```bash
-pip install torchao>=0.10.0
+Tonic/petite-elle-L-aime-3-sft/
+├── int4/
+│   ├── config.json
+│   ├── pytorch_model.bin
+│   ├── tokenizer.json
+│   └── ...
+├── README.md
+└── ...
 ```
 
 ## Deployment Notes
 
-1. **Serialization**: TorchAO models should be saved with `safe_serialization=False`
-2. **Device Compatibility**: Int4 models are device-specific, Int8 models are portable
-3. **Memory**: Monitor memory usage during deployment
-4. **Performance**: Use `cache_implementation="static"` for best performance
+1. **No Additional Quantization**: The model is already quantized
+2. **Cache Implementation**: Always use `cache_implementation="static"`
+3. **Memory Monitoring**: Pre-quantized models use less memory
+4. **Performance**: Optimized for deployment without quantization overhead
 
 ## Troubleshooting
 
-### Check TorchAO Version
-```python
-import torchao
-print(torchao.__version__)
-```
-
-### Verify Quantization
+### Check Model Quantization
 ```python
 # Check if model is quantized
 for name, module in model.named_modules():
@@ -169,4 +146,34 @@ import torch
 print(f"GPU Memory: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
 ```
 
-This implementation ensures proper TorchAO quantization for both loading and inference, with the critical `cache_implementation="static"` parameter for correct generation. 
+### Verify Model Loading
+```python
+# Check model config
+print(f"Model dtype: {model.dtype}")
+print(f"Model device: {model.device}")
+```
+
+## Alternative: TorchAO Quantization
+
+If you want to use TorchAO quantization instead of pre-quantized models:
+
+1. **Load the base model** (not from int4 subfolder)
+2. **Apply TorchAO quantization** during loading
+3. **Use appropriate quantization configs** for your device
+
+```python
+from transformers import TorchAoConfig
+from torchao.quantization import Int4WeightOnlyConfig
+
+quant_config = Int4WeightOnlyConfig(group_size=128)
+quantization_config = TorchAoConfig(quant_type=quant_config)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,  # Not subfolder="int4"
+    quantization_config=quantization_config,
+    device_map="auto",
+    torch_dtype=torch.float32,
+)
+```
+
+This implementation ensures proper handling of pre-quantized models without quantization conflicts, with the critical `cache_implementation="static"` parameter for correct generation. 

app.py

@@ -1,8 +1,6 @@
 import gradio as gr
 import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
-from torchao.quantization import Int4WeightOnlyConfig, Int8WeightOnlyConfig, Int8DynamicActivationInt8WeightConfig
-from torchao.dtypes import Int4CPULayout
+from transformers import AutoModelForCausalLM, AutoTokenizer
 import re
 import json
 from typing import List, Dict, Any, Optional
@@ -23,27 +21,59 @@ model = None
 tokenizer = None
 DEFAULT_SYSTEM_PROMPT = "Tu es TonicIA, un assistant francophone rigoureux et bienveillant."
 title = "# 🤖 Petite Elle L'Aime 3 - Chat Interface"
-description = "A fine-tuned version of SmolLM3-3B optimized for French conversations. This is the torchao quantized version for efficient deployment."
+description = "A fine-tuned version of SmolLM3-3B optimized for French conversations. This is the pre-quantized int4 version for efficient deployment."
 presentation1 = """
 ### 🎯 Features
 - **Multilingual Support**: English, French, Italian, Portuguese, Chinese, Arabic
-- **TorchAO Quantization**: Optimized for deployment with memory reduction
+- **Pre-Quantized Int4**: Optimized for deployment with memory reduction
 - **Interactive Chat Interface**: Real-time conversation with the model
 - **Customizable System Prompt**: Define the assistant's personality and behavior
 - **Thinking Mode**: Enable reasoning mode with thinking tags
+- **Tool Calling**: Support for function calling with XML and Python tools
 """
 presentation2 = """### 🎯 Fonctionnalités
 * **Support multilingue** : Anglais, Français, Italien, Portugais, Chinois, Arabe
-* **Quantification TorchAO** : Optimisé pour un déploiement avec réduction de mémoire
+* **Pré-quantifié Int4** : Optimisé pour un déploiement avec réduction de mémoire
 * **Interface de chat interactive** : Conversation en temps réel avec le modèle
 * **Invite système personnalisable** : Définissez la personnalité et le comportement de l'assistant
 * **Mode Réflexion** : Activez le mode raisonnement avec des balises de réflexion
+* **Appel d'outils** : Support pour l'appel de fonctions avec XML et Python
 """
 joinus = """
 ## Join us :
 🌟TeamTonic🌟 is always making cool demos! Join our active builder's 🛠️community 👻 [![Join us on Discord](https://img.shields.io/discord/1109943800132010065?label=Discord&logo=discord&style=flat-square)](https://discord.gg/qdfnvSPcqP) On 🤗Huggingface:[MultiTransformer](https://huggingface.co/MultiTransformer) On 🌐Github: [Tonic-AI](https://github.com/tonic-ai) & contribute to🌟 [Build Tonic](https://git.tonic-ai.com/contribute)🤗Big thanks to Yuvi Sharma and all the folks at huggingface for the community grant 🤗
 """
 
+# Default tool definition for demonstration
+DEFAULT_TOOLS = [
+    {
+        "name": "get_weather",
+        "description": "Get the weather in a city",
+        "parameters": {
+            "type": "object", 
+            "properties": {
+                "city": {
+                    "type": "string", 
+                    "description": "The city to get the weather for"
+                }
+            }
+        }
+    },
+    {
+        "name": "calculate",
+        "description": "Perform mathematical calculations",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "expression": {
+                    "type": "string",
+                    "description": "Mathematical expression to evaluate"
+                }
+            }
+        }
+    }
+]
+
 
 def download_chat_template():
     """Download the chat template from the main repository"""
@@ -66,20 +96,8 @@ def download_chat_template():
         return None
 
 
-def get_quantization_config():
-    """Get the appropriate quantization config based on device"""
-    if DEVICE == "cuda":
-        # For CUDA, use Int8WeightOnlyConfig for better performance
-        quant_config = Int8WeightOnlyConfig(group_size=128)
-    else:
-        # For CPU, use Int4WeightOnlyConfig with CPU layout
-        quant_config = Int4WeightOnlyConfig(group_size=128, layout=Int4CPULayout())
-    
-    return TorchAoConfig(quant_type=quant_config)
-
-
 def load_model():
-    """Load the model and tokenizer with torchao quantization"""
+    """Load the pre-quantized model and tokenizer"""
     global model, tokenizer
     
     try:
@@ -90,18 +108,14 @@ def load_model():
             tokenizer.chat_template = chat_template
         logger.info("Chat template downloaded and set successfully")
 
-        logger.info(f"Loading model with torchao quantization from {MAIN_MODEL_ID}")
-        
-        # Get quantization config
-        quantization_config = get_quantization_config()
-        logger.info(f"Using quantization config: {quantization_config}")
+        logger.info(f"Loading pre-quantized int4 model from {MAIN_MODEL_ID}/int4")
         
+        # Load the pre-quantized model without additional quantization config
         model_kwargs = {
             "device_map": "auto" if DEVICE == "cuda" else "cpu",
-            "torch_dtype": torch.bfloat16 if DEVICE == "cuda" else torch.float32,
+            "torch_dtype": torch.float32,  # Use float32 for compatibility
             "trust_remote_code": True,
             "low_cpu_mem_usage": True,
-            "quantization_config": quantization_config,
         }
         
         logger.info(f"Model loading parameters: {model_kwargs}")
@@ -110,7 +124,7 @@ def load_model():
         if tokenizer.pad_token_id is None:
             tokenizer.pad_token_id = tokenizer.eos_token_id
             
-        logger.info("Model loaded successfully with torchao quantization")
+        logger.info("Pre-quantized model loaded successfully")
         return True
         
     except Exception as e:
@@ -119,21 +133,39 @@ def load_model():
         return False
 
 
-def create_prompt(system_message, user_message, enable_thinking=True):
-    """Create prompt using the model's chat template"""
+def create_prompt(system_message, user_message, enable_thinking=True, tools=None, use_xml_tools=True):
+    """Create prompt using the model's chat template with SmolLM3 features"""
     try:
         formatted_messages = []
         if system_message and system_message.strip():
+            # Check if thinking flags are already present
+            has_think_flag = "/think" in system_message
+            has_no_think_flag = "/no_think" in system_message
+            
+            # Add thinking flag to system message if needed
+            if not enable_thinking and not has_no_think_flag:
+                system_message += "/no_think"
+            elif enable_thinking and not has_think_flag and not has_no_think_flag:
+                system_message += "/think"
             formatted_messages.append({"role": "system", "content": system_message})
-        formatted_messages.append({"role": "user", "content": user_message})        
-        prompt = tokenizer.apply_chat_template(
-            formatted_messages,
-            tokenize=False,
-            add_generation_prompt=True,
-            enable_thinking=enable_thinking
-        )        
-        if not enable_thinking:
-            prompt += " /no_think"
+        
+        formatted_messages.append({"role": "user", "content": user_message})
+        
+        # Apply chat template with SmolLM3 features
+        template_kwargs = {
+            "tokenize": False,
+            "add_generation_prompt": True,
+            "enable_thinking": enable_thinking
+        }
+        
+        # Add tool calling if tools are provided
+        if tools and len(tools) > 0:
+            if use_xml_tools:
+                template_kwargs["xml_tools"] = tools
+            else:
+                template_kwargs["python_tools"] = tools
+        
+        prompt = tokenizer.apply_chat_template(formatted_messages, **template_kwargs)
         
         return prompt
         
@@ -142,14 +174,23 @@ def create_prompt(system_message, user_message, enable_thinking=True):
         return ""
 
 @spaces.GPU(duration=94)
-def generate_response(message, history, system_message, max_tokens, temperature, top_p, do_sample, enable_thinking=True):
-    """Generate response using the torchao quantized model"""
+def generate_response(message, history, system_message, max_tokens, temperature, top_p, repetition_penalty, do_sample, enable_thinking=True, tools=None, use_xml_tools=True):
+    """Generate response using the pre-quantized model with SmolLM3 features"""
     global model, tokenizer
     
     if model is None or tokenizer is None:
         return "Error: Model not loaded. Please wait for the model to load."
     
-    full_prompt = create_prompt(system_message, message, enable_thinking)
+    # Parse tools from string if provided
+    parsed_tools = None
+    if tools and tools.strip():
+        try:
+            parsed_tools = json.loads(tools)
+        except json.JSONDecodeError as e:
+            logger.error(f"Error parsing tools JSON: {e}")
+            return "Error: Invalid tool definition JSON format."
+    
+    full_prompt = create_prompt(system_message, message, enable_thinking, parsed_tools, use_xml_tools)
      
     if not full_prompt:
         return "Error: Failed to create prompt."
@@ -166,6 +207,7 @@ def generate_response(message, history, system_message, max_tokens, temperature,
             max_new_tokens=max_tokens,
             temperature=temperature,
             top_p=top_p,
+            repetition_penalty=repetition_penalty,
             do_sample=do_sample,
             attention_mask=inputs['attention_mask'],
             pad_token_id=tokenizer.eos_token_id,
@@ -178,6 +220,19 @@ def generate_response(message, history, system_message, max_tokens, temperature,
         if not enable_thinking:
             assistant_response = re.sub(r'<think>.*?</think>', '', assistant_response, flags=re.DOTALL)
         
+        # Handle tool calls if present
+        if parsed_tools and ("<tool_call>" in assistant_response or "<code>" in assistant_response):
+            if "<tool_call>" in assistant_response:
+                tool_call_match = re.search(r'<tool_call>(.*?)</tool_call>', assistant_response, re.DOTALL)
+                if tool_call_match:
+                    tool_call = tool_call_match.group(1)
+                    assistant_response += f"\n\n🔧 Tool Call Detected: {tool_call}\n\nNote: This is a simulated tool call. In a real scenario, the tool would be executed and its output would be used to generate a final response."
+            elif "<code>" in assistant_response:
+                code_match = re.search(r'<code>(.*?)</code>', assistant_response, re.DOTALL)
+                if code_match:
+                    code_call = code_match.group(1)
+                    assistant_response += f"\n\n🐍 Python Tool Call: {code_call}\n\nNote: This is a simulated Python tool call. In a real scenario, the function would be executed and its output would be used to generate a final response."
+        
         assistant_response = assistant_response.strip()
         
         return assistant_response
@@ -188,14 +243,20 @@ def user(user_message, history):
         history = []
     return "", history + [{"role": "user", "content": user_message}]
 
-def bot(history, system_prompt, max_length, temperature, top_p, advanced_checkbox, enable_thinking):
+def bot(history, system_prompt, max_length, temperature, top_p, repetition_penalty, advanced_checkbox, enable_thinking, tools, use_xml_tools, use_tools):
     """Generate bot response"""
     if not history:
         return history    
     user_message = history[-1]["content"] if history else ""
     
     do_sample = advanced_checkbox
-    bot_message = generate_response(user_message, history, system_prompt, max_length, temperature, top_p, do_sample, enable_thinking)
+    
+    tools_to_use = tools if use_tools else None
+    
+    bot_message = generate_response(
+        user_message, history, system_prompt, max_length, temperature, top_p, repetition_penalty,
+        do_sample, enable_thinking, tools_to_use, use_xml_tools
+    )
     history.append({"role": "assistant", "content": bot_message})
     return history
 
@@ -241,25 +302,41 @@ with gr.Blocks() as demo:
                 max_length = gr.Slider(
                     label="📏 Longueur de la réponse", 
                     minimum=10, 
-                    maximum=556, 
-                    value=120, 
+                    maximum=556,   # maximum=32768,
+                    value=56, 
                     step=1
                 )
                 temperature = gr.Slider(
                     label="🌡️ Température", 
                     minimum=0.01, 
                     maximum=1.0, 
-                    value=0.5, 
+                    value=0.6,  # Updated to SmolLM3 recommended
                     step=0.01
                 )
                 top_p = gr.Slider(
                     label="⚛️ Top-p (Echantillonnage)", 
                     minimum=0.1, 
                     maximum=1.0, 
-                    value=0.95, 
+                    value=0.95,  
+                    step=0.01
+                )
+                repetition_penalty = gr.Slider(
+                    label="🔄 Répétition Penalty", 
+                    minimum=1.0, 
+                    maximum=2.0, 
+                    value=1.1, 
                     step=0.01
                 )
                 enable_thinking = gr.Checkbox(label="Mode Réflexion", value=True)
+                use_tools = gr.Checkbox(label="🔧 Enable Tool Calling", value=False)
+                use_xml_tools = gr.Checkbox(label="📋 Use XML Tools (vs Python)", value=True)
+                with gr.Column(visible=False) as tool_options:
+                    tools = gr.Code(
+                        label="Tool Definition (JSON)",
+                        value=json.dumps(DEFAULT_TOOLS, indent=2),
+                        lines=15,
+                        language="json"
+                    )
             
             generate_button = gr.Button(value="🤖 Petite Elle L'Aime 3")
 
@@ -273,7 +350,7 @@ with gr.Blocks() as demo:
         queue=False
     ).then(
         bot,
-        [chatbot, system_prompt, max_length, temperature, top_p, advanced_checkbox, enable_thinking],
+        [chatbot, system_prompt, max_length, temperature, top_p, repetition_penalty, advanced_checkbox, enable_thinking, tools, use_xml_tools, use_tools],
         chatbot
     )
 
@@ -282,6 +359,12 @@ with gr.Blocks() as demo:
         inputs=[advanced_checkbox],
         outputs=[advanced_settings]
     )
+    
+    use_tools.change(
+        fn=lambda x: gr.update(visible=x),
+        inputs=[use_tools],
+        outputs=[tool_options]
+    )
 
 if __name__ == "__main__":
     demo.queue()

test_pre_quantized_model.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""
+Test script for pre-quantized model inference
+"""
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+def test_pre_quantized_model():
+    """Test the pre-quantized model loading and generation"""
+    
+    model_id = "Tonic/petite-elle-L-aime-3-sft"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    
+    logger.info(f"Testing pre-quantized model on device: {device}")
+    
+    try:
+        # Load tokenizer
+        logger.info("Loading tokenizer...")
+        tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="int4")
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+        
+        # Load pre-quantized model
+        logger.info("Loading pre-quantized model...")
+        model_kwargs = {
+            "device_map": "auto" if device == "cuda" else "cpu",
+            "torch_dtype": torch.float32,
+            "trust_remote_code": True,
+            "low_cpu_mem_usage": True,
+        }
+        
+        model = AutoModelForCausalLM.from_pretrained(model_id, subfolder="int4", **model_kwargs)
+        
+        # Test generation
+        test_prompt = "Bonjour, comment allez-vous?"
+        inputs = tokenizer(test_prompt, return_tensors="pt")
+        
+        if device == "cuda":
+            inputs = {k: v.cuda() for k, v in inputs.items()}
+        
+        logger.info("Generating response...")
+        with torch.no_grad():
+            output_ids = model.generate(
+                inputs['input_ids'],
+                max_new_tokens=50,
+                temperature=0.7,
+                top_p=0.95,
+                do_sample=True,
+                attention_mask=inputs['attention_mask'],
+                pad_token_id=tokenizer.eos_token_id,
+                eos_token_id=tokenizer.eos_token_id,
+                cache_implementation="static"  # Important for quantized models
+            )
+        
+        response = tokenizer.decode(output_ids[0], skip_special_tokens=True)
+        assistant_response = response[len(test_prompt):].strip()
+        
+        logger.info("✅ Pre-quantized model test successful!")
+        logger.info(f"Input: {test_prompt}")
+        logger.info(f"Output: {assistant_response}")
+        
+        # Check model quantization status
+        logger.info("Checking model quantization status...")
+        quantized_layers = 0
+        total_layers = 0
+        for name, module in model.named_modules():
+            if hasattr(module, 'weight'):
+                total_layers += 1
+                if module.weight.dtype != torch.float32:
+                    quantized_layers += 1
+                    logger.info(f"Quantized layer: {name} - {module.weight.dtype}")
+        
+        logger.info(f"Quantized layers: {quantized_layers}/{total_layers}")
+        
+        # Clean up
+        del model
+        torch.cuda.empty_cache() if device == "cuda" else None
+        
+    except Exception as e:
+        logger.error(f"❌ Pre-quantized model test failed: {e}")
+        import traceback
+        traceback.print_exc()
+
+if __name__ == "__main__":
+    test_pre_quantized_model() 

test_smollm3_features.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""
+Test script for SmolLM3 features in the Petite Elle L'Aime 3 app
+"""
+
+import json
+import sys
+import os
+
+# Add the current directory to the path so we can import from app.py
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+def test_smollm3_features():
+    """Test the SmolLM3 features implementation"""
+    
+    # Test tool definitions
+    test_tools = [
+        {
+            "name": "get_weather",
+            "description": "Get the weather in a city",
+            "parameters": {
+                "type": "object", 
+                "properties": {
+                    "city": {
+                        "type": "string", 
+                        "description": "The city to get the weather for"
+                    }
+                }
+            }
+        }
+    ]
+    
+    print("✅ Test tool definition format:")
+    print(json.dumps(test_tools, indent=2))
+    
+    # Test thinking flags
+    test_system_prompts = [
+        "Tu es TonicIA, un assistant francophone rigoureux et bienveillant./think",
+        "Tu es TonicIA, un assistant francophone rigoureux et bienveillant./no_think",
+        "Tu es TonicIA, un assistant francophone rigoureux et bienveillant."
+    ]
+    
+    print("\n✅ Test system prompts with thinking flags:")
+    for i, prompt in enumerate(test_system_prompts, 1):
+        print(f"{i}. {prompt}")
+    
+    # Test generation parameters
+    recommended_params = {
+        "temperature": 0.6,
+        "top_p": 0.95,
+        "repetition_penalty": 1.1,
+        "max_new_tokens": 2048,
+        "do_sample": True
+    }
+    
+    print("\n✅ SmolLM3 recommended generation parameters:")
+    for param, value in recommended_params.items():
+        print(f"  {param}: {value}")
+    
+    print("\n✅ SmolLM3 features implemented:")
+    print("  - Thinking mode with /think and /no_think flags")
+    print("  - Tool calling with XML and Python tools")
+    print("  - Recommended generation parameters")
+    print("  - Long context support (up to 32,768 tokens)")
+    print("  - Agentic usage with tool calling")
+    
+    return True
+
+if __name__ == "__main__":
+    test_smollm3_features()
+    print("\n🎉 All SmolLM3 features are properly configured!")