Initial commit: Voice Assistant demo

.gitattributes

@@ -1,35 +1,29 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
+# Git LFS attributes for Hugging Face Spaces
+# Track large files with Git LFS
+
+# Model files
 *.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+
+# Pickle files (model weights)
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+
+# Audio files (if you want to include examples)
+*.wav filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+
+# Archive files
+*.tar.gz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+
+# Data files
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,13 +1,163 @@
 ---
-title: Contact Search Assistant
-emoji: 📈
-colorFrom: yellow
-colorTo: indigo
+title: Voice Assistant - Multi-language Division Matching & Contact Search
+emoji: 🎙️
+colorFrom: purple
+colorTo: blue
 sdk: gradio
-sdk_version: 6.0.0
+sdk_version: 4.0.0
 app_file: app.py
 pinned: false
 license: mit
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🎙️ Voice Assistant Demo
+
+A powerful multi-language voice assistant that helps users find divisions and contacts within an organization using natural language queries.
+
+## 🌟 Features
+
+### 🗣️ Multi-language Voice Input
+- **99+ languages** supported (auto-detected)
+- Automatic speech-to-text using OpenAI Whisper
+- Arabic-to-English translation for seamless processing
+- Works with various audio formats
+
+### 🎯 Smart Division Matching
+- Semantic search using sentence embeddings
+- Confidence-based routing with intelligent thresholds
+- Department-level expansion (searches all divisions in a department)
+- Fast matching (~50ms) using `all-MiniLM-L6-v2`
+
+### 👤 Name Extraction
+- Extracts person names from queries using GLiNER
+- Supports English and Arabic names
+- Zero-shot NER for robust extraction
+
+### 📞 Contact Search
+- 500+ contacts across 23 departments and 67 divisions
+- Intelligent matching combining name and division
+- Confidence scoring with match reasoning
+- Fuzzy name matching for typos and variations
+
+## 🚀 How to Use
+
+### Division Matching (Text)
+Find the right division for your query:
+```
+"I need help from IT Security"
+"Find someone in Finance"
+"Connect me to Human Resources"
+```
+
+### Division Matching (Voice)
+Speak your query in any language - it will be transcribed and processed automatically.
+
+### Contact Search (Text)
+Search for specific people or teams:
+```
+"Find Dima in Information Technology"
+"Ahmed Al-Malek"
+"I need to talk to someone in Legal"
+```
+
+### Contact Search (Voice)
+Speak your contact search query in any language.
+
+## 📊 Example Queries
+
+### Department-Level Queries
+These queries search across ALL divisions in a department:
+- ✅ "Find someone in Information Technology" → Searches 8 IT divisions
+- ✅ "I need help from Finance" → Searches all Finance divisions
+- ✅ "Connect me to Human Resources" → Searches all HR divisions
+
+### Division-Level Queries
+These match specific divisions:
+- ✅ "Find Ahmed in App Dev" → Applications Development & Integrations
+- ✅ "I need help from IT Security" → IT Security Implementation & Operations
+- ✅ "Connect me to Legal" → Legal divisions
+
+### Name-Only Queries
+- ✅ "Find Dima" → Searches all contacts named Dima
+- ✅ "Ahmed Al-Malek" → Exact name match
+- ✅ "I need to talk to Rashed" → Fuzzy name matching
+
+### Combined Queries (Name + Department/Division)
+Priority given to division accuracy:
+- ✅ "Find Dima in Information Technology" → Perfect match (confidence: 1.00)
+- ✅ "Find Ahmed in App Dev" → Shows App Dev team members
+
+## 🔧 Technical Details
+
+### Models Used
+- **Embeddings**: `sentence-transformers/all-MiniLM-L6-v2` - Fast, lightweight semantic search
+- **Name Extraction**: `urchade/gliner_small-v2.1` - Zero-shot NER for person names
+- **Speech-to-Text**: `openai/whisper-tiny` - Optimized for speed on CPU
+
+### Confidence Scoring
+
+| Score | Meaning | Example |
+|-------|---------|---------|
+| **1.00** | Perfect match (name + division) | Dima in IT |
+| **0.95** | Exact name match | Ahmed Al-Malek |
+| **0.66** | Strong division match | People in requested division |
+| **0.59** | Good division match | Close division match |
+| **< 0.30** | Low confidence | Wrong division penalty |
+
+### Match Reasons
+- `name_and_division_match` - Both name AND division match ✅
+- `division_match` - Division/department matches (no name match)
+- `exact_name_match` - Exact name match (100%)
+- `fuzzy_name_match` - Partial name match (75%+)
+- `name_match_wrong_division` - Name matches but WRONG division ⚠️
+
+## 📦 Database Stats
+- **500 contacts** across the organization
+- **23 departments** (Information Technology, Finance, HR, etc.)
+- **67 divisions** (specific teams and units)
+- **Multi-language support** (English + Arabic names)
+
+## 🌍 Supported Languages
+
+The voice assistant supports **99+ languages** including:
+- English
+- Arabic (العربية)
+- Spanish, French, German, Italian
+- Chinese (中文), Japanese (日本語), Korean (한국어)
+- Hindi, Urdu, Bengali
+- And many more...
+
+Language is automatically detected - just speak naturally!
+
+## ⚡ Performance
+
+- **Division Matching**: ~50ms per query
+- **Name Extraction**: ~100-200ms per query
+- **Voice Processing**: ~1-3 seconds (depends on audio length)
+- **Contact Search**: ~100-300ms per query
+
+## 🛠️ Built With
+
+- **Gradio** - Interactive web interface
+- **FastAPI** - Backend API (original implementation)
+- **Sentence Transformers** - Semantic search
+- **OpenAI Whisper** - Speech recognition
+- **GLiNER** - Named Entity Recognition
+- **PyTorch** - Deep learning framework
+
+## 📝 License
+
+MIT License
+
+## 🙏 Acknowledgments
+
+- OpenAI for Whisper
+- Hugging Face for model hosting
+- URCHADE for GLiNER
+- Sentence Transformers team
+
+---
+
+**Version:** 4.0.0
+**Status:** ✅ Production Ready
+**Demo Type:** Interactive Gradio Demo

app.py

@@ -0,0 +1,382 @@
+"""
+Hugging Face Spaces Demo - Voice Assistant API
+Multi-language voice assistant with division matching and contact search
+"""
+
+import gradio as gr
+import logging
+from typing import Optional, Tuple
+import numpy as np
+
+# Import existing services
+from embedding_service import EmbeddingService
+from name_extraction_service import NameExtractor
+from voice_processing_service import VoiceProcessor
+from contact_search_service import ContactSearchService
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+# Global services (initialized once)
+embedding_service: Optional[EmbeddingService] = None
+name_extractor: Optional[NameExtractor] = None
+voice_processor: Optional[VoiceProcessor] = None
+contact_search_service: Optional[ContactSearchService] = None
+
+
+def initialize_services():
+    """Initialize all AI services (called once on startup)"""
+    global embedding_service, name_extractor, voice_processor, contact_search_service
+
+    logger.info("🚀 Initializing services...")
+
+    # Initialize embedding service (fast & lightweight)
+    logger.info("Loading embedding model...")
+    embedding_service = EmbeddingService(model_name="all-MiniLM-L6-v2")
+    logger.info("✓ Embedding service ready!")
+
+    # Initialize name extractor
+    logger.info("Loading name extraction model...")
+    name_extractor = NameExtractor(model_name="urchade/gliner_small-v2.1")
+    logger.info("✓ Name extractor ready!")
+
+    # Initialize voice processor (using tiny model for HF Spaces)
+    logger.info("Loading Whisper model...")
+    voice_processor = VoiceProcessor(model_size="tiny")  # Using tiny for faster inference
+    logger.info("✓ Voice processor ready!")
+
+    # Initialize contact search
+    logger.info("Loading contact database...")
+    contact_search_service = ContactSearchService(name_extractor, embedding_service)
+    stats = contact_search_service.get_contact_stats()
+    logger.info(f"✓ Loaded {stats['total_contacts']} contacts across {stats['divisions']} divisions")
+
+    return stats
+
+
+def format_division_matches(matches, names):
+    """Format division matching results for display"""
+    if not matches:
+        return "No matches found."
+
+    output = []
+
+    if names:
+        output.append(f"**Extracted Names:** {', '.join(names)}\n")
+
+    output.append("### 🎯 Division Matches:\n")
+
+    for i, match in enumerate(matches[:3], 1):
+        confidence_pct = match.confidence * 100
+        confidence_bar = "🟢" * int(confidence_pct / 20) + "⚪" * (5 - int(confidence_pct / 20))
+
+        output.append(f"**{i}. {match.division}**")
+        output.append(f"   - Confidence: {confidence_pct:.1f}% {confidence_bar}")
+        output.append(f"   - Department: {match.department}")
+        output.append(f"   - Keywords: {', '.join(match.keywords[:3])}")
+        output.append("")
+
+    return "\n".join(output)
+
+
+def format_contact_results(contacts, extracted_names, matched_divisions):
+    """Format contact search results for display"""
+    if not contacts:
+        return "No contacts found."
+
+    output = []
+
+    if extracted_names:
+        output.append(f"**Extracted Names:** {', '.join(extracted_names)}\n")
+
+    if matched_divisions:
+        output.append(f"**Matched Divisions:** {', '.join(matched_divisions[:3])}\n")
+
+    output.append(f"### 👥 Found {len(contacts)} Contact(s):\n")
+
+    for i, contact in enumerate(contacts[:10], 1):
+        confidence_pct = contact['confidence'] * 100
+        confidence_bar = "🟢" * int(confidence_pct / 20) + "⚪" * (5 - int(confidence_pct / 20))
+
+        output.append(f"**{i}. {contact['name']}**")
+        output.append(f"   - Position: {contact['position']}")
+        output.append(f"   - Division: {contact['division']}")
+        output.append(f"   - Department: {contact['department']}")
+        output.append(f"   - Phone: {contact['phone']}")
+        output.append(f"   - Email: {contact['email']}")
+        output.append(f"   - Confidence: {confidence_pct:.1f}% {confidence_bar}")
+        output.append(f"   - Match Reason: {contact['match_reason']}")
+        output.append("")
+
+    return "\n".join(output)
+
+
+def search_divisions_text(query: str) -> str:
+    """Search for divisions based on text query"""
+    if not query or not query.strip():
+        return "Please enter a query."
+
+    try:
+        # Extract names
+        names = name_extractor.extract_names(query)
+
+        # Find matching divisions
+        matches = embedding_service.find_division(query, top_k=3)
+
+        return format_division_matches(matches, names)
+
+    except Exception as e:
+        logger.error(f"Error in division search: {e}")
+        return f"Error: {str(e)}"
+
+
+def search_divisions_voice(audio: Optional[Tuple[int, np.ndarray]]) -> str:
+    """Search for divisions based on voice query"""
+    if audio is None:
+        return "Please record audio first."
+
+    try:
+        # Save audio to temporary file
+        sample_rate, audio_data = audio
+        temp_path = voice_processor.save_audio_array(audio_data, sample_rate)
+
+        # Process voice query
+        voice_result = voice_processor.process_voice_query(temp_path)
+        query = voice_result['query']
+
+        # Extract names
+        names = name_extractor.extract_names(query)
+
+        # Find matching divisions
+        matches = embedding_service.find_division(query, top_k=3)
+
+        # Format output
+        output = []
+        output.append(f"**🎤 Transcribed Text:** {query}")
+        output.append(f"**🌍 Language:** {voice_result['language_name']}")
+        if voice_result['was_translated']:
+            output.append(f"**📝 Original:** {voice_result['original_text']}")
+        output.append("")
+        output.append(format_division_matches(matches, names))
+
+        # Cleanup
+        voice_processor.cleanup_temp_file(temp_path)
+
+        return "\n".join(output)
+
+    except Exception as e:
+        logger.error(f"Error in voice division search: {e}")
+        return f"Error: {str(e)}"
+
+
+def search_contacts_text(query: str) -> str:
+    """Search for contacts based on text query"""
+    if not query or not query.strip():
+        return "Please enter a query."
+
+    try:
+        # Search contacts
+        contacts = contact_search_service.search_contacts(query, top_k=10, min_confidence=0.3)
+
+        # Extract names and divisions
+        names = name_extractor.extract_names(query)
+        division_matches = embedding_service.find_division(query, top_k=3)
+        matched_divisions = [m.division for m in division_matches]
+
+        return format_contact_results(contacts, names, matched_divisions)
+
+    except Exception as e:
+        logger.error(f"Error in contact search: {e}")
+        return f"Error: {str(e)}"
+
+
+def search_contacts_voice(audio: Optional[Tuple[int, np.ndarray]]) -> str:
+    """Search for contacts based on voice query"""
+    if audio is None:
+        return "Please record audio first."
+
+    try:
+        # Save audio to temporary file
+        sample_rate, audio_data = audio
+        temp_path = voice_processor.save_audio_array(audio_data, sample_rate)
+
+        # Process voice query
+        voice_result = voice_processor.process_voice_query(temp_path)
+        query = voice_result['query']
+
+        # Search contacts
+        contacts = contact_search_service.search_contacts(query, top_k=10, min_confidence=0.3)
+
+        # Extract names and divisions
+        names = name_extractor.extract_names(query)
+        division_matches = embedding_service.find_division(query, top_k=3)
+        matched_divisions = [m.division for m in division_matches]
+
+        # Format output
+        output = []
+        output.append(f"**🎤 Transcribed Text:** {query}")
+        output.append(f"**🌍 Language:** {voice_result['language_name']}")
+        if voice_result['was_translated']:
+            output.append(f"**📝 Original:** {voice_result['original_text']}")
+        output.append("")
+        output.append(format_contact_results(contacts, names, matched_divisions))
+
+        # Cleanup
+        voice_processor.cleanup_temp_file(temp_path)
+
+        return "\n".join(output)
+
+    except Exception as e:
+        logger.error(f"Error in voice contact search: {e}")
+        return f"Error: {str(e)}"
+
+
+def create_demo():
+    """Create the Gradio demo interface"""
+
+    # Initialize services on startup
+    stats = initialize_services()
+
+    # Create the interface
+    with gr.Blocks(title="Voice Assistant Demo", theme=gr.themes.Soft()) as demo:
+
+        gr.Markdown(f"""
+        # 🎙️ Voice Assistant Demo
+        ### Multi-language voice assistant with division matching and contact search
+
+        **Database:** {stats['total_contacts']} contacts • {stats['departments']} departments • {stats['divisions']} divisions
+
+        **Features:**
+        - 🗣️ Speech-to-text in 99+ languages
+        - 🔍 Smart division matching
+        - 👤 Name extraction (English & Arabic)
+        - 📞 Contact search with confidence scoring
+        """)
+
+        with gr.Tabs():
+
+            # Tab 1: Division Matching (Text)
+            with gr.Tab("📝 Division Matching (Text)"):
+                gr.Markdown("""
+                ### Search for divisions by text query
+                Try queries like:
+                - "I need help from IT Security"
+                - "Find someone in Finance"
+                - "Connect me to Human Resources"
+                - "Find Ahmed in App Dev"
+                """)
+
+                with gr.Row():
+                    with gr.Column():
+                        div_text_input = gr.Textbox(
+                            label="Enter your query",
+                            placeholder="e.g., I need help from IT Security",
+                            lines=2
+                        )
+                        div_text_btn = gr.Button("🔍 Search Divisions", variant="primary")
+
+                    with gr.Column():
+                        div_text_output = gr.Markdown(label="Results")
+
+                div_text_btn.click(
+                    fn=search_divisions_text,
+                    inputs=[div_text_input],
+                    outputs=[div_text_output]
+                )
+
+            # Tab 2: Division Matching (Voice)
+            with gr.Tab("🎤 Division Matching (Voice)"):
+                gr.Markdown("""
+                ### Search for divisions by voice
+                Speak your query in any language - it will be automatically transcribed and translated.
+                """)
+
+                with gr.Row():
+                    with gr.Column():
+                        div_voice_input = gr.Audio(
+                            sources=["microphone"],
+                            type="numpy",
+                            label="Record your voice query"
+                        )
+                        div_voice_btn = gr.Button("🔍 Search Divisions", variant="primary")
+
+                    with gr.Column():
+                        div_voice_output = gr.Markdown(label="Results")
+
+                div_voice_btn.click(
+                    fn=search_divisions_voice,
+                    inputs=[div_voice_input],
+                    outputs=[div_voice_output]
+                )
+
+            # Tab 3: Contact Search (Text)
+            with gr.Tab("👥 Contact Search (Text)"):
+                gr.Markdown("""
+                ### Search for contacts by text query
+                Try queries like:
+                - "Find Dima in Information Technology"
+                - "Ahmed Al-Malek"
+                - "I need to talk to someone in Legal"
+                - "Find Rashed in Finance"
+                """)
+
+                with gr.Row():
+                    with gr.Column():
+                        contact_text_input = gr.Textbox(
+                            label="Enter your query",
+                            placeholder="e.g., Find Dima in Information Technology",
+                            lines=2
+                        )
+                        contact_text_btn = gr.Button("🔍 Search Contacts", variant="primary")
+
+                    with gr.Column():
+                        contact_text_output = gr.Markdown(label="Results")
+
+                contact_text_btn.click(
+                    fn=search_contacts_text,
+                    inputs=[contact_text_input],
+                    outputs=[contact_text_output]
+                )
+
+            # Tab 4: Contact Search (Voice)
+            with gr.Tab("🎙️ Contact Search (Voice)"):
+                gr.Markdown("""
+                ### Search for contacts by voice
+                Speak your query in any language to find contacts.
+                """)
+
+                with gr.Row():
+                    with gr.Column():
+                        contact_voice_input = gr.Audio(
+                            sources=["microphone"],
+                            type="numpy",
+                            label="Record your voice query"
+                        )
+                        contact_voice_btn = gr.Button("🔍 Search Contacts", variant="primary")
+
+                    with gr.Column():
+                        contact_voice_output = gr.Markdown(label="Results")
+
+                contact_voice_btn.click(
+                    fn=search_contacts_voice,
+                    inputs=[contact_voice_input],
+                    outputs=[contact_voice_output]
+                )
+
+        gr.Markdown("""
+        ---
+        **Models:**
+        - Embeddings: `sentence-transformers/all-MiniLM-L6-v2`
+        - Name Extraction: `urchade/gliner_small-v2.1`
+        - Speech-to-Text: `openai/whisper-tiny`
+
+        **Supported Languages:** 99+ languages (auto-detected)
+        """)
+
+    return demo
+
+
+if __name__ == "__main__":
+    demo = create_demo()
+    demo.launch()

contact_search_service.py

@@ -0,0 +1,407 @@
+# contact_search_service.py
+"""
+Contact search service with intelligent matching:
+- Name-based search (exact and fuzzy matching)
+- Division-based search
+- Combined search (name + division)
+- Confidence scoring
+"""
+
+import logging
+from typing import List, Dict, Optional, Tuple
+from difflib import SequenceMatcher
+import re
+
+from contacts_data import (
+    get_all_contacts,
+    get_contacts_by_division,
+    get_contact_by_name
+)
+from name_extraction_service import NameExtractor
+from embedding_service import EmbeddingService
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class ContactSearchService:
+    """
+    Service for searching contacts with intelligent matching.
+
+    Features:
+    - Exact name matching (100% confidence)
+    - Fuzzy name matching (partial names, typos)
+    - Division-based matching
+    - Combined search (name + division)
+    - Multi-language support (English and Arabic)
+    """
+
+    def __init__(
+        self,
+        name_extractor: NameExtractor,
+        embedding_service: EmbeddingService
+    ):
+        """
+        Initialize the contact search service.
+
+        Args:
+            name_extractor: NameExtractor service for extracting names from queries
+            embedding_service: EmbeddingService for division matching
+        """
+        self.name_extractor = name_extractor
+        self.embedding_service = embedding_service
+        self.all_contacts = get_all_contacts()
+
+        logger.info(f"ContactSearchService initialized with {len(self.all_contacts)} contacts")
+
+    def search_contacts(
+        self,
+        query: str,
+        top_k: int = 10,
+        min_confidence: float = 0.3
+    ) -> List[Dict]:
+        """
+        Search for contacts based on query.
+
+        Process:
+        1. Extract names from query
+        2. Find matching divisions
+        3. Match contacts by:
+           - Exact name match (if name found) → confidence = 1.0
+           - Fuzzy name match → confidence based on similarity
+           - Division match → confidence from embedding service
+           - Combined match (name + division) → boosted confidence
+        4. Sort by confidence (exact matches first)
+
+        Args:
+            query: Search query (English or Arabic)
+            top_k: Maximum number of results to return
+            min_confidence: Minimum confidence threshold (0.0-1.0)
+
+        Returns:
+            List of matched contacts with confidence scores
+        """
+        logger.info(f"Searching contacts for query: '{query}'")
+
+        # Step 1: Extract names from query
+        extracted_names = self.name_extractor.extract_names(query)
+        logger.info(f"Extracted names: {extracted_names}")
+
+        # Step 2: Find matching divisions
+        division_matches = self.embedding_service.find_division(query, top_k=3)
+        logger.info(f"Found {len(division_matches)} division matches")
+
+        # Step 3: Match contacts
+        matched_contacts = []
+        has_names = len(extracted_names) > 0
+        has_divisions = len(division_matches) > 0
+        requested_divisions = [dm.division for dm in division_matches] if has_divisions else []
+
+        # Strategy A: If we have names, search by name
+        name_matches = {}  # Track name matches by contact ID
+        if extracted_names:
+            for name in extracted_names:
+                # Try exact match first
+                exact_match = get_contact_by_name(name)
+                if exact_match:
+                    contact_id = exact_match["id"]
+                    name_matches[contact_id] = {
+                        "contact": exact_match,
+                        "confidence": 1.0,
+                        "similarity": 1.0,
+                        "match_type": "exact"
+                    }
+                    logger.info(f"✓ Exact name match: {exact_match['full_name_en']}")
+                else:
+                    # Fuzzy name matching
+                    fuzzy_matches = self._fuzzy_name_search(name, top_k=10)
+                    for contact, similarity in fuzzy_matches:
+                        contact_id = contact["id"]
+                        # Only keep best match for each contact
+                        if contact_id not in name_matches or similarity > name_matches[contact_id]["similarity"]:
+                            name_matches[contact_id] = {
+                                "contact": contact,
+                                "confidence": round(0.5 + (similarity * 0.45), 2),
+                                "similarity": round(similarity, 2),
+                                "match_type": "fuzzy"
+                            }
+                            logger.info(
+                                f"Fuzzy name match: {contact['full_name_en']} "
+                                f"(similarity: {similarity:.2f})"
+                            )
+
+        # Strategy B: Division-based search
+        division_matches_dict = {}  # Track division matches by contact ID
+        for div_match in division_matches:
+            division = div_match.division
+            division_confidence = div_match.confidence
+
+            # Get contacts in this division
+            division_contacts = get_contacts_by_division(division)
+
+            for contact in division_contacts:
+                contact_id = contact["id"]
+                # Only keep best division match for each contact
+                if contact_id not in division_matches_dict or division_confidence > division_matches_dict[contact_id]["confidence"]:
+                    division_matches_dict[contact_id] = {
+                        "contact": contact,
+                        "confidence": division_confidence,
+                        "division": division
+                    }
+
+        # Strategy C: Combine matches intelligently
+        # Priority when BOTH name and division are specified:
+        # 1. Name + Correct Division = HIGHEST (both match)
+        # 2. Correct Division only = HIGH (division is most important)
+        # 3. Name + Wrong Division = LOW (penalize wrong division)
+
+        all_contact_ids = set(name_matches.keys()) | set(division_matches_dict.keys())
+
+        for contact_id in all_contact_ids:
+            has_name_match = contact_id in name_matches
+            has_division_match = contact_id in division_matches_dict
+
+            if has_name_match and has_division_match:
+                # BOTH name and division match - BEST CASE
+                name_data = name_matches[contact_id]
+                div_data = division_matches_dict[contact_id]
+                contact = name_data["contact"]
+
+                # When both match: take MAX of the two confidences and add a boost
+                # This ensures division + name is always better than division alone
+                combined_confidence = max(name_data["confidence"], div_data["confidence"]) + 0.15
+                combined_confidence = min(1.0, combined_confidence)
+
+                contact_result = contact.copy()
+                contact_result["confidence"] = round(combined_confidence, 2)
+                contact_result["match_reason"] = "name_and_division_match"
+                contact_result["name_confidence"] = name_data["confidence"]
+                contact_result["division_confidence"] = div_data["confidence"]
+                matched_contacts.append(contact_result)
+
+                logger.info(
+                    f"✓ BOTH match: {contact['full_name_en']} in {div_data['division']} "
+                    f"(final confidence: {contact_result['confidence']})"
+                )
+
+            elif has_division_match:
+                # Division match only (no name specified, or name didn't match this person)
+                div_data = division_matches_dict[contact_id]
+                contact = div_data["contact"]
+
+                contact_result = contact.copy()
+                contact_result["confidence"] = div_data["confidence"]
+                contact_result["match_reason"] = "division_match"
+                contact_result["division_confidence"] = div_data["confidence"]
+                matched_contacts.append(contact_result)
+
+            elif has_name_match:
+                # Name match but WRONG division (or no division specified)
+                name_data = name_matches[contact_id]
+                contact = name_data["contact"]
+
+                # If division was specified in query AND has reasonable confidence (>= 0.58)
+                # Apply penalty for being in wrong division
+                # If division confidence is very low (< 0.58), treat as name-only search
+                # This threshold helps avoid false division matches from words like "Find" (which scores ~0.56)
+                # while still catching abbreviations like "App Dev" (which scores ~0.59)
+                has_strong_division_match = has_divisions and division_matches[0].confidence >= 0.58
+
+                if has_strong_division_match:
+                    # Heavy penalty for wrong division when division was specified with confidence
+                    penalized_confidence = name_data["confidence"] * 0.3  # 70% penalty
+                    contact_result = contact.copy()
+                    contact_result["confidence"] = round(penalized_confidence, 2)
+                    contact_result["match_reason"] = "name_match_wrong_division"
+                    contact_result["name_confidence"] = name_data["confidence"]
+                    contact_result["requested_division"] = ", ".join(requested_divisions[:2])
+                    matched_contacts.append(contact_result)
+
+                    logger.info(
+                        f"Name match with WRONG division: {contact['full_name_en']} "
+                        f"in {contact['division']} (wanted: {requested_divisions[0]}, "
+                        f"confidence: {contact_result['confidence']})"
+                    )
+                else:
+                    # No division specified OR weak division match - use name confidence as-is
+                    contact_result = contact.copy()
+                    contact_result["confidence"] = name_data["confidence"]
+                    contact_result["match_reason"] = f"{name_data['match_type']}_name_match"
+                    contact_result["name_confidence"] = name_data["confidence"]
+                    matched_contacts.append(contact_result)
+
+        # Step 4: Remove duplicates (keep highest confidence)
+        unique_contacts = {}
+        for contact in matched_contacts:
+            contact_id = contact["id"]
+            if contact_id not in unique_contacts:
+                unique_contacts[contact_id] = contact
+            else:
+                # Keep the one with higher confidence
+                if contact["confidence"] > unique_contacts[contact_id]["confidence"]:
+                    unique_contacts[contact_id] = contact
+
+        # Convert back to list
+        matched_contacts = list(unique_contacts.values())
+
+        # Step 5: Filter by minimum confidence
+        matched_contacts = [
+            c for c in matched_contacts if c["confidence"] >= min_confidence
+        ]
+
+        # Step 6: Sort by confidence (descending) - exact matches will be first
+        matched_contacts.sort(key=lambda x: x["confidence"], reverse=True)
+
+        # Step 7: Limit to top_k
+        matched_contacts = matched_contacts[:top_k]
+
+        logger.info(f"✓ Returning {len(matched_contacts)} matched contacts")
+
+        return matched_contacts
+
+    def _fuzzy_name_search(
+        self,
+        name: str,
+        top_k: int = 5,
+        min_similarity: float = 0.75  # Increased from 0.6 to avoid false matches
+    ) -> List[Tuple[Dict, float]]:
+        """
+        Fuzzy name matching using string similarity with stricter rules.
+
+        Args:
+            name: Name to search for
+            top_k: Number of top matches to return
+            min_similarity: Minimum similarity threshold (0.0-1.0)
+
+        Returns:
+            List of (contact, similarity_score) tuples
+        """
+        matches = []
+
+        # Normalize name for comparison
+        name_normalized = self._normalize_name(name)
+
+        # Get first letter for initial matching (helps avoid false positives)
+        name_first_letter = name_normalized[0] if name_normalized else ''
+
+        for contact in self.all_contacts:
+            # Check against both Arabic and English names
+            full_name_en_normalized = self._normalize_name(contact["full_name_en"])
+            full_name_ar_normalized = self._normalize_name(contact["full_name_ar"])
+            first_name_en_normalized = self._normalize_name(contact["first_name_en"])
+            first_name_ar_normalized = self._normalize_name(contact["first_name_ar"])
+            last_name_en_normalized = self._normalize_name(contact["last_name_en"])
+            last_name_ar_normalized = self._normalize_name(contact["last_name_ar"])
+
+            # Calculate similarity against various name combinations
+            name_candidates = [
+                (full_name_en_normalized, "full_en"),
+                (full_name_ar_normalized, "full_ar"),
+                (first_name_en_normalized, "first_en"),
+                (first_name_ar_normalized, "first_ar"),
+                (last_name_en_normalized, "last_en"),
+                (last_name_ar_normalized, "last_ar"),
+            ]
+
+            best_similarity = 0
+            best_match_type = None
+
+            for candidate_name, match_type in name_candidates:
+                if not candidate_name:
+                    continue
+
+                similarity = self._string_similarity(name_normalized, candidate_name)
+
+                # Apply stricter rules for fuzzy matching:
+                # 1. Names should start with the same letter (for English names)
+                # 2. Or have very high similarity (>= 0.85)
+                if match_type.endswith('_en'):
+                    candidate_first_letter = candidate_name[0] if candidate_name else ''
+                    # Require same first letter OR very high similarity
+                    if candidate_first_letter != name_first_letter and similarity < 0.85:
+                        continue  # Skip this match
+
+                if similarity > best_similarity:
+                    best_similarity = similarity
+                    best_match_type = match_type
+
+            if best_similarity >= min_similarity:
+                matches.append((contact, best_similarity))
+
+        # Sort by similarity (descending)
+        matches.sort(key=lambda x: x[1], reverse=True)
+
+        return matches[:top_k]
+
+    def _normalize_name(self, name: str) -> str:
+        """Normalize name for comparison (lowercase, remove extra spaces)"""
+        return re.sub(r'\s+', ' ', name.strip().lower())
+
+    def _string_similarity(self, s1: str, s2: str) -> float:
+        """
+        Calculate string similarity using SequenceMatcher.
+
+        Returns:
+            Similarity score between 0.0 and 1.0
+        """
+        return SequenceMatcher(None, s1, s2).ratio()
+
+    def get_contact_stats(self) -> Dict:
+        """Get statistics about the contact database"""
+        from collections import Counter
+
+        dept_counts = Counter(contact["department"] for contact in self.all_contacts)
+        div_counts = Counter(contact["division"] for contact in self.all_contacts)
+
+        return {
+            "total_contacts": len(self.all_contacts),
+            "departments": len(dept_counts),
+            "divisions": len(div_counts),
+            "contacts_by_department": dict(dept_counts),
+            "contacts_by_division": dict(div_counts),
+        }
+
+
+if __name__ == "__main__":
+    # Test the contact search service
+    from name_extraction_service import NameExtractor
+    from embedding_service import EmbeddingService
+
+    print("Initializing services...")
+    name_extractor = NameExtractor()
+    embedding_service = EmbeddingService()
+    search_service = ContactSearchService(name_extractor, embedding_service)
+
+    print("\nContact Database Stats:")
+    stats = search_service.get_contact_stats()
+    print(f"Total contacts: {stats['total_contacts']}")
+    print(f"Departments: {stats['departments']}")
+    print(f"Divisions: {stats['divisions']}")
+
+    # Test queries
+    test_queries = [
+        "Find Ahmed in IT",
+        "I need to talk to someone in HR",
+        "محمد في المالية",  # "Mohammed in Finance" in Arabic
+        "Finance accounting help",
+    ]
+
+    print("\n" + "="*80)
+    print("Testing Contact Search")
+    print("="*80)
+
+    for query in test_queries:
+        print(f"\nQuery: '{query}'")
+        print("-" * 80)
+
+        results = search_service.search_contacts(query, top_k=3)
+
+        if results:
+            for i, contact in enumerate(results, 1):
+                print(f"{i}. {contact['full_name_en']} ({contact['full_name_ar']})")
+                print(f"   {contact['title_en']} - {contact['division']}")
+                print(f"   {contact['email']} | Ext: {contact['extension']}")
+                print(f"   Confidence: {contact['confidence']:.2f} | Reason: {contact['match_reason']}")
+        else:
+            print("No matches found.")

contacts_data.py

@@ -0,0 +1,294 @@
+# contacts_data.py
+"""
+Contact database with 500 fake contacts covering all departments and divisions.
+Each contact has Arabic and English names for better search support.
+"""
+
+from typing import List, Dict
+from division_hierarchy import DIVISION_TO_DEPARTMENT
+import random
+
+# Common Arabic first names (male and female)
+ARABIC_FIRST_NAMES_MALE = [
+    "محمد", "أحمد", "عبدالله", "عمر", "خالد", "سعد", "فيصل", "سلطان", "ناصر", "طلال",
+    "عبدالعزيز", "فهد", "تركي", "سلمان", "بندر", "مشعل", "ماجد", "يوسف", "حسن", "علي",
+    "وليد", "زياد", "رامي", "كريم", "عادل", "راشد", "مازن", "طارق", "إبراهим", "عيسى",
+    "نواف", "سامي", "بدر", "عاصم", "وسام", "هاني", "ثامر", "صالح", "ياسر", "جاسم",
+    "هشام", "فواز", "معاذ", "عثمان", "أسامة", "باسل", "عمار", "نبيل", "توفيق", "جمال"
+]
+
+ARABIC_FIRST_NAMES_FEMALE = [
+    "فاطمة", "نورة", "سارة", "منى", "هند", "ريم", "لينا", "دانة", "شهد", "جود",
+    "رهف", "غلا", "عبير", "أمل", "ندى", "رنا", "لمى", "ديمة", "بشرى", "سمية",
+    "هيفاء", "ليلى", "زينب", "خلود", "شروق", "أريج", "جميلة", "رباب", "سلمى", "وفاء",
+    "عائشة", "خديجة", "مريم", "رقية", "زهراء", "نجود", "حصة", "عزة", "صفية", "ملاك",
+    "روان", "تالا", "جنى", "لين", "ريتاج", "أسماء", "سديم", "لمار", "بيان", "شيماء"
+]
+
+# Common Arabic last names
+ARABIC_LAST_NAMES = [
+    "العتيبي", "الدوسري", "القحطاني", "الشهري", "الغامدي", "الزهراني", "العنزي", "الحربي",
+    "المطيري", "العسيري", "السبيعي", "الشمري", "الجهني", "العمري", "البقمي", "الفهد",
+    "السديري", "الثبيتي", "الصقري", "الأحمد", "الخالد", "السليمان", "العبدالله", "الفهيد",
+    "الشايع", "الرشيد", "العجمي", "المالك", "الفريح", "الحمود", "الناصر", "الشريف",
+    "البلوي", "اليامي", "الوادعي", "الفيفي", "الشهراني", "البكري", "العسكر", "الراشد",
+    "الفايز", "الخليف", "المنيع", "العبيد", "السحيم", "الغنام", "السلمان", "الهاجري",
+    "النهدي", "الرويلي", "المري", "السواط", "الربيعان", "الدغيثر", "الفضلي", "القرني",
+    "الثنيان", "العريفي", "الهويدي", "الجريسي", "البدراني", "المهيدب", "السالم", "الحارثي",
+    "العطوي", "الصخري", "الرحيلي", "السعيد", "الحافظ", "الوهيبي", "البراك", "الضويان"
+]
+
+# Job titles in English and Arabic by category
+JOB_TITLES = {
+    "executive": [
+        ("Chief Executive Officer", "المدير التنفيذي"),
+        ("Executive Director", "المدير التنفيذي"),
+        ("Vice President", "نائب الرئيس"),
+        ("Senior Vice President", "نائب الرئيس الأول"),
+    ],
+    "management": [
+        ("Director", "مدير"),
+        ("Senior Manager", "مدير أول"),
+        ("Manager", "مدير"),
+        ("Assistant Manager", "مساعد مدير"),
+        ("Team Leader", "قائد فريق"),
+        ("Supervisor", "مشرف"),
+    ],
+    "specialist": [
+        ("Senior Specialist", "أخصائي أول"),
+        ("Specialist", "أخصائي"),
+        ("Senior Analyst", "محلل أول"),
+        ("Analyst", "محلل"),
+        ("Senior Consultant", "مستشار أول"),
+        ("Consultant", "مستشار"),
+        ("Senior Officer", "موظف أول"),
+        ("Officer", "موظف"),
+    ],
+    "technical": [
+        ("Senior Engineer", "مهندس أول"),
+        ("Engineer", "مهندس"),
+        ("Technical Lead", "قائد تقني"),
+        ("Developer", "مطور"),
+        ("Architect", "مهندس معماري"),
+    ],
+    "support": [
+        ("Coordinator", "منسق"),
+        ("Administrator", "إداري"),
+        ("Assistant", "مساعد"),
+        ("Associate", "معاون"),
+    ]
+}
+
+# Phone extensions (4-digit)
+def generate_extension() -> str:
+    """Generate a 4-digit phone extension"""
+    return str(random.randint(1000, 9999))
+
+# Email generation
+def generate_email(first_name_en: str, last_name_en: str) -> str:
+    """Generate an email address"""
+    # Remove spaces and special characters
+    first = first_name_en.lower().replace(" ", "").replace("-", "")
+    last = last_name_en.lower().replace(" ", "").replace("-", "")
+    return f"{first}.{last}@sidf.gov.sa"
+
+
+def transliterate_arabic_name(arabic_name: str) -> str:
+    """
+    Simple transliteration of Arabic names to English.
+    This is a basic mapping for common names.
+    """
+    transliteration_map = {
+        # Male names
+        "محمد": "Mohammed", "أحمد": "Ahmed", "عبدالله": "Abdullah", "عمر": "Omar", "خالد": "Khalid",
+        "سعد": "Saad", "فيصل": "Faisal", "سلطان": "Sultan", "ناصر": "Nasser", "طلال": "Talal",
+        "عبدالعزيز": "Abdulaziz", "فهد": "Fahad", "تركي": "Turki", "سلمان": "Salman", "بندر": "Bandar",
+        "مشعل": "Mishaal", "ماجد": "Majed", "يوسف": "Yousef", "حسن": "Hassan", "علي": "Ali",
+        "وليد": "Waleed", "زياد": "Ziyad", "رامي": "Rami", "كريم": "Kareem", "عادل": "Adel",
+        "راشد": "Rashed", "مازن": "Mazen", "طارق": "Tariq", "إبراهim": "Ibrahim", "عيسى": "Issa",
+        "نواف": "Nawaf", "سامي": "Sami", "بدر": "Badr", "عاصم": "Asim", "وسام": "Wissam",
+        "هاني": "Hani", "ثامر": "Thamer", "صالح": "Saleh", "ياسر": "Yasser", "جاسم": "Jasim",
+        "هشام": "Hisham", "فواز": "Fawaz", "معاذ": "Muath", "عثمان": "Othman", "أسامة": "Osama",
+        "باسل": "Basel", "عمار": "Ammar", "نبيل": "Nabil", "توفيق": "Tawfiq", "جمال": "Jamal",
+
+        # Female names
+        "فاطمة": "Fatima", "نورة": "Noura", "سارة": "Sarah", "منى": "Mona", "هند": "Hind",
+        "ريم": "Reem", "لينا": "Lina", "دانة": "Dana", "شهد": "Shahad", "جود": "Joud",
+        "رهف": "Rahaf", "غلا": "Ghala", "عبير": "Abeer", "أمل": "Amal", "ندى": "Nada",
+        "رنا": "Rana", "لمى": "Lama", "ديمة": "Dima", "بشرى": "Bushra", "سمية": "Somaya",
+        "هيفاء": "Haifa", "ليلى": "Layla", "زينب": "Zainab", "خلود": "Kholoud", "شروق": "Shorouq",
+        "أريج": "Areej", "جميلة": "Jamila", "رباب": "Rabab", "سلمى": "Salma", "وفاء": "Wafa",
+        "عائشة": "Aisha", "خديجة": "Khadija", "مريم": "Maryam", "رقية": "Ruqaya", "زهراء": "Zahra",
+        "نجود": "Nujoud", "حصة": "Hessa", "عزة": "Azza", "صفية": "Safiya", "ملاك": "Malak",
+        "روان": "Rawan", "تالا": "Tala", "جنى": "Jana", "لين": "Leen", "ريتاج": "Ritaj",
+        "أسماء": "Asma", "سديم": "Sadeem", "لمار": "Lamar", "بيان": "Bayan", "شيماء": "Shaima",
+
+        # Last names
+        "العتيبي": "Al-Otaibi", "الدوسري": "Al-Dosari", "القحطاني": "Al-Qahtani", "الشهري": "Al-Shahri",
+        "الغامدي": "Al-Ghamdi", "الزهراني": "Al-Zahrani", "العنزي": "Al-Anazi", "الحربي": "Al-Harbi",
+        "المطيري": "Al-Mutairi", "العسيري": "Al-Asiri", "السبيعي": "Al-Subaie", "الشمري": "Al-Shammari",
+        "الجهني": "Al-Juhani", "العمري": "Al-Omari", "البقمي": "Al-Buqami", "الفهد": "Al-Fahad",
+        "السديري": "Al-Sudairi", "الثبيتي": "Al-Thubaiti", "الصقري": "Al-Saqri", "الأحمد": "Al-Ahmad",
+        "الخالد": "Al-Khalid", "السليمان": "Al-Sulaiman", "العبدالله": "Al-Abdullah", "الفهيد": "Al-Fahaid",
+        "الشايع": "Al-Shaya", "الرشيد": "Al-Rasheed", "العجمي": "Al-Ajmi", "المالك": "Al-Malek",
+        "الفريح": "Al-Fraihi", "الحمود": "Al-Hamoud", "الناصر": "Al-Nasser", "الشريف": "Al-Shareef",
+        "البلوي": "Al-Balawi", "اليامي": "Al-Yami", "الوادعي": "Al-Wadei", "الفيفي": "Al-Faifi",
+        "الشهراني": "Al-Shahrani", "البكري": "Al-Bakri", "العسكر": "Al-Askar", "الراشد": "Al-Rashed",
+        "الفايز": "Al-Fayez", "الخليف": "Al-Khleif", "المنيع": "Al-Manie", "العبيد": "Al-Obaid",
+        "السحيم": "Al-Suhaim", "الغنام": "Al-Ghannam", "السلمان": "Al-Salman", "الهاجري": "Al-Hajri",
+        "النهدي": "Al-Nahdi", "الرويلي": "Al-Ruwaili", "المري": "Al-Marri", "السواط": "Al-Sawat",
+        "الربيعان": "Al-Rabian", "الدغيثر": "Al-Dughither", "الفضلي": "Al-Fadhli", "القرني": "Al-Qarni",
+        "الثنيان": "Al-Thuniyan", "العريفي": "Al-Arifi", "الهويدي": "Al-Huwaidi", "الجريسي": "Al-Juraysi",
+        "البدراني": "Al-Badrani", "المهيدب": "Al-Muhaidib", "السالم": "Al-Salem", "الحارثي": "Al-Harthi",
+        "العطوي": "Al-Atawi", "الصخري": "Al-Sakhri", "الرحيلي": "Al-Rahili", "السعيد": "Al-Saeed",
+        "الحافظ": "Al-Hafiz", "الوهيبي": "Al-Wahaibi", "البراك": "Al-Barrak", "الضويان": "Al-Dhuwayan",
+    }
+    return transliteration_map.get(arabic_name, arabic_name)
+
+
+def generate_contacts() -> List[Dict]:
+    """
+    Generate 500 fake contacts distributed across all divisions.
+    Returns a list of contact dictionaries.
+    """
+    contacts = []
+    contact_id = 1000  # Starting ID
+
+    # Get all divisions
+    divisions = list(DIVISION_TO_DEPARTMENT.keys())
+
+    # Calculate contacts per division (aim for ~7-8 per division)
+    contacts_per_division = 500 // len(divisions)
+    extra_contacts = 500 % len(divisions)
+
+    for div_index, division in enumerate(divisions):
+        department_name, department_id = DIVISION_TO_DEPARTMENT[division]
+
+        # Number of contacts for this division
+        num_contacts = contacts_per_division
+        if div_index < extra_contacts:
+            num_contacts += 1
+
+        # Determine seniority distribution (more junior staff than senior)
+        # 10% executive, 20% management, 50% specialist, 15% technical, 5% support
+        seniority_distribution = []
+        seniority_distribution.extend(["executive"] * max(1, int(num_contacts * 0.10)))
+        seniority_distribution.extend(["management"] * max(1, int(num_contacts * 0.20)))
+        seniority_distribution.extend(["specialist"] * max(1, int(num_contacts * 0.50)))
+        seniority_distribution.extend(["technical"] * max(1, int(num_contacts * 0.15)))
+        seniority_distribution.extend(["support"] * max(1, int(num_contacts * 0.05)))
+
+        # Ensure we have exactly num_contacts
+        while len(seniority_distribution) < num_contacts:
+            seniority_distribution.append("specialist")
+        seniority_distribution = seniority_distribution[:num_contacts]
+
+        random.shuffle(seniority_distribution)
+
+        for i in range(num_contacts):
+            # Mix male and female names (60% male, 40% female)
+            is_male = random.random() < 0.6
+
+            if is_male:
+                first_name_ar = random.choice(ARABIC_FIRST_NAMES_MALE)
+            else:
+                first_name_ar = random.choice(ARABIC_FIRST_NAMES_FEMALE)
+
+            last_name_ar = random.choice(ARABIC_LAST_NAMES)
+
+            # Transliterate to English
+            first_name_en = transliterate_arabic_name(first_name_ar)
+            last_name_en = transliterate_arabic_name(last_name_ar)
+
+            # Full names
+            full_name_ar = f"{first_name_ar} {last_name_ar}"
+            full_name_en = f"{first_name_en} {last_name_en}"
+
+            # Get job title based on seniority
+            seniority = seniority_distribution[i]
+            title_en, title_ar = random.choice(JOB_TITLES[seniority])
+
+            # Generate contact info
+            extension = generate_extension()
+            email = generate_email(first_name_en, last_name_en)
+
+            contact = {
+                "id": contact_id,
+                "first_name_ar": first_name_ar,
+                "last_name_ar": last_name_ar,
+                "full_name_ar": full_name_ar,
+                "first_name_en": first_name_en,
+                "last_name_en": last_name_en,
+                "full_name_en": full_name_en,
+                "title_en": title_en,
+                "title_ar": title_ar,
+                "division": division,
+                "department": department_name,
+                "department_id": department_id,
+                "email": email,
+                "extension": extension,
+                "phone": f"+966-11-218-{extension}",
+            }
+
+            contacts.append(contact)
+            contact_id += 1
+
+    return contacts
+
+
+# Generate contacts on module load
+CONTACTS_DATABASE = generate_contacts()
+
+# Create indexes for fast lookup
+CONTACTS_BY_NAME_AR = {contact["full_name_ar"]: contact for contact in CONTACTS_DATABASE}
+CONTACTS_BY_NAME_EN = {contact["full_name_en"]: contact for contact in CONTACTS_DATABASE}
+CONTACTS_BY_DIVISION = {}
+for contact in CONTACTS_DATABASE:
+    division = contact["division"]
+    if division not in CONTACTS_BY_DIVISION:
+        CONTACTS_BY_DIVISION[division] = []
+    CONTACTS_BY_DIVISION[division].append(contact)
+
+
+def get_all_contacts() -> List[Dict]:
+    """Get all contacts"""
+    return CONTACTS_DATABASE
+
+
+def get_contacts_by_division(division: str) -> List[Dict]:
+    """Get contacts for a specific division"""
+    return CONTACTS_BY_DIVISION.get(division, [])
+
+
+def get_contact_by_name(name: str) -> Dict:
+    """Get contact by exact name (Arabic or English)"""
+    # Try Arabic first
+    contact = CONTACTS_BY_NAME_AR.get(name)
+    if contact:
+        return contact
+
+    # Try English
+    contact = CONTACTS_BY_NAME_EN.get(name)
+    if contact:
+        return contact
+
+    return None
+
+
+if __name__ == "__main__":
+    # Test the contact generation
+    contacts = get_all_contacts()
+    print(f"Generated {len(contacts)} contacts")
+    print(f"\nSample contacts:")
+    for i, contact in enumerate(contacts[:5]):
+        print(f"{i+1}. {contact['full_name_en']} ({contact['full_name_ar']})")
+        print(f"   {contact['title_en']} - {contact['division']}")
+        print(f"   {contact['email']} | Ext: {contact['extension']}")
+        print()
+
+    # Show distribution by department
+    from collections import Counter
+    dept_counts = Counter(contact["department"] for contact in contacts)
+    print("\nContacts by Department:")
+    for dept, count in sorted(dept_counts.items(), key=lambda x: -x[1]):
+        print(f"  {dept}: {count}")

division_hierarchy.py

@@ -0,0 +1,187 @@
+# division_hierarchy.py
+"""
+Mapping of specific divisions to parent departments
+"""
+
+# Parent Department IDs and Names
+DEPARTMENTS = {
+    "40000003": "Information Technology",
+    "40000010": "Finance",
+    "40000005": "Human Resources",
+    "40000009": "Legal",
+    "40000011": "Communication",
+    "40000012": "Business Development",
+    "40000004": "Strategic Planning",
+    "40000245": "Project Studies",
+    "40000246": "Market Studies",
+    "40020002": "Services and Facilities",
+    "40020003": "Purchasing and Contracts",
+    "40000794": "Governance and Compliance",
+    "40000638": "Credit Risk",
+    "40000565": "Portfolio Management",
+    "40000564": "Credit Relationship",
+    "40020004": "Credit Control Division",
+    "40000013": "Loans Support Division",
+    "40000667": "Enterprise Risk Management",
+    "40001089": "Audit Excellence Unit",
+    "40000096": "Operations Audit Division",
+    "40001088": "Credit Audit Division",
+    "40000097": "IT Audit Division",
+    "40000520": "Cyber Security Division",
+    "40020001": "SIDF Academy",
+    "40001188": "Corporate Excellence",
+    "40000522": "CEO Advisory Office",
+    "40000523": "Board Secretary",
+    "40020005": "Chief of Staff Office",
+    "40020000": "ILab",
+    "40000488": "Secondment",
+    "40001063": "Credit Analysis & Advisory Division",
+    "40000089": "VP Office Credit",
+}
+
+# Mapping: Division Name -> (Department Name, Department ID)
+DIVISION_TO_DEPARTMENT = {
+    # FINANCE
+    "General Accounting Division": ("Finance", "40000010"),
+    "Accounts Payable Division": ("Finance", "40000010"),
+    "Accounts Receivable Division": ("Finance", "40000010"),
+    "Financial Reporting & Control Division": ("Finance", "40000010"),
+    "Audit & Financial Analysis": ("Finance", "40000010"),
+
+    # INFORMATION TECHNOLOGY
+    "IT Governance & Quality Division": ("Information Technology", "40000003"),
+    "Applications Development & Integrations": ("Information Technology", "40000003"),
+    "IT Infrastructure & Operations Div.": ("Information Technology", "40000003"),
+    "Applications Maintenance & Support Div.": ("Information Technology", "40000003"),
+    "IT Security Implementation & Operations": ("Information Technology", "40000003"),
+    "IT Manager's Office": ("Information Technology", "40000003"),
+    "Enterprise Architecture Team": ("Information Technology", "40000003"),
+    "Reporting & Data Analytics Unit": ("Information Technology", "40000003"),
+
+    # HUMAN RESOURCES
+    "Rewards & Hr Operations Division": ("Human Resources", "40000005"),
+    "HR Business Partner": ("Human Resources", "40000005"),
+    "Talent Development Division": ("Human Resources", "40000005"),
+    "Od & Talent Acquisition Division": ("Human Resources", "40000005"),
+    "Org.Culture & Initiatives Division": ("Human Resources", "40000005"),
+
+    # LEGAL
+    "Legal Agreements and Consultancy Division": ("Legal", "40000009"),
+    "Contracts & Mortgages and Guarantees Division": ("Legal", "40000009"),
+    "Cases Division": ("Legal", "40000009"),
+
+    # COMMUNICATION
+    "Public Relations & Media Division": ("Communication", "40000011"),
+    "Internal Communication Unit": ("Communication", "40000011"),
+
+    # BUSINESS DEVELOPMENT
+    "Marketing Division": ("Business Development", "40000012"),
+    "Partnerships Dev&Advisory Services Div": ("Business Development", "40000012"),
+
+    # STRATEGIC PLANNING
+    "Strategy Division": ("Strategic Planning", "40000004"),
+    "Corporate Performance Division": ("Strategic Planning", "40000004"),
+    "Knowledge Management Unit": ("Strategic Planning", "40000004"),
+
+    # PROJECTS
+    "Project Management Division": ("Project Studies", "40000245"),
+    "Projects Consultancy Division": ("Project Studies", "40000245"),
+    "Construction Consultancy Division": ("Project Studies", "40000245"),
+
+    # MARKET RESEARCH
+    "Market Research Div.": ("Market Studies", "40000246"),
+    "Market Studies Division": ("Market Studies", "40000246"),
+    "Business Intelligence Div.": ("Market Studies", "40000246"),
+
+    # FACILITIES
+    "Facilities Management Division": ("Services and Facilities", "40020002"),
+    "Documents and Administrative Communications Center": ("Services and Facilities", "40020002"),
+    "Security and Safety Division": ("Services and Facilities", "40020002"),
+    "Industrial Safety & Loss Prevention Div": ("Services and Facilities", "40020002"),
+
+    # PURCHASING
+    "Procurements Contracts and Vendors Division": ("Purchasing and Contracts", "40020003"),
+    "Purchasing Division": ("Purchasing and Contracts", "40020003"),
+
+    # GOVERNANCE
+    "Enterprise Governance Division": ("Governance and Compliance", "40000794"),
+    "Compliance Regulatory Division": ("Governance and Compliance", "40000794"),
+    "Compliance Operations Division": ("Governance and Compliance", "40000794"),
+    "Operational Risk Management Division": ("Enterprise Risk Management", "40000667"),
+    "Financial Risk Management Division": ("Enterprise Risk Management", "40000667"),
+    "Cybersecurity Governance Unit": ("Cyber Security Division", "40000520"),
+    "Cybersecurity Defense Unit": ("Cyber Security Division", "40000520"),
+
+    # CREDIT
+    "Credit Risk Division": ("Credit Risk", "40000638"),
+    "Credit Relationships Division": ("Credit Relationship", "40000564"),
+    "Credit Control Team - A": ("Credit Control Division", "40020004"),
+    "Credit Control Team - B": ("Credit Control Division", "40020004"),
+    "Collection Team": ("Credit Control Division", "40020004"),
+    "Loan Follow-Up Team": ("Loans Support Division", "40000013"),
+    "C&Lm Info Team": ("Credit Risk", "40000638"),
+
+    # PORTFOLIO
+    "Portfolio Manager'S Office": ("Portfolio Management", "40000565"),
+    "Portfolio Division": ("Portfolio Management", "40000565"),
+    "Special Assets Division": ("Portfolio Management", "40000565"),
+
+    # EXCELLENCE
+    "Programs Delivery Operations Division": ("Corporate Excellence", "40001188"),
+    "Customer Excellence Division": ("Corporate Excellence", "40001188"),
+    "Programs Design And Needs Assessment Division": ("Corporate Excellence", "40001188"),
+    "Programs Evaluation & Quality Assurance Division": ("Corporate Excellence", "40001188"),
+    "Product Dev For National Priorities Div": ("Corporate Excellence", "40001188"),
+    "Product Dev For Customer Empowerment Div": ("Corporate Excellence", "40001188"),
+    "Solutions & Design Division": ("Corporate Excellence", "40001188"),
+
+    # AUDIT
+    "Operations Audit Team": ("Operations Audit Division", "40000096"),
+
+    # ACADEMY
+    "Academy Strategic Partnerships Division": ("SIDF Academy", "40020001"),
+
+    # EXECUTIVE
+    "CEO Office": ("CEO Advisory Office", "40000522"),
+}
+
+
+def get_department(division_name: str) -> tuple:
+    """
+    Get the parent department for a division.
+
+    Args:
+        division_name: Name of the specific division
+
+    Returns:
+        Tuple of (department_name, department_id) or (None, None) if not found
+    """
+    return DIVISION_TO_DEPARTMENT.get(division_name, (None, None))
+
+
+def get_department_name(division_name: str) -> str:
+    """
+    Get just the department name for a division.
+
+    Args:
+        division_name: Name of the specific division
+
+    Returns:
+        Department name or "Unknown" if not found
+    """
+    dept_name, _ = get_department(division_name)
+    return dept_name or "Unknown"
+
+
+def get_department_id(division_name: str) -> str:
+    """
+    Get just the department ID for a division.
+
+    Args:
+        division_name: Name of the specific division
+
+    Returns:
+        Department ID or None if not found
+    """
+    _, dept_id = get_department(division_name)
+    return dept_id

embedding_service.py

@@ -0,0 +1,347 @@
+# embedding_service.py
+import logging
+import numpy as np
+from typing import List, Dict, Any
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+from models import ExtractedInfo
+from division_hierarchy import get_department_name
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class EmbeddingService:
+    """
+    Service for fast division matching using sentence embeddings.
+
+    How it works:
+    1. At startup: Encode all divisions into vectors (one-time cost)
+    2. For each query: Encode query and find most similar division (fast!)
+
+    Speed: ~50-100ms (vs 4 seconds with LLM)
+    Size: ~150MB (vs 4.8GB with LLM)
+    """
+
+    def __init__(self, model_name: str = "all-MiniLM-L6-v2"):
+        """
+        Initialize the embedding service.
+
+        Args:
+            model_name: Name of the sentence-transformers model to use.
+                       'all-MiniLM-L6-v2' is fast, small (22MB), and accurate.
+        """
+        logger.info(f"Loading embedding model: {model_name}")
+
+        # Load the pre-trained model
+        # This downloads the model on first run (~22MB)
+        self.model = SentenceTransformer(model_name)
+
+        logger.info("Model loaded successfully")
+
+        # Define all 67 divisions with COMPREHENSIVE keywords
+        # Format: (division_name, keywords_for_matching)
+        # Updated with exhaustive keywords for better accuracy
+        self.divisions_data = [
+            # FINANCE
+            ("General Accounting Division", "accounting general ledger bookkeeping financial records journal entries posting reconciliation account balancing closing books financial statements invoices receipts vouchers ledgers trial balance balance sheet accounting errors wrong entries posting mistakes account discrepancies accounting software ERP SAP financial systems submit receipts accounting question ledger issue fiscal year closing"),
+
+            ("Accounts Payable Division", "payable payments vendors bills invoices vendor payments bill processing payment approval invoice matching payment scheduling purchase orders vendor invoices payment vouchers remittance advice late payment vendor complaint unpaid invoice payment delay wrong payment payment systems vendor portal invoice processing pay vendor supplier payment creditors outstanding payments payment terms net 30 payment run"),
+
+            ("Accounts Receivable Division", "receivable collections revenue income customer payments invoicing customers collecting payments revenue recognition aging reports dunning sales invoices receipts credit memos statements of account collection letters unpaid invoice overdue payment customer not paying bad debt payment dispute billing system collection software customer portal debtors outstanding invoices aging report payment terms cash collection DSO"),
+
+            ("Financial Reporting & Control Division", "reporting control financial statements disclosure compliance monthly closing financial reporting variance analysis budget vs actual consolidation P&L income statement balance sheet cash flow statement management reports board reports reporting error wrong numbers financial discrepancy statement correction reporting software BI tools consolidation systems Excel models financial report monthly statements quarterly results budget variance GAAP IFRS financial controls internal controls SOX compliance reconciliation"),
+
+            ("Audit & Financial Analysis", "audit analysis review financial review internal audit financial audit account analysis variance investigation trend analysis ratio analysis audit reports findings recommendations analytical reports management letters audit finding control weakness financial irregularity discrepancy investigation audit software data analytics tools sampling tools need financial analysis audit request investigate transaction variance explanation external audit internal audit SOX testing control testing substantive testing"),
+
+            # INFORMATION TECHNOLOGY
+            ("IT Governance & Quality Division", "IT governance quality standards policies IT strategy IT planning governance framework quality assurance standards compliance IT policies procedures standards governance framework compliance reports policy violation non-compliance governance issue quality problem governance platforms policy management tools compliance software IT policy question IT standards governance requirement quality issue ITIL COBIT ISO 27001 IT framework service management change management IT department information technology IT division policy review IT controls IT quality"),
+
+            ("Applications Development & Integrations", "development apps integration software programming coding software development app building system integration API development custom development requirements design docs technical specs user stories sprint plans app not working software bug integration failure API issue development request IDE Git DevOps tools integration platforms development frameworks need new app software development integrate systems build application API connection agile scrum CI/CD microservices REST API web services mobile app web app IT department information technology IT division app dev application development software engineering technical development IT development IT team"),
+
+            ("IT Infrastructure & Operations Div.", "infrastructure servers operations IT systems hardware server management network operations system monitoring infrastructure maintenance capacity planning system docs network diagrams runbooks incident reports change requests server down network problem system slow hardware failure connectivity issue mouse broken keyboard not working monitor problem printer issue laptop problem desktop issue equipment malfunction computer broken screen not working monitoring tools server management network tools ticketing system server not working network down system issue hardware problem my mouse is broken keyboard not responding monitor not working printer offline laptop won't start computer broken data center cloud infrastructure virtualization VMware storage SAN NAS backup disaster recovery computers laptops desktops monitors keyboards mice printers scanners docking stations cables adapters IT department information technology IT operations system controls system performance performance issues slow system data management database management IT infrastructure IT team system monitoring monitoring systems technical infrastructure"),
+
+            ("Applications Maintenance & Support Div.", "maintenance support help desk application support software support incident management problem resolution ticket handling user support application fixes tickets incident reports knowledge base articles support documentation app not working software error login problem access issue system crash user can't login ticketing system remote support tools monitoring tools help desk software app not working software problem need help can't login error message application support service desk L1 support L2 support incident problem service request bug fix IT department information technology IT support IT help technical support IT assistance application support IT team need help with IT IT issues technical issues IT helpdesk support desk"),
+
+            ("IT Security Implementation & Operations", "security implementation IT security cyber protection information security security monitoring threat detection vulnerability management security implementation access control security policies incident reports vulnerability assessments security logs security breach hacked account virus malware suspicious activity phishing email password reset firewall antivirus SIEM IDS IPS security tools encryption my account is hacked security problem virus on computer suspicious email password issue can't access system cybersecurity InfoSec data protection encryption authentication authorization MFA SSO identity management IT department information technology IT security cyber security cybersecurity security team security division access issues access control access management security incident security breach IT security team"),
+
+            ("IT Manager's Office", "IT manager IT leadership IT management technology management IT strategy IT planning resource management vendor management IT budget IT plans strategy documents budget reports vendor contracts escalation IT complaint strategic IT question vendor issue speak to IT manager IT strategy IT escalation major IT issue IT director CIO IT leadership technology roadmap digital transformation"),
+
+            ("Enterprise Architecture Team", "architecture enterprise architecture IT design system architecture architecture design system planning technology standards solution architecture enterprise planning architecture diagrams blueprints technical standards architecture principles architecture question design review technical standards system design system architecture enterprise design technical architecture solution design EA TOGAF architecture framework reference architecture technology stack design patterns"),
+
+            ("Reporting & Data Analytics Unit", "analytics data reporting insights business intelligence data analysis report generation dashboard creation data visualization predictive analytics reports dashboards analytics KPI reports data insights visualizations report error wrong data dashboard not working data quality issue BI tools Tableau Power BI SQL data warehouse analytics platforms need a report data analysis dashboard KPI report business intelligence data insights big data data mining machine learning data science ETL data warehouse OLAP IT department information technology data analytics data team data management data systems"),
+
+            # HUMAN RESOURCES
+            ("Rewards & Hr Operations Division", "rewards HR operations salary compensation payroll benefits payroll processing salary administration benefits management compensation planning bonus calculation payslips salary letters benefits statements payroll reports tax documents salary delay payroll error wrong salary salary not received benefits issue bonus question pay stub problem my salary is delayed salary not received payroll error wrong amount paid benefits question bonus calculation pay raise HRIS payroll system benefits platform time and attendance total rewards variable pay incentives stock options pension health insurance leave balance"),
+
+            ("HR Business Partner", "HR business partner HRBP employee relations HR support employee support HR consultation performance management employee issues HR advice employee files HR policies performance docs employee relations cases employee complaint HR question manager support employee issue workplace problem HRIS case management HR portal HR help employee issue manager question HR advice workplace problem employee engagement organizational development change management workforce planning HR department human resources HR division HR team employee performance performance issues performance review performance management HR help HR support need HR assistance"),
+
+            ("Talent Development Division", "talent development training learning development employee growth training programs skills development career development learning paths competency building training materials course catalogs development plans training records certificates training request course enrollment development opportunity skill gap LMS e-learning platforms training management systems training course learning opportunity development program skills training professional development L&D upskilling reskilling leadership development technical training soft skills certifications"),
+
+            ("Od & Talent Acquisition Division", "acquisition recruitment hiring talent acquisition jobs candidates recruiting hiring candidate sourcing interviewing job posting onboarding job descriptions CVs resumes offer letters recruitment reports hiring request job opening recruitment question candidate issue onboarding problem ATS recruitment platforms LinkedIn job boards need to hire job opening recruitment new position hiring process interview candidate talent acquisition sourcing headhunting employer branding candidate experience assessment"),
+
+            ("Org.Culture & Initiatives Division", "culture initiatives employee engagement organizational culture workplace culture engagement programs culture building employee initiatives recognition programs wellness programs engagement surveys culture reports initiative proposals recognition materials engagement issue culture problem initiative request employee morale employee engagement culture initiative recognition program employee wellness team building employee experience values mission vision culture transformation employee satisfaction DEI"),
+
+            # LEGAL
+            ("Legal Agreements and Consultancy Division", "legal agreements consultancy legal advice contracts consultation legal review contract drafting legal consultation agreement negotiation legal advice contracts agreements legal opinions legal memos NDAs MOUs legal question contract review needed agreement issue legal advice request need legal advice contract review legal question agreement help legal consultation legal counsel commercial law corporate law legal advisory contract law"),
+
+            ("Contracts & Mortgages and Guarantees Division", "contracts mortgages guarantees security collateral mortgage management guarantee processing contract administration collateral management security documentation mortgage documents guarantee letters security agreements pledge documents collateral records mortgage question guarantee issue security problem contract question mortgage document guarantee letter security agreement collateral question pledge loan security real estate property liens hypothecation financial guarantees"),
+
+            ("Cases Division", "cases litigation legal cases lawsuits disputes court litigation management case handling dispute resolution court proceedings legal claims lawsuits court filings legal claims case files judgments settlements legal case lawsuit dispute court matter legal claim legal case lawsuit dispute court matter litigation legal claim arbitration litigation arbitration mediation legal proceedings court case claims management judgments"),
+
+            # COMMUNICATION
+            ("Public Relations & Media Division", "PR media public relations press communications announcements media relations press releases public announcements media monitoring crisis communication press releases media statements communication materials press kits media inquiry press question public statement needed PR crisis media request press release public announcement PR question media inquiry corporate communications external communications media coverage press conference spokesperson"),
+
+            ("Internal Communication Unit", "internal communication employee communication announcements internal comms employee messaging internal announcements town halls newsletters intranet management internal memos newsletters announcements employee updates town hall materials communication request announcement needed employee messaging internal news employee announcement internal communication newsletter staff message company news employee communications change communications internal messaging intranet employee engagement"),
+
+            # BUSINESS DEVELOPMENT
+            ("Marketing Division", "marketing branding promotion advertising campaigns brand management marketing campaigns market promotion advertising digital marketing content marketing marketing plans campaign materials brand guidelines promotional materials marketing reports marketing request branding question campaign support promotional material marketing support branding promotion advertising marketing campaign social media digital marketing SEO SEM social media marketing content marketing email marketing events"),
+
+            ("Partnerships Dev&Advisory Services Div", "partnerships advisory business partnerships strategic partnerships alliances partnership development alliance management collaboration partnership agreements joint ventures partnership agreements MOU collaboration agreements partnership proposals partnership opportunity collaboration request alliance question partnership opportunity business collaboration strategic alliance joint venture partnership strategic partnerships business alliances co-marketing channel partnerships ecosystem"),
+
+            # STRATEGIC PLANNING
+            ("Strategy Division", "strategy strategic planning business strategy corporate strategy strategic planning strategy development business planning strategic initiatives roadmap planning strategic plans business plans strategy documents roadmaps strategic initiatives strategy question strategic planning business direction strategic planning business strategy strategic initiative corporate strategy long-term planning corporate strategy business model strategic objectives strategic goals vision mission"),
+
+            ("Corporate Performance Division", "performance corporate performance KPIs metrics objectives performance management KPI tracking metrics monitoring objective setting performance review KPI reports scorecards performance dashboards objective tracking performance reviews performance tracking KPI question metrics issue objective setting KPI tracking performance metrics corporate performance objectives scorecards BSC balanced scorecard OKRs performance indicators strategic metrics targets goals"),
+
+            ("Knowledge Management Unit", "knowledge management information management documentation knowledge sharing knowledge capture documentation knowledge sharing content management knowledge repository knowledge base documentation procedures best practices lessons learned documentation request knowledge access information search knowledge base documentation procedure how to best practices process documentation knowledge base wiki document management content management information architecture"),
+
+            # PROJECTS
+            ("Project Management Division", "project management projects PMO project planning project execution project planning project execution project monitoring project control project delivery project plans schedules WBS project charters status reports project documentation project issue project delay project question project planning project management project planning PMO project delivery project status project delivery PMO PMP project lifecycle project portfolio agile waterfall project governance"),
+
+            ("Projects Consultancy Division", "projects consultancy project advisory project consultation project services project consulting advisory services project expertise project guidance project assessment consulting reports project assessments advisory recommendations project studies project consultation needed project advice expert guidance project consultation project advisory project expert project guidance project assessment consulting advisory project expertise technical assistance project evaluation"),
+
+            ("Construction Consultancy Division", "construction building construction consultancy construction projects engineering construction planning building supervision construction management site supervision construction quality construction plans building specs drawings blueprints construction reports inspection reports construction issue building problem site question construction quality construction project building construction construction supervision site management engineering civil engineering structural engineering construction management building codes specifications BOQ"),
+
+            # MARKET RESEARCH
+            ("Market Research Div.", "market research research market analysis market intelligence market research data collection market surveys research analysis market intelligence research reports market studies survey results research findings market analysis research request market data needed research question market research market data research study market analysis market intelligence market trends market research surveys focus groups market data industry analysis competitive intelligence"),
+
+            ("Market Studies Division", "market studies industry analysis market assessment market evaluation market studies industry research market assessment feasibility studies market evaluation market study reports industry reports assessment reports feasibility studies market study request industry analysis needed feasibility question market study industry analysis market assessment feasibility study market evaluation industry research market sizing market segmentation market opportunity market attractiveness"),
+
+            ("Business Intelligence Div.", "business intelligence market intelligence competitive intelligence market insights intelligence gathering competitor analysis market monitoring intelligence reporting intelligence reports competitor profiles market briefings intelligence updates intelligence request competitor information market intelligence business intelligence competitor analysis market intelligence competitive intelligence market insights CI competitive analysis market monitoring industry intelligence strategic intelligence"),
+
+            # FACILITIES
+            ("Facilities Management Division", "facilities building management office management facilities services facility maintenance building operations space management facility services office management maintenance schedules facility reports work orders service requests facility problem building issue office space maintenance request AC not working lights not working office equipment furniture request space issue cleaning request AC not working office too cold lights broken need furniture office space facility maintenance building problem room too hot chair broken desk request CMMS facility management software work order systems HVAC air conditioning heating ventilation lighting furniture office supplies workspace meeting rooms office equipment janitorial cleaning waste management"),
+
+            ("Documents and Administrative Communications Center", "documents administrative communications center document management records document processing records management filing archiving document distribution mail handling official documents correspondence letters memos records archives document request filing question records access document retrieval need a document official letter document filing records request archive access correspondence records management archiving filing system document control correspondence management official documents document filing document archiving paper documents document center administrative documents"),
+
+            ("Security and Safety Division", "security safety protection guards access control security operations access control visitor management security monitoring emergency response security reports incident reports visitor logs access logs security procedures security incident lost badge access card problem visitor escort security concern suspicious activity emergency safety hazard lost my badge access card not working need visitor pass security escort security incident emergency safety concern suspicious person physical security badge ID card access card visitor management CCTV guards patrols emergency procedures evacuation physical security building security premises security facility security incident report security event safety incident safety event"),
+
+            ("Industrial Safety & Loss Prevention Div", "industrial safety loss prevention occupational safety workplace safety HSE safety management risk assessment safety inspections incident investigation loss prevention safety reports incident reports risk assessments safety procedures investigation reports safety incident accident injury safety hazard unsafe condition near miss safety incident workplace accident safety hazard injury report unsafe condition PPE safety training HSE occupational health safety compliance OSHA safety regulations PPE accident prevention safety training"),
+
+            # PURCHASING
+            ("Procurements Contracts and Vendors Division", "procurement contracts vendors suppliers purchasing sourcing procurement vendor management contract management supplier management sourcing tendering purchase orders contracts RFP RFQ vendor agreements tender documents vendor issue contract question procurement request supplier problem procurement vendor contract supplier agreement tender RFP purchase order sourcing strategic sourcing supplier relationship contract negotiation vendor evaluation procurement process"),
+
+            ("Purchasing Division", "purchasing buying procurement purchase orders requisitions purchasing buying order processing purchase requisitions goods receipt purchasing approval purchase orders requisitions quotes purchase requests delivery notes purchase request buying question order status delivery problem purchase approval need to buy purchase request order status buying approval purchase something get a quote purchase requisition PO purchase approval ordering buying process goods receipt"),
+
+            # GOVERNANCE
+            ("Enterprise Governance Division", "enterprise governance governance policies corporate governance board governance governance framework policy development board governance corporate governance governance compliance policies governance framework board papers governance reports charters governance question policy issue compliance question governance requirement governance corporate policies board governance governance framework governance compliance corporate governance board of directors governance structure governance best practices King IV"),
+
+            ("Compliance Regulatory Division", "regulatory compliance regulations regulatory compliance legal compliance regulatory compliance regulation monitoring compliance assessment regulatory reporting compliance review compliance reports regulatory filings compliance certificates regulatory updates regulatory requirement compliance question regulation change compliance issue regulatory compliance regulations compliance requirement regulatory filing compliance report regulatory requirements industry regulations compliance standards regulatory framework regulatory authorities"),
+
+            ("Compliance Operations Division", "compliance operations compliance monitoring compliance management compliance controls compliance monitoring compliance testing compliance controls compliance verification compliance tracking compliance reports monitoring reports compliance checklists test results compliance logs compliance breach control failure compliance issue monitoring finding compliance monitoring compliance testing compliance controls compliance verification compliance tracking compliance program compliance testing monitoring controls compliance assurance compliance activities"),
+
+            ("Operational Risk Management Division", "operational risk risk management operational risk management risk controls risk identification risk assessment risk mitigation control testing risk monitoring risk registers risk assessments control matrices risk reports mitigation plans operational risk risk event control failure risk issue operational failure operational risk risk management risk assessment control issue risk event operational failure risk framework COSO ERM risk appetite risk tolerance key risk indicators KRIs system controls controls management risk control internal controls control framework assessment needed risk assessment risk division risk department risk team control systems controls review"),
+
+            ("Financial Risk Management Division", "financial risk risk management credit risk market risk liquidity risk financial risk assessment risk modeling stress testing scenario analysis risk measurement risk reports stress test results risk models VaR reports risk metrics financial risk market risk credit risk event liquidity issue financial risk market risk credit risk liquidity risk risk modeling stress testing Basel capital adequacy VaR credit risk market risk liquidity risk ALM risk metrics"),
+
+            ("Cybersecurity Governance Unit", "cybersecurity governance security governance information security governance security policies security governance security policy development security framework security standards security compliance security policies security framework security standards security guidelines security charters security policy question security governance security framework security compliance cybersecurity governance security policies security framework information security governance security standards ISO 27001 NIST security framework security governance information security management ISMS"),
+
+            ("Cybersecurity Defense Unit", "cybersecurity defense security operations cyber defense threat detection security monitoring threat monitoring incident response security operations threat hunting vulnerability management security incidents threat reports incident response plans security alerts IOCs cyber attack security breach hacked malware virus phishing ransomware data breach security incident cyber attack hacked virus phishing email ransomware security incident data breach malware suspicious email SOC security operations center threat intelligence incident response cyber threats APT zero-day exploit vulnerability"),
+
+            # CREDIT
+            ("Credit Risk Division", "credit risk credit assessment credit evaluation credit analysis default risk credit assessment credit evaluation credit scoring default probability credit rating credit review credit reports credit assessments credit scores rating reports credit analysis credit risk concern default risk credit quality credit deterioration credit risk credit assessment credit evaluation default risk credit quality credit rating PD LGD EAD credit scoring credit rating credit underwriting NPL non-performing loans"),
+
+            ("Credit Relationships Division", "credit relationships customer relations client management customer service account management customer relationship management client servicing account management customer support relationship building customer profiles relationship reports service records customer communications customer complaint client issue relationship problem customer service customer service client support customer complaint account manager relationship manager customer issue CRM customer relationship client servicing relationship manager account management customer satisfaction"),
+
+            ("Credit Control Team - A", "credit control collections monitoring credit monitoring team A credit monitoring collection activities payment follow-up credit limits exposure monitoring collection reports payment schedules credit memos monitoring reports dunning letters overdue payment collection issue credit limit payment delay delinquency credit control collections overdue payment payment monitoring credit limit collection collections receivables management credit monitoring payment tracking delinquency write-off"),
+
+            ("Credit Control Team - B", "credit control collections monitoring credit monitoring team B credit monitoring collection activities payment follow-up credit limits exposure monitoring collection reports payment schedules credit memos monitoring reports dunning letters overdue payment collection issue credit limit payment delay delinquency credit control collections overdue payment payment monitoring credit limit collection collections receivables management credit monitoring payment tracking delinquency write-off"),
+
+            ("Collection Team", "collection collections recovery debt collection payment recovery debt collection payment recovery collection activities recovery process collection calls collection letters payment plans recovery reports collection logs non-payment collection case recovery issue difficult customer debt collection payment recovery collection recover payment non-paying customer collections debt recovery recovery payment collection delinquent accounts bad debt"),
+
+            ("Loan Follow-Up Team", "loan follow-up loan monitoring loan servicing loan administration loan monitoring loan servicing payment follow-up loan administration loan review loan files payment schedules loan reports servicing records loan reviews loan payment loan question payment issue loan servicing loan problem loan payment loan help loan issue loan servicing loan monitoring loan question payment schedule loan servicing loan administration loan payments loan portfolio loan monitoring loan review disbursement"),
+
+            ("C&Lm Info Team", "credit info information credit information credit data information management credit data information services data management credit information credit reports information reports data files credit information information request credit data information access credit information credit data information request credit reports data access credit bureau credit information credit data information services"),
+
+            # PORTFOLIO
+            ("Portfolio Manager'S Office", "portfolio manager portfolio management office portfolio leadership portfolio management portfolio strategy portfolio oversight portfolio governance portfolio reports portfolio strategy portfolio reviews management reports portfolio question portfolio strategy portfolio oversight portfolio manager portfolio strategy portfolio management portfolio oversight portfolio governance portfolio leadership portfolio strategy asset management"),
+
+            ("Portfolio Division", "portfolio portfolio management asset management investment portfolio portfolio management asset management portfolio monitoring portfolio optimization portfolio reporting portfolio reports asset statements portfolio analysis performance reports portfolio question asset issue portfolio performance investment question portfolio asset management portfolio performance investment portfolio portfolio monitoring my portfolio portfolio management asset allocation investment management portfolio optimization asset management portfolio performance"),
+
+            ("Special Assets Division", "special assets distressed assets problem assets asset recovery asset recovery workout restructuring distressed asset management asset resolution workout plans restructuring plans asset reports recovery reports problem asset distressed loan asset recovery troubled asset special assets problem asset distressed asset asset recovery workout restructuring NPL management asset resolution workout restructuring distressed debt asset recovery"),
+
+            # EXCELLENCE
+            ("Programs Delivery Operations Division", "programs delivery operations program execution program operations program delivery program execution operational delivery program operations program implementation program reports delivery schedules operational reports program status program delivery operational issue program execution delivery problem program delivery program operations program execution delivery operations program implementation program management program delivery operational excellence program execution"),
+
+            ("Customer Excellence Division", "customer excellence customer service customer experience customer satisfaction customer service customer experience management service excellence customer complaints customer feedback service quality customer feedback service reports satisfaction surveys complaint reports NPS reports customer complaint service issue poor service customer dissatisfaction complaint bad experience customer complaint service problem poor service complaint customer service not satisfied bad experience complaint about service customer satisfaction NPS customer experience CX service quality customer feedback complaints handling customer care"),
+
+            ("Programs Design And Needs Assessment Division", "programs design needs assessment program development program planning program design needs analysis program planning program development requirements gathering needs assessments program designs requirements documents program proposals program design needs assessment program planning development request program design needs assessment program development program planning requirements analysis program development needs analysis program design program planning feasibility"),
+
+            ("Programs Evaluation & Quality Assurance Division", "programs evaluation quality assurance program assessment QA program review program evaluation quality assurance program assessment program review performance evaluation evaluation reports QA reports assessment reports review reports program audits program evaluation quality issue assessment request program review program evaluation quality assurance program assessment QA program review program quality program evaluation quality control QA quality assurance program effectiveness program impact"),
+
+            ("Product Dev For National Priorities Div", "product development national priorities new products product innovation product development new product creation product innovation product design national programs product proposals product specs development plans product roadmaps new product product development product idea innovation request new product product development product innovation develop new product national priorities product idea product management product innovation new products product design product strategy national initiatives"),
+
+            ("Product Dev For Customer Empowerment Div", "product development customer empowerment customer products customer-focused products customer product development customer-centric design product innovation customer empowerment programs product proposals customer research product specs customer feedback customer product product for customers customer empowerment customer-focused development customer products customer empowerment customer-focused products products for customers customer innovation customer empowerment customer-centric products customer solutions customer value"),
+
+            ("Solutions & Design Division", "solutions design solution design solution development solutions architecture solution design solution development solution architecture solution delivery design thinking solution designs design documents solution proposals solution blueprints solution design solution request design question solution development solution design solution development design solution solutions custom solution solution architecture design thinking solution engineering solution delivery custom solutions"),
+
+            # AUDIT
+            ("Operations Audit Team", "operations audit operational audit audit internal audit operational review operational auditing process audit operational review compliance audit operational assessment audit reports audit findings audit recommendations operational audit reports audit request operational audit audit finding audit question operations audit operational audit audit request internal audit operational review internal audit operational audit process audit compliance audit operational controls"),
+
+            # ACADEMY
+            ("Academy Strategic Partnerships Division", "academy training education learning development partnerships training education programs learning professional development courses workshops certifications partnerships course catalogs training materials certificates learning plans partnership agreements training request course enrollment certification learning opportunity education program training education learning courses workshop certification professional development academy programs training courses SIDF Academy corporate university training programs learning center professional development certifications workshops seminars e-learning leadership development"),
+
+            # EXECUTIVE
+            ("CEO Office", "CEO chief executive executive office CEO office leadership executive management strategic leadership executive decisions CEO communications executive governance executive reports board papers CEO communications strategic documents executive escalation CEO office executive question strategic matter CEO executive office chief executive CEO office executive escalation top management C-suite executive leadership CEO chief executive officer executive management strategic leadership"),
+        ]
+
+        # Extract just the division names and search texts
+        self.division_names = [div[0] for div in self.divisions_data]
+        self.division_search_texts = [
+            f"{div[0]} {div[1]}" for div in self.divisions_data
+        ]
+
+        logger.info(f"Pre-encoding {len(self.division_names)} divisions...")
+
+        # PRE-ENCODE all divisions (this is the magic!)
+        # This happens once at startup, then queries are super fast
+        self.division_embeddings = self.model.encode(
+            self.division_search_texts,
+            convert_to_numpy=True,
+            show_progress_bar=True
+        )
+
+        logger.info(f"✓ Encoded {len(self.division_names)} divisions")
+        logger.info(f"Embedding shape: {self.division_embeddings.shape}")
+
+        # ALSO ENCODE DEPARTMENTS
+        # Get unique departments and build department → divisions mapping
+        from contacts_data import get_all_contacts
+        from collections import defaultdict
+
+        contacts = get_all_contacts()
+        dept_to_divisions = defaultdict(set)
+
+        for contact in contacts:
+            dept_to_divisions[contact["department"]].add(contact["division"])
+
+        self.dept_to_divisions = {dept: list(divs) for dept, divs in dept_to_divisions.items()}
+        self.department_names = list(self.dept_to_divisions.keys())
+
+        # Create search texts for departments (department name + common keywords)
+        self.department_search_texts = []
+        for dept in self.department_names:
+            # Add department name and common keywords
+            search_text = f"{dept} department team group unit"
+            self.department_search_texts.append(search_text)
+
+        logger.info(f"Pre-encoding {len(self.department_names)} departments...")
+        self.department_embeddings = self.model.encode(
+            self.department_search_texts,
+            convert_to_numpy=True,
+            show_progress_bar=False
+        )
+
+        logger.info(f"✓ Encoded {len(self.department_names)} departments")
+        logger.info("EmbeddingService ready!")
+
+    def find_division(self, query: str, top_k: int = 3) -> List[ExtractedInfo]:
+        """
+        Find the best matching divisions for a query.
+        Also checks department-level matches and expands them to divisions.
+
+        Args:
+            query: User's search query (e.g., "app development", "HR help", "Information Technology")
+            top_k: Number of top matches to return (default: 3)
+
+        Returns:
+            List of ExtractedInfo objects with division and confidence scores
+
+        How it works:
+        1. Encode the query into a vector (fast: ~10ms)
+        2. Calculate similarity with all division vectors (fast: ~5ms)
+        3. ALSO calculate similarity with all department vectors
+        4. If department match is stronger, expand to all divisions in that department
+        5. Return top matches sorted by similarity score
+
+        Total time: ~15-50ms
+        """
+        logger.info(f"Processing query: {query}")
+
+        # STEP 1: Encode the query
+        query_embedding = self.model.encode([query], convert_to_numpy=True)
+
+        # STEP 2: Calculate similarity with all divisions
+        division_similarities = cosine_similarity(
+            query_embedding,
+            self.division_embeddings
+        )[0]
+
+        # STEP 2B: ALSO calculate similarity with departments
+        department_similarities = cosine_similarity(
+            query_embedding,
+            self.department_embeddings
+        )[0]
+
+        # STEP 3: Check if any department has significantly better match than divisions
+        best_division_similarity = np.max(division_similarities)
+        best_dept_similarity = np.max(department_similarities)
+
+        # Convert to confidence (0-1 range)
+        best_division_conf = (best_division_similarity + 1) / 2
+        best_dept_conf = (best_dept_similarity + 1) / 2
+
+        logger.info(f"Best division match confidence: {best_division_conf:.2f}")
+        logger.info(f"Best department match confidence: {best_dept_conf:.2f}")
+
+        results = []
+
+        # If department match is better (even slightly >= 0.01 higher), use department
+        # Lower threshold (0.01 instead of 0.05) to catch department-level queries
+        # e.g., "Information Technology" should expand to all IT divisions
+        if best_dept_conf > best_division_conf and (best_dept_conf - best_division_conf) >= 0.01:
+            best_dept_idx = np.argmax(department_similarities)
+            dept_name = self.department_names[best_dept_idx]
+            divisions_in_dept = self.dept_to_divisions[dept_name]
+
+            logger.info(f"✓ Department match: {dept_name} ({best_dept_conf:.2f}) - Expanding to ALL {len(divisions_in_dept)} divisions")
+
+            # Return ALL divisions in this department with the department's confidence
+            # This ensures contact search can find people across all divisions in the department
+            for division_name in divisions_in_dept:
+                results.append(ExtractedInfo(
+                    division=division_name,
+                    department=dept_name,
+                    confidence=round(best_dept_conf, 2)
+                ))
+
+                logger.info(f"  - {division_name} (dept match, confidence: {best_dept_conf:.2f})")
+
+        else:
+            # Use regular division matching
+            top_indices = np.argsort(division_similarities)[::-1][:top_k]
+
+            for idx in top_indices:
+                division_name = self.division_names[idx]
+                similarity_score = float(division_similarities[idx])
+                confidence = (similarity_score + 1) / 2
+
+                # Get parent department
+                department_name = get_department_name(division_name)
+
+                results.append(ExtractedInfo(
+                    division=division_name,
+                    department=department_name,
+                    confidence=round(confidence, 2)
+                ))
+
+                logger.info(
+                    f"Match: {division_name} [{department_name}] "
+                    f"(similarity: {similarity_score:.3f}, "
+                    f"confidence: {confidence:.2f})"
+                )
+
+        logger.info(f"✓ Found {len(results)} matches")
+        return results

name_extraction_service.py

@@ -0,0 +1,181 @@
+# name_extraction_service.py
+import logging
+from typing import List, Dict, Any, Optional
+from gliner import GLiNER
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class NameExtractor:
+    """
+    Service for extracting person names from text using GLiNER.
+
+    GLiNER is a zero-shot NER model that can extract entities without
+    being limited to predefined entity types. It's especially good for:
+    - Multilingual name extraction (English + Arabic)
+    - Flexible entity extraction
+    - Lightweight and fast (~100-200ms)
+
+    Size: ~150MB model
+    Speed: ~100-200ms per query
+    """
+
+    def __init__(self, model_name: str = "urchade/gliner_small-v2.1"):
+        """
+        Initialize the name extraction service.
+
+        Args:
+            model_name: GLiNER model to use. Options:
+                - "urchade/gliner_small-v2.1" (150MB, balanced)
+                - "urchade/gliner_multi-v2.1" (multilingual, better for Arabic)
+                - "urchade/gliner_large-v2.1" (larger, more accurate)
+        """
+        logger.info(f"Loading GLiNER model: {model_name}")
+
+        # Load the pre-trained model
+        # This downloads the model on first run (~150MB)
+        self.model = GLiNER.from_pretrained(model_name)
+
+        # Define the entity labels we want to extract
+        self.labels = ["person", "name", "employee"]
+
+        logger.info(f"✓ GLiNER model loaded successfully")
+        logger.info(f"Entity labels: {self.labels}")
+
+    def extract_names(self, text: str, threshold: float = 0.3) -> List[str]:
+        """
+        Extract person names from text.
+
+        Args:
+            text: Input text (e.g., "find Ahmed in IT")
+            threshold: Confidence threshold (0-1). Lower = more names but less precise.
+                      Default 0.3 is good for most cases.
+
+        Returns:
+            List of extracted names
+
+        Example:
+            >>> extractor.extract_names("find Ahmed Hassan in IT")
+            ['Ahmed Hassan']
+
+            >>> extractor.extract_names("connect me with Sarah from HR")
+            ['Sarah']
+        """
+        logger.info(f"Extracting names from: {text}")
+
+        # Predict entities using GLiNER
+        entities = self.model.predict_entities(
+            text,
+            self.labels,
+            threshold=threshold
+        )
+
+        # Extract just the text of person entities
+        names = [entity["text"] for entity in entities]
+
+        # Remove duplicates while preserving order
+        unique_names = list(dict.fromkeys(names))
+
+        logger.info(f"✓ Found {len(unique_names)} name(s): {unique_names}")
+
+        return unique_names
+
+    def extract_names_with_context(
+        self,
+        text: str,
+        threshold: float = 0.3
+    ) -> List[Dict[str, Any]]:
+        """
+        Extract person names with additional context (position, confidence).
+
+        Args:
+            text: Input text
+            threshold: Confidence threshold (0-1)
+
+        Returns:
+            List of dictionaries with name details:
+            [
+                {
+                    "name": "Ahmed Hassan",
+                    "start": 5,
+                    "end": 17,
+                    "confidence": 0.95,
+                    "label": "person"
+                }
+            ]
+        """
+        logger.info(f"Extracting names with context from: {text}")
+
+        # Predict entities
+        entities = self.model.predict_entities(
+            text,
+            self.labels,
+            threshold=threshold
+        )
+
+        # Format results
+        results = []
+        for entity in entities:
+            results.append({
+                "name": entity["text"],
+                "start": entity["start"],
+                "end": entity["end"],
+                "confidence": round(entity["score"], 2),
+                "label": entity["label"]
+            })
+
+        logger.info(f"✓ Found {len(results)} name(s) with context")
+
+        return results
+
+    def extract_from_query(
+        self,
+        query: str,
+        extract_divisions: bool = False
+    ) -> Dict[str, Any]:
+        """
+        Extract names and optionally division keywords from a query.
+
+        Args:
+            query: User query text
+            extract_divisions: Whether to also extract division/department mentions
+
+        Returns:
+            Dictionary with extracted information:
+            {
+                "names": ["Ahmed", "Sarah"],
+                "has_names": True,
+                "count": 2,
+                "divisions": ["IT", "HR"] (if extract_divisions=True)
+            }
+        """
+        # Extract names
+        names = self.extract_names(query)
+
+        result = {
+            "names": names,
+            "has_names": len(names) > 0,
+            "count": len(names)
+        }
+
+        # Optionally extract division keywords
+        if extract_divisions:
+            # Common division/department keywords
+            division_keywords = [
+                "IT", "HR", "Finance", "Legal", "Accounting",
+                "Marketing", "Sales", "Operations", "Engineering",
+                "Security", "Facilities", "Purchasing", "Audit"
+            ]
+
+            query_upper = query.upper()
+            found_divisions = [
+                kw for kw in division_keywords
+                if kw in query_upper
+            ]
+
+            result["divisions"] = found_divisions
+            result["has_divisions"] = len(found_divisions) > 0
+
+        return result

requirements.txt

@@ -0,0 +1,20 @@
+# Hugging Face Spaces Requirements
+# Optimized for Gradio deployment
+
+# Gradio for the web interface
+gradio>=4.0.0
+
+# AI/ML Core Libraries
+sentence-transformers>=2.2.0  # For division matching embeddings
+openai-whisper>=20231117      # For speech-to-text
+gliner>=0.1.0                 # For name extraction
+
+# Scientific Computing
+numpy>=1.24.0
+scipy>=1.10.0                 # For audio file handling
+
+# Text Processing
+torch>=2.0.0                  # Required by models (CPU version for HF Spaces)
+
+# Standard Libraries (usually included but good to specify)
+# Note: logging, os, tempfile, pathlib, typing, difflib, re are standard library

voice_processing_service.py

@@ -0,0 +1,294 @@
+# voice_processing_service.py
+import logging
+import whisper
+import os
+import tempfile
+from typing import Dict, Any, Optional
+from pathlib import Path
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class VoiceProcessor:
+    """
+    Service for processing voice queries with speech-to-text and translation.
+
+    Features:
+    - Speech-to-text using OpenAI Whisper
+    - Automatic language detection
+    - Arabic-to-English translation
+    - Supports 99+ languages
+    - Works offline
+
+    Whisper Model Sizes:
+    - tiny: 39M params, ~1GB RAM, fast but less accurate
+    - base: 74M params, ~1GB RAM, balanced (RECOMMENDED for quick start)
+    - small: 244M params, ~2GB RAM, good accuracy
+    - medium: 769M params, ~5GB RAM, better accuracy
+    - large: 1550M params, ~10GB RAM, best accuracy
+    """
+
+    def __init__(self, model_size: str = "base"):
+        """
+        Initialize the voice processing service.
+
+        Args:
+            model_size: Whisper model to use. Options:
+                - "tiny" (39M) - Fast, less accurate
+                - "base" (74M) - Balanced, recommended for development
+                - "small" (244M) - Good accuracy
+                - "medium" (769M) - Better accuracy
+                - "large" (1550M) - Best accuracy, slowest
+        """
+        logger.info(f"Loading Whisper model: {model_size}")
+        logger.info("This may take a few minutes on first run (downloading model)...")
+
+        # Load Whisper model
+        # This downloads the model on first run
+        self.model = whisper.load_model(model_size)
+
+        self.model_size = model_size
+        logger.info(f"✓ Whisper model '{model_size}' loaded successfully")
+        logger.info(f"Supported languages: 99+ (auto-detected)")
+
+    def transcribe_audio(
+        self,
+        audio_path: str,
+        language: Optional[str] = None
+    ) -> Dict[str, Any]:
+        """
+        Transcribe audio file in its original language.
+
+        Args:
+            audio_path: Path to audio file (mp3, wav, m4a, etc.)
+            language: Optional language code (e.g., "en", "ar"). If None, auto-detect.
+
+        Returns:
+            Dictionary with transcription results:
+            {
+                "text": "transcribed text",
+                "language": "en",
+                "language_name": "English",
+                "confidence": 0.95
+            }
+        """
+        logger.info(f"Transcribing audio: {audio_path}")
+
+        # Transcribe with Whisper
+        result = self.model.transcribe(
+            audio_path,
+            language=language,
+            fp16=False  # Use fp32 for better compatibility
+        )
+
+        transcription = {
+            "text": result["text"].strip(),
+            "language": result["language"],
+            "language_name": self._get_language_name(result["language"]),
+            "confidence": self._calculate_confidence(result)
+        }
+
+        logger.info(f"✓ Transcribed: '{transcription['text'][:100]}...'")
+        logger.info(f"  Language: {transcription['language_name']} ({transcription['language']})")
+        logger.info(f"  Confidence: {transcription['confidence']:.2f}")
+
+        return transcription
+
+    def translate_to_english(self, audio_path: str) -> Dict[str, Any]:
+        """
+        Transcribe audio and translate to English (if not already English).
+
+        This is optimized for the use case where you always want English output,
+        regardless of the input language.
+
+        Args:
+            audio_path: Path to audio file
+
+        Returns:
+            Dictionary with translation results:
+            {
+                "original_text": "النص الأصلي",
+                "english_text": "translated text",
+                "original_language": "ar",
+                "original_language_name": "Arabic",
+                "was_translated": true
+            }
+        """
+        logger.info(f"Processing audio for English output: {audio_path}")
+
+        # First, transcribe in original language to detect it
+        original = self.model.transcribe(audio_path, fp16=False)
+
+        # Then translate to English
+        translated = self.model.transcribe(
+            audio_path,
+            task="translate",  # This translates to English
+            fp16=False
+        )
+
+        result = {
+            "original_text": original["text"].strip(),
+            "english_text": translated["text"].strip(),
+            "original_language": original["language"],
+            "original_language_name": self._get_language_name(original["language"]),
+            "was_translated": original["language"] != "en"
+        }
+
+        if result["was_translated"]:
+            logger.info(f"✓ Detected {result['original_language_name']}, translated to English")
+            logger.info(f"  Original: '{result['original_text'][:100]}...'")
+            logger.info(f"  English: '{result['english_text'][:100]}...'")
+        else:
+            logger.info(f"✓ Already in English, no translation needed")
+
+        return result
+
+    def process_voice_query(self, audio_path: str) -> Dict[str, Any]:
+        """
+        Complete pipeline: transcribe, translate if needed, return query text.
+
+        This is the main method for the voice assistant use case.
+
+        Args:
+            audio_path: Path to audio file
+
+        Returns:
+            Dictionary ready for division extraction:
+            {
+                "query": "english text for processing",
+                "original_text": "original text if different",
+                "language": "ar",
+                "language_name": "Arabic",
+                "was_translated": true,
+                "audio_duration": 5.2
+            }
+        """
+        logger.info(f"Processing voice query: {audio_path}")
+
+        # Get audio duration
+        audio_info = whisper.load_audio(audio_path)
+        duration = len(audio_info) / whisper.audio.SAMPLE_RATE
+
+        # Translate to English (works for all languages)
+        result = self.translate_to_english(audio_path)
+
+        return {
+            "query": result["english_text"],  # Always English for processing
+            "original_text": result["original_text"],
+            "language": result["original_language"],
+            "language_name": result["original_language_name"],
+            "was_translated": result["was_translated"],
+            "audio_duration": round(duration, 2)
+        }
+
+    def _get_language_name(self, lang_code: str) -> str:
+        """Get full language name from code."""
+        language_names = {
+            "en": "English",
+            "ar": "Arabic",
+            "es": "Spanish",
+            "fr": "French",
+            "de": "German",
+            "zh": "Chinese",
+            "ja": "Japanese",
+            "ko": "Korean",
+            "ru": "Russian",
+            "pt": "Portuguese",
+            "it": "Italian",
+            "nl": "Dutch",
+            "tr": "Turkish",
+            "pl": "Polish",
+            "uk": "Ukrainian",
+            "vi": "Vietnamese",
+            "th": "Thai",
+            "hi": "Hindi",
+            "ur": "Urdu",
+            # Add more as needed
+        }
+        return language_names.get(lang_code, lang_code.upper())
+
+    def _calculate_confidence(self, whisper_result: Dict) -> float:
+        """
+        Calculate confidence score from Whisper result.
+
+        Whisper doesn't directly provide confidence, so we estimate it
+        based on available metrics.
+        """
+        # If segments are available, average their probabilities
+        if "segments" in whisper_result and whisper_result["segments"]:
+            avg_logprob = sum(s.get("avg_logprob", -1.0) for s in whisper_result["segments"])
+            avg_logprob /= len(whisper_result["segments"])
+            # Convert log probability to approximate confidence (0-1)
+            # logprob ranges from -inf to 0, typically -2 to 0 for good transcriptions
+            confidence = max(0.0, min(1.0, (avg_logprob + 2.0) / 2.0))
+            return round(confidence, 2)
+
+        # Default confidence
+        return 0.85
+
+    def save_uploaded_audio(self, audio_bytes: bytes, filename: str) -> str:
+        """
+        Save uploaded audio file to temporary location.
+
+        Args:
+            audio_bytes: Audio file bytes
+            filename: Original filename
+
+        Returns:
+            Path to saved file
+        """
+        # Create temp directory if it doesn't exist
+        temp_dir = Path(tempfile.gettempdir()) / "voice_assistant_uploads"
+        temp_dir.mkdir(exist_ok=True)
+
+        # Save file
+        file_extension = Path(filename).suffix
+        temp_file = temp_dir / f"upload_{os.urandom(8).hex()}{file_extension}"
+
+        temp_file.write_bytes(audio_bytes)
+        logger.info(f"Saved uploaded audio to: {temp_file}")
+
+        return str(temp_file)
+
+    def save_audio_array(self, audio_data, sample_rate: int) -> str:
+        """
+        Save audio numpy array to temporary WAV file (for Gradio integration).
+
+        Args:
+            audio_data: Audio data as numpy array
+            sample_rate: Sample rate of the audio
+
+        Returns:
+            Path to saved WAV file
+        """
+        import numpy as np
+        import scipy.io.wavfile as wavfile
+
+        # Create temp directory if it doesn't exist
+        temp_dir = Path(tempfile.gettempdir()) / "voice_assistant_uploads"
+        temp_dir.mkdir(exist_ok=True)
+
+        # Save as WAV file
+        temp_file = temp_dir / f"gradio_{os.urandom(8).hex()}.wav"
+
+        # Ensure audio_data is in the correct format
+        if isinstance(audio_data, np.ndarray):
+            # Normalize to int16 if needed
+            if audio_data.dtype == np.float32 or audio_data.dtype == np.float64:
+                audio_data = (audio_data * 32767).astype(np.int16)
+
+        wavfile.write(str(temp_file), sample_rate, audio_data)
+        logger.info(f"Saved Gradio audio to: {temp_file}")
+
+        return str(temp_file)
+
+    def cleanup_temp_file(self, file_path: str):
+        """Delete temporary audio file."""
+        try:
+            if os.path.exists(file_path):
+                os.remove(file_path)
+                logger.info(f"Cleaned up temp file: {file_path}")
+        except Exception as e:
+            logger.warning(f"Failed to cleanup temp file {file_path}: {e}")