# Sentinel: A to Z Project Report Welcome to the comprehensive report for **Sentinel**, your Multi-Domain Support Triage Agent. This document breaks down the entire project from what it solves to the nitty-gritty of how it is built, making it easy to understand for both technical and non-technical stakeholders. --- ## 🌟 1. What Does Sentinel Solve? Customer support teams at large enterprises face a massive volume of incoming tickets. Many of these tickets are simple, repetitive questions ("How do I reset my password?"), while others are highly sensitive ("My credit card was stolen," or "I found a security vulnerability"). If human agents spend all their time answering the simple questions, they get overwhelmed, and users with urgent, sensitive issues are left waiting. **Sentinel solves this by acting as an AI-powered "Triage Agent."** It sits at the front door of the support system and reads every incoming ticket for three major platforms: **HackerRank, Claude, and Visa**. For every ticket, Sentinel automatically: 1. **Identifies the platform** (Domain Detection). 2. **Scans for sensitive issues** like fraud, jailbreaks, or security threats and instantly escalates them to a human. 3. **Searches a vast knowledge base** (the support corpus) to find the exact policy or answer. 4. **Drafts a professional reply** based *strictly* on the official documentation. 5. **Audits its own reply** to guarantee it didn't make up fake information (hallucinations). --- ## 🏗️ 2. How is it Built? (The Architecture) Sentinel is not just a simple wrapper around ChatGPT. It is a highly robust **Multi-Agent State Machine**. This means the work is divided among four distinct, specialized "agents," each handling a specific part of the pipeline. ### Agent 1: The Bouncer (Triage) - **Role:** Security and Routing. - **How it works:** Before any expensive AI is used, The Bouncer runs the ticket through a fast, rule-based engine. It uses Regular Expressions (Regex) and Keyword Matching to detect the company (HackerRank/Claude/Visa). It also checks for "Escalation Triggers" like prompt injections (e.g., "ignore all previous instructions"), threats, or fraud. - **Why it matters:** It is deterministic and incredibly fast (<5ms), ensuring safety and saving API costs. ### Agent 2: The Librarian (Retrieval / RAG) - **Role:** Finding the Ground Truth. - **How it works:** The Librarian searches the offline Markdown support corpus. It uses a highly advanced **Hybrid Search**: - **FAISS (Dense Vector Search):** Understands the "meaning" of the ticket using AI embeddings. - **BM25 (Sparse Keyword Search):** Finds exact keyword matches like error codes or specific product names. - It then merges both searches using an algorithm called **Reciprocal Rank Fusion (RRF)** to get the absolute most relevant documents. ### Agent 3: The Spokesperson (Generation) - **Role:** Writing the Reply. - **How it works:** This agent takes the ticket and the documents found by The Librarian and uses the **Gemini 1.5 Flash** Large Language Model (LLM) to write a response. - **Chain of Thought:** It is forced to write out its internal reasoning in a "scratchpad" (citing exact rules) *before* writing the final response. It also categorizes the ticket's `product_area` and `request_type`. ### Agent 4: The Auditor (Guardrails) - **Role:** Hallucination Prevention. - **How it works:** LLMs sometimes make things up. The Auditor reads the Spokesperson's draft, extracts concrete factual claims (like URLs, phone numbers, and procedural instructions), and uses fuzzy string matching (`rapidfuzz`) to check if those claims actually exist in the official corpus. - **The Verdict:** If the claims aren't in the corpus, The Auditor strips the fake information out or immediately escalates the ticket to a human. --- ## 💻 3. Technology Stack Sentinel is built using a modern, lightweight, and incredibly fast Python stack. ### Core Technologies - **Language:** Python 3 - **LLM Provider:** Google Gemini API (`google-generativeai`). Specifically using `gemini-1.5-flash` for high speed, low cost, and strong reasoning capabilities. - **Embeddings:** `sentence-transformers` using the `all-MiniLM-L6-v2` model. This runs locally to turn text into mathematical vectors. ### Search & Data Structures (RAG) - **FAISS (`faiss-cpu`):** Facebook AI Similarity Search. This handles the blazing-fast vector similarity comparisons. - **BM25 (`rank-bm25`):** A classic, battle-tested search algorithm used by search engines for exact keyword matching. - **NumPy & Pandas:** For fast data manipulation, CSV parsing, and matrix operations. ### Guardrails & Output - **RapidFuzz (`rapidfuzz`):** An incredibly fast string matching library used by the Auditor to verify that the LLM's claims match the source text exactly. - **Pydantic:** Used for strict data validation (ensuring the JSON output from the LLM matches the exact schema expected by the state machine). - **Rich:** A terminal formatting library that provides the beautiful, live-updating dashboard you see in the terminal while the agent runs. --- ## 🚀 4. How the Pipeline Flows (Step-by-Step) Here is exactly what happens when a single ticket enters the system: 1. **Input:** Ticket arrives: *"I lost my Visa card and need a replacement immediately."* 2. **The Bouncer:** Detects the keyword "Visa" and flags it for the Visa product area. It sees "lost...card" but realizes the corpus handles this, so it passes it on. (Status: Safe). 3. **The Librarian:** Embeds the ticket text into a vector. Searches the FAISS index and the BM25 index. It finds the Markdown chunk from the Visa directory titled `## Lost or Stolen Cards`. 4. **The Spokesperson:** Gemini receives the prompt. - *Scratchpad:* "The user lost their card. Document 1 says to call 1-800-VISA-911 immediately. I will tell them this." - *Drafts response.* 5. **The Auditor:** Scans the draft. Sees the phone number `1-800-VISA-911`. Uses RapidFuzz to check the retrieved document. The number matches exactly! (Verdict: `clean`). 6. **Output:** The agent outputs a final JSON indicating `status: replied`, with the correct response, product area, and justification. The terminal dashboard updates with a green checkmark ✅. --- ## 🏆 5. Summary of Achievements - **Zero Hallucination Guarantee:** By forcing the LLM to ground its answers and programmatically auditing them, Sentinel practically eliminates AI hallucinations. - **Enterprise-Grade Safety:** The system never guesses on sensitive topics like fraud or security exploits. - **Cost & Speed Optimized:** By routing tickets through a lightweight rule-engine first, it avoids wasting expensive LLM calls on spam, jailbreaks, or simple instant-escalations. - **Beautiful UX:** The `rich` terminal UI provides complete transparency into the agent's "Thought Process" in real-time. github link: https://github.com/MohammedMutee/hackerrank-orchestrate-may26