Streamline RAG Testing Using LLM Feedback

Interactive debugging for Retrieval-Augmented Generation (RAG) pipelines just took a leap forward with the introduction of "raggy," an innovative tool developed collaboratively by experts from the University of Pittsburgh and University of California, Berkeley. This new approach tackles the pervasive problem of debugging complexity in RAG systems head-on. RAG pipelines integrate retrieval (pulling relevant data chunks) with generation (leveraging LLMs like OpenAI's GPT-4o to craft accurate responses). Yet, debugging these intertwined components has traditionally been cumbersome, involving lengthy re-indexing and unclear identification of error sources. "raggy" addresses these issues by combining a Python library of composable RAG primitives with a dynamic, interactive debugging interface. Under the hood, "raggy" pre-computes vector indexes and strategically checkpoints pipeline states to allow instantaneous feedback on parameter adjustments-eliminating hours-long re-indexing delays typically associated with modifying chunk sizes or retrieval methods. Technical highlights include: - Real-time visualization of retrieval chunk distributions and similarity scores - Immediate interactive modification of retrieval parameters (e.g., chunk size, overlap, retrieval method) - Flexible query rewriting using intermediate LLM steps for handling ambiguous user inputs - "What-if" scenario analysis without latency Through an insightful user study involving 12 experienced developers, "raggy" demonstrated clear efficiency gains-71.3% of parameter changes would typically demand re-indexing in traditional workflows but were instantly testable using "raggy". Developers praised the system's capability to rapidly iterate and validate pipeline changes in seconds rather than hours. "raggy" not only accelerates the RAG development cycle but also aligns intuitively with developers' existing Python workflows, significantly enhancing productivity and reducing time-to-deployment. Explore how interactive debugging can streamline your RAG pipeline development. This tool embodies the future of AI system debugging.

Imagine waking up to 40% of your LLM budget used. All because you missed one layer in your app. The fix is simpler than you think. 𝗧𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 A user asks your RAG app: "What's your refund policy?" The app embeds the query, searches your vector database, retrieves the relevant chunks, passes them to the LLM, and returns a grounded answer. An hour later, someone else asks: "How do refunds work?" Same intent. Same answer. But your app doesn't know that. It runs the entire pipeline from scratch. Full compute. Full tokens. Full cost. Now multiply this across your user base. The same questions, asked in slightly different ways, 50 times a day, 100 times, 200 times. Your app is answering the same thing over and over again. It just has no memory. 𝗧𝗵𝗲 𝗙𝗶𝘅 Add a semantic cache layer before your RAG pipeline. When a query comes in, embed it and compare against previously seen queries using cosine similarity. If a match exists above your threshold (start with 0.9), return the cached answer immediately. No retrieval. No LLM call. Response in milliseconds. If no match is found, run your normal RAG pipeline. Then store the query embedding and the generated answer in the cache for next time. The key insight: this layer sits before everything. Before retrieval, reranking and generation. A cache hit means the entire pipeline is skipped. 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 ↳ 40% reduction in LLM API costs ↳ 80% latency reduction on repeat queries ↳ Millisecond responses on cache hits vs seconds for full pipeline 𝗣𝗼𝗶𝗻𝘁 𝘁𝗼 𝗥𝗲𝗺𝗲𝗺𝗯𝗲𝗿 If your similarity threshold is too low, "cancel my order" might match "cancel my account." Start conservative at 0.9 cosine similarity. Tune based on your data. For high-stakes queries, add a lightweight LLM verification step before serving the cached response. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗰𝗸 ↳ Redis — semantic cache (query embeddings as keys, answers as values) ↳ Qdrant — your main vector store for document retrieval ↳ Same embedding model you already use Redis handles the cache. Qdrant handles your documents. Two separate layers, one pipeline. — I teach this in extreme detail with tradeoffs in Week 5 of my RAG cohort. The full production stack. Caching. Observability. Deployment. The layers that separate a working demo from a system that actually scales. 𝗧𝗵𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿'𝘀 𝗥𝗔𝗚 𝗔𝗰𝗰𝗲𝗹𝗲𝗿𝗮𝘁𝗼𝗿 Join The Waitlist → https://academy.neosage.io — ♻️ Repost if this saves someone from a painful LLM bill

🚀 Exciting improvement in RAG (Retrieval-Augmented Generation) systems! This post is about a smart retrieval method that uses LLM-based query expansion to dramatically improve accuracy—proactively, not reactively. **The Challenge:** Traditional RAG systems often struggle with semantic gaps between user queries and document content. A single query might miss relevant information due to vocabulary mismatches or conceptual variations. **The Solution:** Instead of waiting for retrieval failures, our system proactively expands queries using an LLM before searching. The expansion generates multiple query variations, synonyms, and conceptual alternatives that capture the user's intent from different angles. **Key Benefits:** ✅ **Proactive Accuracy** - Addresses potential retrieval gaps before they occur ✅ **Better Coverage** - Multiple query perspectives ensure comprehensive results ✅ **Semantic Understanding** - LLM understands context and generates relevant expansions ✅ **Reduced False Negatives** - Finds documents that would otherwise be missed ✅ **Universal Improvement** - Works for both semantic-only AND hybrid semantic+keyword searches ✅ **Enhanced Top-K Retrieval** - Increased top-k values capture more relevant documents across expanded queries **How It Works:** 1. User submits a query 2. LLM generates expanded query set (synonyms, related concepts, alternative phrasings) 3. Parallel retrieval across all query variations with increased top-k 4. Intelligent result fusion and ranking from multiple query perspectives 5. More accurate, comprehensive responses **What Makes It Special:** This method delivers improvements across the board—whether you're using pure semantic search (vector similarity) or hybrid approaches that combine semantic and keyword-based retrieval. By expanding queries proactively and increasing top-k retrieval, we capture more relevant documents that traditional single-query methods would miss. This provides a significant improvement over reactive methods that evaluation-based of the query responses. The results? Significantly improved retrieval accuracy and user satisfaction across both semantic and hybrid search architectures. By proactively expanding queries and increasing top-k retrieval, we're seeing fewer "I couldn't find that" moments and more relevant, comprehensive answers—regardless of the underlying search methodology. This proactive approach transforms RAG from a reactive search tool into an intelligent information discovery system that works seamlessly with any retrieval strategy. #AI #RAG #LLM #MachineLearning #NLP #RetrievalAugmentedGeneration #QueryExpansion #TechInnovation #ArtificialIntelligence

⚖️ 𝗟𝗟𝗠 𝗮𝘀 𝗔 𝗝𝘂𝗱𝗴𝗲 𝗳𝗼𝗿 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗰 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 of machine learning systems is one of the most time-consuming yet critical steps in development. But what if we could automate that? 🤖 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 — a powerful, and often underestimated, use case of large language models. Instead of only generating content, LLMs can evaluate outputs: 𝗮𝘀𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝘀𝗰𝗼𝗿𝗲𝘀, 𝗰𝗼𝗺𝗽𝗮𝗿𝗶𝗻𝗴 𝗮𝗹𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝘃𝗲𝘀, 𝗼𝗿 𝗲𝘃𝗲𝗻 𝗴𝗶𝘃𝗶𝗻𝗴 𝗮 𝘀𝗶𝗺𝗽𝗹𝗲 ✅ pass / ❌ fail verdict. 💡 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: ⚡ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗶𝘁𝗲𝗿𝗮𝘁𝗶𝗼𝗻 → Automating evaluations reduces manual review time. 🎯 𝗛𝗶𝗴𝗵𝗲𝗿 𝗿𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆 → Standardized evaluation criteria make results more consistent. 💰 𝗖𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 → While LLM calls aren’t free, using them strategically can dramatically reduce human evaluation cycles. 🔑 𝗧𝗵𝗿𝗲𝗲 𝗖𝗼𝗿𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗵𝗼𝗱𝘀 1️⃣ 𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝘁𝘄𝗼 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 Useful when testing prompt variations, different models, or RAG embeddings. The 𝗟𝗟𝗠 𝗷𝘂𝗱𝗴𝗲 decides whether outputs are equal, or which one is better. 2️⃣ 𝗦𝗰𝗼𝗿𝗲 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 (1–10 or simplified scale) Ideal for experiments with multiple prompt versions or models. Anchoring with example scores improves accuracy. 3️⃣ 𝗣𝗮𝘀𝘀/𝗙𝗮𝗶𝗹 𝗰𝗵𝗲𝗰𝗸𝘀 Especially powerful in RAG systems — did the answer correctly reflect the retrieved context? Clear definitions and few-shot examples improve reliability. 📝 𝗞𝗲𝘆 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀 👥 𝗛𝘂𝗺𝗮𝗻 𝗖𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻 → Always benchmark against human evaluators to ensure alignment. Blind tests are best. 💸 𝗖𝗼𝘀𝘁 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀 → Frequent evaluations can add up. Use cheaper models for bulk checks or reduce test sizes. 🔧 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 → No single method fits all. The right evaluation strategy depends on your system (QA, classification, extraction, etc.). 🚀 𝗪𝗵𝘆 𝗜𝘁’𝘀 𝗣𝗼𝘄𝗲𝗿𝗳𝘂𝗹 Think about deploying a new prompt in production. Instead of manually checking hundreds of responses, you can let the LLM judge decide whether the new version performs as well — or better — than the old one. If results hold, deploy confidently. ✅ 𝗙𝗶𝗻𝗮𝗹 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 isn’t just a clever trick; it’s a scalable evaluation framework. By offloading repetitive validation to LLMs, teams can move faster, reduce bottlenecks, and still maintain quality. 𝗜𝘁’𝘀 𝗻𝗼𝘁 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 — careful alignment with human evaluators is essential — but it’s a tool every AI practitioner should have in their arsenal. 🔹 Have you experimented with 𝗟𝗟𝗠𝘀 𝗮𝘀 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿𝘀 in your workflows? What challenges or benefits have you seen? #AI #LLM #MachineLearning #GenerativeAI #ArtificialIntelligence #AICommunity #RAG #AgenticAI #Automation

🔍 Integrating LLM as a Judge in Your RAG Workflow Building upon our exploration of Enterprise RAG architecture and design best practices from two weeks ago (https://lnkd.in/eSggTNyE), and expanding on our examination of evaluation-driven development from last week (https://lnkd.in/eAsiprjH), we'll continue to delving further into the concept of LLM as a Judge. In my earlier post this week, we explored the function of Large Language Models (LLMs) as evaluators and how your specialized teams can contribute to training the LLM Judge (https://lnkd.in/eVz2i_4n). In today's discussion, we'll focus on how to integrate the trained LLM as a Judge in your RAG workflow. We will continue to leverage the Payer specific domain examples to help explain core concepts. Here's a step-by-step guide to integrating an LLM judge: 1/ RAG Response Generation: Generate response from user query and context. 2/ Prepare Evaluation Input: Compile question, response, and context into structured format. 3/ Domain-specific LLM Judge prompt: Use appropriate prompt for evaluation inference. 4/ LLM Judge Evaluation: Submit prepared input with specific evaluation prompt. 5/ Interpret Judge's Output: Analyze assessment ("Correct", "Incorrect", "Unclear"). 6/ Action Based on Evaluation: Handle outputs:If "Correct": Deliver to user. If "Incorrect"/"Unclear": Trigger review or fallback. 7/ Feedback Loop: Store evaluations for continuous improvement of RAG and judge models. Let's walk through this process using a healthcare payer example: User question: "What is the copay for a specialist visit under the Gold Plan?" 1/ RAG response: "Under the Gold Plan, the copay for a specialist visit is $40." 2/ Evaluation input: QUESTION: "What is the copay for a specialist visit under the Gold Plan?" RESPONSE: "Under the Gold Plan, the copay for a specialist visit is $40." CONTEXT: "Gold Plan specialist visits have a $40 copay as of January 1, 2024." 3/ LLM judge prompt: "Given the QUESTION about health insurance, is the RESPONSE correct based on the CONTEXT? Return 'Correct' or 'Incorrect'." 4/ LLM judge evaluation: Make inference call. 5/ Judge's Output: "Correct". 6/ Action: Approve response. 7/ Feedback: If "Incorrect" or "Unclear", trigger human review or use fallback response. By integrating an LLM judge into your RAG workflow, you create a powerful system that combines the efficiency of AI with the reliability of expert-guided evaluation. 💬 How are you planning to integrate AI-driven evaluation in your RAG systems? ♻️ Subscribe to my newsletter & repost if you find value in these insights: https://lnkd.in/g3bdneR7 #enterpriserag #evaluationtechniques #aievaluation #genai #datascience #machinelearning #aistrategy #cto #cdo #aicouncil #aws #enterpriseai #aiadoption #digitaltransformation #healthcarepayers #healthcareai #insurtech

Throw out the old #RAG approaches; use Corrective RAG instead! Corrective RAG introduces the additional layer of checking and correcting retrieved documents, ensuring more accurate and relevant information before generating a final response. This approach enhances the reliability of the generated answers by refining or correcting the retrieved context dynamically. The key idea here is to retrieve document chunks from the vector database as usual and then use an LLM to check if each retrieved document chunk is relevant to the input question. The process roughly goes as below, ⮕ Step 1: Retrieve context documents from vector database from the input query. ⮕ Step 2: Use an LLM to check if retrieved documents are relevant to the input question. ⮕ Step 3: If all documents are relevant (Correct), no specific action is needed. ⮕ Step 4: If some or all documents are not relevant (Ambiguous or Incorrect), rephrase the query and search the web to get relevant context information. ⮕ Step 5: Send rephrased query and context documents or information to the LLM for response generation. I have made a complete video on corrective RAG using LangGraph: https://lnkd.in/gKaEjEvk Know more in-depth about corrective RAG in this paper: https://lnkd.in/g8FkrMzS

Probing-RAG: RAG approach with efficient adaptive retrieval pipeline using LLM Hidden States for Selective Document Retrieval This paper propose Probing-RAG, which utilizes hidden state representations from the intermediate layers of language models to adaptively determine the necessity of additional retrievals for a given query. 𝗠𝗲𝘁𝗵𝗼𝗱 Similar to the conventional retrieval-augmented generation pipeline, this approach comprises a generating language model and a retriever. Different from the general pipeline, the generator of Probing-RAG leverages the output from the prober and adaptively calls the retriever based on the model’s internal hidden state. i) Prober - Given the LLM’s hidden state during answer generation, the prober assesses whether an additional retrieval step is necessary - designed prober as a feed-forward network with a single hidden layer and an output layer for binary classification - prober utilizes the hidden states corresponding to the model-generated rationale (r) and answer ii) Training Prober - requires pair of two data points: input derived from hidden states, output denoting whether additional retrieval is needed - Chain-of-Thought (CoT) prompting is used to generate these pairs - final dataset consists of 26,060 training and 500 validation samples. iii) Probing based Retrieval-Augmented Generation - After generating the initial rationale and answer, the prober assesses whether retrieval is necessary. - To do this, extract hidden state representation, feed them into probers assigned to each layer to generate logit values. - If difference between logit for retrieval necessity and logit indicating no need for retrieval is higher than the threshold, additional documents are retrieved 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - Probing-RAG demonstrates best performance, with improvements of approximately 6.59% points and 8.35% points in accuracy compared to no-retrieval and single-step approaches, respectively -  Probing-RAG outperforms all of these previous adaptive retrieval methods by avoiding redundant retrieval 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 - LLM-based, FLARE, Adaptive-RAG, and DRAGIN perform 1.17, 2.67, 1.54, and 6.83 times more retrieval calls, respectively, compared to Probing-RAG - prober trained on just 1k data points outperforms all of the previous methods, indicating that that it is possible to effectively train the prober using a small dataset. 𝗕𝗹𝗼𝗴: https://lnkd.in/egY7w5v9 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eHtMBTdM

RAG is good. Evaluation makes it better. The question is: 𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐞 𝐚 𝐑𝐀𝐆 𝐬𝐲𝐬𝐭𝐞𝐦? Retrieval-augmented generation (RAG) systems change how LLM generate responses by integrating real-time data retrieval into the generative process. But how do we ensure these systems are reliable in production? At the core of every RAG system are two key components: 1️⃣ Retriever: Identifies relevant information from a vector database using similarity search. 2️⃣ Generator: Generate retrieved documents with the user query to generate accurate responses. For RAG to work seamlessly, both components must perform optimally. This is where evaluation comes in. One way to evaluate RAG systems is using the TRIAD Framework by Trulens, which consists of three metrics: 🔹 Context Relevance: Ensures retrieved documents align with the query. 🔹 Faithfulness (Groundedness): Verifies if the response is factually accurate and grounded in the retrieved documents. 🔹 Answer Relevance: Measures how well the response addresses the query. But here’s the challenge: Traditional evaluation requires significant data collection and ground truth, which can be resource-intensive. Enter LLM-as-a-Judge—a faster, cost-effective alternative to human evaluation. 𝐇𝐨𝐰 𝐃𝐨𝐞𝐬 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞 𝐖𝐨𝐫𝐤? The LLM evaluates generated outputs based on predefined guidelines. It can assess: ✅ Context Relevance: Is the retrieved document relevant to the query? ✅ Faithfulness: Is the response factually accurate? ✅ Answer Relevance: Does the response address the query effectively? 𝐖𝐡𝐲 𝐃𝐨𝐞𝐬 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞 𝐖𝐨𝐫𝐤? Critiquing text is inherently easier than generating it. By leveraging the LLM’s classification capabilities, we can evaluate RAG systems effectively, even in production environments. 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬 𝐰𝐢𝐭𝐡 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞: 🔸 Pairwise Comparison: Compare two responses and choose the better one. 🔸 Reference-Free Evaluation: Assess responses based on criteria like tone, bias, or correctness. 🔸 Reference-Based Evaluation: Judge responses against a reference document or context. While LLM-as-a-Judge isn’t perfect, it provides a robust framework for improving RAG systems. Key Takeaways: 🔑 RAG systems require rigorous evaluation to ensure reliability in production. 🔑 The TRIAD framework offers a structured approach to evaluating context relevance, faithfulness, and answer relevance. 🔑 LLM-as-a-Judge is a powerful tool for scalable, cost-effective evaluation. If you want to know about building the RAG Evaluation System with LLM-as-a-Judge, I recently wrote about them in my latest Newsletter post. ✍️Article Link: https://lnkd.in/gMpmWFj3 🔗RAG-To-Know Repository: https://lnkd.in/gQqqQd2a What are your thoughts on using LLMs to evaluate RAG systems? Let’s discuss it!