Evaluating RAG’s — Some more learnings

A simplified RAG system to automate customer self-service

I previously spoke about how to evaluate your RAG system by citing the research paper related to it. But this time, I want to speak from experience, talk about few approaches I took. Also, AI/ML tools are coming a long way, and slowly making it possible to test and evaluate your models/RAG systems.

Retrieval Augmented Generation (RAG) emerges as a potent tool in bolstering output quality by tapping into relevant context from an external vector database.

Yet, the development and assessment of a RAG system pose formidable challenges, particularly in gauging performance metrics.

In this discourse, we delve into the efficacious metrics for each stage of the RAG pipeline and elucidate their application in evaluating the entire system.

RAG Evaluation: A Primer

Evaluating a RAG system entails scrutinizing its adeptness in sourcing pertinent information from a knowledge repository to furnish dependable and precise outputs.

Conducting these evaluations proves invaluable during the initial stages of RAG development, but their utility extends post-deployment. Continuous evaluation in a production environment aids in comprehending the system’s prevailing performance vis-a-vis potential enhancements achievable through prompt modifications.

This iterative process is indispensable for optimizing RAG system efficacy and necessitates thorough evaluation mechanisms.

Evaluating Your RAG System: A Methodical Approach

Evaluating a RAG system mandates meticulous examination of its two cardinal components: retrieval and content generation. However, overlooking ancillary aspects integral to the system’s business logic would be remiss.

Let’s dissect the evaluation components:

1. Context Retrieval: Evaluating the “Context Retrieval” segment entails discerning the system’s proficiency in consistently retrieving salient knowledge from vast textual corpora. This involves optimizing chunking strategies, embedding models, and search algorithms for optimal performance.

2. Content Generation: Assessing the quality of generated content entails conducting experiments with diverse prompts and models. The goal is to leverage metrics such as faithfulness and relevancy to ascertain the production of cogent responses based on retrieved knowledge.

3. Business Logic: While context retrieval and content generation are imperative, other facets of the AI workflow crucial to the specific use case demand evaluation. Metrics like intent verification, output length, and rule compliance play pivotal roles in evaluating segments of the RAG pipeline.

Evaluation necessitates access to human-annotated ground truth data or leveraging synthetic data generation methods. Alternatively, real-time evaluation utilizing proficient language models like GPT-4 serves as a viable option, prevalent in the NLP community.

RAG Evaluation Metrics: A Glimpse

Determining optimal metrics for RAG system evaluation remains an evolving field, with certain metrics proving indispensable for production-grade AI applications.

Context Retrieval Evaluation:

• Context Relevance: Measures the relevance of retrieved context to the query.
• Context Adherence: Ensures generated answers align with retrieved context exclusively.
• Context Recall: Validates the accuracy of retrieved context against ground truth data.

Content Generation Evaluation:

• Answer Relevancy: Evaluates the relevance of generated answers to queries.
• Faithfulness: Gauges the factual accuracy of answers vis-a-vis context.
• Correctness: Measures answer accuracy against ground truth data.
• Semantic Similarity: Assesses the contextual semantic alignment of answers.

While the enumerated metrics provide a foundational framework, customized evaluation tailored to specific business needs remains paramount.

Conclusion: Iterative Improvement

Deploying a RAG system mandates a cyclical process of evaluation and refinement. Real-time user feedback and continuous evaluation post-deployment serve as linchpins for augmenting system efficacy and fostering user trust.

Harnessing apt tools like Vellum and embracing a data-driven approach underscore the path to perpetually enhancing RAG system performance.