A Comprehensive Analysis of Evaluation Frameworks, Metrics, and Tracing Standards for Retrieval-Augmented Generation Systems

Note

Created with Google Gemini Deep Research on 2025/08/11.

Initial prompt:

I’m trying to find good frameworks for running evaluations on different LLM RAG systems and models. I want something similar to promptfoo, or at least a comparison of available tools. I would like to know what frameworks, if any, the latest and top papers use to run their evaluations. I would also like to know what schema or format people are using that preserves the entire trace for a set of queries.

The Anatomy of RAG Evaluation: Deconstructing for Quality

Retrieval-Augmented Generation (RAG) has emerged as a pivotal architecture for enhancing the capabilities of Large Language Models (LLMs) by grounding them in external, verifiable knowledge sources.1 By retrieving relevant information before generating a response, RAG systems aim to reduce hallucinations, provide up-to-date information, and enable domain-specific expertise.2 However, the hybrid nature of these systems, which combines information retrieval with generative modeling, introduces unique and complex evaluation challenges.2 A single, monolithic score of “correctness” is insufficient for diagnosing and improving RAG performance. A more granular, component-wise approach is necessary to isolate failure modes and guide effective optimization. This section deconstructs the RAG pipeline to establish the foundational principles of its evaluation, introducing the core metrics that have become the industry standard for quality assessment.

Deconstructing the RAG Pipeline for Evaluation: Isolating Failure Modes

At its core, a RAG system is composed of two primary, interdependent components: a Retriever and a Generator.1 The final quality of the system’s output is a direct function of the performance of both. A failure in either component can cascade, resulting in an inaccurate, irrelevant, or nonsensical response. Therefore, effective evaluation hinges on the ability to independently assess each component to perform accurate credit assignment when a failure occurs.
The Two-Component Problem:
The retriever’s responsibility is to query a knowledge base—typically a vector database—and fetch a set of context documents deemed relevant to the user’s input.7 The generator, usually a powerful LLM, then takes this retrieved context along with the original query and synthesizes a final answer.1 This separation of concerns creates distinct points of potential failure that must be monitored.
Retriever Failure Modes: The quality of the entire RAG process is fundamentally gated by the quality of the retrieved context. If the information provided to the generator is flawed, no amount of generative capability can produce a correct answer. Common failure modes for the retriever include:

• Low Precision (Noise): The retriever returns documents that are irrelevant to the user’s query. This irrelevant context can confuse the generator, leading it to produce off-topic or incorrect answers.9
  
• Low Recall (Missing Information): The retriever fails to fetch documents that contain the necessary information to answer the query. In this scenario, the generator is forced to rely solely on its parametric knowledge, which may be outdated or incorrect, leading to hallucinations.6
  
• Poor Ranking: The retriever successfully fetches relevant documents, but it ranks irrelevant or less important documents higher than the most critical ones. Given that LLMs can exhibit a “lost in the middle” problem, where they pay less attention to information in the middle of a long context, poor ranking can cause the generator to overlook key facts.6

Generator Failure Modes: Even with perfectly retrieved context, the generator component can fail in several ways:

• Hallucination / Lack of Faithfulness: The generator produces statements that are not supported by or directly contradict the provided context. This is a critical failure mode that undermines the trustworthiness of the RAG system.6
  
• Poor Synthesis: The generator fails to correctly synthesize information from multiple retrieved documents, especially if they contain conflicting or nuanced details. It may cherry-pick information or fail to construct a coherent narrative.2
  
• Irrelevance to Query: The generator produces an answer that is factually grounded in the context but does not actually address the user’s specific question or intent.6

The Need for Component-Wise Metrics: The existence of these distinct failure modes necessitates a suite of evaluation metrics that can isolate the performance of the retriever and the generator. This diagnostic capability is not merely an academic exercise; it is a critical requirement for practical debugging and system improvement. For instance, a low score on a metric like Faithfulness clearly indicates a problem with the generator’s ability to adhere to the provided context. Conversely, a low score on Contextual Recall points directly to a deficiency in the retriever’s ability to find all necessary information.6 Without this separation, a developer faced with a “wrong answer” is left to guess whether to re-index the documents, fine-tune the embedding model, engineer the generator’s prompt, or replace the LLM entirely. A component-wise evaluation framework transforms this guesswork into a data-driven, methodical process of elimination.

Foundational Pillars of RAG Quality: The “RAG Triad”

In response to the need for a standardized, component-wise evaluation methodology, the industry has largely converged on a set of three core metrics, often referred to as the “RAG Triad”.14 These metrics are typically implemented using the “LLM-as-a-judge” pattern, where a powerful model (like GPT-4) is used to score the outputs of the RAG system based on specific criteria. This approach provides a scalable way to assess the semantic qualities of the system’s inputs and outputs.
1. Context Relevance (Query ↔︎ Retrieved Context):
This metric is the first line of defense in RAG evaluation and assesses the retriever. It measures the alignment between the user’s query and the documents retrieved from the knowledge base.10 The fundamental question it answers is: “Is the information fetched by the retriever pertinent to the user’s question?” This is the most critical check in the pipeline, as irrelevant context makes a correct and faithful answer nearly impossible. A low context relevance score indicates that the retrieval stage is the primary bottleneck, pointing to potential issues with the embedding model, chunking strategy, or search algorithm. For enterprise applications, a target score of over 70% for context relevance is often considered a baseline for acceptable performance.14
2. Faithfulness / Groundedness (Context ↔︎ Response):
This metric evaluates the generator and is the primary tool for detecting hallucinations. It assesses whether the generated response is factually supported by the retrieved context.3 The core question is: “Is the LLM making things up, or is every claim in its answer grounded in the provided source documents?” To calculate this, frameworks like RAGAs will often decompose the generated answer into a set of individual statements and then verify each statement against the context.11 High faithfulness is paramount for building user trust and ensuring the reliability of the RAG system, especially in high-stakes domains like finance or medicine. The enterprise target for faithfulness is typically very high, often exceeding 90%.14
3. Answer Relevance (Query ↔︎ Response):
This metric also assesses the generator, but from the perspective of user intent. It measures how well the generated response addresses the specific question asked by the user.6 An answer can be perfectly faithful to the context but still be unhelpful if it fails to answer the question. For example, if a user asks “What was the company’s revenue in Q4 2024?”, and the retrieved context contains the full financial report, a faithful but irrelevant answer might be “The company released its financial report on January 28, 2025.” This statement is factually correct and grounded in the context but fails to answer the user’s question. Answer relevance ensures the system is not only factually correct but also useful. The enterprise target for this metric is generally above 85%.14
Together, these three pillars form a robust framework for diagnosing the health of a RAG system. By systematically evaluating Context Relevance, then Faithfulness, and finally Answer Relevance, developers can efficiently pinpoint whether a failure originates in the retrieval or generation stage and take targeted action to improve the system.

The Landscape of RAG Evaluation Frameworks: A Comparative Analysis

The rapid proliferation of RAG systems has spurred the development of a diverse ecosystem of evaluation frameworks. These tools range from lightweight, open-source libraries designed for developer-centric testing to comprehensive observability platforms built for production monitoring and enterprise-grade, managed solutions. Understanding this landscape is crucial for selecting the appropriate toolchain for each stage of the LLM application lifecycle, from initial prototyping to production deployment and continuous improvement. A clear trend of both specialization and convergence is shaping this landscape, creating a powerful, albeit complex, set of options for developers and MLOps professionals.

Open-Source Suites for Metric-Driven Development

These frameworks are primarily designed for developers and data scientists to integrate directly into their development and testing workflows. They are often Python-native, highly modular, and built to support a “metric-driven development” approach, where quantitative scores guide the iterative improvement of RAG components.

• RAGAs (Retrieval Augmented Generation Assessment):
  • Core Focus: RAGAs has become one of the most popular open-source frameworks due to its specific focus on the reference-free evaluation of RAG pipelines.3 This is a significant advantage, as it does not always require a manually curated “golden” answer for every test case. Instead, it can use the retrieved context as the source of truth for metrics like
    Faithfulness, making evaluation more scalable.
    
  • Key Features: It provides a suite of core RAG metrics, including its own implementations of Faithfulness, Answer Relevancy, Contextual Precision, and Contextual Recall.3 A standout feature is its ability to synthetically generate question-context-answer triplets from a corpus of documents, which helps bootstrap the creation of evaluation datasets.20
    
  • Academic Backing and Integrations: The framework’s credibility is bolstered by its presentation at academic conferences like the European Chapter of the Association for Computational Linguistics (EACL).5 It offers seamless integrations with major LLM development frameworks like LangChain and LlamaIndex, allowing for easy adoption into existing projects.11
    
• DeepEval:
  • Core Focus: DeepEval positions itself as “Pytest for LLMs,” emphasizing its design for unit testing LLM outputs and integrating evaluations directly into Continuous Integration/Continuous Deployment (CI/CD) pipelines.12 Its syntax, using decorators like
    @assert_test, is intentionally designed to be familiar to Python developers accustomed to frameworks like Pytest.12
    
  • Key Features: It offers a broad library of over 14 metrics, covering not only RAG-specific evaluations but also general LLM capabilities like summarization and hallucination detection. It can also be used to run standard academic benchmarks such as MMLU, HellaSwag, and HumanEval directly within the framework.12 Like RAGAs, it also includes functionality to help generate “golden datasets” for evaluation.10
    
  • Integrations: Its primary integration is with Pytest, which is its core value proposition for automated testing. It also provides integrations for LlamaIndex and Hugging Face, enabling real-time evaluations during fine-tuning or within RAG pipelines.12
    
• TruLens:
  • Core Focus: TruLens is an open-source evaluation tool backed by the enterprise data company Snowflake, which lends it significant credibility in corporate environments.14 Its evaluation philosophy is heavily centered on the “RAG Triad”.1
    
  • Key Features: The framework provides instrumentation for logging and evaluating RAG applications, with a strong emphasis on the core triad of Context Relevance, Groundedness (its term for Faithfulness), and Answer Relevance.15 A notable strategic direction for TruLens is its shift towards adopting OpenTelemetry as its underlying tracing standard, which will enhance its interoperability with other observability tools and telemetry backends.15

Observability Platforms for Production Monitoring

While the open-source suites excel during development, a different class of tools is required to monitor, debug, and evaluate RAG systems once they are deployed to production. These observability platforms are built around the concept of tracing, capturing the end-to-end execution flow of every request and building evaluation capabilities on top of this rich, real-time data.

• Langfuse:
  • Core Focus: Langfuse is a comprehensive, open-source LLM engineering platform that unifies tracing, evaluation, prompt management, and analytics into a single system.22
    
  • Key Features: Its strength lies in its detailed trace visualization, which allows developers to inspect and debug complex, multi-step chains and agentic workflows.24 It supports both custom and LLM-as-a-judge evaluations that can be run on production data for continuous monitoring. A key differentiator is its integrated prompt management UI, which allows teams to version, test, and collaboratively manage prompts, a critical component of RAG system maintenance.22 It can also integrate with evaluation libraries like RAGAs to attach scores to its traces.25
    
• Arize Phoenix:
  • Core Focus: Phoenix is an open-source AI observability tool from Arize AI, built from the ground up on the OpenTelemetry standard.18 This native OTEL compliance makes it highly interoperable and vendor-agnostic.
    
  • Key Features: Phoenix excels at visualizing traces and, uniquely, at analyzing the underlying embedding data of a RAG system.27 It leverages the
    OpenInference specification—a set of semantic conventions for AI on top of OTEL—to auto-instrument popular frameworks like LangChain and LlamaIndex with minimal code changes.26 It includes a built-in suite of evaluations for common issues like Q&A accuracy, hallucination, and toxicity.22
    
• LangSmith:
  • Core Focus: Developed by the team behind LangChain, LangSmith is an observability and evaluation platform designed to work with any LLM application but offers exceptionally tight integration for systems built with LangChain.22
    
  • Key Features: LangSmith is renowned for its detailed, hierarchical trace views, which are particularly valuable for debugging the complex, nested execution paths of agents and multi-step chains.31 It features a robust built-in evaluation suite for running tests on datasets and provides powerful mechanisms for logging and analyzing user feedback and other metadata directly against production traces.30
    
• promptfoo:
  • Core Focus: While the user query mentioned promptfoo as a point of comparison, it is important to characterize it correctly. promptfoo is an open-source toolkit that excels at prompt engineering and testing.22 Its primary use case is the systematic, side-by-side comparison of different prompts, models, and configurations against a defined set of test cases. While it is a powerful tool for optimizing the “G” (Generation) part of RAG, it is less specialized for evaluating the end-to-end RAG pipeline, particularly the retrieval component, compared to dedicated frameworks like RAGAs or DeepEval.

Enterprise-Grade and Commercial Solutions

For large organizations with stringent requirements for security, scalability, and support, a range of commercial platforms offer managed, enterprise-grade solutions for RAG evaluation and monitoring.

• Galileo AI: This platform is cited in industry reports as having the “Highest Enterprise Adoption”.14 It is a purpose-built, managed service designed specifically for enterprise RAG use cases, offering comprehensive production monitoring, evaluation, and dedicated support.14
  
• Other Commercial Platforms: The broader commercial landscape includes a variety of tools that blend prompt engineering, observability, and evaluation. Platforms such as Klu.ai, Latitude, and Pezzo provide hosted environments with features aimed at streamlining the LLM development lifecycle, typically offering free tiers for individual developers and scalable plans for enterprise teams.23 Major cloud providers also offer extensive MLOps solutions, such as
  Google’s Vertex AI, which includes tools for building, deploying, and evaluating ML models, including RAG systems.34

The distinction between these categories is not always rigid. A developer’s journey often begins with a lightweight, open-source library like RAGAs for local development and CI/CD checks. This allows for rapid, cost-effective iteration. However, once the application is deployed, the challenges shift from static dataset evaluation to understanding real-world performance, latency, and cost. At this stage, the detailed, request-level insights provided by an observability platform like Langfuse, Arize Phoenix, or LangSmith become indispensable. These platforms capture production traces, which in turn become the most valuable source of data for identifying failure modes. Recognizing this, the observability platforms have built powerful evaluation features directly into their products, allowing developers to run LLM-as-a-judge evaluators on live or sampled production data. This creates a powerful, continuous feedback loop: deploy the application, observe its behavior in production, identify failures from traces, add those failure cases to the CI/CD evaluation suite (using RAGAs or DeepEval), fix the underlying issue, and redeploy with confidence that the regression has been addressed. This full-cycle workflow represents the maturation of LLMOps, moving the field from ad-hoc testing to a systematic, data-driven process of continuous evaluation and improvement.

Table 1: Comparative Matrix of RAG Evaluation Frameworks

Framework	License	Primary Focus	Key RAG Metrics	Core Integrations	Tracing Standard
RAGAs	Apache-2.0	Reference-Free RAG Evaluation	Faithfulness, Answer Relevancy, Context Precision/Recall	LangChain, LlamaIndex	N/A (Library)
DeepEval	Apache-2.0	CI/CD Unit Testing for LLMs	RAG Triad, G-Eval, Bias, Toxicity	Pytest, LlamaIndex, Hugging Face	Custom Decorators
TruLens	Apache-2.0	Enterprise-Backed RAG Evaluation	RAG Triad (Context Relevance, Groundedness, Answer Relevance)	Snowflake, LangChain, LlamaIndex	OpenTelemetry (planned)
Langfuse	MIT	Open-Source LLM Observability	Custom Evaluations, LLM-as-a-Judge, RAGAs Scores	LangChain, LlamaIndex, OpenAI	OpenTelemetry
Arize Phoenix	Apache-2.0	Open-Source AI Observability	Hallucination, Q&A Accuracy, Toxicity	LangChain, LlamaIndex, OpenAI	OpenTelemetry, OpenInference
LangSmith	Proprietary	LLM Observability & Evaluation	Correctness, Relevance, Groundedness, Retrieval Relevance	LangChain, OpenAI	Custom
Galileo AI	Proprietary	Enterprise RAG Monitoring	Comprehensive Production Metrics	Enterprise Systems	Proprietary
promptfoo	MIT	Prompt Engineering & Comparison	Custom Assertions, LLM-as-a-Judge	N/A (CLI/Library)	N/A (Testing Tool)

A Lexicon of RAG Metrics: From Information Retrieval to Factual Consistency

A robust evaluation of a RAG system requires a nuanced vocabulary of metrics capable of dissecting its performance at each stage of the pipeline. These metrics have evolved from traditional information retrieval (IR) concepts to sophisticated, LLM-driven assessments of semantic quality. This section provides a detailed taxonomy of these metrics, categorizing them by the component they evaluate—the retriever or the generator—and highlighting the latest advancements from academic research that push beyond the standard “RAG Triad” to offer deeper diagnostic capabilities.

Evaluating the Retriever: Is the Context Correct?

The performance of the retriever is the foundation of any RAG system. These metrics are designed to quantify the quality of the documents returned from the knowledge base, before they are passed to the generator LLM.

• Contextual Precision: This metric assesses the ranking quality of the retrieval process. It measures whether the most relevant retrieved documents are ranked higher than irrelevant ones.3 High contextual precision is critical because LLMs can be sensitive to the order of information in their context window, often paying more attention to documents appearing at the beginning or end of a prompt. A low score suggests that the reranking component of the RAG pipeline may be ineffective, potentially causing the LLM to focus on noisy or less important information.6
  
• Contextual Recall: This metric evaluates the completeness of the retrieved information. It measures whether the retrieved context contains all the necessary facts required to formulate the ideal, ground-truth answer.6 A high contextual recall score indicates that the retriever is successfully finding all the relevant pieces of the puzzle. A low score, however, signifies a critical failure where the retriever is missing key information, which will inevitably force the generator to either provide an incomplete answer or hallucinate to fill the gaps.
  
• Contextual Relevancy: While precision and recall focus on ranking and completeness against a known target, contextual relevancy provides a more general measure of the signal-to-noise ratio in the retrieved context.3 It asks a simpler question: “Of all the text retrieved, how much of it is actually pertinent to the user’s query?” This metric is useful for identifying retrievers that are overly “chatty,” returning large volumes of text that contain only a few relevant sentences, thereby increasing processing costs and the risk of confusing the generator.
  
• Classic Information Retrieval (IR) Metrics: When a “golden” dataset with explicit relevance labels for each query-document pair is available, traditional IR metrics can be employed for a more rigorous, non-LLM-based evaluation of the retriever’s ranking performance.
  • Mean Reciprocal Rank (MRR): This metric is focused on the performance of retrieving the first correct document. It calculates the reciprocal of the rank of the first relevant document, averaged across all queries. MRR is particularly useful in scenarios where finding a single correct answer quickly is the primary goal.14
    
  • Normalized Discounted Cumulative Gain (NDCG): NDCG is a more sophisticated ranking metric that evaluates the entire ranked list. It assigns higher scores to relevant documents that appear earlier in the results and uses a logarithmic discount to penalize relevant documents that are ranked lower. It is one of the standard metrics for evaluating search and recommendation systems.14

Evaluating the Generator: Is the Answer Correct?

Once the context has been retrieved, the focus of evaluation shifts to the generator LLM. These metrics assess the quality of the final, user-facing answer.

• Faithfulness: This is arguably the most important generator metric, as it directly measures the factual consistency of the output and serves as a primary detector of hallucination. Faithfulness evaluates whether every claim made in the generated answer is explicitly supported by the information present in the provided context.3 To implement this, frameworks like RAGAs programmatically break down the generated answer into a series of individual statements and then use an LLM-as-a-judge to verify each statement against the source context.11 A low faithfulness score is a clear signal that the generator is failing to adhere to its instructions to ground its response in the provided evidence.
  
• Answer Relevancy: This metric assesses whether the generated answer is pertinent to the user’s query and addresses their specific intent. An answer can be 100% faithful to the context but still be completely useless if it doesn’t answer the question that was asked.6 This metric ensures that the generator is not only factually accurate but also helpful and on-topic.
  
• Answer Correctness: This is a stricter, reference-based metric that compares the generated answer to a pre-defined, ground-truth “golden” answer.10 It measures the factual alignment between the system’s output and the ideal response. This is distinct from Faithfulness. For example, if the retriever fetches an outdated document, the generator might produce an answer that is
  faithful to that incorrect context but is ultimately incorrect when compared to the ground truth. Answer Correctness captures this end-to-end factual accuracy.

The Frontier of Evaluation: Novel Metrics from Academic Research

While the RAG Triad provides a strong foundation, recent academic research has identified more subtle failure modes that require a new generation of more granular, explainable metrics. These novel metrics aim to provide deeper insights into the internal workings of the generator.

• Context Utilization: Introduced by the RAGBench paper’s TRACe framework, this metric measures the proportion of the retrieved context that the generator actually used to formulate its response.3 This is a powerful diagnostic tool. For example, a system might retrieve a large amount of relevant context (high Context Relevance), but the generator might lazily use only the first sentence to construct its answer. A low utilization score would immediately flag this “lazy” behavior, which would be missed by the standard RAG Triad.
  
• Completeness: Proposed by both the RAGBench/TRACe and RAGEval frameworks, this metric assesses whether the generated answer incorporates all of the relevant and necessary information from the provided context.1 This addresses a key failure mode where a RAG system might provide an answer that is technically correct and faithful but is frustratingly superficial or incomplete. A system could have high context relevance and high faithfulness but low completeness if it only addresses part of the user’s query, even when all the necessary information was available in the context.
  
• Adherence: This is the term used within the TRACe framework and is synonymous with Faithfulness, Groundedness, and Attribution.38 Its inclusion in a comprehensive framework alongside Utilization and Completeness signifies the academic push towards a more holistic and multi-faceted evaluation of the generator’s behavior.

The progression of these metrics tells a story about the maturing understanding of RAG systems. The initial phase relied on classic IR metrics borrowed from search engine evaluation. The second phase, enabled by the power of LLMs, introduced the semantic “RAG Triad” to judge qualities like faithfulness and relevance at scale. The current, third phase, driven by academic research, is now moving towards even more granular and explainable metrics. This evolution is necessary because as RAG systems improve, their failure modes become more subtle. A developer might use the RAG Triad and find that all scores are high: the context is relevant, the answer is faithful, and it addresses the query. Yet, users may still find the answers superficial. The standard metrics cannot explain this. By applying a newer metric like Completeness, the developer might discover that the score is low. This reveals the true problem: the generator is correctly using a piece of the context to answer the question but is ignoring other, equally relevant pieces of information that would provide a more comprehensive and satisfying answer. This diagnosis points not to a retrieval problem, but to a generator-side synthesis problem, guiding the developer to focus on prompt engineering or model selection to encourage more thorough reasoning. This demonstrates that as RAG systems become more sophisticated, the metrics used to evaluate them must evolve from simple right/wrong checks to nuanced diagnostics of system behavior.

Insights from the Academic Vanguard: Benchmarks and Methodologies

The academic research community plays a crucial role in pushing the boundaries of Retrieval-Augmented Generation, not only by developing new architectures but also by creating more rigorous and realistic methods for their evaluation. Recent work presented at top-tier conferences like NeurIPS, ICLR, and ACL reveals a clear trend away from simplistic QA datasets towards dynamic, domain-specific, and explainable benchmarks. These new tools and methodologies are designed to expose the subtle failure modes of modern RAG systems and to question the very foundations of common evaluation practices like the “LLM-as-a-judge” paradigm.

The New Wave of RAG Benchmarks

Recognizing that traditional question-answering datasets are often insufficient for testing the unique challenges of RAG (such as robustness to noisy context or the ability to synthesize information), researchers have developed a new generation of specialized benchmarks.

• RAGBench:
  • Contribution: Developed by researchers at Galileo and Rungalileo, RAGBench is a large-scale (100,000 examples) benchmark explicitly designed for explainable RAG evaluation.1 Its key innovation is its focus on real-world applicability, sourcing its data from industry-specific corpora like user manuals, legal contracts, and financial documents across five domains.38
    
  • The TRACe Framework: RAGBench introduces a novel evaluation framework called TRACe, which stands for T-utilization, R-elevance, A-dherence, and C-ompleteness. This suite of metrics is designed to move beyond a single score and provide actionable, component-wise feedback on both the retriever and the generator.1
    
  • Key Finding: The paper presents a significant and potentially disruptive finding: a fine-tuned RoBERTa model, a much smaller and older transformer architecture, consistently outperforms large, powerful LLM-based judges (like GPT-3.5) on the task of evaluating RAG outputs within the RAGBench dataset. This result directly challenges the prevailing industry assumption that larger LLMs are inherently better evaluators and suggests that smaller, specialized models may be more reliable and cost-effective for this task.1
    
• CRAG (Comprehensive RAG Benchmark):
  • Contribution: Originating from Facebook Research, CRAG is a benchmark designed to test RAG systems against the diverse and dynamic nature of real-world information needs.43
    
  • Features: The benchmark consists of 4,409 factual question-answer pairs that are specifically curated to include entities with varying levels of popularity (from well-known facts to long-tail knowledge) and high temporal dynamism (i.e., facts that change frequently over time, from years down to seconds).43 This directly tests a RAG system’s ability to handle up-to-date and obscure information.
    
  • Key Finding: The evaluation on CRAG highlights a significant performance gap in current state-of-the-art systems. Even advanced RAG solutions were found to answer only 63% of questions without any hallucination, revealing substantial weaknesses in handling dynamic, complex, or long-tail queries and pointing towards clear directions for future research.43
    
• RAGEval:
  • Contribution: This framework, presented in an arXiv paper, addresses a major practical bottleneck in RAG evaluation: the high cost and effort of creating high-quality test datasets. RAGEval is designed to automatically generate scenario-specific evaluation data—including documents, questions, ground-truth answers, and supporting references—using a sophisticated, schema-based pipeline.36
    
  • Features: RAGEval also proposes a novel set of metrics focused on factual accuracy, calculated against keypoints extracted from the ground-truth answer: Completeness, Hallucination, and Irrelevance. This provides a structured way to assess the factual content of a generated response without relying on simple lexical overlap metrics like ROUGE or BLEU.36
    
• Other Notable Research:
  • RAGGED: This ICLR paper introduces an evaluation framework for systematically analyzing the interactions between different RAG components. It investigates how the choice of retriever (e.g., BM25 vs. ColBERT), reader model (e.g., LLaMa vs. GPT), and context length impacts overall performance across various tasks, providing empirical evidence for task-specific configuration.44
    
  • Multi-modal Benchmarks: As RAG systems expand to handle more than just text, new benchmarks are emerging to evaluate their multi-modal capabilities. REAL-MM-RAG and RAG-Check are two such examples, designed to assess systems that retrieve and reason over a combination of text, images, and tables.45

Synthesis of Methodologies and Findings from Top Venues (NeurIPS, ICLR, ACL)

Across these leading research venues, several key themes and methodologies are apparent, signaling a maturation of the field.

• Critique of LLM-as-a-Judge: There is a growing and critical examination of the reliability of using large, general-purpose LLMs as evaluators. The RAGBench paper’s finding that a smaller, fine-tuned model can be a more accurate judge is a cornerstone of this critique.1 This suggests that the task of evaluation is a specialized skill that may not be perfectly captured by the general pre-training of models like GPT-4.
  
• Bridging Component and End-to-End Evaluation: Researchers are actively seeking to close the gap between component-level metrics (like retrieval precision) and final, end-to-end task performance. The eRAG paper, for instance, proposes a novel method for evaluating retrieval quality that shows a much higher correlation with the downstream RAG system’s performance compared to traditional IR metrics, offering a more predictive measure of retriever quality.47
  
• Focus on Realism and Actionability: The most impactful new benchmarks are defined by their move towards greater realism. They use industry-specific documents, require multi-hop reasoning, and test against dynamic, changing information.38 The explicit goal of frameworks like RAGBench’s TRACe is not just to produce a score but to provide
  actionable insights that help developers diagnose and fix problems in their systems.1

The rapid industry adoption of the LLM-as-a-judge paradigm was driven by its scalability and apparent effectiveness for semantic evaluation.14 However, the academic community, in its pursuit of greater rigor, has begun to stress-test this approach with more challenging and nuanced benchmarks. The resulting discovery—that a smaller, specialized, fine-tuned model can be a superior evaluator—is significant. It implies that the general-purpose reasoning abilities of a massive LLM do not necessarily translate perfectly to the specific and subtle task of judging the quality of another AI’s output. This opens up a new and promising research direction: instead of relying on a single, proprietary, black-box model for all tasks, the community can now focus on developing smaller, cheaper, more transparent, and potentially more accurate
evaluator models. These models can be specifically fine-tuned on high-quality, human-annotated benchmarks like RAGBench. This creates a virtuous cycle where open, reproducible benchmarks are used to train open-source evaluator models, which can then be used by the entire community for more reliable and cost-effective evaluation, reducing the dependency on a single provider and fostering a more robust and trustworthy AI ecosystem.

Table 2: Overview of Key Academic RAG Benchmarks

Benchmark	Primary Authors/Institution	Key Contribution	Domains	Novel Metrics
RAGBench	Galileo / Rungalileo	Large-scale (100k), explainable benchmark from real-world industry corpora.	Biomedicine, General Knowledge, Legal, Customer Support, Finance	TRACe Framework: Utilization, Relevance, Adherence, Completeness
CRAG	Facebook Research	Factual QA benchmark focused on entity popularity and high temporal dynamism.	5 diverse domains (e.g., Finance, Sports)	N/A (focus on dataset characteristics)
RAGEval	arXiv	Framework for automatically generating scenario-specific RAG evaluation datasets.	Domain-agnostic generation pipeline	Completeness, Hallucination, Irrelevance (based on keypoints)
RAGGED	ICLR Submission	Framework for systematic analysis of RAG component choices and their interactions.	NQ, HotPotQA, BioASQ	N/A (focus on framework methodology)
REAL-MM-RAG	IBM Research	Benchmark for multi-modal RAG focusing on text, tables, and images.	Finance, General	N/A (focus on multi-modal data)

The Digital Thread: Tracing Schemas for Full-Stack RAG Observability

While quantitative metrics provide a vital scorecard for RAG system performance, they often fail to tell the whole story. A low faithfulness score indicates hallucination, but it doesn’t reveal why the hallucination occurred. To answer this, developers need to inspect the entire execution flow of a request—the digital thread that connects the user’s query to the final response. This is the role of tracing. Tracing provides a qualitative, step-by-step narrative of a RAG execution, making it an indispensable tool for debugging complex failures, understanding agentic behavior, and pinpointing performance bottlenecks.24 The emergence of OpenTelemetry as a unifying standard for tracing is revolutionizing RAG observability, enabling interoperable and detailed inspection of these complex AI systems.

The Imperative of Tracing: Beyond Metrics

A single trace represents the complete, end-to-end journey of a single request through the RAG system. It captures every intermediate step, including the initial query, the calls to the retriever, the documents that were returned, the construction of the prompt, the invocation of the LLM, and the final generated output.22 This level of granularity is essential for deep debugging. For example, by examining a trace, a developer can see the exact context that was passed to the LLM, immediately verifying whether a hallucination was caused by faulty retrieval (bad context) or a flawed generation step (good context, bad answer). In agentic RAG systems that may involve multiple tool calls and decision-making loops, tracing is not just helpful—it is the only feasible way to understand the agent’s reasoning path.

OpenTelemetry as the Lingua Franca

Historically, different LLM frameworks and observability platforms each had their own proprietary logging and tracing formats, leading to a fragmented ecosystem and vendor lock-in. Recognizing that a RAG application is a form of distributed system, the industry is rapidly converging on OpenTelemetry (OTEL) as the vendor-neutral, open-source standard for instrumenting applications to collect traces, metrics, and logs.26 This standardization is a critical step towards a mature, interoperable LLMOps ecosystem.

• Core OTEL Concepts:
  • Trace: A trace is a collection of all operations that belong to a single request as it propagates through the system. It is identified by a unique trace_id.53
    
  • Span: A span represents a single, named, and timed unit of work within a trace, such as a database query or an LLM call. Spans are organized into a hierarchy through parent_id references, forming a tree structure that reflects the execution flow.15
    
  • Attributes: Attributes are key-value pairs attached to a span that contain metadata about the operation it represents. This is where the rich, domain-specific information about a RAG process is stored.53
    
• GenAI Semantic Conventions: To ensure that a “retrieval” span from a LlamaIndex application looks the same as one from a LangChain application, the OTEL community is actively defining a set of standardized attribute names for AI and LLM operations. This specification, often referred to as semconv.ai, provides a common vocabulary for describing LLM calls, vector database interactions, tool usage, and agent activities.57 Adherence to these conventions is key to achieving true interoperability between different tools and platforms.

The adoption of OpenTelemetry is a transformative trend. It effectively commoditizes the data collection layer, allowing developers to instrument their RAG application once using the standard OTEL SDKs and semantic conventions. That same stream of standardized trace data can then be sent to multiple backends simultaneously—for example, an open-source tool like Jaeger for local debugging and a commercial platform like Arize or Langfuse for production monitoring—without requiring any changes to the application code. This decouples the application from the observability platform, preventing vendor lock-in and shifting the competitive focus of platforms from data ingestion to the value they provide through their UI, analytics, and evaluation features built on top of the standardized trace data.

Trace Schemas in Practice: A Comparative Look

Leading observability platforms are increasingly aligning with the OpenTelemetry standard, though they may still have their own internal data models that map to OTEL concepts. Examining their practical schemas reveals what a complete RAG trace looks like.

• Anatomy of a LangSmith Trace:
  • LangSmith uses the internal concepts of Runs and Traces, which map directly to OTEL’s spans and traces, respectively.55 A trace is a collection of runs.
    
  • A typical RAG trace in LangSmith is visualized as a hierarchical tree. The root Run represents the overall RAG chain. This root run has child Runs for each major step, such as the retriever call and the LLM call.
    
  • Key Data Schema: Each Run object contains inputs and outputs fields. For a retriever run, the inputs would contain the user’s query string, while the outputs would contain a list of retrieved Document objects (including their content and metadata). For an LLM run, the inputs would contain the fully formatted prompt (including the system message and the retrieved context), and the outputs would contain the generated AI message object.30 When creating evaluation datasets, the schema often follows a pattern of
    {“inputs”: {“question”:…}, “outputs”: {“answer”:…}}.7
    
• Anatomy of an Arize Phoenix / OpenInference Trace:
  • As a natively OpenTelemetry-based platform, Phoenix’s schema adheres closely to the OTEL specification, augmented by the OpenInference semantic conventions for AI-specific data.26
    
  • Key Data Schema: A RAG trace in Phoenix is a collection of OTEL spans, each with specific attributes:
    • A retrieval span would have a span.kind of TOOL and would be decorated with attributes like retrieval.query.text for the input query and retrieval.documents for the output. The retrieval.documents attribute would typically be an array of objects, each containing the document’s content and score.15
      
    • An LLM span would have attributes defined by the semantic conventions, such as llm.model.name, llm.prompt (or an array of llm.messages), llm.usage.prompt_tokens, llm.usage.completion_tokens, and llm.output.value for the final generated text.57
      
    • In an agentic RAG trace, there would be additional spans for tool calls, with attributes like tool.name, tool.input, and tool.output to capture the agent’s interaction with its tools.63

To make this concrete, a simplified JSON representation of a single RAG trace might look like this:

JSON

}  

},
{
“trace_id”: “trace-123”,
“span_id”: “span-C”,
“parent_id”: “span-A”,
“name”: “ChatOpenAI”,
“start_time”: “…”,
“end_time”: “…”,
“attributes”: {
“llm.model.name”: “gpt-4o”,
“llm.messages”: [
{ “role”: “system”, “content”: “Answer based on context.” },
{ “role”: “user”, “content”: “Context: [Paris is the capital…]\nQuestion: What is the capital of France?” }
],
“llm.usage.prompt_tokens”: 150,
“llm.usage.completion_tokens”: 5,
“output.value”: “The capital of France is Paris.”
}
}
]

Table 3: A Cross-Walk of RAG Trace Attributes

This table provides a practical mapping from the conceptual data points in a RAG pipeline to the specific attribute names used in the OpenTelemetry semantic conventions and leading platform implementations.

Conceptual Data Point	OpenTelemetry Semantic Convention (semconv.ai)	LangSmith Field (in Run object)	Arize/OpenInference Attribute
User Query	input.value	inputs[‘question’]	input.value or retrieval.query.text
Retrieved Documents	retrieval.documents (array of objects)	outputs (of retriever run)	retrieval.documents (array of objects)
Final Generated Answer	output.value	outputs[‘answer’]	output.value
LLM Model Name	llm.model.name	extra.metadata.model_name	llm.model.name
Prompt Template	llm.prompt.template	inputs (of prompt run)	llm.prompt.template
System Prompt	llm.messages (role: system)	inputs[‘messages’]	llm.messages (role: system)
Prompt Tokens	llm.usage.prompt_tokens	prompt_tokens	llm.usage.prompt_tokens
Completion Tokens	llm.usage.completion_tokens	completion_tokens	llm.usage.completion_tokens
Total Tokens	llm.usage.total_tokens	total_tokens	llm.usage.total_tokens
Latency	(Calculated from start_time, end_time)	latency	(Calculated from span duration)
Cost	(Calculated from tokens and model)	total_cost	(Calculated metric)

Strategic Recommendations and Future Outlook

The comprehensive analysis of the RAG evaluation landscape reveals a rapidly maturing ecosystem. To navigate this complexity and build robust, reliable RAG systems, a multi-layered evaluation strategy is required, spanning the entire application lifecycle from development to production. Furthermore, emerging trends in agentic AI, multi-modality, and evaluation methodologies signal the future direction of the field, demanding that developers and researchers anticipate and adapt to these new frontiers.

A Multi-Layered Strategy for Comprehensive RAG Evaluation

A one-size-fits-all approach to evaluation is insufficient. Instead, a phased strategy that employs different tools and methodologies at each stage of the development lifecycle provides the most effective path to building and maintaining high-quality RAG applications.

• Phase 1: Development & CI/CD (Pre-flight Checks):
  • Objective: Enable rapid iteration and prevent regressions.
    
  • Methodology: During local development and in automated CI/CD pipelines, employ lightweight, open-source evaluation frameworks like RAGAs or DeepEval. Focus on establishing a baseline of quality using a “golden dataset” of expert-validated question-answer pairs.10 Run automated tests that calculate the core “RAG Triad” metrics (
    Context Relevance, Faithfulness, Answer Relevance) and other component-wise scores (Contextual Precision, Contextual Recall). To expand test coverage beyond the golden dataset, leverage the synthetic data generation capabilities of frameworks like RAGAs to create a broader set of test cases automatically.20 This phase is about catching bugs early and ensuring code changes do not degrade performance.
    
• Phase 2: Pre-Production Benchmarking (Stress Testing):
  • Objective: Assess system robustness and performance on challenging, out-of-distribution data.
    
  • Methodology: Before deploying a major new model or architecture, benchmark the system against rigorous academic datasets like RAGBench or CRAG.1 These benchmarks are specifically designed to test for failure modes that may not be present in a curated golden dataset, such as handling long-tail knowledge, temporally dynamic facts, or noisy, irrelevant context from real-world documents. This phase provides a more realistic assessment of how the system will perform under pressure and can reveal weaknesses in generalization that need to be addressed before a production release.
    
• Phase 3: Production Observability & Continuous Improvement (In-flight Monitoring):
  • Objective: Monitor real-world performance, detect live failures, and create a data-driven feedback loop for continuous improvement.
    
  • Methodology: Instrument the production application using the OpenTelemetry standard. Ingest the resulting trace data into a dedicated observability platform such as Langfuse, Arize Phoenix, or LangSmith. Use the platform’s dashboards to monitor key operational metrics like latency, token usage, and cost, as well as quality indicators.14 The most critical part of this phase is closing the loop:
    1. Capture and analyze production traces to identify specific user queries where the system failed.
       
    2. Log user feedback (e.g., thumbs up/down) and associate it directly with the corresponding trace.
       
    3. When a clear failure pattern is identified from production data, extract those query-response pairs and add them as new test cases to the Phase 1 evaluation dataset.
       
    4. This ensures that real-world failures are systematically captured and used to prevent future regressions, creating a robust, full-loop continuous evaluation process.

The Future of RAG Evaluation: What’s Next?

The field of RAG evaluation is evolving as rapidly as the underlying models and architectures. Several key trends are poised to shape the future of how these systems are built and validated.

• Agentic RAG Evaluation: As RAG systems evolve from simple Q&A bots into complex, autonomous agents that can use tools, plan multi-step tasks, and make decisions, the focus of evaluation will necessarily shift.14 Evaluating a single, final answer will no longer be sufficient. Instead, evaluation will need to assess the entire reasoning process of the agent. Tracing will become the primary artifact for evaluation, and new metrics will emerge to judge the quality of the agent’s plan, the correctness of its tool selection, and the efficiency of its path to a solution.59
  
• Multi-Modal RAG: The advent of powerful multi-modal models that can understand and reason over text, images, tables, and other data formats is giving rise to multi-modal RAG systems. This creates a significant evaluation challenge. New benchmarks, such as REAL-MM-RAG and RAG-Check, are the first step in this direction, but the field will need to develop a new suite of metrics that can assess, for example, whether an answer is faithfully grounded in both a text paragraph and a corresponding chart in a document.16
  
• The Rise of Specialized Evaluator Models: The finding from the RAGBench paper—that a smaller, fine-tuned model can be a more reliable evaluator than a massive, general-purpose LLM—is likely to be a seminal moment for the field.1 This will likely spur a new wave of research and development focused on creating open-source, specialized evaluator models. These models, fine-tuned on high-quality, human-annotated benchmarks, could offer a cheaper, faster, and more accurate alternative to the “LLM-as-a-judge” paradigm, democratizing access to rigorous evaluation.
  
• Standardization and Interoperability: The convergence on OpenTelemetry for tracing is the harbinger of a broader trend towards standardization in the LLMOps ecosystem. As the GenAI semantic conventions mature, they will provide a common language that enables seamless interoperability between different LLM frameworks, vector databases, evaluation tools, and observability platforms. This will reduce friction for developers, prevent vendor lock-in, and foster a more competitive and innovative marketplace of tools built upon a shared, open foundation.

Works cited

1. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems - arXiv, accessed August 12, 2025, https://arxiv.org/html/2407.11005v2
   
2. Evaluation of Retrieval-Augmented Generation: A Survey - arXiv, accessed August 12, 2025, https://arxiv.org/html/2405.07437v1?ref=chitika.com
   
3. Understanding RAG Part IV: RAGAs & Other Evaluation Frameworks - MachineLearningMastery.com, accessed August 12, 2025, https://machinelearningmastery.com/understanding-rag-part-iv-ragas-evaluation-framework/
   
4. Retrieval augmented generation: Keeping LLMs relevant and current - Stack Overflow, accessed August 12, 2025, https://stackoverflow.blog/2023/10/18/retrieval-augmented-generation-keeping-llms-relevant-and-current/
   
5. RAGAs: Automated Evaluation of Retrieval Augmented Generation - ACL Anthology, accessed August 12, 2025, https://aclanthology.org/2024.eacl-demo.16/
   
6. RAG Evaluation Metrics: Assessing Answer Relevancy, Faithfulness, Contextual Relevancy, And More - Confident AI, accessed August 12, 2025, https://www.confident-ai.com/blog/rag-evaluation-metrics-answer-relevancy-faithfulness-and-more
   
7. A Beginner’s Guide to Evaluating RAG Systems with LangSmith - FutureSmart AI Blog, accessed August 12, 2025, https://blog.futuresmart.ai/a-beginners-guide-to-evaluating-rag-systems-with-langsmith
   
8. Evaluating RAG Architectures on Benchmark Tasks — LangChain …, accessed August 12, 2025, https://langchain-ai.github.io/langchain-benchmarks/notebooks/retrieval/comparing_techniques.html
   
9. RAG evaluation: Complete guide 2025 - SuperAnnotate, accessed August 12, 2025, https://www.superannotate.com/blog/rag-evaluation
   
10. RAG Evaluation Metrics: Best Practices for Evaluating RAG Systems - Patronus AI, accessed August 12, 2025, https://www.patronus.ai/llm-testing/rag-evaluation-metrics
    
11. RAGAS: Automated Evaluation of Retrieval … - ACL Anthology, accessed August 12, 2025, https://aclanthology.org/2024.eacl-demo.16.pdf
    
12. confident-ai/deepeval: The LLM Evaluation Framework - GitHub, accessed August 12, 2025, https://github.com/confident-ai/deepeval
    
13. Evaluating RAG with DeepEval and LlamaIndex, accessed August 12, 2025, https://www.llamaindex.ai/blog/evaluating-rag-with-deepeval-and-llamaindex
    
14. The Complete Enterprise Guide to RAG Evaluation and Benchmarking - AiExponent, accessed August 12, 2025, https://aiexponent.com/the-complete-enterprise-guide-to-rag-evaluation-and-benchmarking/
    
15. Telemetry for the Agentic World: TruLens + OpenTelemetry, accessed August 12, 2025, https://www.trulens.org/blog/otel_for_the_agentic_world/
    
16. every LLM metric you need to know : r/LangChain - Reddit, accessed August 12, 2025, https://www.reddit.com/r/LangChain/comments/1j3gllj/every_llm_metric_you_need_to_know/
    
17. RAGAS: Automated Evaluation of Retrieval Augmented Generation - -ORCA - Cardiff University, accessed August 12, 2025, https://orca.cardiff.ac.uk/id/eprint/166780/1/_EACL_2024__demo_RAGAs-2.pdf
    
18. Top 6 Open Source LLM Evaluation Frameworks : r/LLMDevs - Reddit, accessed August 12, 2025, https://www.reddit.com/r/LLMDevs/comments/1i6r1h9/top_6_open_source_llm_evaluation_frameworks/
    
19. Evaluating Naive RAG and Advanced RAG pipeline using langchain v.0.1.0 and RAGAS, accessed August 12, 2025, https://medium.aiplanet.com/evaluating-naive-rag-and-advanced-rag-pipeline-using-langchain-v-0-1-0-and-ragas-17d24e74e5cf
    
20. explodinggradients/ragas: Supercharge Your LLM … - GitHub, accessed August 12, 2025, https://github.com/explodinggradients/ragas
    
21. RAGAs: Automated Evaluation of Retrieval Augmented Generation | Request PDF, accessed August 12, 2025, https://www.researchgate.net/publication/393020278_RAGAs_Automated_Evaluation_of_Retrieval_Augmented_Generation
    
22. LLM Evaluation Frameworks: Head-to-Head Comparison - Comet, accessed August 12, 2025, https://www.comet.com/site/blog/llm-evaluation-frameworks/
    
23. Top promptfoo Alternatives in 2025 - Slashdot, accessed August 12, 2025, https://slashdot.org/software/p/promptfoo/alternatives
    
24. LLM Observability & Application Tracing (open source) - Langfuse, accessed August 12, 2025, https://langfuse.com/docs/tracing
    
25. Evaluation of RAG pipelines with Ragas - Langfuse, accessed August 12, 2025, https://langfuse.com/guides/cookbook/evaluation_of_rag_with_ragas
    
26. Arize-ai/phoenix: AI Observability & Evaluation - GitHub, accessed August 12, 2025, https://github.com/Arize-ai/phoenix
    
27. Evaluating and Analyzing Your RAG Pipeline with Ragas - Arize AI, accessed August 12, 2025, https://arize.com/blog/ragas-how-to-evaluate-rag-pipeline-phoenix
    
28. Evaluating RAG Retrieval Quality and Correctness | Arize Docs, accessed August 12, 2025, https://arize.com/docs/ax/cookbooks/evaluation/evaluating-rag
    
29. Trace. Detect. Improve. RAG Chatbots with Arize Phoenix | by Ajeethkumar - Medium, accessed August 12, 2025, https://medium.com/@ajeethk67/observability-for-rag-chatbots-using-langchain-and-arize-phoenix-9d47681d8f9e
    
30. Evaluate a RAG application | 🦜️🛠️ LangSmith - LangChain, accessed August 12, 2025, https://docs.smith.langchain.com/evaluation/tutorials/rag
    
31. Tracing LLM Workflows with LangSmith (RAG Demo with Anime!) | by Subhasmita Sahoo | Jun, 2025 | Medium, accessed August 12, 2025, https://medium.com/@subhasmitasahoo.247/tracing-llm-reasoning-with-langsmith-rag-demo-with-anime-4144c4fc8698
    
32. [Starters-friendly] Build an RAG application with LangChain and LangSmith - Medium, accessed August 12, 2025, https://medium.com/@zljdanceholic/starters-friendly-build-an-rag-application-with-langchain-and-langsmith-2cf6d720a86a
    
33. Ultimate Langsmith Guide for 2025 - Analytics Vidhya, accessed August 12, 2025, https://www.analyticsvidhya.com/blog/2024/07/ultimate-langsmith-guide/
    
34. Best promptfoo Alternatives & Competitors - SourceForge, accessed August 12, 2025, https://sourceforge.net/software/product/promptfoo/alternatives
    
35. Evaluating Multi-Modal RAG - LlamaIndex, accessed August 12, 2025, https://docs.llamaindex.ai/en/stable/examples/evaluation/multi_modal/multi_modal_rag_evaluation/
    
36. arXiv:2408.01262v5 [cs.CL] 3 Mar 2025, accessed August 12, 2025, https://arxiv.org/pdf/2408.01262
    
37. Evaluating - LlamaIndex, accessed August 12, 2025, https://docs.llamaindex.ai/en/stable/module_guides/evaluating/
    
38. [Literature Review] RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems - Moonlight, accessed August 12, 2025, https://www.themoonlight.io/en/review/ragbench-explainable-benchmark-for-retrieval-augmented-generation-systems
    
39. RAGEval: Scenario Specific RAG Evaluation Dataset Generation …, accessed August 12, 2025, https://arxiv.org/abs/2408.01262
    
40. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems, accessed August 12, 2025, https://www.promptlayer.com/research-papers/ragbench-explainable-benchmark-for-retrieval-augmented-generation-systems
    
41. (PDF) RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems, accessed August 12, 2025, https://www.researchgate.net/publication/382301929_RAGBench_Explainable_Benchmark_for_Retrieval-Augmented_Generation_Systems
    
42. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems - arXiv, accessed August 12, 2025, https://arxiv.org/html/2407.11005v1
    
43. Comprehensive RAG Benchmark - NeurIPS Poster CRAG, accessed August 12, 2025, https://neurips.cc/virtual/2024/poster/97703
    
44. RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems, accessed August 12, 2025, https://openreview.net/forum?id=KDXj60FpJr
    
45. REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark for ACL 2025, accessed August 12, 2025, https://research.ibm.com/publications/real-mm-rag-a-real-world-multi-modal-retrieval-benchmark
    
46. RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance - arXiv, accessed August 12, 2025, https://arxiv.org/abs/2501.03995
    
47. [2404.13781] Evaluating Retrieval Quality in Retrieval-Augmented Generation - arXiv, accessed August 12, 2025, https://arxiv.org/abs/2404.13781
    
48. MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems for ACL 2025 - IBM Research, accessed August 12, 2025, https://research.ibm.com/publications/mtrag-a-multi-turn-conversational-benchmark-for-evaluating-retrieval-augmented-generation-systems
    
49. RAG Evaluation - Hugging Face Open-Source AI Cookbook, accessed August 12, 2025, https://huggingface.co/learn/cookbook/rag_evaluation
    
50. Tracing LangChain applications with OpenTelemetry - New Relic, accessed August 12, 2025, https://newrelic.com/blog/how-to-relic/tracing-langchain-applications-with-opentelemetry
    
51. Improve your RAG Results using LangSmith - Focused Labs, accessed August 12, 2025, https://focused.io/lab/improve-your-rag-results-using-langsmith-2
    
52. GenerativeAIExamples/RAG/tools/observability/README.md at main - GitHub, accessed August 12, 2025, https://github.com/NVIDIA/GenerativeAIExamples/blob/main/RAG/tools/observability/README.md
    
53. Traces | OpenTelemetry, accessed August 12, 2025, https://opentelemetry.io/docs/concepts/signals/traces/
    
54. Overview: Tracing | Phoenix - Arize AI, accessed August 12, 2025, https://arize.com/docs/phoenix/tracing/llm-traces
    
55. Concepts - ️🛠️ LangSmith - LangChain, accessed August 12, 2025, https://docs.smith.langchain.com/observability/concepts
    
56. OpenTelemetry (OTEL) Concepts: Span, Trace, Session - Arize AI, accessed August 12, 2025, https://arize.com/opentelemetry-otel-concepts-span-trace-session/
    
57. Setup using base OTEL | Phoenix - Arize AI, accessed August 12, 2025, https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans
    
58. AI Agent Observability - Evolving Standards and Best Practices - OpenTelemetry, accessed August 12, 2025, https://opentelemetry.io/blog/2025/ai-agent-observability/
    
59. OpenTelemetry for AI Systems: Implementation Guide - Uptrace, accessed August 12, 2025, https://uptrace.dev/blog/opentelemetry-ai-systems
    
60. Open Source LLM Observability via OpenTelemetry - Langfuse, accessed August 12, 2025, https://langfuse.com/docs/opentelemetry/get-started
    
61. Build a Retrieval Augmented Generation (RAG) App: Part 1 | 🦜️ LangChain, accessed August 12, 2025, https://python.langchain.com/docs/tutorials/rag/
    
62. Agentic RAG - GitHub Pages, accessed August 12, 2025, https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/
    
63. Agentic RAG Tracing | Phoenix - Arize AI, accessed August 12, 2025, https://arize.com/docs/phoenix/cookbook/tracing-and-annotations/agentic-rag-tracing
    
64. RAG Evaluation: The Definitive Guide to Unit Testing RAG in CI/CD - Confident AI, accessed August 12, 2025, https://www.confident-ai.com/blog/how-to-evaluate-rag-applications-in-ci-cd-pipelines-with-deepeval
    
65. Understanding Agentic RAG - Arize AI, accessed August 12, 2025, https://arize.com/blog/understanding-agentic-rag/