Evaluating Non-Deterministic Results From RAG Systems

By Alejandro Aroca Escribano, at Parser

Introduction

The Challenge of Testing AI Systems

In traditional software testing, creating automated test cases involves validating predictable outputs: the same input data should always produce the same results. This deterministic behaviour allows us to write simple assertions such as A == B.

However, AI systems behave differently. Their non-deterministic nature means the same input data can generate multiple valid outputs. These outputs may vary in wording, structure, or style, yet they could be equally valid based on their semantic content.

Look at the following example from our Pre-Sales agent in Parser, which contains documentation and lessons learned from previous projects:

• Question: What are common challenges faced in project implementations and how are they typically addressed?

Answer 1: Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are typically addressed by improving data governance, adopting agile methodologies, and enhancing communication and coordination across teams.

Answer 2: Challenges in project implementations, such as inconsistent data quality, inefficient processes, and inadequate communication, are typically resolved by strengthening data governance, implementing agile methodologies, and enhancing team coordination.

Answer 3: Project implementations face challenges like inconsistent data quality, inefficient processes, and poor communication, addressed by enhancing data governance, agile practices, and team coordination. Ensuring scalability and translating complex requirements also require robust planning and testing.

All three answers are correct, despite the variations in length, phrasing and details provided. Traditional assertion methods fall short in evaluating such outputs. This is where tools like Ragas come into play.

What is Ragas?

Ragas (Retrieval Augmented Generation Assessment) is a useful tool for evaluating the outputs of a RAG system, where additional documentation is included in the context of an LLM request.

In this approach, we use AI to evaluate AI. Ragas communicates with an LLM model that acts as an evaluator. Given a dataset containing prompts, contexts, actual responses and expected outputs, Ragas performs the evaluation and produces scores for each included metric.

Why Ragas?

Compared to other approaches, like LLM-as-judge, no prompt engineering is needed. It uses optimized, internally managed LLM prompts for each metric, ensuring repeatable and consistent results using objective and measurable criteria (each metric is well explained and documented). It’s also optimized for batch evaluation of large datasets, minimizing API usage and reducing cost, as less LLM calls are needed.

Compared to other similar available tools (like TruLens, BEIR and Promptfoo) Ragas is not a general-purpose LLM evaluator, but a purpose-built tool for evaluating end-to-end RAG pipelines. It focuses specifically on the quality of retrieval and generation within the context of retrieved documents, evaluating both the retriever and the LLM response, and does not require extensive human annotation for every sample.

Use case | Step-by-Step: Evaluating a RAG System with Ragas

Let’s walk you through how to implement Ragas in a real-world project:

1. Installation

Ragas is a Python tool, and you can easily install it using pip.

pip install ragas

1. Importing required modules

In a Python project we will need at least the following:

from ragas import evaluate

This is the basic import used for evaluating the dataset and generating results.

It is possible to use Ragas without specifying any metrics or LLM models, in which case it will apply default values. Otherwise, we will need to import the desired metrics and the wrappers for the LLMs (both the Model and the Embeddings).

Ragas provides many metrics that can be used out of the box, and we can also create other custom metrics. See the Ragas documentation for more details.

1. Set API Key

Depending on the evaluator LLM provider (e.g. OpenAI, Amazon Bedrock, Azure OpenAI) used, the corresponding API key needs to be set, either as an environment variable in the system or programmatically.

For OpenAI:

import os
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")

1. Metrics Object

A dictionary must be created, including each metric to be analysed and the LLM Model / Embeddings to be used for each of them.

metrics = [
AnswerRelevancy(llm=evaluator_llm),

ContextPrecision(llm=evaluator_llm),

FactualCorrectness(llm=evaluator_llm),

LLMContextRecall(llm=evaluator_llm),

SemanticSimilarity(embeddings=evaluator_embeddings)
]

This object will be used to evaluate the results, along with the dataset.

1. Dataset

The dataset can be sourced externally (e.g. from a previous generated file) or created in-memory, if it was generated as part of our test.

The expected input is a Python dictionary with arrays of questions, answers, contexts, and ground truths. For contexts, we have a bi-dimensional array, as we can include more than one entry for each request.

Example:

question = ["What are common challenges faced in project implementations and how are they typically addressed?"]

answer = ["Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are typically addressed by improving data governance, adopting agile methodologies, and enhancing communication and coordination across teams."]

contexts = [["Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are addressed by improving data governance, adopting agile methodologies, and enhancing team coordination. Additionally, translating complex requirements into actionable plans and ensuring scalability and integration are significant challenges. These require robust planning, execution strategies, and thorough testing. Overall, strategic planning and agile practices are key to overcoming these challenges."]]

ground_truth = ["Project implementations face challenges like inconsistent data quality, inefficient processes, and poor communication, addressed by enhancing data governance, agile practices, and team coordination. Ensuring scalability and translating complex requirements also require robust planning and testing."]

This example contains just one element. In a real scenario, we would have a list of questions/prompts with the corresponding answers, contexts, and ground truths.

When testing a RAG system, three of these inputs are typically known in advance (question, context, and ground truth), and could be obtained from a CSV file, a database, or the code itself. The answers will be dynamically generated as we send each question to the LLM that we are evaluating. Once obtained, we can complete the dictionary by appending the actual response to each row.

1. Run the evaluation

To run the evaluation, use the evaluate method from Ragas, including the Dataset and Metrics.

results = evaluate(dataset=dataset, metrics=metrics)

After a while, we will obtain the Evaluation Result object with the scores for every row and metric.

If printed directly, it will show a JSON object with the average score for each metric, but we can convert it into a Pandas object to obtain results in different formats, such as CSV or HTML.

When using a reporting tool like Allure (for example, with Behave), these results can be attached to the report for easier review.

Analysing Trends in The Medium to Long Term

A useful approach in the medium to long term is to store these scores in a database or CSV file, enabling trend analysis and comparison against a Baseline.

This is especially useful when introducing changes to your RAG system, such as a new model, configuration adjustments, prompts updates, and so on.

When running these tests regularly, we’ll obtain scores that can be compared against previous iterations, helping to identify whether the system is performing better or worse after the changes. This enables us to make informed decisions and prioritise efforts.

Using Plotly

We can use a tool like Plotly to generate charts that provide quick, visual feedback about the metrics.

Plotly can be installed using pip:

pip install plotly

In the following example, we add two traces to a radar plot. The first one (current_results) contains the current test results, and the second one (baseline_scores) represents the baseline scores, allowing a side-by-side comparison.

Basic Test with Ragas and Ploty

The following metrics have been used in this example:

• Answer Relevancy: It shows how well a reply matches the user’s question. A higher score means the answer is clear, complete, and focused, while a lower score means it might be missing information or includes unnecessary details.
• Context Precision: It tells how much of the background information given is truly helpful for answering a question. A high score means that most of the information provided (retrieved chunks) is truly relevant to the question being asked.
• Factual Correctness: This metric measures how true and accurate a response is by comparing it to a reliable answer. The score goes from 0 to 1, where a higher number means the response is more factually accurate and trustworthy.
• Context Recall: It checks if the system found all the important information needed to answer a question. A high score means it included most or all of the useful details, without leaving anything important out.
• Semantic similarity: This metric measures how similar the meaning of a response is to the correct or expected answer. The score ranges from 0 to 1, where higher values mean the response better captures the intended meaning.

Colab Notebook including this example: https://colab.research.google.com/drive/1kpnHV65ZkrGsrfzLugemogmUb4egJh4I

Analysis of Results

Implementing this approach allows teams to:

• Automatically evaluate non-deterministic outputs: This tool is useful for evaluating responses of Generative AI applications, such as RAG systems, which are non-deterministic. Rather than relying on boolean assertion (correct/incorrect), we can obtain different metrics that can be used to determine the correctness of responses.
• Analyse trends over time: Some specific score from a metric can be useful, but it will be really powerful when we use it to compare different iterations over time. As we introduce changes in our system, this approach can be useful for identifying how better/worse some components are performing after the modifications. This way we can approve or discard these changes, being aware about the impact of certain updates and prioritising the next steps.
• Obtain visual feedback: Using a tool like Plotly to generate charts is highly effective for obtaining quick feedback about the metrics, also for comparing against a baseline to identify which of them have improved or worsened, and by how much.

Conclusions

This is an effective approach for addressing the challenges of testing non-deterministic applications built on top of Generative AI models, such as in the client’s project, where Ragas proved to be a valuable asset during the evaluation process. As lessons learnt, the most challenging aspect was the generation of evaluation data from our dataset, which required domain expertise and considerable time investment, but the insight and automation that Ragas brought to our evaluation workflow were well worth the effort.

The practical example provided is just one way of applying this method, but it can be extrapolated and adapted to suit different contexts and requirements. As the AI landscape is rapidly evolving, tools like Ragas are constantly being updated, and new tools continue to emerge. It’s therefore important to stay up-to-date with the trends and technologies to refine and enhance your testing strategy for Generative AI systems.