Evaluating Non-Deterministic Results From RAG Systems

By Alejandro Aroca Escribano, at Parser

Introduction

The Challenge of Testing AI Systems

In traditional software testing, creating automated test cases involves validating predictable outputs: the same input data should always produce the same results. This deterministic behaviour allows us to write simple assertions such as A == B.

However, AI systems behave differently. Their non-deterministic nature means the same input data can generate multiple valid outputs. These outputs may vary in wording, structure, or style, yet they could be equally valid based on their semantic content.

Look at the following example from our Pre-Sales agent in Parser, which contains documentation and lessons learned from previous projects:

• Question: What are common challenges faced in project implementations and how are they typically addressed?

Answer 1: Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are typically addressed by improving data governance, adopting agile methodologies, and enhancing communication and coordination across teams.

Answer 2: Challenges in project implementations, such as inconsistent data quality, inefficient processes, and inadequate communication, are typically resolved by strengthening data governance, implementing agile methodologies, and enhancing team coordination.

Answer 3: Project implementations face challenges like inconsistent data quality, inefficient processes, and poor communication, addressed by enhancing data governance, agile practices, and team coordination. Ensuring scalability and translating complex requirements also require robust planning and testing.

All three answers are correct, despite the variations in length, phrasing and details provided. Traditional assertion methods fall short in evaluating such outputs. This is where tools like Ragas come into play.

What is Ragas?

Ragas (Retrieval Augmented Generation Assessment) is a useful tool for evaluating the outputs of a RAG system, where additional documentation is included in the context of an LLM request.

In this approach, we use AI to evaluate AI. Ragas communicates with an LLM model that acts as an evaluator. Given a dataset containing prompts, contexts, actual responses and expected outputs, Ragas performs the evaluation and produces scores for each included metric.

Why Ragas?

Compared to other approaches, like LLM-as-judge, no prompt engineering is needed. It uses optimized, internally managed LLM prompts for each metric, ensuring repeatable and consistent results using objective and measurable criteria (each metric is well explained and documented). It’s also optimized for batch evaluation of large datasets, minimizing API usage and reducing cost, as less LLM calls are needed.

Compared to other similar available tools (like TruLens, BEIR and Promptfoo) Ragas is not a general-purpose LLM evaluator, but a purpose-built tool for evaluating end-to-end RAG pipelines. It focuses specifically on the quality of retrieval and generation within the context of retrieved documents, evaluating both the retriever and the LLM response, and does not require extensive human annotation for every sample.

Use case | Step-by-Step: Evaluating a RAG System with Ragas

Let’s walk you through how to implement Ragas in a real-world project:

1. Installation

Ragas is a Python tool, and you can easily install it using pip.

pip install ragas

1. Importing required modules

In a Python project we will need at least the following:

from ragas import evaluate

This is the basic import used for evaluating the dataset and generating results.

It is possible to use Ragas without specifying any metrics or LLM models, in which case it will apply default values. Otherwise, we will need to import the desired metrics and the wrappers for the LLMs (both the Model and the Embeddings).

Ragas provides many metrics that can be used out of the box, and we can also create other custom metrics. See the Ragas documentation for more details.

1. Set API Key

Depending on the evaluator LLM provider (e.g. OpenAI, Amazon Bedrock, Azure OpenAI) used, the corresponding API key needs to be set, either as an environment variable in the system or programmatically.

For OpenAI:

import os
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")

1. Metrics Object

A dictionary must be created, including each metric to be analysed and the LLM Model / Embeddings to be used for each of them.

metrics = [
        AnswerRelevancy(llm=evaluator_llm),

        ContextPrecision(llm=evaluator_llm),

        FactualCorrectness(llm=evaluator_llm),

        LLMContextRecall(llm=evaluator_llm),

        SemanticSimilarity(embeddings=evaluator_embeddings)
 ]

This object will be used to evaluate the results, along with the dataset.

1. Dataset

The dataset can be sourced externally (e.g. from a previous generated file) or created in-memory, if it was generated as part of our test.

The expected input is a Python dictionary with arrays of questions, answers, contexts, and ground truths. For contexts, we have a bi-dimensional array, as we can include more than one entry for each request.

Example:

question = ["What are common challenges faced in project implementations and how are they typically addressed?"]

  answer = ["Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are typically addressed by improving data governance, adopting agile methodologies, and enhancing communication and coordination across teams."]

  contexts = [["Common challenges in project implementations include inconsistent data quality, inefficient processes, and poor communication. These are addressed by improving data governance, adopting agile methodologies, and enhancing team coordination. Additionally, translating complex requirements into actionable plans and ensuring scalability and integration are significant challenges. These require robust planning, execution strategies, and thorough testing. Overall, strategic planning and agile practices are key to overcoming these challenges."]]

  ground_truth = ["Project implementations face challenges like inconsistent data quality, inefficient processes, and poor communication, addressed by enhancing data governance, agile practices, and team coordination. Ensuring scalability and translating complex requirements also require robust planning and testing."]

This example contains just one element. In a real scenario, we would have a list of questions/prompts with the corresponding answers, contexts, and ground truths.

When testing a RAG system, three of these inputs are typically known in advance (question, context, and ground truth), and could be obtained from a CSV file, a database, or the code itself. The answers will be dynamically generated as we send each question to the LLM that we are evaluating. Once obtained, we can complete the dictionary by appending the actual response to each row.

1. Run the evaluation

To run the evaluation, use the evaluate method from Ragas, including the Dataset and Metrics.

results = evaluate(dataset=dataset, metrics=metrics)

After a while, we will obtain the Evaluation Result object with the scores for every row and metric.

If printed directly, it will show a JSON object with the average score for each metric, but we can convert it into a Pandas object to obtain results in different formats, such as CSV or HTML.

When using a reporting tool like Allure (for example, with Behave), these results can be attached to the report for easier review.