# Retrieval Verification: RAGAS_Automated_Evaluation_of_Retrieval_Augmented_Generation_parsed

## Pilar: Background
**Query:** _Latar belakang, masalah, dan motivasi penelitian ini. Background, research problem, motivation, context, introduction, underlying issue._

### Rank 1 (Score: 0.2904)
**Header:** **3 Evaluation Strategies**
**Content:**
```text
7
AR
sim(q, qi)
(1)
n
=1  
This metric evaluates how closely the generated answer aligns with the initial question or instruction.  
**Context relevance** The context _c_ ( _q_ ) is considered relevant to the extent that it exclusively contains information that is needed to answer the question. In particular, this metric aims to penalise the inclusion of redundant information. To estimate context relevance, given a question _q_ and its context _c_ ( _q_ ), the LLM extracts a subset of sentences, _Sext_ , from _c_ ( _q_ ) that are crucial to answer _q_ , using the following prompt:  
- _Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you’re not allowed to make any changes to sentences from given context._  
The context relevance score is then computed as:  
- number of extracted sentences  
- CR = (2) total number of sentences in _c_ ( _q_ )
```

### Rank 2 (Score: 0.2402)
**Header:** **5 Experiments**
**Content:**
```text
_answer 1_ : [answer 1] _answer 2_ : [answer 2]  
The results in Table 4 show that our proposed metrics are much closer aligned with the human judgements than the predictions from the two baselines. For faithfulness, the RAGAS prediction are in general highly accurate. For answer relevance, the agreement is lower, but this is largely due to the fact that the differences between the two candidate answers are often very subtle. We found context relevance to be the hardest quality dimension to evaluate. In particular, we observed that ChatGPT often struggles with the task of selecting the sentences from the context that are crucial, especially for longer contexts.
```

### Rank 3 (Score: 0.2377)
**Header:** **4 The WikiEval Dataset**
**Content:**
```text
_6. Do not use phrases that ‘provided context’, etc in the question_  
- _context:_  
We also used ChatGPT to answer the generated question, when given the corresponding introductory section as context, using the following prompt:  
_Answer the question using the information from the given context._  
_question:_ [question] _context:_ [context]  
All questions were annotated along the three considered quality dimensions by two annotators. Both annotators were fluent in English and were given clear instructions about the meaning of the three considered quality dimensions. For faithfulness and context relevance, the two annotators agreed in around 95% of cases. For answer relevance, they agreed in around 90% of the cases. Disagreements were resolved after a discussion between the annotators. Tables 1, 2 and 3 show examples from the WikiEval dataset, focusing in particular on answers with high and low faithfulness (Table 1), high and low answer relevance (Table 2), and high and low context relevance (Table 3).  
**Faithfulness** To obtain human judgements about faithfulness, we first used ChatGPT to answer the question without access to any additional context. We then asked the annotators to judge which of the two answers was the most faithful (i.e. the standard one or the one generated without context), given the question and corresponding Wikipedia page.
```

## Pilar: Methodology
**Query:** _Metode, pendekatan, arsitektur, algoritma, eksperimen, atau dataset yang digunakan. Methodology, proposed method, approach, system architecture, dataset, materials, experimental setup, procedure._

### Rank 1 (Score: 0.3875)
**Header:** **6 Python API**
**Content:**
```text
RAGAS provides access to metrics and datasets via an easy-to-use Python API. Its syntax is similar to other well-known libraries such as transformers or datasets. As an example, once installed, loading a dataset, evaluating a pipeline with the desired metrics, and exporting the results to a pandas dataframe can be accomplished with the snippet below. The metrics available at ragas.metrics use OpenAI’s API by default, which requires having the appropriate environment variables set up. It is however possible to experiment with other LLMs
```

### Rank 2 (Score: 0.3827)
**Header:** **7 Conclusions** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
We have highlighted the need for automated reference-free evaluation of RAG systems. In particular, we have argued the need for an evaluation framework that can assess faithfulness (i.e. is the answer grounded in the retrieved context), answer relevance (i.e. does the answer address the question) and context relevance (i.e. is the retrieved context sufficiently focused). To support the development of such a framework, we have introduced _WikiEval_ , a dataset which human judgements of these three different aspects. Finally, we have also described RAGAS, our implementation of the three considered quality aspects. This framework is easy to use and can provide developers of RAG systems with valuable insights, even in the absence of any ground truth. Our evaluation on WikiEval has shown that the predictions from RAGAS are closely aligned with human judgments, especially for faithfulness and answer relevance.
```

### Rank 3 (Score: 0.3727)
**Header:** **8 Limitations** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
This paper introduces a toolkit aimed at providing an end-to-end evaluation framework for RAG  
> 6See https://github.com/explodinggradients/ ragas/blob/main/docs/howtos/customisations/llms. ipynb.  
systems. It relies heavily on the performance of the LLMs used for evaluating the different components. While the current set of experiments demonstrate high correlation of these metrics with human judgements, we acknowledge that relying on LLMs comes with known limitations. Therefore, careful reviewing of the suitability of LLMs (ideally prioritizing open models over full-fledged products behind paid APIs) is critical for RAGAS, and other contributions in this space, to nurture a healthy environment. We are also aware of the potential implications of enabling large user bases into using more accurate RAG systems, and therefore we will continue to encourage applications of fair systems on top of RAGAS.
```

## Pilar: Gap
**Query:** _Gap penelitian, keterbatasan, kelemahan, atau masalah yang belum terselesaikan. Research gap, limitations, weakness, unresolved problem, remains a challenge, yet to be explored, however._

### Rank 1 (Score: 0.2750)
**Header:** **References** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
- Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. _CoRR_ , abs/2207.05221.  
- Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. _CoRR_ , abs/2211.08411.  
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ . OpenReview.net.  
- Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. _CoRR_ , abs/2212.14024.
```

### Rank 2 (Score: 0.2683)
**Header:** **References** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
- Amos Azaria and Tom M. Mitchell. 2023. The internal state of an LLM knows when its lying. _CoRR_ , abs/2304.13734.  
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.  
- Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. _arXiv preprint arXiv:2303.12712_ .
```

### Rank 3 (Score: 0.2645)
**Header:** **7 Conclusions** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
We have highlighted the need for automated reference-free evaluation of RAG systems. In particular, we have argued the need for an evaluation framework that can assess faithfulness (i.e. is the answer grounded in the retrieved context), answer relevance (i.e. does the answer address the question) and context relevance (i.e. is the retrieved context sufficiently focused). To support the development of such a framework, we have introduced _WikiEval_ , a dataset which human judgements of these three different aspects. Finally, we have also described RAGAS, our implementation of the three considered quality aspects. This framework is easy to use and can provide developers of RAG systems with valuable insights, even in the absence of any ground truth. Our evaluation on WikiEval has shown that the predictions from RAGAS are closely aligned with human judgments, especially for faithfulness and answer relevance.
```

## Pilar: Objective
**Query:** _Tujuan utama, sasaran, atau pertanyaan penelitian yang ingin dijawab. Main objective, research goal, purpose of this study, aims to investigate, to evaluate, this paper proposes._

### Rank 1 (Score: 0.3665)
**Header:** **7 Conclusions** / export results to pandas dataframe df = result.to_pandas ()
**Content:**
```text
We have highlighted the need for automated reference-free evaluation of RAG systems. In particular, we have argued the need for an evaluation framework that can assess faithfulness (i.e. is the answer grounded in the retrieved context), answer relevance (i.e. does the answer address the question) and context relevance (i.e. is the retrieved context sufficiently focused). To support the development of such a framework, we have introduced _WikiEval_ , a dataset which human judgements of these three different aspects. Finally, we have also described RAGAS, our implementation of the three considered quality aspects. This framework is easy to use and can provide developers of RAG systems with valuable insights, even in the absence of any ground truth. Our evaluation on WikiEval has shown that the predictions from RAGAS are closely aligned with human judgments, especially for faithfulness and answer relevance.
```

### Rank 2 (Score: 0.3401)
**Header:** **5 Experiments**
**Content:**
```text
Table 4 analyses the agreement between the metrics proposed in Section 3 and the human assessments from the proposed WikiEval dataset. Each WikiEval instance requires the model to compare two answers or two context fragments. We count how often the answer/context preferred by the model (i.e. with highest estimated faithfulness, answer relevance, or context relevance) coincides with the answer/context preferred by the human annotators. We report the results in terms of accuracy (i.e. the fraction of instances on which the model agrees with the annotators).  
To put the results in context, we compare our proposed metrics (shown as RAGAS in Table 4) with two baseline methods. For the first method, shown as _GPT Score_ , we ask ChatGPT to assign a score between 0 and 10 for the three quality dimensions. To this end, we use a prompt that describes  
the meaning of the quality metric and then asks to score the given answer/context in line with that definition. For instance, for evaluating faithfulness, we used the following prompt:  
- _Faithfulness measures the information consistency of the answer against the given context. Any claims that are made in the answer that cannot be deduced from context should be penalized. Given an answer and context, assign a score for faithfulness in the range 0-10. context_ : [context] _answer_ : [answer]
```

### Rank 3 (Score: 0.2837)
**Header:** **4 The WikiEval Dataset**
**Content:**
```text
To evaluate the proposed framework, we ideally need examples of question-context-answer triples which are annotated with human judgments. We can then verify to what extent our metrics agree with human assessments of faithfulness, answer relevance and context relevance. Since we are not aware of any publicly available datasets that could be used for this purpose, we created a new dataset, which we refer to as _WikiEval_[4] . To construct the dataset, we first selected 50 Wikipedia pages covering events that have happened since the start of 2022[5] . In selecting these pages, we prioritised those with recent edits. For each of the 50 pages, we then asked ChatGPT to suggest a question that can be answered based on the introductory section of the page, using the following prompt:  
_Your task is to formulate a question from given context satisfying the rules given below:_  
_1. The question should be fully answered from the given context._  
4https://huggingface.co/datasets/ explodinggradients/WikiEval.  
- 5That is, beyond the reported training cutoff of the model  
- we used in our experiments.  
_2. The question should be framed from a part that contains non-trivial information._  
_3. The answer should not contain any links._  
_4. The question should be of moderate difficulty._  
_5. The question must be reasonable and must be understood and responded to by humans._  
_6. Do not use phrases that ‘provided context’, etc in the question_  
- _context:_
```

## Pilar: Futurework
**Query:** _Pekerjaan di masa depan, saran untuk peneliti lain, atau langkah selanjutnya. Future work, recommendations, future research directions, next steps, further investigation._

### Rank 1 (Score: 0.2993)
**Header:** **5 Experiments**
**Content:**
```text
Ties, where the same score is assigned by the LLM to both answer candidates, were broken randomly. The second baseline, shown as _GPT Ranking_ , instead asks ChatGPT to select the preferred answer/context. In this case, the prompt again includes a definition of the considered quality metric. For instance, to evaluate answer relevance, we used the following prompt:  
- _Answer Relevancy measures the degree to which a response directly addresses and is appropriate for a given question. It penalizes the present of redundant information or incomplete answers given a question. Given an question and answer, rank each answer based on Answer Relevancy._  
_question_ : [question]  
|**Question**|**Context**|
|---|---|
|When was the Chimnabai|_High context relevance_: The Chimnabai Clock Tower, also known as the Raopura Tower, is|
|Clock Tower completed,<br>and who was it named af-|a clock tower situated in the Raopura area of Vadodara, Gujarat, India. It was completed<br>in 1896 and named in memory of Chimnabai I (1864–1885), a queen and the frst wife of|
|ter?|Sayajirao Gaekwad III of Baroda State.|
||_Low context relevance_: The Chimnabai Clock Tower, also known as the Raopura Tower, is|
||a clock tower situated in the Raopura area of Vadodara, Gujarat, India. It was completed<br>in 1896 and named in memory of Chimnabai I (1864–1885), a queen and the frst wife of|
||Sayajirao Gaekwad III of Baroda State. It was built in Indo-Saracenic architecture style.|
```

### Rank 2 (Score: 0.2993)
**Header:** **5 Experiments**
**Content:**
```text
||Sayajirao Gaekwad III of Baroda State. It was built in Indo-Saracenic architecture style.|
||History. Chimnabai Clock Tower was built in 1896. The tower was named after Chimnabai<br>I (1864–1885), a queen and the frst wife of Sayajirao Gaekwad III of Baroda State. It was|
||inaugurated by Mir Kamaluddin Hussainkhan, the last Nawab of Baroda. During the rule of|
||Gaekwad, it was a stoppage for horse drawn trams. The clock tower was erected at the cost|
||of 25,000 (equivalent to 9.2 million or USD 120,000 in 2023).|  
Table 3: Example from WikiEval, showing answers with high and low context relevance.  
||**Faith.**|**Ans. Rel.**|**Cont. Rel.**|
|---|---|---|---|
|RAGAS|**0.95**|**0.78**|**0.70**|
|GPT Score|0.72|0.52|0.63|
|GPT Ranking|0.54|0.40|0.52|  
Table 4: Agreement with human annotators in pairwise comparisons of faithfulness, answer relevance and context relevance, using the WikEval dataset (accuracy).  
erate outputs in structured JSON format. We found that this largely makes RAGAS compatible with different LLMs, and ultimately lowers the error rate when consuming LLM generated text. To measure the effectiveness of the JSON-formatting, we measured the correlation between RAGAS scores in successive runs with and without JSON formatting. As shown in Figure 1, the scores are clearly more consistent when JSON formatted outputs are used.  
_answer 1_ : [answer 1] _answer 2_ : [answer 2]
```

### Rank 3 (Score: 0.2502)
**Header:** **1 Introduction**
**Content:**
```text
While the usefulness of retrieval-augmented strategies is clear, their implementation requires a significant amount of tuning, as the overall performance will be affected by the retrieval model, the considered corpus, the LM and the prompt formulation, among others. Automated evaluation of retrieval-augmented systems is thus paramount. In practice, RAG systems are often evaluated in terms of the language modelling task itself, i.e. by measuring perplexity on some reference corpus. However, such evaluations are not always predictive of downstream performance (Wang et al., 2023c). Moreover, this evaluation strategy relies on the LM probabilities, which are not accessible for some closed models (e.g. ChatGPT and GPT-4). Question answering is another common evaluation task, but usually only datasets with short extractive answers are considered, which may not be representative of how the system will be used.  
To address these issues, in this paper we present RAGAS, a framework for the automated assess-  
ment of retrieval augmented generation systems. We focus on settings where reference answers may not be available, and where we want to estimate different proxies for correctness, in addition to the usefulness of the retrieved passages. The RAGAS framework provides an integration with both llamaindex and Langchain, the most widely used frameworks for building RAG solutions, thus enabling developers to easily integrate RAGAS into their standard workflow.
```
