# Retrieval Verification: Retrieval_Augmented_Generation_for_Knowledge_Intensive_NLP_Tasks_parsed

## Pilar: Background
**Query:** _Fenomena ini merupakan isu krusial. Oleh karena itu, studi ini dilakukan untuk merespons urgensi permasalahan yang melatarbelakangi penelitian._

### Rank 1 (Score: 0.8026)
**Header:** **1 Introduction** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Margin-
Fact Verification:
alize
Label Generation
The Divine
Comedy
(x)
MIPS
Pe
This 14th century work
is divided into 3
Jeopardy Question
sectlons: "Inferno"
Generation:
"purgatorio" &
Answer Query
"paradiso"
(y)
Question Generation  
**----- Start of picture text -----**<br>
Define�"middle�ea�" (x) The�middle�ea��incl�de�<br>Q�e��i�� A���e�i�g: End-to-End Backprop through  �  and� pθ �he���mpanic�ca�i���and�he��h�ee�o��icle�.�� (y)<br>Q�e��i�� Q�e�� E�c�de�Q�e�� Re��ie�e� �η D�c�me��I�de� Gene�a�o���� Q�e��i�� A���e�i�g:A���e� Ge�e�a�i��<br>Ba�ack�Obama��a� (N��-Pa�a�e���c) (Pa�a�e���c)<br>bo�n�in�Ha�aii. (x) �(�) d(�) z4 ��ppo���� (y)<br>Fac� Ve�i�ca�i��: Fac� Q�e�� z2 z3 Ma�gin-ali�e Labe� Ge�e�a�i��Fac� Ve�i�ca�i��:<br>The�Di�ineComed�� (x) � MIPS z1 pθ Thi��14�h�cen������o�ki��di�ided�in�o�3<br>Je��a�d� Q�e��i�� �ec�ion�:�"Infe�no",<br>Ge�e�a�i��: "P��ga�o�io"�&<br>A���e� Q�e�� "Pa�adi�o"��������� (y)<br>Q�e��i�� Ge�e�a�i��<br>**----- End of picture text -----**<br>  
Figure 1: Overview of our approach. We combine a pre-trained retriever ( _Query Encoder_ + _Document Index_ ) with a pre-trained seq2seq model ( _Generator_ ) and fine-tune end-to-end. For query _x_ , we use Maximum Inner Product Search (MIPS) to find the top-K documents _zi_ . For final prediction _y_ , we treat _z_ as a latent variable and marginalize over seq2seq predictions given different documents.
```

### Rank 2 (Score: 0.8019)
**Header:** **Broader Impact** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
This work offers several positive societal benefits over previous work: the fact that it is more strongly grounded in real factual knowledge (in this case Wikipedia) makes it “hallucinate” less with generations that are more factual, and offers more control and interpretability. RAG could be employed in a wide variety of scenarios with direct benefit to society, for example by endowing it with a medical index and asking it open-domain questions on that topic, or by helping people be more effective at their jobs.  
With these advantages also come potential downsides: Wikipedia, or any potential external knowledge source, will probably never be entirely factual and completely devoid of bias. Since RAG can be employed as a language model, similar concerns as for GPT-2 [50] are valid here, although arguably to a lesser extent, including that it might be used to generate abuse, faked or misleading content in the news or on social media; to impersonate others; or to automate the production of spam/phishing content [54]. Advanced language models may also lead to the automation of various jobs in the coming decades [16]. In order to mitigate these risks, AI systems could be employed to fight against misleading content and automated spam/phishing.
```

### Rank 3 (Score: 0.8001)
**Header:** **Abstract** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, and another which can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
```

## Pilar: Methodology
**Query:** _Penelitian ini menggunakan pendekatan terstruktur di mana data dikumpulkan dan analisis dilakukan menggunakan teknik serta metode yang dijelaskan pada bagian ini._

### Rank 1 (Score: 0.8418)
**Header:** **1 Introduction** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Margin-
Fact Verification:
alize
Label Generation
The Divine
Comedy
(x)
MIPS
Pe
This 14th century work
is divided into 3
Jeopardy Question
sectlons: "Inferno"
Generation:
"purgatorio" &
Answer Query
"paradiso"
(y)
Question Generation  
**----- Start of picture text -----**<br>
Define�"middle�ea�" (x) The�middle�ea��incl�de�<br>Q�e��i�� A���e�i�g: End-to-End Backprop through  �  and� pθ �he���mpanic�ca�i���and�he��h�ee�o��icle�.�� (y)<br>Q�e��i�� Q�e�� E�c�de�Q�e�� Re��ie�e� �η D�c�me��I�de� Gene�a�o���� Q�e��i�� A���e�i�g:A���e� Ge�e�a�i��<br>Ba�ack�Obama��a� (N��-Pa�a�e���c) (Pa�a�e���c)<br>bo�n�in�Ha�aii. (x) �(�) d(�) z4 ��ppo���� (y)<br>Fac� Ve�i�ca�i��: Fac� Q�e�� z2 z3 Ma�gin-ali�e Labe� Ge�e�a�i��Fac� Ve�i�ca�i��:<br>The�Di�ineComed�� (x) � MIPS z1 pθ Thi��14�h�cen������o�ki��di�ided�in�o�3<br>Je��a�d� Q�e��i�� �ec�ion�:�"Infe�no",<br>Ge�e�a�i��: "P��ga�o�io"�&<br>A���e� Q�e�� "Pa�adi�o"��������� (y)<br>Q�e��i�� Ge�e�a�i��<br>**----- End of picture text -----**<br>  
Figure 1: Overview of our approach. We combine a pre-trained retriever ( _Query Encoder_ + _Document Index_ ) with a pre-trained seq2seq model ( _Generator_ ) and fine-tune end-to-end. For query _x_ , we use Maximum Inner Product Search (MIPS) to find the top-K documents _zi_ . For final prediction _y_ , we treat _z_ as a latent variable and marginalize over seq2seq predictions given different documents.
```

### Rank 2 (Score: 0.8304)
**Header:** **4.5 Additional Results** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Effect of Retrieving more documents** Models are trained with either 5 or 10 retrieved latent documents, and we do not observe significant differences in performance between them. We have the flexibility to adjust the number of retrieved documents at test time, which can affect performance and runtime. Figure 3 (left) shows that retrieving more documents at test time monotonically improves Open-domain QA results for RAG-Sequence, but performance peaks for RAG-Token at 10 retrieved documents. Figure 3 (right) shows that retrieving more documents leads to higher Rouge-L for RAG-Token at the expense of Bleu-1, but the effect is less pronounced for RAG-Sequence.
```

### Rank 3 (Score: 0.8282)
**Header:** **3.2 Abstractive Question Answering** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
RAG models can go beyond simple extractive QA and answer questions with free-form, abstractive text generation. To test RAG’s natural language generation (NLG) in a knowledge-intensive setting, we use the MSMARCO NLG task v2.1 [43]. The task consists of questions, ten gold passages retrieved from a search engine for each question, and a full sentence answer annotated from the retrieved passages. We do not use the supplied passages, only the questions and answers, to treat  
MSMARCO as an open-domain abstractive QA task. MSMARCO has some questions that cannot be answered in a way that matches the reference answer without access to the gold passages, such as “What is the weather in Volcano, CA?” so performance will be lower without using gold passages. We also note that some MSMARCO questions cannot be answered using Wikipedia alone. Here, RAG can rely on parametric knowledge to generate reasonable responses.
```

## Pilar: Gap
**Query:** _Meskipun banyak studi telah dilakukan, masih terdapat kekosongan literatur (gap), kelemahan, dan batasan pada penelitian sebelumnya yang belum dieksplorasi._

### Rank 1 (Score: 0.7982)
**Header:** **Abstract** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, and another which can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
```

### Rank 2 (Score: 0.7952)
**Header:** **1 Introduction** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Margin-
Fact Verification:
alize
Label Generation
The Divine
Comedy
(x)
MIPS
Pe
This 14th century work
is divided into 3
Jeopardy Question
sectlons: "Inferno"
Generation:
"purgatorio" &
Answer Query
"paradiso"
(y)
Question Generation  
**----- Start of picture text -----**<br>
Define�"middle�ea�" (x) The�middle�ea��incl�de�<br>Q�e��i�� A���e�i�g: End-to-End Backprop through  �  and� pθ �he���mpanic�ca�i���and�he��h�ee�o��icle�.�� (y)<br>Q�e��i�� Q�e�� E�c�de�Q�e�� Re��ie�e� �η D�c�me��I�de� Gene�a�o���� Q�e��i�� A���e�i�g:A���e� Ge�e�a�i��<br>Ba�ack�Obama��a� (N��-Pa�a�e���c) (Pa�a�e���c)<br>bo�n�in�Ha�aii. (x) �(�) d(�) z4 ��ppo���� (y)<br>Fac� Ve�i�ca�i��: Fac� Q�e�� z2 z3 Ma�gin-ali�e Labe� Ge�e�a�i��Fac� Ve�i�ca�i��:<br>The�Di�ineComed�� (x) � MIPS z1 pθ Thi��14�h�cen������o�ki��di�ided�in�o�3<br>Je��a�d� Q�e��i�� �ec�ion�:�"Infe�no",<br>Ge�e�a�i��: "P��ga�o�io"�&<br>A���e� Q�e�� "Pa�adi�o"��������� (y)<br>Q�e��i�� Ge�e�a�i��<br>**----- End of picture text -----**<br>  
Figure 1: Overview of our approach. We combine a pre-trained retriever ( _Query Encoder_ + _Document Index_ ) with a pre-trained seq2seq model ( _Generator_ ) and fine-tune end-to-end. For query _x_ , we use Maximum Inner Product Search (MIPS) to find the top-K documents _zi_ . For final prediction _y_ , we treat _z_ as a latent variable and marginalize over seq2seq predictions given different documents.
```

### Rank 3 (Score: 0.7874)
**Header:** **3.2 Abstractive Question Answering** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
RAG models can go beyond simple extractive QA and answer questions with free-form, abstractive text generation. To test RAG’s natural language generation (NLG) in a knowledge-intensive setting, we use the MSMARCO NLG task v2.1 [43]. The task consists of questions, ten gold passages retrieved from a search engine for each question, and a full sentence answer annotated from the retrieved passages. We do not use the supplied passages, only the questions and answers, to treat  
MSMARCO as an open-domain abstractive QA task. MSMARCO has some questions that cannot be answered in a way that matches the reference answer without access to the gold passages, such as “What is the weather in Volcano, CA?” so performance will be lower without using gold passages. We also note that some MSMARCO questions cannot be answered using Wikipedia alone. Here, RAG can rely on parametric knowledge to generate reasonable responses.
```

## Pilar: Objective
**Query:** _Tujuan utama dan sasaran spesifik dari artikel ini adalah untuk memberikan kontribusi baru, mengeksplorasi, dan menyelesaikan permasalahan tersebut._

### Rank 1 (Score: 0.8166)
**Header:** **Broader Impact** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
This work offers several positive societal benefits over previous work: the fact that it is more strongly grounded in real factual knowledge (in this case Wikipedia) makes it “hallucinate” less with generations that are more factual, and offers more control and interpretability. RAG could be employed in a wide variety of scenarios with direct benefit to society, for example by endowing it with a medical index and asking it open-domain questions on that topic, or by helping people be more effective at their jobs.  
With these advantages also come potential downsides: Wikipedia, or any potential external knowledge source, will probably never be entirely factual and completely devoid of bias. Since RAG can be employed as a language model, similar concerns as for GPT-2 [50] are valid here, although arguably to a lesser extent, including that it might be used to generate abuse, faked or misleading content in the news or on social media; to impersonate others; or to automate the production of spam/phishing content [54]. Advanced language models may also lead to the automation of various jobs in the coming decades [16]. In order to mitigate these risks, AI systems could be employed to fight against misleading content and automated spam/phishing.
```

### Rank 2 (Score: 0.8139)
**Header:** **1 Introduction** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Margin-
Fact Verification:
alize
Label Generation
The Divine
Comedy
(x)
MIPS
Pe
This 14th century work
is divided into 3
Jeopardy Question
sectlons: "Inferno"
Generation:
"purgatorio" &
Answer Query
"paradiso"
(y)
Question Generation  
**----- Start of picture text -----**<br>
Define�"middle�ea�" (x) The�middle�ea��incl�de�<br>Q�e��i�� A���e�i�g: End-to-End Backprop through  �  and� pθ �he���mpanic�ca�i���and�he��h�ee�o��icle�.�� (y)<br>Q�e��i�� Q�e�� E�c�de�Q�e�� Re��ie�e� �η D�c�me��I�de� Gene�a�o���� Q�e��i�� A���e�i�g:A���e� Ge�e�a�i��<br>Ba�ack�Obama��a� (N��-Pa�a�e���c) (Pa�a�e���c)<br>bo�n�in�Ha�aii. (x) �(�) d(�) z4 ��ppo���� (y)<br>Fac� Ve�i�ca�i��: Fac� Q�e�� z2 z3 Ma�gin-ali�e Labe� Ge�e�a�i��Fac� Ve�i�ca�i��:<br>The�Di�ineComed�� (x) � MIPS z1 pθ Thi��14�h�cen������o�ki��di�ided�in�o�3<br>Je��a�d� Q�e��i�� �ec�ion�:�"Infe�no",<br>Ge�e�a�i��: "P��ga�o�io"�&<br>A���e� Q�e�� "Pa�adi�o"��������� (y)<br>Q�e��i�� Ge�e�a�i��<br>**----- End of picture text -----**<br>  
Figure 1: Overview of our approach. We combine a pre-trained retriever ( _Query Encoder_ + _Document Index_ ) with a pre-trained seq2seq model ( _Generator_ ) and fine-tune end-to-end. For query _x_ , we use Maximum Inner Product Search (MIPS) to find the top-K documents _zi_ . For final prediction _y_ , we treat _z_ as a latent variable and marginalize over seq2seq predictions given different documents.
```

### Rank 3 (Score: 0.8044)
**Header:** **4.5 Additional Results** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Generation Diversity** Section 4.3 shows that RAG models are more factual and specific than BART for Jeopardy question generation. Following recent work on diversity-promoting decoding [33, 59, 39], we also investigate generation diversity by calculating the ratio of distinct ngrams to total ngrams generated by different models. Table 5 shows that RAG-Sequence’s generations are more diverse than RAG-Token’s, and both are significantly more diverse than BART without needing any diversity-promoting decoding.  
**Retrieval Ablations** A key feature of RAG is learning to retrieve relevant information for the task. To assess the effectiveness of the retrieval mechanism, we run ablations where we freeze the retriever during training. As shown in Table 6, learned retrieval improves results for all tasks. We compare RAG’s dense retriever to a word overlap-based BM25 retriever [53]. Here, we replace RAG’s retriever with a fixed BM25 system, and use BM25 retrieval scores as logits when calculating _p_ ( _z|x_ ). Table 6 show the results. For FEVER, BM25 performs best, perhaps since FEVER claims are heavily entity-centric and thus well-suited for word overlap-based retrieval. Differentiable retrieval improves results on all other tasks, especially for Open-Domain QA, where it is crucial.
```

## Pilar: Futurework
**Query:** _Keterbatasan dalam penelitian ini membuka peluang bagi studi selanjutnya. Penelitian di masa depan disarankan untuk memperluas cakupan dan menyelesaikan isu terkait._

### Rank 1 (Score: 0.8126)
**Header:** **Abstract** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, and another which can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
```

### Rank 2 (Score: 0.7871)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Learned Retrieval** There is significant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models [44, 26] similar to ours. Some work optimizes the retrieval module to aid in a specific, downstream task such as question answering, using search [46], reinforcement learning [6, 63, 62], or a latent variable approach [31, 20] as in our work. These successes leverage different retrieval-based architectures and optimization techniques to achieve strong performance on a single task, while we show that a single retrieval-based architecture can be fine-tuned for strong performance on a variety of tasks.  
**Memory-based Architectures** Our document index can be seen as a large external memory for neural networks to attend to, analogous to memory networks [64, 55]. Concurrent work [14] learns to retrieve a trained embedding for each entity in the input, rather than to retrieve raw text as in our work. Other work improves the ability of dialog models to generate factual text by attending over fact embeddings [9, 13] or, closer to our work, over retrieved text directly [15]. A key feature of our memory is that it is comprised of raw text rather distributed representations, which makes the memory both (i) human-readable, lending a form of interpretability to our model, and (ii) human-writable, enabling us to dynamically update the model’s memory by editing the document index.
```

### Rank 3 (Score: 0.7859)
**Header:** **4.1 Open-domain Question Answering** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
Table 1 shows results for RAG along with state-of-the-art models. On all four open-domain QA tasks, RAG sets a new state of the art (only on the T5-comparable split for TQA). RAG combines the generation flexibility of the “closed-book” (parametric only) approaches and the performance of "open-book" retrieval-based approaches. Unlike REALM and T5+SSM, RAG enjoys strong results without expensive, specialized “salient span masking” pre-training [20]. It is worth noting that RAG’s retriever is initialized using DPR’s retriever, which uses retrieval supervision on Natural Questions and TriviaQA. RAG compares favourably to the DPR QA system, which uses a BERT-based “crossencoder” to re-rank documents, along with an extractive reader. RAG demonstrates that neither a re-ranker nor extractive reader is necessary for state-of-the-art performance.  
There are several advantages to generating answers even when it is possible to extract them. Documents with clues about the answer but do not contain the answer verbatim can still contribute towards a correct answer being generated, which is not possible with standard extractive approaches, leading  
Table 1: Open-Domain QA Test Scores. For TQA, left column uses the standard test set for OpenDomain QA, right column uses the TQA-Wiki test set. See Appendix D for further details.
```
