# Retrieval Verification: Retrieval_Augmented_Generation_for_Knowledge_Intensive_NLP_Tasks_parsed

## Pilar: Background
**Query:** _Latar belakang, masalah, dan motivasi penelitian ini. Background, research problem, motivation, context, introduction, underlying issue._

### Rank 1 (Score: 0.3463)
**Header:** **References** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
- [44] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. _arXiv preprint arXiv:1901.04085_ , 2019. URL `https://arxiv.org/abs/1901.04085` .  
- [45] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-4009. URL `https://www.aclweb. org/anthology/N19-4009` .  
- [46] Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&a models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2402–2411, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1244. URL `https://www.aclweb.org/anthology/D19-1244` .
```

### Rank 2 (Score: 0.2894)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Single-Task Retrieval** Prior work has shown that retrieval improves performance across a variety of NLP tasks when considered in isolation. Such tasks include open-domain question answering [5, 29], fact checking [56], fact completion [48], long-form question answering [12], Wikipedia article generation [36], dialogue [41, 65, 9, 13], translation [17], and language modeling [19, 27]. Our work unifies previous successes in incorporating retrieval into individual tasks, showing that a single retrieval-based architecture is capable of achieving strong performance across several tasks.  
**General-Purpose Architectures for NLP** Prior work on general-purpose architectures for NLP tasks has shown great success without the use of retrieval. A single, pre-trained language model has been shown to achieve strong performance on various classification tasks in the GLUE benchmarks [60, 61] after fine-tuning [49, 8]. GPT-2 [50] later showed that a single, left-to-right, pre-trained language model could achieve strong performance across both discriminative and generative tasks. For further improvement, BART [32] and T5 [51, 52] propose a single, pre-trained encoder-decoder model that leverages bi-directional attention to achieve stronger performance on discriminative and generative tasks. Our work aims to expand the space of possible tasks with a single, unified architecture, by learning a retrieval module to augment pre-trained, generative language models.
```

### Rank 3 (Score: 0.2761)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Retrieve-and-Edit approaches** Our method shares some similarities with retrieve-and-edit style approaches, where a similar training input-output pair is retrieved for a given input, and then edited to provide a final output. These approaches have proved successful in a number of domains including Machine Translation [18, 22] and Semantic Parsing [21]. Our approach does have several differences, including less of emphasis on lightly editing a retrieved item, but on aggregating content from several pieces of retrieved content, as well as learning latent retrieval, and retrieving evidence documents rather than related training pairs. This said, RAG techniques may work well in these settings, and could represent promising future work.
```

## Pilar: Methodology
**Query:** _Metode, pendekatan, arsitektur, algoritma, eksperimen, atau dataset yang digunakan. Methodology, proposed method, approach, system architecture, dataset, materials, experimental setup, procedure._

### Rank 1 (Score: 0.3583)
**Header:** **References** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
- [25] Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stackaugmented recurrent nets. In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, page 190–198, Cambridge, MA, USA, 2015. MIT Press. URL `https://papers.nips.cc/paper/ 5857-inferring-algorithmic-patterns-with-stack-augmented-recurrent-nets` .  
- [26] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. _arXiv preprint arXiv:2004.04906_ , 2020. URL `https://arxiv.org/abs/2004.04906` .  
- [27] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In _International Conference on Learning Representations_ , 2020. URL `https://openreview.net/forum?id=HklBjCEKvH` .  
- [28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015. URL `http://arxiv.org/abs/1412.6980` .
```

### Rank 2 (Score: 0.3253)
**Header:** **3 Experiments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
We experiment with RAG in a wide range of knowledge-intensive tasks. For all experiments, we use a single Wikipedia dump for our non-parametric knowledge source. Following Lee et al. [31] and Karpukhin et al. [26], we use the December 2018 dump. Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents. We use the document encoder to compute an embedding for each document, and build a single MIPS index using FAISS [23] with a Hierarchical Navigable Small World approximation for fast retrieval [37]. During training, we retrieve the top _k_ documents for each query. We consider _k 2 {_ 5 _,_ 10 _}_ for training and set _k_ for test time using dev data. We now discuss experimental details for each task.
```

### Rank 3 (Score: 0.3137)
**Header:** **References** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
- [51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv e-prints_ , 2019. URL `https://arxiv.org/abs/1910.10683` .  
- [52] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? _arXiv e-prints_ , 2020. URL `https://arxiv.org/abs/ 2002.08910` .  
- [53] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. _Found. Trends Inf. Retr._ , 3(4):333–389, April 2009. ISSN 1554-0669. doi: 10.1561/ 1500000019. URL `https://doi.org/10.1561/1500000019` .  
- [54] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jian-Bing Wang. Release strategies and the social impacts of language models. _ArXiv_ , abs/1908.09203, 2019.  
- [55] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 2440–2448. Curran Associates, Inc., 2015. URL `http://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf` .
```

## Pilar: Gap
**Query:** _Gap penelitian, keterbatasan, kelemahan, atau masalah yang belum terselesaikan. Research gap, limitations, weakness, unresolved problem, remains a challenge, yet to be explored, however._

### Rank 1 (Score: 0.3354)
**Header:** **Acknowledgments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
The authors would like to thank the reviewers for their thoughtful and constructive feedback on this paper, as well as HuggingFace for their help in open-sourcing code to run RAG models. The authors would also like to thank Kyunghyun Cho and Sewon Min for productive discussions and advice.
```

### Rank 2 (Score: 0.3333)
**Header:** **References** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
- [44] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. _arXiv preprint arXiv:1901.04085_ , 2019. URL `https://arxiv.org/abs/1901.04085` .  
- [45] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-4009. URL `https://www.aclweb. org/anthology/N19-4009` .  
- [46] Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&a models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2402–2411, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1244. URL `https://www.aclweb.org/anthology/D19-1244` .
```

### Rank 3 (Score: 0.3039)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Retrieve-and-Edit approaches** Our method shares some similarities with retrieve-and-edit style approaches, where a similar training input-output pair is retrieved for a given input, and then edited to provide a final output. These approaches have proved successful in a number of domains including Machine Translation [18, 22] and Semantic Parsing [21]. Our approach does have several differences, including less of emphasis on lightly editing a retrieved item, but on aggregating content from several pieces of retrieved content, as well as learning latent retrieval, and retrieving evidence documents rather than related training pairs. This said, RAG techniques may work well in these settings, and could represent promising future work.
```

## Pilar: Objective
**Query:** _Tujuan utama, sasaran, atau pertanyaan penelitian yang ingin dijawab. Main objective, research goal, purpose of this study, aims to investigate, to evaluate, this paper proposes._

### Rank 1 (Score: 0.3638)
**Header:** **Acknowledgments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
The authors would like to thank the reviewers for their thoughtful and constructive feedback on this paper, as well as HuggingFace for their help in open-sourcing code to run RAG models. The authors would also like to thank Kyunghyun Cho and Sewon Min for productive discussions and advice.
```

### Rank 2 (Score: 0.2791)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Single-Task Retrieval** Prior work has shown that retrieval improves performance across a variety of NLP tasks when considered in isolation. Such tasks include open-domain question answering [5, 29], fact checking [56], fact completion [48], long-form question answering [12], Wikipedia article generation [36], dialogue [41, 65, 9, 13], translation [17], and language modeling [19, 27]. Our work unifies previous successes in incorporating retrieval into individual tasks, showing that a single retrieval-based architecture is capable of achieving strong performance across several tasks.  
**General-Purpose Architectures for NLP** Prior work on general-purpose architectures for NLP tasks has shown great success without the use of retrieval. A single, pre-trained language model has been shown to achieve strong performance on various classification tasks in the GLUE benchmarks [60, 61] after fine-tuning [49, 8]. GPT-2 [50] later showed that a single, left-to-right, pre-trained language model could achieve strong performance across both discriminative and generative tasks. For further improvement, BART [32] and T5 [51, 52] propose a single, pre-trained encoder-decoder model that leverages bi-directional attention to achieve stronger performance on discriminative and generative tasks. Our work aims to expand the space of possible tasks with a single, unified architecture, by learning a retrieval module to augment pre-trained, generative language models.
```

### Rank 3 (Score: 0.2700)
**Header:** **3 Experiments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
We experiment with RAG in a wide range of knowledge-intensive tasks. For all experiments, we use a single Wikipedia dump for our non-parametric knowledge source. Following Lee et al. [31] and Karpukhin et al. [26], we use the December 2018 dump. Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents. We use the document encoder to compute an embedding for each document, and build a single MIPS index using FAISS [23] with a Hierarchical Navigable Small World approximation for fast retrieval [37]. During training, we retrieve the top _k_ documents for each query. We consider _k 2 {_ 5 _,_ 10 _}_ for training and set _k_ for test time using dev data. We now discuss experimental details for each task.
```

## Pilar: Futurework
**Query:** _Pekerjaan di masa depan, saran untuk peneliti lain, atau langkah selanjutnya. Future work, recommendations, future research directions, next steps, further investigation._

### Rank 1 (Score: 0.3935)
**Header:** **5 Related Work** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
**Retrieve-and-Edit approaches** Our method shares some similarities with retrieve-and-edit style approaches, where a similar training input-output pair is retrieved for a given input, and then edited to provide a final output. These approaches have proved successful in a number of domains including Machine Translation [18, 22] and Semantic Parsing [21]. Our approach does have several differences, including less of emphasis on lightly editing a retrieved item, but on aggregating content from several pieces of retrieved content, as well as learning latent retrieval, and retrieving evidence documents rather than related training pairs. This said, RAG techniques may work well in these settings, and could represent promising future work.
```

### Rank 2 (Score: 0.3626)
**Header:** **Acknowledgments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
The authors would like to thank the reviewers for their thoughtful and constructive feedback on this paper, as well as HuggingFace for their help in open-sourcing code to run RAG models. The authors would also like to thank Kyunghyun Cho and Sewon Min for productive discussions and advice.
```

### Rank 3 (Score: 0.2584)
**Header:** **3 Experiments** / **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**
**Content:**
```text
We experiment with RAG in a wide range of knowledge-intensive tasks. For all experiments, we use a single Wikipedia dump for our non-parametric knowledge source. Following Lee et al. [31] and Karpukhin et al. [26], we use the December 2018 dump. Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents. We use the document encoder to compute an embedding for each document, and build a single MIPS index using FAISS [23] with a Hierarchical Navigable Small World approximation for fast retrieval [37]. During training, we retrieve the top _k_ documents for each query. We consider _k 2 {_ 5 _,_ 10 _}_ for training and set _k_ for test time using dev data. We now discuss experimental details for each task.
```
