# Retrieval Verification: PaperQA_Retrieval_Augmented_Generative_Agent_for_Scientific_Research_parsed

## Pilar: Background
**Query:** _Latar belakang, masalah, dan motivasi penelitian ini. Background, research problem, motivation, context, introduction, underlying issue._

### Rank 1 (Score: 0.3132)
**Header:** G.2 ABSENCE OF KEY INFORMATION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
As a second example, we ask PaperQA: “What temperature does water turn to plasma?” and provide it with context that is relevant but insufficient to answer the question. Then, we provide the model with the following background information:  
“Water turns into plasma under extreme conditions of temperature and pressure. Plasma is the fourth state of matter, beyond solid, liquid, and gas. It’s a super-heated state where atoms are ionized, meaning electrons are stripped from atoms. This typically occurs at temperatures exceeding 20,000 Kelvin.”  
In this example, the model answers:  
Water transitions into the plasma state under extreme conditions of temperature and pressure, typically at temperatures exceeding 20,000 Kelvin.  
Excluding the background information results in PaperQA responding that it is unable to answer.  
Thus, the role of parametric knowledge is crucial in shaping the model’s responses: when it contradicts the context, the model may refrain from answering due to the conflict, whereas supportive parametric knowledge enables the model to provide detailed and informed responses when context is lacking.
```

### Rank 2 (Score: 0.2691)
**Header:** G IMPACT OF PARAMETRIC KNOWLEDGE / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
In order to test whether including parametric knowledge in the gathered context is useful, we prompt PaperQA:  
Write an answer (about 200 words, but can be longer if necessary) for the question below based on the provided context. If the context provides insufficient information and the question cannot be directly answered, reply ‘‘I cannot answer’’. For each part of your answer, indicate which sources most support it via valid citation markers at the end of sentences, like (Example2012).  
Followed by pre-gathered context with relevance scores, some extra background information, and the question. The extra background information represents the LLMs’ parametric knowledge.
```

### Rank 3 (Score: 0.2585)
**Header:** 1 INTRODUCTION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
The rate of papers published yearly grows at an exponential rate, with over 5 million academic articles published in 2022 [1], and over 200 million articles in total [2]. The difficulty of navigating this extensive literature means significant scientific findings have gone unnoticed for extended periods [3, 4, 5]. Work in the last 10 years has sought to make the space of literature more manageable for scientists, with the introduction of keyword search systems [6, 7], vector similarity embeddings [8, 9, 10, 11] and recommender systems [12, 13]. The process of scientific discovery from literature is still, however, highly manual.  
The use of Large Language Models (LLMs) to answer scientific questions is increasingly seen in academia and research-heavy professions such as medicine [14, 15]. While LLMs can produce answers faster and encompass a broader, deeper scope than manual searching, there is a high risk of hallucination in responses, which can lead to potentially dangerous outcomes [16, 17]. Incorrect information can be more damaging than no information at all, as the time to verify veracity can take just as long as retrieving it from research papers in the first place [18]. Reliance on pre-trained LLMs also prevents the discovery of new information published after a training cutoff date. Given the rapidly moving pace of science, this can lead to misconceptions persisting.
```

## Pilar: Methodology
**Query:** _Metode, pendekatan, arsitektur, algoritma, eksperimen, atau dataset yang digunakan. Methodology, proposed method, approach, system architecture, dataset, materials, experimental setup, procedure._

### Rank 1 (Score: 0.4173)
**Header:** 5 EXPERIMENTS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
In this section we measure the performance of PaperQA on LitQA and other datasets, and compare to existing commercial products. We ablate all components of PaperQA on the LitQA dataset, showing their importance. Finally, we investigate the ability to retrieve relevant papers and the rate of hallucinations.
```

### Rank 2 (Score: 0.3810)
**Header:** REFERENCES / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
- [9] Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP)_ , Hong Kong, China, November 2019. Association for Computational Linguistics. URL https://aclanthology.org/D19-1371.  
- [10] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017.  
- [11] Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. Scirepeval: A multi-format benchmark for scientific document representations. _ArXiv_ , abs/2211.13308, 2022.  
- [12] Paul Resnick and Hal R Varian. Recommender systems. _Communications of the ACM_ , 40(3): 56–58, 1997.  
- [13] Gediminas Adomavicius and Alexander Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. _IEEE transactions on knowledge and data engineering_ , 17(6):734–749, 2005.  
- [14] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. _Nature_ , pages 1–9, 2023.
```

### Rank 3 (Score: 0.3602)
**Header:** 4 THE LITQA DATASET / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
Table 1: **LitQA Dataset Examples.** Three example questions from the dataset with correct answers bolded. The difficulty annotated comes from our observations in experiments given in Section 5.  
|ID|Question|Answers|Diffculty|
|---|---|---|---|
|7|Has anyone performed a base editing screen against splice sites|**Yes**|Easy|
||in CD33 before?|No||
|32|How diffuse are the laminar patterns of the axonal terminations|**More diffuse**|Medium|
||of lower Layer 5/Layer 6 intratelencephalic neurons compared|About the same||
||to Layer 2-4 intratelencephalic neurons in mouse cortex?|Less diffuse||
|21|Which of these glycoRNAs does NOT show an increase in M0|**U8**|Hard|
||macrophages upon stimulation with LPS: U1, U35a, Y5 or U8?|U1||
|||U35a||
|||Y5||  
cutoff and then devising a question from that paper that both refers to a novel finding in the paper and that is not presented in the paper abstract. We do not expect these findings to be presented in other papers or to be available in sources preceding the cutoff. For every question, distractor answers were also created, either by the question writer alone, or by using an LLM to provide a plausible answer to the given question. We take special care to only collect questions from papers published _after_ the GPT-3.5/4 cutoff date in September 2021. This ensures the questions cannot be reliably answered using the latent knowledge of GPT-4. Each question was then independently reviewed by at least one other co-author.
```

## Pilar: Gap
**Query:** _Gap penelitian, keterbatasan, kelemahan, atau masalah yang belum terselesaikan. Research gap, limitations, weakness, unresolved problem, remains a challenge, yet to be explored, however._

### Rank 1 (Score: 0.3917)
**Header:** 1 INTRODUCTION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
The rate of papers published yearly grows at an exponential rate, with over 5 million academic articles published in 2022 [1], and over 200 million articles in total [2]. The difficulty of navigating this extensive literature means significant scientific findings have gone unnoticed for extended periods [3, 4, 5]. Work in the last 10 years has sought to make the space of literature more manageable for scientists, with the introduction of keyword search systems [6, 7], vector similarity embeddings [8, 9, 10, 11] and recommender systems [12, 13]. The process of scientific discovery from literature is still, however, highly manual.  
The use of Large Language Models (LLMs) to answer scientific questions is increasingly seen in academia and research-heavy professions such as medicine [14, 15]. While LLMs can produce answers faster and encompass a broader, deeper scope than manual searching, there is a high risk of hallucination in responses, which can lead to potentially dangerous outcomes [16, 17]. Incorrect information can be more damaging than no information at all, as the time to verify veracity can take just as long as retrieving it from research papers in the first place [18]. Reliance on pre-trained LLMs also prevents the discovery of new information published after a training cutoff date. Given the rapidly moving pace of science, this can lead to misconceptions persisting.
```

### Rank 2 (Score: 0.3843)
**Header:** 7 CONCLUSION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
We introduced PaperQA, a Retrieval-Augmented Generative (RAG) agent that can answer scientific questions better than other LLMs and commercial products. We found PaperQA to be more costefficient than humans, while still retaining its accuracy on par with human researchers. We measured the hallucination rate of citations for recent LLMs to be between 40-60%, whereas we were not able to find a single hallucinated citation in PaperQA’s responses. We also introduced LitQA – a benchmark of 50 questions that require retrieving information from full-text scientific papers. The PaperQA system works independently of the underlying model, with various combinations of Claude-2, GPT-3.5, and GPT-4 providing strong results on LitQA. The most important attributes of PaperQA are its ability to dynamically use RAG tools, retrieve full-text papers, and iterate on the answer through the agent’s decision-making. We hope this open-source implementation of a scientific question-answering system illuminates the design of future RAG agents and tools that reduce hallucinations in LLMs. With such advancements, we believe scientific research will be carried at a fraction of the cost and a multiple of the speed, spurring faster innovation within the natural sciences. By augmenting researchers with a literature review tool, we aim to minimise the time spend searching literature, and rather maximize the amount of productive hours spend carrying out deep thinking for scientific research.
```

### Rank 3 (Score: 0.3785)
**Header:** REFERENCES / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
- [1] Dimitrije Curcic. Dimitrije curcic, Jun 2023. URL https://wordsrated.com/number-ofacademic-papers-published-per-year/#: _[∼]_ :text=As%20of%202022%2C%20over%205.14,5. 03%20million%20papers%20were%20published.  
- [2] Michael Fire and Carlos Guestrin. Over-optimization of academic publishing metrics: observing goodhart’s law in action. _GigaScience_ , 8(6):giz053, 2019.  
- [3] Santo Fortunato, Carl T Bergstrom, Katy B¨orner, James A Evans, Dirk Helbing, Staˇsa Milojevi´c, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. Science of science. _Science_ , 359(6379):eaao0185, 2018.  
- [4] Gunther S Stent. Prematurity and uniqueness in scientific discovery. _Scientific American_ , 227 (6):84–93, 1972.  
- [5] Eugene Garfield. Premature discovery or delayed recognition-why. _Current contents_ , (21): 5–10, 1980.  
- [6] Google Scholar. Google scholar. URL https://scholar.google.com/.  
- [7] Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, et al. The semantic scholar open data platform. _arXiv preprint arXiv:2301.10140_ , 2023.  
- [8] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document-level representation learning using citation-informed transformers. _arXiv preprint arXiv:2004.07180_ , 2020.
```

## Pilar: Objective
**Query:** _Tujuan utama, sasaran, atau pertanyaan penelitian yang ingin dijawab. Main objective, research goal, purpose of this study, aims to investigate, to evaluate, this paper proposes._

### Rank 1 (Score: 0.3037)
**Header:** **Andrew D White** / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
Future House University of Rochester andrew@futurehouse.org  
Dec 8, 2023 ABSTRACT  
Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question-answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA’s matches expert human researchers on LitQA.
```

### Rank 2 (Score: 0.2958)
**Header:** G IMPACT OF PARAMETRIC KNOWLEDGE / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
In order to test whether including parametric knowledge in the gathered context is useful, we prompt PaperQA:  
Write an answer (about 200 words, but can be longer if necessary) for the question below based on the provided context. If the context provides insufficient information and the question cannot be directly answered, reply ‘‘I cannot answer’’. For each part of your answer, indicate which sources most support it via valid citation markers at the end of sentences, like (Example2012).  
Followed by pre-gathered context with relevance scores, some extra background information, and the question. The extra background information represents the LLMs’ parametric knowledge.
```

### Rank 3 (Score: 0.2923)
**Header:** G.2 ABSENCE OF KEY INFORMATION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
As a second example, we ask PaperQA: “What temperature does water turn to plasma?” and provide it with context that is relevant but insufficient to answer the question. Then, we provide the model with the following background information:  
“Water turns into plasma under extreme conditions of temperature and pressure. Plasma is the fourth state of matter, beyond solid, liquid, and gas. It’s a super-heated state where atoms are ionized, meaning electrons are stripped from atoms. This typically occurs at temperatures exceeding 20,000 Kelvin.”  
In this example, the model answers:  
Water transitions into the plasma state under extreme conditions of temperature and pressure, typically at temperatures exceeding 20,000 Kelvin.  
Excluding the background information results in PaperQA responding that it is unable to answer.  
Thus, the role of parametric knowledge is crucial in shaping the model’s responses: when it contradicts the context, the model may refrain from answering due to the conflict, whereas supportive parametric knowledge enables the model to provide detailed and informed responses when context is lacking.
```

## Pilar: Futurework
**Query:** _Pekerjaan di masa depan, saran untuk peneliti lain, atau langkah selanjutnya. Future work, recommendations, future research directions, next steps, further investigation._

### Rank 1 (Score: 0.2755)
**Header:** REFERENCES / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
- [54] Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: A benchmark for molecular machine learning, 2017. URL https://arxiv.org/abs/1703.00564.  
- [55] Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. _arXiv preprint arXiv:2304.05376_ , 2023.  
- [56] Debadutta Dash, Rahul Thapa, Juan M Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H Chen, Saurabh Gombar, Lance Downing, et al. Evaluation of gpt-3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. _arXiv preprint arXiv:2304.13714_ , 2023.  
- [57] D. Soong, S. Sridhar, Han Si, J. Wagner, Ana Caroline Costa S’a, Christina Y. Yu, Kubra Karagoz, Meijian Guan, Hisham K Hamadeh, and Brandon Higgs. Improving accuracy of gpt-3/4 results on biomedical data using a retrieval-augmented language model. _ArXiv_ , abs/2305.17116, 2023.  
- [58] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. _arXiv preprint arXiv:2212.14024_ , 2022.
```

### Rank 2 (Score: 0.2753)
**Header:** PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
**Jakub L´ala Odhran O’Donoghue Aleksandar Shtedritski** Future House Align to Innovate Align to Innovate Francis Crick Institute Francis Crick Institute Francis Crick Institute University of Oxford University of Oxford  
**Sam Cox Samuel G Rodriques** Future House Future House University of Rochester Francis Crick Institute
```

### Rank 3 (Score: 0.2491)
**Header:** 4 THE LITQA DATASET / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
**Human performance** We recruited five biomedical researchers with an undergraduate degree or higher to solve LitQA. They were given access to the internet and given three minutes per question (2.5 hours in total) to answer all questions, specifying the paper they used to choose the corresponding answer. Additionally, they were instructed to answer _unsure_ if they cannot find the answer to a given question.
```
