# Retrieval Verification: PaperQA_Retrieval_Augmented_Generative_Agent_for_Scientific_Research_parsed

## Pilar: Background
**Query:** _Penelitian ini dilatarbelakangi oleh urgensi masalah, konteks situasi saat ini, serta motivasi mendasar yang mendorong perlunya dilakukan kajian._

### Rank 1 (Score: 0.8342)
**Header:** 5 EXPERIMENTS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
In this section we measure the performance of PaperQA on LitQA and other datasets, and compare to existing commercial products. We ablate all components of PaperQA on the LitQA dataset, showing their importance. Finally, we investigate the ability to retrieve relevant papers and the rate of hallucinations.
```

### Rank 2 (Score: 0.8259)
**Header:** 7 CONCLUSION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
We introduced PaperQA, a Retrieval-Augmented Generative (RAG) agent that can answer scientific questions better than other LLMs and commercial products. We found PaperQA to be more costefficient than humans, while still retaining its accuracy on par with human researchers. We measured the hallucination rate of citations for recent LLMs to be between 40-60%, whereas we were not able to find a single hallucinated citation in PaperQA’s responses. We also introduced LitQA – a benchmark of 50 questions that require retrieving information from full-text scientific papers. The PaperQA system works independently of the underlying model, with various combinations of Claude-2, GPT-3.5, and GPT-4 providing strong results on LitQA. The most important attributes of PaperQA are its ability to dynamically use RAG tools, retrieve full-text papers, and iterate on the answer through the agent’s decision-making. We hope this open-source implementation of a scientific question-answering system illuminates the design of future RAG agents and tools that reduce hallucinations in LLMs. With such advancements, we believe scientific research will be carried at a fraction of the cost and a multiple of the speed, spurring faster innovation within the natural sciences. By augmenting researchers with a literature review tool, we aim to minimise the time spend searching literature, and rather maximize the amount of productive hours spend carrying out deep thinking for scientific research.
```

### Rank 3 (Score: 0.8200)
**Header:** 5.2 RESULTS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
and Perplexity – in Table 2. All commercial tools are specifically tailored to answering questions by retrieving scientific literature. We give them the same prompt as to PaperQA. From Table 2 we see that PaperQA outperforms all competing models and products, and is on par with that of human experts with access to the internet. Furthermore, we see the lowest rate of incorrectly answered questions out of all tools, which rivals that of humans. This emphasizes that PaperQA is better calibrated to express uncertainty when it actually is uncertain. Surprisingly, GPT-4 and Claude-2 perform better than random although the questions are from papers after their training cut-off date, suggesting they have latent knowledge, leading to useful bias towards answers that are more plausible.
```

## Pilar: Methodology
**Query:** _Tahapan penelitian, arsitektur sistem, desain eksperimen, dan prosedur pengujian dieksekusi menggunakan metodologi teknis berikut._

### Rank 1 (Score: 0.8253)
**Header:** 5.2 RESULTS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
and Perplexity – in Table 2. All commercial tools are specifically tailored to answering questions by retrieving scientific literature. We give them the same prompt as to PaperQA. From Table 2 we see that PaperQA outperforms all competing models and products, and is on par with that of human experts with access to the internet. Furthermore, we see the lowest rate of incorrectly answered questions out of all tools, which rivals that of humans. This emphasizes that PaperQA is better calibrated to express uncertainty when it actually is uncertain. Surprisingly, GPT-4 and Claude-2 perform better than random although the questions are from papers after their training cut-off date, suggesting they have latent knowledge, leading to useful bias towards answers that are more plausible.
```

### Rank 2 (Score: 0.8199)
**Header:** C.1 ABSTRACT RETRIEVAL METRIC / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
A concatenated example of such abstract-question pair (red being the LLM completion) follows:  
Title: DeepDTA: deep drug-target binding affinity prediction Abstract: Motivation: The identification of novel drug-target (DT) interactions is a substantial part of the drug discovery process. [...] One novel approach used in this work is the modeling of protein sequences and compound 1D representations with convolutional neural networks (CNNs). Results: The results show that the proposed deep learning based model that uses the 1D representations of targets and drugs is an effective approach for drug target binding affinity prediction. [...]  
Question: Are there any models that use 1D representations of targets and drugs for drug target binding affinity prediction?  
This dataset is then used to search for the top-10 papers using 20 keywords that were generated using the following prompt:  
We want to answer the following question: question Provide num_keywords unique keyword searches (one search per line) and year ranges that will find papers to help answer the question. Do not use boolean operators. Make sure not to repeat searches without changing the keywords or year ranges. Make some searches broad and some narrow. The current year is get_year(). Use this format: ‘X. [keywords], [start year]-[end year]‘ For example, a list of 3 keywords would be formatted as: 1. ‘keyword1, keyword2, 2020-2021 2. ‘keyword3, keyword4, keyword5, 2020-2021 3. ‘keyword1, keyword2, 2020-2021‘
```

### Rank 3 (Score: 0.8167)
**Header:** REPRODUCIBILITY STATEMENT / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
Discussions on the inherent limitations and challenges faced, along with the mitigations applied to maintain the robustness and reliability of the models, are available in section 6. For comprehensive understanding and replicability of our approach, we recommend a thorough review of the mentioned section, coupled with the supplemental materials provided.
```

## Pilar: Gap
**Query:** _Apa kelemahan, keterbatasan ruang lingkup, atau masalah spesifik pada metode penelitian terdahulu yang belum terselesaikan?_

### Rank 1 (Score: 0.8376)
**Header:** 6 LIMITATIONS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
PaperQA leverages fact, processes and concepts from underlying research papers; transforming that information into human and machine interpretable context. We have an underlying assumption from this that the information in the underlying papers is correct, which may not hold. Although we give some “signals” to the model like the journal name and citation count, these are not faithful indicators of quality.  
Our models and benchmarks are affected by the changing nature of science and the availability of scientific literature: some of the questions in LitQA may have new correct answers or become invalid over time. Users of this benchmark should therefore limit papers used to answer the questions to be cutoff at September 15, 2023.  
While recent work has been conducted on prompt optimization [75, 76], the complex setting of multiple agents with individual prompts is unsolved. Specifically, the task becomes a non-trivial, bi-level optimization problem. Consequently, discerning the impact of manual prompt adjustments becomes difficult. Thus, it is unlikely our prompts are optimal and it is difficult to assess which pieces of the prompts are necessary.
```

### Rank 2 (Score: 0.8337)
**Header:** 5.2 RESULTS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
and Perplexity – in Table 2. All commercial tools are specifically tailored to answering questions by retrieving scientific literature. We give them the same prompt as to PaperQA. From Table 2 we see that PaperQA outperforms all competing models and products, and is on par with that of human experts with access to the internet. Furthermore, we see the lowest rate of incorrectly answered questions out of all tools, which rivals that of humans. This emphasizes that PaperQA is better calibrated to express uncertainty when it actually is uncertain. Surprisingly, GPT-4 and Claude-2 perform better than random although the questions are from papers after their training cut-off date, suggesting they have latent knowledge, leading to useful bias towards answers that are more plausible.
```

### Rank 3 (Score: 0.8289)
**Header:** 2 RELATED WORKS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
continue
Do you know
context
answering
anything
answer
library
LLM
show
propose
to
if sufficient,
about this
answer
agent
give answer
question?  
Figure 1: **PaperQA Workflow Diagram.** PaperQA is an agent that transforms a scientific question into an answer with cited sources. The agent utilizes three tools – search, gather evidence, and answer question. The tools enable it to find and parse relevant full-text research papers, identify specific sections in the paper that help answer the question, summarize those section with the context of the question (called evidence), and then generate an answer based on the evidence. It is an agent, so that the LLM orchestrating the tools can adjust the input to paper searches, gather evidence with different phrases, and assess if an answer is complete.  
**Evaluating LLM Scientists** Assessing the scientific capabilities of LLMs often relies on QA benchmarks, such as general science benchmarks [50, 51], or those specializing in medicine [21], biomedical science [52] or chemistry [53, 54]. In contrast, open-ended tasks, such as conducting chemical synthesis planning [55] or offering healthcare support [56], necessitate manual evaluation to effectively measure an LLM’s capabilities. We discuss the limitation of these datasets and introduce a new QA dataset for retrieval-based science.
```

## Pilar: Objective
**Query:** _Apa tujuan utama, sasaran akhir, atau kontribusi spesifik yang dihasilkan oleh penelitian ini untuk menyelesaikan masalah tersebut?_

### Rank 1 (Score: 0.8298)
**Header:** 5.1 EXPERIMENTAL DETAILS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
All of the experiments were implemented within LangChain’s agent framework [72]. As described above, we use four different LLM instances with the following settings unless stated otherwise: agent LLM: GPT-4 OpenAIFunctions Agent with temperature _τ_ agent = 0 _._ 5; summary LLM: GPT3.5 always with temperature _τ_ sum = 0 _._ 2; answer LLM: GPT-4 with temperature _τ_ ans = 0 _._ 5; ask LLM: GPT-4 with temperature _τ_ ask = 0 _._ 5. For the search engine, we use Google Scholar, where we collect the top-5 papers that are accessible. The implementation details are explained in an experiment in the appendix. We access papers through publicly available APIs, such as Arxiv, PMC, OpenAccess, PubMed, and our own local database of papers. Lastly, we set the number of sources to consider at each round of gather ~~e~~ vidence to 20, and the count of evidence context to include in the final answer within answer ~~q~~ uestion to 8.
```

### Rank 2 (Score: 0.8287)
**Header:** 3.2 TOOLS / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
Write an answer (answer_length) for the question below based on the provided context. If the context provides insufficient information, reply ‘‘I cannot answer’’. For each part of your answer, indicate which sources most support it via valid citation markers at the end of sentences, like (Example2012). Answer in an unbiased, comprehensive, and scholarly tone. If the question is subjective, provide an opinionated answer in the concluding 1-2 sentences.  
context Extra background information: ask LLM Question: question Answer:  
The question in this prompt is the original question fed to the higher-level PaperQA agent, and the context comes from the collection of relevant chunks within the context library. The response from the the tool is returned to the agent, which may reject or accept the answer based as given in the agent’s initialisation prompt given above.
```

### Rank 3 (Score: 0.8279)
**Header:** C.1 ABSTRACT RETRIEVAL METRIC / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
We thus evaluate recall, i.e. finding the original paper. Figure 2 shows the cumulative recall curve, where Google Scholar and Semantic Scholar show outstanding ability to retrieve the original paper. Although CORE API [78] does not perform on par with the other two, we include it here as their main contribution to the field of scientific literature is their standardisation of open-access article repositories.
```

## Pilar: Futurework
**Query:** _Apa saran, rekomendasi penelitian lanjutan, atau agenda di masa depan yang diusulkan oleh penulis untuk mengatasi keterbatasan studi saat ini?_

### Rank 1 (Score: 0.8261)
**Header:** 7 CONCLUSION / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
We introduced PaperQA, a Retrieval-Augmented Generative (RAG) agent that can answer scientific questions better than other LLMs and commercial products. We found PaperQA to be more costefficient than humans, while still retaining its accuracy on par with human researchers. We measured the hallucination rate of citations for recent LLMs to be between 40-60%, whereas we were not able to find a single hallucinated citation in PaperQA’s responses. We also introduced LitQA – a benchmark of 50 questions that require retrieving information from full-text scientific papers. The PaperQA system works independently of the underlying model, with various combinations of Claude-2, GPT-3.5, and GPT-4 providing strong results on LitQA. The most important attributes of PaperQA are its ability to dynamically use RAG tools, retrieve full-text papers, and iterate on the answer through the agent’s decision-making. We hope this open-source implementation of a scientific question-answering system illuminates the design of future RAG agents and tools that reduce hallucinations in LLMs. With such advancements, we believe scientific research will be carried at a fraction of the cost and a multiple of the speed, spurring faster innovation within the natural sciences. By augmenting researchers with a literature review tool, we aim to minimise the time spend searching literature, and rather maximize the amount of productive hours spend carrying out deep thinking for scientific research.
```

### Rank 2 (Score: 0.8204)
**Header:** C.1 ABSTRACT RETRIEVAL METRIC / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
We thus evaluate recall, i.e. finding the original paper. Figure 2 shows the cumulative recall curve, where Google Scholar and Semantic Scholar show outstanding ability to retrieve the original paper. Although CORE API [78] does not perform on par with the other two, we include it here as their main contribution to the field of scientific literature is their standardisation of open-access article repositories.
```

### Rank 3 (Score: 0.8196)
**Header:** C.1 ABSTRACT RETRIEVAL METRIC / PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
**Content:**
```text
As part of the development of PaperQA, we evaluate the efficacy of paper searching using different search API engines. In order to do that, we create a new metric composed of 500 questions. The goal is to create questions that are likely to have only a small number of papers that would include an answer. The exact methodology is:  
1. Search PubMed with 20,000 unique tuples of five keywords within the field of medicine, biology and artificial intelligence.  
2. Only keep the search results with a single paper that is not a review (where a review is assumed to have more than 20 authors).  
3. Instruct GPT-4 to generate a question that could be answered by the abstract only.  
The question generation follows a 5-shot prompting technique, the system prompt being:  
You are a helpful assistant helping completing the following task. The goal is to create a question that could be answered by the paper with these abstracts. Be creative and think of a question that a scientist would ask without knowing the paper he is looking for.  
A concatenated example of such abstract-question pair (red being the LLM completion) follows:
```
