# Retrieval Verification: SciLit_Joint_Literature_Discovery_Summarization_and_Citation_Generation_parsed

## Pilar: Background
**Query:** _Latar belakang, masalah, dan motivasi penelitian ini. Background, research problem, motivation, context, introduction, underlying issue._

### Rank 1 (Score: 0.2462)
**Header:** **2.3 Citation Generation Module**
**Content:**
```text
The citation generation module acts as an abstract summarizer that takes as input the context, the keywords, and the target paper to be cited; it then generates a sentence that cites the target paper and narrates it in context.  
> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0004-00.png]**:
User
frontend
Interface
response
request
gateway
API
Gateway
context, eywords and iteD
query
citation sentences
reranked IDs
highlights
Literature
Extractive
Citation
Discovery
Summarization
Content
query & prefetched IDs
Generation
reranked IDs
fultext
backend
prefetched IDs
abstract
Prefetch &re-
Rerank
IDs
早
Database
move duplicates
titles & abstracts
Interface
query
IDs
contents
IDS
titles & authors
Index
Index
Index
DB
DB
DB
(S20RC)
(PMCOA)
(arXiv)
(S2ORC)
(PMCOA)
(arXiv)  
**----- Start of picture text -----**<br>
```

### Rank 2 (Score: 0.2316)
**Header:** **`Quadricuspid Aortic Valve: A Comprehensive Review`**
**Content:**
```text
```
Generated Citation:  The clinicalmanifestations of QAVdepend on the functionalstatus of the QAV and the associate
d disorders [ #CIT ].
```  
```
Real Citation:  Clinical manifestations, such as palpitations, dyspnea, fatigue, and chest pain, depend on the funct
ional status of QAV and usually present in the fifth or sixth decade of life [#CIT].
```
```

### Rank 3 (Score: 0.2289)
**Header:** **3.1 Demonstration**
**Content:**
```text
modifying the pretraining tasks.
surprisingly good results even when only an order of a hundred examples are available.
For example, recurring span selection #CIT masks all recurring spans but one, and asks the model to select the correct span in
the passage for each masked span.
VIEW FULLTEXT
s
M Joshi, D Chen, Y Liu, et al. Transactions of the Association for Computational Linguistics, 2019
国
Keywords
uo suje isiel u ym 1a soednno Auasisuoo jeu yooidde buujeaid jenl-ueds e aonpoul am saded su u
span selection
span selection tasks such as question answering and coreference resolution.
lixa1 jo sueds topeud pue juesaudeu jeeq oi peubisep s teu pouew bujujen-aid e 1t3gueds jueseud a *
SEARCH Q
FINETUNEGENERATION
• Our method differs from BERT in both the masking scheme and the training objectives.
• Second, we introduce a novel span-boundary objective (SBO) to train the model to predict the entire masked span from the
Selected Citation Text
Span-BERT #CIT masks contiguous random
observed tokens at its boundary.
spans instead of random tokens, and trains
eedde Aau yom ui xauo eu bujsn Aielos sueds euue sopaud on lepow eu seoy buyse pesequeds •
the span boundary representation to predict
jeue asueds wopue, snonbguo upyseu (t) Aq Ltag spuejxe yoqm Sujujeqeud peseq-ueds so, pouew mau e pejuasaud a/ •
the entire content of the masked span.
than random tokens, and (2) training the span boundary representations to predict the entire content of ..
```

## Pilar: Methodology
**Query:** _Metode, pendekatan, arsitektur, algoritma, eksperimen, atau dataset yang digunakan. Methodology, proposed method, approach, system architecture, dataset, materials, experimental setup, procedure._

### Rank 1 (Score: 0.3732)
**Header:** **2.1 Literature Discovery**
**Content:**
```text
The literature discovery module takes as inputs the context and keywords and recommends papers that are worth citing in the current context. To strike a balance between query accuracy and speed on large scientific corpora, our document discovery module employs a two-stage prefetching-ranking strategy (Gu et al., 2022b) (Figure 2). For each scientific corpus, we build a database and create an efficient prefetching model that we use to pre-filter _Np_ (see the discussion of _Np_ in Table 2 and Section 3.2) candidate documents based on the provided keywords and context. After removing duplicates, we re-rank the prefetched documents from each database to produce the final order.  
**Databases.** We dump each corpus into a separate SQLite (Hipp, 2000) database to allow flexibility in deploying and updating of independent prefetching servers. We further process documents from different corpora into a unified JSON schema so that we can use the same codebase to index, query, summarize, and display documents from different corpora. The JSON schema includes “Title”, “Author”, etc., for metadata, and “Content.Abstract_Parsed”, “Content.Fullbody_Parsed” for parsed full text, The details are given in Appendix B.  
**Prefetching.** The prefetching model of a given SQLite database consists of an inverted index and an embedding index. The inverted index stores the paper IDs of all publications that contain a given
```

### Rank 2 (Score: 0.3720)
**Header:** **1 Introduction**
**Content:**
```text
tuned SciBERT (Beltagy et al., 2019). Such an architecture allows us to dynamically add or remove databases and update one database and its index without significantly affecting the others. From a user-chosen document (i.e., target paper), we extract highlights using a light-weight extractive summarization model proposed in Gu et al. (2022a). Furthermore, using a fine-tuned T5 model (Raffel et al., 2020), we generate a citing sentence based on the abstract of the target paper, the context (the text surrounding the original citation sentence), and the keywords provided by users. We also develop a microservice-based architecture that allows easy updating of algorithms.  
In summary, our main contributions are:  
- We demonstrate SCILIT, a platform for searching, summarizing, and citing scientific papers.  
- We evaluate SCILIT on scientific literature retrieval, paper summarization, and contextaware citation sentence generation, and showcase the generation of a related-work paragraph.  
- A live demo website of our system is at https://scilit.vercel.app and our implementation and data are at https://github.com/nianlonggu/ SciLit and a video demonstrating the system can be viewed at https://youtu.be/PKvNaY5Og1Y
```

### Rank 3 (Score: 0.3463)
**Header:** **3.2 Performance**
**Content:**
```text
**Evaluation Dataset.** We evaluated SCILIT on a test set containing 1530 samples, mainly from papers published in 2022 in the fields of computer science and biomedical science. Each sample contains the following information: 1) context, up to 6 sentences preceding the citation sentence and within the same section; 2) keywords, up to 2 unior bi-grams that occur in all of the context, the citation sentence, and the cited paper; 3) ID of the cited paper; 4) the citation sentence following the context, which is the ground truth for evaluating generated citations. For quality control, we only include citation sentences in the test set that cite one paper.  
**Paper Retrieval.** For each sample in the evaluation dataset, we use context and keywords as queries and invoke the literature search service to  
||Model<br>BertSum (Liu,2019)|Rouge-1<br>42.53|Rouge-2<br>16.89|Rouge-L<br>39.18|
|---|---|---|---|---|
||MemSum (Gu et al.,2022a)|**46.40***|**19.61***|**42.66***|  
Table 3: The extractive summarization performance. “*" indicates statistical significance in comparison to baselines with a 95% bootstrap confidence interval.
```

## Pilar: Gap
**Query:** _Gap penelitian, keterbatasan, kelemahan, atau masalah yang belum terselesaikan. Research gap, limitations, weakness, unresolved problem, remains a challenge, yet to be explored, however._

### Rank 1 (Score: 0.3286)
**Header:** **2.1 Literature Discovery**
**Content:**
```text
|**Corpus**|**Databases**<br># of<br>papers<br># papers with<br>fullbody<br>until<br>date|**Inverted Index**<br>keywords<br>length<br># of<br>keywords<br>data<br>format<br>storage<br>size|**Embedding Index**<br>embedding<br>dimension<br>storage<br>size|
|---|---|---|---|
|S2ORC<br>PMCOA<br>arXiv|136.60 M<br>12.44 M<br>2020-04-14<br>2.89 M<br>2.73 M<br>2022-06-17<br>1.69 M<br>1.69 M<br>2022-07-28|unigram, bigram<br>1.20 B<br>sqlitedict<br>769 GB<br>0.30 B<br>145 GB<br>0.15 B<br>77 GB|256<br>169 GB<br>2.9 GB<br>1.7 GB|  
Table 1: Statistics of our literature discovery system. We indexed S2ORC (Lo et al., 2020), PMCOA (of Medicine, 2003), and arXiv (Kaggle, 2022), which contain large numbers of recent scientific papers in diverse fields.  
keyword, such as a unigram like “computer” or a bigram like “machine learning”, where the paper ID is a unique identifier using which we can retrieve the paper’s content from the database. The embedding index is formed by the embeddings of all papers in the database. Embeddings are 256-dimensional vectors computed by Sent2Vec (Pagliardini et al., 2018) (we simply average the embeddings of all words in a document). We train Sent2Vec using sentences obtained from the full text of the papers contained in S2ORC.
```

### Rank 2 (Score: 0.2923)
**Header:** **1 Introduction**
**Content:**
```text
When we compose sentences like “Our experiments show that XXX performs significantly worse than YYY” in a manuscript, we may want to find papers that report similar performance evaluations (Cohan et al., 2019) and discuss these in our manuscript. This process is a non-trivial task requiring in-depth human involvement in finding, summarizing, and citing papers, which raises the question whether it is possible to partly automate this process to reduce users’ cognitive load in searching, retrieving, reading, and rephrasing related findings.  
Recent advances in natural language processing (NLP) help answer this question. First, releases of large scientific corpora such as S2ORC (Lo et al., 2020) and General Index (Else, 2021) provide opportunities for building large databases of scientific  
> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0001-10.png]**:
ContextText
candidate
Highlights
we observe that
papers
In this paper, we show
the pooling strategy
Extractive
how universal sentence
has a large impact
Summarization
Our experiments show .
There, the MAX
LSTM architecture with
strategy perform
max pooling, trained on
significantly worse
Literature
the SNLI dataset, yields
than MEAN or
state-of-the-art...
CLS-token strategy.
Retrieval
Citation
Citation Sentence
Keywords
Generation
This is in contrast to [CIT]
who found it beneficial for
MAX pooling;
the BiLSTM-layer of
Pooling strategy
InferSent to use MAX
```

### Rank 3 (Score: 0.2887)
**Header:** **B JSON Schema for Database**
**Content:**
```text
- Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020a. Automatic Generation of Citation Texts in Scholarly  
The details of our unified JSON schema is shown in Listing 1. As metadata we define “Author”, “Title”, “Abstract”, “Venue”, “DOI”, “URL”, and as parsed full text we define “PublicationDate”, "Content.Abstract_Parsed", and “Content.Fullbody_Parsed”. The parsed abstract or full body contains a list of parsed sections. Each section contains a list of parsed paragraphs, each including a list of parsed sentences. If a sentence cites a paper, we create a “cite span” that records the citation marker such as “[1]”, the position of the citation marker in the sentence, and the cited paper’s index in the “Reference” list.  
We implemented a S2ORC parser to convert documents in the S2ORC corpus to our JSON format. For PDFs in the arXiv corpus, we first used the s2orc-doc2json (Lo et al., 2020) to convert them into S2ORC format and then we applied our S2ORC parser. For XML files in the PMCOA corpus, we implemented an XML parser  
2https://vercel.com/  
> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0009-00.png]**:
NLP
machinetranslation
AND
OR
NMT
Pub.Year:2020
OR
Pub.Year:2021
Pub.Year:2022  
**----- Start of picture text -----**<br>
NLP<br>machine translation<br>AND OR<br>NMT<br>Pub. Year: 2020<br>OR<br>Pub. Year: 2021<br>Pub. Year: 2022<br>**----- End of picture text -----**<br>
```

## Pilar: Objective
**Query:** _Tujuan utama, sasaran, atau pertanyaan penelitian yang ingin dijawab. Main objective, research goal, purpose of this study, aims to investigate, to evaluate, this paper proposes._

### Rank 1 (Score: 0.2805)
**Header:** **3.1 Demonstration**
**Content:**
```text
the entire content of the masked span.
than random tokens, and (2) training the span boundary representations to predict the entire content of ..
yoan pue sysl jo Aleen e uo sauljesen 1iio ie woedino jey sjepow spielA ssaooid buure-aid ano dseujeoo •
substantially better performance on span selection tasks in particular.
S66
Span-BERT #CIT masks contiguous random spans instead of random tokens, and trains the span boundary representation to
predict the entire content of the masked span.
VIEW FULLTEXT
EXPORT CITATION
50  
Figure 4: Overview of the user interface. The context text comes from the related work section in Glass et al. (2020).  
|_Np_<br>time<br>(_s_/query)|Recall@_K_ (R@_K_)<br>R@1<br>R@5<br>R@10<br>R@20<br>R@50<br>R@100|
|---|---|
|50<br>**2.02**<br>100<br>2.55<br>200<br>3.26<br>300<br>3.93|**0.107**<br>0.208<br>0.263<br>0.305<br>0.327<br>0.331<br>0.096<br>0.215<br>**0.278**<br>0.328<br>0.384<br>0.401<br>0.095<br>**0.220**<br>0.275<br>**0.339**<br>0.420<br>0.452<br>0.095<br>0.204<br>0.273<br>0.330<br>**0.422**<br>**0.482**|  
Table 2: Paper retrieval performance measured by the recall of the top _K_ recommendations. _Np_ denotes the number of prefetched candidates per corpus.
```

### Rank 2 (Score: 0.2715)
**Header:** **3.2 Performance**
**Content:**
```text
**Evaluation Dataset.** We evaluated SCILIT on a test set containing 1530 samples, mainly from papers published in 2022 in the fields of computer science and biomedical science. Each sample contains the following information: 1) context, up to 6 sentences preceding the citation sentence and within the same section; 2) keywords, up to 2 unior bi-grams that occur in all of the context, the citation sentence, and the cited paper; 3) ID of the cited paper; 4) the citation sentence following the context, which is the ground truth for evaluating generated citations. For quality control, we only include citation sentences in the test set that cite one paper.  
**Paper Retrieval.** For each sample in the evaluation dataset, we use context and keywords as queries and invoke the literature search service to  
||Model<br>BertSum (Liu,2019)|Rouge-1<br>42.53|Rouge-2<br>16.89|Rouge-L<br>39.18|
|---|---|---|---|---|
||MemSum (Gu et al.,2022a)|**46.40***|**19.61***|**42.66***|  
Table 3: The extractive summarization performance. “*" indicates statistical significance in comparison to baselines with a 95% bootstrap confidence interval.
```

### Rank 3 (Score: 0.2544)
**Header:** **3.3 Ablation Study**
**Content:**
```text
To analyze the impact of keywords, we evaluated retrieval and generation systems without keywords. For document retrieval, we first prefetch _Np_ = 100 candidates from each corpus and then rank them based on context only. For citation generation, we trained a T5 model to learn to generate citation sentences with only the context and the title and abstract of the cited paper and evaluated it on the  
Recently, AI-driven platforms focused on literature recommendation and scientific paper summarization have been proposed. (keywords: platform, paper: #2) _One such platform is AI Research Navigator (Fadaee et al., 2020), which combines classical keyword search with neural retrieval to discover and organize relevant literature._ (keywords: scientific; summarization; platform, paper #3) _Another platform is Anne O’Tate, which supports user-driven summarization, drill-down and mining of search results from PubMed, the leading search engine for biomedical literature (Smalheiser et al., 2021)._ (keywords: related work generation, paper #9) _Chen and Zhuge (2019) automatically generates related work by comparing the main text of the paper being written with the citations of other papers that cite the same references._
```

## Pilar: Futurework
**Query:** _Pekerjaan di masa depan, saran untuk peneliti lain, atau langkah selanjutnya. Future work, recommendations, future research directions, next steps, further investigation._

### Rank 1 (Score: 0.2804)
**Header:** **3.1 Demonstration**
**Content:**
```text
the entire content of the masked span.
than random tokens, and (2) training the span boundary representations to predict the entire content of ..
yoan pue sysl jo Aleen e uo sauljesen 1iio ie woedino jey sjepow spielA ssaooid buure-aid ano dseujeoo •
substantially better performance on span selection tasks in particular.
S66
Span-BERT #CIT masks contiguous random spans instead of random tokens, and trains the span boundary representation to
predict the entire content of the masked span.
VIEW FULLTEXT
EXPORT CITATION
50  
Figure 4: Overview of the user interface. The context text comes from the related work section in Glass et al. (2020).  
|_Np_<br>time<br>(_s_/query)|Recall@_K_ (R@_K_)<br>R@1<br>R@5<br>R@10<br>R@20<br>R@50<br>R@100|
|---|---|
|50<br>**2.02**<br>100<br>2.55<br>200<br>3.26<br>300<br>3.93|**0.107**<br>0.208<br>0.263<br>0.305<br>0.327<br>0.331<br>0.096<br>0.215<br>**0.278**<br>0.328<br>0.384<br>0.401<br>0.095<br>**0.220**<br>0.275<br>**0.339**<br>0.420<br>0.452<br>0.095<br>0.204<br>0.273<br>0.330<br>**0.422**<br>**0.482**|  
Table 2: Paper retrieval performance measured by the recall of the top _K_ recommendations. _Np_ denotes the number of prefetched candidates per corpus.
```

### Rank 2 (Score: 0.2739)
**Header:** **3.1 Demonstration**
**Content:**
```text
modifying the pretraining tasks.
surprisingly good results even when only an order of a hundred examples are available.
For example, recurring span selection #CIT masks all recurring spans but one, and asks the model to select the correct span in
the passage for each masked span.
VIEW FULLTEXT
s
M Joshi, D Chen, Y Liu, et al. Transactions of the Association for Computational Linguistics, 2019
国
Keywords
uo suje isiel u ym 1a soednno Auasisuoo jeu yooidde buujeaid jenl-ueds e aonpoul am saded su u
span selection
span selection tasks such as question answering and coreference resolution.
lixa1 jo sueds topeud pue juesaudeu jeeq oi peubisep s teu pouew bujujen-aid e 1t3gueds jueseud a *
SEARCH Q
FINETUNEGENERATION
• Our method differs from BERT in both the masking scheme and the training objectives.
• Second, we introduce a novel span-boundary objective (SBO) to train the model to predict the entire masked span from the
Selected Citation Text
Span-BERT #CIT masks contiguous random
observed tokens at its boundary.
spans instead of random tokens, and trains
eedde Aau yom ui xauo eu bujsn Aielos sueds euue sopaud on lepow eu seoy buyse pesequeds •
the span boundary representation to predict
jeue asueds wopue, snonbguo upyseu (t) Aq Ltag spuejxe yoqm Sujujeqeud peseq-ueds so, pouew mau e pejuasaud a/ •
the entire content of the masked span.
than random tokens, and (2) training the span boundary representations to predict the entire content of ..
```

### Rank 3 (Score: 0.2534)
**Header:** **B JSON Schema for Database**
**Content:**
```text
- Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020a. Automatic Generation of Citation Texts in Scholarly  
The details of our unified JSON schema is shown in Listing 1. As metadata we define “Author”, “Title”, “Abstract”, “Venue”, “DOI”, “URL”, and as parsed full text we define “PublicationDate”, "Content.Abstract_Parsed", and “Content.Fullbody_Parsed”. The parsed abstract or full body contains a list of parsed sections. Each section contains a list of parsed paragraphs, each including a list of parsed sentences. If a sentence cites a paper, we create a “cite span” that records the citation marker such as “[1]”, the position of the citation marker in the sentence, and the cited paper’s index in the “Reference” list.  
We implemented a S2ORC parser to convert documents in the S2ORC corpus to our JSON format. For PDFs in the arXiv corpus, we first used the s2orc-doc2json (Lo et al., 2020) to convert them into S2ORC format and then we applied our S2ORC parser. For XML files in the PMCOA corpus, we implemented an XML parser  
2https://vercel.com/  
> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0009-00.png]**:
NLP
machinetranslation
AND
OR
NMT
Pub.Year:2020
OR
Pub.Year:2021
Pub.Year:2022  
**----- Start of picture text -----**<br>
NLP<br>machine translation<br>AND OR<br>NMT<br>Pub. Year: 2020<br>OR<br>Pub. Year: 2021<br>Pub. Year: 2022<br>**----- End of picture text -----**<br>
```
