## **SCILIT: A Platform for Joint Scientific Literature Discovery, Summarization and Citation Generation** 

**Nianlong Gu** 

Institute of Neuroinformatics, University of Zurich and ETH Zurich nianlong@ini.ethz.ch 

## **Richard H.R. Hahnloser** 

Institute of Neuroinformatics, University of Zurich and ETH Zurich rich@ini.ethz.ch 

## **Abstract** 

Scientific writing involves retrieving, summarizing, and citing relevant papers, which can be time-consuming processes. Although in many workflows these processes are serially linked, there are opportunities for natural language processing (NLP) to provide end-to-end assistive tools. We propose SCILIT, a pipeline that automatically recommends relevant papers, extracts highlights, and suggests a reference sentence as a citation of a paper, taking into consideration the user-provided context and keywords. SCILIT efficiently recommends papers from large databases of hundreds of millions of papers using a two-stage prefetching and re-ranking literature search system that flexibly deals with addition and removal of a paper database. We provide a convenient user interface that displays the recommended papers as extractive summaries and that offers abstractively-generated citing sentences which are aligned with the provided context and which mention the chosen keyword(s). Our assistive tool for literature discovery and scientific writing is available at https://scilit.vercel.app 

## **1 Introduction** 

When we compose sentences like “Our experiments show that XXX performs significantly worse than YYY” in a manuscript, we may want to find papers that report similar performance evaluations (Cohan et al., 2019) and discuss these in our manuscript. This process is a non-trivial task requiring in-depth human involvement in finding, summarizing, and citing papers, which raises the question whether it is possible to partly automate this process to reduce users’ cognitive load in searching, retrieving, reading, and rephrasing related findings. 

Recent advances in natural language processing (NLP) help answer this question. First, releases of large scientific corpora such as S2ORC (Lo et al., 2020) and General Index (Else, 2021) provide opportunities for building large databases of scientific 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0001-10.png]**:
ContextText
candidate
Highlights
we observe that
papers
 In this paper, we show
the pooling strategy
Extractive
how universal sentence
has a large impact
Summarization
 Our experiments show .
There, the MAX
LSTM architecture with
strategy perform
max pooling, trained on
significantly worse
Literature
the SNLI dataset, yields
than MEAN or
state-of-the-art...
CLS-token strategy.
Retrieval
Citation
Citation Sentence
Keywords
Generation
This is in contrast to [CIT]
who found it beneficial for
MAX pooling;
the BiLSTM-layer of
Pooling strategy
InferSent to use MAX
instead of MEAN pooling



**----- Start of picture text -----**<br>
Context Text candidate  Highlights<br>papers<br>we observe that • In this paper, we show<br>the pooling strategy  Extractive   how universal sentence ...<br>has a large impact. There, the MAX  Summarization •  LSTM architecture with  Our experiments show ...<br>strategy perform   max pooling, trained on<br>significantly worse than MEAN or  Literature  the SNLI dataset, yields    state-of-the-art ...<br>CLS-token strategy. Retrieval<br>Citation  Citation Sentence<br>Generation<br>Keywords This is in contrast to [CIT],<br>who found it beneficial for<br>MAX pooling; Pooling strategy the BiLSTM-layer of InferSent to use instead of MEAN pooling.MAX<br>**----- End of picture text -----**<br>


Figure 1: The main workflow of our platform. 

papers. Second, such databases can be linked to systems for text retrieval (Guo et al., 2020), citation recommendation (Färber and Jatowt, 2020; Gu et al., 2022b; Medi´c and Snajder, 2020), extractive summarization (Zhong et al., 2020; Gidiotis and Tsoumakas, 2020; Gu et al., 2022a), and citation generation (Xing et al., 2020a; Ge et al., 2021; Wang et al., 2022), all of which can be tailored to meet the requirements of an author’s manuscript. 

To build a comprehensive system that helps authors with finding, reading, and summarizing of literature, the following challenges must be overcome: The system must index many papers (e.g., S2ORC has over 136 million papers (Lo et al., 2020)) to achieve good coverage, it must respond quickly to queries, and it must be flexible to handle database additions and deletions. In addition, the overall architecture should be modular to make it simple to upgrade components when better algorithms become available. 

To meet these challenges, we developed SCILIT, a platform for concurrent literature discovery, summarization, and citation generation. We propose a hierarchical architecture for paper retrieval that efficiently retrieves papers from multiple large corpora. On each corpus (e.g., S2ORC and PMCOA (of Medicine, 2003)), we build an efficient prefetching system based on a keyword inverted index and a document embedding index. The prefetched documents are then re-ordered (re-ranked) by a fine- 

tuned SciBERT (Beltagy et al., 2019). Such an architecture allows us to dynamically add or remove databases and update one database and its index without significantly affecting the others. From a user-chosen document (i.e., target paper), we extract highlights using a light-weight extractive summarization model proposed in Gu et al. (2022a). Furthermore, using a fine-tuned T5 model (Raffel et al., 2020), we generate a citing sentence based on the abstract of the target paper, the context (the text surrounding the original citation sentence), and the keywords provided by users. We also develop a microservice-based architecture that allows easy updating of algorithms. 

In summary, our main contributions are: 

- We demonstrate SCILIT, a platform for searching, summarizing, and citing scientific papers. 

- We evaluate SCILIT on scientific literature retrieval, paper summarization, and contextaware citation sentence generation, and showcase the generation of a related-work paragraph. 

- A live demo website of our system is at https://scilit.vercel.app and our implementation and data are at https://github.com/nianlonggu/ SciLit and a video demonstrating the system can be viewed at https://youtu.be/PKvNaY5Og1Y 

## **2 SciLit** 

Figure 1 shows the workflow of our system. A literature discovery module receives a context text and keywords provided by a user and recommends a list of relevant papers that are semantically similar with the context and that match the keywords used as Boolean filters (Gökçe et al., 2020). For each recommended paper, an extractive summarizer selects a short list of sentences from the full text as highlights. From the target paper selected by the user, a citation generation module takes the abstract together with the context and keywords as inputs and generates a citation sentence that references the target paper and that fits the context and keywords. 

We define the context as the text before a citation sentence because we focus on the workflow of first finding papers and then writing citation sentences, rather than finding the missing citation in a 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0002-08.png]**:
Prefetch
Database
keyword
Neares Neighbor
Boolean filter
Search
Remove
duplicates
Rerank
Prefetch
Database



**----- Start of picture text -----**<br>
Prefetch<br>Database keyword  Neares Neighbor<br>Boolean filter  Search  Remove Rerank<br>duplicates<br>Prefetch<br>Database<br>**----- End of picture text -----**<br>


Figure 2: Schematic of literature retrieval. From each database, candidate documents are prefetched by a cascade of keyword boolean filter and embedding-based nearest neighbor search. Then, candidate documents are reranked by a fine-tuned SciBERT. 

given sentence as in Gu et al. (2022b); Medi´c and Snajder (2020). The user-provided keywords are optional. When no keywords are explicitly given during training and evaluation of our system, we use the keywords occurring in both the context, the cited paper, and the citation sentence as a substitutes for user-provided keywords. 

## **2.1 Literature Discovery** 

The literature discovery module takes as inputs the context and keywords and recommends papers that are worth citing in the current context. To strike a balance between query accuracy and speed on large scientific corpora, our document discovery module employs a two-stage prefetching-ranking strategy (Gu et al., 2022b) (Figure 2). For each scientific corpus, we build a database and create an efficient prefetching model that we use to pre-filter _Np_ (see the discussion of _Np_ in Table 2 and Section 3.2) candidate documents based on the provided keywords and context. After removing duplicates, we re-rank the prefetched documents from each database to produce the final order. 

**Databases.** We dump each corpus into a separate SQLite (Hipp, 2000) database to allow flexibility in deploying and updating of independent prefetching servers. We further process documents from different corpora into a unified JSON schema so that we can use the same codebase to index, query, summarize, and display documents from different corpora. The JSON schema includes “Title”, “Author”, etc., for metadata, and “Content.Abstract_Parsed”, “Content.Fullbody_Parsed” for parsed full text, The details are given in Appendix B. 

**Prefetching.** The prefetching model of a given SQLite database consists of an inverted index and an embedding index. The inverted index stores the paper IDs of all publications that contain a given 

|**Corpus**|**Databases**<br># of<br>papers<br># papers with<br>fullbody<br>until<br>date|**Inverted Index**<br>keywords<br>length<br># of<br>keywords<br>data<br>format<br>storage<br>size|**Embedding Index**<br>embedding<br>dimension<br>storage<br>size|
|---|---|---|---|
|S2ORC<br>PMCOA<br>arXiv|136.60 M<br>12.44 M<br>2020-04-14<br>2.89 M<br>2.73 M<br>2022-06-17<br>1.69 M<br>1.69 M<br>2022-07-28|unigram, bigram<br>1.20 B<br>sqlitedict<br>769 GB<br>0.30 B<br>145 GB<br>0.15 B<br>77 GB|256<br>169 GB<br>2.9 GB<br>1.7 GB|



Table 1: Statistics of our literature discovery system. We indexed S2ORC (Lo et al., 2020), PMCOA (of Medicine, 2003), and arXiv (Kaggle, 2022), which contain large numbers of recent scientific papers in diverse fields. 

keyword, such as a unigram like “computer” or a bigram like “machine learning”, where the paper ID is a unique identifier using which we can retrieve the paper’s content from the database. The embedding index is formed by the embeddings of all papers in the database. Embeddings are 256-dimensional vectors computed by Sent2Vec (Pagliardini et al., 2018) (we simply average the embeddings of all words in a document). We train Sent2Vec using sentences obtained from the full text of the papers contained in S2ORC. 

Using the keywords and a specific syntax, we first perform Boolean filtering (Gökçe et al., 2020) of the inverted index. For example, given “POS tag;2010..2022”, we will filter papers published between 2010 and 2022 that mention “POS tag”. The filtered papers are then ranked based on the cosine similarity between the papers’ Sent2Vec embeddings and the context embedding. Such a hybrid of lexical filtering and semantic ranking allows users to find papers that are semantically similar to the context and that flexibly meet a constrained search scope. 

Statistics for the database and indexing system are reported in Table 1. Details of the indexing implementation are shown in Appendix C. 

**Duplicate Removal.** Since corpora can overlap, the prefetched candidates from multiple corpora can contain duplicate items. To remove duplicated candidates, we check the title and authors and keep only one record per paper for reranking. 

**Reranking.** We use SciBERT (Beltagy et al., 2019) to rerank prefetched candidates, aiming at highly ranking papers that can be cited given the context and keywords. We follow Gu et al. (2022b) to compute an affinity score as follows: we pass an input text “[CLS] _query_ [PAD] _paper_ [PAD]” to SciBERT, where the _query q_ is a concatenation of the context and the keywords, and _paper d_ is a concatenation of the title and the abstract of the candidate paper. The encoded output of the “[CLS]” token is passed to a linear layer, which outputs a scalar _s_ ( _q, d_ ) that 

we interpret as the affinity score between the query _q_ and the paper _d_ . To train the reranker, we use the cross-entropy loss: 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0003-08.png]**:
exp s(q,d+)
L=
-log
exp s(q, d )
(1)



where _d_[+] is the paper actually cited in the query, and _d[−] i_[is one of] _[ N]_[(] _[N]_[=][10)][ uncited papers that] are randomly sampled from prefetched candidate at each training iteration. 

## **2.2 Extractive Summarization** 

The extractive summarization module extract a short list of sentences from the full text of a paper to highlight the main points to a reader. We choose the summary to be extractive rather than abstractive to prevent readers from being misled by the potential hallucinations introduced in abstractive summarization models (Nan et al., 2021; Xu et al., 2020; Wang et al., 2020). The extractive summarization model must efficiently select sentences from a given document so that users do not experience obvious delays. 

In this paper, we employ MemSum, an RNNbased extractive summarizer that models the extraction process as a Markov decision process in a reinforcement learning framework. MemSum has been trained on the PubMed dataset Gu et al. (2022a) and it can summarize long papers without exhausting GPU memory due to its lightweight model structure. Also, MemSum is computationally efficient, taking only 0.1 sec on average to summarize a paper. These features make it a suitable model for our extractive summarization module. 

## **2.3 Citation Generation Module** 

The citation generation module acts as an abstract summarizer that takes as input the context, the keywords, and the target paper to be cited; it then generates a sentence that cites the target paper and narrates it in context. 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0004-00.png]**:
User
frontend
Interface
response
request
gateway
API
Gateway
context, eywords and iteD
query
citation sentences
reranked IDs
highlights
Literature
Extractive
Citation
Discovery
Summarization
Content
query & prefetched IDs
Generation
reranked IDs
fultext
backend
prefetched IDs
abstract
Prefetch &re-
Rerank
IDs
早
Database
move duplicates
titles & abstracts
Interface
query
IDs
contents
IDS
titles & authors
Index
Index
Index
DB
DB
DB
(S20RC)
(PMCOA)
(arXiv)
(S2ORC)
(PMCOA)
(arXiv)



**----- Start of picture text -----**<br>
User<br>frontend Interface<br>response request<br>gateway API<br>Gateway<br>Literature Extractive  Citation<br>Discovery Summarization  Generation<br>backend<br>Prefetch & re-  Rerank IDs Database<br>move duplicates  titles & abstracts  Interface<br>IDs<br>titles & authors<br>Index Index Index DB DB DB<br>(S2ORC) (PMCOA) (arXiv) (S2ORC) (PMCOA) (arXiv)<br>citation sentencescontext, keywords and cited ID<br>reranked IDs query<br>ID<br>fulltext<br>IDs<br>query<br>reranked IDsquery & prefetched IDs<br>IDs<br>contents<br>prefetched IDs query<br>ID<br>Content<br>ID<br>abstract<br>highlights ID<br>**----- End of picture text -----**<br>


Figure 3: The architecture of our platform. The direction of an arrow represents the direction of data flow. 

By providing keywords as inputs to a sequenceto-sequence model, our input differs from previous works on automatic citation generation (Ge et al., 2021; Xing et al., 2020b), which use only the context as inputs but no keywords. We consider keywords to be an important source of input because we believe that authors usually have a clear intention when citing a paper, and a keyword can sometimes more easily convey this intention than a long text. In the case shown in Figure 1, for example, after writing the context “MAX pooling performs worse than MEAN pooling”, the author naturally intends to discuss papers about “MAX pooling”. Therefore, the keyword “MAX pooling” should be used as a thematic cue for citation sentence generation. Moreover, making the citation generation model conditional on keywords also allows users to fine-tune the generated citation text by simply adjusting the keywords, thus making the system interactive and conveniently tunable. 

To make the generation conditional on context, keywords, and cited papers, we fine-tuned a T5 (Raffel et al., 2020) so that its input is a concatenation of three attributes: keywords, context, and the abstract of a cited paper, each preceded by a special field name to make attributes distinguishable to the model: keywords: XXX. context: XXX. target abstract: XXX. The corresponding decoding output is the actual citation sentence that cites the target paper. 

## **2.4 Microservice-based Architecture** 

We build our platform as a network of microservices (Figure 3). An API gateway routes requests from the frontend to the target microservice on the backend. The microservices run separate modules on their respective Flask servers (Aggarwal, 2014) and communicate with each other by sending HTTP requests and waiting for responses. When a query request arrives, the API gateway forwards the query to the literature discovery service, which calls the prefetching and reranking services to get the reranked IDs. The API gateway then sends the paper IDs to the extractive summarization service to receive the highlights of each recommended paper. The gateway also sends the context, keywords, and recommended paper IDs to the citation generation service to suggest citation sentences. The database interface service manages the databases of multiple scientific corpora and provides a unified interface to access the paper content given its ID. Each microservice runs in an independent environment, which makes it easy to upgrade backend systems online, such as adding or removing a database or updating an algorithm. 

## **3 Evaluation** 

In this section, we first show how SCILIT works and then we evaluate its performance. 

## **3.1 Demonstration** 

Our user interface runs on a web page (Figure 4) created with ReactJS[1] . The left sidebar is an input panel where users can enter context and keywords and trigger a query by clicking the _search_ button. Retrieved papers are displayed in the search-results panel on the right. Users can scroll up and down or paginate to browse through the recommended papers. Each paper is accompanied by highlights and a suggested citation sentence generated by our extractive summarization and citation generation services, respectively. Users can cite a paper by clicking on the _cite_ button and the suggested citation sentence will jump to the editing area on the left where users can tweak the sentence by changing keywords and clicking on the _fine-tune generation_ button, or they can edit the sentences manually. Exporting citation information is also supported. 

> 1https://reactjs.org/ 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0005-00.png]**:
ifJointScientificLiterature Discovery,Summarization and Citation Generation
• We explore the few-shot setting of extractive question answering, and demonstrate that existing methods, based on fine-tuning
Context Text
large pretrained language models, fail in this setup. I
Since the development of BERT there have
been many efforts towards adding or
modifying the pretraining tasks.
surprisingly good results even when only an order of a hundred examples are available. 
 For example, recurring span selection #CIT masks all recurring spans but one, and asks the model to select the correct span in
the passage for each masked span.
VIEW FULLTEXT 
s
M Joshi, D Chen, Y Liu, et al. Transactions of the Association for Computational Linguistics, 2019
国
Keywords
uo suje isiel u ym 1a soednno Auasisuoo jeu yooidde buujeaid jenl-ueds e aonpoul am saded su u 
span selection
span selection tasks such as question answering and coreference resolution. 
lixa1 jo sueds topeud pue juesaudeu jeeq oi peubisep s teu pouew bujujen-aid e 1t3gueds jueseud a *
SEARCH Q
FINETUNEGENERATION
• Our method differs from BERT in both the masking scheme and the training objectives. 
 • Second, we introduce a novel span-boundary objective (SBO) to train the model to predict the entire masked span from the
Selected Citation Text
Span-BERT #CIT masks contiguous random
observed tokens at its boundary. 
spans instead of random tokens, and trains 
eedde Aau yom ui xauo eu bujsn Aielos sueds euue sopaud on lepow eu seoy buyse pesequeds •
the span boundary representation to predict
jeue asueds wopue, snonbguo upyseu (t) Aq Ltag spuejxe yoqm Sujujeqeud peseq-ueds so, pouew mau e pejuasaud a/ •
the entire content of the masked span.
than random tokens, and (2) training the span boundary representations to predict the entire content of .. 
yoan pue sysl jo Aleen e uo sauljesen 1iio ie woedino jey sjepow spielA ssaooid buure-aid ano dseujeoo •
substantially better performance on span selection tasks in particular. 
S66
Span-BERT #CIT masks contiguous random spans instead of random tokens, and trains the span boundary representation to
predict the entire content of the masked span.
VIEW FULLTEXT 
EXPORT CITATION
50



Figure 4: Overview of the user interface. The context text comes from the related work section in Glass et al. (2020). 

|_Np_<br>time<br>(_s_/query)|Recall@_K_ (R@_K_)<br>R@1<br>R@5<br>R@10<br>R@20<br>R@50<br>R@100|
|---|---|
|50<br>**2.02**<br>100<br>2.55<br>200<br>3.26<br>300<br>3.93|**0.107**<br>0.208<br>0.263<br>0.305<br>0.327<br>0.331<br>0.096<br>0.215<br>**0.278**<br>0.328<br>0.384<br>0.401<br>0.095<br>**0.220**<br>0.275<br>**0.339**<br>0.420<br>0.452<br>0.095<br>0.204<br>0.273<br>0.330<br>**0.422**<br>**0.482**|



Table 2: Paper retrieval performance measured by the recall of the top _K_ recommendations. _Np_ denotes the number of prefetched candidates per corpus. 

## **3.2 Performance** 

**Evaluation Dataset.** We evaluated SCILIT on a test set containing 1530 samples, mainly from papers published in 2022 in the fields of computer science and biomedical science. Each sample contains the following information: 1) context, up to 6 sentences preceding the citation sentence and within the same section; 2) keywords, up to 2 unior bi-grams that occur in all of the context, the citation sentence, and the cited paper; 3) ID of the cited paper; 4) the citation sentence following the context, which is the ground truth for evaluating generated citations. For quality control, we only include citation sentences in the test set that cite one paper. 

**Paper Retrieval.** For each sample in the evaluation dataset, we use context and keywords as queries and invoke the literature search service to 

||Model<br>BertSum (Liu,2019)|Rouge-1<br>42.53|Rouge-2<br>16.89|Rouge-L<br>39.18|
|---|---|---|---|---|
||MemSum (Gu et al.,2022a)|**46.40***|**19.61***|**42.66***|



Table 3: The extractive summarization performance. “*" indicates statistical significance in comparison to baselines with a 95% bootstrap confidence interval. 

first prefetch _Np_ candidates from each of the three corpora (S2ORC, PMCOA, and arXiv). We remove duplicates and then we rank the prefetched candidates. The top _K_ recommendations serve to evaluate the retrieval performance (Table 2). We observed that for large _K_ ( _K_ = 50 _,_ 100), the recall increases as _Np_ increases, whereas for small _K_ ( _K_ = 5 _,_ 10 _,_ 20), the recall first increases and then starts to decrease, indicating that the reranking performance is impacted by more prefetched candidates. We choose _Np_ = 100 as the default value, which was fast and achieved the best performance for R@10. 

**Extractive Summarization.** To evaluate the summaries, following Zhong et al. (2020); Xiao and Carenini (2019), we computed the ROUGE F1 scores between the summary sentences extracted from the full body and the corresponding abstract. MemSum significantly outperformed BertSum (Liu, 2019), a Bert-based summarizer that 

|generation pipeline|Rouge-1|Rouge-2|Rouge-L|
|---|---|---|---|
|generation-only|32.96|9.19|24.52|
|Best of top1paper|28.62|6.00|21.05|
|Best of top5papers|34.92|9.59|26.23|
|Best of top10papers|**36.83***|**10.98***|**28.10***|



Table 4: The performance of citation generation. 

evaluation dataset. We observe a significant degradation in the performance of literature retrieval and citation generation (Table 5), which demonstrates the utility of keywords for recommending relevant papers and generating accurate citations on our platform. 

## **4 Related Work** 

|retrieval<br>(_Np_ = 100)<br>R@1|R@5|R@10 R@20|R@50 R@100|
|---|---|---|---|
|w keywords<br>0.096<br>w/o keywords 0.013<br>citation generation<br>w keywords<br>w/o keywords|0.215<br>0.050<br>Rouge-1<br>32.96<br>26.57|0.278<br>0.328<br>0.085<br>0.125<br>Rouge-2<br>9.19<br>5.56|0.384<br>0.401<br>0.199<br>0.250<br>Rouge-L<br>24.52<br>20.39|



Table 5: Ablation study on retrieval and citation generation performance. 

requires truncation of long documents, indicating the effectiveness of MemSum in extractively summarizing scientific documents. 

**Citation Generation.** To evaluate our joint retrieval and citation generation pipeline, we let our system first recommend papers based on context and keywords and then we let it generate _K_ citation sentences, one for each of the top _K_ recommended papers. Then, we calculate the ROUGE F1 score between the ground truth citation sentence and each of the _K_ sentences and record the highest ROUGE F1 score of them. We compared the “Best-of-top- _K_ ” pipeline to the “generation-only” pipeline, where we directly provide the truly cited paper for citation generation. 

We observed that for _K_ = 5 and 10, the “Bestof-top- _K_ ” pipeline achieved significantly higher ROUGE scores than the "generation only" pipeline (Table 4), indicating that the paper retrieval module contributes positively to the citation generation process and increases the chance of suggesting appropriate citation sentences. We believe that this result further supports our idea of developing an integrated system for joint retrieval and generation. 

## **3.3 Ablation Study** 

To analyze the impact of keywords, we evaluated retrieval and generation systems without keywords. For document retrieval, we first prefetch _Np_ = 100 candidates from each corpus and then rank them based on context only. For citation generation, we trained a T5 model to learn to generate citation sentences with only the context and the title and abstract of the cited paper and evaluated it on the 

Recently, AI-driven platforms focused on literature recommendation and scientific paper summarization have been proposed. (keywords: platform, paper: #2) _One such platform is AI Research Navigator (Fadaee et al., 2020), which combines classical keyword search with neural retrieval to discover and organize relevant literature._ (keywords: scientific; summarization; platform, paper #3) _Another platform is Anne O’Tate, which supports user-driven summarization, drill-down and mining of search results from PubMed, the leading search engine for biomedical literature (Smalheiser et al., 2021)._ (keywords: related work generation, paper #9) _Chen and Zhuge (2019) automatically generates related work by comparing the main text of the paper being written with the citations of other papers that cite the same references._ 

In the previous paragraph, the italicized citation sentences are generated from SCILIT. In generating each sentence, we use all the preceding sentences in the paragraph as contexts and use the keywords in parentheses to obtain the recommended papers and the corresponding citation sentences. The paper index in parentheses indicates the ranked order of recommended papers. 

## **5 Conclusion and Future Work** 

This paper demonstrates SCILIT, a platform for joint scientific literature retrieval, paper summarization, and citation generation. SCILIT can efficiently recommend papers from hundreds of millions of papers and proactively provides highlights and suggested citations to assist authors in reading and discussing the scientific literature. In addition, our prefetching, reranking, and citation generation system can be conditioned on user-provided keywords, which provides flexibility and adjusts the platform’s response to user intention. In the future, we will further improve the performance of each module, especially the citation generation part, and collect feedback from users to improve the overall workflow and the frontend user experience. 

## **Acknowledgements** 

We thank the anonymous reviewers for their useful comments. 

## **References** 

- Titipat Achakulvisut, Daniel Acuna, and Konrad Kording. 2020. Pubmed parser: A python parser for pubmed open-access xml subset and medline xml dataset xml dataset. _Journal of Open Source Software_ , 5(46):1979. 

- Shalabh Aggarwal. 2014. _Flask framework cookbook_ . Packt Publishing Ltd. 

- Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. 

- Jingqiang Chen and Hai Zhuge. 2019. Automatic generation of related work through summarizing citations. _Concurrency and Computation: Practice and Experience_ , 31(3):e4261. E4261 CPE-16-0462.R2. 

- Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics. 

- Holly Else. 2021. Giant, free index to world’s research papers released online. available online at https://www. nature. com/articles/d41586-02102895-8, accessed 2021-12-15. _Nature (Oct 2021). https://doi. org/10.1038/d41586-021-02895-8_ . 

- Marzieh Fadaee, Olga Gureenkova, Fernando Rejon Barrera, Carsten Schnober, Wouter Weerkamp, and Jakub Zavrel. 2020. A new neural search and insights platform for navigating and organizing AI research. In _Proceedings of the First Workshop on Scholarly Document Processing_ , pages 207–213, Online. Association for Computational Linguistics. 

- Michael Färber and Adam Jatowt. 2020. Citation recommendation: approaches and datasets. _International Journal on Digital Libraries_ , 21(4):375–405. 

- Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International_ 

_Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1466–1478, Online. Association for Computational Linguistics. 

- Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of long documents. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:3029–3040. 

- Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, and Avi Sil. 2020. Span selection pretraining for question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2773–2782, Online. Association for Computational Linguistics. 

- Onur Gökçe, Jonathan Prada, Nikola I. Nikolov, Nianlong Gu, and Richard H.R. Hahnloser. 2020. Embedding-based scientific literature discovery in a text editor application. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 320–326, Online. Association for Computational Linguistics. 

- Nianlong Gu, Elliott Ash, and Richard Hahnloser. 2022a. MemSum: Extractive summarization of long documents using multi-step episodic Markov decision processes. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6507–6522, Dublin, Ireland. Association for Computational Linguistics. 

- Nianlong Gu, Yingqiang Gao, and Richard H. R. Hahnloser. 2022b. Local Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERTBased Reranking. In _Advances in Information Retrieval_ , pages 274–288, Cham. Springer International Publishing. 

- Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W Bruce Croft, and Xueqi Cheng. 2020. A deep look into neural ranking models for information retrieval. _Information Processing & Management_ , 57(6):102067. 

Richard D Hipp. 2000. Sqlite. https://www. sqlite.org/index.html. Version: 3.39.2, Accesses: 2022-07-31. 

Kaggle. 2022. arXiv Dataset. https: //www.kaggle.com/datasets/ Cornell-University/arxiv. Accesses: 2022-08-01. 

- Yang Liu. 2019. Fine-tune bert for extractive summarization. 

- Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983, Online. Association for Computational Linguistics. 

- Zoran Medi´c and Jan Snajder. 2020. Improved local citation recommendation based on context enhanced with global information. In _Proceedings of the First Workshop on Scholarly Document Processing_ , pages 97–103, Online. Association for Computational Linguistics. 

- Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entitylevel factual consistency of abstractive text summarization. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733, Online. Association for Computational Linguistics. 

- Bethesda (MD): National Library of Medicine. 2003. Pmc open access subset [internet]. https://www.ncbi.nlm.nih.gov/pmc/ tools/openftlist/. Accesses: 2022-07-30. 

- Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 528–540. ArXiv: 1703.02507. 

- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_ , 21(140):1–67. 

Papers: A Pilot Study. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6181–6190, Online. Association for Computational Linguistics. 

- Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020b. Automatic generation of citation texts in scholarly papers: A pilot study. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6181–6190, Online. Association for Computational Linguistics. 

- Xinnuo Xu, Ondˇrej Dušek, Jingyi Li, Verena Rieser, and Ioannis Konstas. 2020. Fact-based content weighting for evaluating abstractive summarisation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5071–5081, Online. Association for Computational Linguistics. 

- Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive Summarization as Text Matching. _arXiv:2004.08795 [cs]_ . ArXiv: 2004.08795. 

## **A Hardware Information** 

We run the backend of SCILIT on a server with dual 64-Core AMD EPYC 7742 2.25GHz Processors, 2TB DDR4 3200MHz ECC Server Memory, and 4 _×_ 7.68TB NVME GEN4 PM9A3 for storage. The server is also equipped with two nVidia RTX A6000 48GB GPU. The frontend is hosted on Vercel[2] . 

## **B JSON Schema for Database** 

- Neil R Smalheiser, Dean P Fragnito, and Eric E Tirk. 2021. Anne o’tate: Value-added pubmed search engine for analysis and text mining. _PloS one_ , 16(3):e0248335. 

- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. _arXiv:2004.04228 [cs]_ . ArXiv: 2004.04228. 

- Yifan Wang, Yiping Song, Shuai Li, Chaoran Cheng, Wei Ju, Ming Zhang, and Sheng Wang. 2022. DisenCite: Graph-Based Disentangled Representation Learning for Context-Specific Citation Generation. _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):11449–11458. 

- Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3011–3021, Hong Kong, China. Association for Computational Linguistics. 

- Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020a. Automatic Generation of Citation Texts in Scholarly 

The details of our unified JSON schema is shown in Listing 1. As metadata we define “Author”, “Title”, “Abstract”, “Venue”, “DOI”, “URL”, and as parsed full text we define “PublicationDate”, "Content.Abstract_Parsed", and “Content.Fullbody_Parsed”. The parsed abstract or full body contains a list of parsed sections. Each section contains a list of parsed paragraphs, each including a list of parsed sentences. If a sentence cites a paper, we create a “cite span” that records the citation marker such as “[1]”, the position of the citation marker in the sentence, and the cited paper’s index in the “Reference” list. 

We implemented a S2ORC parser to convert documents in the S2ORC corpus to our JSON format. For PDFs in the arXiv corpus, we first used the s2orc-doc2json (Lo et al., 2020) to convert them into S2ORC format and then we applied our S2ORC parser. For XML files in the PMCOA corpus, we implemented an XML parser 

2https://vercel.com/ 




> **[Teks Gambar Hasil OCR - SciLit_Joint_Literature_Discovery,_Summarization_and_Citation_Generation.pdf-0009-00.png]**:
NLP
machinetranslation
AND
OR
NMT
Pub.Year:2020
OR
Pub.Year:2021
Pub.Year:2022



**----- Start of picture text -----**<br>
NLP<br>machine translation<br>AND OR<br>NMT<br>Pub. Year: 2020<br>OR<br>Pub. Year: 2021<br>Pub. Year: 2022<br>**----- End of picture text -----**<br>


Figure 5: The parsed tree structure of the given keywords string: NLP; machine learning|NMT; 2020..2022”. 

based on Achakulvisut et al. (2020) to convert XML to S2ORC format and then we applied again the S2ORC parser to convert it into our JSON format. 

## **C Prefetching Indexing Implementation** 

## **C.1 Inverted Index** 

The inverted index is a mapping table from keywords (unigrams and bigrams) to paper IDs. We extract keywords from the full text of each document and keep a bigram only if neither word is a stopword. We use sqlitedict[3] to store the inverted index for each corpus, which is an on-disk hashmap based on an SQLite database that allows us to efficiently obtain the paper ID for a given keyword without loading the entire inverted index into RAM. **Syntax Parsing.** Our platform allows users to filter documents using syntax-rich keyword strings. For example, to filter papers that contain the keywords ’NLP’ and either ’machine translation’ or ’NMT’ and that has been published between 2020 and 2022, one can compose a keyword string NLP; machine learning|NMT; 2020..2022”. We transform this keyword string into a tree of logical operations (Figure 5), wherein each node we denote the logical operations applied to the sub-nodes, and each leaf node contains a keyword. We implemented the tree using a Python dictionary (Listing 2). Then, we recursively traverse all nodes in the tree in a depth-first search, obtain the paper IDs with the keyword in each leaf node, and apply the logical operations indicated in each node to obtain the final paper ID at the root node. 

## **C.2 Embedding Index** 

**Structure of the Embedding Index.** The embedding index consists of three main components: 

The first component is a matrix _M ∈R[N][×][D]_ , where _N_ is the number of documents and _D_ is the dimensionality of document embeddings. Each document’s embedding is L2-normalized so that given an L2-normalized query embedding _eq ∈ R[D][×]_[1] , the matrix multiplication _Meq ∈R[N][×]_[1] represents the cosine similarity between the query embedding and all document embeddings. We use _Meq_ to rank documents and to obtain the indices of most similar paper embeddings. 

The second component is a mapping table from the index of a paper embedding in the matrix _M_ to the corresponding paper ID in our databases. With this mapping table we can get the papers’ content given the top ranked indices during _K_ nearest neighbor search (KNN). 

The last component is a reversed mapping table from the paper ID to the corresponding index in the embedding matrix. In our prefetching system, we first use the inverted index to pre-filter a subset of paper IDs based on given keywords. Then we use this reversed mapping table to obtain the corresponding paper embeddings and perform KNN among them. 

**Multi-Processing Speedup for Brute-Force Nearest Neighbor Search.** For a large corpus like S2ORC, the embedding matrix contains up to 136.6 million vectors, and performing matrix multiplication in a single thread is very time-consuming. To take full advantage of the multiple CPU cores on our server, we divide the embedding matrix into 137 shards, each containing about 1 million embeddings. We first run a brute-force nearest neighbor search in parallel to obtain _Np_ candidates on each shard, and then we rank the 137 _× Np_ candidates again to obtain the final _Np_ candidates. Given that our server has 128 cores, we can achieve a nearly linear speedup using multiprocessing KNN with slicing, and mathematically it is equivalent to performing a single KNN over the entire embedding matrix to obtain the closest _Np_ candidates. 

## **D Joint Retrieval and Citation Generation Examples** 

We show some specific results of joint paper retrieval and automatic generation of citation sentences. The contexts and keywords we used were obtained from papers in arXiv (Figure 6) and PM- 

1 {’Author’: [{’GivenName’: ’Daisuke’, ’FamilyName’: ’Ida’}, ...], 2 ’Title’: ’Topology Change of Black Holes’, 3 ’Abstract’: ’The topological structure of the event horizon has been investigated ...’, 4 ’Venue’: ’’, 5 ’DOI’: ’’, 6 ’URL’: ’’, 7 ’PublicationDate’: {’Year’: ’2007’, ’Month’: ’3’}, 8 ’Content’: { 9 ’Abstract’: ’’, 10 ’Abstract_Parsed’: [{ 11 ’section_id’: ’0’, 12 ’section_title’: ’Abstract’, 13 ’section_text’: [{ 14 ’paragraph_id’: ’0’, 15 ’paragraph_text’: [{ 16 ’sentence_id’: ’0’, 17 ’sentence_text’: ’The topological structure of the event horizon has been investigated in terms of the Morse theory.’, 18 ’cite_spans’: []}, 19 # ... 20 ]}, 21 # ... 22 ] 23 }], 24 ’Fullbody’: ’’, 25 ’Fullbody_Parsed’: [{ 26 ’section_id’: ’0’, 27 ’section_title’: ’Introduction’, 28 ’section_text’: [{ 29 ’paragraph_id’: ’0’, 30 ’paragraph_text’: [ 31 # ..., 32 { 33 ’sentence_id’: ’2’, 34 ’sentence_text’: ’[1, 2] This follows from the fact that the total curvature, which is the integral of the intrinsic scalar curvature over the horizon, is positive under the dominant energy condition and from the GaussBonnet theorem.’, 35 ’cite_spans’: [{’start’: ’4’, ’end’: ’6’, ’text’: ’2]’,’ref_id’: ’0 ’}] 36 }, 37 # ... 38 ] 39 }] 40 }]}, 41 ’Reference’: [{ 42 ’Title’: ’The large scale structure of space-times’, 43 ’Author’: [{’GivenName’: ’S’, ’FamilyName’: ’Hawking’}, 44 {’GivenName’: ’G’, ’FamilyName’: ’Ellis’}], 45 ’Venue’: ’’, 46 ’PublicationDate’: {’Year’: ’1973’}, 47 ’ReferenceText’: ’2. Hawking, S, and G Ellis. "The large scale structure of space-times." (1973).’}, 48 # ... 49 ] 50 } 

Listing 1: An example of the JSON schema that we used for parsing and storing scientific papers. 

1 { 2 ’operation’: ’AND’, 3 ’elements’: [ 4 {’operation’: ’AND’, 5 ’elements’: [{’operation’: None, ’elements’: [’nlp’]}]}, 6 {’operation’: ’OR’, 7 ’elements’: [ 8 {’operation’: ’AND’, 9 ’elements’: [{’operation’: None, 10 ’elements’: [’machine translation’]}]}, 11 {’operation’: ’AND’, 12 ’elements’: [{’operation’: None, ’elements’: [’nmt’]}]}]}, 13 {’operation’: ’OR’, 14 ’elements’: [ 15 {’operation’: None, ’elements’: [’publicationdate.year:2020’]}, 16 {’operation’: None, ’elements’: [’publicationdate.year:2021’]}, 17 {’operation’: None, ’elements’: [’publicationdate.year:2022’]} 18 ] 19 }] 20 } 

Listing 2: The dictionary representation of the tree structure shown in Figure 5. 

COA (Figure 7), respectively. In each example, the actual cited paper occurs in the top 5 paper recommendations, which we have highlighted with an underline. 

## **`Context:`** 

```
#OTHERCIT apply a multi-stream CNN model to extract and fuse deep features from the designed complementary shape-mot
ion representations. Zhu et al. #OTHERCIT organize the pairwise displacements between all body joints to obtain a cu
boid action representation and use attention-based deep CNN models to focus analysis on actions. Inspired by the fac
t that the skeleton data is naturally a topological graph, where the joints and bones are regarded as the nodes and
edges, Graph Convolutional Network (GCN) is adopted to boost the performance of skeleton based action recognition #O
THERCITS .
```

## **`Keywords:`** 

```
skeleton
```

## **`Recommended Papers and Generated Citations:`** 

## **`UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition`** 

```
Generated Citation:  UNIK #CIT is a generic skeleton-based action recognition model pre-trained on Posetics, a large
-scale human skeleton video dataset.
```

## **`ANUBIS: Skeleton Action Recognition Dataset, Review, and Benchmark`** 

```
Generated Citation:  The ANUBIS dataset #CIT is a large-scale 3D skeleton dataset, which mainly consists of 3D joint
s, bones, and limbs.
```

## **`SkeletonNet: Mining Deep Part Features for 3-D Action Recognition`** 

```
Generated Citation:  SkeletonNet #CIT is proposed to extract body-part-based features from each frame of the skeleto
n sequence, which are translation, rotation, and scale invariant.
```

## **`Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition`** 

```
Generated Citation:  ST-GCN #CIT learns both the spatial and temporalpatterns from the skeleton data, which leads t
o greater expressive power and stronger generalization capability.
```

```
Real Citation:  ST-GCN #CIT firstly constructs the skeleton as a graph and uses GCN to automatically capture the pat
terns embedded in the spatial configuration of the joints as well as their temporal dynamics.
```

```
SKELETON-BASED ACTION RECOGNITION WITH CONVOLUTIONAL NEURAL NETWORKS
Generated Citation:  In #CIT, a novel skeleton transformer module is designed to rearrange and select important skel
etonjoints automatically.
```

Figure 6: Example of joint paper retrieval and citation generation. The context text was obtained from arXiv. 

## **`Context:`** 

```
Quadricuspid aortic valve (QAV) is a rare congenital heart disease that is less common than bicuspid and unicuspid a
ortic valve [#OTHERCITS]. Coronary artery and coronary ostium anomalies are present in 2–10% of patients with QAV
[#OTHERCIT, #OTHERCIT, #OTHERCITS]. The functionalstatus of QAV is predominantly pure aortic regurgitation (AR) [#O
THERCIT, #OTHERCIT].
```

```
Keywords:
```

```
functionalstatus
```

## **`Recommended Papers and Generated Citations:`** 

```
ESC Guidelines for the management of grown-up congenital heart disease (new version 2010): The Task Force on the Man
agement of Grown-up Congenital Heart Disease of the European Society of Cardiology (ESC)
```

```
Generated Citation:  The functionalstatus of QAV is characterized by the presence of aortic regurgitation (AR), aor
tic stenosis (AR), and aortic stenosis (AR) [ #CIT ].
```

## **`Quadricuspid Aortic Valve: A Comprehensive Review`** 

```
Generated Citation:  The clinicalmanifestations of QAVdepend on the functionalstatus of the QAV and the associate
d disorders [ #CIT ].
```

```
Real Citation:  Clinical manifestations, such as palpitations, dyspnea, fatigue, and chest pain, depend on the funct
ional status of QAV and usually present in the fifth or sixth decade of life [#CIT].
```

## **`A classification system for the bicuspid aortic valve from 304 surgical specimens.`** 

```
Generated Citation:  The functionalstatus of QAV is characterized by the presence of a number of raphes, spatial po
sition of cusps or raphes, and number of raphes [ #CIT ].
```

```
Functional status of the quadricuspid aortic valve/an uncommon coincidence of congenital quadricuspid aortic valve a
ccompanied by hypertrophic obstructive cardiomyopathy.
```

```
Generated Citation:  The functionalstatus of QAV is characterized by hypertrophic obstructive cardiomyopathy (HOMC)
and hypertrophic obstructive cardiomyopathy (HOCM) [ #CIT ].
```

## **`Congenital coronary artery fistulae: a rare cause of heart failure in adults`** 

```
Generated Citation:  The functionalstatus of QAV can be improved by epicardial fistula ligation and coronary artery
bypass grafting with marked improvement in functionalstatus [ #CIT ].
```

~~Figure 7: Example of joint paper retrieval and citation generation. The context text was obtained from PMCOA.~~