LScD (Leicester Scientific Dictionary)
LScD (Leicester Scientific Dictionary)
August 2019 by Neslihan Suzen, PhD student at the University of Leicester (firstname.lastname@example.org)
Supervised by Prof Alexander Gorban and Dr Evgeny Mirkes
This document provides the pre-processing steps for creating an ordered list of words from the LSC (Leicester Scientific Corpus)  and the description of LScD (Leicester Scientific Dictionary). The dictionary is built to be used by Neslihan Suzen for her PhD project and to allow researchers accessible and reproducible data use for academic studies. R code for producing the dictionary from LSC and instructions for usage of the code are available in . The code can be also used for list of texts from other sources. In this case, amendments to the code may be required.
LSC is a collection of abstracts of articles and proceeding papers published in 2014 and indexed by the Web of Science (WoS) database . Each document contains title, list of authors, list of categories, list of research areas, and times cited. The corpus contains only documents in English. The corpus was collected in July 2018 and contains the number of citations from publication date to July 2018. The total number of documents in LSC is 1,673,824. There are 119 documents with empty authors list, we did not exclude these documents
LScD is an ordered list of words from texts of abstracts in LSC.The dictionary stores 974,238 unique words, is sorted by the number of documents containing the word in descending order. All words in the LScD are in stemmed form of words. The LScD contains the following information:
1.Unique words in abstracts
2.Number of documents containing each word
3.Number of appearance of a word in the entire corpus
Processing the LSC
Step 1.Downloading the LSC Online: Use of the LSC is subject to acceptance of request of the link by email. To access the LSC for research purposes, please email to email@example.com. The data is extracted from Web of Science . You may not copy or distribute this data in whole or in part without the written consent of Clarivate Analytics.
Step 2.Importing the Corpus to R: The full code for processing the corpus including reading the LSC to R can be found in the GitHub .
All following steps can be applied for arbitrary list of texts from any source with changes of parameter. We note that the structure of the corpus such as file format and names (also the position) of fields should be taken into account to apply our code. The organisation of CSV files of LSC is described in data description file for LSC .
Step 3.Extracting Abstracts and Saving Metadata: Metadata that includes all fields in a document excluding abstracts and the field of abstracts are separated. Metadata is then saved as MetaData.R to the output directory. Fields of metadata are: List_of_Authors, Title, Categories, Research_Areas, Total_Times_Cited and Times_cited_in_Core_Collection.
Step 4.Text Pre-processing Steps on the Collection of Abstracts: In this section, we presented our approaches to pre-process abstracts of the LSC.
1.Removing punctuations and special characters: This is the process of substitution of all non-alphanumeric characters by space. We did not substitute the character “-” in this step, because we need to keep words like “z-score”, “non-payment” and “pre-processing” in order not to lose the actual meaning of such words. A processing of uniting prefixes with words are performed in later steps of pre-processing.
2.Lowercasing the text data: Lowercasing is one of the most effective pre-processing step in text mining problems to avoid considering same words like “Corpus”, “corpus” and “CORPUS” differently. Entire collection of texts are converted to lowercase.
3.Uniting prefixes of words: Words containing prefixes joined with character “-” are united as a word. The list of prefixes united for this research are listed in the file “list_of_prefixes.csv”. The most of prefixes are extracted from . We also added commonly used prefixes: ‘e’, ‘extra’, ‘per’, ‘self’ and ‘ultra’.
4.Substitution of words: Some of words joined with “-” in the abstracts of the LSC require an additional process of substitution to avoid losing the meaning of the word before removing the character “-”. Some examples of such words are “z-test”, “well-known” and “chi-square”. These words have been substituted to “ztest”, “wellknown” and “chisquare”. Identification of such words is done by sampling of abstracts form LSC. The full list of such words and decision taken for substitution are presented in the file “list_of_substitution.csv”.
5.Removing the character “-”: All remaining character “-” are replaced by space.
6.Removing numbers: All digits which are not included in a word are replaced by space. All words that contain digits and letters are kept because alphanumeric characters such as chemical formula might be important for our analysis. Some examples for words containing numbers are “co2”, “h2o” and “21st”.
7.Stemming: Stemming is the process of converting inflected words into their word stem. As stemming removes suffixes and reduces the number of words in corpus, this step results in uniting several forms of words with similar meaning into one form and also saving memory space and time . All words in the LScD are stemmed to their word stem.
8.Stop words removal: In natural language processing, stop words are defined as words that are extreme common but provide little value in a language. Some common stop words in English are ‘I’, ‘the’, ‘a’ etc. In our research, we used ‘tm’ package in R to remove stop words . There are 174 English stop words listed in the package.
Step 5.Writing the LScD into CSV Format: There are 1,673,824 plain pre-processed texts for further analysis. All unique words in the corpus are extracted and written in the file “LScD.csv”.
The Organisation of the LScD
The total number of words in the file “LScD.csv” is 974,238. Each field in the file is described below:
Word: It contains unique words from the corpus. All words are in lowercase and their stem forms. The field is sorted by the number of documents that contain words in descending order.
Number of Documents Containing the Word: In this content, binary calculation is used: if a word exists in an abstract then there is a count of 1. If the word exits more than once in a document, the count is still 1. Total number of document containing the word is counted as the sum of 1s in the entire corpus.
Number of Appearance in Corpus: It contains how many times a word occurs in the corpus when the corpus is considered as one large document.
Instructions for R Code
LScD_Creation.R is an R script for processing the LSC to create an ordered list of words from the corpus . All outputs of the code are saved as RData file. LScD is also saved in CSV format. Outputs of the code are:
Metadata File: It includes all fields in a document from LSC excluding abstracts. Fields are List_of_Authors, Title, Categories, Research_Areas, Total_Times_Cited and Times_cited_in_Core_Collection.
File of Abstracts: It contains all abstracts after pre-processing steps defined in the step 4.
DTM: It is the Document Term Matrix constructed from the LSC. Each entry of the matrix is the number of times the word occurs in the corresponding document.
LScD: This file contains of ordered list of words from LSC as defined in the previous section.
The code can be used by:
1.Download the folder ‘LSC’, ‘list_of_prefixes.csv’ and ‘list_of_substitution.csv’
2.Open LScD_Creation.R script
3.Change parameters in the script: replace with the full path of the directory with source files and the full path of the directory to which you write output files
4.Run the full code.
N. Suzen. (2019). LSC (Leicester Scientific Corpus) [Dataset]. Available: https://doi.org/10.25392/leicester.data.9449639.v1
N. Suzen. (2019). LScD-LEICESTER SCIENTIFIC DICTIONARY CREATION. Available: https://github.com/neslihansuzen/LScD-LEICESTER-SCIENTIFIC-DICTIONARY-CREATION
Web of Science. (15 July). Available: https://apps.webofknowledge.com/
A. Thomas, "Common Prefixes, Suffixes and Roots," Center for Development and Learning, 2013.
C. Ramasubramanian and R. Ramya, "Effective pre-processing activities in text mining using improved porter’s stemming algorithm," International Journal of Advanced Research in Computer and Communication Engineering, vol. 2, no. 12, pp. 4536-4538, 2013.
I. Feinerer, "Introduction to the tm Package Text Mining in R," Accessible en ligne: https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf, 2013.