Content uploaded by Debarshi Kr. Sanyal
Author content
All content in this area was uploaded by Debarshi Kr. Sanyal on Feb 16, 2019
Content may be subject to copyright.
Beyond Search: An Overview of Emerging Applications
for Scholarly Digital Libraries in the Era of Big Data
Debarshi Kumar Sanyal
Indian Institute of Technology Kharagpur, Kharagpur-721302, West Bengal, INDIA
Email: debarshisanyal@gmail.com
ABSTRACT
Nowadays, scholarly digital libraries are common. On the one hand, there are
domain specific libraries like IEEE Xplore, ACM Digital Library, on the other
hand there are multidisciplinary digital libraries like the newly built National
Digital Library of India that cater to a more general audience but index
significant volume of scholarly works. These libraries provide access to
millions of documents but do not always provide discovery mechanisms beyond
browse and keyword-based search. Innovative retrieval services and analytic
tools are needed to help users explore these rich repositories and keep abreast of
the latest developments in the academic landscape. In this chapter, we review
the emerging applications that are of critical importance to the usability of these
libraries. In particular, we consider information visualization, impact
measurement and recommendation services as the key emerging applications.
We also briefly discuss how they can be introduced in the National Digital
Library of India.
Keywords: Digital library, Big scholarly data, Information visualization,
Recommendation Systems, Usability
1 INTRODUCTION
Scholarly data normally refer to works reporting research results, are written by
subject matter experts and published in peer reviewed venues like edited books,
conference proceedings and journals (Cornell University, 2018). They present new
knowledge or novel perspectives on existing knowledge. Scholarly data are being
produced at stupendous rates. It has been reported that global scientific output
doubles every nine years (Noorden, 2014a). No doubt, data scientists are talking
about big scholarly data (Xia et al., 2017). This definition is generally widened to
include other academic resources like presentation slides, audio and video lectures,
project descriptions, patents, and datasets. A scholarly digital library (SDL) stores or
indexes scholarly works. An example is the e-print service arXiv
(https://arxiv.org/) that contains north of 1.3 million e-prints in Physics,
Mathematics, Computer Science, Quantitative Biology, Quantitative Finance,
Statistics, Electrical Engineering and Systems Science, and Economics. In this
chapter, we focus on large SDLs which index upwards of several thousand scholarly
works. The digital libraries of many technical publishers like IEEE
(https://ieeexplore.ieee.org/Xplore/home.jsp)
and Springer
(https://link.springer.com/) fall comfortably in this category. Another
example is the National Digital Library of India (NDLI)
(https://ndl.iitkgp.ac.in/) designed by the Indian Institute of
Technology Kharagpur to provide a single search window to a wide variety of digital
resources present in different knowledge repositories in India and abroad. It indexes
more than 17 million resources of more than 60 types including articles, scholarly as
well as popular publications, theses, question papers, presentations, datasets, audio
and video lectures, albums, historical records, law judgments and patents. Unlike
many national libraries, NDLI indexes a multitude of scholarly works like research
papers; so we include it under SDL.
As more and more digital contents get added to SDLs, it becomes difficult to
explore their contents effectively. Most of them offer simple search, browse and in
some cases, recommendation services. Although keyword and faceted search are
common, semantic search (e.g., using machine learning (Xiong et al., 2017)) is
gradually being introduced in digital libraries. Multimodal search, though rare today,
is also likely to become prevalent in near future.
Given the issues of big data, it is important to have sophisticated tools beyond
search to explore the repositories and discover knowledge from them. We identify
the following three categories of emerging applications that we believe can enhance
the usability of SDLs: information visualization, impact measurement, and
recommendation services.
Recommendation services are present in some digital libraries like IEEE Xplore
and Scopus; they are absent in arXiv and many other libraries. They offer a
personalized discovery mechanism for users guiding them in the wilderness of big
data. Hence, they are very useful and are likely to be more popular in future.
Measuring the impact of different venues and using them to rank the search results is
also important as the number of venues is growing very fast. It helps users identify
the results published in the top-tier venues and hence, presumably, the most
significant ones. Although many commercial search engines like Google Scholar
display citation data, they are hardly found in SDLs like NDLI. For large digital
libraries, it is beneficial if users can visually examine the contents at a certain level
of abstraction. This helps them understand whether the source will satisfy her
information needs. Again, most SDLs like IEEE Xplore, arXiv and NDLI do not
support it. Thus, these applications are not widely implemented despite their
potential benefit. There is also a lack of consensus on the best approach to each of
them. As big scholarly data make their presence more prominent, they will become
ever more important. These three categories capture a large gamut of potential tools
for SDLs. We survey these applications and highlight the existing issues in them.
Towards the end of the chapter, we briefly describe the challenges in incorporating
these services in NDLI.
2 LARGE SCHOLARLY DIGITAL LIBRARIES
Nowadays, SDLs are quite common and range from focused, domain-specific ones
to world-scale multi-disciplinary repositories. Well known SDLs in Computer
Science are the ACM Digital Library (https://dl.acm.org/) and CiteSeerX
(http://citeseerx.ist.psu.edu/index). IEEE Xplore covers Computer
Science, Electrical and Electronics Engineering while PubMed Central archives
biomedical and life science journal literature. ScienceDirect
(https://www.sciencedirect.com/) and arXiv are multi-disciplinary
databases (spanning technology, science and social science). Most of them allow free
access to metadata like title, authors, and abstract while restricting full text access to
subscribers only. In fact, many of these libraries like ACM Digital Library index a
large number of resources but host only a subset of them; the remaining resources
come from related digital repositories.
Scholarly search: A digital library is typically equipped with a search engine so
that users can explore its contents. In addition, there are web-scale academic search
engines like Google Scholar (https://scholar.google.co.in/), Microsoft
Academic (https://academic.microsoft.com/)
and Semantic Scholar
(https://www.semanticscholar.org/)
that index scholarly works. Anurag
Acharya, the key inventor of Google Scholar, defined scholarly works as,
“‘Scholarly’ is what everybody else in the scholarly field considers scholarly. It
sounds like a recursive definition but it does settle down.” (Noorden, 2014c). There
is no silver bullet to hit it but one can start with a paper known for its scholarly
content (as judged by experts), collect the papers it cites and repeat the process.
These engines provide powerful means to discover scholarly content from anywhere
on the Web. Hence, they often serve as the gateway to more specific SDLs.
However, the metadata and content furnished by search engines are usually more
noisy than those available from specific publishers or libraries. Academic search is
usually classified as either navigational (where the user is looking for a specific
scholarly document; she usually uses precise metadata like digital identifiers such as
DOI or ISBN, or full title or author name with title or author name with venue/year,
etc.) or informational (where the user is seeking some information in the library)
(Khabsa et al., 2016). A third kind - transactional query - is also possible in an SDL,
though it is less commonplace. This term is used in taxonomy of web search to
indicate queries in which the user’s intent is to perform some web-mediated activity
like search for best sites to download music or buy gadgets (Broder, 2002). If an
SDL is linked to multiple book sellers or to physical libraries that lend books, users
might issue transactional queries, too. The SDL should then display the various sites
from where the user can borrow or buy the desired book. However, at the time, this
is not so common. To satisfy users’ needs, a scholarly search engine uses lexical
search (where keywords taken as free text or as values of specific metadata fields are
matched against the document corpora) or semantic search (where the search engine
tries to infer the intent of the query and retrieve results accordingly). Search engines
might construct elaborate linked data graph and knowledge graph offline to make
semantic search possible. Although many national libraries (like Library of Congress
USA, British Library, National Library of France and Europeana) do maintain linked
data (Hallo et al., 2016)
and some of them support SPARQL queries, semantic search
is not frequently seen in SDLs (e.g., in most of the technology SDLs mentioned
earlier).
Academic social networks: Closely related to SDLs are academic social networks
like ResearchGate (https://researchgate.net), Academia.edu
(https://www.academia.edu/)
and Mendeley
(https://www.mendeley.com/). They allow researchers to create personal
profiles, upload papers and related artifacts (like datasets, presentations, and code),
interact with other researchers, and track the influence (e.g., citations, downloads,
views, etc.) of their papers. These initiatives encourage unrestricted deliberations
among scientists (which might create an alternative to peer review) and sharing
papers openly (which helps many researchers cross over the paywalls in
subscription-based SDLs) (Noorden, 2014b). SDLs might find it profitable to
integrate with them or at least borrow some of their features on collaborative
interaction.
Curating content: While SDLs and academic social networks normally require
maintainers to manually upload papers and other artifacts, some SDLs like
CiteSeerX and Web-scale academic search engines typically deploy web crawlers to
harvest scholarly resources from the Web. Most articles are in Portable Document
Format (PDF). Usually SDLs, search engines and social networks extract metadata
and citations from the articles so that users can easily access them without viewing
the entire paper (Wu et al., 2015). Some of them also parse the articles for more
specific components like mathematical equations, chemical formulae, figures, tables,
algorithms, etc. using rule-based or machine learning techniques. They form the
basis for higher level representations and analytics on these works.
3 EMERGING APPLICATIONS FOR LARGE SDLS
Several applications and analytic tools can be built on top of large SDLs. They help
to understand the patterns hidden in the voluminous scholarly data stored in the
repositories. We look at some of the popular and emerging applications below that
we believe are critical to the usability of a large SDL.
3.1 Information Visualization
Search and retrieval form the most important functionality offered by a digital library
for without a proper discovery service, the scholarly works will remain inaccessible
to the users. Keyword-based search is most common although most SDLs allow
faceted search that applies specific filters on the metadata. Clustering and
visualization of search results can help users navigate through them easily. When the
user’s information needs are less well-defined, she would prefer to browse the
library. Again, visualizations can aid this process enormously (Greene et al., 2000).
Visualization tools must consider what data source to use, what analysis (e.g.,
discourse analysis, text summarization, etc.), what kind of visualization task to
execute (refers to lower level tasks like categorization, overview, etc. on the analysis
result), what kind of graphics to use (e.g., scatter plot, maps, etc.) and how many
dimensions to use (2D or 3D) (Kucher and Kerren, 2015).
In an SDL, visualization can be used at different levels, namely, collection level,
document level and intra-document level (Herrmannova and Knoth, 2012).
Attributes of a collection can be visualized using tag clouds (Hassan-Montero and
Herrero-Solana, 2006), scatter plots (Marks et al., 2005), and line graphs (Michel et
al., 2011). It can provide overviews of the topics or, simply, the top words in the
collection, or the temporal variation in topics/words in the collection. It can also
show thematic partitions of the collection with each partition containing documents
of a specific theme.
Fig. 1. The above map developed by Infobaleen (wikimap.infobaleen.com) shows
Wikipedia topics; each box is a cluster of related Wikipedia pages (Retrieved on December 31,
2017).
Figure 1 shows an information map of Wikipedia; each box is a cluster of related
Wikipedia pages. Similar visualizations could be designed for collections in an SDL.
Collection level views are very useful in digital humanities, for example, to track the
variation in the frequency of a word’s use (Michel et al., 2010)
or trace the temporal
change of a word’s linguistic sense. At the second level, one is more interested to see
how the documents connect with each other, e.g., through correlation of topics, co-
citation or co-authorship. The third level focuses on visualizing the contents of a
single document at some level of abstraction. For example, it may show the topics in
it or build a semantic model (like a knowledge graph) of the document (Ronzano and
Saggion, 2015).
We feel visualizations should support several zoom levels and furnish precise
numerical statistics whenever requested by the user. Numerous techniques available
for text mining, generation of ontology and linked data, and information
visualization can be used to design graphical representations of data within an SDL
(Liu et al., 2014). However, visualization methods have been traditionally tested on
small data sources and may face scalability issues with big data, especially if
computation needs to be done in real-time. They are hardly found in real-world
SDLs. Efforts should also be made to visualize other resources like datasets,
question banks, lectures, etc. available in SDLs (like NDLI) and present them in an
integrated canvas with text documents.
3.2 Impact Measurement
Researchers have spent huge efforts to measuring the impact of scholarly
publications (Mingers and Leydesdorff, 2015). It is an important activity as it helps
to identify significant works and devote even further research on their proposed
ideas. It also helps to identify the influential leaders in different branches of
knowledge. This information can be used by different stakeholders like funding
agencies for dispensation of research grants, academic and research institutes in
deciding promotions for their staff, authors in identifying the most influential
publication venues, researchers in understanding the state-of-the-art, accreditation
bodies in determining relative positions of research institutes, and academic search
engines in ranking retrieval results. Impact measurement has been typically at the
following levels.
Fig. 2. Google Scholar shows citation count for each article; it is also considered when ranking
retrieved results. (Retrieved December 31, 2017).
Articles: The most common metric in this area is citation count. It is widely
accepted as an indicator of the influence of a scholarly article. For example, as
shown in Figure 2, Google Scholar shows citation counts for every scholarly article.
It is also shown in CiteSeerX, IEEE Xplore and ACM Digital Library. However, it
has many limitations including inflation by self-citations, giving equal importance to
all citations, allowing obvious benefits to old papers and inability to account for
variations in citation patterns across disciplines. So, several alternatives have been
suggested that consider the nature (for example, appreciation, criticism, etc.) of the
citations, consider the age of the paper, and adjust for discipline-specific citation
patterns. Researchers have also tried to predict the citation counts of new
publications (Yan et al., 2011).
The increasing presence of scholarly works on social networks and other online
platforms has motivated the design of altmetrics to measure research impact (Priem,
2010). They are alternatives to the traditional citation-based metrics discussed above.
They can be roughly divided into (1) usage (i.e., views or downloads of the paper),
(2) captures (e.g., bookmarks), (3) mentions (e.g., comments, blog posts, Wikipedia
articles, reviews), (4) social media presence (e.g., tweets, likes, shares), and (5)
citations (e.g., in Web of Science) (Melero, 2015). Their proponents argue that they
can be accumulated fast (unlike the huge gestation time for citations), capture
influence of a paper from a rich diversity of sources, apply to many forms of
publishing like papers, datasets, code, experimental designs, etc., are amenable to
algorithmic processing and are more robust to fraudulent manipulation. They are
already visible in BioMed Central and Scopus. See Figure 3.
Fig. 3. Altmetrics for an article in Scopus (Retrieved December 31, 2017).
Authors: The most common metric here is the h-index. An author’s h-index is h if
she has h publications that have been cited at least h times each. Thus, it overlooks
the citation counts of her very highly cited articles. The g-index improvises on it.
Given a set of articles of an author arranged in decreasing order of their citation
counts, the g-index is the (unique) largest number such that the top g articles
received (together) at least g
2
citations.
Venues: Journals have long been the established as the authoritative venues to
archive scholarly work. Several metrics to measure their impact have been proposed.
Examples include Journal Impact Factor (defined for journals listed in Journal
Citation Reports (subset of Web of Science); JIF of a journal for a given year is the
number of citations, received in that year, of articles published in that journal during
the two preceding years, divided by the total number of articles published in that
journal during the two preceding years), Scimago Journal Rank (SJR) (based on
Scopus database, SJR of a journal is the ratio of the number of weighted citations
received in a year to the total number of publications in the last 3 years; the weight is
related to the importance of the source from where the citation comes), Source-
Normalized Impact per Paper (SNIP) (based on Scopus, SNIP of a journal for a year
divides the journal’s citation count per paper by the citation potential in its subject
area; it compensates for the skew in citation counts across disciplines), and the
recently proposed CiteScore (based on Scopus database, CiteScore of a journal for a
year is the ratio of the number of citations received in the year to the number of all
documents published in the journal in the preceding three years; here, computation is
done over all documents indexed by Scopus) (Elsevier, n.d.). Several other metrics
are proposed in the literature (Bradshaw and Brook, 2016). There are hardly any
popular ranking metrics for a conference series although variants of h-index are
sometimes used (modeling the conference as an author) (Google Scholar, n.d.).
Since the above metrics are not free of criticism and depend on the volume of the
indexed literature, SDLs should ideally display multiple metrics. But that is
practically difficult due to the huge data and computation needed.
3.3 Recommendation Services
The extraordinary rate at which scholarly literature is being produced baffles even
the most avid of readers. It is almost impossible to keep aware of all publications
even in a focused sub-field. Hence, digital libraries are increasingly providing
recommendation services that suggest scholarly resources to the users based on their
personal profiles. Figure 4 shows how Elsevier shows recommendations when a user
looks at a paper. Additionally, recommendations on potential author collaborations,
reviewers, publication venues, queries, courses, and tags for articles are useful in
digital libraries that index research works and are frequently used by active
researchers. Recommendations can be personalized for individual users or groups of
users (e.g., researchers in a specific domain).
Primarily, three techniques have been used to produce research paper
recommendations: content-based filtering (CBF) (a user is recommended an item
similar to the one she liked earlier), collaborative filtering (CF) (based on the notion
that people who agreed in their evaluation of certain items in the past will probably
agree again in the future) and hybrid (uses features of CBF and CF). CBF methods
typically use words or n-grams from article title, keywords, abstract and full text and
compare them across articles using techniques like vector space models (commonly
using term frequency-inverse document frequency as a weighting scheme and cosine
similarity to infer relatedness), probabilistic topic models (like Latent Dirichlet
Allocation), etc. Content analysis and comparison can be improved with external
knowledge bases. More generally, they may also use author names and co-citation
statistics. CBF is the most common method, possibly, due to lack of sufficient
transaction logs (Beel et al., 2016). In this comprehensive review (Beel et al., 2016),
Beel et al. point to the lack of systematic comparison of different research paper
recommender system approaches proposed in the literature. This is due to diversity
in evaluation methods (metrics; strategy, i.e., online or offline), low number of
participants in user studies, lack of information on runtimes and scalability,
heterogeneity in chosen baselines, diversity and unavailability of datasets and last,
but not the least, sparse information on the algorithms used. Hence, it is unclear if
CBF is better than CF or the other way round and which papers have advanced the
best designs. The recent paper (Beel and Dinesh, 2017) elaborates more on the
dismal scenario and encourages researchers to implement and evaluate recommender
systems in the real world and open source their projects. A promising step in this
direction is the Babel project (http://babel.eigenfactor.org/). It is,
though, clear that a scholarly recommender system should deliver provide relevant
recommendations in a timely manner; the recommendations displayed should be
small in number (say, at most five, to avoid information overload) and possess
diversity (e.g., showing related videos and datasets with articles). Recommendation
systems should also succinctly display the rationale behind their recommendations
and allow users to explicitly provide feedback for improvement. It may be noted here
that great care is needed in design of recommendation services; otherwise there is a
risk of trapping the user in a biased zone created by algorithms and artificial
intelligence. This will stifle innovation and discovery instead of encouraging it.
More research is also needed in other types of academic recommenders that we have
briefly mentioned at the beginning of this subsection.
Fig. 4. Recommendations (on right pane) in Elsevier ScienceDirect digital library (shown when
a user selects an article) (Retrieved December 31, 2017).
4 NDLI AND THE SCOPE OF SCHOLARLY APPLICATIONS
NDLI is a metadata hub that connects to different content holders like institutional
repositories and publisher sites. It caters to academic and non-academic readers, a
wide academic age group – from kindergarten to research scholars – and a host of
different languages. It indexes millions of resources of various types including
scholarly (like research papers, academic books, audio-visual lectures, etc.) and
popular publications (like children’s story books). However, in this section, we
restrict our analysis to scholarly resources.
Fig. 5. Search in NDLI; various filters are shown on the left and right panes (Retrieved
December 31, 2017).
Currently NDLI provides a simple browse and search interface with a number of
filters on metadata attributes like author name, language, learning resource type, etc.
See Figure 5. However, it does not contain the advanced services discussed above.
Although it may not be justifiable to compare an indigenous initiative like NDLI
with other advanced international scholarly platforms, it is worthwhile to explore the
scope of including these services in NDLI in future.
NDLI catalogues millions of resources from various sources like institutional
digital repositories from all over India and digital libraries maintained by
international publishers. Given this size and diversity, it would be useful to provide
visualizations of the various sources so that users can get a graphical overview of
their contents.
NDLI does not show any impact figures against publications, authors or venues.
This is indeed difficult because of copyright restrictions; in many cases, NDLI is not
allowed to parse full text or extract references from them. One alternative is to
collaborate with external indexing services like the Web of Science or open citation
databases (https://i4oc.org/).
In regard to recommendation, the overwhelming quantity and diversity of
resources in NDLI hold the promise of rich recommendations. Recommendations
using CBF method can be easily provided by analyzing bibliographic metadata.
However, in many cases metadata are sparse and noisy. Moreover, there are multiple
domains which should be used in clustering the resources before recommendations
are produced (West et al., 2016). Click-logs should also be analyzed to discover
library search behavior of users so that effective CF-based recommendations can be
added.
We emphasize that evaluating a digital library is a very complex task and requires
measuring several attributes (Sandusky, 2002; Fuhr et al., 2007). The above tools
alone do not produce a perfect digital library. But if it already has a rich content base
(as is true for NDLI), these tools can make it considerably more usable for
researchers.
5 CONCLUSION
We presented a brief review of tools found in an SDL. The exponential growth rate
of scholarly works makes them absolutely essential for a library user, whether she is
a casual reader or a dedicated researcher. Although a plethora of applications are
possible, we tried to restrict to three primary buckets, namely, information
visualization, impact measurement and recommendation services that, we believe,
are the most important. We have seen there are many approaches to implement each
application, each having its own pros and cons. But for each of them, the huge data
size results in considerable computational load. Hence, it is important to decide
whether to implement them as an offline or an online process (wherever there is a
choice). We also briefly discussed the scope of including them in NDLI which is
envisioned to revolutionize learning across all sections of Indian society.
ACKNOWLEDGEMENTS
This work is supported by Development of National Digital Library of India as a
National Knowledge Asset of the Nation sponsored by Ministry of Human Resource
Development, Government of India.
REFERENCES
Beel, J. and Dinesh, S. (2017). Real-world recommender systems for academia: The pain and gain in
building, operating, and researching them. In Proceedings of BIR@ECIR, pages 6–17.
Beel, J., Gipp, B., Langer, S., and Breitinger, C. (2016). Paper recommender systems: a literature
survey. International Journal on Digital Libraries, 17(4):305–338.
Bradshaw, C. J. and Brook, B. W. (2016). How to rank journals. PloS ONE, 11(3):e0149852.
Broder, A. (2002). A taxonomy of web search. In ACM SIGIR Forum, volume 36, pages 3–10.
ACM.
Cornell University. (Apr 16, 2018). Distinguishing scholarly from non-scholarly periodicals: A
checklist of criteria. Retrieved July 11, 2018, from
http://guides.library.cornell.edu/c.php?g=31867&p=201758.
Elsevier. (n.d.). Measuring a journal’s impact. Retrieved Jul 11, 2018, from
https://www.elsevier.com/authors/journal-authors/measuring-a-journals-impact.
Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., Klas, C.-P., Kovács, L.,
Landoni, M., Micsik, A., et al. (2007). Evaluation of digital libraries. International Journal on
Digital Libraries, 8(1):21–38.
Google Scholar. (n.d.). Retrieved July 11, 2018, from
https://scholar.google.co.in/citations?view_op=top_venues&hl=en&vq=eng_computatio
nallinguistics.
Greene, S., Marchionini, G., Plaisant, C., and Shneiderman, B. (2000). Previews and overviews in
digital libraries: Designing surrogates to support visual information seeking. Journal of the
American Society for Information Science, 51(4):380–393.
Hallo, M., Luján-Mora, S., Maté, A., and Trujillo, J. (2016). Current state of linked data in digital
libraries. Journal of Information Science, 42(2):117–127.
Hassan-Montero, Y. and Herrero-Solana, V. (2006). Improving tag-clouds as visual information
retrieval interfaces. In Proceedings of the International Conference on Multidisciplinary
Information Sciences and Technologies, pages 25–28.
Herrmannova, D. and Knoth, P. (2012). Visual search for supporting content exploration in large
document collections. D-Lib Magazine, 18(7):8.
Khabsa, M., Wu, Z., and Giles, C. L. (2016). Towards better understanding of academic search. In
Proceedings of the IEEE/ACM Joint Conference on Digital Libraries (JCDL), pages 111–114.
IEEE.
Kucher, K. and Kerren, A. (2015). Text visualization techniques: Taxonomy, visual survey, and
community insights. In Proceedings of IEEE Pacific Visualization Symposium (PacificVis),
pages 117–121. IEEE.
Liu, S., Cui, W., Wu, Y., and Liu, M. (2014). A survey on information visualization: recent advances
and challenges. The Visual Computer, 30(12):1373–1393.
Marks, L., Hussell, J. A., McMahon, T. M., and Luce, R. E. (2005). ActiveGraph: A digital library
visualization tool. International Journal on Digital Libraries, 5(1):57–69.
Melero, R. (2015). Altmetrics–a complement to conventional metrics. Biochemia Medica,
25(2):152–160.
Michel, J.-B., Shen, Y. K., Aiden, A. P., Veres, A., Gray, M. K., Pickett, J. P., Hoiberg, D., Clancy,
D., Norvig, P., Orwant, J., et al. (2011). Quantitative analysis of culture using millions of
digitized books. Science, 331:176–182.
Mingers, J. and Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European
Journal of Operational Research, 246(1):1–19.
Noorden, R, V. (2014a). Global scientific output doubles every nine years. Retrieved from
http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html.
Noorden, R, V. (2014b). Online collaboration: Scientists and the social network. Nature news,
512:126-129. Retrieved from https://www.nature.com/news/online-collaboration-scientists-and-
the-social-network-1.15711.
Noorden, R, V. (2014c). Google Scholar pioneer on search engine’s future. Retrieved from
https://www.nature.com/news/google-scholar-pioneer-on-search-engine-s-future-1.16269.
Priem, J., Taraborelli, D., Groth, P., Neylon, C. (2010), Altmetrics: A manifesto, 26 October
2010. Retrieved from http://altmetrics.org/manifesto.
Ronzano, F. and Saggion, H. (2015). Dr. Inventor framework: Extracting structured information
from scientific publications. In Proceedings of the International Conference on Discovery
Science, pages 209–220. Springer.
Sandusky, R. J. (2002). Digital library attributes: Framing usability research. In Proceedings of
Workshop on Usability of Digital Libraries at JCDL, volume 2, pages 35–38.
West, J. D., Wesley-Smith, I., and Bergstrom, C. T. (2016). A recommendation system based on
hierarchical clustering of an article-level citation network. IEEE Transactions on Big Data,
2(2):113–123.
Wu, J., Williams, K. M., Chen, H.-H., Khabsa, M., Caragea, C., Tuarob, S., Ororbia, A. G., Jordan,
D., Mitra, P., and Giles, C. L. (2015). CiteSeerX: AI in a digital library search engine. AI
Magazine, 36(3):35–48.
Xia, F., Wang, W., Bekele, T. M., and Liu, H. (2017). Big scholarly data: A survey. IEEE
Transactions on Big Data, 3(1):18–35.
Xiong, C., Power, R., and Callan, J. (2017). Explicit semantic ranking for academic search via
knowledge graph embedding. In Proceedings of the 26th International Conference on World
Wide Web, pages 1271–1279.
Yan, R., Tang, J., Liu, X., Shan, D., and Li, X. (2011). Citation count prediction: learning to estimate
future citations for literature. In Proceedings of the 20th ACM International Conference on
Information and Knowledge Management, pages 1247–1252. ACM.
