NLP for Student and Teacher: Concept for an AI based Information Literacy Tutoring System

Abstract

We present the concept of an intelligent tutoring system which combines web search for learning purposes and state-of-the-art natural language processing techniques. Our concept is described for the case of teaching information literacy, but has the potential to be applied to other courses or for independent acquisition of knowledge through web search. The concept supports both, students and teachers. Furthermore, the approach integrates issues like AI explainability, privacy of student information, assessment of the quality of retrieved information and automatic grading of student performance.

Full text

NLP for Student and Teacher: Concept for an AI based
Information Literacy Tutoring System
P. Libbrechta, T. Declerckb, T. Schlippea, T. Mandlcand D. Schinerd
aIUBH Fernstudium, Bad Reichenhall, Germany
bDFKI GmbH, Saarbrücken, Germany
cUniversity of Hildesheim, Hildesheim, Germany
dDIPF | Leibniz Institute for Educational Research and Information, Frankfurt, Germany
Abstract
We present the concept of an intelligent tutoring system which combines web search for learning purposes and state-of-the-
art natural language processing techniques. Our concept is described for the case of teaching information literacy, but has
the potential to be applied to other courses or for independent acquisition of knowledge through web search. The concept
supports both, students and teachers. Furthermore, the approach integrates issues like AI explainability, privacy of student
information, assessment of the quality of retrieved information and automatic grading of student performance.
1. Motivation
Information literacy is a core skill for the digital age.
In modern education and work environments it is of
growing importance as knowledge work is increas-
ingly based on large and rapidly changing knowledge
sources. Search and organization of knowledge is a
constant requirement. Higher education teaches in-
formation literacy sometimes in dedicated courses and
often only within another course. Studies show that
the level is low: E.g. students have diculties in us-
ing operators in search terms, organize literature and
tend not to know appropriate sources to nd scientic
literature.
The potential of Articial Intelligence (AI) in higher
education still needs to be explored and innovative
applications need to be developed. Can computers
support teaching sta in coaching information com-
petency? – The research area “AI in Education” ad-
dresses the application and evaluation of AI meth-
ods in the context of education and training. One
of the main focuses of this research is to analyze
and improve teaching and learning processes. On the
one hand, deep learning – learning in multi-layered
(“deep”) articial neural networks – has become a cen-
tral component of AI research and numerous libraries
or frameworks1have been created that simplify the
Proceedings of the CIKM 2020 Workshops, October 19–20, Galway,
Ireland
email: p.libbrecht@iubh-fernstudium.de ( P. Libbrecht);
declerck@dfki.de ( T. Declerck); t.schlippe@iubh.de ( T. Schlippe);
mandl@uni-hildesheim.de ( T. Mandl); schiner@dipf.de (
D. Schiner)
orcid:
©2020 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
1To name a few: TensorFlow,Keras ,Cae,PyTorch
work and support the creation of own experiments.
On the other hand, many educational institutions al-
ready conduct their courses, exercises, and examina-
tions online. This means that student assessments are
already available in digital, machine-readable form,
oering a wide range of analysis options. Focusing
on information literacy, a course typically consists of
teaching material in text form and the course partici-
pants themselves practice information skills and gen-
erate text in online research and essays. However, the
evaluation of free texts such as essays, references and
research methodology still requires intensive manual
work.
Consequently, we deal with the question which
methods of Natural Language Processing (NLP) can
support coaching of information competency and how
they can be applied for the teacher, and for the stu-
dent. The focus is on the combination of various deep
learning approaches to automatically help students to
accelerate the learning process by automatic feedback,
but also to support teachers by pre-evaluating free text
and suggesting corresponding scores or grades.
2. Related Work
Lazonder [1] showed that searching and talking about
the search has led to positive learning eects. In a
similar fashion, providing feedback and suggestions
around a search activity can support the reection on
search tools’ usage, on one’s information needs and
on the goals of the task at hand. In his keynote at
ECIR 20202C. Shah envisioned the next decade of re-
search in search and recommendation where modeling
2https://ecir2020.org/keynote-speakers/

the tasks is central to raise the quality of the results.
Dened tasks around the learning of information liter-
acy are a good example of context where recommen-
dations can be made more relevant to the process. In-
telligent Tutoring Systems (ITS) follow a long tradi-
tion of environments where AI supports learning. The
most widespread didactic situation where ITS has been
employed, is as exercises where a direct feedback (e.g.
in form of a score or recommendation) is oered fol-
lowing interactions with a dedicated system. Multiple
example intelligent tutoring systems exist and many
follow the model of [2]. Our research aims to observe
the work of the students instead of requiring exercise
specic actions.
State-of-the-art research and the basis for the devel-
opment of an NLP system to coach information com-
petency include sentiment analysis [3], topic identi-
cation [4], named entity recognition [5], text sum-
marization [6], word sense disambiguation [7] and in-
formation retrieval [8]. A major scientic challenge
is the explainability of system outputs [9]. The NLP
methods can be combined with knowledge graphs to
include ontology-based knowledge coding in the pro-
cesses [10]. This can be enhanced by including visu-
alizations that represent both the inputs of the stu-
dent and the results of the NLP analysis. Lachner [11]
shows that graph representations can support the un-
derstanding of a topic. To represent ontologies or com-
plex topics, knowledge graphs such as [12] help to
identify context.
For the processing in deep learning architectures,
sequences of words are encoded into vector spaces
in order to perform computations in neural networks.
Tools for text vectorization are Word2Vec , GloVe [13]
or fastText [14]. The concept of the skip-thought
vectors [15], universal sentence encoder (USE) and
bidirectional encoder representations from transform-
ers (BERT) [16] are methods also supporting sentence
embeddings in the semantic vector space. [17] in-
vestigates and compares state-of-the-art deep learn-
ing techniques for automatic short answer grading.
Their experiments demonstrate that systems based on
BERT [16] performed best for English and German. On
their German data set they report a Mean Average Er-
ror of 1.2 points, i.e. 31% of the student answers are
correctly graded and in 40% the system deviates by 1
out of 10 points.
3. Information Literacy Courses
The term Information Literacy is often used synony-
mous with Media Literacy. According to the UNESCO
it “constitutes a composite set of knowledge, skills,
attitudes, competencies and practices that allow ef-
fectively access, analyze, critically evaluate, interpret,
use, create and disseminate information and media
products with the use of existing means and tools on a
creative, legal and ethical basis. It is an integral part of
so-called “21st century skills” or “transversal compe-
tencies”” [18]. In higher education, these information
skills are highly relevant for students. Nevertheless,
the information literacy of students is often measured
as low [19]. Courses on basic scientic work cover sev-
eral domains of information literacy. Often, there is a
strong focus on searching skills, correct citing and as-
sembling short abstracts based on scientic texts. The
practice of teaching information literacy skills has not
developed much towards digital formats. Some open
online courses exist [20], but there is no use of AI tools
yet.
An example is the ILO-MOOC (informationliter-
acy.eu). It allows to study in a self-paced manner. The
feedback for students is show right or wrong after an-
swering multiple choice questions. Another example
is at IUBH University where bachelor students with
a diverse background are trained on the basics of sci-
entic work. While the focus of the assignments is in
the production of written texts, it involves all aspects
of information literacy. The course is made for both
remote and on-site attendance and involves various
communication channels, many of them happening on
the web. The resulting competencies expect an inde-
pendent and self-condent scientic work which may
be strongly supported by an automatic evaluation.
4. Proposed System Architecture
Our concept proposes an integration within the web
activities of the learner attending an assignment task
which includes searching, reading, evaluating, and
writing: Using JavaScript or web extensions, the text
and timestamps of the search results, of the viewed
publications, and of the input text can be used as fea-
tures for the NLP models. Based on the assignment’s
objective, the feature vectors generated from the stu-
dent’s behavior and text is processed by our NLP mod-
els. The models were trained by annotated text data
from previous course members (model solution, al-
ready graded works, other annotated works) to gen-
erate textual feedback.
The concept comprises several tasks for which sup-
port for students can be provided. For the sake of
brevity we illustrate two of them:
A core task in scientic work and, thus, in teaching

information literacy is web search. Students are of-
ten required to search for documents fullling certain
requirements e.g. within a closed collection of docu-
ments. An AI system observes the search terms input
by the student and compares the strategies to identi-
ed objectives. The system tracks the actions of the
students regarding search terms, observed documents,
headers and time spent. It then suggests the most
appropriate steps toward reaching better results. In
the example of searching in a closed collection with a
pre-dened goal, suggestions for further search terms
leading to relevant documents can be made.Text vec-
torization and a deep learning model-based classi-
cation can be used for keyword extraction [21]. As
such, the student learns in the direct interaction with
a search system and improves skills based on previous
activities by providing automatic feedback.
Another exemplary task in teaching information lit-
eracy is related to academic writing. Students are
asked to assemble a short summary and synthesis of
several papers. The system supports them in analyz-
ing the writing, recognizing the parts in a certain pa-
per, checking whether the short summary is adequate
and without plagiarism. Siamese neural networks can
be used to detect similarities there [22]. The system
also uses NLP to analyze the coherence of the text.
Here an AI based system also gives context-dependent
suggestions on how to improve the text. The sugges-
tions provided can refer to documents, showing a title,
a time when the student saw it, and a link to the docu-
ment as last accessed. This is eective for the learner
as the reading is kept in memory. Such support within
the writing process can help students more than a the-
oretical unit on academic writing.
In our suggested AI based information literacy tu-
toring system, word sequences in search terms, re-
trieved documents, and reference documents are en-
coded into vector spaces in order to perform compu-
tations in deep learning architectures. A ne-tuning
architecture, such as BERT, which has proven itself in
many NLP tasks, provides the basis of our system: It is
based on a pre-trained deep learning model, which —
supplemented by a linear regression layer — is adapted
to our specic tasks, e.g. grading short summaries or
retrieved documents from the web search, and the pa-
rameters of the embeddings are tuned accordingly. A
data set with labeled and graded documents and sum-
maries from old information literacy courses serves to
optimize the model for predicting scores.
To achieve a steeper learning curve and to guaran-
tee explainability, we suggest two methods: (1) High-
lighting keywords and (2) displaying the condence
score of the system’s output. The keywords can be
retrieved by adapting the BERT ne-tuning architec-
ture to extract named entities as proposed in [23] and
[24]. The condence score can be retrieved by map-
ping the predicted scores to classes and output a vector
that contains a probability for each class.
Further feedback to both teachers and learners can
be given by the visualizations of individual states and
congurations of the system. Given the exemplary
tasks, the trial and error processes can be shown in
dierent paths along a timeline that can be shared by
the student with a teacher to allow for better feedback.
Also the categorization and identication of correct
steps are to be shown to the learner using clear visu-
alizations to help understanding the decision making.
The analysis of the evolving students’ work gath-
ers data that should not necessarily be shown to fel-
low students or teachers: The data is made of personal
trial and error processes and is to be considered as pri-
vate. Other content, e.g. from chat rooms and forums,
can be considered as public. It is adequate that a bot
can provide answers using information that all chat
members have seen (e.g. lecture scripts, assignments’,
posts). Similarly, submitted assignments’ text data can
be automatically graded based on existing information
such as earlier assignments or expert texts.
5. Conclusion and Future Work
We have described the architecture of an intelligent tu-
toring system which combines web search and natural
language processing techniques on the basis of infor-
mation competency. After implementing the system
and training the machine learning models, the system
needs to be evaluated and optimized for students and
teachers regarding usability and eciency for which
several courses exist. With the help of metrics, we re-
duce the error rate for training the models. Finally, we
intend to speedup the system. Throughout the imple-
mentation usability tests are repeatedly performed to
ensure the quality of the proposed system.
We plan to apply the architecture within several
courses, adjust the tasks to be automatically measur-
able and annotate corpora of articles so that an au-
tomatic evaluation yields productive feedback. Using
this system, we expect to answer the following ques-
tions: How to adequately capture the students’ activ-
ity, select information to store and evaluate it, and how
to oer support which is timely and relevant for the
learning process.

References
[1] A. Lazonder, Do two heads search better than
one? Eects of student collaboration on web
search behaviour and search outcomes, BJET 36
(2005) 465–475.
[2] K. VanLehn, The Behavior of Tutoring Systems,
I. J. Articial Intelligence in Education 16 (2006)
227–265.
[3] H. Liu, Sentiment Analysis of Citations Using
Word2vec, arXiv:1704.00177 (2017).
[4] M. Lamba, M. Margam, Metadata Tagging
of Library and Information Sciences Theses:
Shodhganga (2013-2017), 2018. doi:10.5281/
zenodo.1475795.
[5] X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, J. Li, A Uni-
ed MRC Framework for Named Entity Recogni-
tion, arXiv:1910.11476 (2019).
[6] A. Padmakumar, A. Saran, Unsupervised Text
Summarization Using Sentence Embeddings, in:
Technical Report, University of Texas at Austin,
2016.
[7] Y. Wang, M. Wang, H. Fujita, Word Sense Dis-
ambiguation: A comprehensive knowledge ex-
ploitation framework, Knowledge-Based Sys-
tems (2020).
[8] H. Zhang, G. Cormack, M. Grossman,
M. Smucker, Evaluating Sentence-Level
Relevance Feedback for High-Recall Information
Retrieval, Inf. Retr. J. (2020).
[9] H. Liu, Q. Yin, W. Wang, Towards Explainable
NLP: A Generative Explanation Framework for
Text Classication, in: A. Korhonen, D. Traum,
L. Màrquez (Eds.), ACL, 2019.
[10] D. Gromann, L. Anke, T. Declerck, Special Issue
on Semantic Deep Learning, Semantic Web 10
(2019) 815–822.
[11] A. Lachner, C. Neuburg, Learning by writing
explanations: Computer-based Feedback about
the Explanatory Cohesion Enhances Students’
Transfer, Instructional Science 47 (2019) 19–37.
[12] M. Jaradeh, A., Oelen, M. Prinz, M. Stocker,
S. Auer, Open research knowledge graph: A sys-
tem walkthrough, in: TPDL, 2019.
[13] J. Pennington, R. Socher, C. Manning, Glove:
Global Vectors for Word Representation., in:
EMNLP, 2014.
[14] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov,
Enriching Word Vectors with Subword Informa-
tion, Transactions of the Association for Compu-
tational Linguistics 5 (2017) 135–146.
[15] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel,
A. Torralba, R. Urtasun, S. Fidler, Skip-Thought
Vectors, in: NIPS, MIT Press, Cambridge, MA,
USA, 2015.
[16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,
BERT: Pre-training of Deep Bidirectional Trans-
formers for Language Understanding, in:
ACL:HLT, 2019.
[17] J. Sawatzki, T. Schlippe, M. Benner-Wickner,
Deep Learning Techniques for Automatic Short
Answer Grading, in: submitted to: COLING,
ACL, 2020.
[18] The Moscow Declaration on Media and Informa-
tion Literacy, 2020. https://iite.unesco.org/mil,
2020-05-02.
[19] A. Hebert, Information literacy skills of rst-
year Library and Information Science graduate
students: An exploratory study, Evidence Based
Library and Information Practice 13 (2018) 32–52.
[20] S. Nowrin, L. Robinson, D. Bawden, Multi-
lingual and Multi-cultural Information Literacy:
Perspectives, Models and Good Practice, Global
Knowledge, Memory and Communication (2019).
[21] Y. Zhang, M. Tuo, Q. Yin, L. Qi, X. Wang, T. Liu,
Keywords Extraction with Deep Neural Network
Model, Neurocomputing 383 (2020) 113 – 121.
[22] E. Hambi, F. Benabbou, A Multi-Level
Plagiarism-Detection-System Based on Deep
Learning Alg., in: IJCSNS, 2019.
[23] K. Hakala, S. Pyysalo, Biomedical Named En-
tity Recognition with Multilingual BERT, in:
BioNLP-OST, 2019.
[24] E. Taher, S. Hoseini, M. S., Beheshti-NER: Per-
sian named entity recognition Using BERT, in:
ICNLSP, 2019.