What Do Academic Users Really Want from an Adaptive
Martin Harrigan1, Miloš Kravčík2, Christina Steiner3, and Vincent Wade1
1Department of Computer Science, Trinity College Dublin, Ireland
2Open Universiteit Nederland, The Netherlands
3Department of Psychology, University of Graz, Austria
Abstract. When developing an Adaptive Learning System (ALS), users are generally consulted
(if at all) towards the end of the development cycle. This can limit users’ feedback to the charac-
teristics and idiosyncrasies of the system at hand. It can be diﬃcult to extrapolate principles and
requirements, common to all ALSs, that are rated highly by users. To address this problem, we
have elicited requirements from learners and teachers across several European academic institu-
tions through explorative, semi-structured interviews . The goal was to provide a methodology
and an appropriate set of questions for conducting such interviews and to capture the essential
requirements for the early iterations of an ALS design. In this paper we describe the methodol-
ogy we employed while preparing, conducting, and analyzing the interviews and we present our
ﬁndings along with objective and subjective analysis.
The development of an Adaptive Learning System (ALS) is a challenging task [2, 3]. There exist many
prototypical systems with domain-speciﬁc adaptive functionality. However, there is no established
strategy for incorporating adaptivity in a system. This makes the process of requirements elicitation
quite diﬃcult. To address this problem, we have collected and aggregated the needs of users involved
in higher education (learners and teachers) in a systematic form through interviews. Our approach is
to illustrate the concept of adaptivity during the interviews through a hypothetical scenario involving
a learner, a teacher (author and tutor), and a fully-functional ALS. A semi-structured interview allows
the interviewees to evaluate an ALS’s potential merits, short-comings and usefulness with respect to
their individual needs.
Prototypical ALSs are often assessed through user evaluations during or after the system devel-
opment stage [4, 5]. However, this can frame the user’s evaluation; they comment on what has been
developed and oﬀer criticisms. Our hypothetical scenario is intentionally vague to promote a ‘green
ﬁelds approach’. It is the intention of this work to involve the users before any design or development
commences and to later assess the utility of their input through user trials when a system is being
This paper is organized as follows. Section 2 details the requirements elicitation methodology.
Section 3 describes the interviews themselves. In particular, we discuss current usage of learning systems
(both adaptive and non-adaptive) and ratings of the various features and dimensions of adaptivity.
Section 4 analyzes the interviews subjectively by highlighting some of the pertinent and interesting
This work was performed within the EU FP7 GRAPPLE (Generic Responsive Adaptive Personalized Learn-
ing Environment) Project. The authors would like to acknowledge the help of the following people in orga-
nizing and conducting the interviews: Françoise Docq (Université Catholique de Louvain), Maurice Hendrix
(University of Warwick), Riccardo Mazza (Università della Svizzera Italiana), Luca Mazzola (Università
della Svizzera Italiana), Ekaterina Pechenezhskaya (Technische Universiteit Eindhoven), Bram Pellens (Vrije
Universiteit Brussel), Kees van der Sluijs (Technische Universiteit Eindhoven), and Dominique Verpoorten
(Open Universiteit Nederland).
suggestions made by the interviewees. An accompanying technical report1provides an expanded version
of the sections herein, including the full text of the interview summaries.
2 The Requirements Elicitation Methodology
Interviewees are ﬁrst divided into three groups: learners, teachers, and others. An interview guide
and protocol is produced and distributed to all interviewers to ensure consistency. The interviews are
documented in two forms: interview summaries (having a narrative character) and interview data
sheets (for quantitative and statistical analysis). The interview questions are both quantitative (closed
questions with a predeﬁned choice of answers) and qualitative (open-ended questions that try to gather
information in an unbiased manner).
Content analysis reduces the large body of text in the interview summaries and data sheets to a
condensed form with essential content. There are two approaches: quantitative content analysis employs
word frequencies to deduce a systematic, objective, and quantitative description of the communication
content; and qualitative content analysis analyzes the texts within their context of communication,
following content analytic rules and step by step models, without rash quantiﬁcation. A combination
of both preserves their respective advantages , thus resulting in a systematic analysis that is guided
by qualitative interpretation in order to get an in-depth understanding of the ideas and views of
the interviewees on the one hand, and quantitative data on the other . To handle the open-ended
questions, categories of answers are built using a combination of inductive category building, where
the categories are formulated a priori and characterized by the relevant aspects of analysis, and
deductive category building, where the categories are formulated a posteriori in terms of the gathered
material [6, 7].
Before conducting the interviews, a hypothetical scenario involving a learner, a tutor, a content
author, and a fully-functional ALS is distributed to the interviewees. The scenario illustrates typical
and possible usage of an ALS. It provides the interviewees with a basic understanding of adaptivity.
Respondents are encouraged to estimate the relevance of each use case to their own personal context
and work. The technical report provides an example of one such scenario. We followed the above
methodology when conducting the interviews reported below.
3 The Interviews
There were 27 interviews conducted in June 2008 across seven European institutions (see Table 1).
The sample size was predominantly due to the data collection instrument and the involved eﬀort.
Learner Teacher Other Total
Open Universiteit Nederland 2 6 2 10
Technische Universiteit Eindhoven 0 4 0 4
Trinity College Dublin 1 2 0 3
Università della Svizzera Italiana 1 2 0 3
Universität Graz 2 1 0 3
University of Warwick 1 1 0 2
Vrije Universiteit Brussel 1 1 0 2
Total 8 17 2 27
Table 1. Summary of the interviews.
3.1 Current Usage of Learning Systems
The ﬁrst section of the interview gauged the current usage of learning systems and ALSs by the
interviewees. We present each question in turn and summarize the results.
A1. Do you use any learning systems? Out of 27 interviewees, 25 were using or had used learning
systems. All of the teachers had experience with learning systems. Only two learners indicated that
they had no experience. Questions A2-A5 were answered by the 25 interviewees with experience; the
remaining questions, unless otherwise indicated, were answered by all 27.
A2. Which learning systems have you used? This was an open-ended question; we did not provide
a list of learning systems to choose from. In the case of customized or heavily modiﬁed systems, we
grouped these under the category ‘in-house’. Other than in-house systems, Moodle and Blackboard
were the most popular learning systems (see Table 2). We note that the most popular Open-Source
and commercial LMSs feature. This question also provided us with information as regards the number
of learning systems in use by each interviewee. On average, each interviewee used two learning systems
(mean = 2.04,s.d.= 1.26). Teachers indicated that they use signiﬁcantly more learning systems
(t(23) = 2.699,p= 0.013), with teachers listing on average 2.5 (s.d.= 2.47) systems and learners
listing on average 1.1 (s.d.= 0.9) learning systems.
Others (AHA!, ALEKS, Dokeos, Educativa, Ilias) 6
Table 2. The learning systems used by interviewees (in descending order by use).
A3. How often do you use a learning system? The majority of the teachers used learning systems daily
or once to several times a week, whereas learners used them less frequently.
A4. How long have you been using learning systems? The teachers had long-term experience in using
learning systems (13 had many years’ experience, 3 had one year’s experience, and 1 had several
months’ experience), whereas learners had considerably less (only 1 has many years’ experience, 2 has
one year’s experience, and 3 had several months’ experience).
A5. Do the learning systems you have used so far provide any adaptive features to users? The responses
to this question show that the majority of learning systems have no adaptive features (no = 15,
yes = 10). The weak support of adaptation by Open Source and commercial LMSs has been conﬁrmed
in the literature .
3.2 Adaptivity – Needs and Preferences
The second section of the interview focused more on adaptivity and the purposes and beneﬁts of an
ALS (whether the interviewee had previously used one or not).
B1. What do you think are the purposes or tasks for which an ALS is especially suited? Table 3
summarizes the results. The top two answers were individualized teaching and guided, individualized
learning. These can be considered the same, but from opposing viewpoints, i.e. the teachers’ and
Individualized Teaching 6
Guided and Individualized Learning 5
Details of Technical Material 4
Clearly Deﬁned Knowledge Domains 2
Identiﬁcation of Strengths and Weaknesses in a Learner 2
Procedural and Vocational Training 2
Table 3. The top seven purposes or tasks for which ALSs are especially suited (in descending order
by the number of interviewees who said so).
User Speciﬁcity 9
Relevant Learning Material 4
Learner Motivation 3
Avoids Information and Cognitive Overload 2
Table 4. The top seven beneﬁts of ALSs (in descending order by the number of interviewees who said
B2. What are the beneﬁts of using an ALS? Do you think adaptivity in a learning system brings added
value to the user? The results are summarized in Table 4.
B3/B4. I list features that are reported in the literature to function as sources of adaptation, i.e.
characteristics of the learner or environment that may be considered by an ALS when adapting to the
individual learner. Please indicate your opinion on the importance of adaptation to each of these
features on a scale from 1 to 10 (1 being unimportant and 10 being very important). The listed
features and the results are shown in Table 5. All adaptation criteria were judged quite important;
each criterion reached at least a mean importance of 5. The criteria judged to be the most important
were adaptation to learner knowledge (mean = 8.85,s.d.= 1.19) and adaptation to learning goals
and tasks (mean = 8.7,s.d.= 1.82). A correlation analysis showed that the judgment of learner
knowledge is highly correlated with learning goals and tasks (r= 0.606,p= 0.001), and features
medium correlations with language, learner qualiﬁcations, user role, background, and experience in the
hyperspace. The importance rating of learner knowledge was not correlated with any other criterion.
The least importantly judged aspects, although still characterized by a mean importance of about
5, were background (mean = 5.3,s.d.= 2.37), learner personality (mean = 5.07,s.d.= 2.37), and
experience in the hyperspace (mean = 5.0,s.d.= 2.56).
B5/B6. I list dimensions that can be the subject of adaptation, i.e. methods and techniques that may
be used for adapting the learning process to the individual learner. Please indicate your opinion on the
importance of each of these dimensions on a scale from 1 to 10 (1 being unimportant and 10 being very
important). The list of dimensions and the results are shown in Table 6. As was the case for the features
of adaptivity, all the dimensions have quite high ratings, with minimum means between 5 and 6. The
dimensions judged to be most important were learning activity selection (mean = 8.37,s.d.= 2.02)
and content selection (mean = 8.33,s.d.= 2.25) in general – and within this dimension, the techniques
of additional explanations (mean = 8.37,s.d.= 1.04) and prerequisite explanations (mean = 8.19,
s.d.= 1.98). Furthermore, adaptive testing (mean = 8.22,s.d.= 1.63) was considered very important.
The dimensions judged to be least important, but still featuring a medium mean importance score,
No. Min. Max. Mean S.D.
Learner Knowledge 26 6 10 8.85 1.190
Learning Goals and Tasks 27 4 10 8.70 1.815
Language 26 5 10 7.96 1.455
Platform 26 3 10 7.77 1.583
Interests 27 2 10 7.22 2.136
Learning and Cognitive Style 27 2 10 7.19 2.403
Learner Qualiﬁcations 26 3 10 7.15 1.974
User Role 27 1 10 7.00 2.370
Motivation 27 1 10 6.96 2.682
Learner Preferences 27 1 10 6.26 2.474
Location 27 1 10 6.04 2.361
Background 27 1 10 5.30 2.367
Learner Personality 27 1 8 5.07 2.368
Experience in Hyperspace 26 1 10 5.00 2.561
Table 5. Speciﬁc features of adaptivity as rated by the interviewees (in descending order by mean
were hiding (mean = 5.22,s.d.= 2.55) and service provision (mean = 5.85,s.d.= 2.71). Hiding is less
popular and desirable in comparison with other techniques within adaptive navigation support. The
learner is deprived of information in this way, which was explicitly criticized by some interviewees.
4 Analysis and Conclusions
The views of our interviewees, comprising learners, teachers and others (researchers and developers)
can be summarized as follows. They require an ALS that provides individualized teaching and learning.
In particular, it should be capable of providing details of technical material that cannot be covered
adequately in a class or lecture. They expect such a system to be eﬃcient with respect to the learners,
tutors and authors, by providing users with relevant learning material. Table 5 and Table 6 provide a
‘most-wanted’ list of speciﬁc features and dimensions of adaptivity as ordered by their mean ratings.
In addition, ALSs are considered particularly suited to well explored and structured content. How-
ever, this is only one part of what a learner needs to learn. They must also learn more abstract and
complex competencies, e.g. social and relational skills, creative problem solving (where the ‘correct’
or ‘best’ solution is possibly unknown), independent critical thinking, etc. The interviewees propose
some areas where an ALS can add value in the academic context: the acquisition of basic knowledge,
the acquisition of technical details that are too cumbersome to cover in lectures and classes, adaptive
testing of basic knowledge, and language skills. Many interviewees insist that learners should be made
aware of the adaptation; they should be able to set adaptation parameters and always feel in control.
There is also a potential conﬂict between a learner’s preferred learning style and an optimal learning
strategy. It appears to be a delicate trade-oﬀ between pleasing the learner and doing what’s best for
them from a pedagogical standpoint. The accompanying technical report draws some more subjective
conclusions from speciﬁc remarks and suggestions made by the interviewees.
 Goguen, J., Linde, C.: Techniques for Requirements Elicitation. In: Proceedings of IEEE Interna-
tional Symposium on Requirements Engineering (RE’93). (1993) 152–164
No. Min. Max. Mean S.D.
Learning Activity Selection 27 1 10 8.37 2.022
Content Selection 27 1 10 8.33 2.253
Additional Explanations 27 7 10 8.37 1.043
Prerequisite Explanations 27 1 10 8.19 1.981
Comparative Explanations 27 5 10 7.56 1.121
Explanation Variants 27 5 10 7.44 1.625
Sorting 27 1 10 7.26 2.177
Problem Solving Support 27 5 10 7.93 1.299
Intelligent Analysis of Solutions 27 5 10 7.74 1.631
Example-Based Problem Solving 27 3 10 7.67 1.687
Interactive Problem Solving Support 27 3 10 7.37 1.822
Assessment 27 1 10 7.89 2.082
Testing 27 3 10 8.22 1.625
Questions 27 1 10 6.52 2.376
Learner Model Matching 27 1 10 7.56 1.888
Collaboration Support 27 3 10 7.78 1.805
Intelligent Class Monitoring 27 6 10 7.70 0.953
Presentation 27 1 10 7.52 2.242
Multimedia Presentation 27 1 10 7.41 2.635
Text Presentation 27 1 10 6.81 1.882
Customization of the Interface 27 1 10 6.63 2.041
Navigation Support 27 1 10 7.33 2.760
Link Generation 27 1 10 7.56 2.225
Sorting 27 1 10 7.04 2.488
Link Annotation 27 1 10 7.00 2.000
Map Annotation 27 1 10 6.96 2.244
Direct Guidance 27 1 10 6.70 2.267
Hiding 27 1 10 5.22 2.547
Service Provision 27 1 10 5.85 2.713
Table 6. Speciﬁc dimensions of adaptivity as rated by the interviewees (in descending order by category
mean ratings (bold terms) and then individual mean ratings).
 Frosch-Wilke, D., Sanchez-Alonso, S.: Composing Adaptive Learning Systems. In: Proceedings
of the 6th IEEE International Conference on Advanced Learning Technologies (ICALT’06), IEEE
Computer Society (2006) 360–362
 Tseng, J., Chu, H., Hwang, G., Tsai, C.: Development of an Adaptive Learning System with Two
Sources of Personalization Information. Computers and Education 51(2) (2008) 776–786
 Ortigosa, A., Carro, R.: The Continuous Empirical Evaluation Approach: Evaluating Adaptive
Web-Based Courses. In Brusilovsky, P., Corbett, A., de Rosis, F., eds.: Proceedings of the 9th
International Conference on User Modeling (UM’03), Springer (2003) 146
 Paramythis, A., Weibelzahl, S.: A Decomposition Model for the Layered Evaluation of Interac-
tive Adaptive Systems. In Ardissono, L., Brna, P., Mitrovic, A., eds.: Proceedings of the 10th
International Conference on User Modeling (UM’05), Springer (2005) 438–442
 Mayring, P.: Qualitative Content Analysis. Forum Qualitative Sozialforschung / Forum: Qualitative
Social Research 1(2) (2000)
 Schilling, J.: On the Pragmatics of Qualitative Assessment. European Journal of Psychological
Assessment 22(1) (2006) 28–37
 Hauger, D., Köck, M.: State of the Art of Adaptivity in E-Learning Platforms. In: Workshop
Adaptivität und Benutzermodellierung in interactiven Systemen (ABIS’07). (2007)