Content uploaded by Dietrich Albert
Author content
All content in this area was uploaded by Dietrich Albert on Oct 01, 2014
Content may be subject to copyright.
A preview of the PDF is not available
Reflections on the Evaluation of Adaptive Learning Technologies
Abstract
The key characteristic of adaptive learning technologies implies specific challenges for evaluation. Despite an increasing awareness of the need for sound evaluations, the current evaluation practice still suggests considerable weaknesses in the approaches applied and calls for further research on appropriate and sophisticated evaluation methodologies. This paper aims at further advancing the maturity of evaluation approaches for adaptive (learning) technologies by stimulating discussion and dialogue on future research directions.
استهدف البحث الحالی استقصاء أثر بیئة تعلم إلکترونیة تکیفیة قائمة على نموذج التلمذة المعرفیة بأسالیبه الستة وفقًا للأسلوب المعرفی (التبسیط مقابل التعقید المعرفی) على تنمیة مهارات إنتاج المحتوى الرقمی وعمق المعرفة لدى طلاب تقنیات التعلیم. استخدم البحث المنهج شبه التجریبی والتصمیم العاملی (2 x 2). طُبق البحث على عینة قوامها (32) طالبًا من طلاب تقنیات التعلیم بمعهد الدراسات العلیا التربویة - جامعة الملک عبد العزیز من خلال مقرر "التعلیم المبرمج" فی الفصل الدراسی الأول من العام الدراسی 1437/1438ه، تم تقسیمهم وفقًا للأسلوب المعرفی باستخدام مقیاس أسلوب التعقید المعرفی، ثم توزیعهم عشوائیًا على مجموعات البحث الأربعة. تمثلت أدوات البحث فی بطاقة ملاحظة الأداء المهاری، واختبار عمق المعرفة.تم رصد النتائج وتحلیلها إحصائیًا باستخدام "اختبار ولکوکسون Wilcoxon Test" لتحدید دلالة الفروق بین مجموعتی البحث الرئیستین فی التطبیق القبلی والبعدی لأدوات البحث، و"اختبار مان وتنی Mann-Whitney Test" لتحدید دلالة الفروق بین کل مجموعتین فرعیتین من نفس ذات الأسلوب المعرفی فی التطبیق البعدی، بالإضافة إلى تحدید حجم الأثر، وأسفرت نتائج البحث عن أن تطبیق نموذج التلمذة المعرفیة من خلال بیئة التعلم الإلکترونیة التکیفیة زاد من قدرتها وفاعلیتها فی تنمیة مهارات إنتاج المحتوى الرقمی وعمق المعرفة مقارنة ببیئة التعلم الإلکترونیة التکیفیة بدون نموذج التلمذة معرفیة. وفی ضوء نتائج البحث قدم الباحث توصیات ومقترحات من شأنها تحسین بیئات التعلم الإلکترونیة التکیفیة وزیادة فاعلیتها.
Adaptive Hypermedia address the fact that each individual user has different preferences and expectations. Hypermedia need adaptive features to provide an improved user experience. This requirement results in an increased complexity of the development process and evaluation methodology. In this article, we first discuss development methodologies used for hypermedia development in general and especially for user-Adaptive hypermedia development. Second, we discuss evaluation methodologies that constitute a very important part of the development process. Finally, we propose a customized development process supported by ASF, a special framework designed to build Adaptive Hypermedia Systems.
Adaptive hypermedia is a relatively new direction of research on the crossroads of hypermedia and user modeling. Adaptive hypermedia systems build a model of the goals, preferences and knowledge of each individual user, and use this model throughout the interaction with the user, in order to adapt to the needs of that user. The goal of this paper is to present the state of the art in adaptive hypermedia at the eve of the year 2000, and to highlight some prospects for the future. This paper attempts to serve both the newcomers and the experts in the area of adaptive hypermedia by building on an earlier comprehensive review (Brusilovsky, 1996; Brusilovsky, 1998).
This chapter describes recent and ongoing research to automatically personalize a learning experience through adaptive educational hypermedia. The Web has made it possible to give a very large audience access to the same learning material. Rather than offering several versions of learning material about a certain subject, for different types of learners, adaptive educational hypermedia offers personalized learning material without the need to know a detailed classification of users before starting the learning process. We describe different approaches to making a learning experience personalized, all using adaptive hypermedia technology. We include research on authoring for adaptive learning material (the AIMS and MOT projects) and research on modeling adaptive educational applications (the LAOS project). We also cover some of our ongoing work on the AHA! system, which has been used mostly for educational hypermedia but has the potential to be used in very different application areas as well.
This paper suggests an alternative to the traditional 'as a whole' approach of evaluating adaptive learning systems (ALS), and adaptive systems, in general. We argue that the commonly recognised models of adaptive systems can be used as a basis for a layered evaluation that offers certain benefits to the developers of ALS. Therefore, we propose the layered evaluation framework, where the success of adaptation is addressed at two distinct layers: • user modelling • adaptation decision making. We outline how layered evaluation can improve the current evaluation practice of ALS. To build a stronger case for a layered evaluation we re-visit the evaluation of the InterBook where the layered approach can provide a difference and provide an example of its use in KOD learning system.
A crucial factor for successful digital educational games, particularly for older children and adolescents, is an appropriate balance; balance between learning and gaming and balance between challenge and ability. These factors are important to maintain fun, immersion, flow experience, and motivation – the motivation to play and therefore to learn. Moreover, it is important to realize a gaming experience that can compete with that of commercial, non-educational games. A special challenge in this context arises from the need for pedagogical support during learning -and therefore during gaming. At many staves of the learning ladder, from a psych-pedagogical perspective, support and feedback is necessary in order to ensure successful, effective, and complacent learning. Considering the importance of not destroying immersion with the game, the assessment of the learning progress and psycho-pedagogical feedback must occur in a non-invasive way. This, however, requires an intelligent system that is capable of assessing individual competencies and learning progress by observing and interpreting the learner's behaviour in the learning situations within the game. In ELEKTRA, a project funded by the European Commission and aiming at developing a sound psycho-pedagogical framework for immersive educational games, we developed a formal cognitive framework for the non-invasive assessment and interventions within complex learning situations, that is, micro adaptivity. Attuned to the assessed competencies or lack of competencies, meaningful feedback, for example hints, suggestions, reminders, critical questions, or praise, can be triggered, without destroying the gaming experience. Two questions arise with respect to feedback. First, does feedback, although designed to be non-invasive, on educational issues impair gaming experience? Second, can feedback in gaming situations facilitate the learning progress or does it increase the learner's cognitive load, which was suggested be several researchers. In the context of the ELEKTRA project, we implemented the theoretical framework of micro adaptivity in the game demonstrator. This demonstrator is a state-of-the art 3D adventure game teaching physics in relation to national school curricula for the age group of 12 o 14 years. For evaluation purposes, log files of the gaming sessions were recorded and, in addition, questionnaires and performance tests were presented. In this work, we present results from an evaluation session. The results indicate that (micro) adaptive interventions (i.e., appropriate and meaningful interventions/feedback for an individual learner, his/her knowledge and learning progress) are superior to neutral (i.e., non-individualized but semantically correct interventions) and inappropriate interventions (i.e., non-individualized, unsuited interventions) in terms of learning and gaming measures. In addition, we analysed the relationships between learning progress and socio-emotional variables. The results indicate that adaptive feedback not only facilitates learning but also attitude and immersion.
The assessment of knowledge and learning progress in the context of game-based learning requires novel, noninvasive, and embedded approaches. In the present paper, we introduce a mathematical framework which relates the (problem solution) behavior of a learner in the game context to the learner's available and lacking competencies. We argue that a problem situation and its status at a certain point in time can be described by a set of game props and their current properties or states. In the course of the game, the learner can perform different actions to modify the props and, consequently, change the problem situation. Each action is evaluated with respect to its correctness or appropriateness for accomplishing a given task which, in turn, enables conclusions about the competence state of the learner. This assessment procedure serves as the basis for adaptive interventions, for instance, by providing the learner with guidance or feedback.
Learning spaces offer a rigorous mathematical foundation for various practical systems of knowledge assessment. An example is offered by the ALEKS system (Assessment and LEarning in Knowledge Spaces), a software for the assessment of mathematical knowledge. From a mathematical standpoint, learning spaces as well as knowledge spaces (which made the title of the first edition) generalize partially ordered sets. They are investigated both from a combinatorial and a stochastic viewpoint. The results are applied to real and simulated data. The book gives a systematic presentation of research and extends the results to new situations. It is of interest to mathematically oriented readers in education, computer science and combinatorics at research and graduate levels. The text contains numerous examples and exercises, and an extensive bibliography. © Springer-Verlag Berlin Heidelberg 2011. All rights are reserved.
The academic user base of an Adaptive Learning System (ALS) can be partitioned in two: the learners and the teachers (encompassing both content authors and tutors). Learners come from a diverse set of backgrounds with varying abilities and motivation and hence, have very individual learning requirements [2, 6, 7]. The time and eort they can devote to learning are nite. On the other hand, teachers can create and adapt learning material to individual learning requirements. However, the time and eort they can devote to teaching are also nite. An ALS, through the virtues of adaptivity [4, 5, 3], can reconcile this mismatch by delivering individualized educational experiences to the learners while making the best use of the time and eort invested by the teachers. The rst step in developing an ALS is requirements elicitation [9]. As part of this endeavor, it is instructive to interview the user base. Interviews help crystalize the expec-tations of learners and teachers with respect to an ALS. They may reveal requirements not previously envisaged as being key, requirements purported to be useful but are con-sidered otherwise by the interviewees, and requirements from dierent partitions of the user base that are contradictory. Within the scope of the GRAPPLE Project 1 , we have elicited requirements from learners and teachers across several European academic insti-tutions through explorative, semi-structured interviews. In this report we describe the methodology we employed while preparing, conducting, and analyzing the interviews and we present our ndings along with some objective and subjective analysis. 1 The GRAPPLE Project (Generic Responsive Adaptive Personalized Learning Environment) is an EU FP7 STREP project that aims to deliver a technology-enhanced learning (TEL) environment that guides learners through a learning experience, automatically adapting to personal preferences, prior knowledge, skills and competences, learning goals, and the personal or social context in which the learning takes place. This functionality will be provided as a set of adaptive learning services that will be integrated with existing Open Source and commercial Learning Management Systems (LMSs).
Intelligent tutoring systems are highly interactive learning environments that have been shown to improve upon typical classroom
instruction. Cognitive Tutors are a type of intelligent tutor based on cognitive psychology theory of problem solving and
learning. Cognitive Tutors provide a rich problem-solving environment with tutorial guidance in the form of step-by-step feedback,
specific messages in response to common errors, and on-demand instructional hints. They also select problems based on individual
student performance. The learning benefits of these forms of interactivity are supported, to varying extents, by a growing
number of results from experimental studies. As Cognitive Tutors have matured and are being applied in new subject-matter
areas, they have been used as a research platform and, particularly, to explore interactive methods to support metacognition.
We review experiments with Cognitive Tutors that have compared different forms of interactivity and we reinterpret their results
as partial answers to the general question: How should learning environments balance information or assistance giving and withholding to achieve optimal student learning? How best to achieve this balance remains a fundamental open problem in instructional
science. We call this problem the “assistance dilemma” and emphasize the need for further science to yield specific conditions
and parameters that indicate when and to what extent to use information giving versus information withholding forms of interaction.