No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Contexts for evaluating educational software
Abstract
The evaluation of educational software is of concern to two particular academic communities: HCI and educational technology.
There is a danger that usability features are considered at the expense of educational issues (and the converse of this is
of course equally true). This paper considers how the notion and practice of evaluation in the educational community differs
from that in HCI and also identifies areas of overlap. It then describes how particular influences and contexts have led one
group of evaluators from the educational community to develop a context, interactions, attitudes and outcomes (CIAO!) model
of evaluation for computer assisted learning (CAL) evaluation in distance education. The application of this model is illustrated
by case studies from a recent evaluation project and related research. The paper concludes with a discussion of the issues
raised for both communities by this model for evaluation.
... For answering the above questions, a thorough evaluation of the system was conducted. For the evaluation, we combined two evaluation frameworks, the CIAO! framework (Jones et al., 1999) and the evaluation framework that was proposed by Lynch and Ghergulescu (2016), that were developed for evaluating educational software. In this way, we accomplish to assess multiple aspects of the tutoring system that include the intelligent features as well as the necessary educational aspects. ...
... However, in the literature review, there is not a widely approved evaluation framework and technique for the assessment of an ITS, especially since ITSs need to be evaluated concerning their intelligent features as awell as their educational effectiveness and usability aspects. Therefore, after a thorough investigation in the literature review, we decided to perform the evaluation of the fuzzy-based ITS following well-known and accepted evaluation methodologies: the CIAO! framework (Jones et al., 1999) and the evaluation framework that was proposed by Lynch and Ghergulescu (2016). We chose to use these frameworks because the CIAO! framework was developed especially for the evaluation of general educational aspects of computer assisted learning systems and the Lynch and Chergulescu framework concerns the evaluation of adaptive and intelligent learning systems. ...
... Therefore, its evaluation has to include both aspects that concern in general an educational software and aspects that concern its intelligent operation. To succeed it we combined two evaluation frameworks: the CIAO! framework (Jones et al., 1999), which evaluates in general aspects of a computer assisted learning (CAL) system, and the evaluation framework that was proposed by Lynch and Ghergulescu (2016), which evaluates aspects of adaptive and intelligent learning systems. ...
Nowadays, the improvement of digital learning with Artificial Intelligence has attracted a lot of research, as it provides solutions for individualized education styles which are independent of place and time. This is particularly the case for computer science, as a tutoring domain, which is rapidly growing and changing and as such, learners need frequent update courses. In this paper, we present a thorough evaluation of a fuzzy-based intelligent tutoring system (ITS), that teaches computer programming. The evaluation concerns multiple aspects of the ITS. The evaluation criteria are: (i) context, (ii) effectiveness, (iii) efficiency, (iv) accuracy, (v) usability and satisfaction, and (vi) engagement and motivation. In the evaluation process students of an undergraduate program in Informatics of the University of Piraeus in Greece participated. The evaluation method that was used included questionnaires, analysis of log files and experiments. Also, t-tests were conducted to certify the validity of the evaluation results. Indeed, the evaluation results are very positive and show that the incorporated fuzzy mechanism to the presented ITS enhances the system with Artificial Intelligence and through this, it increases the learners’ satisfaction and new knowledge learning and mastering, improves the recommendation accuracy of the system, the efficacy of interactions, and contributes positively to the learners’ engagement in the learning process.
... This phase lasted throughout the whole process of the design and the development of the presented system. At the second phase of the evaluation, the established framework CIAO! (Jones et al., 1999) and t-test were utilised. At this phase, instructors and students participated. ...
... At the second phase of the evaluation, the CIAO! framework (Jones et al., 1999) was utilised. The reason why we used CIAO! is because it is a well-established framework which has been used extensively for evaluation with great results (Scanlon et al., 2000;Owston, 2008). ...
... For the evaluation of each one of the three dimensions, several questions have been formulated with a six-point Likert scale from -3 (absolutely negative) to 3 (absolutely positive). Specifically, questions 1-3 belong to the 'context' dimension, questions 4-9 belong to the 'outcomes' dimension and questions 10-13 belong to the 'interaction' dimension (Jones et al., 1999). The reliability of the questionnaire has been explored in the authors' previous work (Troussas et al., 2021). ...
... This phase lasted throughout the whole process of the design and the development of the presented system. At the second phase of the evaluation, the established framework CIAO! (Jones et al., 1999) and t-test were utilised. At this phase, instructors and students participated. ...
... At the second phase of the evaluation, the CIAO! framework (Jones et al., 1999) was utilised. The reason why we used CIAO! is because it is a well-established framework which has been used extensively for evaluation with great results (Scanlon et al., 2000;Owston, 2008). ...
... For the evaluation of each one of the three dimensions, several questions have been formulated with a six-point Likert scale from -3 (absolutely negative) to 3 (absolutely positive). Specifically, questions 1-3 belong to the 'context' dimension, questions 4-9 belong to the 'outcomes' dimension and questions 10-13 belong to the 'interaction' dimension (Jones et al., 1999). The reliability of the questionnaire has been explored in the authors' previous work (Troussas et al., 2021). ...
... Various theories have stressed the importance of context and user tasks in evaluating digital library design efforts [2]. Jones et al. proposed the CIAO! framework which evaluates educational software through context, interaction, attitude, and outcomes [3]. However, this framework has not been adopted in the evaluation of digital libraries. ...
... Following the CIAO! framework, surveys, observations, videotaping, software logging, and web logging were used to capture the context, interaction, attitude, and outcomes of the use of K-MODDL [3]: (1) Context: at the beginning of this research, classroom observations were used to explore the structure and style of the class in order to put the use of K-MODDL in context; ...
... Learning: Outcomes of learning are always difficult to assess in the use of technologies in education settings [3]. In this class, the methods of users' subjective evaluation and critical incident analysis on the process of learning were adopted [4]. ...
This paper describes an evaluation effort of the use of the Kinematic Model for Design Digital Library (K-MODDL) in an undergraduate mathematics class. Based on CIAO! framework, the research revealed usability problems and users' subjective experience when using K-MODDL, confirmed the usefulness of various physical and digital models in facilitating learning, and revealed interesting relationships among usability, learning, and subjective experience.
... The utility, as well as the educational and pedagogical orientation of the developed system were assessed using the CIAO! framework (Jones et al., 1999). CIAO! is an evaluation framework which is used to evaluate students' use of technology-based teaching and learning (Jones et al., 1999). ...
... The utility, as well as the educational and pedagogical orientation of the developed system were assessed using the CIAO! framework (Jones et al., 1999). CIAO! is an evaluation framework which is used to evaluate students' use of technology-based teaching and learning (Jones et al., 1999). The CIAO! framework is considered as the most appropriate evaluation model for our case, since it focuses on evaluating the learner's experience using the new digital learning environment. ...
Social Media have become a tool for communication among people, the majority of whom belong to the younger generation. Hence, exploiting them in the educational field can promote communication and collaboration among students and instructors. Moreover, expanding them with more pedagogical tools, such as assessments and adaptivity, can improve further the learning outcomes. In view of the above, this paper presents i-LearnC#, an intelligent tutoring application over Facebook, for learning programming. This Facebook app extends the capabilities of Facebook platform to be used as an integrated tutoring system. In particular, i-LearnC# embodies an intelligent Virtual Coach offering individualised advice, covering students’ knowledge gaps and solving queries on unknown learning concepts. Regarding students’ assessment, i-LearnC# applies the Revised Bloom Taxonomy and delivers personalised assistance to them through the Virtual Coach. Finally, i-LearnC# employs cluster analysis for recommending the optimal groups for collaboration to students. i-LearnC# was evaluated using a well-known framework, the statistical hypothesis test and system's log files. The results are very promising, showing that i-LearnC# has been educationally beneficial to students in higher education promoting effective learning and teaching of computing education and achieving high acceptance rate as an adaptive intelligent social learning environment.
... One such evaluation framework outlines three dimensions to evaluate (Jones et al., 1999): ...
... The second phase was based on the framework presented in (Jones et al., 1999) and involved in total 40 users, both undergraduate and graduate. Mobile applications tend to be more convenient and flexible to use, as students spend much time in the use of mobile phones. ...
This paper proposes a student-oriented approach tailored to effective collaboration between students using mobile phones for language learning within the life cycle of an intelligent tutoring system. For this reason, in this research, a prototype mobile application has been developed for multiple language learning that incorporates intelligence in its modeling and diagnostic components. One of the primary aims of this research is the construction of student models which promote the misconception diagnosis. Furthermore, they are the key for collaboration, given that students can cooperate with their peers, discuss complex problems from various perspectives and use knowledge to answer questions and/or to solve problems. Summarizing, in this paper, a mobile tutoring framework, built up in the context of student collaboration, is presented. Collaborative student groups are created with respect to the corresponding user models. Finally, the prototype was evaluated and the results confirmed the usefulness of collaborative learning.
... The aforementioned questions followed the contexts of CIAO! Framework. 32 The questions of the evaluation questionnaire were grouped based on the three dimensions of CIAO!, 32 and an overall evaluation result per dimension was conducted. These results are illustrated in Fig. 4. Table 1 summarizes a basic set of questions that were asked to the students after their interaction with the application. ...
... The aforementioned questions followed the contexts of CIAO! Framework. 32 The questions of the evaluation questionnaire were grouped based on the three dimensions of CIAO!, 32 and an overall evaluation result per dimension was conducted. These results are illustrated in Fig. 4. Table 1 summarizes a basic set of questions that were asked to the students after their interaction with the application. ...
Mobile learning offers, with the help of handheld devices, a continuous access to the learning process. With the advent of mobile learning, educational systems are changing, offering the possibility of distance education without the restrictions of place and time. As such, new technological advancements are employed by mobile learning. This paper presents the design, development and evaluation of a novel artificial conversational entity, incorporated in a mobile learning system for personalized English language instruction. More specifically, it offers amelioration of the domain knowledge model by adapting it to the students’ needs and to the pace that they prefer to receive learning. Moreover, it creates personalized tutoring advice in order to support students in the educational process. Finally, it can assist the procedure of assessments since it automatically generates questions to assess the knowledge level of students. The evaluation of the mobile tutoring system presents promising results regarding the incorporation of this new technology in digital education with the aim of creating a student-centric learning experience.
... While there are heuristics specially designed for educational software, for example, the CIAO! Framework by Jones et al. (1999), there are relatively few heuristics available for researchers to use in usability studies for instructional games. ...
... Observing how prospective users interact with an application provides a deeper understanding on their learning processes (Jones et al., 1999). Nonetheless, it can be argued that to obtain a more credible evaluation, some of the heuristics should be triangulated with expert reviews, namely the teachers who are well-versed in teaching social skills. ...
... It can not only integrate teaching, counseling, self-test, consolidation, review, and assessment but also plays a great role in cultivating students' self-study ability, mobilizing students' learning enthusiasm, improving the learning effect, and teaching students according to their aptitude. Multimedia technology-assisted instruction can let students see the interactive integrated information with pictures, texts, and audio-visual integration [11,12]. They can not only read the teaching content in the multimedia courseware but also obtain the sound or video information associated with classroom teaching. ...
... Perceived usability of educational technology is an important factor determining the overall effectiveness of user-system interaction and consequently the learning itself (Mahmodi et al., 2020;Virvou and Katsionis, 2008;Ardito et al., 2006;Jones et al., 1999;Mayes and Fowler, 1999;Squires, 1999). Although there is general agreement among researchers on the relationship between perceived usability of a technology system and user's learning effectiveness, research data investigating this relationship are extremely limited. ...
Purpose
The purpose of this study was to determine the impact of perceived usability and students' personality traits on their learning gain in an e-learning context at the university level.
Design/methodology/approach
The factors examined are related to individual characteristics such as students' personality traits, as well as to perceptual characteristics such as the perceived usability of the platform used. A total of 110 undergraduate students participated in the study. A one-group pretest-posttest research design was adopted. Big 5 personality test, System Usability Scale (SUS) and a general knowledge assessment questionnaire were used.
Findings
Perceived usability of Zoom platform is statistically significantly correlated with students' learning gain ( r = 0.294, p = 0.002, s). Concerning learning effectiveness in the current e-learning scenario, students' final performance was found to be statistically significantly higher than the initial ( p = 0.000, s). A hierarchical regression analysis ( R ² = 0.146) unveiled that Zoom's perceived usability and personality traits are significant predictors for learning gain ( p = 0.011, s).
Research limitations/implications
The findings of this research provide important implications regarding the design of lessons in an e-learning context.
Social implications
A substantial fraction of the educational process is going online, especially in higher education. Thus, a thorough understanding of the factors which influence learning gain in an e-learning context is of significance.
Originality/value
The main contribution of this study is that it quantifies the variance of the learning gain explained by two factors, namely, SUS and personality.
... Markopoulos and Bekker (2003) also suggested that system usability may be a significant drawback for students using VR games in learning or enjoyment. However, researchers often neglect the system usability of VR games, whereas the evaluation is purely conducted based on the achievement of educational outcomes (Jones et al., 1999;Mayes and Fowler, 1999). To ensure that the developed VR environment is sophisticated, attractive and at the same time useable, the expense of student effort for engagement in educational games should be well-defined before VR game development and evaluation (Yacci et al., 2004). ...
Purpose
The construction site operates under a hazardous environment that requires a high level of understanding in building systems to minimise accidents. However, the current building education generally adopts paper-based learning approaches that lack hands-on experiences. Furthermore, to achieve Industrial Revolution 4.0 in line with any unforeseen pandemic, the most optimum solution is to transition from physical to technological-based building education. This paper aims to address the problems by proposing a game-based virtual reality (GBVR) for building utility inspection training.
Design/methodology/approach
The feasibility of the GBVR for building the utility inspection training approach is validated on a sample of undergraduate engineering students through user experience (survey) and performance-based comparisons against traditional paper-based training method.
Findings
The results show that the developed GBVR training has higher system usability in terms of visual output and knowledge retention than paper-based training due to visualisation technologies. The GBVR training method has also higher user-friendliness because of the higher motivational and engagement factors through the adoption of virtual reality and game-based learning.
Research limitations/implications
GBVR training required a longer training duration and achieved a lower performance score (effectiveness) but can be improved by transitioning into hands-on tasks. This study has the potentials to be extended to vocational training platforms for competency development in the construction workforce by using cutting-edge extended reality technologies.
Originality/value
This paper portrays the benefits of integrating virtual reality technology in building education to overcome the low practicality and engagement of paper-based training.
... It is a crucial characteristic for user acceptance and therefore for the technology to be further diffused (Holden and Rada, 2011). Furthermore, a significant body of research demonstrates that the perceived usability of educational technological systems greatly affects the learning effectiveness and the overall learning experience (Bettayeb et al., 2020;Ifinedo et al., 2018;Katsanos et al., 2012;Wong et al., 2003;Grani c, 2008;Kukulska-Hulme and Shield, 2004;Mayes and Fowler, 1999;Orfanou et al., 2015;Shitkova et al., 2015;Jones et al., 1999;Squires, 1999). In specific, when a technological system responds to the user's needs in a direct, intuitive and effective manner it will lead to increased use and adoption (Tselios et al., 2008;Ardito et al., 2006). ...
Purpose
The purpose of this paper is to investigate the impact of personality traits on the perceived usability evaluation of e-learning platforms. In specific, perceived usability levels of the educational platforms and tools used in primary and secondary education in Greece are demonstrated. The impact which personality traits and other individual-related factors have on the perceived usability were also examined.
Design/methodology/approach
In total, 2,239 Greek primary and secondary education teachers participated in the study through an online survey. The system usability scale (SUS) and Big 5 personality test questionnaires were adopted, as well as a demographics questionnaire and nine platforms were examined: e-me, eclass, Google Classroom, Microsoft Teams, Cisco Webex, Edmodo, Padlet, Skype and Zoom.
Findings
Most platforms were rated as satisfactory in terms of perceived usability as measured by SUS. SUS scores were not found to be significantly related with participants’ age, gender, private/public school, working relationship. Furthermore, openness to new experience and extraversion demonstrated the strongest positive correlation with perceived usability evaluation.
Research limitations/implications
The current study used a sample of Greek teachers as participants, so the generalizability of results without considering cultural or organizational issues is of questionable validity. The Big 5 personality test is widely adopted; however, it seems that it does not capture comprehensively all possible dimensions of personality.
Practical implications
Acknowledging the correlations between personality traits and perceived usability for each platform, teachers may now choose the most user-friendly one depending on the personality traits. Designers can adapt their systems to meet users’ needs accordingly. Moreover, the reported results provide a frame of reference for the respective organizations and companies to compare the quality of user experience of their products and services. Thus, development costs of an e-learning platform/tool can be reduced accordingly.
Social implications
Text.
Originality/value
Perceived usability of a technological system affects the way users interact with it and consequently the whole learning experience. Thus, factors which may affect perceived usability and, in turn, the learning outcomes are of paramount importance and should be exhaustively examined. Giving that personality affects or is related to, other parameters related with technology (technology adoption, perceived and actual use, acceptance, interaction, perceived ease of use), this study addresses a significant research gap and sheds light to the before-mentioned issues.
... e b) Qual o conjunto de evidências sobre o uso do Robomind no processo de ensino e aprendizagem de programação?. Seguindo as diretrizes de [Jones et al. 1999], entende-se por contribuições pedagógicas: ganhos de aprendizagem (aquisição de conhecimento sobre um determinado tema), capacidade de motivar estudantes, como também o potencial em transformar (favoravelmente) opiniões sobre um determinado tema. ...
O Robomind é uma tecnologia educacional apoiada em recursos lúdicos e pedagógicos para dar suporte ao aprendizado de programação. Sumarizar as evidências acadêmicas já publicadas sobre o uso deste software é fundamental para avaliar sua eficácia. Uma revisão sistemática da literatura foi realizada e observou-se que os estudantes que utilizaram o Robomind apresentaram maior motivação com os estudos, interesse pela área de computação e menor evasão. Por outro lado, os ganhos de aprendizagem foram divergentes. Por fim, discutem-se as fragilidades científicas e estatísticas observadas na maioria dos estudos e contribuições são apresentadas para aperfeiçoar pesquisas na área de informática na educação.
... Traditionally, highly detailed evaluation approaches are labor and time-intensive (Jones et A., 1999). In our design study, the heuristic approach to evaluation using the Learning Object Review Instrument (LORI) was combined with usability testing. ...
... They also refer to their use of the CIAO! Framework, developed within the CALRG for evaluating CAL, and how it draws on a variety of sources, using both qualitative and quantitative approaches (Jones et al., 1999). In evaluating student use of computers, particularly collaborative learning, they argued for the need to observe students interacting with the educational innovation, and also note Issroff 's holistic approach (Issroff, 1995) which in addition to recording interactions emphasises the importance of affective measures. ...
... The ultimate objective for educational software is to be educationally beneficial. So it is important to understand how usability can contribute to educational goals [19]. Squires and Preece [20] argue that software developers do not consider enough the implications of usability features of an educational package in order to achieve educational goals. ...
This paper deals with quality enhancement of e-learning activities through the satisfaction analysis expressed by university learners while using specific learning modules. The paper considers a newly engineered survey that evaluates not only quality in use but also quality in learning and learner involvement. The quality of the interface, the friendliness of the contents, the accessibility of concepts and the capability to provide new knowledge and skills are all considered. Through the paper, the various aspects concerning the different discriminating factors that characterize e-learning processes are investigated. The results, obtained during university teaching activities, show the usefulness of the survey in individualizing the more improvable factors by understanding effective learners' needing.
... While the frameworks that exist for the evaluation of educational software are suitable and relevant for the evaluation of educational technologies such Computer Assisted Learning (CAL) systems (Jones et al., 1999;Squires & Preece, 1996), the existing evaluative methods and frameworks for CASE tools are mainly suitable for evaluation in industrial environments. The existing frameworks for the evaluation of CASE tools are mainly intended for its evaluation and selection in organizations (Misra, 1990;Mosley, 1992). ...
The aim of the research is to derive a framework for the evaluation of Computer Aided Software Engineering (CASE) tool learnability in educational environments. Drawing from the literature of Human Computer Interaction and educational research, a framework for evaluating CASE tool learnability in educational environments is derived. The two main differences between this framework and existing evaluation frameworks is that 1) it is specific to educational environments and 2) it recognizes the significance of contextual factors such as learning environment and teaching and learning methodologies in evaluating learnability. The framework is oriented towards learnability and is customisable. It helps to define the important issues related to CASE tool learnability, and allows conclusions to be drawn about all influential factors on learning in the given context. A detailed case study describes the application of the framework to evaluate the learnability of Rational Rose, a CASE tool used in an undergraduate Systems Analysis and Design course.
... With regard to usability, several authors have considered usability to be an important factor that affects educational effectiveness [44,45,46]. Sun et al. [47] argued that systems that are easy to use help students to focus their attention on the content. ...
Therapeutic education in diabetes helps patients take responsibility for self-control of their disease, and providing technological support systems facilitates this education. In this paper, we present an augmented reality game to support therapeutic education for patients with diabetes. Our game helps children (aged 5–14 years) to learn carbohydrate (carb) content of different foods. The game shows virtual foods on a real dish. The number of carb choices corresponding to the visualized food is also shown (1 carb choice = 10 grams of carbs). A study to determine the effectiveness of the game in terms of learning and perceived satisfaction and usability was carried out. A total of seventy children with diabetes participated in the study. From the results, we observed that the initial knowledge about carb choices of the children who participated in the study was low (a mean of 2 on a scale from 0 to 9). This indicates that therapeutic education for patients with diabetes is needed. When the results for the pre-knowledge questionnaire and the post-knowledge questionnaire were compared, it was shown that the children learned about carb choices by playing our game. We used two post-knowledge questionnaires (one post-knowledge questionnaire that contained the same foods as the pre-knowledge questionnaire and a second post-knowledge questionnaire that contained foods that were different from the ones on the pre-knowledge questionnaire). There were no statistically significant differences between these two different post-knowledge questionnaires. Moreover, the knowledge acquired was independent of gender and age. We also evaluated usability and perceived satisfaction. The children were satisfied with the game and considered that the game offers a high degree of usability. This game could be a valuable therapeutic education tool for patients with diabetes.
... Its sections cover instructional goals, instructional design, program design, hardware and software, management benefits, and documentation. The checklist applies to educators, and has been compiled from Doll[11]and Sloane and associates[12]. It seems complete, well organized, and understandable by the educators, without pedagogic terminology, guiding the users to identify the instructional objectives of the software. ...
This work comments on an overall approach to the design, development and evaluation of advanced learning environments. After a critical review of the related bibliography, it seems that there are no guidelines and questionnaires for an integrated design and evaluation process, covering all the factors involved and addressed to different groups of evaluators. This article tries to give an overall approach proposing three groups of interconnected factors for the design and evaluation of educational software. These are the information content, the context, and the evaluation process. These groups of factors are represented in three-dimensional graphs. The space under each graph integrates the design and evaluation criteria for educational software.
... Η επηινγή θαηάιιειεο κεζφδνπ αμηνιφγεζεο ελφο δηδαθηηθνχ ζπζηήκαηνο, πέξαλ ησλ πεξηνξηζκψλ θαη ησλ ηδηαηηεξνηήησλ πνπ πεξηγξάθηεθαλ ζηελ εηζαγσγηθή ελφηεηα, επεξεάδεηαη απφ ην γλσζηνζεσξεηηθφ πιαίζην πνπ δηέπεη ηε ζρεδίαζε ηνπ. πλεπψο, θαηά ηε δηαδηθαζία ζρεδίαζεο θαη αμηνιφγεζεο εθπαηδεπηηθψλ ζπζηεκάησλ ζα πξέπεη λα έρεη δηεξεπλεζεί ζπζηεκαηηθά ν ξφινο ηνπο θαζψο θαη ην πξνηηζέκελν πιαίζην ρξήζεο (Jones et al., 1999). ...
Το κεφάλαιο αυτό εστιάζει στην αξιολόγηση ευχρηστίας (usability) εκπαιδευτικών συστημάτων που βασίζονται σε διαφορετικές σύγχρονες τεχνολογίες. Εξετάζει τη σημασία της ευχρηστίας και την επίπτωσή της στη μαθησιακή αποτελεσματικότητα των περιβαλλόντων αυτών. Επιχειρείται επισκόπηση διαφορετικών τεχνικών μέτρησης της ευχρηστίας και εξετάζεται η εφαρμογή τους σε διάφορες κατηγορίες εκπαιδευτικών συστημάτων, όπως πολυμεσικών/ υπερμεσικών εφαρμογών και συστημάτων στηριγμένων σε εποικοδομητικό γνωσιοθεωρητικό πλαίσιο. Παρατίθενται παραδείγματα μέτρησης ευχρηστίας από ειδικούς, καθώς και από τους ίδιους τους χρήστες (μαθητές και εκπαιδευτικούς). Στόχος είναι να περιγραφούν μέθοδοι τις οποίες μπορεί να εφαρμόσει ο ίδιος ο εκπαιδευτικός που πρόκειται να χρησιμοποιήσει ένα διδακτικό σύστημα, ώστε να μετρήσει την ευχρηστία του και να την συσχετίσει με τους εκπαιδευτικούς στόχους του. Τέλος στο κεφάλαιο αυτό περιγράφονται προηγμένες τεχνικές μέτρησης ευχρηστίας σύγχρονων περιβαλλόντων μάθησης και γίνεται μια επισκόπηση των ερευνητικών εξελίξεων της περιοχής ευχρηστίας συνεργατικών συστημάτων.
... While this research belongs to the broad field of software evaluation the main focus has been on evaluating educational software and specifically evaluating the usability of social media blends for connectivist learning in ODeL environments. Usability limitations of educational software, in general, have been extensively researched and resulted in evaluation frameworks as discussed in (Leacock & Nesbit 2007, Jones et al. 1999, Belyk & Feist 2002, Georgiadou, Economides, Michailidou & Mosha 2001, Nesbit and Belfer 2004. Fox and Naidu (2009) and Owens, Lenz and Speagle (2009) evaluate the usability of the most commonly used social media blends and find firsttime users to be negatively affected by the lack of compliance with generic usability guidelines. ...
Distance education has by nature always depended on technology as a vehicle for various media used to facilitate learning. Today as technology has taken centre stage in our lives especially Internet-based technology, distance education continues to grow in its use of state-of-the-art tools and hence the need to effectively adopt these online tools for online education remains inevitable. This study presents an evaluation of the usability of social media blends in e-learning environments. Comprehensive guidelines for improving the usability of social media blends are proposed in order to facilitate learner networks where knowledge can be created that is based on the leaners’ experiences and their peer connections.
In this meta-analysis study literature was searched to identify qualitative research articles that discuss social media blends most commonly used in e-learning. The search was done on South African and international academic databases which included SAe-Publications, EbscoHost, ProQuest and Google Scholar, among others.
The metadata analysis was conducted following the online collaborative learning theory as a conceptual framework and the findings agreed with previous studies that the use of social media blends still lacks important empirical data. This study recommends a set of phases in designing curriculum for social media use in e-learning. The proposed guidelines should be useful to instructional designers interested in using modern learning theories in e-learning. Since African qualitative research could not be found, further work in this field could involve qualitative studies on the use of e-learning in African institutions.
... As the emphasis on education, the teacher should evaluate the usefulness of a particular software and its impact on learning. Continuous training of teachers should enable teachers to acquire skills in the use of educational software and estimating its value (Bitter & Camuze, 1984 Jones (1993) points out several essential characteristics of a good software for children: ...
In the era of information technology and general digitization of society, invasion of every kind of software is evident. No matter how laudable is the existence and development of educational software, taking into account its role, its quality and whether it achieves the desired goal is very important. In addition to programming experts it is necessary to include educational experts in certain fields when developing educational software. Those who should take an active part and be able to distinguish good from bad educational software are certainly teachers. Especially as they should be the most competent persons from which parents will seek the advice and opinion when searching for appropriate educational software.
... We developed the context, interaction, attitudes and outcomes (CIAO!) framework to support the evaluation of CAL developments, mainly for the Open University courses (Jones, Scanlon, Tosunoglu, Morris, Ross, Butcher and Greenberg, 1999). In this framework, information from a variety of sources and both qualitative and quantitative techniques are used. ...
This paper will discuss the approach to the evaluation of computer-supported collaborative learning developed in our group over the past ten years. This approach depends on the collection of video data to allow the analysis of key features of problemsolving behaviour within groups of students working on collaborative learning tasks. Our theoretical framework derives from two sources- the CIAOl framework for evaluating examples of CAL and an analysis of appropriate methods of evaluating computersupported collaboration. Our work in this area has been supported by developing the data capture facilities for the CALRG (Computers and Learning Research Group) at the Open University. We will draw on a number of studies to illustrate this approach and will present a brief case study from work done on a computer-supported learning environment for statistics where we use video records of video-mediated collaboration. This case study gives an example of the rich data that can be collected using video recording and analysed to increase understanding of computer-supported collaboration.
... Educational software differentiates to other application software to the fact that during the development of an educational oriented software the way students learn should be taken into account [3], [4]. Moreover, the usability factor of the designed educational software must also be taken under consideration during the development phase since it is tightly connected with the learning process [3], [5] and plays a very important role with regards to the acquisition of the educational software [6], [7]. ...
In this paper the complicated task of educational software evaluation is revisited and examined from a
different point of view. By the means of Educational Data Mining (EDM) techniques, in the present study
177 of the most common evaluation standards that have been proposed by various researchers are
examined and evaluated with regards to the degree they affect the effectiveness of educational software.
More specifically, via the employment of prediction, feature selection and relationship mining techniques
we investigate for the underlying rationale hidden within the data collected from experiments conducted at
the Department of Education of the University of Patras with regards to the software evaluation task and
the results of this study are presented and discussed in a quantitative and qualitative way.
... To achieve this objective, we used an evaluation framework which is completely adjusted to educational software and outlines three basic dimensions to evaluate [12]: ...
Social networks are addressed to a very large and heterogeneous audience of people. When trying to incorporate an intelligent language learning system in social networks, a problem of user diversity emerges and thus, user clustering based on their characteristics is necessary. In view of this compelling need, this paper concerns the pattern discovery of user clusters in social networks. In this research, we have modeled the Facebook user characteristics that determine the clustering process. An unsupervised clustering algorithm was used so that coherent groups of users with the same learning styles and capabilities are generated. This algorithm clusters users by taking as input their several fundamental characteristics, such as their age, educational level, number of languages spoken and computer knowledge level. The general objective of this data mining process is to extract important information and to gain knowledge from the user data set and transform it into a manageable and intelligible structure with a view to ameliorating the learning process. These experimental results show that the Facebook user characteristics, which were chosen at the clustering process, seem to be significant determinants for the clusters and the whole learning experience of each user.
... 462 Diese Zielwerte können sich auf Erfahrungen vorhergehender Lernarrangements beziehen, dem Vergleich mit Aktivitäten ähnlicher Lernservices (Benchmarking) entstammen oder Resultat begründeter Annahmen sein. Das Vorgehen ist auch analog zuCo- Grob et al. 2010, S. 39. 461 Rosemann und Brocke 2010 Auf Mikroebene des Lernservices illustrierenJones et al. (1999) die Entwicklung erwünschter Interaktionen in einzelnen Lehr-/ Lernkomponenten am Beispiel mehrerer Fallstudien. Da sich im ESIC die Verbesserung eines Lernszenarios über die prototypische Entwicklung von Lernarrangements vollzieht, ergibt sich daraus, dass alle notwendigen Daten zum Zeitpunkt der jeweiligen Aktivitäten des ESIC gesammelt werden müssen. ...
Terms like e-, m- or Blended Learning show, currently many educational services are supported by web technologies. Within such services predominant parts of learner’s usage processes are hidden from the educator’s perception. In front of a service-dominant logic understanding usage processes is essential to comprehend the value-co-creation of educators and learners. Learning analytics may hold methods to enable a continual improvement process by collecting and analyzing usage data. A systematic literature review reveals that neither educational service nor learning analytics literature present a suitable or adequately specified procedural model for this purpose. Following a design science research approach this dissertation introduces a new procedural model to systematically improve educational services. It is called ’Educational Service Improvement Cycle (ESIC)’. Design parameters are derived from the literature review. As part of an iterative design process two learning scenarios from higher education are used to develop the procedural model. The ESIC consists of six activities, which enable a systematic analysis of usage processes. Recommended methods, a role concept and a broad overview on possible indicators are presented to clarify the ESIC. Besides their demonstrative purposes both learning scenarios are also part of an ex ante evaluation. The ex post evaluation contains another single case study, where students make use of the ESIC and create a learning analytics dashboard for advanced training of entrepreneurs. Additional interviews with experts of the field also indicate its perceived usefulness and ease-of-use.
... Therefore, data was collected from learners" interaction with the application. In fact, evaluation models such as the CIAO model (Jones et al. 1999) have outlined that while evaluating educational technology one should consider data about learners" interaction with the software. This points to measuring performance because performance is all about what the user actually does in interacting with the product (Albert & Tullis 2008) and consists of five types of metrics: task success; time-on-task; errors; efficiency; and learnability. ...
The ubiquity of mobile phones provides an opportunity to use them as a resource for construction of programs beyond the classroom. However, limitations of mobile phones impede their use as typical programming environments. This research proposes that programming environments on mobile phones should include scaffolding techniques specifically designed for mobile phones, and designed based on learners' needs. This paper discusses the effectiveness of theoretically-derived scaffolding techniques to construct Java programs on a mobile phone. The results indicate that even though scaffolding techniques could support learners to program on a mobile phone, further modifications of the designed scaffolding techniques may be necessary in order to more effectively support programming on a mobile phone.
... One such evaluation framework outlines three dimensions to evaluate [16]: ...
Social networking services tend to promote social relations among people sharing interests, activities, backgrounds, or real-life connections. They consist of a representation of each user, namely a profle, along with of his/her social links and a variety of additional information. Such kind of information can be used as a means for highlighting the educational aspect of social networking services. To this direction, the authors of this chapter used Facebook as a testbed for this research and implemented a multi-language learning application. The main focus of this chapter is on the automatic classifcation of Facebook users, utilizing the aforementioned application, based on their profles. In that way, coherent user clusters can be created and effcient cooperative learning among them can be achieved. Finally, the educational process can be further ameliorated given that cooperation among user clusters is recommended by the system.
... Therefore, data was collected from learners' interaction with the application. In fact, evaluation models such as the CIAO model (Jones et al. 1999) have outlined that while evaluating educational technology one should consider data about learners' interaction with the software. This points to measuring performance because performance is all about what the user actually does in interacting with the product (Albert & Tullis 2008) and consists of five types of metrics: task success; time-on-task; errors; efficiency; and learnability. ...
PROCEEDINGS OF THE 11th INTERNATIONAL CONFERENCE ON MOBILE LEARNING 2015, Page 99
... We would also like to discuss the ease of use and its implications. Several authors argued that perceived ease of use is an important technical factor that affects educational effectiveness [52][53][54]. Based on their findings, Sun et al. [55] stated that learning systems that are easy to use help students to focus their attention on the learning content and they are more motivated to learn. ...
Playful interactions facilitate the development of engaging applications for different purposes. This aspect is very important for serious games, and especially when these games are for children. Another aspect to consider is the interaction among children, which could be a great reinforcement in learning environments. Children enjoy playing, and they like playing with other children. This relationship could encourage their motivation and their learning outcomes. In this paper, a playful interaction system for learning about a period of history is presented. The interaction of the system was achieved using natural gestures and the visualization was autostereoscopic. A study was carried out to determine whether their learning outcomes were greater playing collaboratively or playing individually. Forty six children from 7 to 10 years old participated in the study. The analysis of the pre-tests and the post-tests indicate that the children increased their knowledge about historical periods after playing with the two modes. Therefore, the game could be used as an effective transmitter of knowledge both collaboratively and individually. When the post-knowledge scores for the two modes were compared, statistically significant differences were found in favor of the collaborative mode. Therefore, the collaborative mode facilitates learning to a greater extent than the individual mode. The rest of the questions indicated that the children had a lot of fun while playing the game; they found the game easy to play; they would recommend the game to their friends; and they scored the game as a mean of 9.57 over 10. Finally, we believe that the combination of playful interaction and autostereoscopy is an option that should be exploited not only for the development of computer-supported learning systems, but also for the development of systems for different purposes.
Gelişen teknoloji ve bunun eğitime yansımalarıyla sürekli sayısı artan eğitsel yazılımların seçimlerinin ve etkililiklerinin belirlenmesinin eğitimde önemli bir yerinin olduğu bilinmektedir. Bu sebeple eğitsel yazılımların değerlendirilebilmeleri için farklı amaçlara yönelik ve farklı açılardan incelemelerde bulunan değerlendirme araçları geliştirilmektedir. Bu çalışmanın amacı eğitsel yazılımları değerlendirmek için geliştirilen bu araçların kapsamlı incelemesini ve değerlendirmesini yaparak gelecek çalışmalar için önerilerde bulunmaktır. Bu amaç doğrultusunda “Education Resources Information Center, Ulusal Tez Merkezi, Computers & Education” vb. veri tabanlarında, “Eğitsel Yazılım Değerlendirme Ölçeği, Eğitsel Yazılım Değerlendirme Formu, Educational Software Evaluation, Educational Software Evaluation Scale, Educational Software Evaluation Criteria” gibi anahtar kelimeleriyle taramalar yapılmış ve 81 çalışmaya ulaşılmıştır. Belirlenen kriterler neticesinde bu çalışmalardan 39 tanesi çalışma kapsamı dışında tutulmuş ve toplam 42 değerlendirme aracı incelenmeye alınmıştır. Bu incelemelerin neticesinde eğitsel yazılım değerlendirme araçlarının geliştirilmesinde 2006 – 2010 yılları arasında bir artışın olduğu tespit edilmiştir. Buna karşın literatürde eğitsel oyunların öğrenci başarısını arttırdığı birden fazla çalışmada belirtilmiş olmasına rağmen eğitsel oyunlar gibi önemli bir alanda yeterli değerlendirme aracının bulunmadığı sonucuna varılmıştır. Eğitsel yazılım değerlendirme araçlarının geliştirilmesi aşamasında ve geçerlik güvenirlik çalışmalarında da ortak bir yöntemin olmaması göze çarpmaktadır. İncelenen eğitsel yazılım değerlendirme araçlarında tespit edilen bir diğer nokta ise “İçerik, Tasarım, Öğretimsel Uygunluk” gibi kavramların ortak noktalar olduğudur. Elde edilen verilerin ışığında ve eğitsel oyunların önemi dikkate alındığında gelecek çalışmalarda bu alanda değerlendirme araçlarının geliştirilmesi önerilmektedir. Aynı zamanda geliştirilecek değerlendirme araçlarında literatürde mevcut olan örneklerde görüldüğü üzere bazı boyutlara ve kavramlara dikkat edilmesinin gerekliliği ön plana çıkmaktadır. Eğitsel yazılım değerlendirme araçlarının geliştirilmesinde görülen ve en büyük eksilik olan ortak bir yöntemin ve metodolojinin olmayışının dikkate alınması, bu alandaki eksikliği giderebilecek çalışmalar yapılması önerilmektedir. Bunların yanı sıra geçerlik ve güvenirlik çalışmalarının da yapılması araştırmacılara tavsiye edilmektedir.
This chapter looks at the implementation and evaluation of mobile learning scenarios with location-based features, done under the project COLLAGE (Collaborative Learning Platform Using Game-like Enhancements). Five scenarios are examined. Evaluating different implementations within the same project gives an appraisal of the technical and pedagogical value of the mobile learning scenarios, but also informs the research community of appropriate evaluation models and evaluation parameters, which are lacking in the game-based mobile learning with location-based components. The project responds to questions about designing mobile learning scenarios that are pedagogically sound and attractive to secondary school students, and presents the pedagogical evaluation of the effectiveness of mobile game-based learning scenarios in the COLLAGE platform. The project makes a case for a close collaboration with teachers and students in elaborating and validating complex mobile learning scenarios.
Qualität in Bildungsmaßnahmen - als Legitimation oder aus wissenschaftlichem Interesse - ist ein Thema von hoher Bedeutung, auch - und gerade - im E-Learning. E-Learning, als eine noch relativ junge, nicht völlig etablierte, aber - so stellt sich immer häufiger heraus – kostenintensive Bildungsform, muss sich immer wieder der kritischen Frage nach der Qualität stellen, zumeist assoziiert mit Lernerfolg. Vielfältige Instrumente, um die Qualität zu überprüfen oder zu entwickeln, begleitend oder im Nachhinein, sind bislang konzipiert worden (vgl. Ehlers/Pawlowski/Goertz 2003). Der Artikel gibt einen Überblick zum derzeitigen Stand der Dinge in Bezug auf zwei dieser Verfahrensweisen: zum einen geht es dabei um die Beurteilungen anhand von Kriterienkatalogen bzw. Checklisten und zum anderen um Evaluationsverfahren. Obwohl vielfach bereits umfassendere Managementsysteme zum Einsatz kommen, die die Qualität bei Anbietern kontinuierlich verbessern sollen, haben beide Verfahren eine steigende Bedeutung - da auch Managementverfahren immer wieder auf Kriterien oder Evaluationsverfahren zurückgreifen.
Qualität in Bildungsmaßnahmen – als Legitimation oder aus wissenschaftlichem Interesse – ist ein Thema von hoher Bedeutung, auch – und gerade – im E-Learning.1 ELearning, als eine noch relativ junge, nicht völlig etablierte, aber – so stellt sich immer
häufiger heraus – kostenintensive Bildungsform, muss sich immer wieder der kritischen
Frage nach der Qualität stellen, zumeist assoziiert mit Lernerfolg.2 Vielfältige Instrumente, um die Qualität zu überprüfen oder zu entwickeln, begleitend oder im Nachhinein, sind bislang konzipiert worden (vgl. Ehlers/Pawlowski/Goertz 2003).
Der Artikel gibt einen Überblick zum derzeitigen Stand der Dinge in Bezug auf
zwei dieser Verfahrensweisen: zum einen geht es dabei um die Beurteilungen anhand
von Kriterienkatalogen bzw. Checklisten und zum anderen um Evaluationsverfahren.
Obwohl vielfach bereits umfassendere Managementsysteme zum Einsatz kommen, die
die Qualität bei Anbietern kontinuierlich verbessern sollen, haben beide Verfahren eine
steigende Bedeutung – da auch Managementverfahren immer wieder auf Kriterien oder
Evaluationsverfahren zurückgreifen.
Educational environments bring new challenges for their design and evaluation, particularly in considering their interfaces. One issue that has already been identified is the importance of the teacher's involvement. This article presents a proposal based on Semiotic Engineering theory, the Semiotic Inspection Intermediated Method (SIIM). The
method combines two known methods, Semiotic Inspection Method (SIM) and Underlying Discourse Unveiling Method (UDUM) to enable the evaluation of an educational environment by an evaluator from the point of view of the teacher.
Grading App is a cross-platform desktop application for Windows, Linux, Mac OS X for evaluating student assignments and providing them with a grade and detailed feedback on TITANium. The Grading App can be used for any course in any department. The faculty used the app successfully to grade three sets of assignments (in Fall 2015) and two sets (in Spring 2016) and her grading time was reduced by 80% as compared to manual grading, while providing detailed feedback to the students.
Purpose: Information & Communication Technologies (ICT) can be a fundamental aid for the design of new teaching methods that better adapt to the framework of the European Higher Education Area. In this context, this study aims to develop and assess a complex and truly interactive ICT-based teaching tool for instruction in OM.Design/methodology/approach: A multimedia application for Just-in-Time (JIT) / Lean Production has been conceived, designed and assessed. A constructivist focus was followed in its conception and design to encourage active and flexible learning adapted to each individual’s own requirements. Using empirical research the tool has been assessed by students and compared to the traditional teaching methods.Findings: The interactive multimedia application has been clearly valued for the way it conveys information and for its usability, for the way the application is structured and the improvements to students’ understanding of the knowledge. Students are also in favour of ICT being incorporated into teaching over more traditional methods. The assessment took students’ gender and the average overall mark on their academic records as control variables but, broadly-speaking, no significant differences were found. Research limitations/implications: The study was carried out in a controlled environment and not in the normal on-site university teaching process. Conclusions could be extended to OM and other related subjects, especially if they make use of similar tools to the one described in this paper. Practical implications: This study provides a contribution that allows reflections to be made on the design of specific software for OM and students’ perceptions when using it.Originality/value: Through this paper we contribute to an improvement in learning methods in general and to higher education in OM in particular.
Most learners in resource-constrained environments own mobile phones that they could use to learn programming while outside the classroom. However, limitations of mobile phones, such as small screens and small keypads, impede their use as typical programming environments. This study proposed that programming environments on mobile phones could include scaffolding techniques specifically designed for mobile phones, and designed based on learners' needs. Scaffolding should be designed with some essential techniques that are mandatory for learners to use. Hence, one type of scaffolding technique that was designed to support programming on the mobile phone is static scaffolding that does not fade. Experiments were conducted with 64 learners of programming from three universities in Kenya and South Africa in order to investigate how they used the designed static scaffolding techniques to construct Java programs on a mobile phone. The results show that programming on mobile phones can be supported by providing scaffolding techniques that never fade, in order to address the limitations of mobile phones and to meet learners' needs.
Do-It-Yourself Electronic Lectures (DIYELs) are intended to allow academics to easily and effeciently convert lecture material in Microsoft Powerpoint into an e-lecture that can be delivered to students on their own computer. Our current design of DIYELs is described, and how these can be created. Example tricks when using Powerpoint are provided to illustrate how to increase the ease and efficiency of DIYEL production. The potential, global market for e-lectures is discussed.
Mit diesem Beitrag möchten wir ein heuristisches Modell zur Bewertung von Bildungssoftware vorschlagen. Es handelt sich dabei um Zwischenergebnisse aus einem zweijährigen Forschungsprojekt „Didaktik von Bildungssoftware“ (Okt. 1991 bis Sept. 1993), das die Autorinnen im Auftrag des österr. Bundesministeriums für Wissenschaft und Forschung, Abteilung Hochschuldidaktik, durchführen.
Software that is meant to help an educational process can be considered successful only if it is accepted and approved by the interacting targeted participants. For educational software, there are two basic groups of users who use it, namely instructors and students. As a result, in this chapter, we try to find out how helpful our mobile learning software might be for human instructors and also how educationally beneficial it is to their students. On a second level, our evaluation is targeted to measure the effectiveness of a mobile approach to learning compared with the "traditional" computer based, e-learning process.
Revolutionary advancements have been observed in e-learning technologies though an amalgamated evaluation methodology for new generation e-learning content development tools is not available. The evaluation of educational software for online use must consider its usability and as well as its pedagogic effectiveness. This study is a first step towards the definition of criteria for evaluating e-learning tools. A preliminary user study involving a group of pre-service instructional designers, observed during their interaction with e-learning tools, is reported. Throughout the study, specific usability attributes of these e-learning tools were identified. Participants were assigned to rate the importance of functional and pedagogical competences proposed during the criteria development phase. The findings of the study revealed 31 evaluation criteria under the headings of technical, media, and assessment competences. Among the groups of benchmarks proposed and rated by the users, assessment was considered as the most important one while technical and media features were even. The following step was actual implemention of the usability criteria into evaluation of fifteen leading software used in e-learning across the world. Mostly, tools were observed as having limitations in terms of capabilities. Comparing to the other software, Captivate, Softchalk, and Lectora were regarded as outstanding tools by the participants. Following the discussion on the limitations of the study, some implications for further research were proposed.
The paper describes the development of a research-based framework that can be used to design the online component of a hybrid engineering course. This framework was tested in an undergraduate course entitled Construction Equipment and Heavy Construction Methods. The multimedia design decisions are grounded on evidence-based findings from cognitive theories and experimental evaluation studies. Instructional design decisions are matched with course objectives and supportive findings from the literature review. The online component is designed as a series of interactive modules that teach students how to solve engineering problems with definitive answers. Each module is a stand-Alone unit that can be used as preparation for the onsite portion of the class, after the student has completed reading the related chapter in the text book. The modules are used to prepare students for solving open-ended questions in a face to face part of the course. Also, the module can serve as a study guide for review purposes. In the modules, students view the video demonstrations of how to solve problems and then practice problem solving step by step with verification (correct/incorrect) and explanatory program feedback presented for each step. The pilot development of the modules with six participants was conducted, and a plan for the refinement of the modules design was laid out. The educational design research methodology was used, which not only targets the problem of designing an online component for this specific course, but at the same time seeks new knowledge that can inform the work of others facing similar problems. The lessons learned from this experience and the future directions that the authors intend to take are presented in this paper.
Experimental results of usability evaluation of a distance learning system are presented in this article. An experiment is described that took place in the frame of a University course. The main goal of the experiment was to evaluate the usability of the Testing and Self-evaluation component of the system. A complementary research goal was to explore the eventual impact of system usability on student performance. For this purpose, two alternative software components were compared that shared similar functionality, implemented in different ways (IDLE, WebCT). The usability evaluation was based on user questionnaires. From this experiment correlation between the software usability and student performance has emerged, underlining the importance of usability evaluation of systems supporting distance learning.
Flash memory game based on multitouch technology is designed with richness of artistic effects which enhances children learning experience. It offers an opportunity for children to learn more effectively in a fun and enjoyable environment which improve their thinking, memorization and social skills. The objective of this paper is to provide an evaluation of children learning experience of multitouch flash memory game. An active learning experience offered by this game may contribute to the right brain functions development, thus unlocked children's innate intelligence. This preliminary study was conducted with four preschool children. Our results shown that the children had an average learning experience with the game. They had exhibited strong interest at playing the game but the excitement wore off gradually over time. This study had shown that the children exposure to such game in their education setting remains a challenge today.
The design of learning materials and researching their efficacy involves the application of both theoretical learning principles and ways of working or practices to move towards evidence based improvement. This paper abstracts 4 categories from our on-going work of educational technology research which we have found to be important in considering what constitutes a successful Technology-Enhanced Learning implementation. These considerations influence the likelihood or feasibility of the wider adoption a particular Technology-Enhanced Learning implementation in the longer term. We also discuss how these considerations relate to the scalability of the development.
This article presents the structure and theoretical foundations of the Learning Object Review Instrument (LORI), an evaluation aid available through the E-Learning Research and Assessment Network at http://www.elera.net. A primary goal of LORI is to balance assessment validity with efficiency of the evaluation process. The instrument enables learning object users to create reviews consisting of ratings and comments on nine dimensions of quality: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance. The article presents research and practices relevant to these dimensions and describes how each dimension can be interpreted to evaluate multimedia learning resources.
The use of computers in the teaching of statistics is receiving increasing attention from teachers and researchers (Shaughnessy, Garfield, & Greer, 1996). The introduction of computers is encouraged in different curricula, such as in the "Standards" of the National Council of Teachers of Mathematics (1989) and the "Diseño Curricular Base" in Spain (M.E.C., 1989), not only to extend what mathematics is taught, but also to affect how that mathematics is learned. However, there is still scarce research reported concerning the introduction of computers for teaching statistical concepts. This paper presents the results of an experimental research project that investigated the effects of a computer-based teaching environment on students' understanding of statistical association. The experimental sample consisted of nineteen, 20 year-old university students who were enrolled in their first year course on exploratory data analysis and descriptive statistics. The teaching experiment included 21, 1.5 hour sessions. For seven of the sessions, the students worked in the statistical laboratory, solving problems whose solution required them to analyze different datasets provided by the teacher or collected by themselves. The planning and instruction involved the organization of an instructional sequence to meet the learning goals and contents, the selection of appropriate datasets, and a sequence of problems with increasing difficulty, including the main task variables relevant to understanding association. The changes in the students' conceptions were assessed using two different approaches. For one approach, two equivalent versions of a questionnaire were given to the students as a pretest and posttest instrument. For the second approach, the interactions of two students with the computer were recorded and analyzed together with their written responses and their discussions during the problem-solving process. As a result, we identified some resistant statistical misconceptions as well as different acts of understanding concerning statistical association, which these students demonstrated during their learning process. All of these results are presented below, starting with a summary of our previous research on students' preconceptions and strategies concerning association (Batanero, Estepa, Godino, & Green, 1996; Estepa & Batanero, 1996), which was conducted on the experimental sample as well as on an additional sample of 213 students. This additional sample was taken to compare how typical the experimental students' responses were with regard to a more typical group.
Develops generalizations for empirical evaluation of software based upon suitability of several research designs--pretest posttest control group, single-group pretest posttest, nonequivalent control group, time series, and regression discontinuity--to type of software being evaluated, and on circumstances under which evaluation is conducted. (MBR)
This paper critically examines current thinking about whether learning computer programming promotes the development of general higher mental functions. We show how the available evidence, and the underlying assumptions about the process of learning to program, fail to address this issue adequately. Our analysis is based on a developmental cognitive science perspective on learning to program, incorporating developmental and cognitive science considerations of the mental activities involved in programming. It highlights the importance for future research of investigating students' interactions with instructional and programming contexts, developmental transformations of their programming skills, and their background knowledge and reasoning abilities.
Describes evaluations of CAL (computer-assisted learning) at the Open University (Great Britain). Highlights include evaluation of science teaching; evaluation of the Home Computing Policy; CAL evaluation in the literature; and a framework for a principled approach to evaluation, including interactions with software, outcomes, and a case study. (LRW)
This paper reviews the work of a team over two and a half years whose remit has been to “evaluate” a diverse range of CAL—computer assisted learning—in use in a university setting. It gives an overview of the team's current method, including some of the instruments most often used, and describes some of the painful lessons from early attempts. It then offers a critical discussion of what the essential features of the method are, and of what such studies are and are not good for. One of the main conclusions, with hindsight, is that its main benefit is as integrative evaluation: to help teachers make better use of the CAL by adjusting how it is used, rather than by changing the software or informing purchasing decisions.
Editor's Notes : The computer is no longer one option, or one choice of medium among others in a media mix as it was considered to be a few years back, when computers were primarily used for word processing, e-mail, and computer conferencing. It has now become the vehicle for multimedia. It is able to handle both signs, visuals and sounds, are no longer a 'machine' like radio and television but a 'medium' using digital communications system. WorldWide Web (WWW) and CD-ROM are being used for production of learning materials for the computer. Both of them also serve as a medium for storage of huge amounts of data (information). They are specially important for the transference of knowledge-learning. The hyperstructure of these media supports the interaction between the learner and the learning material, while at the same time, the learner remains in a position where he/she controls the learning process. We would like students of distance education to get more insights on resource-based learning. To achieve this objective, we selected t:tis article which is a case study of U.K. Open University Students registered,fn* the course 'Information Technology and Society' which was presented for the first time in 1995. The students' experiences and their attitudes towards resource-based learning provided through the computer have been studied through a survey of students' from two tutorial groups on completion of the above mentioned course. The move towards resource-based learning reflects a trend away from transmissive models of education, towards a more student-centred approach, where the individual has the responsibility for finding and synthesising relevant information from a variety of sources.
This paper summarizes the results of a recent evaluation study on the use of computer assisted learning at the Open University. Our main finding is that students have a realistic view of the educational benefits of computer assisted learning and also of the practical problems associated with its use. These include: physical access, a variety of user interface standards and frustrating computer experiences leading to a fear of its use. We conclude by making some recommendations which would allow computer assisted learning to achieve its potential in the Open University and in distance learning generally.
At the Open University (OU), the perception of computer assisted learning (CAL) is shifting from its current position as an optional extra in a battery of course components to that of a key teaching medium. This paper presents the authors’ experiences of the evaluation of a range of current OU produced computer assisted learning materials, mainly but not exclusively in science and technology reported in terms of evaluation framework. These materials include interactive tutorial packages, simulations used largely in a support role to the main teaching and multimedia packages which are expected to carry more of the initial presentation of key ideas in the courses. The overall purpose of the evaluations was to gain insight into how computers can have an impact on learning, particularly of science and technology subjects, and to explore a portfolio of evaluation techniques to be used in the development and assessment of computer based learning materials. In this paper the techniques are discussed and illustrated with reference to a number of case studies of a range of programs used by students. Some conclusions are drawn about how the use of programs leads to the support of students’ science and technology learning.
The evaluation of the use of educational software must take account of usability as well as learning and, crucially, the integration of tissues concerned with usability and learning. Checklists, a very common approach to the predictive evaluation of educational software, do not take account of this integration. A review of two well known checklists illustrate this. A structured evaluation model known as the Jigsaw Model is described. Empirical evidence is provided to demonstrate that teachers' awareness of integration can be greatly improved by using the model.
Many teaching practices implicitly assume that conceptual knowledge can be abstracted from the situations in which it is learned and used. This article argues that this assumption inevitably limits the effectiveness of such practices. Drawing on recent research into cognition as it is manifest in everyday activity, the authors argue that knowledge is situated, being in part a product of the activity, context, and culture in which it is developed and used. They discuss how this view of knowledge affects our understanding of learning, and they note that conventional schooling too often ignores the influence of school culture on what is learned in school. As an alternative to conventional practices, they propose cognitive apprenticeship (Collins, Brown, & Newman, in press), which honors the situated nature of knowledge. They examine two examples of mathematics instruction that exhibit certain key features of this approach to teaching.
Developing interactive multimedia courseware and networks: some current issues Interactive Multimedia: Practice and promise
- A Romiszowski
- C Latchem
- J Williamson
- L Henderson
A. Romiszowski, Developing interactive multimedia courseware and networks: some current issues. in: C. Latchem, J. Williamson, L. Henderson-Lancett (Eds.), Interactive Multimedia: Practice and promise, Kogan Page, London, 1993, pp. 57–78. A. Jones et al. / Interacting with Computers 11 (1999) 499–516 516
Evaluation as illumination: a new approach to the study of innovatory programs (originally published as Occasional Paper Number Nine, University of Edinburgh Centre for Research in the educational Sciences Introduction to Illuminative Evaluation: Studies in Higher Education
- M Parlett
- D Hamilton
M. Parlett, D. Hamilton, Evaluation as illumination: a new approach to the study of innovatory programs (originally published as Occasional Paper Number Nine, University of Edinburgh Centre for Research in the educational Sciences, 1972), in: M. Parlett, G. Dearden (Eds.), Introduction to Illuminative Evaluation: Studies in Higher Education, Guildford, SRHE, University of Surrey, 1981.
Choosing and Using Educational Software: A Teachers' Guide
- D Squires
- A Mcdougall
D. Squires, A. McDougall, Choosing and Using Educational Software: A Teachers' Guide, The Falmer Press, London, 1994.
Methodological considerations for investigating computer-supported collaborative learning
- K Issroff
- E Scanlon
- A Jones
K. Issroff, E. Scanlon, A. Jones, Methodological considerations for investigating computer-supported collaborative learning, in: Proceedings of WICCE '95, Birmingham, UK, 1995.
Anchored instruction: why we need it and how technology can help Advances in computer-video technol-ogy, computers, cognition, and multimedia: explorations in high technology
- J D Bransford
- R S Sherwood
- T S Hasselbring
- C K Kinzer
- S M Williams
J.D. Bransford, R.S. Sherwood, T.S. Hasselbring, C.K. Kinzer, S.M. Williams, Anchored instruction: why we need it and how technology can help, in: D. Nix, R. Spiro (Eds.), Advances in computer-video technol-ogy, computers, cognition, and multimedia: explorations in high technology. Lawrence Erlbaum Associ-ates, Hillsdale, 1990, pp. 115–141.
- A Jones
- E Scanlon
- C Tosunoglu
- S Ross
- P Butcher
- J Murphy
- Greenberg
A. Jones, E. Scanlon, C. Tosunoglu, S. Ross, Butcher, P. Murphy, J. Greenberg, Evaluating CAL at the
Open University: 15 years on, Computers and Education 26 (1-3) (1996) 5–15.
Personal Computers for Distance Learning
- A Jones
- G Kirkup
- A Kirkwood
Advances in computer-video technology, computers, cognition, and multimedia: explorations in high technology
- J D Bransford
- R S Sherwood
- T S Hasselbring
- C K Kinzer
- S M Williams
Supporting Resource Based Learning
- J Taylor
- D Laurillard
A formative evaluation of the program
- E J Morris
Developing interactive multimedia courseware and networks: some current issues
- A Romiszowski