No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A Convergent Participation Model for Evaluation of Learning Objects
Abstract
The properties that distinguish learning objects from other forms of educational software - global accessibility, metadata standards, finer granularity and reusability - have implications for evaluation. This article proposes a convergent participation model for learning object evaluation in which representatives from stakeholder groups (e.g., students, instructors, subject matter experts, instructional designers, and media developers) converge toward more similar descriptions and ratings through a two-stage process supported by online collaboration tools. The article reviews evaluation models that have been applied to educational software and media, considers models for gathering and meta-evaluating individual user reviews that have recently emerged on the Web, and describes the peer review model adopted for the MERLOT repository. The convergent participation model is assessed in relation to other models and with respect to its support for eight goals of learning object evaluation: (1) aid for searching and selecting, (2) guidance for use, (3) formative evaluation, (4) influence on design practices, (5) professional development and student learning, (6) community building, (7) social recognition, and (8) economic exchange.
... Due to the nature of the research aim and questions, the study followed a summative quantitative external evaluation approach (i.e. the first of the aforementioned approaches to evaluation), using a standard closed response instrument and 'employing' PE teachers as evaluators. This type of evaluation is deemed to be efficient and objective (Williams 2000;Nesbit et al. 2002). Furthermore, such evaluation activities are considered to contribute to the professional development of the participating teachers (Nesbit et al. 2002), who are the primary prospective users of LOs (Williams 2000), and to build communities of practice among them on the basis of the LOs (Nesbit et al. 2002). ...
... This type of evaluation is deemed to be efficient and objective (Williams 2000;Nesbit et al. 2002). Furthermore, such evaluation activities are considered to contribute to the professional development of the participating teachers (Nesbit et al. 2002), who are the primary prospective users of LOs (Williams 2000), and to build communities of practice among them on the basis of the LOs (Nesbit et al. 2002). ...
... This type of evaluation is deemed to be efficient and objective (Williams 2000;Nesbit et al. 2002). Furthermore, such evaluation activities are considered to contribute to the professional development of the participating teachers (Nesbit et al. 2002), who are the primary prospective users of LOs (Williams 2000), and to build communities of practice among them on the basis of the LOs (Nesbit et al. 2002). ...
The aim of this study was twofold: (a) the design and development of a set of open, interactive, digital Learning Objects (LOs) for high school Physical Education (PE), and (b) the evaluation of the newly-created LOs by PE teachers, experienced both in teaching PE and in Information and Communication Technology (ICT), as to their quality and usefulness, before the dissemination of the use of the LOs in schools. The design and development framework, which was included in an innovative national project, is first presented, followed by descriptions of the 19 LOs that were produced, grouped in two categories (instructivist-oriented and constructivist-oriented LOs). Thirteen (13) of the LOs were subjected to external, summative evaluation by 19 qualified PE teachers, each of whom evaluated all 13 LOs through completing an online evaluation form for each. The form consisted of a standard LO evaluation instrument comprising 4 criteria of quality (interactivity, design, engagement, usability), and of an additional item assessing the usefulness of the LO in the teaching of PE. The LOs were found to be adequate in terms of interactivity, design, engagement, usability and usefulness, which corroborates their suitability for use in schools. Instructivist-oriented LOs and constructivist-oriented LOs were not found to differ significantly in terms of design and usability. However, the latter were found to be significantly more interactive, engaging, and useful in PE teaching than the former. The study provides useful guidance for the future design and development of LOs for PE.
... El diseño del prototipo ha tenido en cuenta las aportaciones teórico-práctica de los siguientes modelos: constructivismo y conectivismo (Simmons, 2002, Siemens, 2005, Downes, 2005, que han contribuido a la definición de las distintas capsulas de aprendizaje; cognitivismo, sobre todo la teoría de la carga cognitiva (Sweller, 1988), a tono con los principios del microaprendizaje; el modelo de diseño instruccional ADDIE (Mullins, 2017;Guo-Hua, 2019;Budoya et al., 2019), utilizado para elaborar la app mediante las fases de análisis, diseño, desarrollo, implementación, y evaluación; and los modelos centrados en el usuario como (Nesbit et al., 2002;Nokelainen, 2006;Pinto et al., 2017;Hartson and Pyla, 2019). También, se han tenido en cuenta los resultados recogidos en algunos focus groups realizados a estudiantes y profesores (Gretter y Yadav, 2018;Sales et al., 2020). ...
... Del mismo modo, el diseño instruccional ADDIE de Mullins (2017); Guo-Hua (2019) y Budoya et al. (2019) han contribuido a los contenidos. La consolidación de una estructuración sólida, coherente y útil garantiza la fortaleza de los contenidos (Nesbit et al., 2002;Nokelainen, 2006;Pinto et al., 2017). Al mismo tiempo, contribuye al éxito y difusión de los aprendizajes. ...
Objetivo. Crear, evaluar e implementar la eficacia de la aplicación MoILCaps en el contexto de la educación superior. Con fundamento en las teorías cognitiva, constructivista y conectivista, se ha desarrollado una app con un modelo de diseño instruccional basado en la experiencia del usuario. Está destinada al autoaprendizaje por parte del estudiantado de Ciencias Sociales. Método. Con la colaboración tanto del profesorado como del estudiantado se elaboró la aplicación, siguiendo el modelo Análisis, Diseño, Desarrollo y Evaluación (ADDIE), por medio de las fases de análisis, diseño, desarrollo, implementación y evaluación. La última fase proporcionó las propuestas de mejora para la optimización de la versión final de la herramienta, una web progresiva de acceso abierto. Resultados. La aplicación desarrollada se organiza en seis cápsulas que siguen el marco de ACRL (Association of College and Research Libraries) para la alfabetización informacional en educación superior (2015): aprender, buscar, evaluar, crear, investigar y difundir. Incluye recursos multimedia en forma de microcontenidos que ponen en relieve la legibilidad, organización y visualización como características claves. Este modelo de app centrada en la experiencia del usuario constituye un instrumento relevante para fomentar la docencia de ALFIN (Alfabetización Informacional) por parte del profesorado y la biblioteca. Conclusiones. Los diferentes análisis, seguidos de las propuestas de mejora y las revisiones, conllevaron la consecución de una aplicación de gran utilidad para el estudiantado, el profesorado y la biblioteca. Las bibliotecas, junto al profesorado, juegan un papel crucial como agentes activos en la formación de la alfabetización informacional, que se ha de adaptar a la incorporación de las tecnologías móviles en respuesta a las necesidades de la sociedad actual.
... En relación a su calidad, además de posibilitar el logro objetivos de aprendizaje, Nesbit et al. (2002) destacan otros beneficios que traen consigo los procesos de evaluación: ayudan a los usuarios a encontrar OA en los repositorios, orientan sobre la mejor manera de emplearlos, pueden mejorar la calidad de los OA de manera temprana mediante evaluaciones en etapas de diseño y desarrollo, mejoran el desarrollo profesional de quienes participan en actividades de evaluación, los estándares de calidad guían a diseñadores y desarrolladores, las habilidades de éstos pueden ser reconocidas socialmente frente a evaluaciones positivas, entre otras. ...
... refiere a "Validez"; y "Conf." a "Confiabilidad". LORI (Learning Object Review Instrument) (Nesbit et al., 2002(Nesbit et al., , 2009Leacock y Nesbit, 2007). Evalúa 9 heurísticas: Calidad del contenido; Motivación; Alineación de los objetivos de aprendizaje; Adaptación y retroalimentación; Diseño de presentación; Usabilidad de la interacción; Accesibilidad; Reusabilidad; Conformidad con estándares de metadatos e interoperabilidad. ...
En este trabajo se presenta una revisión exhaustiva de instrumentos de evaluación de calidad de objetos de aprendizaje propuestos en la literatura con el objetivo de determinar si es necesario elaborar un nuevo instrumento que contemple necesidades no previstas en aquéllos; o si, por el contrario, alguno puede ser adoptado para valorar esta clase particular de recursos educativos en la Universidad Tecnológica Nacional de Argentina. El instrumento a ser elegido debe guardar un adecuado equilibrio entre cobertura, extensión y facilidad de aplicación, a la vez que dar garantías de confiabilidad y validez. Del análisis de resultados se concluyó que el modelo de la norma de la Agencia Española de Normalización, UNE 71362 Calidad de los materiales educativos digitales, verifica los requisitos de selección.
... This can be done by using both technological and social approaches. Social tools include user generated collaborative quality methods such as rating (Nesbit, Belfer & Vargo, 2002), and technological tools that automatically check for indicators of quality reflected in the interface (Clements, Pawlowski & Manouselis, 2015). ...
... A number of the issues represented in the affinity diagram also reaffirm what has been seen in the literature, presented in chapter 2. For example, participants brought up views regarding ownership of material (see Bates et al., 2007). Additionally, participants emphasized the importance of maintaining quality of the resources (Nesbit, Belfer & Vargo, 2002;Clements, Pawlowski & Manouselis, 2015), and shared several ideas of how this could be done. ...
The Association for Computing Machinery (ACM) Special Interest Group on Computer-Human Interaction (SIGCHI) has been supporting research into Human-Computer Interaction (HCI) education for many years, most actively in the last six years. This thesis proposes the preliminary framework for a Community of Practice (CoP) of HCI scholars and educators, sharing and collaborating to develop teaching materials. Two studies were carried out. The first involved a set of 12 interviews with HCI educators and practitioners that examined requirements for the HCI Living Curriculum. The second study was a questionnaire with 28 respondents which carried out further requirements analysis. Based on the findings of the two studies, use cases and design requirements for the HCI Living Curriculum are presented. Guided by these use cases and design requirements, a prototype HCI Living Curriculum was then constructed. The thesis concludes with a discussion of lessons learned and prospects for future development of the Living Curriculum.
... Evaluation serves several different purposes. Nesbit, Belfer, and Vargo (2002) list eight purposes for developing effective learning object evaluation systems. In Table 1, we connect articles from the field to these purposes. ...
... Purposes for developing effective learning object evaluation systems, fromNesbit et al. (2002) ...
In this article, we describe the development of a classification system providing a framework for analysis of, and communication about, a subgroup of learning objects. The objects we consider are highly visual, animated, interactive, and mathematics-related, and we call them VaniMaps. Secondly, we discuss the use of the system. In the first phase, the development was based on literature studies and discussions on examples of VaniMaps. In the second phase, the classification system was tested by students and their responses were analyzed to identify possible improvements. Now, the system is developed further based on experience gained while using it for different purposes. We see several possible uses of the classification system, or selected parts of it: (a) to facilitate communication between the orderer and the developer, (b) to initiate discussions on VaniMaps in teacher education, (c) to analyze and choose between VaniMaps for teaching and learning activities, and (d) to establish a database for VaniMaps labeled using classification statements. We will discuss all these uses and especially emphasize the use in teacher education, illustrated with a case study.
... learning objects is the Learning Object Review Instrument (LORI) evaluation tool. A number of works developed within the international scope, such as in Canada and in the United States, have already used the LORI evaluation tool, which was developed to search for quality in a learning object made available at the Internet (computers world network) (Nesbit et. al., 2002;Krauss & Ally, 2005). ...
... In order to achieve the goals in this research, the VLOs were evaluated following the LORI methodology (Nesbit, Belfer & Vargo, 2002;Nesbit, Belfer & Leacock, 2004;Leacock & Nesbit, 2007). ...
p class="3">The constant technological development in education, and the potentiality of the resources offered by Information and Communication Technologies (ICTs), are challenges faced by teaching institutions in Brazil, especially by those institutions, which by the very nature of their services intend to provide distance education courses. In such a scene, one sees the use of technology as a tool to give support and to take part in the process of teaching activities, such as the Virtual Learning Objects (VLOs), which offer an opportunity to contribute to the teaching and learning process. Considering this, the present work aims at analyzing the VLOs used in the distance education courses of Economic Sciences and of Accounting at the Universidade Federal de Santa Catarina (Federal University of Santa Catarina), under the quality criteria indicated by the Learning Object Review Instrument (LORI) methodology proposed by Nesbit, Belfer, and Vargo (2002), Nesbit, Belfer, and Leacock (2004), and Leacock & Nesbit (2007), in order to learn how to better take profit of efforts and resources.</p
... Moreover, different evaluators can also give more importance to one specific dimension than to the others. In order to soften this situation, Nesbit et al. [9] propose applying LORI through the use of a convergent model, where several evaluators from distinct areas (instructors, instructional designers, and multimedia developers) collaborate to achieve a single and unique quality rating for a given resource. In fact, this concept was applied in eLera as it will be shown in next section. ...
... In Organic.Edunet, quality is assured by the community of users who are allowed to rate the resources, and to improve their metadata translations (the portal is multilingual, the interface is available in nine (9) languages and the metadata about the resources is manually translated in up to eight (8) languages)(see Figure 4). Moreover, any user can give direct feedback about a given resource to the portal as well as to report inappropriate contents. ...
Assessing quality of learning objects (LOs) is a difficult and complex task that normally revolves around multiples and different aspects that need to be addressed. Nowadays, quality assessment of LOs inside repositories is based on the information provided by the same community of users and experts that use such platforms. These kinds of information are known as evaluative metadata and constitute a value body of knowledge about Los that is normally used inside the repositories during the process of searching and retrieving. The present work aims to present a brief overview about how LOs quality is being assessed inside some of the most important repositories existent nowadays, as well as some limitations of these existing approaches.
... In a recent review of 58 articles on learning objects (Kay & Knaack, 2007a), only eight evaluated the actual use of learning objects (Adams, Lubega, Walmsley, & Wil-liams, 2004; Bradley & Boyle, 2004;Cochrane, 2005;Kenny, et al., 1999;Krauss & Ally, 2005;Macdonald et al., 2005;Nesbit, Belfer, & Vargo, 2002;Van Zele, Vandaele, Botteldooren, Lenaerts & 2003). Various methods of evaluation were used including informal or qualitative feedback (Adams et al., 2004;Bradley & Boyle, 2004;Cochrane, 2005;Macdonald et al., 2005), descriptive analysis (Krauss & Ally, 2005;Macdonald et al., 2005), convergent participation (Nesbit et al., 2002), formal surveys (Cochrane, 2005;Krauss & Ally, 2005), and learning outcomes (Adams et al., 2005;Bradley & Boyle, 2004;Macdonald et al., 2005;Van Zele et al., 2003). ...
... In a recent review of 58 articles on learning objects (Kay & Knaack, 2007a), only eight evaluated the actual use of learning objects (Adams, Lubega, Walmsley, & Wil-liams, 2004; Bradley & Boyle, 2004;Cochrane, 2005;Kenny, et al., 1999;Krauss & Ally, 2005;Macdonald et al., 2005;Nesbit, Belfer, & Vargo, 2002;Van Zele, Vandaele, Botteldooren, Lenaerts & 2003). Various methods of evaluation were used including informal or qualitative feedback (Adams et al., 2004;Bradley & Boyle, 2004;Cochrane, 2005;Macdonald et al., 2005), descriptive analysis (Krauss & Ally, 2005;Macdonald et al., 2005), convergent participation (Nesbit et al., 2002), formal surveys (Cochrane, 2005;Krauss & Ally, 2005), and learning outcomes (Adams et al., 2005;Bradley & Boyle, 2004;Macdonald et al., 2005;Van Zele et al., 2003). ...
In the past 10 years, a considerable amount of money and effort has been directed toward distance education, with growth estimated as high as 30%-40% annually (Harper, Chen, & Yen, 2004; Hurst, 2001; Newman, 2003). The popularity of distance learning appears to be founded on personal control over instruction (Burgess & Russell, 2003, Pierrakeas, 2003), the variety of multimedia formats available to students (Hayes & Jamrozik, 2001), and customized support (Harper et al., 2004). However, the success of distance education is anything but a foregone conclusion. Multiple obstacles have impeded acceptance including reluctance to use technology (Harper et al., 2004), time required to develop course resources (Harper et al., 2004; Hayes & Jamrozik, 2001) and to support students (Levine & Sun, 2002), lack of technology skills (Berge & Smith, 2000), and cost (Burgess & Russell, 2003; Levine & Sun, 2002). In addition the promise of interactivity and constructive learning in distance learning has not been realized. Most distance learning offerings resemble traditional classroom courses (Coates & Humpeys, 2003; Levine & Sun, 2002, Navaro, 2000). When interaction does take place, it is usually in the form of online discussion, however, a number of studies have reported that true social interaction leading to cognitive development is rare (e.g., Berge and Muilenburg, 2000; Bisenbach-Lucas, 2003; Garrison, Anderson, and Archer, 2001; Hara, Bonk and Angeli, 1998; Meyer, 2003; Wickstrom, 2003).
... • LORI: Stands for Learning Object Review Instrument. It was developed in 2002 by academic researchers from the University of California and Simon Fraser University of Canada [14]. ...
... "Evaluation is the process of determining the merit, worth and value of things, and evaluations are the products of that process" (Vargo et al. 2003, 1). Evaluating online formative assessment strategies as learning objects is one of the most important quality assurance measures in teaching and learning online (Nesbit, Belfer, and Vargo 2002). As learning objects in the educational fraternity, online formative assessment strategies can provide pedagogically and digitally rich reusable learning environments. ...
Online formative assessment strategies are an important element in the level of engagement needed for motivating students learning online in open distance learning (ODL) because they are key in offering large-scale interactive participation. While many studies have looked at the advantages of online formative assessment strategies, they have neglected demotivating online formative assessment strategies that students struggle with. To ensure the quality of the online assessment, it is also important to understand the nature of assessment practices that demotivate students from persisting with their studies. This is because the type of online assessment that students struggle with should guide the choice and design of the online formative assessment strategies. Therefore, this article identifies online formative assessment strategies that demotivate students from participating persistently in online formative assessment in ODL in order to improve the design and development of online learning. This study used a developmental research approach to carry out a descriptive quantitative case study survey involving 112 purposefully sampled students, of which 58 responded, who were registered for a master’s in education in ODL course at an ODL university in South Africa. A thematic coding process was adopted during the analysis of students’ responses to an online Google form. Though students differed in their choices of the online formative assessment strategies that demotivated them in their studies, there was consensus on seven significant themes. The socio-technological perspective and the self-determination theory were used as the theoretical frameworks to drive the investigation since they encompassed all relevant aspects of the design of online learning and motivation to learn online. Information regarding demotivating online formative assessment strategies provide insight to course leaders and instructional designers attempting to build successful online learning strategies that motivate students to participate persistently in online learning environments.
... Para llevar a cabo la evaluación de las aplicaciones móviles relacionadas con el tema de balanceo de reacciones químicas, se realizó una búsqueda en Play Store y se seleccionaron 15 aplicaciones. Se evaluaron teniendo en cuenta aspectos enmarcados en la norma ISO/ IEC 25040 (ISO, 2021) y en Cháves et al. (2021), según se describe en la figura 1. Con el objetivo y los pasos de la evaluación definidos por Nesbit et al. (2002) y Otamendi et al. (2018), se diseñó una encuesta de evaluación para estudiantes y docentes (ver figura 2). Los docentes evaluaron las aplicaciones en primer lugar, seguidos por los estudiantes. ...
La presencia de recursos digitales como las apps o aplicaciones móviles con un propósito educativo preciso y utilizado como herramienta complementaria a la labor docente flexibiliza y dinamiza, entre otras, el proceso de enseñanza/aprendizaje. En este estudio, se seleccionaron y evaluaron diferentes apps educativas relacionadas con el tema de balanceo de reacciones químicas. Las apps mejor evaluadas por los docentes e incluidas dentro del proceso de enseñanza/aprendizaje de estudiantes universitarios fueron Chem Calc - Equation Balancer & Element Dictionary y Chemical Equation Balance.La percepción general, después del uso de estos dos recursos, fue favorable y la calificación descriptiva que les otorgaron fue: motivantes, lúdicas, que diversifican la conducta del docente en cuanto a uso de recursos se refiere, que aumentan los espacios de participación por parte del estudiante e idóneas porque ayudan a mejorar su comprensión en el tema y a verificar los ejercicios que se hubiesen realizado.
... This is important to provide designers and course leaders with feedback that could guide them into adapting the new practices aligned to the needs of students for the improvement of DOFAS (Nicholas & Thomas, 2000). Evaluating online formative assessment strategies as learning objects to "determine the merit, worth and value of things, as the products of that process", is one of the most important quality assurance measures in teaching and learning (Scriven, 1994:1;Nesbit, Belfer & Vargo, 2002). Evaluation must be an ongoing process during any stage in the design or the implementation of the learning object since it focuses on the usability and the pedagogical design of the learning object (Williams, 2000). ...
There is a growing body of research that suggests that improving the quality of online formative assessment strategies increases students’ motivation to participate in online assessment. However, the way in which course leaders at open distance learning universities communicate the expectations of learning to students through online formative assessment strategies largely determines how students approach online formative assessment tasks. The design of online formative assessment strategies is an important element that is needed to promote an interactive level of engagement needed for motivating students learning online. It is against this background that this paper investigated the experiences of students with demotivating online formative assessment strategies at an open distance learning institution. The study achieved this through phenomenography, a developmental, qualitative and non-dualistic case study research. Twelve purposively selected students from a postgraduate course were interviewed to understand their experiences with demotivating online formative assessment strategies. Students identified seven demotivating online formative assessment strategies that provided insight to course leaders in the production of more motivating assessment strategies. The study achieved this through phenomenography, a developmental, qualitative and non-dualistic research approach that used a case study, to interview twelve purposively selected students from a postgraduate course to understand their experiences with demotivating online formative assessment strategies. From the students’ own perspective, seven online formative assessment strategies that demotivated them to learn were identified to provide insight to course leaders to produce motivating assessment strategies.
... An in-depth study of the indicators proposed in these models revealed that their meanings differ depending on their author and the methodology used. While, for example, Nesbit et al. (2002) equate the "content quality" indicator with truthfulness, accuracy, a balanced presentation of ideas, and an adequate level of detail; Bravo Ramos (2005) regards content quality as both the scientific rigor of the expressive resources and techniques they use. According to Opdenacker et al. (2009), the authors of the QuADEM model, this indicator assesses the correspondence of a didactic material with the training objectives and the suitability of the material to its recipients. ...
The quality of didactic materials is a source of concern for teachers, users, and educational institutions that offer online education. There is a lack of indicators to help assess the quality of three key types of didactic materials commonly used in online education: didactic units (i.e., materials that contain program contents), didactic guides (i.e., materials that provide information), and additional didactic materials (materials to deepen knowledge). The objective of this article is to present a system of indicators designed to assess the quality of these types of didactic materials and guide their creation process. The system was developed based on a critical analysis of existing models designed to assess the quality of digital didactic materials. The system was validated by 16 international experts in online education, and a trial application of the system assessed five didactic guides and didactic units used by online universities in three different countries. Results of the validation process were triangulated with relevant literature, allowing the authors to make decisions regarding changes to the system in terms of maintaining, reformulating, or removing indicators. The resulting system comprises 43 assessment indicators and serves as a guide for designers, teachers, and users in the creation and selection of didactic materials for use in online education and in the assessment of their quality.
... We evaluated the instructional part of the MTLP using the Learning objective review instrument. 27 The participants scored the content of MTLP positively and valued the interdisciplinary approach with a mean of 4.14 (SD: 0.64) on a Likert scale of 1=totally disagree and 5=totally agree. They acknowledged the added value of a training programme that addressed their personal growth and leadership development (alignment with learning goals) with a mean score of 4.14 (SD: 0.70, see table 3). ...
As we witness an alarming change in the delivery of healthcare worldwide, there is a need for transformational leadership that can inspire and empower healthcare professionals, and patients alike. Due to the lack of economic, infrastructural and human resources especially in developing countries, the organisation and delivery of healthcare services is even more challenging. Hence, the focus of leadership development in resource-limited environments should be to facilitate knowledge building, interprofessional collaboration, empowerment and inclusion.
In this paper, we use two exemplary cases to illustrate the impact of, and the challenges facing leadership capacity building in non-Western contexts with special reference to India and Curaçao. These cases demonstrate that there is a place for validated medical leadership models in developing countries with promising outcomes for the collective health status of communities at large. Such models, however, need to be contextualised to fit the individual country's economic and sociopolitical context. Also, the stakeholders should be prepared to create healthy professional cultures that embrace respect (for self and others) and focus on effective communication practices within their local environments.
... While the moderator urged them to do their utmost in resolving differences based on domain-specific cognition, he also cautioned against undue coercion or giving in for the sole purpose of conforming rather than personal conviction. This is akin to a convergent participation model used by Nesbit, Belfer and Vargo 35 . Figure 3 shows the results of a typical "norming" session on a white board for 2 assignments. ...
... Research data sources include: the application of learning media assessmentbased material chemical reaction rate with android-based by the Validator (expert content, media experts, the chemical educators, peer reviewers), readability of media applications android on a small-scale trial. Assessment of the feasibility of using an instrument of validation of the modified eligibility (Crozat, Hu, Trigano &, 1999) by Evaluation of Multimedia and Pedagogical Interactive Software (EMPI) and Learning Objects Review Instrument (LORI) developed by (Nesbit interpreted evidence Belfer, & Vargo, 2002). The data obtained were analyzed using the validator from the data analysis of Aiken's to find out the validity of the content of an item (Aiken, 1985) States the level of quality of the media learning that qualitatively changed by conversion and reliability guidelines validation of developed media analyzed with make a score assessment service from expert lecturers, educators and peer reviewers. ...
... The criteria and questions used in the expert review for evaluating the learning objects were drawn from five main sources: the CLOE draft guidelines (in Haughey & Muirhead, 2005); the Le@rning Federation Soundness Specification (Freebody, Muspratt, 2007); the Learning Object Review Instrument (Nesbit, Belfer & Vargo, 2002) and its revised version (Li, Nesbit & Richards, 2006); Nielsen's (2003) usability guidelines as well as Lund (2001) ...
This paper reports on two open source survey tools that were used to gather data related to Athabasca University's educational development activities within a qualitative evaluation framework. A comparative review of a Moodle module questionnaire and Lime Survey aims to analyze the multiple uses of evaluative instruments as part of a broader discussion on 'utilization-focused evaluation' for Higher Education projects.
... Renata Marciniak y Joaquín Gairín Sallán. Página 9 de 30 estudio documental sobre estándares, normas, directrices, modelos y guías ya existentes para llevar a cabo la evaluación de educación virtual (Abdulaziz, Almetair y Alzeer, 2014;ADL, 2014;AENOR, 2012;AICCJC, 2013;Attwell, 2006;AVU, 2014;CALED, 2010;CEN, 2007;CIEES, 2008;EADTU, 2016;EFQUEL, 2010;Ehlers, 2013;García Aretio, 1998;Hilera y Hoya, 2010;IEEE, 2014;ISO/IEC 19796-1:2005;Kirkpatrick, 2007;Latchem, 2014;Marshall, 2007;Masoumi y Lindström, 2012;MSCHE, 2011;Marciniak, 2015;Nesbit, Belfer, y Vargo, 2002;UNIQUe, 2014;Zhang y Jiang, 2012; entre otros). A partir de dicho análisis pudimos extraer las variables y dimensiones que se deben tener en cuenta a la hora de evaluar un programa formativo impartido bajo la modalidad virtual y diseñar el primer borrador del modelo (modelo provisional I) compuesto por dos variables, catorce dimensiones y ciento veinte indicadores. ...
La mejora de la calidad de programas formativos impartidos bajo la modalidad virtual es un gran desafío para todas las universidades que los ofrecen, si consideramos las crecientes expectativas de los estudiantes y los requisitos de calidad y mejora permanente de las agencias de acreditación. Uno de los métodos que permiten identificar el nivel actual de la calidad del programa y mejorarlo es la autoevaluación que debería abarcar la evaluación de la calidad propia del programa, así como su evaluación continua. El objetivo del artículo es presentar un modelo aplicativo para la autoevaluación de programas de educación universitaria virtual, que integra los dos tipos de evaluación y de esta manera permite evaluar la calidad tanto de los elementos pedagógicos y tecnológicos que constituyen un programa virtual como cada una de las tres fases de su aplicación: la fase inicial, de desarrollo y la fase final. La validación del modelo ha incluido varios momentos e implicado a 23 expertos internacionales y su aplicación en la autoevaluación de cuatro programas virtuales. El modelo está dirigido a los responsables de la ejecución de programas formativos impartidos bajo la modalidad virtual, directores/ coordinadores de las carreras a las que pertenece el programa. Asimismo, puede servir a los diseñadores de programas virtuales como pauta para diseñar un programa virtual de buena calidad.
... a) Perspectiva sistémica, que considera cada uno de los elementos como parte de un sistema que adquiere sentido gracias a su interacción y en su contexto. Dentro de esta corriente destacan Nesbit, Belfer y Vargo (2002), que proporcionan un conjunto de ocho criterios para el desarrollo efectivo de sistemas de evaluación de objetos aprendizaje, entre los que se pueden destacar los métodos y normas de evaluación y/o los sistemas de evaluación fiables para la rentabilización económica en el intercambio de objetos de aprendizaje. b) Perspectiva heurística, centrada en los resultados más que en los métodos. ...
El objetivo de este trabajo es presentar el desarrollo de la herramienta Evaluareed, diseñada para evaluar
recursos electrónicos educativos, en especial aquellos orientados a la enseñanza superior. A partir de la revisión de la literatura científica relativa a esta cuestión y utilizando una metodología con un doble enfoque
cualitativo-cuantitativo, se definen un conjunto de criterios y subcriterios que conforman el checklist. Los
indicadores de calidad son los siguientes: 1) calidad del contenido, 2) objetivos y metas de aprendizaje, 3)
feedback, 4) usabilidad, 5) motivación, 6) accesibilidad, 7) requerimientos técnicos, 8) propiedad intelectual, y
9) efectividad del recurso desde el punto de vista del aprendizaje. El instrumento permite la evaluación de los recursos y genera un informe con la valoración de la calidad de los mismos y recomendaciones sobre su mejora. Se encuentra disponible en un portal web que proporciona además un repositorio de OCW evaluados de universidades españolas, una selección bibliográfica especializada e información relativa al desarrollo del proyecto.
... A diferencia de otros repositorios, MERLOT fue uno de los primeros en implementar un enfoque para la evaluación de la calidad de los OA sobre la base de un proceso de revisión por pares (Nesbit, Belfer, & Vargo, 2002) Hoy en día, el sitio Web de MERLOT da soporte más de veinte comunidades propias de la disciplina (por ejemplo, biología, historia, tecnología de la información, formación del profesorado, etc.). Cada comunidad disciplinaria tiene asociado un subconjunto de la colección completa de OA y cuenta con un consejo editorial que guía las políticas de revisión por pares y las prácticas en esa comunidad. ...
En el campo del e-learning se realizan grandes esfuerzos dedicados al mejoramiento del proceso de enseñanza ? aprendizaje, uno de ellos está dirigido a motivar el uso y reutilización de los recursos digitales en repositorios. En un comienzo, los esfuerzos se dirigieron a aumentar la cantidad de recursos disponibles, hoy en día existe una gran cantidad de recursos almacenados en repositorios heterogéneos, por lo tanto el desafío se traslada a mejorar y hacer más eficientes las formas de buscar, seleccionar, localizar y acceder a recursos dispersos y distribuidos en repositorios. Dentro de esta línea, el objetivo de esta tesis es proponer una estrategia para la expansión de consultas basadas en ontologías de dominio que permita al diseñador instruccional obtener, desde un repositorio, objetos de aprendizaje relevantes para el diseño de sus cursos o la composición de otros recursos más complejos. Para lograr este objetivo se analizan las propuestas de expansión de consultas ya sea en el campo de la recuperación de información en general o específicamente, en los repositorios de objetos de aprendizaje. A partir de lo anterior, se establecen los criterios para la expansión de consultas basada en ontologías, se define la forma como serán abordados los problemas detectados, y por último, se formula, diseña e implementa la estrategia de expansión de consultas basada en ontología de dominio aplicada en el contexto de la búsqueda de objetos de aprendizaje en repositorios. Para la evaluación de nuestra propuesta se diseña un experimento dentro del dominio de genética, utilizando la ontología Gene como base de conocimiento y el repositorio MERLOT como proveedor de los objetos de aprendizaje en este dominio. Las consultas de prueba se definen a partir de los contenidos tratados en un conjunto de cursos de genética publicados en la Web por instituciones de educación superior para el año 2009. La evaluación de la relevancia de los resultados es realizada por 3 expertos en el dominio. El análisis de la concordancia y asociación entre las evaluaciones de los expertos es realizado por medio del análisis de Kappa de Cohen y el coeficiente de correlación de Spearman. Finalmente, la efectividad de la propuesta de expansión se evalúa a partir de las métricas de cobertura y novedad aplicadas a los resultados recuperados de las consultas con y sin expansión. La principal aportación de nuestra propuesta es una estrategia para la expansión de consultas basada en ontologías de dominio que permita al diseñador instruccional obtener resultados relevantes que sin la expansión no podrían ser recuperados desde los repositorios de objetos de aprendizaje. Suponemos que en la medida que los diseñadores intruccionales puedan acceder a recursos relevantes es posible contribuir en la calidad de los cursos e-learning o en la calidad de los nuevos recursos creados a partir de ellos. Cabe destacar que la efectividad de nuestra propuesta se ve afectada por el sistema de recuperación utilizado en cada repositorio, la calidad de los recursos almacenados y su etiquetado, así como la completitud y calidad de la base de conocimiento utilizada para la expansión.
... However, when content changes rapidly and dynamically, it is essential to plan incremental quality checks. Ratings (Nesbit, Belfer, & Vargo, 2002) Recommender Systems (Manouselis & Sampson, 2004) Peer Reviews (Neven & Duval, 2002) Trust (Pawlowski & Clements, 2010) As a conclusion, we can state that the broad concept of quality needs to be defined so that the scope and levels of quality are clearly understood. While OER quality mechanisms obviously need to be built on previous experience, it is also essential to assess how they can be applied to new situations, i.e., to the rapidly changing, dynamic environment. ...
Teacher education nowadays needs a change in vision and action to cater to the demands of changing societies. Reforms, improvements, and new approaches in teacher education are an immediate need. Online training of teachers using OER has emerged as a new approach in this direction. This approach is based on the assumption that online training will facilitate mass training of teachers as per their convenience and ease and will be helpful in filling the huge gap in demand for skilled teachers. The other assumption is that being OER-based training, it will be easy to adapt it to different cultures and languages. Considering these assumptions, the present paper discusses about on-going initiatives of online training of teachers in different parts of the world; focuses on promises of using OER to make online training of teachers more relevant and up-to-date; and suggests potential strategies for effectively using OER for initial training and continuing professional learning of teachers in online settings.
This paper presents an evaluation of the online interactive multimedia educational resources from three prominent publishers in Croatia (PROFIL Klett, Alfa and Školska knjiga) that are offering additional digital contents for learning the Croatian language in primary schools. Evaluation criteria were applied in three areas: Croatian language and Communication, Literacy, and Culture and Media. Digital contents represent cognitive auxiliary tools and supplementary teaching and learning material and can have different modes of application in the teaching and learning process, depending on the teacher's methodical approach. Studies have shown that pupils have better achievements in content retention and comprehension if multimedia resources accommodate different learning styles (visual, aural, read/write, and kinesthetic) and follow fundamental multimedia instructional design principles (multimedia, spatial and temporal contiguity, coherence, modality, redundancy, signalling, animation and interactivity), in order to capture pupils' attention, reduce split-attention effect and cognitive load. The aim of the evaluation is to provide guidelines for the publishers and teachers in building an effective conceptual model for developing teaching and learning materials in accordance with relevant research in multimedia design and education. This should enable and contribute to a better understanding of the potential benefits and limitations of the multimedia design process with respect to existing and future digital contents.
Розповсюдження на ринку навчальної літератури друкованих шкільних підручників із цифровими навчальними матеріалами вносить зміни в редакційно-видавничий процес та вимагає додаткових умінь і навичок редакторів. Мета дослідження – зібрати досвід редакційного колективу одного видавництва в роботі над кросмедійними шкільними підручниками та на його основі описати зміни в структурі редакторської роботи.
Методологія дослідження. У статті представлено результати дослідження,
проведеного методом опитування редакційного колективу УОВЦ «Оріон» за допомогою анкети на самозаповнення з подальшими напівструктурованими інтерв’ю. Методом порівняльного аналізу було описано зміни в структурі роботи редактора за умов кросмедійного виробництва навчальних видань.
Результати. Зроблено висновок, що класичні етапи редагування доповнюються новими процесами опрацювання цифрового навчального контенту, завданням побудови взаємодії друкованого й електронного видання або цифрового контенту та зміщуються в порядку виконання. За результатами опитування та інтерв’ю редактори відчувають новизну кросмедійного виробництва, розширення процесу редагування, усвідомлюють брак компетенцій для роботи із цифровим навчальним контентом. Дослідження дало змогу окреслити контрольні точки роботи редактора шкільних підручників з електронним
контентом.
Новизна дослідження полягає у спробі упорядкувати редакційні процеси видавничої підготовки кросмедійних підручників за етапами редагування та виявити дискусійні питання підготовки кросмедійного шкільного підручника.
Objectives:
To conduct a pilot investigation about the alignment between didactic multimedia materials utilized by pharmacy faculty, with Mayer's Principles for Multimedia Learning and faculty characteristics associated with greater alignment.
Methods:
An investigatory systematic process was used which included a modified Learning Object Review Instrument (LORI) to evaluate the faculty video-recorded lectures for alignment with Mayer's Principles of Multimedia Learning, hence capturing the number and type of misalignments. Correlations were performed to evaluate the association between faculty characteristics; and ratings and proportions of misalignments.
Results:
Five hundred fifty-five PowerPoint slides of 13 lectures from 13 faculty members were reviewed. The average (SD) LORI score per slide was 4.44 (0.84) out of 5 with an average score per lecture ranging from 3.83 (0.96) to 4.95 (0.53). Across all lecture slides, misalignments with multimedia principles were captured in 20.2% of slides. For each lecture, the average percentage of misalignments was 27.6% ranging from 0% to 49%. Principal misalignments included violation of the principles of coherence (66.1%), signaling (15.2%), and segmenting (8%). No faculty characteristics were significantly associated with LORI ratings or proportion of misalignments within lectures.
Conclusions:
Faculty had high LORI ratings for their multimedia material but these varied significantly between lectures. Misalignments with multimedia principles were identified and were related primarily to extraneous processing. These misalignments, when addressed, have the potential to improve learning, thus suggesting an opportunity for the faculty to develop ways to optimize multimedia educational delivery. Future investigation is needed to clarify how clinical pharmacy faculty can develop multimedia material and the impact of faculty development on the application of multimedia principles and learning outcomes.
A practicum is beneficial in science learning to provide a concrete manifestation of a topic. This study aimed at creating a simulation-based interactive learning multimedia product to assist students’ practicum. The product was developed for the topic of cell metabolism process focusing on the energy in living systems. The study employed the Research and Development referring to the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model. During the development process, both product testing and users’ response testing were carried out. To reveal the product impact, students’ results of the pre-test and post-test were analyzed. The product was developed by using Macromedia Flash 8. The results of Alpha and Beta analysis showed that the results are ‘feasible’ and ‘very feasible’ for the product. Furthermore, based on the average value of 39.2 for the initial test and 82.5 for the final test it was obtained an N-gain value of 0.712 with the category of "High". Therefore, it proved that the developed interactive learning multimedia is effective in increasing students' analytical skills for the Science Learning materials in junior high schools.
The Open Education movement has gained substantial traction since the term
Open Educational Resources (OER) was coined in 2000. However, there remains
much scope for further advocacy and promotion of Open Education generally and
of the principles and values that the concept embodies. Open Education is a broad
canvas that is able to accommodate a range of understandings of the term. It is
also a term that gathers an array of different elements beneath its umbrella, of
which OER is one, although one that is much discussed. OERs are generally stored
in a Learning Object Repository (LOR).
The focus of this paper is to undertake a short evaluation of quality issues in relation to OER, with a focus on the K12 sector where that is possible, and to examine any techniques that can be identified, either in proposals by educators and academics or in actual use in OER repositories, that might support educators in integrating digital learning resources into their teaching practice. Much of the literature and research into Open Educational Practices (OEP) generally, and into OER in particular, has been produced and carried out from a higher education (HE) or technical & vocational education & training (TVET) perspective, and, to date, with a few honourable exceptions, less so from the perspective of teachers and students in the schools sector. For whatever reasons, there seems to have been an identifiably lower level of engagement with Open Education from within the K12 sector generally over many years. Of course, many of the Open Education lessons learnt and applied within HE and TVET will apply just as much to OEP in schools, but we should be cautiously aware that their application in K12 might raise some contextually specific issues too.
The Basque Government has published two calls to create digital educational objects for the programme called Eskola 2.0. After having provided schools with technological equipment, these calls aim to increase the use of learning technology in the classroom. More than 300 didactic sequences have been developed, which vary greatly in visual design, content structure, organization, and pedagogical aspects. Even though accessibility is one of the quality criteria, the reality is that they are hardly accessible and inclusive. DeustoTech Learning research group has carried out a survey of the educational objects approved in these calls up to November 2011. The authors evaluated pedagogical and technological aspects to find out how inclusive they are. In this chapter, they provide the results of the survey and propose a set of guidelines for designing more accessible and inclusive objects in the future.
https://explore.researchgate.net/display/support/Generating+a+DOI
With the rise of the Internet and the proliferation of online content, the design and evaluation of educational digital resources (EDRs) are pressing and challenging issues. They warrant an investigation of what exactly are the features that increase the quality of EDRs. In a previous professional development program, we trained and supported teachers in evaluating and selecting EDRs with the support of a scientifically validated rubric. In this present study, through quantitative, qualitative, and text-mining methods, we analyzed the review data of 1200 resources produced that professional development program in order to provide a big picture of the quality of currently available products, and to identify the features that characterize quality digital resources. Our findings suggest the need for digital repositories to reflect or make visible how resources fit particular instructional design models.
Este trabajo describe el proceso para la creación de un modelo de calidad que evalúe Objetos de Aprendizaje en aspectos tecnológicos y didácticos utilizando la notación i*. Para cumplir con este objetivo se revisaron modelos de calidad en ambos aspectos y conjuntamente con los conceptos de OA y calidad se han determinado las características que debe cumplir un OA, de esta manera considerar las características de evaluación. Se ha utilizado el método IQMC para la construcción del diagrama de dependencias estratégicas (SD) de i* para el modelo de calidad final. Con el modelo SD creado se establece la trazabilidad entre los criterios del modelo generado y los atributos de la matriz de la ISO/IEC 25010 para mantener una correlación entre las características de la notación i* con el estándar de la matriz mencionada de la ISO. La matriz obtenida es enriquecida con características conceptuales y los atributos que debe cumplir un OA. Se establecen las diferentes métricas en la matriz de la ISO para las evaluaciones de OA. Como resultado se obtiene un modelo generado mediante la evaluación de un OA con cada una de las métricas seleccionadas.
In dem neu veröffentlichten Sonderband des Fachmagazins Synergie befassen sich Prof. Dr. Olaf Zawacki-Richter und Prof. Dr. Kerstin Mayrberger mit der Qualität von Open Educational Resources (OER). Die Studie umfasst eine internationale Bestands- aufnahme von Instrumenten zur Qualitätssicherung von OER sowie die Entwicklung von Schritten zu einem deutschen Modell am Beispiel der Hamburg Open Online Uni- versity. Im Fokus steht die Frage nach der systematischen Qualitätssicherung und -entwicklung von OER im Allgemeinen und im deutschprachigen Raum.
Learning objects are not visualization tools but they are the essence of a learning environment and are of various forms that in turn serve various purposes. Therefore, they can dictate what visualization tools will be needed and must be understood to an appropriate level. Like many concepts, learning objects can mean or be defined differently by different people.
Information and materials on practically any topic can be currently found on the Internet. However, the advances to classify and measure the quality of materials have been minimal and scattered. Anyone can create materials and publish them, but no one ensures or guarantees that those materials will be accessible or their contents will be suitable for study. This means that, even though a huge amount of materials are available, it is difficult to retrieve those that have a minimum level of quality. This general problem worsens in educational domains, such as the university, where teachers continuously create and update digital educational materials with no guidance as to the minimum requirements for their materials not to pose technical problems — such as portability and durability —, teaching problems — such as the consistency between goals and recipients, on the one hand, and suitable design and format for study, on the other — and accessibility problem — such as ensuring the perceptibility of audiovisual and textual materials. In this context, an AENOR working group has developed a proposal of a Spanish quality standard for educational materials (UNE 71632). On the basis of this standard, a web tool has been developed to implement the standard and facilitate the assessment of the quality of digital educational materials. This paper presents the tool developed and its main features.
Learning Objects (LOs) are web based learning resources presented by Learning Object Repositories (LOR). For recent years LOs have begun to take place on web and it is suggested that appropriate design of LOs can make positive impact on learning. In order to support learning, research studies recommends LOs should have been evaluated pedagogically and technologically, and the content design created by using LOs should have been designed through appropriate instructional models. Since the use of LOs have recently begun, an exact pedagogical model about efficient use of LOs has not been developed. In this study a LOR is designed in order to be used in mathematics education. The LOs in this LOR have been evaluated pedagogically and technologically by mathematics teachers and field experts. In order to evaluate the designed LO based environment, two different questionnaires have been used. These questionnaires are developed by using the related literature about web based learning environments evaluation criteria and also the items are discussed with the field experts for providing the validity. The reliability of the questionnaires is calculated cronbach alpha = 0.715 for the design properties evaluation survey and cronbach alpha =0.726 for pedagogic evaluation. Both of two questionnaires are five point Likert type. The first questionnaire has the items about “Learning Support of LOs, Competency of LOR, The importance of LOs in mathematics education, the usability of LOs by students”. “The activities on LOs are related to outcomes of subjects, there are activities for students have different learning styles. There are activities for wondering students.” are examples for items about learning support of LOs. “System helps for exploration of mathematical relations”, “I think teaching mathematics with this system will be enjoyable.” are example items for importance of LOs in mathematics education. In the competency of LOR title, “System can be used in problem design about daily life.”, “Using systems can take much time for teachers” and in the title Students‟ using of LOR, “System is not appropriate for collaboration”, “The learning environment can help communication between students.” are the example items. In the technological dimension, the title design principles consist of items like; “The text and tables on the LOs are readable. The design of the menus makes the system usable. , The design is original.” In technological consistency title “LOs can be found by search options.”, “The tools are used for users to continue on the system.” , “Many kinds of files can be uploaded in to the system.” and in security title, the questionnaire has items like; the upload and download systems do not have problems, the user control system works confident. By using these surveys 64 educators in the field and mathematics teacher evaluated the LOR in pedagogy and content dimensions, and 46 material design expert and web expert evaluated LOR in design, technology and security dimensions. As a result of pedagogical and content evaluation, the participants revealed positive views about the LO based learning environment. In pedagogical dimension; it is found that the number of LOs is enough, the content design system must be eliminated from the details, the LOs have enough interaction, LOs can help exploring mathematical relations, and the activities related affective domain must be increased and by using of LOs an exciting and funny learning environment may be designed. In addition; according to the design experts, the learning environment is basic and useful, also most of them agreed on the system that it is complied with design criteria. Besides experts evaluated the security options are feasible. By the results of this study, the LOR will be updated and revised to a form a web based learning environment for mathematics education and the real impact of LO based mathematics learning environment will be investigated in future studies.
Content management is not an issue most senior administrators in educational institutions will be familiar with. In this chapter, I want to take a strategic view of content management, especially for those institutions that have or are about to make a major commitment to the development and delivery of online teaching and learning materials, or what I will call digital learning materials.
Architects, development practitioners and designers are working in a global environment and issues such as environmental and cultural sustainability matter more than ever. Past interactions and interventions between developed and developing countries have often been unequal and inappropriate. We now need to embrace fresh design practices based on respect for diversity and equality, participation and empowerment.
Within the last five years, governments and education authorities worldwide have developed and implemented approaches to facilitate access to a wide range of quality digital resources and reduce the costs of production. This chapter reports on a study which invited school teachers and university academics in Egypt, as a developing and Arabic-speaking country, to cooperate in establishing a learning object repository to store, locate, and share quality learning objects for class teaching and e-learning programs. The proposed solution is originally a vendor hosted web-based groupware, file management, and sharing system that meets the basic criteria of instructional learning object repositories called eStudio. Motivators and inhibitors to using the repository, factors that determine locating, using, and sharing learning objects within the repository and their qualities are assessed to help in developing repositories that demonstrate an understanding of the existing needs and the work practices of Egyptian teachers and other user groups.
Web-based learning tools (WBLTs), also known as learning objects, have been evaluated with a wide range of metrics, but rarely with respect to pedagogical design. The current study evaluated the impact of instructional architecture (direct instruction vs. constructive-based) on middle (n=333) and secondary (n=389) school student attitudes (learning, design, and engagement) and learning performance (remembering, understanding, and application tasks) with respect to WBLTs. Students rated WBLTs with a direct instruction architecture significantly higher than WBLTs with a constructive-based architecture in the areas of learning effectiveness, design, and engagement. Students also commented that a direct instruction format had better organization and visual supports, higher quality graphics and animations, and superior help features. Students performed significantly better in the understanding and application knowledge categories when a direct instruction format was employed. No significant differences were observed between direct instruction and constructive-based architectures with respect to the remembering knowledge category. It is speculated that a direct-instruction design may be better suited to younger students who are learning basic level concepts, although more research is needed to explore higher level knowledge areas.
There is a need for better and more objective assessments of student cognition especially at the engineering college level. This is important to capture essential intellectual abilities that may be missed through conventional testing, produce assessments that are far more descriptive of student cognition than a single holistic grade, link learning outcomes and professional competencies, enable multiple evaluators to apply the same criteria to judge the same work and design better formative interventions. This paper reports on the development and subsequent deployment of a rubric based on Fink's cognitive dimensions of Foundational Knowledge (F), Application (A) and Integration (I) in a Fluid Mechanics and Heat Transfer (FMHT) class though it has potential broader multi-disciplinary applicability. Professors from Chemical Engineering, Mechanical Engineering and Education alongside graduate students from Chemical Engineering were involved in the development of the rubric. Definitions of what exactly constitute F, A and I in the subject domain were negotiated, scales indicating levels of performance were agreed upon and a minimum competency "anchor" line was drawn. Deployment of the rubric was done following a Convergent Participation Model (CPM) to be described in the body of the paper. Three main types of assessment artifacts namely traditional text-book problems, professor-crafted worksheets and final examination problems were rated for the aforementioned cognitive dimensions. A portion (25%) of the anonymous artifacts were selected and rated by 2 different panels consisting of professors and graduates students involved in the rubric development, or otherwise trained in its use. Our premise was that we would see consistent ratings for the F, A, and I dimensions and that ratings for the worksheets would be identical to those for traditional textbook problems. However, the three dimensions were found to produce different ratings, depending on the depth and complexity of the assignment, and the three types of assessment artifacts were found to differ significantly in rubric outcomes. These and other interesting findings are discussed with a view to designing better learning experiences and assessments.
Despite the fact that digital technologies are more and more used in the learning and education process, there is still lack of professional evaluation tools capable of assessing the quality of used digital teaching aids in a comprehensive and objective manner. Construction of the Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) tool was preceded by several surveys and knowledge obtained in the course of creation of digital learning and teaching aids and implementation thereof in the teaching process. The evaluation tool as such consists of sets (catalogues) of criteria divided into four separately assessed areas - the area of technical, technological and user attributes; the area of criteria evaluating the content, operation, information structuring and processing; the area of criteria evaluating the information processing in terms of learning, recognition, and education needs; and, finally, the area of criteria evaluating the psychological and pedagogical aspects of a digital product. The specified areas are assessed independently, separately, by a specialist in the given science discipline. The final evaluation of the assessed digital product objectifies (quantifies) the overall rate of appropriateness of inclusion of a particular digital teaching aid in the teaching process.
El artículo presenta los resultados de la adaptación de indicadores cuantitativos al Instrumento para Evaluar Objetos de Aprendizaje LORI. El aporte que ofrece esta investigación al área educativa consiste en la posibilidad de medir, reportar, comunicar , actuar y dar seguimiento a la calidad de los Recursos Educativos Digitales en sus etapas de diseño, desarrollo y post implementación, tanto por los docentes, desarrolladores, administradores de calidad como por usuarios finales del recurso digital; al permitir realizar revisiones y comparaciones sistemáticas con las mediciones obtenidas para seleccionar y mejorar los recursos educativos digitales.
ResearchGate has not been able to resolve any references for this publication.