Figure - uploaded by Jeffrey Buckley
Content may be subject to copyright.
Similar publications
Esta obra complementa los contenidos de análisis de circuitos eléctricos abordados en el libro Circuitos eléctricos – Parte I , editado en 2017 por EDULP. Se presentan temas de electrónica básica, transductores y la resolución gráfica de circuitos lineales y alineales; disponiéndose así, con ambos volúmenes, de una obra completa con los fundamentos...
Citations
... First-year engineering students are typically required to engage in team and problem-based activities through introductory coursework to support the development of design capabilities [1], [11]. This type of activity is typically assessed using rubrics, portfolios, and criteriongrading tools [2]- [6]. However, there are issues in assessing open-ended and divergent tasks in this manner including; reliability, teacher bias, excessive time investment, and timeliness of feedback [1]- [5]. ...
... Validity has been demonstrated in both approaches to making judgements [9]. In past studies, judges have been students, professionals, and instructors/professors [1], [3], [4], [6], [8]- [10]. The number of judgements made by each judge depends on the number of portfolios and judges. ...
... This feedback is collated and provided to each student [7]. By judging portfolios on multiple occasions, a consensus is reached on the positioning of the work within the cohort, and a rank-ordered list is produced that can be used to determine grades or monitor progress (see [3], [4], [6], [7], [9] for detailed descriptions of tabulation and grade computation using ACJ assessment). Where students serve as judges, they may be assigned their work to judge. ...
Design projects are an important part of many first-year engineering programs. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities (e.g., professional constructs) that are not typically conducive to standard assessment rubrics. Adaptive Comparative Judgement (ACJ) is an assessment approach that is used to assess design projects holistically. The assessment of projects using ACJ can be carried out by experts or students to scaffold their learning experience. This Work-in-Progress paper explores the use and benefits of ACJ for assessing design projects specifically focusing on first-year engineering students and educators. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software to experience how this system can work in practice for assessing student design projects.
... What determined the hierarchical structure and how this varied between students would shed additional light and perspective on the mental processes involved with ACJ. F. Seery et al. (2016), gave students a design project that was followed by student-made judgments of work using ACJ. Students also hand graded (on a scale of 1-10) select portfolios for points. ...
... Additionally, university students peer-graded a mock examination question using ACJ and generally praised the approach for learning and were more inclined to this approach of peer-grading than assigning traditional marks (scores). Further investigation into the selection of judgment criteria, or the lack thereof (see Canty et al. 2017Canty et al. , 2019Seery et al. 2016) and its impact on the selection process is needed. Judges selecting the better of two items, with different criteria for selection in mind, may arrive at the same conclusion, or not; regardless, an understanding of the impact of the criteria for selection is needed to further explore the ramifications of ACJ for assessment and learning. ...
Adaptive Comparative Judgment (ACJ), an approach to the assessment of open-ended problems which utilizes a series of comparisons to produce a standardized score, rank order, and a variety of other statistical measures, has demonstrated high levels of reliability and validity and the potential for application in a wide variety of areas. Further, research into using ACJ, both as a formative and summative assessment tool, has been conducted in multiple contexts across higher education. This systematized review of ACJ research outlines our approach to identifying, classifying, and organizing findings from research with ACJ in higher education settings as well as overarching themes and questions that remain. The intent of this work is to provide readers with an understanding of the current state of the field and several areas of potential further inquiry related to ACJ in higher education.
... According to Pollitt (2012b), this holistic approach embedded in CJ with multiple judges rules out personal biases, leading to higher consistency in judgment among the assessors. Seery and Buckley (2016). Each data point represents one portfolio. ...
There is a growing demand for the use of digital tools in assessment. Few approaches show innovative benefits beyond being logistical aids. Comparative judgment (CJ) has the potential to enhance educational practices by providing a mechanism for reliable assessment, supporting formative feedback, and by supporting critical discourse on evidence of learning. This chapter provides an overview of CJ as it has been used in educational assessment and describes how it can be facilitated by digitalization by providing illustrative examples of research studies, mainly undertaken for formative purposes. Specifically, this chapter provides an introduction to CJ and a description of its theoretical roots, presents possible approaches and agendas for the use of CJ ranging from being a pedagogical tool in a classroom to being a mediator for continuing professional development and discusses implications for practice and future research needs. Ultimately, it is envisaged that this chapter will act as a source of inspiration for educational stakeholders who wish to use CJ to add value to their practice.
... ACJ has been tested, implemented, researched, and scrutinized extensively in summative assessment settings with student portfolios (e.g., Bartholomew, Strimel, & Jackson, 2017;Newhouse, 2011;Seery, Delahunty, Canty, & Buckley, 2017), student essays (e.g., Pollitt & Whitehouse, 2012;Steedle & Ferrara, 2016;van Daal, Lesterhuis, Coertjens, Donche, & De Maeyer, 2019), and mathematics (Jones & Alcock, 2012). Additionally, ACJ has also been piloted with students in formative assessment scenarios (e.g., Bartholomew, Yoshikawa, & Strimel, 2018;Bartholomew, Zhang, Garcia-Bravo, & Strimel, 2019;Kimbell, 2018;Seery, Buckley, Doyle, & Canty, 2016;Seery et al., 2012). ...
This chapter focuses on the development, adaptation, and revising of the Taiwanese-based STEM2TV (STEM for Taiwan, Thailand, and Vietnam) module toward the needs of New Asia. The STEM2TV project aims to investigate and develop the next generation of STEM education for this region. After the introduction to this chapter, we present our STEM2TV project in four sections: (1) context setting—how our research project starts by leveraging similarities, accommodating the differences between the partner countries, as well as presenting our struggles in implementing STEM courses in different classrooms; (2) research framework—how we gather knowledge from other (mostly western) countries as well as from our previous experiences to guide STEM education research toward our STEM2TV project; (3) preliminary findings from the pilot data—how we are building strong relationships and partnerships by trialing STEM modules and assessments together, to fit the needs of “New Asia’s” countries, based on more comprehensive and contextual views toward STEM (this section also shares some educators’ and teachers’ feedback from two collaborative case studies in Vietnam and Thailand in 2018); and (4) future directions of the project—the benefits, contributions, and future vision of this research for STEM education communities.
... ACJ has been tested, implemented, researched, and scrutinized extensively in summative assessment settings with student portfolios (e.g., Bartholomew, Strimel, & Jackson, 2017;Newhouse, 2011;Seery, Delahunty, Canty, & Buckley, 2017), student essays (e.g., Pollitt & Whitehouse, 2012;Steedle & Ferrara, 2016;van Daal, Lesterhuis, Coertjens, Donche, & De Maeyer, 2019), and mathematics (Jones & Alcock, 2012). Additionally, ACJ has also been piloted with students in formative assessment scenarios (e.g., Bartholomew, Yoshikawa, & Strimel, 2018;Bartholomew, Zhang, Garcia-Bravo, & Strimel, 2019;Kimbell, 2018;Seery, Buckley, Doyle, & Canty, 2016;Seery et al., 2012). ...
While Science, Technology, Engineering, and Mathematics (STEM) education is being lobbied as a mechanism for teaching students both conceptual knowledge and procedural skills, the assessment of these skills can be difficult. The open-ended nature of STEM education assignments contributes to challenges related to reliability, validity, and feasibility. Adaptive comparative judgment (ACJ), an approach to assessment which relies on pairwise comparisons rather than value-based decisions, has shown promise in relieving some of the aforementioned difficulties with STEM conceptual knowledge and procedural skill assessment. Utilizing a holistic approach to assessment through ACJ has shown promise, especially in the potential for identifying, and assessing, student competencies in important STEM skill areas. Several case studies, with accompanying explanations and potential application directions, are included in line with direction for further inquiry and discussion.
... The assignments were underpinned by broad criteria including (1) quality and coherency in communication, (2) innovation, (3) knowledge of stages of design, (4) knowledge of functions of design, and (5) an understanding of geometric principles. A separate study conducted by Seery and Buckley (2016) determined that performance in a similar conceptual graphical design task is indicative of learning and competency with respect to these criteria. ...
Educational assessment has profound effects on the nature and depth of learning that students engage in. Typically there are two core types discussed within the pertinent literature; criterion and norm referenced assessment. However another form, ipsative assessment, refers to the comparison between current and previous performance within a course of learning. This paper gives an overview of an ipsative approach to assessment that serves to facilitate an opportunity for students to develop personal constructs of capability and to provide a capacity to track competence based gains both normatively and ipsatively. The study cohort (n = 128) consisted of undergraduate students in a Design and Communication Graphics module of an Initial Technology Teacher Education programme. Four consecutive design assignments were designed to elicit core graphical skills and knowledge. An adaptive comparative judgment method was employed to rank responses to each assignment which were subsequently analysed from an ipsative perspective. The paper highlights the potential of this approach in developing students’ epistemological understanding of graphical and technological education. Significantly, this approach demonstrates the capacity of ACJ to track performance over time and explores this relative to student ability levels in the context of conceptual design.
... The design worksheets and prototypes were evaluated by the teachers using the Adaptive Comparative Judgment (ACJ) assessment technique-a method that has recently gained attention as a reliable and valid method of evaluating open-ended design problems (Bartholomew 2017;Bartholomew et al. 2017;Kimbell 2012;Seery et al. 2016;Bartholomew et al. 2018). Through the ACJ software CompareAssess, teachers assessed students design portfolios at the conclusion of the assignment (the portfolios consisted of the design worksheet and an image of their prototype). ...
This study examined the design cognition and achievement results of both kindergarten and fourth grade students engaged in engineering design-based instructional activities. Relationships between design cognition and student grade level, as well as quality of student work, were investigated. 30 concurrent think-aloud protocols were collected from individual primary students as they worked in groups to design and make a solution to a design task. The concurrent think-aloud protocols were examined and coded to determine the duration of time the participants devoted to a pre-established set of mental processes for technological problem solving. Significant differences between kindergarten and fourth grade participants were found in the amount of time various cognitive processes were employed. Fourth grade students dedicated significantly more time to the mental processes of Creating, Defining Problems, Measuring, and Testing than kindergarten students. In addition, when examining the think-aloud protocols along with the evaluations of the participant’s design work, it was found that more time devoted to the cognitive process of Managing could be a significant predictor of lower design achievement. These findings can highlight potential areas for improving educational practice based on the cognitive abilities of students at different grade levels and the quality of their design work. As engineering design-based activities become more prevalent for the teaching of STEM-related content and practices, the results of this research, and the employed methodology, may demonstrate a promising practice for better understanding and assessing such education efforts.
... According to Pollitt (2012b), this holistic approach embedded in CJ with multiple judges rules out personal biases, leading to higher consistency in judgment among the assessors. Seery and Buckley (2016). Each data point represents one portfolio. ...
Eva will present comparative judgement and findings from an on-going international comparative study focusing on unpacking teachers’ assessment practices in USA, UK and Sweden
... The use of Adaptive Comparative Judgement (ACJ) (Pollitt, 2012b) as a method of assessment has been proven to be both valid and reliable in the assessment of design based competencies (Kimbell, 2012;Pollitt, 2012aPollitt, , 2012bSeery & Buckley, 2016;Seery, Canty, & Phelan, 2012). Based on Thurstone's (1927) Law of Comparative Judgement, assessment is carried out by a group of 'judges' making binary decisions on of quality of work evidenced in multiple pairs of portfolios. ...
One of the leading frameworks in engineering education specifically associated with design based competencies is the CDIO framework. This has been incorporated internationally into many institutions offering engineering education courses. Characterized by four unique stages, the CDIO framework affords an ideal scenario to incorporate a continuous assessment model. This paper presents a proposed synthesis between CDIO and Adaptive Comparative Judgement (ACJ). In particular, the opportunity to provide feedback through the ACJ system is theorized to have potentially positive educational effects. As part of a larger study, this approach is in the process of being refined prior to implementation as a pilot study for feasibility which will ultimately be succeeded by large-scale implementation to determine any potentially positive effect sizes.
... The use of Adaptive Comparative Judgement (ACJ) (Pollitt, 2012b) as a method of assessment has been proven to be both valid and reliable in the assessment of design based competencies (Kimbell, 2012;Pollitt, 2012aPollitt, , 2012bSeery & Buckley, 2016;Seery, Canty, & Phelan, 2012, Ryan et al. 2017. Based on Thurstone's (1927) Law of Comparative Judgement, assessment is carried out by a group of 'judges' making binary decisions on of quality of work evidenced in multiple pairs of portfolios. ...
One of the leading frameworks in engineering education specifically associated with design based competencies is the CDIO framework. This has been incorporated internationally into many institutions offering engineering education courses. Characterized by four unique stages, the CDIO framework affords an ideal scenario to incorporate a continuous assessment model. This paper presents a proposed synthesis
between CDIO and Adaptive Comparative Judgement (ACJ). In particular, the opportunity to provide feedback through the ACJ system is theorized to have potentially positive educational effects. As part of a larger study, this approach is in the process of being refined prior to implementation as a pilot study for feasibility which will ultimately be succeeded by large-scale implementation to determine any potentially positive effect sizes.
