Content uploaded by Richard Edward Clark
Author content
All content in this area was uploaded by Richard Edward Clark on Jan 26, 2016
Content may be subject to copyright.
A preview of the PDF is not available
Cognitive Task Analysis for Expert-Based Instruction in Healthcare
Abstract and Figures
This chapter presents an overview or the rationale and ev idence for the use of cognitive task analysis (CTA) in health care including the following: It presents a brief history and definition of CTA, the reason it is being adopted for healthcare education, evidence for its learning benefits when used in evidence-based instructional design and medical simulators, an example of how one of the evidence-based CTA methods was implemented in healthcare, and suggestions for future research. The point is made that when evidence-based CTA methods are used, learning from CTA-based healthcare instruction increases an average of 45 % when compared with current task analysis methods.
Figures - uploaded by Richard Edward Clark
Author content
All figure content in this area was uploaded by Richard Edward Clark
Content may be subject to copyright.
... Instructional design development typically starts with an analysis of the desired learning outcomes, which, in competency-based medical education, are also termed "whole-tasks," "complex tasks," or "entrustable professional activities" (EPAs) [3,11]. To describe such a task from an expert's perspective, educators have used cognitive task analysis (CTA) to analyze and structure the relevant knowledge, skills, and attitudes of complex tasks in a range of disciplines-including medicine [12]. Research in other educational fields has shown that instructional designs based on CTA have resulted in a 31-46% post-training performance gain [12]. ...
... To describe such a task from an expert's perspective, educators have used cognitive task analysis (CTA) to analyze and structure the relevant knowledge, skills, and attitudes of complex tasks in a range of disciplines-including medicine [12]. Research in other educational fields has shown that instructional designs based on CTA have resulted in a 31-46% post-training performance gain [12]. The next step in instructional design development is to select and orchestrate specific instructional strategies and sequences. ...
... The research question was: "What instructional design strategies for teaching the psychiatric interview and MSE have been used or studied in the education of health professionals?" For charting the data and collating and summarizing the results, both a cognitive task analysis (CTA) [12] and the fourcomponent instructional design (4C/ID) [11,16] approach were used to derive instructional design implications and identify research gaps. ...
Objective
The psychiatric mental status examination is a fundamental aspect of the psychiatric clinical interview. However, despite its importance, little emphasis has been given to evidence-based instructional design. Therefore, this review summarizes the literature from an instructional design perspective with the aim of uncovering design strategies that have been used for teaching the psychiatric interview and mental status examination to health professionals.
Methods
The authors conducted a scoping review. Multiple databases, reference lists, and the gray literature were searched for relevant publications across educational levels and professions. A cognitive task analysis and an instructional design framework was used to summarize and chart the findings.
Results
A total of 61 articles from 17 countries in six disciplines and three educational levels were identified for data extraction and analysis. Most studies were from the USA, presented as educational case reports, and carried out in undergraduate education in the field of psychiatry. Few articles described the instructional rationale for their curriculum. None of the studies compared the effectiveness of different instructional design components. Reported learning activities for each task domain (knowledge, skills, and attitudes) and for each step of an instructional design process were charted. Most articles reported the use of introductory seminars or lectures in combination with digital learning material (videos and virtual patients in more recent publications) and role-play exercises.
Conclusions
Educators in psychiatry should consider all task domains of the psychiatric interview and mental status examination. Currently, there is a lack of empirical research on expertise acquisition and use of instructional design frameworks in this context.
... More broadly, a large body of research has examined the relations between student behavior, cognitive processes, and learning in different contexts (i.e., the relations between student actions, speech, gestures; e.g., Chu andKita (2011), Church, Kelly, andHolcombe (2014), Clark (2014), Goldin-Meadow and Singer (2003), Kelly, Healey, Özyürek, and Holler (2015)). Previous research on the coupling between speech and physical movements in communication has shown a stronger relationship between speech and gesture than between speech and action in the context of language production and language comprehension (Church et al., 2014;Kelly et al., 2015). ...
... This project aims to advance research on student behavior, reasoning, and learning by using machine-learning methods to analyze the interplay between learners' actions, speech, and gestures while completing measurement estimation tasks. Since physical behavior during problem solving (Clark, 2014) and gestures (Goldin-Meadow & Singer, 2003) reveal implicit knowledge, we hypothesize that observing students' physical actions while problem solving will also reveal valuable implicit knowledge of measurement concepts. Our goal is to use machine-learning methods to discern how student behavior is indicative of student performance and, consequently, conceptual understanding in the context of measurement, in order to inform the design of future activities and instructional support for students to develop procedural and conceptual measurement skills. ...
Although interdisciplinary collaborations are becoming increasingly common, researchers typically use data analysis methods specific to their field in order to uncover how students learn. We present affordances of integrating theories of embodied cognition and design with machine-learning methods to study student learning in mathematics and inform the design of embodied learning activities. By increasing such collaborative research efforts, learning scientists can incorporate regularization in computational models and ultimately draw reliable conclusions to further inform theory and practice through the design of technology-augmented learning activities. To illustrate this point, we explored students’ conceptual understanding of measurement since limited research has identified measurement estimation strategies that should be emphasized in classroom instruction. By uniquely applying machine-learning methods to a small, multimodal dataset from a study on student behavior in mathematics, we identified behavioral profiles, patterns in speech, and specific actions and gestures that are predictive of performance. These findings may be used to inform the design of embodied learning activities for measurement. We discuss the contribution of these findings to the field of embodied design, and the affordances and challenges of conducting collaborative research in the learning sciences.
... Meta-analyses after 2000 examining learning outcomes (Bernard et al., 2004;Shachar and Neumann, 2003) typically concluded that modality, per se, was not a significant factor in learning outcomes; findings Clark drew upon to re-iterate his original claims (Clark and Feldon, 2005;Clark, 2007). This position statement remains essentially unchanged (Becker, 2010;Clark, 2014), despite some recent meta-analyses showing that university learning outcomes are generally better with OL modes (Jayakumar et al., 2015;Jurewitsch, 2012;Nguyen, 2015). Critics of this status-quo argued that Clark's commentary reflected a lack of understanding regarding educational applications of "new media" and their educational applications (e.g. ...
... The primary focus of our review was to determine, based on the weight of evidence over the past two decades, whether F2F or online teaching modalities provide greater efficacy regarding university learning outcomes or whether the "no significant difference" position (Clark, 2007(Clark, , 2014 continues to reflect their relative status. Our combined analysis provided a clear answer to our research question. ...
Purpose – Following the rapid shift to online learning due to COVID-19, this paper aims to compare the relative efficacy of face-to-face and online university teaching methods.
Design/methodology/approach – A scoping review was conducted to examine the learning outcomes within and between online and face-to-face (F2F) university teaching programmes.
Findings – Although previous research has supported a “no significant difference” position, the review of 91 comparative studies during 2000–2020 identified 37 (41%) which found online teaching was associated with better learning outcomes, 17 (18%) which favoured F2F and 37 (41%) reporting no significant difference. Purpose-developed online content which supports “student-led” enquiry and cognitive challenge were cited as
factors supporting better learning outcomes.
Research limitations/implications – This study adopts a pre-defined methodology in reviewing literature which ensures rigour in identifying relevant studies. The large sample of studies (n = 91) supported the comparison of discrete learning modes although high variability in key concepts and outcome variables made it difficult to directly compare some studies. A lack of methodological rigour was observed in some studies.
Originality/value – As a result of COVID-19, online university teaching has become the “new normal” but also re-focussed questions regarding its efficacy. The weight of evidence from this review is that online learning is at least as effective and often better than, F2F modalities in supporting learning outcomes, albeit these differences are often modest. The findings raise questions about the presumed benefits of F2F learning
and complicate the case for a return to physical classrooms during the pandemic and beyond.
... Thus, a needs assessment may not provide all the necessary information about which nonrecurrent and recurrent skills learners must coordinate in a whole task. In addition, evidence shows that clinical experts who teach recall only about 30% of their automated decisions and strategies, leading them to fill these memory gaps with perhaps faulty assumptions when explaining procedures to learners [10]. A more reliable approach is cognitive task analysis (CTA). ...
Background
Healthcare simulation education often aims to promote transfer of learning: the application of knowledge, skills, and attitudes acquired during simulations to new situations in the workplace. Although achieving transfer is challenging, existing theories and models can provide guidance.
Recommendations
This paper provides five general recommendations to design simulations that foster transfer: (1) emphasize whole-task practice, (2) consider a cognitive task analysis, (3) embed simulations within more comprehensive programs, (4) strategically combine and align simulation formats, and (5) optimize cognitive load. We illustrate the application of these five recommendations with a blueprint for an educational program focusing on simulation activities.
Conclusions
More evidence-informed approaches to healthcare simulation might require a paradigm shift. We must accept that a limited number of simulations is not enough to develop complex skills. It requires comprehensive programs that combine simulation sessions with workplace learning.
... Several studies in the health sciences domain have shown that educational programmes based on CTA are particularly effective. 16 Thus, CTA seems to provide a strong basis for the design of educational programmes, but more research can contribute to improvements of the CTA process and its outcomes for instructional design. For example, the growing prevalence of competency frameworks such as CanMeds or descriptions of Entrustable Professional Activities (i.e., EPAs) 17 ...
This chapter discusses research themes that are pertinent to the field of instructional design. It provides a brief description of the ADDIE model that characterises the main phases in instructional design: analysis, design, development, implementation and evaluation. Instructional design is both a practical field and a research field. The chapter focuses on two approaches for the specification of final attainment levels: the traditional use of taxonomies of learning for setting learning goals and the use of cognitive task analysis for developing integrative learning goals. Several studies in the health sciences domain have shown that educational programmes based on cognitive task analysis are particularly effective. The chapter also discusses whole‐task models because, in health professions education, many instructional design projects will deal with the development of professional competencies or complex skills. Successful implementation of whole‐task curricula requires careful preparation of teachers, staff, and organisation, and evaluations that lead to continuous improvement.
... (P3, Foldit, Expert) In fact, this professional vision is so ingrained, so intuitive, that participants struggled to talk about their expertise and learning process, which is perhaps an obvious finding for researchers familiar with cognitive task analysis (CTA). As CTA researcher Clark writes, "experts don't know what [others] don't know," in other words, experts significantly distort or omit details of their own expertise without specialized knowledge probing [33]. P12 (Foldit, Intermediate) says that they "can't really put into words what makes a well-designed protein." ...
... A much simpler, potentially cheaper strategy to develop a skill chain would be directly asking the expert players to draw the skill chain as they understand it. This direct, unguided approach is typically avoided in CTA literature because it can lead to reduced or less structured results [13]. However, two critical factors differ in this context. ...
For citizen science games (CSGs) to be successful in advancing scientific research, they must effectively train players. Designing tutorials for training can be aided through developing a skill chain of required skills and their dependencies, but skill chain development is an intensive process. In this work, we hypothesized that free recall may be a simpler yet effective method of directly eliciting skill chains. We elicited 23 skill chains from players and developers and augmented our reflexive thematic analysis with 11 semi-structured interviews in order to determine how players and developers conceptualize skill trees and whether free recall can be used as an alternative to more resource-intensive cognitive task analyses. We provide three main contributions: (1) a comparison of skill chain conceptualizations between players and developers and across prior literature; (2) insights to the process of free recall in eliciting CSG skill chains; and (3) a preliminary toolkit of CSG skill-based design recommendations based on our findings. We conclude CSG developers should: give the big picture up front; embrace social learning and paratext use; reinforce the intended structure of knowledge; situate learning within applicable, meaningful contexts; design for discovery and self-reflection; and encourage practice and learning beyond the tutorial. Free recall was ineffective for determining a traditional skill chain but was able to elicit the core gameplay loops, tutorial overviews, and some expert insights.
... A CTA is a technique for identifying, analyzing, and structuring the skills and knowledge used by experts to perform a complex task [16], and leads to an integrative, coherent description of everything required to perform professional tasks adequately [17]. In this article, skills and constituent skills are identified by means of lesson observations and a cued retrospective interview with various expert teachers, and summarized in a skills hierarchy. ...
Although many schools in the Netherlands have purchased adaptive learning systems (ALSs) to reduce workload and improve differentiated instruction, the use of ALSs with teacher dashboards in the classroom does not in itself necessarily improve differentiated instruction. The question is, what skills and knowledge do teachers need to provide differentiated instruction with the help of ALSs with teacher dashboards and how does this differ from a context in which such technologies are not used By means of a cognitive task analysis (CTA), consisting of classroom observations, interviews and several expert meetings, teachers skills and knowledge necessary for providing differentiated instruction when using teacher dashboards and ALSs were disentangled. These findings were and compared with the skills and knowledge needed by teachers providing differentiated instruction without these tools. Results show that teachers dashboards can support teachers when providing differentiated instruction by providing up-to-date information about students progress. Nevertheless, coping with the new information provided on the dashboards also requires new skills and knowledge such as interpreting information from the dashboard and understanding how the information in the ALS is calculated. Although there are some differences, the core principles of differentiated instruction1) being goal-oriented, 2) continually monitoring, 3) adapting instruction and assignments, 4) being ambitious, 5) stimulating students self-regulationare critical for high-quality differentiated instruction regardless of the available tools in the classroom.
... Studies spanning a wide variety of domains, including but not limited to medical field, business sector, and software design, indicated that 30-45% of learning performance was effectively increased using CTA-informed instruction compared to traditional methods of observation and analysis (Clark, 2014;Lim, 2015). Therefore, CTA can help in capturing observable expert knowledge, processes, and goals, eventually leading to using that captured knowledge in instruction, training, performance assessment, and systems development (Mutie, 2015). ...
This empirical research aims to explore physical education teachers’ readiness and develop an evaluation platform for wearable technology as a digital innovation in physical education. The research adopts a mixed-method approach with qualitative and quantitative methodologies for collecting data through systematic literature review, case study, survey questionnaire, and an evaluation platform. 115 peer-reviewed articles formed the basis of the systematic review. Responses of 38 expert physical education teachers in Kuwait, collected from semi-structured interviews and cognitive task analysis, formed the basis for qualitative analysis. The responses of 346 physical education teachers in Kuwait, collected from a questionnaire, formed the basis for quantitative analysis. In addition, usability testing and responses of 13 physical education teachers shaped the evaluation platform. The systematic review identifies different research foci for educational research and loci for technological innovation with insights that pose theory-based behavioral and technological conundrums. The case study findings identifies cognitive tasks, organizational and technological conditions for wearables. The survey develops a measurement scale for teacher readiness and tests the influence of readiness on perceived innovativeness. Using case study and survey insights, an evaluation platform was developed to support physical education teachers. The research offers theoretical, practical, and policy implications, shaping policy for strategies to introduce digital innovations in physical education.
Ophthalmoscopy is a core component of a complete ophthalmic examination. Due to its complex technical aspects and patients becoming uncooperative if the procedure is prolonged, it can be a difficult skill for a novice to learn and develop proficiency in. Skills instruction is typically provided by subject matter experts (SMEs) through free recall without an agreed-upon instructional framework. This can lead to unintentional omission of essential steps and knowledge required to perform skills correctly. Cognitive task analysis (CTA) allows for construction of standardized instructional protocols that encompass the knowledge and skills experts apply when performing tasks. The objectives of this study were to (a) develop a CTA-based teaching protocol for canine indirect ophthalmoscopy and (b) compare the steps verbalized or demonstrated by SMEs during free recall instruction versus those ultimately identified by CTA. Four SMEs participated in free recall instructional sessions and interviews used for the development of a CTA-based teaching protocol for novice learners. The CTA-based protocol identified 66 steps and sub-steps considered essential for successfully performing canine indirect ophthalmoscopy. During instructional sessions, SMEs on average failed to verbalize 57.1% of clinical knowledge steps, did not verbalize or demonstrate 68.3% and 9.5% of action steps and did not verbalize or demonstrate 73.2% and 40.4% of decision steps, respectively. This study demonstrates that SMEs teaching indirect ophthalmoscopy by free recall may unintentionally omit important steps, suggesting that compared with free recall, CTA may generate more comprehensive and thus potentially more effective instructional materials for teaching technical skills in veterinary medicine.
This volume is the first comprehensive history of task analysis, charting its origins from the earliest applied psychology through to modern forms of task analysis that focus on the study of cognitive work. Through this detailed historical analysis, it is made apparent how task analysis has always been cognitive.
Developing effective instruction for complex problem-solving tasks requires analysis of the cognitive processes and structures that contribute to task performance. This report describes the data collection procedures associated with a cognitive task analysis technique known as the PARI (precursor, action, result, and interpretation) methodology. The methodology is being developed under the Basic Job Skills (BJS) program and constitutes one component of an integrated technology for developing and delivering training of cognitively complex tasks. The data collection procedures can be considered an extension of existing task analysis techniques and are based on studies of over 200 Air Force technicians in aircraft maintenance specialties whose primary task is troubleshooting. The procedures derived from these studies impose a structure on the knowledge acquisition task which captures the cognitive as well as the behavioral components of troubleshooting skill. The structured interview approach yields data that allow qualitative comparisons of problem-solving performances within and across technical skill levels. Such analyses have informed instruction developed under the BJS program by revealing the developmental course of skill acquisition and the components of expertise which are the training targets. More recent analyses have identified skill and knowledge commonalities across maintenance specialties and are informing training designed to facilitate knowledge transfer. A future goal of the BJS program is to examine the generality of the PARI methodology and the extent to which it can be applied to problem-solving tasks in nonmaintenance domains.