Content uploaded by Laura G Militello
Author content
All content in this area was uploaded by Laura G Militello
Content may be subject to copyright.
A preview of the PDF is not available
Applied Cognitive Task Analysis (ACTA): A Practitioner's Toolkit for Understanding Cognitive Task Demands
Ergonomics
Abstract
Cognitive task analysis (CTA) is a set of methods for identifying cognitive skills, or mental demands, needed to perform a task proficiently. The product of the task analysis can be used to inform the design of interfaces and training systems. However, CTA is resource intensive and has previously been of limited use to design practitioners. A streamlined method of CTA, Applied Cognitive Task Analysis (ACTA), is presented in this paper. ACTA consists of three interview methods that help the practitioner to extract information about the cognitive demands and skills required for a task. ACTA also allows the practitioner to represent this information in a format that will translate more directly into applied products, such as improved training scenarios or interface recommendations. This paper will describe the three methods, an evaluation study conducted to assess the usability and usefulness of the methods, and some directions for future research for making cognitive task analysis accessible to practitioners. ACTA techniques were found to be easy to use, flexible, and to provide clear output. The information and training materials developed based on ACTA interviews were found to be accurate and important for training purposes.
... The HMT Knowledge Audit (McDermott et al., 2018) is an interview technique designed to uncover and understand HMT requirements. With origins in the Applied Cognitive Task Analysis toolkit's knowledge audit (Militello and Hutton, 1998), the questions are driven by the themes identified in the HMT Framework and the need to collect information to flesh out the general requirements. ...
... Some tools for HMT and related fields have focused explicitly on their usability and usefulness for systems development. Well-known examples include the System Usability Scale (Brooke, 1996), Cognitive Walkthrough (Rieman et al., 1995), Applied Cognitive Task Analysis (Militello and Hutton, 1998), and Project Premortem (Klein, 2007). These techniques have each distilled dense theoretical concepts into methods that can be readily implemented with minimal training. ...
This paper was completed as part of a comprehensive exam milestone for PhD candidacy in Human Systems Engineering at Arizona State University.
Abstract:
The US Department of Defense (DoD)’s future missions increasingly include Autonomous and AI-Enabled Systems (AAIS) that work in partnership with warfighters. To realize this partnership, effective Human-Machine Teaming (HMT) design needs to be integrated into the DoD acquisition of AAIS. Within the complex ecosystem of DoD acquisition, assessment through Test and Evaluation (T&E) is critical for ensuring engineering outcomes like HMT. Despite the prioritization of HMT evident in DoD strategic efforts and arguably sufficient capability to assess HMT, there are low levels of adoption of HMT assessment in DoD T&E of AAIS. To unpack and better understand the disparity between priority, capability, and adoption of HMT assessment, this narrative review explores barriers to adoption of current HMT assessment methods that can be gleaned from research, DoD guidance and policy, and the HMT community’s discourse on adoption. This synthesis led to eight challenges that are organized into themes of systems engineering policies and practices, the test workforce who would implement test methods and tools, and agreement about HMT assessment. In the long-term, achieving levels of adoption that meet the demands of HMT risks in AAIS may involve corresponding changes to policies, workforce development, cultural shifts, and development of HMT assessment tools intentionally designed to support DoD acquisition. In the near term, practical HMT assessment guidance may support DoD T&E stakeholders to adopt current methods and tools.
... The process continued until saturation was achieved, at which point no additional relevant concepts were extracted from the narratives. The interview protocol was adapted from Militello and Hutton (1998). All interviews were audio-recorded and transcribed for analysis. ...
... This research, which is inherently exploratory, aimed to delve into various facets of leadership tasks and responsibilities. A Cognitive Task Analysis (CTA) framework, as delineated by Militello and Hutton (1998) and Crandall et al. (2006), structured the interview protocol. Meanwhile, a mixed method containing Interpretative Phenomenological Analysis (IPA) and Cluster Analysis on the codes were used. ...
Wisdom is proposed as a bridge between leader and leadership. It is necessary to explore what characteristics wise leaders have and what approaches they use to solve problems. In this study, we suggest a unique approach involving qualitative and quantitative research methods to seek the essence of wise leadership and its styles. The research employed Cognitive Task Analysis (CTA) to conduct interviews with nine wise leaders in management positions. The data were then analyzed using a circular mixed method combining Interpretative Phenomenological Analysis (IPA) to delve into the participants’ job experiences and cluster analysis through R software to extract the categories. Results identified five key themes: personal traits, processes, conscientiousness, going beyond assigned responsibilities, and intrapersonal skills. Additionally, two distinct approaches-- transformational wise leadership, and transactional wise leadership--were highlighted. The wise-leader model and its styles can be used for leadership-development programs, recruitment, and evaluation of leaders.
... However, reports about CEiED approaches are readily available in addition to those cited, as well as specific methods such as cognitive task analysis. 22,25,33,34 Lastly, comparative studies with other self-debriefing approaches are needed to identify specific advantages of the CEiED approaches. Specifically, the CEiED approaches should be compared to other self-discovery-oriented approaches as well as more systematic methods that tightly control the debriefing process. ...
Introduction
Online experiential learning can benefit learners with scalable techniques to self-debrief and to develop cognitive skills in recognizing clinical cues.
Methods
We developed techniques for cue-detection exercises and self-debriefing, based on cognitive engineering-inspired expertise development approaches that focus on tacit knowledge, sensemaking, and mental models. Self-debriefing was structured by asking the learners to compare and then reflect on their choices and rationales against those provided by a panel of experts. Using these techniques, we generated scenario-based experiential learning materials in a virtual environment for a 2-hour module on deteriorating patient conditions that can lead to imminent cardiac arrests. The module was tested in a senior nursing course as an optional assignment. The volume of voluntarily submitted reflections by learners was used to assess engagement and subsequent thematic analysis to assess feasibility of the techniques.
Results
The module was completed by 189 of the 197 students invited (95.9%). Engagement level was high with all completed students submitting reflections after self-debriefing, most of which (between 53.4% and 87.8%) were specific enough for thematic analysis. The main theme of reflections was “missing something” in the scenario, followed by the themes of importance of reading the patient monitor and refining actions and priorities.
Conclusions
We demonstrated the feasibility of the techniques based on cognitive engineering-inspired approaches for virtual simulation learning in health care that structures self-debriefing by comparing a learner's situation assessment and responses with those of experts. The techniques have the potential to help learners in health care efficiently and consistently develop key critical thinking skills, especially those based on tacit knowledge to detect cues.
... To fill this knowledge gap, the MITRE Corporation used Applied Cognitive Task Analysis [24] to interview eleven experienced IP CTS users. Study participants were asked to (1) describe challenges with the technology, (2) why it is difficult, (3) the strategies they use to deal with these issues, and (4) potential errors a novice might make in the same situation. ...
The Telecommunications Relay Service (TRS) industry, comprised of organizations supported and regulated by the Federal Communications Commission (FCC), strives to provide individuals who are d/Deaf, Hard of Hearing, DeafBlind, or have speech impediments with telecommunication services that are “functionally equivalent” to those used by the broader population. There has been much debate within the TRS community about what “functionally equivalent” means. In response to the need to determine if TRS are functionally equivalent, this paper describes the Effective Communication Framework (ECF), which extends prior communications models that address the components and processes of telecommunication between callers with varying abilities to hear and speak. Qualitative and quantitative research applying the ECF is detailed as evidence of its validity and utility for researchers and practitioners. The ECF can help provide a basis for reliable research design, identify gaps in knowledge and capabilities, guide data collection and analysis, and facilitate communication among researchers and practitioners, especially those looking to improve assistive communication technologies like TRS.
This chapter explored the use of an electronic learning prototype to enhance the pedagogical knowledge and skills of Malaysian higher education educators. It assessed the benefits of an e-learning platform for addressing teaching challenges. Guided by constructivism, connectivism, activity theory, and adult learning theories, the study used design-based research (DBR) methodology, adapted from Reeves, in eight phases: analysis, design, validation, development, implementation, evaluation, impact, and continuous learning. Eight experts identified educators' professional development needs, while 43 educators participated in a training course using the prototype. Post-course surveys revealed readiness and motivation to integrate technology into teaching, indicating that electronic training can offer a scalable, sustainable solution for continuous professional development.
This chapter outlines the design and development of a mobile flipped learning module aimed at enhancing teaching practices among educators in TVET higher learning institutions through the flipped classroom approach. The mobile flipped learning (MFL) module plays a significant role in fostering the professional development of lecturers, enabling them to incorporate the flipped classroom strategy into their teaching practices. Utilizing a design-based research technique, the module development was divided into four major phases – analysis, design and develop with validation using ACTA technique, implementation, and usability. It involves 30 lecturers with a mixed method data collection technique. Findings indicate a positive impact on lecturers' professional development, enhancing their performance, attitude, behavior, and effort in adopting new practices with the MFL module. This strategy offers new insights into integrating technology into teaching practices and enhancing students' learning experiences.
This chapter discusses the design and development of TaSPoD (electronic training system for professional development) using design-based research (DBR) approach. TaSPoD is essential for enhancing professional development in higher education by integrating electronic training into development plans, benefiting institutions, experts, and trainees. The DBR approach merges design principles with solutions tailored to the needs of e-training environments, facilitating the design, development, and evaluation of TaSPoD. The initial analysis phase identifies training issues, while the design phase focuses on creating effective training modules and the system framework. Experts gather further information to refine TaSPoD's design using the applied cognitive task analysis (ACTA) method. After validating the cognitive tasks, the TaSPoD prototype will be developed and evaluated for effectiveness by experts. Ultimately, TaSPoD aims to bridge cultural, religious, and technological gaps, enabling educators of all genders to engage in professional development and maximize their potential.
This chapter explores the development of microlearning pedagogical practices aimed at enhancing student learning in higher education through a design-based research (DBR) approach. Recognizing microlearning's potential for quick, focused learning, the study addresses the lack of strong pedagogical foundations and practical implementation challenges. The research progresses through four phases: analyzing current teaching practices, designing key principles, developing pedagogical strategies, and evaluating their usability via expert feedback. Using surveys and applied cognitive task analysis (ACTA), the study introduces the AMPLIFY framework, which enhances microlearning with eight components: clear objectives, personalized learning, continuous feedback, efficient delivery, engaging content, interactive experiences, social collaboration, and seamless curriculum integration. This research offers practical insights and a systematic approach for implementing microlearning strategies in higher education, bridging the literature gap and setting the stage for future studies.
Experts, who were the sole active dispensers of certain kinds of knowledge in the days before AI, have now often assumed a rather passive role. They relay their knowledge to various novices, knowledge engineers, experimental psychologists or cognitivists - or other experts! - involved in the development and understanding of expert systems. This book achieves a perfect marriage between experimentalists and theoreticians who deal with expertise. It tries to establish the benefits to society of an advanced technology for representing and disseminating the knowledge and skills of the best corporate managers, the most seasoned pilots, and the most renowned medical diagnosticians. This book interests psychologists as well as all those out in the trenches developing expert systems, and everyone pondering the nature of expertise and the question of how it can be studied scientifically. Its scope, the pivotal concepts which it elucidates and brilliantly summarizes and appraises in the final chapter, as well as the references it includes, make this book a landmark in the field.
Cognitive Systems Engineering (CSE) is primarily a blend of technological opportunities, findings from cognitive research, and Cognitive Task Analysis. Using CSE, we were able to produce an efficient and effective redesign of the AWACS Weapons Director (WD) station. The design effort was completed in a relatively short period of time.
A Cognitive Task Analysis was performed using two interview techniques: Concept Mapping and the Critical Decision method (CDM). The information obtained using these methods pinpointed specific cognitive areas which any redesign of the WD interface must address. A revised interface was developed and evaluated.
During the evaluation, the training of the participants on the revised interface was quite brief (4.5 hours). As a result, the WDs did not achieve the same degree of familiarity or automatization with the revised interface that they have with the current interface. Yet, when WDs were using the revised system their performance improved. This was indicated by an increase in performance for a number of process and outcome measures. Also, a skilled WD provided blind ratings of WD performance. These global ratings were significantly higher for the revised interface.
The effectiveness of the revised interface suggests that it is possible to pinpoint cognitive task requirements and to make these the driving factors in a design effort. Moreover, these Cognitive Systems Engineering activities do not consume a great deal of time or effort. The use of CSE may be a feasible aspect of the design process, enabling system developers to achieve a much stronger effectiveness at relatively low cost.
This paper presents the findings of a study on how experienced naval officers make decisions in a complex, time-pressured command and control setting, the Combat Information Center of AEGIS cruisers. The decision processes invoked by the officers were consistent with the recognition-primed decision model. The majority of decisions concerned situation awareness and diagnosis in which the decision makers used feature-matching and story generation strategies to build situation awareness. Furthermore, awareness of the situation enabled the officers to recognize appropriate actions from published procedures or past experience. A recognitional strategy was used to identify 95% of the actions taken; decision makers compared multiple options in only 4% of the cases. These findings are discussed in terms of their implications for framing command-and-control problems, for emphasizing situation awareness, for a descriptive model of decision making, and for designing decision supports.
Two studies supporting the use of the Critical Decision method (CDM) in eliciting knowledge from expert neonatal intensive care unit (NICU) nurses are presented. The first examines the utility of CDM in the nursing profession. In this study, significantly more information was elicited in CDM interviews than in non-CDM interviews. In the second study, cues, indicators, and exemplars were extracted from CDM incident accounts to form a guide to early sepsis assessment in the NICU that contains information not available in the current literature. All evaluators rated the guide as useful. Implications for future research, including generalizability to other areas of nursing, are discussed.
Part 1: the task analysis process. Part 2: task analysis techniques task data collection task description methods task simulation methods task behaviour assessment methods task requirements evaluation methods. Part 3: task analysis case studies balancing automation and human action through task analysis a preliminary communications system assessment a plant local panel review a staffing assessment for a local control room task simulation to predict operator workload in a command systenm task analysis of operator safety actions maintenance training a method for quantifying ultrasonic inspection effectiveness operational safety review of a solid waste storage plant a task analysis programme for a large nuclear chemical plant.