Article

4. Some guidelines for conducting a cognitive task analysis

Authors:
  • ShadowBox LLC & MacroCognition LLC
  • Applied Decision Science, LLC
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Cognitive Task Analysis (CTA) attempts to explain the mental processes involved in performing a task. These processes include the knowledge, skills and strategies that are needed to accomplish the task functions. The criteria for success in a CTA study are: making a useful discovery about the cognitive skills being studied; being able to communicate the discovery to the users (i.e. those who will need to use the CTA for design); and having a meaningful impact on the eventual design.Currently, a wide variety of CTA methods are being used. As we learn how to define the cognitive demands presented by a task/situation, we hope we will be able to map CTA methods onto these demands, so that we can more efficiently select and apply the appropriate methods. This should result in more efficient studies, and greater user satisfaction. It should also help move the field of CTA into becoming more of a technology.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Moreover, meaningful evaluation of human understanding is challenging [60,61]. Human evaluation is essential to assess interpretability and produce actionable design knowledge, but this requires researchers to find ways to peer inside the heads of those who use data systems [44,53]. For most studies that evaluate human understanding, researchers often rely on numerical self-reported measures of trust administered once at the end of the session [83] or throughout the task [46,69]. ...
... Because mental models tend to be fuzzy [65], providing some clear examples helps users bring clarity to their interpretation. Typically, a constrained task is presented to users that helps to dissect the cognitive processes underlying their mental state [53]. The matching mental models method asks users to select an explanation or diagram that is the "Nearest Neighbor" to their beliefs. ...
... Typically interviews are agnostic to temporal phenomena, and this lack of context can lead users to overgeneralize and potentially forget key details. In addition to post-task interviews, Klein and Mitello describe cognitive task analysis, which attempts to derive the cognitive skills involved with a task [53]. Generally, experts are prompted to repeatedly walk through the decision making steps in higher and higher detail to help researchers identify strategies these experts have learned to use in their domain. ...
Preprint
Many interactive data systems combine visual representations of data with embedded algorithmic support for automation and data exploration. To effectively support transparent and explainable data systems, it is important for researchers and designers to know how users understand the system. We discuss the evaluation of users' mental models of system logic. Mental models are challenging to capture and analyze. While common evaluation methods aim to approximate the user's final mental model after a period of system usage, user understanding continuously evolves as users interact with a system over time. In this paper, we review many common mental model measurement techniques, discuss tradeoffs, and recommend methods for deeper, more meaningful evaluation of mental models when using interactive data analysis and visualization systems. We present guidelines for evaluating mental models over time that reveal the evolution of specific model updates and how they may map to the particular use of interface features and data queries. By asking users to describe what they know and how they know it, researchers can collect structured, time-ordered insight into a user's conceptualization process while also helping guide users to their own discoveries.
... 14 Deep conceptual knowledge of the domain is fundamental to expertise. 15 The mental models of experts incorporate a broader set of causal relationships 16 and support better anticipation and inferential reasoning. 14 Experts are able to better detect and use cues 14,16 while thinking more strategically 11,14,16 and understanding the entire perspective. ...
... 15 The mental models of experts incorporate a broader set of causal relationships 16 and support better anticipation and inferential reasoning. 14 Experts are able to better detect and use cues 14,16 while thinking more strategically 11,14,16 and understanding the entire perspective. 17 The high turnover among emergency medical technician (EMT) workers 18 deprives units of this expertise and provides further reason for addressing the shortage of research on paramedic cognition. ...
... 15 The mental models of experts incorporate a broader set of causal relationships 16 and support better anticipation and inferential reasoning. 14 Experts are able to better detect and use cues 14,16 while thinking more strategically 11,14,16 and understanding the entire perspective. 17 The high turnover among emergency medical technician (EMT) workers 18 deprives units of this expertise and provides further reason for addressing the shortage of research on paramedic cognition. ...
Article
Full-text available
Out-of-hospital care is becoming more complex, thus placing greater reliance on the cognitive abilities of paramedics to manage difficult situations. In adapting to the challenges in their work, paramedics develop expertise. We study the cognitive strategies used by expert paramedics to contribute to understanding how paramedics and the EMS system can adapt to new challenges. We conducted a "staged-world" cognitive task analysis to explore paramedics' handling of cognitive challenges related to sense-making and to resource and task management. A mixed-fidelity simulation was used to present paramedics with 2 challenging scenarios: a pulmonary embolism initially presenting as a myocardial infarction and a 2-person shooting with limited resources available. Participants were 10 paramedics, 6 more experienced and 4 less experienced. Analysis involved comparing the performance of the 2 groups to identify strategies associated with expertise. The more experienced paramedics made more assessments, explored a wider variety of presumptive diagnoses, and identified the pulmonary embolism earlier. They switched attention between the 2 shooting victims more, used their emergency medical technician-basic level partners more, and provided more advanced level care for both patients. Their patients arrived at the emergency department more prepared for specialized emergency care. Our findings correspond to general cognitive attributes of expertise: greater cue gathering and inferential reasoning, and more functional and strategic thinking. These results suggest potential areas and methods to facilitate development of expertise, as well as ways to better support use of expertise. Future studies should expand on these findings through larger sample sizes and more complex scenarios.
... In identifying the cognitive processes underlying expert tactical decision making, this study highlights skills found to be most appropriate and successful in British police firearms domains, and as a result, instructors can concentrate their training accordingly (Klein & Militello, 2001;WBI Evaluation Group, 2007). In other professions requiring rapid decision making in risky environments, there has been significant research to understand and train operational thinking skills-for example, aviation (Orasanu & Fischer, 1997;Seamster et al., 1993) and prehospital/medical emergency (Gunnarsson & Stomberg, 2009;Wong & Blandford, 2002). ...
... The CDM interview is a multistage process that utilizes multiple "sweeps" through an incident. These sweeps build in intensity-from brief and general incident recall to an intensive examination guided by the creation of a visual timeline, identification of decision points, and subsequent probing and hypothetical questioning regarding those decision points (Hoffman, Crandall, & Shadbolt, 1998;Klein & Militello, 2001). ...
... Mental modeling was described by 11 expert SFOs as a preparatory process, which aided later adaptation (Klein & Militello, 2001;Pirolli & Card, 2005). Eleven expert SFOs used mental models to mentally simulate potential solutions and the associated outcomes to assess these options and adapt their behavior accordingly. ...
Article
Identifying the cognitive processes underlying tactical decision making is vital for two purposes: (a) reducing risk through improved training and (b) facilitating the public’s attitudes toward the legitimacy of the police and criminal justice system. Despite this, very little research has been conducted into British police decision making involving the use of firearms. This study begins to address this gap by examining the impact that expertise has on British police’s use-of-force decisions during armed confrontations. To do so, the tactical decision-making processes of 12 expert specialized firearms officers and 11 novice authorized firearms officers during armed confrontations were compared through cognitive task analysis methods. Data were coded via categories derived from theory and patterns inductively emergent within the data. The results found expert specialized firearms officers to be more flexible in adaptive responding to situational changes, while novice authorized firearms officers reported a more sequential and linear process of tactical decision making. In identifying the key features of expertise within this environment (“adaptive flexibility”), this study has theoretical and practical implications for the acceleration of authorized firearms officers’ expertise acquisition to bridge the existing gap resulting from a lack of available qualified operational force commanders.
... Researchers developing a simulation environment need a thorough understanding of the task and its key characteristics [24]. Therefore a team consisting of clinicians, simulator developers, engineers, and human factor experts should collaboratively deduce and define key characteristics of the simulator environment. ...
... Our CTA-based approach that encompassed in situ observations in the OR, expert interviews, and an expert consensus panel consisted of two parts: knowledge elicitation and knowledge representation [24]. For the first part, elicitation aims to capture experts' cognitive knowledge that is often tacit and automated [1]. ...
... In preparation of step III, we first determined appropriate methods to elicit knowledge from experts. To decide on CTA methods, it is important to consider which types of cognitive processes and tasks should be investigated [24]. We defined requirements of the CTA methods to obtain essential information on tasks and simulation requirements. ...
Article
Full-text available
Background: Despite the growing importance of medical simulation in education, there is limited guidance available on how to develop medical simulation environments, particularly with regard to technical and non-technical skills as well as to multidisciplinary operating room (OR) team training. We introduce a cognitive task analysis (CTA) approach consisting of interviews, structured observations, and expert consensus to systematically elicit information for medical simulator development. Specifically, our objective was to introduce a guideline for development and application of a modified CTA to obtain task demands of surgical procedures for all three OR professions with comprehensive definitions of OR teams' technical and non-technical skills. Methods: To demonstrate our methodological approach, we applied it in vertebroplasty, a minimally invasive spine procedure. We used a CTA consisting of document reviews, in situ OR observations, expert interviews, and an expert consensus panel. Interviews included five surgeons, four OR nurses, and four anesthetists. Ten procedures were observed. Data collection was carried out in five OR theaters in Germany. Results: After compiling data from interviews and observations, we identified 6 procedural steps with 21 sub-steps for surgeons, 20 sub-steps for nurses, and 22 sub-steps for anesthetists. Additionally, we obtained information on 16 predefined categories of intra-operative skills and requirements for all three OR professions. Finally, simulation requirements for intra-operative demands were derived and specified in the expert panel. Conclusions: Our CTA approach is a feasible and effective way to elicit information on intra-operative demands and to define requirements of medical team simulation. Our approach contributes as a guideline to future endeavors developing simulation training of technical and non-technical skills for multidisciplinary OR teams.
... There are many CTA methods used to facilitate knowledge engineering and each varies in terms of their knowledge representation, how information is extracted from participants, and the types of tasks the methods work best for. These facets are further divided into their own categories of methods for knowledge elicitation and representation [9]. When used for designing training systems, CTA is used two-fold, to first define the difficulty of the task in the problem space, determine common errors encountered by expert in the field, and produce ways to mitigate those errors [10]. ...
... Some specific CTA methodologies employed for knowledge elicitation include unstructured interviews, critical decision analyses, direct observation and questioning, and simulations [9]. In these approaches, researchers will either ask participants a series of questions, allowing for feedback and revising of further questions, or observe participants as they attempt to solve a problem or walk through a process, usually with some verbal component from either party. ...
Preprint
Full-text available
Computer-aided diagnosis for medical imaging is a well-studied field that aims to provide real-time decision support systems for physicians. These systems attempt to detect and diagnose a plethora of medical conditions across a variety of image diagnostic technologies including ultrasound, x-ray, MRI, and CT. When designing AI models for these systems, we are often limited by little training data, and for rare medical conditions, positive examples are difficult to obtain. These issues often cause models to perform poorly, so we needed a way to design an AI model in light of these limitations. Thus, our approach was to incorporate expert domain knowledge into the design of an AI model. We conducted two qualitative think-aloud studies with doctors trained in the interpretation of lung ultrasound diagnosis to extract relevant domain knowledge for the condition Pneumothorax. We extracted knowledge of key features and procedures used to make a diagnosis. With this knowledge, we employed knowledge engineering concepts to make recommendations for an AI model design to automatically diagnose Pneumothorax.
... This makes a knowledge audit suitable for an initial set of interviews to obtain information from pilots. The knowledge audit method is well suited for identifying perceptual skills including cues and patterns needed for a given task (Klein and Militello, 2001). As helicopter landings are very reliant on perceptual cues (Smith, 2006), the knowledge audit fits well with the objectives of our study. ...
... This simulation environment presents pilots with a helicopter task requiring them to demonstrate the task of approaching and landing a helicopter at a specific point. Operators proficient in a given task often find it challenging to describe the perceptual skills and mental models they possess (Klein and Militello, 2001). By using a simulation interview, the researchers can directly observe and query the operators. ...
Article
The task of landing helicopters on ships is cognitively complex and is bounded by several limitations for safe operation. To design technologies to support helicopter pilots in shipboard operations, a better understanding of cognitive processes underlying helicopter piloting in shipboard landing maneuvers is required. Limitations in prior work on ship-based helicopter pilots motivate more studies to better understand how pilots compensate for difficulties associated with ship motion, air turbulence among other factors. We report a preliminary study that used the Applied Cognitive Task Analysis (ACTA) method. We interviewed three participants who have ground-based helicopter piloting experience. The results indicate that ACTA can be used in eliciting knowledge from helicopter pilots. However, we have identified several caveats with the use of the method for this application. The revised method will be used to interview helicopter pilots with shipboard landing experience. Our study has identified a number of cognitive elements in the task which are associated with task management, divided attention, and noticing anomalies.
... This makes a knowledge audit suitable for an initial set of interviews to obtain information from pilots. The knowledge audit method is well suited for identifying perceptual skills including cues and patterns needed for a given task (Klein and Militello, 2001). As helicopter landings are very reliant on perceptual cues (Smith, 2006), the knowledge audit fits well with the objectives of our study. ...
... This simulation environment presents pilots with a helicopter task requiring them to demonstrate the task of approaching and landing a helicopter at a specific point. Operators proficient in a given task often find it challenging to describe the perceptual skills and mental models they possess (Klein and Militello, 2001). By using a simulation interview, the researchers can directly observe and query the operators. ...
Conference Paper
The task of landing helicopters on ships is cognitively complex and is bounded by several limitations for safe operation. To design technologies to support helicopter pilots in shipboard operations, a better understanding of cognitive processes underlying helicopter piloting in shipboard landing maneuvers is required. Limitations in prior work on ship-based helicopter pilots motivate more studies to better understand how pilots compensate for difficulties associated with ship motion, air turbulence among other factors. We report a preliminary study that used the Applied Cognitive Task Analysis (ACTA) method. We interviewed three participants who have ground-based helicopter piloting experience. The results indicate that ACTA can be used in eliciting knowledge from helicopter pilots. However, we have identified several caveats with the use of the method for this application. The revised method will be used to interview helicopter pilots with shipboard landing experience. Our study has identified a number of cognitive elements in the task which are associated with task management, divided attention, and noticing anomalies.
... Ergonomics has many tools and design models to promote the practical acceptance of technologies, such as for example the Cognitive Task Analysis and the User-Centered Design. Cognitive Task Analysis attempts to identify and explain the mental processes implemented in performing a task (Klein and Militello 2001) with the aim of defining the functionalities and optimizing the ease of use of the technologies by the users. User-centered design (Norman and Draper 1986;ISO 9241-210: 2019), which seeks to make technologies more useful, usable and acceptable to users, is of particular interest in relation to practical acceptance, insofar as it takes account of the dynamic nature of both instrumental and non-instrumental aspects, meaning that users can be involved at various stages of the design process. ...
Article
Ergonomics has several theoretical frameworks at its disposal for assessing the acceptability of technologies, and it uses numerous methodologies that exist within the discipline. Authors generally base their studies on one or other of these theoretical and methodological approaches, which are sometimes seen as mutually incompatible. The present article, however, seeks to demonstrate the interest of combining three methodological approaches (i.e., experimental, ecological, and prospective methodologies) in designing emerging technologies that will be accepted. Our first focus is conceptual, and the arguments we present for taking a holistic view of the acceptability of emerging technologies are based on the foundations, the contributions and the limitations of three theoretical frameworks of acceptance (i.e., social acceptance, practical acceptance, and situated acceptance). Our second focus is methodological: using our own work as evidence we argue that an experimental methodology can further both the practical acceptance and the social acceptance of Virtual Reality, that an ecological methodology can further its situated acceptance, and that a prospective methodology has relevance to all three facets. We discuss and suggest some recommendations regarding the use of these methodologies according to the type of design project.
... The primary users of implementation strategies often include both the targets of those strategies as well as the implementation practitioners who deliver them. For instance, testing components of a leadership-focused implementation strategy (e.g., Leadership and Organizational Change for Implementation; [55]) could include representative leaders from the organizations in which the strategy is likely to be applied as well as leadership coaches from the implementation team. If a broader strategy (e.g., change record systems; [5]) is selected for testing, then multiple groups re ecting different user types may need to be recruited (e.g., clinicians, supervisors, patients). ...
Preprint
Full-text available
Background. Implementation strategies have flourished over the last decade in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially-mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which implementation strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to surface aspects of strategies that may inhibit their usability and, ultimately, their effectiveness. This article presents a novel cognitive walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support mental health clinicians in the education sector to adopt measurement-based care. Method. The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially-mediated implementation strategies in health. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed a quantitative assessment of strategy usability. Results. Average subtask success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale: 0-100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative coding and classified by severity and problem type to explain the ratings. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. Conclusions. Ratings indicated that usability of the consultation protocol was at the low end of the “acceptable” range. Collectively, the 21 usability issues explained the ISUS quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.
... Interview protocols loosely followed the procedures outlined in a cognitive task analysis, or critical decision method (Klein & Militello, 2001;Klein, Calderwood, MacGregor, 1989). Cognitive task analysis methods seek to describe cognitive demands of the task and situation, define task constraints, and provide a framework for the systematic interpretation of the qualitative results. ...
Article
Full-text available
The present paper outlines an approach to representing cognitive, cultural, and physiological variability in the computational representation an individual U.S. peacekeeper as he interacts with an unexpected target (two young Iraqi girls) in an ambiguous situation while faced with a high-consequence decision that will greatly impact subsequent events. This project sought to demonstrate steps toward a realistic computational representation of the variability encountered in individual human behavior. Realism, as conceptualized in this project, required that the human representation address the underlying psychological, cultural, physiological, and environmental stressors. A software model of a peacekeeping scenario adapted from a Desert Storm incident was developed in which the framework consisted of a computational instantiation of Recognition Primed Decision Making in the context of a Naturalistic Decision Making model (Klein, 1997). Recognition Primed Decision Making was augmented with an underlying foundation based on an understanding of human neurophysiology and it's relationship to human cognitive processes. The goal was to provide initial steps toward a computational representation of human variability in cultural, cognitive, and physiological (arousal, emotions, etc.) state order to attain a better understanding of the full depth of human decision-making processes in the context of ambiguity, novelty, and heightened arousal.
... Separately, VR is also proving it can support collaborative tasks. When used as a medium for telepresence [16,17] -the feeling of being there -VR has been shown to increase situation awareness [18] and the vividness [19], interactivity [17] and media richness [20,21], as well as the proprioceptive [22] or kinesthetic [23] properties of remote experiences. These qualities have been shown to enhance collaborative teleoperation [12] scenarios like remote driving [24] and surgery [25], again, just to cite a few. ...
Article
The growing complexity of scientific data poses serious challenges for an effective visualization. Data sets, e.g., catalogs of objects detected in sky surveys, can have a very high dimensionality, ~ 100 - 1000. Visualizing such hyper-dimensional data parameter spaces is essentially impossible, but there are ways of visualizing up to ~ 10 dimensions in a pseudo-3D display. We have been experimenting with the emerging technologies of immersive virtual reality (VR) as a platform for a scientific, interactive, collaborative data visualization. Our initial experiments used the virtual world of Second Life, and more recently VR worlds based on its open source code, OpenSimulator. There we can visualize up to ~ 100,000 data points in ~ 7 - 8 dimensions (3 spatial and others encoded as shapes, colors, sizes, etc.), in an immersive virtual space where scientists can interact with their data and with each other. We are now developing a more scalable visualization environment using the popular (practically an emerging standard) Unity 3D Game Engine, coded using C#, JavaScript, and the Unity Scripting Language. This visualization tool can be used through a standard web browser, or a standalone browser of its own. Rather than merely plotting data points, the application creates interactive three-dimensional objects of various shapes, colors, and sizes, and of course the XYZ positions, encoding various dimensions of the parameter space, that can be associated interactively. Multiple users can navigate through this data space simultaneously, either with their own, independent vantage points, or with a shared view. At this stage ~ 100,000 data points can be easily visualized within seconds on a simple laptop. The displayed data points can contain linked information; e.g., upon a clicking on a data point, a webpage with additional information can be rendered within the 3D world. A range of functionalities has been already deployed, and more are being added. We expect to make this visualization tool freely available to the academic community within a few months, on an experimental (beta testing) basis.
... CTA identifies the cognitive strategies and patterns from experts that trainees must learn in order to perform effectively through the use of various techniques, such as observation, structured interviews, and verbal protocols (Miller et al., 2006). Research has shown that CTAs can positively impact the design of training (Hoffman & Lintern, 2006;G. Klein & Militello, 2001) and aid in generating relevant training scenarios and practice opportunities (see Salas & Klein, 2001;Schraagen, Chipman, & Shalin, 2000). Conduct a CTA when jobs are knowledge based and when there is a need to unpack the expertise and uncover the cognitive requirements. ...
Article
Full-text available
Organizations in the United States alone spend billions on training each year. These training and development activities allow organizations to adapt, compete, excel, innovate, produce, be safe, improve service, and reach goals. Training has successfully been used to reduce errors in such high-risk settings as emergency rooms, aviation, and the military. However, training is also important in more conventional organizations. These organizations understand that training helps them to remain competitive by continually educating their workforce. They understand that investing in their employees yields greater results. However, training is not as intuitive as it may seem. There is a science of training that shows that there is a right way and a wrong way to design, deliver, and implement a training program. The research on training clearly shows two things: (a) training works, and (b) the way training is designed, delivered, and implemented matters. This article aims to explain why training is important and how to use training appropriately. Using the training literature as a guide, we explain what training is, why it is important, and provide recommendations for implementing a training program in an organization. In particular, we argue that training is a systematic process, and we explain what matters before, during, and after training. Steps to take at each of these three time periods are listed and described and are summarized in a checklist for ease of use. We conclude with a discussion of implications for both leaders and policymakers and an exploration of issues that may come up when deciding to implement a training program. Furthermore, we include key questions that executives and policymakers should ask about the design, delivery, or implementation of a training program. Finally, we consider future research that is important in this area, including some still unanswered questions and room for development in this evolving field.
... Separately, VR is also proving it can support collaborative tasks. When used as a medium for telepresence [8] -the feeling of being there -VR has been shown to increase situation awareness [9] and the vividness [10], interactivity [8] and media richness [11], as well as the proprioceptive [12] or kinesthetic [13] properties of remote experiences. These qualities have been shown to enhance collaborative teleoperation scenarios like remote driving [14] and surgery [15], just to cite a few. ...
Conference Paper
Full-text available
Effective data visualization is a key part of the discovery process in the era of big data. It is the bridge between the quantitative content of the data and human intuition, and thus an essential component of the scientific path from data into knowledge and understanding. Visualization is also essential in the data mining process, directing the choice of the applicable algorithms, and in helping to identify and remove bad data from the analysis. However, a high complexity or a high dimensionality of modern data sets represents a critical obstacle. How do we visualize interesting structures and patterns that may exist in hyper-dimensional data spaces? A better understanding of how we can perceive and interact with multi dimensional information poses some deep questions in the field of cognition technology and human computer interaction. To this effect, we are exploring the use of immersive virtual reality platforms for scientific data visualization, both as software and inexpensive commodity hardware. These potentially powerful and innovative tools for multi dimensional data visualization can also provide an easy and natural path to a collaborative data visualization and exploration, where scientists can interact with their data and their colleagues in the same visual space. Immersion provides benefits beyond the traditional desktop visualization tools: it leads to a demonstrably better perception of a datascape geometry, more intuitive data understanding, and a better retention of the perceived relationships in the data.
... The interviews and observations must highlight cognitive challenges, and articulate how humans meet these challenges. Data must be analyzed with the intent of discovering something new about the role of expertise in accomplishing work tasks and managing complexity (Militello & Klein, 2001). ...
Article
Full-text available
Cognitive Task Analysis (CTA) has become part of the standard tool set of cognitive engineering. CTAs are routinely used to understand the cognitive and collaborative demands that contribute to performance problems, the basis of expertise, as well as the opportunities to improve performance through new forms of training, user interfaces, or decision aids. While the need to conduct CTAs has become well established, there is little in the way of available guidance with respect to ‘best practice’ for how to conduct a CTA or how to evaluate the quality of a CTA that has been conducted by others. This is an important gap as the range of consumers of CTAs is expanding to include program managers and regulators who may need to make decisions based on CTA findings. This panel brings together some of the leaders in development and application of CTA methods to address the question: Given the variety of methods available, and the lack of rigid guidance on how to perform a CTA, how does one judge the quality of a CTA?” The goal of the panel is to explore points of consensus with respect to ‘best practice’ in conducting and evaluating a CTA, in spite of differences in particular CTA methods, as well as to draw insights from unique and provocative perspectives.
... Moreover, whether conducted in real time or retrospective techniques such as "critical decision method" or cognitive task analysis, each require a deliberate and methodical approach for eliciting, explaining, and then representing task cognition in a comprehensive manner. 41 Finally, querying clinicians about their thinking can also be used for education rather than for research. Educational use might choose to use the more invasive technique of asking for explanations for thoughts as a means to enhance the potential learning potential value of the exercise. ...
... Cognitive Task Analysis (CTA) is one of such methodologies that has been used to elicit and understanding of macrocognitive processes involved in performing various clinical tasks (14)(15)(16)(17). It can be defined as a set of methods to elicit, explain, and represent the mental processes involved in performing a task (18). Such techniques have previously been used to improve clinicians' access to data through electronic platforms (16), develop and integrate decision support systems (CDSS) into the clinical space (16,(19)(20)(21), develop simulation training programs to aid in sepsis recognition (22), and to understand decision making among expert clinicians (15,17). ...
Article
Full-text available
Background and Objectives Children with congenital heart disease (CHD) are at risk of deterioration in the face of common childhood illnesses, and their resuscitation and acute treatment requires guidance of CHD experts. Many children with CHD, however, present to their local emergency departments (ED) with gastrointestinal and respiratory symptoms that closely mimic symptoms of CHD related heart failure. This can lead to incorrect or delayed diagnosis and treatment where CHD expertise is limited. An understanding of the differences in cognitive decision-making processes between CHD experts and ED physicians can inform how best to support ED physicians when treating CHD patients. Methods Cardiac intensivists (CHD experts) and pediatric emergency department physicians (ED physicians) in a major academic cardiac center were interviewed using the critical decision method. Interview transcripts were coded deductively based on Schubert and Klein's macrocognitive frameworks and inductively to allow for new or modified characterization of dimensions. Results In total, 6 CHD experts and 7 ED physicians were interviewed for this study. Although both CHD experts and ED physicians spent a lot of time sensemaking, their approaches to sensemaking differed. CHD experts reported readily recognizing the physiology of complex congenital heart disease and focused primarily on ruling out cardiac causes for the presenting illness. ED physicians reported a delay in attributing the signs and symptoms of the presenting illness to congenital heart disease, because these clinical findings were often non-specific, and thus explored different diagnoses. CHD experts moved quickly to treatment and more time anticipating potential problems and making specific contingency plans, while ED physicians spent more time gathering a range of data prior to arriving at a diagnosis. These findings were then applied to develop a prototype web-based decision support application for patients with CHD. Conclusion There are differences in the cognitive processes used by CHD experts and ED physicians when managing CHD patients. An understanding of differences in the cognitive processes used by CHD experts and ED physicians can inform the development of potential interventions, such as clinical decision support systems and training pathways, to support decision making pertaining to the acute treatment of pediatric CHD patients.
... In this context, CTA is viewed as a suitable and necessary method for analysing (shared) cognitive activities in complex and dynamic environments (Seamster, Redding and Kaempf, 1997). It attempts to objectively describe and explain mental components such as knowledge, and mental processes such as strategies required to carry out a task (Klein & Militello, 2001). In order to cover various cognitions and increase the validity of the CTA, Appendix. ...
...  I samband med intervjuerna användes ACTA -Applied Cognitive Task Analysis (Klein & Militello, 2001;Militello & Hutton, 1998). ACTA är en metod för semistrukturerad intervju där deltagarna utifrån en simulering eller ett scenario får beskriva vad som hänt och hur de tänkte i de aktuella situationerna. ...
Technical Report
Full-text available
FOI develops a demonstrator of Command & Control Warfare Simulator (C2W simulator) on a contract awarded by the Swedish Defence Materiel Administration. The purpose is to show and visualize concrete effects of C2W on C2 ability. The current report describes the sixth and seventh user tests or experiments with users from the Swedish Armed Forces. The experiments were conducted with version 4 of the demonstrator of C2W simulator, which includes new functionality compared to previous versions, among other things with extended possibilities to perform computer network operations, simulation of mobile-phone calls and visualization of social networks. The used experimental design was similar to previous experiments, but with the scenario modified and including incidents that invited to use the new functionality. The users, or the participants of the experiments, constituted a command staff of a rapid deployment force responsible for the stability in an operational area in which the activities of an irregular force resulted in the incidents. Both the command staff and the irregular force had access to different types of C2W by means of measures and countermeasures of electronic warfare (EW) and computer network operations (CNO). Data collection was performed with the purpose of determining whether C2 ability can be measured and how. In experiment 6, primarily the new functionality and experimental design and methods were tested. Experiment 7 included an extended data collection to make it possible to determine whether the used measures actually grasp the construct "C2 ability" and whether C2 ability in turn is affected by C2W. The data analyses indicate that C2 ability may be grasped, or measured, by the measures used and that C2W may affect C2 ability. However, in the development of the demonstrator of C2W simulator, critical aspects of the measuring of C2 ability are that correct competencies are included in the command staff, and that training of the staff in using the demonstrator allows an adequately high level of skill to be reached.
... The research focused on the cognitive analysis that was conducted on the TaSPoD system design, using applied cognitive task analysis. The result of the analysis extracted from the users' cognitive skills is valuable for many beneficiaries such as system developers, who appreciate the representation from the users' perspective (Klein & Militello, 2001). The result of the ACTA is essential to uphold cognitive tasks in an e-training system design and support effective designing tasks. ...
Article
Full-text available
E-training system has been a practical method in higher learning institutions to enable a virtual training environment. It is a vigorous effort to corroborate educators to prepare and facilitate themselves with professional knowledge, skill, values, and practices through training programs. These training programs are structured with an essence of competency development, lifelong learning, and career path. The effect of e-training design and process difficulties faced by end-Docens Series in Education ISSN 2583-1054 52 users were less prominent. Therefore, this research's objective is to validate the design of the E-Training System for Professional Development (TaSPoD), using an Applied Cognitive Task Analysis (ACTA) that was performed by a group of experts, to elicit the most difficult cognitive elements. Interview sessions were conducted to answer the research question of the electronic training design recommendation implemented in TaSPoD. The result shows that the experts faced difficulties in two system design elements which are system design and functionality and course content design. Thus, to increase user engagement in the e-training system, experts proposed three recommendations: (i) graphical user interface with simple and appropriate objects, tabs and icons, and video upload tools, (ii) communication tools such as chat programs and video conferencing, (iii) customized application for external documents.
... When job incumbents exist, one approach that could help identify the skills and knowledge required for carrying out a job is to perform a cognitive task analysis (CTA). CTA is a group of task analysis methods that specifically uncovers the cognitive processes required to accomplish a certain task (Crandall, Klein, & Hoffman, 2006;Klein, & Militello, 2001). These methods may be used to determine the cognitive processes involved in operating the CU-170 Heron UAV used by the CF in the ongoing mission in Afghanistan, which could also help determine the skills and knowledge that a UAV operator should possess. ...
Article
Full-text available
The Directorate Technical Airworthiness and Engineering Support 6 tasked Defence Research and Development Canada (DRDC) - Toronto to provide a preliminary summary of human factors issues related to the control of uninhabited aerial vehicles (UAVs) in support of the Canadian Forces Joint Unmanned Aerial Vehicle Surveillance Target Acquisition System (JUSTAS) project. This was carried out by performing a literature review, and consulting with subject matter experts. The human factors topics discussed are organizational influences, operator influences, and human-system integration issues. The key findings were: (1) human factors play a major role in UAV mishaps, (2) operator vigilance is required in automated UAV control, (3) recent increases in long-endurance UAV operations have necessitated shift work schedules to man the GCS around-the-clock that has caused UAV operators to experience fatigue leading to serious implications on health and performance, (4) a ground control station interface that supports a multimodal display (i.e., an interface that uses visual, auditory, and tactile cues) can enhance operator performance, and, (5) prior pilot experience may not be a mandatory criterion for selecting individuals for operating the Predator UAV. This report concludes by proposing short- and long-term recommendations for defining future requirements in support of the JUSTAS project.
... A list of tasks were given to the users, and the screen was recorded while the different users were trying to accomplish each task. At this time, the tester must write with a high level of detail all feelings, obstacles and relevant information that may be helpful in future improvements [18][1] [2]. This test can be performed in different stages of the process [19]. ...
Conference Paper
Full-text available
The purpose of this study is to design and develop an accessible web application, named Playlingua, to improve the language learning process and reading comprehension through gamification, by using Natural Language Processing (NLP) tools and techniques. We will analyze the advantages and positive effects of gamification in learning applications and how it influences users in terms of motivation, engagement, learning quality and entertainment. We will explore how can we use both NLP and gamification together in the same application, paying special attention to user experience (UX). Everything will be developed following the User Centered Design (UCD) process.
... Tasks may be behavioral/physical (e.g., taking notes; speaking) or cognitive (e.g., prioritizing cases) [50,51]. Cognitive tasks are groups of related mental activities directed toward a goal [52]. These activities are often unobservable, but are frequently relevant to the decision making and problem-solving activities that are central to many implementation strategies. ...
Article
Full-text available
Background Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. Method The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users ( N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. Results Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 ( SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. Conclusions CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.
... De Carvalho (2011) suggested using Cognitive Task Analysis (CTA) methods, to understand work activities and how performance variabilities may affect work. CTA is defined as a "set of methods to elicit, explain, and represent the mental processes involved in performing a task" (Klein and Militello, 2001). An example of a CTA method is ACTA (Applied cognitive task analysis), (Militello and Hutton, 1998). ...
Article
Full-text available
It is important to study accidents and their underlying causes, in order to generate recommendations for improving system safety. A range of methods have been developed in various industries, to understand how accidents have occurred, as well as identify potential human errors in systems. Theories of accident causation, and the development of safety models and methods have evolved over the last few decades. However, the majority of accident analysis methods fail to account for the increasing complexity of socio-technical systems (Hollnagel, 2004 and Lindberg et al. 2010). Much of the previous research has taken a safety I perspective, which considers successful performance as reducing the number of adverse outcomes to as low as possible (Hollnagel, 2014). According to Hollnagel (2014) however, it is important to understand how operators actually carry out work (‘work-as-done’), rather than as it should be carried out (‘work-as-imagined’), to understand how normal variabilities and flexibilities in performance contribute towards both successful and unsuccessful performance. Understanding how work is normally carried out is essential for understanding how it can go wrong. This includes understanding how success is obtained, for example how people adjust their performance in the face of changing conditions and demands, and limited resources (such as time and information). Although variability and flexibility in performance are prerequisites for success and productivity, these can also explain why things can go wrong (Hollnagel, 2014). Understanding normal work (or ‘work-as-done’) is the basis of the safety II perspective, which views safety as increasing the number of things that go right. So far however, there seems to be little application of this safety II perspective in the rail industry. Research in this thesis addresses this gap, by examining whether understanding normal performance in rail engineering contexts contributes towards identifying how incidents occur, and measures for improving safety, compared to the use of existing methods. A range of different methods were used to address the aims of this thesis. Rail incident reports were analysed to understand sources of human errors in rail contexts. Observations were also conducted of operators carrying out work, to understand the opportunities for human errors associated with rail engineering processes. To understand cognitive demands and strategies associated with normal work, a cognitive task analysis was carried out. FRAM (Functional Resonance Analysis Method) (Hollnagel, 2012) wasalso used to determine how incidents may develop, and whether everyday performance can contribute towards successful and unsuccessful performance. Participants in semi-structured interviews and workshops were asked to identify strengths and limitations of various human reliability assessment methods, and offer opinions on their practical applicability. Benefits of understanding normal work included a greater understanding of how human errors can occur (by identifying cognitive demands that contribute towards the occurrence of different error types), and how cognitive strategies can reduce human errors and contribute towards acceptable performance. It was demonstrated how variabilities and flexibilities in performance can contribute towards successful and productive performance, as well as explain why things can go wrong (supporting Hollnagel, 2014). This is especially important to consider, since human errors were not easily identified from rail incident reports and observations of operators carrying out work. System safety can therefore be improved by increasing things that can go right, rather than just decreasing the things that can go wrong (Hollnagel, 2014). Participants in a workshop, however, identified that FRAM may be time consuming to apply, especially for more complex systems. Further research is recommended for the development of a toolkit, from which both practitioners and researchers can choose from a range of different methods. To further understand factors affecting acceptable performance, it is recommended that further data are collected to determine whether varying levels of cognitive demands affect performance, and whether these influence the implementation of cognitive strategies.
... In [33], (p. 168) they defined as cognitive task analysis (CTA) as "a set of methods to elicit, explain, and represent the mental processes involved in performing a task". ...
Conference Paper
Full-text available
Cyber security visualization designers can benefit from human factors engineering concepts and principles to resolve key human factors challenges in visual interface design. We survey human factors concepts and principles that have been applied in the past decade of human factors research. We highlight these concepts and relate them to cybersecurity visualization design. We provide guidelines to help cybersecurity visualization designers address some human factors challenges in the context of interface design. We use ecological interface design approach to present human factors-based principles of interface design for visualization. Cyber security visualization designers will benefit from human factors engineering concepts and principles to resolve key human factors challenges in visual interface design.
... ). Dies schließt auch implizites Wissen mit ein, wie zum Beispiel komplexere Fertigkeiten und Problemlösestrategien(Clark et al. 2008;Klein/Militello 2001). Dazu stehen eine Reihe qualitativer Erhebungsverfahren zur Verfügung. ...
Article
Full-text available
Angesichts einer hohen Heterogenität in Klassen berufsschulischer Aus- und Fortbildung stellt die Forderung nach Subjekt- oder Potenzialorientierung (Jenewein 2017; Zinn et al. 2018) eine Herausforderung für Lehrkräfte dar. Unabhängig vom Bildungsgang bringen die Schülerinnen und Schüler unterschiedliche kognitive Voraussetzungen, sozio-kulturelle Annahmen sowie Alltags- und Berufserfahrungen in den Unterricht ein. Kernstück einer Planung eines solchen Unterrichts ist die kognitive Strukturierung des Lerngegenstandes (Lipowsky 2015), zugleich muss dieser in Beziehung zu anderen Erfahrungen gesetzt werden (Einsiedler/Hardy 2010). Lernzielorientierte Ansätze für einen „one size - fits all“-Unterricht reichen daher nicht aus: Die verschiedenen Erfahrungswelten der Schülerinnen und Schüler bedingen individuelle kognitive Strukturierungen. Wir schlagen für die gewerblich-technische Unterrichtsplanung ein Analysetool vor, welches lernpsychologische und technikdidaktische Erkenntnisse aufgreift, und zeigen beispielhaft auf, dass dieses geeignet ist, die verschiedenen kognitiven Ressourcen der jugendlichen Schülerinnen und Schüler hinsichtlich derer lebensweltlichen Erfahrungen zu berücksichtigen.
... CTA seeks to understand and explain the micro-or macrocognitive processes involved in executing tasks by analyzing data collected about what people actually do in naturalistic settings (G. Klein & Militello, 2001). Applying CTA to interview data from the Caring Hearts study of older adults with heart failure, human factors researchers were able to characterize the macrocognitive functions performed by patients to manage their medications (Mickelson et al., 2016). ...
Chapter
Self-care is a process with which all people will engage at some point in their lives, whether healthy, healing from an acute health situation, or living with chronic illness. As such, there are different types of self-care with unique components and challenges. Each challenge presents an opportunity to design interventions to improve the process and outcomes of self-care. This chapter describes the key actors, challenges, and solutions presented in human factors literature addressing self-care.
... When clinicians were asked to describe their goals at the time of assuming care from the ED, the initial goals elicited prior to review of the simulation were almost always incomplete; review of the simulation generated additional clinical goals and decisions. This is consistent with previous literature on cognitive task analysis that suggests that multiple approaches with various levels of event abstraction may be necessary to achieve full understanding of the task (Klein & Militello, 2001). ...
Article
Simulation is an educational approach well suited to development of knowledge and decision-making skills for emergent or infrequent scenarios. Electronic Health Record (EHR) based simulation, in which participants retrieve information from a simulated EHR, provides an authentic training environment with fidelity to the typical clinical decision-making process and has been associated with enduring changes in EHR use patterns. However, we do not know whether these behavior changes reflect better decision-making. We aimed to develop a measure of pediatric resident performance in an EHR based simulation using the Situation Awareness Global Assessment Technique (SAGAT).
... These improved designs would reduce the likelihood of error and allow workers to better leverage the strengths of increasingly powerful technologies. In response to these challenges, researchers developed interview techniques (e.g., Klein, Calderwood, and Macgregor, 1989; Gordon & Gill, 1992; Hall, Gott, and Pokorney, 1994) and adapted ethnographic observation strategies with the goal of explaining the mental processes involved in performing a task (Klein & Militello, 2001). The last 20+ years has seen refinement and extension of these CTA methods (Schraagen, Chipman, and Shalin, 2001) in pursuit of tools to aid in the design of safer and more effective technologies to meet the complex challenges of the modern workplace. ...
Article
Full-text available
Many reports of recent research on topics in cognitive systems engineering describe their methods by distinguishing cognitive task analysis from "traditional" or "behavioral" task analysis. A hallmark of modern cognitive task analysis (CTA) methods is that they place primary focus on understanding the cognitive demands of a task and the knowledge and strategies that underlie performance. While cognitive task analysis may have seemed to be a revolutionary approach introduced in recent years, a review of the history of task analysis reveals many things that have been lost in modern treatments of the history of human factors. This presentation will review some highlights of this forgotten history, including the ideas and methods of the Psychotechnicians, the earliest industrial psychologists, the Taylorists, and others who contributed to modern CTA. Task analysis never lacked cognitive categories. Even microscale "time and motion" studies involved the analysis of the work of domain experts. Basic ideas of human-machine systems and of complexity also appear in some of the earliest literatures - industrial psychology of the first decades of the 20th century.
... Complex decision making tasks require access to sufficient information to make an informed decision, as well as updates to that information when it changes such that decision-makers can update their mental model of the environment and modify their plans accordingly (Woods & Hollnagel, 2006). Techniques such as cognitive task analysis (Klein & Militello, 2001) and cognitive work analysis (Jenkins, Stanton, Salmon, & Walker, 2008) provide guidelines for identifying task demands, including the information required to carry out complex cognitive work. ...
Article
Full-text available
NASA’s Management by Trajectory (MBT) concept will improve trajectory predictability by providing methods to keep aircraft on negotiated trajectories; however, the concept impacts roles and responsibilities. Most notably, MBT envisions a different distribution of responsibilities between the Radar-side and Data-side (herein called the negotiating) controllers. Cognitive engineering provides a rich history of research into understanding the roles that humans and automation play in complex systems and some principles for system design. This paper briefly summarizes this literature, focusing on principles underlying the proposed allocation of responsibilities in MBT and the source of automation and information requirements.
... We chose Cognitive Task Analysis (CTA) as a promising method to grasp this phenomenon. Klein and Militello [48] (p. 164) state that CTA offers the opportunity to understand the way people make key judgments and decisions, interpret situations, make perceptual discriminations, solve problems, generate plans, and use their cognitive skills to carry out challenging tasks. ...
Article
Full-text available
Complexity and uncertainty are framing the modern world, whilst also affecting issues on security and sustainability. There is a need to prepare for known threats and identified risks, but also to improve the ability to cope in situations that are difficult to recognize or describe beforehand. What is at stake—both at the organizational and individual level—is the ability to make sense of uncertain and ambiguous situations. Analyzing two empirical cases, this study aims to shed light on the abilities of experts, who have acted in very challenging situations, in which deviating from established procedures and abandoning politeness have been necessary to respond effectively. The first case deals with a threat of serious violence faced by a police officer. The second case focuses on the actions of an executive fire officer during a rescue operation after an explosion at a shopping mall. This paper concludes by arguing that pre-established procedures require experts to reflect on their usability in exceptional situations as relying on them could also have detrimental effects.
... The questions in the semi-structured interview were not asked in any particular order, but served as a guide to remind the researcher leading the interview of the areas we wanted to cover in the interview. Interview protocols loosely followed the procedures outlined in a cognitive task analysis, or critical decision method [12], [13]. Cognitive task analysis methods seek to describe cognitive demands of the task and situation, define task constraints, and provide a framework for the systematic interpretation of the qualitative results. ...
Article
Full-text available
This report documents an exploratory FY 00 LDRD project that sought to demonstrate the first steps toward a realistic computational representation of the variability encountered in individual human behavior. Realism, as conceptualized in this project, required that the human representation address the underlying psychological, cultural, physiological, and environmental stressors. The present report outlines the researchers' approach to representing cognitive, cultural, and physiological variability of an individual in an ambiguous situation while faced with a high-consequence decision that would greatly impact subsequent events. The present project was framed around a sensor-shooter scenario as a soldier interacts with an unexpected target (two young Iraqi girls). A software model of the ''Sensor Shooter'' scenario from Desert Storm was developed in which the framework consisted of a computational instantiation of Recognition Primed Decision Making in the context of a Naturalistic Decision Making model [1]. Recognition Primed Decision Making was augmented with an underlying foundation based on our current understanding of human neurophysiology and its relationship to human cognitive processes. While the Gulf War scenario that constitutes the framework for the Sensor Shooter prototype is highly specific, the human decision architecture and the subsequent simulation are applicable to other problems similar in concept, intensity, and degree of uncertainty. The goal was to provide initial steps toward a computational representation of human variability in cultural, cognitive, and physiological state in order to attain a better understanding of the full depth of human decision-making processes in the context of ambiguity, novelty, and heightened arousal.
Article
The focus of this article is the background and use of the critical decision method in nursing research. The purpose is to present a synthesis of the body of work whose authors used the critical decision method to explore nurses' cognitive work. For this integrative review of literature, I used the 5-step process: problem formulation, literature search, data evaluation, data analysis, and presentation. Synthesis of the 7 studies uncovered evidence of nurses' cognitive processes and demands of their practice environments and affirmed the method as a valuable tool for eliciting experienced nurses' practice knowledge making nursing expertise explicit.
Article
Using a case study analysis of the Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Treatment of Asthma, this article compares the workflows and knowledge requirements of primary care practice to the structure and content of a well-respected set of clinical guidelines. The authors show that there are discrepancies between physician workflow and the structure of the EPR-3, as well as between physicians' knowledge requirements and the content of the EPR-3. The analysis suggests that closing the gap between medical knowledge and practice will require alternative ways to represent guidelines' knowledge and recommendations.
Article
Future technologies will enable carriers to collect additional flight data for Flight Operational Quality Assurance. This paper describes how analysis of these data using model-based activity tracking can automatically assess the causes of detected deviations to support safety-enhancement efforts. The paper describes the activity tracking methodology implemented in the Crew Activity Tracking System (CATS) using an example drawn from previous research in which CATS analyzed full-mission simulation data online. The paper also discusses current research on using CATS to analyze flight data from a Boeing 757 aircraft.
Article
Studies of performance in medicine are often based on observation. Videotape provides a valuable tool for recording events from both real environments and simulators. When analysing observational data it is important that robust tools are used, particularly when investigating non-technical (cognitive and social) skills. This paper describes the method used to identify the key non-technical skills required in anaesthesia and to de- velop a behavioural marker system for their measure- ment. A prototype taxonomy was designed on the basis of a literature review; an examination of existing marker systems; cognitive task analysis interviews; an iterative development process involving workshops; and cross- checking in theatre. The resulting anaesthetists' non- technical skills (ANTS) system comprises four skill cat- egories (task management, team working, situation awareness, and decision making) that divide into 15 elements, each with example behaviours. Preliminary evaluation using ratings of videotaped scenarios indi- cated that the skills were observable and could be rated with reasonable agreement.
Article
Full-text available
A decision science working group (DSWG) was chartered by AFRL in 2002 to assess the state-of-the-art in the application of decision support science and technology (S&T); and to recommend ways to infuse the latest technologies and methodologies into the Air Force's Information Systems S&T portfolio. This report presents a brief introduction to the field of decision science as it relates to defense command and control (C2) systems and the motivations which led to the establishment of the working group, the activities sponsored and undertaken by the DSWG are described, and the findings and recommendations of the working group are summarized.
Article
Full-text available
Learning designs aimed at supporting transformational change could significantly benefit from the adoption of socio‐historical and socio‐cultural analysis approaches. Such systemic perspectives are gaining more importance in education as they facilitate understanding of complex interactions between learning environments and human activity. The current study utilizes a framework based on activity theory to examine the system elements underlying patient self‐management practices. Specifically, Jonassen and Rohrer‐Murphy’s task analysis methodology was adapted and literature reviews, expert interviews, and patient survey data were collected to ground the descriptions of system components. Results from this research will be discussed in terms of their contribution to the identification of design requirements for the creation of patient self‐management and educational environments.
Chapter
IntroductionScience of Training: Theoretical DevelopmentsInstructional Systems Development ModelTraining AnalysisTraining DesignTraining DevelopmentTraining ImplementationTraining EvaluationTransfer of TrainingConclusions References
Article
Military psychologists have made significant contributions to training methods and media design. The introduction of a systems approach to training has brought order to the system. Development of individual training methods for both the schoolhouse and distance education and training have allowed for continuing skill development over the course of a career. Engagement simulation technology and the After Action Review (AAR) process brought realism and an effective way to discuss lessons learned. Simulator systems such as Simulator Network (SIMNET), Close Combat Tactical Trainer (CCTT), and Distributed Mission Operations (DMO) have lowered the cost of training, while still exposing trainees to the full complexities of combat without the associated danger. Today's emphasis on cognitive tasks and decision making takes training to the next level from knowing "how to" to knowing "when and where to." Future work for military psychologists will include automating training processes and developing training methods to support the teaching of cultural adaptation.
Article
Full-text available
This research aims to develop design guidelines for systems that support investigators and analysts in the exploration and assembly of evidence and inferences. We focus here on the problem of identifying candidate 'influencers' within a community of practice. To better understand this problem and its related cognitive and interaction needs, we conducted a user study using a system called INVISQUE (INteractive Visual Search and QUery Environment) loaded with content from the ACM Digital Library. INVISQUE supports search and manipulation of results over a freeform infinite 'canvas'. The study focuses on the representations user create and their reasoning process. It also draws on some pre-established theories and frameworks related to sense-making and cognitive work in general, which we apply as a 'theoretical lenses' to consider findings and articulate solutions. Analysing the user-study data in the light of these provides some understanding of how the high-level problem of identifying key players within a domain can translate into lower-level questions and interactions. This, in turn, has informed our understanding of representation and functionality needs at a level of description which abstracts away from the specifics of the problem at hand to the class of problems of interest. We consider the study outcomes from the perspective of implications for design.
Article
This volume is the first comprehensive history of task analysis, charting its origins from the earliest applied psychology through to modern forms of task analysis that focus on the study of cognitive work. Through this detailed historical analysis, it is made apparent how task analysis has always been cognitive.
Article
Understanding how experts are able to manage the complexity of modern work environments can inform the strategies and tools used to develop and support effective performance. To that end, this chapter provides a comprehensive review of naturalistic decision making (NDM) research for industrial and organizational (I/O) psychologists. The community of NDM researchers focuses on understanding how experts use their experience to make good decisions in complex real-world situations characterized by stress and uncertainty. NDM and the study of expertise in real-world settings is a rich scientific and practically relevant research area that has yet to be integrated with the I/O field. This chapter is intended to facilitate this cross-fertilization of perspectives. In this chapter, six specific goals are addressed: (a) definitions of the core features of NDM as a field of inquiry as well as the fundamental issues in the domain are reviewed; (b) an introduction to the fundamental theoretical perspectives in NDM is provided; (c) perspectives on the methodological approaches employed in the process of understanding complex cognition in the real world are summarized; (d) NDM research findings are synthesized and presented as a set of performance mechanisms of expert individual and team decision making; (e) applications of NDM research to improving decision-making performance in organizations are reviewed with specific attention to training the expert decision maker; and (f) future directions for NDM as a field of inquiry are outlined.
Article
The design and evaluation of healthcare work systems requires an understanding of the cognitive work involved in healthcare tasks. Previous studies suggest that a formative approach would be particularly useful to examine healthcare activities for this purpose. In the present study, methods from cognitive work analysis and cognitive task analysis are combined in a formative examination of managing acute kidney injury, an activity that occurs across primary and secondary healthcare settings. The analyses are informed by interviews with healthcare practitioners and a review of practice guidelines. The findings highlight ways in which the task setting influenced practitioners’ activity, and ways in which practitioners approached the activity (for example, how they used data to make decisions). The approach taken provided a rich understanding of the cognitive work involved, as well as generating suggestions for the design of work systems to support the clinical task. Practitioner summary: Healthcare tasks often require decision-making in complex and dynamic circumstances, potentially involving collaboration across different practitioner roles and locations. We demonstrate the use of a formative analysis to understand the cognitive work in managing a clinical syndrome across primary and secondary care settings, and consider the implications for work design.
Chapter
Full-text available
This paper presents a review of fundamental concepts behind the shared mental model and its processes. Shared mental model has two properties which are similarity and accuracy where both of these properties have different emphasis. Types of shared mental model refers to four (4) different ways on what type of cognitive process is being shared in team, which includes task-specific knowledge, task-related knowledge, knowledge of teammates and attitude or beliefs. These four (4) types of shared mental model are grouped into two (2) major content domains, i.e. task-work and team-work. It further describe the aspects of shared mental model on how cognitive process is being shared, includes shared vs. overlapping, similar vs. identical, compatible vs. complimentary and distributed. Shared mental model has to be evaluated using three (3) aspects of characteristics in order to show its operationalization: elicitation method, structure representation, and representation of emergence. A specific evaluation technique called cognitive task analysis that focuses on the analysis of difficulties in cognitive structures is introduced in evaluating shared mental model. This paper also discusses Collaborative Visualisation as technological approach in shared mental model, which includes the benefit of user participation in the aspects of joining or leaving, floor control, privacy and global view. Specific areas on big data, visual analytics, multimedia interface, mobility, disability, awareness and learning analytics that can benefit from the shared mental model approach is discussed.
Chapter
As ubiquitous sensing becomes embedded in our everyday world, we can easily obtain multimodal sensor data from our physical surroundings. Objectivity is a core value in the visualization research community. However, personal experiences are characterized not only by objective facts, but by personal emotions. In this paper, we explore immersive data visualization for conveying personal subjective perception and experience by using multidimensional data and multimedia. We introduce a framework to describe narrative structures in immersive data visualization, and provide an example project Baguamarsh as a proof of theme-based creative methods for designing correlations between different information. We hope this study may offer outline opportunities for future research in narrative visualization.
Article
Full-text available
In high-risk industries such as aviation, the skills not related directly to technical expertise, but crucial for maintaining safety (e.g. teamwork), have been categorized as non-technical skills. Recently, research in anaesthesia has identified and developed a taxonomy of the non-technical skills requisite for safety in the operating theatre. Although many of the principles related to performance and safety within anaesthesia are relevant to the intensive care unit (ICU), relatively little research has been done to identify the non-technical skills required for safe practice within the ICU. This review focused upon critical incident studies in the ICU, in order to examine whether the contributory factors identified as underlying the critical incidents, were associated with the skill categories (e.g. task management, teamwork, situation awareness and decision making) outlined in the Anaesthetists' Non-technical Skills (ANTS) taxonomy. We found that a large proportion of the contributory factors underlying critical incidents could be attributed to a non-technical skill category outlined in the ANTS taxonomy. This is informative both for future critical incident reporting, and also as an indication that the ANTS taxonomy may provide a good starting point for the development of a non-technical skills taxonomy for intensive care. However, the ICU presents a range of unique challenges to practitioners working within it. It is therefore necessary to conduct further non-technical skills research, using human factors techniques such as root-cause analyses, observation of behaviour, attitudinal surveys, studies of cognition, and structured interviews to develop a better understanding of the non-technical skills important for safety within the ICU. Examples of such research highlight the utility of these techniques.
Chapter
Full-text available
shows how one can go beyond spartan laboratory paradigms and study complex problem-solving behaviors without abandoning all methodological rigor / describes how to carry out process tracing or protocol analysis methods as a "field experiment" (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
The objective of this study was to examine the way decisions are made by highly proficient personnel, under conditions of extreme time pressure, and where the consequences of the decisions could affect lives and property. Fire Ground Commanders (FGCs), who are responsible for allocating personnel and resources at the scene of a fire, were studied using a critical incident protocol analysis. The major finding was that in less than 12% of the decision points was there any evidence of simultaneous comparisons and relative evaluation of two or more options. Instead the FGCs most commonly relied on their experience to directly identify the situation as typical and to identify a course of action as appropriate for that prototype. A Recognition Primed Decision (RPD) model is proposed which emphasizes the use of recognition rather than calculation or analysis for rapid decision making.
Article
Full-text available
Direct comparisons were made of expert highway engineers' use of analytical, quasirational, and intuitive cognition on three different tasks, each displayed in three different ways. Use of a systems approach made it possible to develop indices for measuring the location of each of the nine information display conditions on a continuum ranging from intuition-inducing to analysis-inducing and for measuring the location of each expert engineer's cognition on a continuum ranging from intuition to analysis. Individual analyses of each expert's performance over the nine conditions showed that the location of the task on the task index induced the expert's cognition to be located at the corresponding region on the cognitive continuum index. Intuitive and quasirational cognition frequently outperformed analytical cognition in terms of the empirical accuracy of judgments. Judgmental accuracy was related to the degree of correspondence between the type of task and the type of cognitive activity on the cognitive continuum.
Article
Full-text available
Retention of skilled search after a nine-year period of nonuse was investigated in letter and digit tasks. The letter task involved search for one of six consistently mapped letter targets in a set of two, three, or four letters. The digit task required subjects to indicate the largest digit in a display of two, three, or four digits (one through nine). In the digit task the digits 4, 5, 6, 7, and 8 could serve as targets or distractors, depending on the other items in the display. Results indicated only minimal loss of speed and no loss in visual search rates after the nine-year interval. In addition, the savings were equivalent for the two tasks in spite of the difference in stimulus-response consistency.
Article
Full-text available
Artificial Intelligence methods can rapidly generate plans for complex situations. However, these plans may be rejected by air campaign planning experts, who judge dimensions such as robustness using operational rather than computational criteria. Our goal is to capture the tactical and strategic concerns of air campaign planners, and incorporate these into planning technology to filter out the unacceptable options and highlight preferred plans. We are in the process of employing a complete Cognitive Task Analysis (CTA), which uses a variety of knowledge elicitation techniques to fully identify the process of plan evaluation and factors underlying judgments of plan robustness. The CTA will form the foundation of an ACP Plan Critic. The Plan Critic will evaluate plans for plan quality and will highlight potential problem areas and vulnerable assumptions. This paper summarizes the current status of this work. There is often a large gap between technological advances and operational feasibility. In the Artificial Intelligence (AI) planning domain, planning technology often approaches the problem via a process that does not match the way the targeted user thinks about and solves planning problems. In some cases, planning technologists and air campaign planners do not define certain concepts identically; what is robust to the planning technologist does not necessarily match robustness to the military planner. Subsequently, there is a considerable risk that current planning technologies will not be accepted in the field because they do not meet the decision-making needs of the commanders and planning staff. The following pages discuss the need for user-centered planning systems in the air campaign planning domain and describe a Decision-Centered Design approach to building systems, which will enable the generation of AI products that meet the operational needs of air campaign planners.
Article
Full-text available
Further simulations of human multiple-task performance have been conducted with computational models that are based on the executive-process interactive control (EPIC) architecture introduced by D. E. Meyer and D. E. Kieras (see record 84-14604). These models account well for patterns of reaction times and psychological refractory-period phenomena (delays of overt responses after short stimulus onset asynchronies) observed in a variety of laboratory paradigms and realistic situations. This supports the claim of the present theoretical framework that multiple-task performance relies on adaptive executive control, which enables substantial amounts of temporal overlap among stimulus identification, response selection, and movement-production processes for concurrent tasks. Such overlap is achieved through optimized task scheduling by flexible executive processes that satisfy prevailing instructions about task priorities and allocate limited-capacity perceptual-motor resources efficiently. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
The impact of technology on many tasks and functions has resulted in greatly increased demands on the cognitive skills of workers. More procedural or predictable tasks are now handled by smart machines, while humans have become responsible for difficult cognitive tasks. The increase in cognitive demands placed on workers has created a need for training that targets cognitive skills. In most cases, however, the task analyses that drive training development are conducted using methodologies that focus primarily on behaviors. The training community needs tools that will allow access to experienced based cognitive skills. The primary goal of this project was to develop streamlined methods of Cognitive Task Analysis that would fill this need. We have made important progression this direction. We have developed streamlined methods of Cognitive Task Analysis. Our evaluation study indicates that the methods are usable and aid in the development of important, accurate training materials addressing cognitive issues. In addition, we have developed a CD-based stand alone instructional package, which will make the Applied Cognitive Task Analysis (ACTA) tools widely accessible. A survey of the software conducted with both Navy instructional Systems Specialists (ISSs) and private sector Instructional Designers indicates that the software is successful in communicating the ACTA techniques.
Article
Full-text available
High-technology production and information systems are becoming increasingly common in the workplace. Those who work with them need formal and knowledge-intensive skills that match the way these systems work. The traditional methods of shopfloor education, learning-by-doing from other skilled operators, or from the equipment supplier, are risky, expensive and inefficient. Classroom training does not work well either. We describe two-day constructive learning workshops designed to help introduce MRP, the widely used computer-based production and inventory management system, into a major transportation maintenance facility. MRP is notoriously difficult to install successfully. These workshops worked well and reduced the operatives' learning period by upwards of a year.
Book
Full-text available
This is an important thorough book. Guy Boy has presented a masterful review and synthesis of the many factors that affect how people and technology interact in the performance of a task, an understanding that is essential for those who design technology. I strongly recommend it for both students and professionals. -Donald A. Norman, Hewlett-Packard; author of The Invisible Computer If it is, as I have claimed that AI systems of the future will be less about artificial' intelligence and more about augmented' intelligence, Dr. Boy has produced a veritable handbook on the design of these cognitive prostheses. So sit down, relax, put on your ocular prosthesis and enjoy the read. -Ken Ford, Associate Director, NASA Ames Research Center This book is a significant first step towards making human-centered design a reality. It provides orientation and guidance for everyone who is concerned with developing systems that integrate people and computers in a context that provides functionality, reliability, flexibility, and responsibility. -Terry Winograd, Professor, Stanford University.
Article
Full-text available
for several years, we have been trying to understand what constitutes radiological expertise and how that expertise is acquired / our goal is to understand the learning of a complex skill and thereby to stretch the limits of existing knowledge about expertise and its acquisition / this chapter is a progress report on that work expert problem-solving (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
the chapter samples a range of recent human factors work primarily in the context of nuclear power plants and the U.S. nuclear industry history of human factors in the U.S. nuclear industry / improved control rooms / displays / controls cognitive factors / decision making / evaluation of decision aids (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Reviews evidence which suggests that there may be little or no direct introspective access to higher order cognitive processes. Ss are sometimes (a) unaware of the existence of a stimulus that importantly influenced a response, (b) unaware of the existence of the response, and (c) unaware that the stimulus has affected the response. It is proposed that when people attempt to report on their cognitive processes, that is, on the processes mediating the effects of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit causal theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response. This suggests that though people may not be able to observe directly their cognitive processes, they will sometimes be able to report accurately about them. Accurate reports will occur when influential stimuli are salient and are plausible causes of the responses they produce, and will not occur when stimuli are not salient or are not plausible causes. (86 ref)
Article
Full-text available
This paper presents the findings of a study on how experienced naval officers make decisions in a complex, time-pressured command and control setting, the Combat Information Center of AEGIS cruisers. The decision processes invoked by the officers were consistent with the recognition-primed decision model. The majority of decisions concerned situation awareness and diagnosis in which the decision makers used feature-matching and story generation strategies to build situation awareness. Furthermore, awareness of the situation enabled the officers to recognize appropriate actions from published procedures or past experience. A recognitional strategy was used to identify 95% of the actions taken; decision makers compared multiple options in only 4% of the cases. These findings are discussed in terms of their implications for framing command-and-control problems, for emphasizing situation awareness, for a descriptive model of decision making, and for designing decision supports.
Chapter
Advances during the last several decades in cognitive science provide a new methodology for task analysis called Cognitive Task analysis (CTA). Cognitive science research in the areas of memory structure, mental models in problem solving, attention allocation, skill acquisition, and the nature of expertise provide the theoretical framework for CTA, while research methods from experimental psychology and knowledge engineering provide many of the data collection and analysis techniques. CTA has been applied in aviation settings for part-task training, air traffic controller curriculum redesign, and the development of computer-based trainers for combat aircrews.
Article
This paper presents the findings of a study on how experienced naval officers make decisions in a complex, time-pressured command and control setting, the Combat Information Center of AEGIS cruisers. The decision processes invoked by the officers were consistent with the recognition-primed decision model. The majority of decisions concerned situation awareness and diagnosis in which the decision makers used feature-matching and story generation strategies to build situation awareness. Furthermore, awareness of the situation enabled the officers to recognize appropriate actions from published procedures or past experience. A recognitional strategy was used to identify 95% of the actions taken; decision makers compared multiple options in only 4% of the cases. These findings are discussed in terms of their implications for framing command-and-control problems, for emphasizing situation awareness, for a descriptive model of decision making, and for designing decision supports.
Chapter
This chapter presents a practical GOMS model methodology for user interface design. The basic approach to user-interface design using the cognitive complexity approach would be that the iterative design process would be followed, but with the evaluation of a proposed design being done with simulation techniques rather than actual human user testing; only a final test of the design would require actual user testing. Additional user testing would be involved to develop aspects of the design, such as screen layout, that are not directly addressed by an analysis of the procedures entailed by the design. There are several problems in using the cognitive complexity approach as a design tool that have become clear from technology transfer. The chapter presents two critical problems: (1) the difficulty of constructing production rule simulation models, and (2) the difficulty of doing, in a standardized and reliable way, the detailed task analysis required to construct the representation of the procedural knowledge that the user should have to operate the system.
Article
A new theoretical framework, executive-process interactive control (EPIC), is introduced for characterizing human performance of concurrent perceptual-motor and cognitive tasks. On the basis of EPIC, computational models may be formulated to simulate multiple-task performance under a variety of circumstances. These models account well for reaction-time data from representative situations such as the psychological refractory-period procedure. EPIC's goodness of fit supports several key conclusions: (a) At a cognitive level, people can apply distinct sets of production rules simultaneously for executing the procedures of multiple tasks; (b) people's capacity to process information at "peripheral" perceptual-motor levels is limited; (c) to cope with such limits and to satisfy task priorities, flexible scheduling strategies are used; and (d) these strategies are mediated by executive cognitive processes that coordinate concurrent tasks adaptively.
Article
Two studies supporting the use of the Critical Decision method (CDM) in eliciting knowledge from expert neonatal intensive care unit (NICU) nurses are presented. The first examines the utility of CDM in the nursing profession. In this study, significantly more information was elicited in CDM interviews than in non-CDM interviews. In the second study, cues, indicators, and exemplars were extracted from CDM incident accounts to form a guide to early sepsis assessment in the NICU that contains information not available in the current literature. All evaluators rated the guide as useful. Implications for future research, including generalizability to other areas of nursing, are discussed.
Book
Part 1: the task analysis process. Part 2: task analysis techniques task data collection task description methods task simulation methods task behaviour assessment methods task requirements evaluation methods. Part 3: task analysis case studies balancing automation and human action through task analysis a preliminary communications system assessment a plant local panel review a staffing assessment for a local control room task simulation to predict operator workload in a command systenm task analysis of operator safety actions maintenance training a method for quantifying ultrasonic inspection effectiveness operational safety review of a solid waste storage plant a task analysis programme for a large nuclear chemical plant.
Article
Presents a theory of norms and normality and applies the theory to phenomena of emotional responses, social judgment, and conversations about causes. Norms are assumed to be constructed ad hoc by recruiting specific representations. Category norms are derived by recruiting exemplars. Specific objects or events generate their own norms by retrieval of similar experiences stored in memory or by construction of counterfactual alternatives. The normality of a stimulus is evaluated by comparing it with the norms that it evokes after the fact, rather than to precomputed expectations. Norm theory is applied in analyses of the enhanced emotional response to events that have abnormal causes, of the generation of predictions and inferences from observations of behavior, and of the role of norms in causal questions and answers. (3 p ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
This paper describes the empirical validation of a knowledge acquisition methodology using conceptual graph analysis. Conceptual graphs are a representational medium used to integrate and organize knowledge obtained from documents, verbal protocols, question probes, and observation of task performance. The method was validated by presenting two groups of subjects with text and graphics on a topic in engineering dynamics. One set of materials was written by a recognized subject matter expert. The other set was developed by subjecting the expert generated text to conceptual graph analysis. The text was first translated into a conceptual graph structure, then the graph was revised via question probes and observation/induction methods. The final graph was returned to a standard text format. Students receiving the knowledge-engineered text solved significantly more problems than students receiving the original text. Conceptual graph analysis is a generalized method that can be used for a broad range of training domains, providing a highly structured means for making explicit the knowledge base to be incorporated into instructional design.
This study presents a critical analysis of the state of current technologies, methods, and tools used in cognitive task-analysis. Methods for cognitive task-analysis, derived from methods used in cognitive science, are relatively new and have not been systematized. Current methodologies demand considerable time and expertise to conduct properly and often yield data which is difficult to readily translate into practical application. This paper examines these problems and proposes some directions for future research and training program development.
Book
The purpose of this book is to convey what is known today about building expert systems and to present concepts, methods and issues necessary for an understanding of the field. Among the topics covered are: an overview of the expert systems field that explains its nature, history, and significance; an analysis of how expert systems differ from conventional software programs, laboratory Artificial Intelligence programs and humans in their problem solving activity; an explanation of basic Artificial Intelligence and knowledge engineering concepts that affect design and implementation; an analysis of the architectural choices faced in building expert systems, including specific design prescriptions for tasks of different kinds; an examination of the evolutionary process of knowledge acquisition necessary to build a system with expertise; and a comparative analysis of the strengths and weaknesses of existing knowledge engineering tools based on experiences with eight different techniques applied to a single problem.
Article
The development of support of decision making processes in complex systems requires a systematic approach based upon knowledge of the human's role, capabilities, and the tasks to be performed. Results from a survey and a workshop show that there is a need for methodologies that systematically address the cognitive factors of complex task situations. COADE provides the developer and cognitive specialist with an approach to the development of cognitively-centered systems. The COADE framework comprises a set of activities for cognitive analysis, design and evaluation. Analysis activities result in the specification of cognitive requirements; design activities translate those into design requirements; evaluation activities control the quality of the intermediate and final products of the development process.
Article
How does a decision support system (DSS) designer identify the kernel(s) or key decision elements of a problem, from the user's perspective, and develop a design for a DSS from them? The problem arises because DSS adaptive design says start small and grow, but there are no suggestions on where to start. This thesis addresses the primary challenge by developing the tool, concept mapping, as a means to identify kernels in a decision process. This is a descriptive process that defines the problem space, describes the decision process, and identifies the key steps of the decision process as potential kernels. Beyond this initial challenge is the challenge of integrating concept mapping into the adaptive design process. This thesis effort will take the technique of concept mapping and use it to extract the information about a problem from the decision maker himself. The result of this extraction should be a well defined problem space, a sufficiently described decision process and a means to identify the kernels. Finally, the effort turns to integrating concept mapping into the adaptive design process. The integration of concept mapping into the adaptive design process is accomplished through a process called the problem definition process (PDP).
Article
The performance of instructor pilots and student pilots was compared in two visual scanning tasks. In the first task both groups were shown slides of T-37 instrument displays in which errors were to be detected. Instructor pilots detected errors faster and with greater accuracy than student pilots, thus providing evidence for the validity of the procedures employed. However, student pilots showed a greater tendency to employ a systematic search pattern than instructor pilots, which suggests the use of a flexible scanning strategy rather than a rigid scanning pattern. In the second experiment, the attention diagnostic method task was employed to determine if the experience in visual scanning obtained in the flight situation would transfer to a novel scanning task. In the first session there were no differences in response latency between instructor pilots, student pilots, and a group of university students. Instructor pilots, however, showed a significant linear decrease in latency over the course of eight sessions, indicating their experience in learning to attend to critical features. The experiment results recommend the use of a variety of scanning tasks in the pilot training program to facilitate the more rapid development of adaptive scanning strategies. (Author/EA)
Article
The volume outlines the overall background, research approach, and paradigm used by TADMUS, with specific focus on how to train decision making at the individual and team levels—especially how to provide training that will prepare individuals to operate in complex team environments. The chapters explore complex, realistic tasks with experienced Navy participants. Throughout the book, the contributors explore the research implications and the lessons learned that may guide those interested in applying the results of research in operational environments. Although TADMUS focused on a military decision-making environment, its program of research has applicability across a variety of task environments that pose similar demands on human operators. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
the major problem confronting the scientific investigation of extraordinary achievements and their creative nature is their uniqueness / by focusing instead on the highly replicable skills of exceptional performers (e.g., professional musicians) one can identify high (expert) levels of performance . . . that correspond to phenomena that are more tractable to analysis with scientific methods / the principal question addressed here is how expert performers attain a successful adaptation to the demands of the critical activities in the corresponding domain in some domains of expertise such as individual sport events (e.g., running the 100-meter dash), the performance is measured by absolute units of time / in other domains, performance is evaluated in relative terms through comparison with other contemporary performers (e.g., gymnastics) / propose methods for measuring and describing even these types of expert performance by absolute standards that are independent of the social and historical context of the studied expert performance / discuss the following characteristics of expert performance: (a) its reliability, (b) its reproducibility in the laboratory, and (c) its measurement in absolute terms expert performance as an empirical phenomenon / deliberate practice: a broader view (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
This article presents a theory in which automatization is construed as the acquisition of a domain-specific knowledge base, formed of separate representations, instances, of each exposure to the task. Processing is considered automatic if it relies on retrieval of stored instances, which will occur only after practice in a consistent environment. Practice is important because it increases the amount retrieved and the speed of retrieval; consistency is important because it ensures that the retrieved instances will be useful. The theory accounts quantitatively for the power-function speed-up and predicts a power-function reduction in the standard deviation that is constrained to have the same exponent as the power function for the speed-up. The theory accounts for qualitative properties as well, explaining how some may disappear and others appear with practice. More generally, it provides an alternative to the modal view of automaticity. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
discusses problems in transfer of training from formal models of decision making / describes remedies for training of naturalistic decision making (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
The highest levels of performance and achievement in sports, games, arts, and sciences have always been an object of fascination, but only within the last couple of decades have scientists been studying these empirical phenomena within a general theoretical framework. [This book] brings together [research] on specific domains of expertise and related theoretical issues, such as the importance of individual differences in ability and innate talent for attaining expert levels of performance. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
The first step in the development of an expert system is the extraction and characterization of the knowledge and skills of an expert. This step is widely regarded as the major bottleneck in the system development process. To assist knowledge engineers and others who might be interested in the development of an expert system, I offer (1) a working classification of methods for extracting an expert's knowledge, (2) some ideas about the types of data that the methods yield, and (3) a set of criteria by which the methods can be compared relative to the needs of the system developer. The discussion highlights certain issues, including the contrast between the empirical approach taken by experimental psychologists and the formalism-oriented approach that is generally taken by cognitive scientists.
Article
The Critical Decision Method (CDM) is an approach to cognitive task analysis. The method involves multiple-pass event retrospection guided by probe questions. The CDM has been used in the elicitation of expert knowledge in diverse domains and for applications including system development and instructional design. The CDM research illustrates the sorts of knowledge representation products that can arise from cognitive task analysis (e.g., Situation Assessment Records, time lines, decision requirements). The research also shows how one can approach methodological issues surrounding cognitive task analysis, including questions about data quality and method reliability, efficiency, and utility. As cognitive task analysis is used more widely in the elicitation, preservation, and dissemination of expert knowledge and is used more widely as the basis for the design of complex cognitive systems, and as projects move into even more field applications and real-world settings, the issues we discuss become increasingly critical.
Article
Information on knowledge elicitation methods is widely scattered across the fields of psychology, business management, education, counseling, cognitive science, linguistics, philosophy, knowledge engineering and anthropology. The purpose of this review is to (1) identify knowledge elicitation techniques and the associated bibliographic information, (2) organize the techniques into categories on the basis of methodological similarity, and (3) summarize for each category of techniques strengths, weaknesses, and recommended applications. The review is intended to provide a starting point for those interested in applying or developing knowledge elicitation techniques, as well as for those more generally interested in exploring the scope of the available methodology.
Book
What does a chessmaster think when he prepartes his next move? How are his thoughts organized? Which methods and strategies does he use by solving his problem of choice? To answer these questions, the author did an experimental study in 1938, to which famous chessmasters participated (Alekhine, Max Euwe and Flohr). This book is still usefull for everybody who studies cognition and artificial intelligence.