Article

The comparative effectiveness of distributed and colocated team after-action reviews

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Despite their frequent use in the military and private sectors, the comparative effectiveness of colocated and distributed after-action reviews (AARs) is relatively unknown. Consequently, this study examined the comparative effectiveness of colocated and distributed AARs across taskwork and teamwork outcomes. Data were obtained from 492 participants randomly assigned to 123 four-person teams who participated in one of six AAR conditions. The results indicated that teams in the AAR conditions had significantly higher performance and team efficacy scores than the teams in the non-AAR conditions. In summary, the findings highlight that regardless of the training environment or type of AAR, the AAR remains an effective method at increasing performance and other outcomes. Therefore, the use of distributed AARs does not engender the posited process losses that were hypothesized.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Meta-analytical findings have demonstrated that virtual teams exhibit lower levels of performance (e.g., Lacerenza et al., 2017). With regard to the benefits of collective reflections within virtual teams, the findings are equivocal; in some instances, the effects of team reflections are stronger in face-to-face teams (Jarrett et al., 2016), yet in others team reflections improve performance in virtual teams (e.g., Gurtner, Tschan, Semmer, & Nagele, 2007;Konradt et al., 2015). For this reason, we will examine the differential effects of team virtuality on the effectiveness of team reflexivity interventions, rather than propose a specific hypothesis. ...
... In studies (K = 3) in which there was an experimental condition unrelated to the aims of the metaanalysis (e.g., individual reflection), the condition was excluded from the analysis. Fourth, two eligible studies were coded as containing two experiments (a and b); in one instance, two experiments were reported within the same study (pilot and experiment; Dyas, 2018), whereas experimental conditions were compared to control conditions in two different location conditions in the second paper (distributed and co-located; Jarrett et al., 2016). ...
... , all studies delivered the intervention in a group setting, with team size ranging from three to eight members (M = 4.5, SD = 4.1). The majority of studies provided interventions using a face-to-face delivery style (64%, n = 16), alternately studies utilised elements of virtuality in the delivery of interventions (28%, n = 7); two studies(Jarrett et al., 2016; Kondradt et al., 2015) utilised both face-to-face and virtual delivery styles. For interventions delivered in a virtual environment, team communication occurred through chat functions or e-mails in all but one study in which team members could communicate via headsets with a voice activated microphone. ...
Article
Cumulating evidence from 24 independent randomized controlled trials or experiments (N = 4,339), we meta-analytically examined questions regarding the effectiveness of team reflexivity on collective performance outcomes and behaviors, and the conditions under which such effects are strongest. We addressed these questions by testing the overall effect of team reflexivity on performance outcomes (i.e., indices or metrics that quantify goal attainment) and behaviors (i.e., those actions or states that precede or influence goal attainment), and assessing the robustness of this pooled effect across study (e.g., team size), outcome (e.g., measurement approach), and intervention characteristics (e.g., virtuality) via a series of moderator analyses. We found a positive and significant medium overall effect of team reflexivity interventions on performance outcomes (g = .549) and performance behaviors (g = .548). Moderator analyses indicated that the team reflexivity-performance outcome effect is contingent upon the measurement approach (in favor of self-reports over objective indices or researcher-assessed outcomes). With regard to performance behaviors, the effect of team reflexivity was strongest for cognitions and behaviors relative to affective dimensions, and when interventions were delivered to teams present physically relative to virtual teams. Collectively, our findings extend existing meta-analytic evidence regarding team reflexivity interventions in terms of resolution (i.e., inability to isolate unique effects), scope (i.e., primary studies missed via the systematic search), and methodological quality of primary evidence (i.e., incorporation of quasi-experimental designs).
... Compared with no AARs, performing AARs after completing multiple problems is more helpful to improve mathematics performance. This is consistent with the results of previous studies, such as that AARs can significantly improve the training effect compared with the absence of AARs (Ellis et al., 2010;Jarrett et al., 2016;Villado & Arthur, 2013). In the AARs groups, students were required to review and explain the problem-solving process and the results based on the three AARs questions supplied after solving a one-variable quadratic equation problem. ...
Article
Reviewing the completed exercises was essential to mathematical learning, but students often jump to the next exercise after finishing the current one without reviewing. It is not clear how to effectively conduct this learning activity. On the other side, after-action reviews (AARs) have proved to be a good learning practice that has been widely used in military and unmilitary training for decades. It helps students learn through reviewing. This study applied AARs in mathematical learning to improve students’ learning effectiveness through reviewing their completed exercises. In addition, the study further investigated how the question types of AARs influence students’ cognitive engagement. Ninth-grade students (N = 125) were randomly assigned to one of three experimental conditions (AARs of general questions, AARs of specific questions, and no AARs). The results revealed that conducting AARs after solving mathematics problems could significantly improve learning performance (p < .05). Compared with AARs of specific questions, AARs of general questions could significantly improve students’ cognitive engagement (p < .001). The findings provide important implications for mathematical education and learning practice.
Book
The main goal of the field of augmented cognition is to research and develop adaptive systems capable of extending the information management capacity of individuals through computing technologies. Augmented cognition research and development is therefore focused on accelerating the production of novel concepts in human-system integration and includes the study of methods for addressing cognitive bottlenecks (e.g., limitations in attention, memory, learning, comprehension, visualization abilities, and decision making) via technologies that assess the user’s cognitive status in real time. A computational interaction employing such novel system concepts monitors the state of the user, through behavioral, psychophysiological, and neurophysiological data acquired from the user in real time, and then adapts or augments the computational interface to significantly improve their performance on the task at hand. The International Conference on Augmented Cognition (AC), an affiliated conference of the HCI International (HCII) conference, arrived at its 16th edition and encouraged papers from academics, researchers, industry, and professionals, on a broad range of theoretical and applied issues related to augmented cognition and its applications. The field of augmented cognition has matured over the years to solve enduring issues such as portable, wearable neurosensing technologies and data fusion strategies in operational environments. These innovations coupled with better understanding of brain and behavior, improved measures of brain state change, and improved artificial intelligence algorithms have helped expand the augmented cognition focus areas to rehabilitation, brain-computer interfaces, and training and education. The burgeoning field of human-machine interfaces such as drones and autonomous agents are also benefitting from augmented cognition research. This volume of the HCII 2022 proceedings is dedicated to this year’s edition of the AC conference and focuses on topics related to understanding human cognition and behavior, brain activity measurement and electroencephalography, human and machine learning, and augmented cognition in extended reality. Papers of this one volume are included for publication after a minimum of two single-blind reviews from the members of the AC Program Board or, in some cases, from members of the Program Boards of other affiliated conferences. We would like to thank all of them for their invaluable contribution, support, and efforts.
Article
Full-text available
This study expands on Keiser and Arthur's (2021) meta-analysis of the after-action review (AAR), or debrief, by examining six additional task and training characteristics that contribute to or attenuate its effectiveness. The findings based on a bare-bones meta-analysis of results from 83 studies (134 ds [955 teams; 4,684 individuals]) indicate that the effectiveness of the AAR (overall d = 0.92) does indeed vary across the pertinent characteristics. The primary impact of this study pertains to the practical implementation of AARs; notably, the findings indicate that the AAR is particularly effective in task environments that are characterized by a combination of high complexity and ambiguity in terms of offering no intrinsic feedback. The types of tasks-often project and decision-making-that more commonly entail these characteristics are frequently used in industries that do not traditionally use the AAR. The results also suggest that more recent variants of the AAR (i.e., a reaction phase, a canned performance review) do not meaningfully add to its effectiveness. These findings are combined with those from prior meta-analyses to derive 11 empirically-based practical guidelines for the use of AARs. In sum, this study highlights the complexity of the AAR that results from the independent and interdependent influence among various components and characteristics, the examination of the effects of novel and ostensibly distinct variants or approaches to AARs, and the extension of AARs to tasks and contexts in which they are less commonly used.
Article
Full-text available
Objective The present study examined the effectiveness of after-action reviews (AARs; also known as debriefing) in mitigating skill decay. Background Research on the long-term effectiveness of AARs is meager. To address this gap in the literature, we conducted an experimental study that also overcomes some research design issues that characterize the limited extant research. Method Eighty-four participants were randomly assigned to an AAR or non-AAR condition and trained to operate a PC-based fire emergency simulator. During the initial acquisition phase, individuals in the AAR condition were allowed to review their performance after each practice session, whereas individuals in the non-AAR condition completed a filler task. About 12 weeks later, participants returned to the lab to complete four additional practice sessions using a similar scenario (i.e., the retention and reacquisition phase). Results The performance of participants in the AAR condition degraded more after nonuse but also recovered faster than the performance of participants in the non-AAR condition, although these effects were fairly small and not statistically significant. Conclusion Consistent with the limited research on the long-term effectiveness of AARs, our findings failed to support their effectiveness as a decay-prevention intervention. Because the present study was conducted in a laboratory setting using a relatively small sample of undergraduate students, additional research is warranted. Application Based on the results of the present study, we suggest some additional strategies that trainers might consider to support long-term skill retention when using AARs.
Article
This study examined the effectiveness of the after-action review (AAR)—also commonly termed debrief—and four training characteristics within the context of Villado and Arthur’s (2013) conceptual framework. Based on a bare-bones meta-analysis of the results from 61 studies (107 ds [915 teams and 3,499 individuals]), the AAR leads to an overall d of 0.79 improvement in multiple training evaluation criteria. This effect is larger than some of the largest training method effects reported in Arthur et al. (2003), and it is also larger than Tannenbaum and Cerasoli’s (2013) estimate of the effect of the AAR on task performance (d = 0.67). Two training characteristics consistently contributed to the effectiveness of the AAR: (1) alignment to the individual or the team, and (2) objective performance review media. The effects of the other training characteristics were often interactive. Most notably, the facilitation approach contributes to the effectiveness of the AAR in combination with the individual vs. the team and the type of review media, with the most effective combinations being the self-led facilitation approach coupled with a team-aligned AAR, and the self-led approach coupled with objective media. Additionally, the AAR that is highly structured is more effective than a less structured AAR in the military, but high and low structured AARs display comparable effectiveness in healthcare. Overall, this study suggests that the effectiveness of the AAR should be understood as a function of the combined influence among multiple interacting characteristics. Future theoretical development and research should be directed at better understanding these interactions.
Article
Full-text available
Debriefs are a type of work meeting in which teams discuss, interpret, and learn from recent events during which they collaborated. In a variety of forms, debriefs are found across a wide range of organizational types and settings. Well-conducted debriefs can improve team effectiveness by 25% across a variety of organizations and settings. For example, the U.S. military adopted debriefs decades ago to promote learning and performance across the various services. Subsequently, debriefs have been introduced in the medical field, the fire service, aviation, education, and in a variety of organizational training and simulation environments. After a discussion of various purposes for which debriefs have been used, we proceed with a historical review of development of the concepts and use in industries and contexts. We then review the psychological factors relevant to debrief effectiveness and the outcomes for individuals, teams, and organizations that deploy debriefs. Future directions of particular interest to team researchers across a variety of psychological disciplines are presented along with a review of how best to implement debriefs from a practical perspective.
Article
Full-text available
We argue herein that typical training procedures are far from optimal. The goat of training in real-world settings is, or should be, to support two aspects of posttraining performance: (a) the level of performance in the long term and (b) the capability to transfer that training to related tasks and altered contexts. The implicit or explicit assumption of those persons responsible for training is that the procedures that enhance performance and speed improvement during training will necessarily achieve these two goals. However, a variety of experiments on motor and verbal learning indicate that this assumption is often incorrect. Manipulations that maximize performance during training can be detrimental in the long term; conversely, manipulations that degrade the speed of acquisition can support the long-term goals of training. The fact that there are parallel findings in the motor and verbal domains suggests that principles of considerable generality can be deduced to upgrade training procedures.
Article
Full-text available
In many high-risk domains, simulators are used for training and evaluating team performance under realistic conditions. Once the simulation is complete, the teams review their performance to identify the lessons that they have learned. These post-training debrief sessions may be either instructor- or team-led. Unfortunately, the relative effectiveness of instructor- versus team-led debriefs remains unclear. To address this question, we surveyed a nationwide, representative sample of over 30,000 pilots from 24 U.S. airlines. Despite having a high degree of statistical power and a reliable scale, we found no statistically or practically significant differences among the four most common approaches to post-training feedback: team debrief with videotape, team debrief without videotape, instructor debrief with videotape, and instructor debrief without videotape. The results suggest that all four approaches may be equally effective.
Article
Full-text available
The commonly used form of rwg. (J) can display irregular behavior, so four variants of this index were examined. An alternative index, r&ast;wg. J, is recommended. This index is an inverse linear function of the ratio of the average obtained variance to the variance of uniformly distributed random error. r&ast;wg.Jis superficially similar to Cronbach’s α, but careful examination confirms that r&ast;wg.Jis an index of agreement, not reliability. Based on an examination of the small-sample behavior of rwgand r&ast;wg.J, sample sizes of 10 or more raters are recommended.
Article
Full-text available
This article uses an ecological approach to analyze factors in the effectiveness of work teams—small groups of interdependent individuals who share responsibility for outcomes for their organizations. Applications include advice and involvement, as in quality control circles and committees; production and service, as in assembly groups and sales teams; projects and development, as in engineering and research groups: and action and negotiation, as in sports teams and combat units. An analytic framework depicts team effectiveness as interdependent with organizational context, boundaries, and team development. Key context factors include (a) organizational culture, (b) technology and task design, (c) mission clarity, (d) autonomy, (e) rewards, (f) performance feedback, (g) training/consultation, and (h) physical environment. Team boundaries may mediate the impact of organizational context on team development. Current research leaves unanswered questions but suggests that effectiveness depends on organizational context and boundaries as much as on internal processes. Issues are raised for research and practice. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower level manager, or a team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For such situations, these procedures furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. The proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency and social desirability) on estimates of interrater reliability. (49 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Meta-analytic techniques were used to examine the effectiveness of Web-based instruction (WBI) relative to classroom instruction (CI) and to examine moderators of the comparative effectiveness of the 2 delivery media. The overall results indicated WBI was 6% more effective than CI for teaching declarative knowledge, the 2 delivery media were equally effective for teaching procedural knowledge, and trainees were equally satisfied with WBI and CI. However, WBI and CI were equally effective for teaching declarative knowledge when the same instructional methods were used to deliver both WBI and CI, suggesting media effects are spurious and supporting Clark's (1983, 1994) theory. Finally, WBI was 19% more effective than CI for teaching declarative knowledge when Web-based trainees were provided with control, in long courses, and when trainees practiced the training material and received feedback during training. Study limitations and directions for future research are discussed.
Article
Full-text available
A meta-analysis of research comparing decision making in face-to-face versus computer-mediated communication groups was conducted. Results suggest that computer-mediated communication leads to decreases in group effectiveness, increases in time required to complete tasks and decreases in member satisfaction compared to face-to-face groups. All of the moderators tested (anonymity in the group process, limited versus unlimited time to reach decisions, group size, and task type) were significant for at least one of the dependent variables. The article concludes with cautions about the unbridled rush by organizations to adopt computer-mediated communication as a medium for group decision making and implications of the present findings for theory and research on computer-mediated communication and group decision making.
Article
Full-text available
This study examines the effect of guided reflection on team processes and performance, based on West’s (1996, 2000) concept of reflexivity. Communicating via e-mail, 49 hierarchically structured teams (one commander and two specialists) performed seven 15 min shifts of a simulated team-based military air-surveillance task (TAST) in two meetings, a week apart. At the beginning of the second meeting, teams were assigned either to a reflexivity (individual or group) or to a control condition. Results show that reflexivity enhanced performance, the link between reflexivity and team performance being mediated by communication and implementation of strategies as well as by similarity of mental models. Contrary to expectations, individual reflexivity was superior to group reflexivity. Additional analyses suggested that group reflexivity decreased the commanders’ active behavior and increased discussion of strategies that were too general to be helpful. Results point to the usefulness of reflexivity as a generic intervention but underscore the importance of focusing on strategies that are task-specific.
Article
Full-text available
An unanswered question in employee development is how reflection can be used for improving performance in organizations. Drawing from research and theory on dual-process models, we develop and test a reflection strategy to stimulate deeper learning after feedback. Results of two studies (N = 640 and N = 488) showed that reflection combined with feedback enhanced performance improvement on a web-based work simulation better than feedback alone. Reflection without feedback did not lead to performance improvement. Further analyses indicated that the proposed reflection strategy was less effective for individuals low in learning goal orientation, low in need for cognition, and low in personal importance as they engaged less in reflection. Together, these findings provide a theoretical basis for the future study of reflection in organizations and suggest a practical and cost-effective strategy for facilitating employee development after feedback in organizations.
Article
Full-text available
Recent advances in networking environments and telecommunications have led to the proliferation of teams that do not work face-to-face but interact over a computer-mediated communications network. Although some have asserted that virtual teams transcend boundaries of time or distance, others have claimed that working remotely in a mediated team environment differs in significant ways from working face-to-face. In this article, the authors examine the effects of technological mediation on team processes such as cohesiveness, status and authority relations, counternormative behavior, and communication. They discuss conditions under which distance matters in virtual team interaction.
Article
Full-text available
Technological innovations in data transfer and communication have given rise to the virtual team where geographically separate individuals interact via one or more technologies to combine efforts on a collective activity. In military, business, and spaceflight settings, virtual teams are increasingly used in training and operational activities; however there are important differences between these virtual collaborations and more traditional face-to-face (FTF) interactions. One concern is the absence of FTF contact may alter team communication and cooperation and subsequently affect overall team performance. The present research examined this issue with a specific focus on how communication modality influences team learning and performance gains. Evidence from a recent study on virtual team performance (Singer, Grant, Commarford, Kring, & Zavod, 2001) indicated local teams, with both members in same physical location in Orlando, Florida which allowed for FTF contact before and after a series of virtual environment (VE) missions, performed significantly better than distributed teams, with team members in separate physical locations in Orlando and Toronto, Canada and no FTF contact. For the first mission, local and distributed teams exhibited no significant difference in performance as measured by the number of rooms properly cleared in the building search exercises. In contrast, for the second mission, occurring after each team had completed the opportunity to discuss mission performance and make plans for future missions, local teams performed significantly better than distributed teams; a pattern that continued for the remaining six missions. Given that the primary difference between local and distributed teams was how they communicated outside of the VE during after action reviews (AARs), and that the localiii distributed difference was first detected on the second mission, after teams had completed one, 10-min discussion of mission performance, a tenable conclusion is that certain team characteristics and skills necessary for performance were communication-dependent and negatively affected by the absence of FTF communication. Although Singer et al. (2001) collected multiple dependent variables related to performance and communication activities, these measures were not designed to detect communication-dependent team factors and therefore incapable of supporting such an explanation. Therefore, the present research replicated Singer et al. (2001) and incorporated additional measures in order to determine if specific communication-dependent factors could explain the inferior performance of distributed teams. Three factors critical to team communication, particularly during the AAR process, are the similarity of team members. shared mental models (SMMs), team cohesion (task and interpersonal), and team trust (cognitive and emotional). Because evidence suggests FTF communication has a positive effect on processes related to each of these factors, the current study tested whether distributed teams exhibit less similar mental models and degraded cohesion and trust in comparison to local teams, which can affect performance. Furthermore, to test the prediction that distributed teams possess degraded communication and would benefit from improved communication skills, brief team communication training (TCT) was administered to half of the teams in each location condition. Thirty two, 2-person teams comprised of undergraduate students were equally distributed into four experimental conditions (n = 8) based on the independent variables of location (local vs. distributed) and training (TCT vs. no-TCT). Teams completed five missions using the same VE system and mission tasks as in Singer et al. (2001), however in the present study distributed team members were in separate rooms in the same building, not separate geographic locations. In iv addition to performance data, participants completed a series of questionnaires to assess SMMs, cohesion, and trust. It was hypothesized that local teams would again exhibit better performance than distributed teams and that the local team advantage could partly be explained by a greater similarity in mental models and higher levels of cohesion and trust. Moreover, TCT teams in both locations were expected to exhibit improved performance over their non-trained counterparts. Analyses of the three team factors revealed the largest location and communication training differences for levels of cognitive trust, with local teams reporting higher levels than distributed teams early after the second VE mission, and TCT teams reporting higher levels than no-TCT teams after the second and fifth VE missions. In contrast, the main effects of location and communication training were only significant for one SMM measure agreement between team members on the strengths of the team�s leader during the AAR sessions. Local teams and TCT teams reported higher levels of agreement after the first VE mission than their distributed v and no-TCT counterparts. Furthermore, on the first administration of the questionnaire, TCT teams reported higher levels of agreement than non-TCT teams on the main goals of the VE missions. Overall, teams in all conditions exhibited moderate to substantial levels of agreement for procedural and personnel responsibility factors, but poor levels of agreement for mental models related to interpersonal interactions. Finally, no significant differences were detected for teams in each experimental condition on levels of task or interpersonal cohesion which suggests cohesion may not mature enough over the course of several hours to be observable. In summary, the first goal of the present study was to replicate Singer et al..s (2001) findings which showed two-person teams conducting VE missions performed better after the first mission if allowed face-to-face (FTF) contact during discussions of the team�s performance. Local and distributed teams in the current study did show a similar pattern of performance, completing a greater total of rooms properly, although when evaluating mission-by-mission performance, this difference was only significant for missions 3 and 4. Even though distributed team members experienced the same experimental conditions as in Singer et al. (no pre-mission contact, no FTF contact during missions or AARs) and were told their partner was at .distant location, familiarity with a teammate's dialect and other environmental cues may have differentially affected perceptions of physical and psychological distance, or social presence, which ultimately altered the distributed team relationship from before. The second goal was to determine if brief TCT could reduce or eliminate the distributed team disadvantage witnessed in Singer et al. (2001). Results did not support this prediction and revealed no significant differences between TCT and no-TCT teams with regard to number of rooms searched over the five missions. Although purposefully limited to 1 hr, the brevity of the TCT procedure (1 hr), and its broad focus, may have considerably reduced any potential benefits of learning how to communicate more effectively with a teammate. In addition, the additional training beyond the already challenging requirements of learning the VE mission tasks may have increased the cognitive load of participants during the mission phase, leading to a detriment in performance due to divided attention. Despite several notable differences from Singer et al. (2001), the present study supports that distributed teams operating in a common virtual setting experience performance deficits when compared to their physically co-located counterparts. Although this difference was not attributed to agreement on SMMs or levels of cohesion, local teams did posses higher levels of cognitive trust early on in the experimental session which may partly explain their superior performance. However additional research that manipulates cognitive trust as an independent variable is needed before implying a cause-and-effect relationship. Ultimately, this study�s most significant contribution is identifying a new set of questions to understand virtual team performance. In addition to a deeper examination of cognitive trust, future research should address how features of the distributed team experience affect perceptions of the physical and psychological distance, or social presence, between team members. It is also critical to understand how broadening the communication channel for distributed teams, such as the inclusion of video images or access to biographical information about one�s distant teammate, facilitates performance in a variety of virtual team contexts.
Article
Full-text available
In the current study, we compared the effect of personal and filmed after-event reviews (AERs) on performance, and the role that self-efficacy plays in moderating and mediating the effects of these 2 types of AER on performance. The setting was one in which 49 men and 63 women participated twice in a simulated business decision-making task. In between, participants received a personal AER, watched a filmed AER, or had a break. We found that individuals who participated in an AER, whether personal or filmed, improved their performance significantly more than those who did not participate in a review. Furthermore, there was no significant difference in performance improvement between the personal and the filmed AER, which suggests that the 2 are quite similar in their effect. We also found that the differences in performance improvement between the personal AER group and the control group were somewhat greater than those found in the filmed AER group. Self-efficacy mediated the effect of AER on performance improvement in both types of AER. In addition, the effect of AER on performance improvement was moderated by initial self-efficacy in the personal but not in the filmed AER: The personal AER was more effective, the higher the initial self-efficacy.
Article
Full-text available
The claim that appropriate after-event review might decrease the relative advantage of drawing lessons from failures over drawing lessons from successes was examined in a quasi-field experiment. The results show that performance of soldiers doing successive navigation exercises improved significantly when they were debriefed on their failures and successes after each training day, compared with others who reviewed their failed events only. The findings also show that, before the manipulation, in both groups, learners' mental models of failed events were richer in constructs and links than were their mental models of successful events. This gap closed gradually in subsequent measurements.
Article
The authors describe a series of experiments that explore 3 major ability determinants of individual differences in skill acquisition in the context of prior theory (e.g., P. L. Ackerman, 1988) and subsequent empirical and theoretical research. Experiment 1 assessed the predictability of individual differences in asymptotic skill levels on the Kanfer-Ackerman Air Traffic Controller (ATC) task. Experiment 2 provided an exploration of the construct space underlying perceptual-speed abilities. Experiment 3 concerned an evaluation of theoretical predictions for individual differences in performance over skill development in a complex air traffic control simulation task (TRACON) and the ATC task, with an extensive battery of general and perceptual-speed measures, along with a newly developed PC-based suite of psychomotor ability measures. Evidence addressing the predictability of individual differences in performance at early, intermediate, and asymptotic levels of practice is presented.
Article
Virtual teams in organizations have now become a reality, but there have been only a handful of quantitative reviews on “virtualness” (i.e., teams that are more or less virtual). We decided to conduct a meta-analytic review of the effects of virtualness on team functioning (conflict, communication frequency, knowledge sharing, performance, and satisfaction). To explain inconsistencies in the results of published material on the topic, we also examined the moderating effects of level of analysis (individual/group), method (experiment/survey), and time frame (short/long). Eighty studies were found that covered some part of this domain. Results seem to differ in the relative importance of the factors. Thus though aggregated findings suggested negative effects of virtualness on team functioning, results varied in strength and direction of the moderators, indicating that it was not possible to generalize. For example, the negative effects held only for short-term teams, while in longer-term teams the effects weakened or disappeared.
Article
Using a longitudinal design, the authors examined the criterion-related validity of two operationalizations of task-specific team efficacy that differed in their approximation to the level of analysis of the criterion, team performance. Data were obtained from 85 highly interdependent dyadic teams trained over a 2-week period to perform a complex perceptual-motor skill task. Results indicated that, as expected, the operationalization with a team-level referent (referent-shift consensus) was superior to the operationalization with an individual-level referent (additive) across all three data collection periods. For the referent-shift consensus operationalization, within-team agreement and the criterion-related validity improved between the first and second data collection periods but not between the second and third. However, for both operationalizations, despite the increased strength of the team efficacy and team performance relationships, efficacy ratings collected later in the study protocol did not explain unique variance in subsequent team performance once the effect of previous performance was statistically controlled.
Article
Managers are challenged to develop strategically flexible organizations in response to increasingly competitive marketplaces. Fortunately, a new generation of information and telecommunications technology provides the foundation for resilient new organizational forms that would have not been feasible only a decade ago. One of the most exciting of these new forms, the virtual team, will enable organizations to become more flexible by providing the impressive productivity of team-based designs in environments where teamwork would have once been impossible. Virtual teams, which are linked primarily through advanced computer and telecommunications technologies, provide a potent response to the challenges associated with today's downsized and lean organizations, and to the resulting geographical dispersion of essential employees. Virtual teams also address new workforce demographics, where the best employees may be located anywhere the world, and where workers demand increasing technological sophistication and personal flexibility. With virtual teams, organizations can build teams with optimum membership while retaining the advantages of flat organizational structure. Additionally, firms benefit from virtual teams through access to previously unavailable expertise, enhanced cross-functional interaction, and the use of systems that improve the quality of the virtual team's work.
Article
Debriefs (or "after-action reviews") are increasingly used in training and work environments as a means of learning from experience. We sought to unify a fragmented literature and assess the efficacy of debriefs with a quantitative review. Used by the U.S. Army to improve performance for decades, and increasingly in medical, aviation, and other communities, debriefs systematize reflection, discussion, and goal setting to promote experiential learning. Unfortunately, research and theory on debriefing has been spread across diverse disciplines, so it has been difficult to definitively ascertain debriefing effectiveness and how to enhance its effectiveness. We conducted an extensive quantitative meta-analysis across a diverse body of published and unpublished research on team- and individual-level debriefs. Findings from 46 samples (N = 2,136) indicate that on average, debriefs improve effectiveness over a control group by approximately 25% (d = .67). Average effect sizes were similar for teams and individuals, across simulated and real settings, for within- or between-group control designs, and for medical and nonmedical samples. Meta-analytic methods revealed a bolstering effect of alignment and the potential impact of facilitation and structure. Organizations can improve individual and team performance by approximately 20% to 25% by using properly conducted debriefs. Debriefs are a relatively inexpensive and quick intervention for enhancing performance. Our results lend support for continued and expanded use of debriefing in training and in situ. To gain maximum results, it is important to ensure alignment between participants, focus and intent, and level of measurement.
Article
The after-action review (AAR; also known as the after-event review or debriefing) is an approach to training based on a review of trainees' performance on recently completed tasks or performance events. Used by the military for decades, nonmilitary organizations' use of AARs has increased dramatically in recent years. Despite the prevalence of AARs, empirical research investigating their effectiveness has been limited. This study sought to investigate the comparative effectiveness of objective AARs (reviews based on an objective recording and playback of trainees' recent performance) and subjective AARs (reviews based on a subjective, memory-based recall of trainees' recent performance). One hundred eighty-eight individuals, participating in 47 4-person teams, were assigned to 1 of 3 AAR conditions and practiced and tested on a cognitively complex performance task. Although there were no significant differences between objective and subjective AAR teams across the 5 training outcomes, AAR teams had higher levels of team performance, team efficacy, openness of communication, and cohesion than did non-AAR teams but no differences in their levels of team declarative knowledge. Our results suggest that AARs are effective at enhancing training outcomes. Furthermore, AARs may not be dependent on objective reviews and therefore may be a viable training intervention when objective reviews are not feasible or possible. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
Article
This paper seeks to evaluate a specific implementation of a reflective approach to teacher education in a pre‐service course in teaching English to speakers of other languages (TESOL) in a UK university. This implementation is the use of delayed debriefing (supervisory conference) after lesson observation, providing the student teacher with support for reflection in the form of a time delay and completion of a structured journal. Six delayed debriefing events are compared with six immediate debriefing events from another institution providing a course leading to the same award. Three analyses of student teacher talk in the debriefing events are presented: topic initiation, modal verb use, and types of ‘reasoning’ talk. The analyses offer some evidence of a higher level of reflective analysis by the student teacher in delayed debriefings.
Article
This article presents a review of the skill retention and skill decay literature that focuses on factors that influence the loss of trained skills or knowledge over extended periods of nonuse. Meta-analytic techniques were applied to a total of 189 independent data points extracted from 53 articles. Results indicate that there is substantial skill loss with nonpractice or nonuse, with the amount of skill loss ranging from an effect size (d) of -0.01 immediately after training to a d of -1.4 after more than 365 days of nonuse. Most of the study's hypotheses for moderators were supported. Physical, natural, and speed-based tasks were less susceptible to skill loss than cognitive, artificial, and accuracy-based tasks. Additionally, certain methodological variables, such as using recognition tests, using similar conditions of retrieval at retention, and using behavioral evaluation criteria, resulted in less skill loss over time. Implications of the results for training and future research are discussed.
Article
Studies suggest that young children are quite limited in their knowledge about cognitive phenomena—or in their metacognition—and do relatively little monitoring of their own memory, comprehension, and other cognitive enterprises. Metacognitive knowledge is one's stored knowledge or beliefs about oneself and others as cognitive agents, about tasks, about actions or strategies, and about how all these interact to affect the outcomes of any sort of intellectual enterprise. Metacognitive experiences are conscious cognitive or affective experiences that occur during the enterprise and concern any aspect of it—often, how well it is going. Research is needed to describe and explain spontaneous developmental acquisitions in this area and find effective ways of teaching metacognitive knowledge and cognitive monitoring skills. (9 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Advances in telecommunications and simulation technologies have created opportunities for conducting distributed team training through networked simulations. In distributed team training, military teams train together in the same battlespace despite the physical separation. in these types of training environments, multiple users are located at multiple sites; consequently, the efficient and effective conduct of training is a challenge. One area that is particularly challenging is the measurement of team performance. Two case studies are reported in which team performance measurement instruments were developed and tested in a distributed training environment. The measurement tools were designed within the context of an instructional approach known as event-based training, which relies on the creation of explicit linkages among learning objectives, exercise events, performance measures, and after-action review or feedback. Active duty, reserve, and National Guard personnel from the Marine Corps, Air Force, and Army participated in several days of training exercises conducted within a network of simulators that were geographically distributed across the United States. The development and use of the measurement instruments are described, data from both case studies are presented, and implications for training are discussed. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Two studies (one field, one experimental) found that the more accurately individuals evaluated their performance, the better they performed on a subsequent task. The first study also found that the more individuals overestimated their previous performance, the lower was their performance on the next task. In contrast, the evaluation accuracy of the underestimators was unrelated to their subsequent performance. The second study found that when participants received feedback from an external authority, the effect of the inaccuracy of self-performance evaluation on subsequent performance was reduced. The results of the 2 studies are explained in motivational and cognitive terms.
Article
[Excerpt] Our objective in this chapter is to provide an integrative perspective on work groups and teams in organizations, one that addresses primary foci of theory and research, highlights applied implications, and identifies key issues in need of research attention and resolution. Given the volume of existing reviews, our review is not intended to be exhaustive. Rather, it uses representative work to characterize key topics, and focuses on recent work that breaks new ground to help move theory and research forward. Although our approach risks trading breadth for depth, we believe that there is much value in taking a more integrative view of the important areas of team research, identifying key research themes, and linking the themes and disparate topics closer together. To the extent that we identify new and necessary areas of theory development and research, the value of this approach will be evident.
ABSTRACT The United States Army currently uses after action reviews (AARs) to give personnel feedback on their performance. However, due to the growing use of geographically distributed teams, the traditional AAR, with participants and a moderator in the same room, is becoming difficult; therefore, distributed AARs are becoming a necessity. However, distributed AARs have not been thoroughly researched. To determine what type of distributed AARs would best facilitate team training in distributed Army operations, feedback media platforms must be compared. This research compared three types of AARs, which are no AAR, teleconference AAR, and teleconference AAR with visual feedback, to determine if there are learning differences among these conditions. Participants completed three search missions and received feedback between missions from one of these conditions. Multiple ANOVAs were conducted to compare these conditions and trials. Results showed that overall the teleconference AAR with visual feedback improved performance the most. A baseline, or no AAR, resulted in the second highest improvement, and the teleconference condition resulted in the worst overall performance. This study has implications for distributed military training and feedback, as well as other domains that use distributed training and feedback.
Article
Using the five factor model with an emphasis on extraversion and conscientiousness, the authors investigated how personality is related to small group processes and outcomes. Graduate students (N = 289) assigned to 4- and 5-person teams in 61 groups engaged in a series of creative problem-solving tasks over a period of several weeks. Extraversion was associated with group processes and outcomes at both individual and group levels of analysis. At the individual level, extraverts were perceived by others as having greater effect than introverts on group outcomes. Covariance structure modeling suggested that extraverts induce these perceptions through the provision of both socioemotional and task-related inputs. At the group level, the proportion of relatively extraverted members was related curvilinearly to task focus and group performance. Contrary to expectations, Conscientiousness was unrelated to processes and outcomes at either the individual or group level.
Article
The effectiveness and efficiency of the active interlocked modeling (AIM) dyadic protocol in training complex skills has been extensively demonstrated. However, past evaluation studies have all used male participants exclusively. Consequently, the present study investigated the generalizability of the effectiveness and efficiency gains to women. We randomly assigned 108 female participants to either the AIM-dyad condition or a standard individual control training condition. The results supported the robustness and viability of the AIM protocol. Although their overall performance was lower than that obtained for men in previous studies, women trained in the AIM-dyad condition performed as well as those trained in the individual condition. Thus, the efficiency gains associated with the AIM-dyad protocol, which result from the ability to train two people simultaneously to reach the same performance level as a single person with no increase in training time or machine cost, are generalizable to female participants. The applied and basic research implications of the present study are discussed within the context of well-documented male/female differences in the performance of complex psychomotor tasks. For instance, given the number of women entering the workforce and the significant proportion of women in professions previously deemed to be male-dominated (e.g., air navigation), it is reassuring to know that sex differences in task performance do not necessarily imply sex differences in the effectiveness of training protocols.
Article
The authors describe a series of experiments that explore 3 major ability determinants of individual differences in skill acquisition in the context of prior theory (e.g., P.L. Ackerman, 1988) and subsequent empirical and theoretical research. Experiment 1 assessed the predictability of individual differences in asymptotic skill levels on the Kanfer-Ackerman Air Traffic Controller (ATC) task. Experiment 2 provided an exploration of the construct space underlying perceptual-speed abilities. Experiment 3 concerned an evaluation of theoretical predictions for individual differences in performance over skill development in a complex air traffic control simulation task (TRACON) and the ATC task, with an extensive battery of general and perceptual-speed measures, along with a newly developed PC-based suite of psychomotor ability measures. Evidence addressing the predictability of individual differences in performance at early, intermediate, and asymptotic levels of practice is presented.
Article
One of the most remarkable changes in aviation training over the past few decades is the use of simulation. The capabilities now offered by simulation have created unlimited opportunities for aviation training. In fact, aviation training is now more realistic, safe, cost-effective, and flexible than ever before. However, we believe that a number of misconceptions--or invalid assumptions--exist in the simulation community that prevent us from fully exploiting and utilizing recent scientific advances in a number of related fields in order to further enhance aviation training. These assumptions relate to the overreliance on high-fidelity simulation and to the misuse of simulation to enhance learning of complex skills. The purpose of this article is to discuss these assumptions in the hope of initiating a dialogue between behavioral scientists and engineers.
Article
The authors used meta-analytic procedures to examine the relationship between specified training design and evaluation features and the effectiveness of training in organizations. Results of the meta-analysis revealed training effectiveness sample-weighted mean ds of 0.60 (k = 15, N = 936) for reaction criteria, 0.63 (k = 234, N = 15,014) for learning criteria, 0.62 (k = 122, N = 15,627) for behavioral criteria, and 0.62 (k = 26, N = 1,748) for results criteria. These results suggest a medium to large effect size for organizational training. In addition, the training method used, the skill or task characteristic trained, and the choice of evaluation criteria were related to the effectiveness of training programs. Limitations of the study along with suggestions for future research are discussed.
Article
The debriefing process during simulation-based education has been poorly studied despite its educational importance. Videotape feedback is an adjunct that may enhance the impact of the debriefing and in turn maximize learning. The purpose of this study was to investigate the value of the debriefing process during simulation and to compare the educational efficacy of two types of feedback, oral feedback and videotape-assisted oral feedback, against control (no debriefing). Forty-two anesthesia residents were enrolled in the study. After completing a pretest scenario, participants were randomly assigned to receive no debriefing, oral feedback, or videotape-assisted oral feedback. The debriefing focused on nontechnical skills performance guided by crisis resource management principles. Participants were then required to manage a posttest scenario. The videotapes of all performances were later reviewed by two blinded independent assessors who rated participants' nontechnical skills using a validated scoring system. Participants' nontechnical skills did not improve in the control group, whereas the provision of oral feedback, either assisted or not assisted with videotape review, resulted in significant improvement (P < 0.005). There was no difference in improvement between oral and video-assisted oral feedback groups. Exposure to a simulated crisis without constructive debriefing by instructors offers little benefit to trainees. The addition of video review did not offer any advantage over oral feedback alone. Valuable simulation training can therefore be achieved even when video technology is not available.
The effectiveness of after-action reviews as a training method: A meta-analysis (Unpublished manuscript)
  • I Schurig
  • S Jarrett
  • W Arthur
  • Jr
  • R M Glaze
  • M Schurig
Schurig, I., Jarrett, S., Arthur, W., Jr., Glaze, R. M., & Schurig, M. (2011). The effectiveness of after-action reviews as a training method: A meta-analysis (Unpublished manuscript). Texas A&M University, College Station.
Team training in virtual environments: An event based approach
  • E Salas
  • R L Oser
  • J A Cannon-Bowers
  • E Daskarolis-Kring
Salas, E., Oser, R. L., Cannon-Bowers, J. A., & Daskarolis-Kring, E. (2002). Team training in virtual environments: An event based approach. In M. K. Stanney (Ed.), Handbook of virtual environments: Design, implementation, and applications. Human factors and ergonomics (pp. 873-892). Mahwah, NJ: Erlbaum.
Beyond facilitation: An improved CRM debrief for safety training
  • C Prince
  • E Salas
  • M Brannick
  • J Orasnau
Prince, C., Salas, E., Brannick, M., & Orasnau, J. (2005). Beyond facilitation: An improved CRM debrief for safety training. Human Factors and Aerospace Safety, 5, 1-22.
Determinants of effective unit performance: Research on measuring and managing unit training readiness
  • L L Meliza
  • D W Bessemer
  • J H Hiller
Meliza, L. L., Bessemer, D. W., & Hiller, J. H. (1994). Providing unit training feedback in the distributed interactive simulation environment. In R. F. Holz, H. Hiller, & H. H. McFann (Ed.), Determinants of effective unit performance: Research on measuring and managing unit training readiness (pp. 257-280). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
Factors that influence skill decay and retention: A quantitative analysis
  • W Arthur
  • B W Jr
  • P L Stanush
  • T L Mcnelly
Arthur, W., Jr., B. W., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative analysis. Human Performance, 11, 57-101.
Steel Beasts Pro PE ver. 2.370 [Computer software]
  • Esim Games
eSim Games. (2007). Steel Beasts Pro PE ver. 2.370 [Computer software]. Mountain View, CA: Author.
A leader's guide to after-action reviews (Training Circular 25-20)
United States Army. (1993). A leader's guide to after-action reviews (Training Circular 25-20). Washington, DC: Author.