ArticlePDF Available

What Will Quantitative Measures of Teamwork Look Like in 10 Years?

Session Chair: Eduardo Salas, Ph.D.
Rice University, Houston, TX
Ron Stevens, Ph.D., UCLA School of Medicine, The Learning Chameleon, Inc., Los Angeles, CA
Jamie Gorman, Ph.D., Georgia Institute of Technology, Atlanta, GA
Nancy J. Cooke, Ph.D., Arizona State University, Mesa, AZ
Stephen Guastello, Ph.D., Marquette University, Milwaukee, WI
Alina A. von Davier, Ph.D., Educational Testing Service, Inc., Princeton, NJ
Nationwide there is a need for cost-effective training
solutions that are highly automated, adaptable, and capable of
producing quantifiable behavioral changes in teams that are
indicative of deep learning. In contrast to what is known
about individual skill acquisition and persistence, relatively
little is known about how team process skills develop; how
well these skills once learned in one context transfer to
another context; how long the skills persist when unused; and,
what interventions or training will most rapidly restore them?
Answering these questions is challenging due to the
limited number of quantitative teamwork measures that track
team performance, cohesion and flexibility across teams, time,
environments and training protocols. Adopting a scientific
approach for studying the effects of training interventions is
problematic without theory and methods that are aligned and
capable of representing and capturing the dynamics of team
A confluence of new technologies will soon generate
enormous amounts of new data at an unprecedented level of
detail about teams. But these data will also raise questions of
their own; principally how researchers will make sense of the
expected onslaught of data and derive general organizing
principles that guide the co-evolution of the complex team and
task interactions. This suggests the need for novel methods
and ways of thinking about team dynamics and measurement.
Our goals are to speculate, given where we are,
where the measurement and assessment of learning and
performance of teams and of individuals in teams might go in
the next decade and how we might get there. As such, some
sample ‘Big’ Questions’ that the panelists were asked to
consider in their presentations include:
What types of high and low level data abstractions
might provide the most useful quantitative
information about teamwork?
Across which biologic and interpersonal scales of
teamwork will the strongest information flows be
Can dynamical clues tell us how well a team is
performing / will perform?
In addition to performance assessment, what can we
learn from dynamics about team flexibility, cohesion,
leadership, and resilience?
How can we disentangle individual contributions
from the team contribution and accurately measure
Toward Quantitative Descriptions of the Neurodynamic
Organizations of Teams
Ron Stevens
UCLA School of Medicine,
The Learning Chameleon, Inc.
Advances in our understanding of the learning
process used by teams while they develop and refine their
team skills have been slowed by a lack of easily understand-
able quantitative approaches that can objectively and
automatically assess collective learning processes and
outcomes over time in training situations.
We have developed symbolic models of teamwork
that capture the brainwave levels of each person of the team,
and situates them in the context of the levels of other team
members as well as the immediate context of the task.
Quantitative estimates of the symbol variations in the data
stream are then made using a moving window of entropy
approach. Periods of decreased entropy represent times of
increased team neurodynamic organization when there were
prolonged and restricted neurodynamic relationships across
members of the teams.
Using this approach we have shown that (a) the
neurodynamic rhythms of six-person US Navy submarine
navigation teams are measurable and are entrained by the task
(Stevens, Gorman, Amazeen, Likens & Galloway, 2013); (b)
the structure of these rhythms is multifractal, resulting from
the meso and micro responses of teams to changes in the task
and the sharing of information across the crew (Likens,
Amazeen, Stevens, Galloway & Gorman, 2014); and (c)
quantitative differences in team’s neurodynamic rhythms are
linked with team expertise, resilience, and communication
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015 235
Copyright 2015 Human Factors and Ergonomics Society. DOI 10.1177/1541931215591048
(Stevens & Galloway, 2014). Consistent with the nonlinear
dynamical systems concepts of elasticity and rigidity in
complex adaptive systems, the expert navigation teams were
positioned at an organizational neurodynamic point midway
between rigid and elastic. These findings suggest the
existence of team-related neurodynamical processes that
quantitatively track across the novice-expert continuum.
Team Performance at the Level of Emergent, Dynamic
Coordinative Relations
Jamie C. Gorman
Georgia Institute of Technology
In a recent series of experiments on dyadic,
interpersonal coordination, we found that the tendency to
spontaneously synchronize one’s movements with those of a
teammate can interfere with human performance in team tasks
like assisted suturing and knot tying (Gorman & Crites, 2014).
Namely, we found that the ability to decouple the hands, such
that they move independently (i.e., to not synchronize), is
fundamental to tying skill and is presumably acquired from an
early age. In the context of unfamiliar, dyadic tying, however,
participants were unable to fight the spontaneous tendency to
synchronize their hand movements, which hurt team
performance. How do we account for phenomena such as this
in the science of team learning? This example is meant to
illustrate the need for team skill acquisition and assessment to
account for high-level, emergent coordinative relations, such
as “sync” (Strogatz, 2003) that structure team performance
beyond the control of the individual.
Interpersonal tying provides a relatively simple
example of how emergent, dynamic coordinative relations
structure team performance, but we see the same phenomena
at work in more complex, cognitive settings, such as
intelligence analysis and planning and team command-and-
control. In those situations, team members self-organize
“cognitive-behavioral” (e.g., communication) patterns without
being consciously aware of it (Dunbar & Gorman, 2014). We
think the key to understanding how emergent coordinative
relations shape team performance is in identifying the
mathematical and statistical dynamics (e.g., sync; self-
organization) that occur as team members interact. In the next
decade, this approach will not produce new models of shared
cognition but will be characterized by mathematical models
that go beyond the individual mental state or top-down
knowledge with the goal of understanding how general
coordinative mechanisms structure team performance. Such
models will also have implications for understanding how
individual psychological processes are structured by emergent,
coordinative relations (Gorman, 2014).
This dynamic approach to understanding team
performance has already provided novel predictions and ways
of thinking about enhancing team performance, including
enhancing team flexibility and resilience. I will briefly
describe research conducted using the dynamic perspective
that has already provided new insights into how teams
develop, what develops, and how to enhance transfer to novel
contexts during skill acquisition in both motor-perceptual and
cognitive-behavioral tasks. I will link these results together by
briefly describing the common theoretical core that underlies
them, and I will briefly describe the types of models and
methodologies that are needed to understand team perform-
ance at the level of emergent, dynamic coordinative relations.
Communication Dynamics for Team Assessment
Nancy J. Cooke
Arizona State University
Teams can be viewed as complex dynamic systems
made up of interacting components that are systems
themselves. As effective teams learn, not only does their
performance improve, but their team process behaviors evolve
to become more flexible, adaptive, and resilient. These
process dynamics have been linked to team effectiveness
(Cooke, Gorman, Myers, & Duran, 2013).Therefore
assessment of team learning can benefit from an under-
standing of the dynamics of team interactions. Team
interactions often take place through communication, though
there are other forms of interaction that are nonverbal
including gestures, facial expressions, and implicit interaction.
Of these data sources, communication is the most straight-
forward to collect and therefore, most commonly collected.
Communication data can be collected unobtrusively. There are
also opportunities to analyze communication data in near-real
time for continuous monitoring of team learning and rapid
intervention. For instance, metrics based on communication
flow from person to person or amount of communication, are
amenable to this real-time processing (Cooke & Gorman,
2009). Research is needed on the association between
communication dynamics and effective team process and
performance in various contexts. Ultimately the discovery of
communication dynamics signatures linked to specific process
and performance across contexts would allow for impactful
and timely interventions.
For example, recent studies in my lab of human-
synthetic teammate teams in the unmanned aerial system
context suggest that coordination can be impacted by a single
team member who knows the ideal communication push and
pull. Future team training may benefit from synthetic team-
mates that are able to serve the role of coordination coach.
Synchronization of Autonomic Arousal in Dyads and
Stephen J. Guastello
Marquette University
Physiological synchronization of autonomic arousal
between people is thought to be an important component of
work team coordination and other interpersonal dynamics. The
group dynamics, in turn, contribute to workload and fatigue
effects at the group level in addition to the individually-
defined work assignments.
The minimum requirements for two living or non-
living entities to synchronize are two oscillators, a feedback
loop between them, and a control parameter that speeds up the
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015 236
process (Strogatz, 2003). When speed reaches a critical level,
a phase shift occurs such that the system goes into phase-lock.
The first challenges for operationalizing these principles to
human systems include identifying the nature of the
oscillators, the feedback loop, and the control parameter. In
the prototype, the feedback is reciprocal between the two
oscillating entities. In human systems such as interactions in
conversations, both one-way and two-way influences are
possible. The coupling is generally loose and moderated by
the empathy levels of the two parties (Guastello, Pincus &
Gunderson, 2006; Marci, Ham, Moran, & Orr, 2007).
Analytic challenges include: (a) determining the
statistical time series models that capture the dynamics of the
teams’ interaction patterns, (b) finding the optimal lag length
between observations for structuring the models, and (c) con-
necting synchronization parameters to performance variables,
subjective ratings of workload and other group dynamics, and
individual differences of the contributing group members.
Equation 1, for instance, is capable of registering self-
organizing and chaotic processes, and it identified the syn-
chronization links in the sample of dyads more often and with
greater accuracy compared to a linear alternative. In Eq. 1
= A exp(Bz
) + exp(CP
) (1)
z is the normalized behavior (autonomic arousal) of the target
person at two successive points in time, P is the normalized
behavior of the partner at time 1, and A, B, and C are nonlinear
regression weights (Guastello et al., 2006).
Lag length denotes how much real time is required to
elapse between the two measurements in order to observe the
coupling effect. A measurement at time 2 is a function of
itself at time 1 and a coupling effect from another source also
at time 1. In a vigilance dual task experiment, 73 undergrad-
uates worked in pairs for 90 min while galvanic skin responses
(GSR) were recorded. Event rates on the vigilance task either
increased or decreased without warning during the work
period. Results based on two criteria supported a lag value of
20 sec (Guastello, Reiter & Malon, in press).
The properties of the linear and nonlinear (Eq.1)
autoregressive models, with and without a synchronization
component were examined. All models were more accurate at
a lag of 20 sec compared to customized lag lengths. Although
the linear models were more accurate overall by a margin of 4-
13% of variance accounted for, the nonlinear synchronization
parameters were more often related to psychological variables
and performance. (Guastello, in press). Importantly, greater
synchronization was observed with the nonlinear model when
the target event rate increased, compared to when it decreased,
which was expected from the general theory of synchroniza-
tion. Equation 1 was also more effective for uncovering
inhibitory or dampening relationships between the co-workers
as well as mutually excitatory relationships.
The latest study on this theme involves teams of four
people who play an emergency response (ER) game against a
single opponent, all with GSR recordings. An example data
stream appears in Figure 1. The participants appear to be in
phase lock. The generalizability of this result remains to be
determined. The adaptive value of high levels of synchrony
has also been questioned (Stevens, Galloway, & Lamb, 2014).
Figure 1. GSR readings for five people in an ER game.
As with many games, the ER team and opponent take
turns. It now appears that the optimal lag length is much
shorter for this task. The ER team members are synchronized
with each other and also with the opponent thus producing the
cluster of linkages shown in Figure 2.
Figure 2. Linkages between ER team members and opponent.
Future research should dissect the experimental tasks
to identify the primary oscillators, feedback loops, possible
control parameters, and conditions that induce higher levels of
synchronization and involve different types of internal group
coordination. Analyses of biometrics need to go beyond dis-
crete event-related potentials, which are commonly used, to
focus instead on continuous streams of data and the analysis of
dynamics therein. The dynamics can then be related to qualita-
tive variables of interest, such as coordination behavior,
communication events, workload manipulations, and ratings
of group processes, thereby building a comprehensive bio-
psychosocial model and generalizable theory.
Collaborative Assessments and Data Analysis
Alina A. von Davier
Educational Testing Service
Educational measurement is undergoing dramatic
change at all levels, with new directions in assessment of
individuals and groups. Some of the most innovative and
exciting features involve conversation-based, technology-
enhanced learning and assessment tools in the areas of
collaborative assessment, game-based learning, and
simulation-based training. Among the advantages of these
approaches is that they support learning of cognitive, social
and affective skills within a common framework and allow for
a detailed collection of the process data in addition to the usual
outcome data in structured log files. Collaborative assessments
in game-like environments integrate many of these different
approaches and tools (Liu, von Davier, Hao, Kyllonen,
Zapata-Rivera, 2014).
Collaboration is one of the skills identified as the
“21st-century skills” and it receives attention among stake-
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015 237
holders in both higher education and the workplace. The
OECD (OECD, 2013) included a test of collaborative problem
solving skills in its PISA 2015 survey of critical skills.
Collaborative assessment is also being promoted by a global
initiative called Assessment and Teaching of 21st Century
Skills, a partnership among Cisco, Intel, Microsoft and the
University of Melbourne to prepare students to live and work
in information-age societies.
The questions for the educational specialists revolve
around the measurement issues: how can we measure
accurately individual contributions to team success? Shall
collaborative tests be domain specific or are the collaborative
problem solving skills transferable from one domain to
another? How can we integrate the data from the dynamic
process of collaboration and the outcome test data and build
valid measurement models?
Collaborative interactions in computerized
educational environments produce data of extraordinarily high
dimensionality (often containing more variables than people
for whom those variables are measured). Extracting key
features from the noise in such data is crucial not only to make
analysis computationally tractable (Masip, Minguillon, &
Mor, 2011), but also to extract relevant features of student
performance from the noise surrounding them (Kim et al.,
2008). Nowadays, with the technological advantages of
systems for recording, capturing, and recognition (e.g., Kinect
for Windows) of multimodal data, the data from collaborative
interactions contain discourse, actions, gestures, tone, body
language that result in a deluge of data (See Figure 3).
To these types of data we can further add the neurodata
collected with (portable) EEG headsets. One way to attempt to
find patterns among these different types of data is to make
use of data mining techniques.
Data mining does not have a long history in educa-
tion or psychology because, until recently, educational and
psychological data were not often of high enough dimension-
ality to require such techniques. However, these techniques
have been used for decades in fields where data with high
dimensionality has long been the norm, such as finance,
marketing, medicine, astronomy, physics, chemistry, and
computer science (Frawley, Piateski-Shapiro, & Matheus,
1992). The purpose of data mining techniques is to reduce the
dimensionality of the dataset something more manageable
(Hand, Mannila, & Smyth, 2001) by extracting implicit,
interesting, and interpretable patterns (Frawley et al, 1992) in
order to allow research questions to be addressed that would
not otherwise be feasible (Romero et al., 2011). Data mining
methods as those developed by Kerr (in press) and Kerr and
Chung (2012) can be applied to identify patterns of
interactions and strategies of success in collaborative problem
solving tasks.
In one of the recent pilot applications at Educational
Testing Service (ETS), we analyzed the behavioral
convergence of test takers in dyads in a science collaborative
problem solving assessment task, the Tetralogue (Luna-
Bazldua, Khan, von Davier, Hao, Liu, Wang, 2015) – see
Figure 3. In order to study evidence of behavioral
convergence, features from log files and video data of 24
study participants were represented as a multi-dimensional
behavioral feature vector composed of cognitive behaviors
(such as the number of messages among the dyad members,
the number of requests for help from the system) and non-
cognitive behaviors (such as engagement, hand-on-face,
anxiety, curiosity, anger, joy, contempt and surprise).
Figure 3. Multimodal data capture including video and action log
files while participants solve a problem collaboratively (using the
ETS’ Tetralogue platform).
A hierarchical cluster analysis was performed on an
Euclidean distance matrix (i.e., a similarity matrix) computed
from the multidimensional behavioral feature data of the study
participants. The cluster analysis revealed that members in the
same dyad tended to group together from the beginning of the
clustering process (i.e., they will be closer to each other in the
feature space than to others). We believe this observed pattern
of agglomeration of the dyad partners could be interpreted as
evidence of convergence of cognitive and non-cognitive states
when people interact in a collaborative task.
Other methods can be considered on these rich data
that exhibit time dependence at the individual level are multi-
variate stochastic processes, time processes, and dynamic
models. Some of these models can accommodate analyzing
the interactions among multiple team members (von Davier &
Halpin, 2013; Halpin & von Davier, 2013); other models are
more appropriate for analyzing the states of collaborative
events (Soller & Stevens, 2007).
Cooke, N. J. & Gorman, J. C. (2009). Interaction-Based
Measures of Cognitive Systems. Journal of Cognitive
Engineering and Decision Making: Special Section on:
Integrating Cognitive Engineering in the Systems
Engineering Process: Opportunities, Challenges and
Emerging Approaches 3, 27-46.
Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J.L.
(2013). Interactive Team Cognition, Cognitive Science,
37, 255-285, DOI: 10.1111/cogs.12009.
Dunbar, T. A. & Gorman, J. C. (2014). Fractal effects of task
constraints in the self-organization of team
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015 238
communication. Talk presented at the Human Factors
and Ergonomics Society 58
Annual Meeting.
Frawley, W. J., Piateski-Shapiro, G., & Matheus, C. J. (1992).
Knowledge discovery in databases: An overview. AI
Magazine, 13(3), 57–70.
Gorman, J. C. (2014). Team coordination and dynamics: Two
central issues. Current Directions in Psychological
Science, 23, 355-360.
Gorman, J. C. & Crites, M. J. (2014). Learning to tie well
with others: Bimanual versus intermanual performance of
a highly practised skill. Ergonomics. Online version
published ahead of print.
Guastello, S. J. (in press). Physiological synchronization in a
vigilance dual task. Nonlinear Dynamics, Psychology,
and Life Sciences.
Guastello, S. J., Pincus, D., & Gunderson, P. R. (2006).
Electrodermal arousal between participants in a
conversation: Nonlinear dynamics for linkage effects.
Nonlinear Dynamics, Psychology, and Life Sciences, 10,
Guastello, S. J., Reiter, K., & Malon, M. (in press). Estimating
appropriate lag length for synchronized physiological
time series: The electrodermal response. Nonlinear
Dynamics, Psychology, and Life Sciences.
Halpin, P. F., & von Davier, A. A. (2013). Evaluating the
roles of individual members in team interactions.
Presented at the National Council on Measurement in
Education, San Francisco, CA.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of data
mining. Cambridge, Massachusetts: The MIT Press.
Kerr, D. (in press). Using data mining results to improve
educational video game design. Journal of Educational
Data Mining.
Kerr, D., & Chung, G. K. W. K. (2012). Identifying key
features of student performance in educational video
games and simulations through cluster analysis. Journal
of Educational Data Mining, 4, 144–182.
Likens, A., Amazeen, P., Stevens, R., Galloway, T., Gorman,
J. (2014). Multifractal scaling reveals the nested nature of
team performance. Social Neuroscience. Vol. 9(3), pp.
Liu, L., Von Davier, A.A., Hao, J., Kyllonen, P. & Rivera-
Zapata, D. (2014). A tough nut to crack: Measuring
collaborative problem solving. Paper presented at the
National Council in Measurement in Education Annual
Meeting, Philadelphia.
Luna-Bazldua, D.A., Khan, S., von Davier, A.A., Hao, J., Liu,
L., & Wang, Z. (2015). On Convergence of Cognitive and
Noncognitive Behavior in Collaborative Activity.
Accepted for Proceedings of the 8
Conference on Educational Data Mining, Madrid, Spain
OECD (2013). PISA 2015 Collaborative Problem Solving
Framework. OECD Publishing.
20Problem%20Solving%20Framework%20.pdf. Retrieve
Dec, 18, 2014.
Marci, C. D., Ham, J., Moran, E., & Orr, S. P. (2007).
Physiologic correlates of perceived therapist empathy and
social-emotional process during psychotherapy. Journal
of Nervous and Mental Disease, 195, 103-111.
Masip, D., Minguillon, J., & Mor, E. (2011). Capturing and
analyzing student behavior in a virtual learning
environment. In C. Romero, S. Ventura, M. Pechenizkiy,
& R. S. J. d. Baker (Eds.), Handbook of educational data
mining (pp. 339–351). Boca Raton, FL: CRC Press.
Romero, C., Gonzalez, P., Ventura, S., del Jesus, M. J., &
Herrera, F. (2009). Evolutionary algorithms for subgroup
discovery in e-learning: A practical application using
Moodle data. Expert Systems with Applications, 39, 1632–
Soller, A. & Stevens, R. (2007). Applications of stochastic
analyses for collaborative learning and cognitive
assessment. Alexandria: Report for the Institute for
Defense Analysis.
Stevens, R. H. & Galloway, T., (2014). Toward a quantitative
description of the neurodynamics organizations of teams.
Social Neuroscience 9:2, 160-173.
Stevens, R.H., Gorman, J.C., Amazeen, P., Likens, A., &
Galloway, T. (2013). The organizational neurodynamics
of teams. Nonlinear Dynamics, Psychology and Life
Sciences, 17, No. 1, pp. 67-86.
Stevens, R. H., Galloway, T., & Lamb, C. (2014). Submarine
navigation team resilience: Linking EEG and behavioral
models. Proceedings of the Human Factors and
Ergonomics Society, 58, 245-249.
Strogatz, S. (2003). Sync: The emerging science of
spontaneous order. New York: Hyperion.
Von Davier, A. A., & Halpin, P. F. (2013). Collaborative
problem solving and the assessment of cognitive skills:
Psychometric considerations (No. ETS RR-13-41).
Princeton, NJ: Educational Testing Service.
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015 239
... Synchrony often reflects team coordination when a situation requires physical movements that are exactly timed, but it can also occur spontaneously when mutual mimicry of behavior is not a performance objective (Delaherche et al., 2012;Guastello, Pincus, & Gunderson, 2006;Sulis, 2016). Studies of autonomic synchrony and other neurodynamic patterns are part of a broader movement to gain an integrated understanding of connections between behavioral outcomes and collective neurocognitive activity (Gorman et al., 2020;Kazi et al., 2021;Salas, Stevens et al., 2015). Synchrony in autonomic arousal levels occurs when two or more people engage a joint activity or conversation through a combination of emotional contagion (Hatfield, Cacioppo, & Rapson, 1993;Hatfield, Rapson, & Le, 2009), empathy, common focus of attention, (Guastello, Mirabito, & Peressini, 2020;Palumbo et al., 2017), or temporal regularities in the activity or environment (Guastello, Marra, Castro, Equi, & Peressini, 2017;Henning, Boucsein, & Gil, 2001;Torrents, Balagué, & Hristovski, 2016). ...
... Teachers may also find it difficult to assess in detail the effectiveness of embodied teamwork as critical events may happen concurrently at different locations. Students may also find it difficult to reflect on their own team dynamics given that they can only observe the events happening around themselves [49]. ...
Conference Paper
Full-text available
Embodied team learning is a form of group learning that occurs in co-located settings where students need to interact with others while actively using resources in the physical learning space to achieve a common goal. In such situations, communication dynamics can be complex as team discourse segments can happen in parallel at different locations of the physical space with varied team member configurations. This can make it hard for teachers to assess the effectiveness of teamwork and for students to reflect on their own experiences. To address this problem, we propose METS (Multimodal Embodied Teamwork Signature), a method to model team dialogue content in combination with spatial and temporal data to generate a signature of embodied teamwork. We present a study in the context of a highly dynamic healthcare team simulation space where students can freely move. We illustrate how signatures of embodied teamwork can help to identify key differences between high and low performing teams: i) across the whole learning session; ii) at different phases of learning sessions; and iii) at particular spaces of interest in the learning space.
... A growing body of contemporary research has aimed at modelling salient aspects of teamwork using sensor-based multimodal innovations in learning contexts [5] and for research purposes [7], [8]. Other works have proposed conceptual frameworks, such as the Input-Process-Outputs (IPO), to align teamwork salient aspects with multimodal and sensor data [9]. ...
Full-text available
Collocated teamwork remains a pervasive practice across all professional sectors. Even though live observations and video analysis have been utilized for understanding embodied interaction of team members, these approaches are impractical for scaling up the provision of feedback that can promote developing high-performance teamwork skills. Enriching spaces with sensors capable of automatically capturing team activity data can improve learning and reflection. Yet, connecting the enormous amounts of data such sensors can generate with constructs related to teamwork remains challenging. This article presents a framework to support the development of human-centered embodied teamwork analytics by 1) enabling hybrid human–machine multimodal sensing; 2) embedding educators’ and experts’ knowledge into computational team models; and 3) generating human-driven data storytelling interfaces for reflection and decision making. This is illustrated through an in-the-wild study in the context of healthcare simulation, where predictive modeling, epistemic network analysis, and data storytelling are used to support educators and nursing teams.
... Cluster analysis is a data reduction method that aids in the analysis of large data sets (Namey et al., 2008). This method is used to group together homogenous participants or groups within a sample and is especially useful for data mining or exploratory research with a multitude of variables (Salas et al., 2015). Cluster analysis can utilize both bottom-up or top-down approaches. ...
Substantially advancing the study of teams will require a new research paradigm complete with methods capable of capturing the complex, dynamic process of teamwork. In this paper, we suggest studying teams with an integrated mixed methods approach (i.e., methods defined by an interconnected mix of quantitative and qualitative characteristics) can help address current methodological shortcomings of our science by promoting sufficiently contextualized research. Through a review of methods, we highlight exemplars of integrated mixed methods that have the potential to be more widely adopted; namely, interaction analysis, content analysis, cluster analysis, state space grids, and agent-based modeling.
... Where along the biometric time scale of team training (i.e., 10 −3 to over 10 5 s) (Salas et al., 2015) would differences be expected? The widespread use of simulations in healthcare would argue against major differences being seen between behavioral and biometric measures as these would have likely already been incorporated into simulation developments. ...
Full-text available
The initial models of team and team member dynamics using biometric data in healthcare will likely come from simulations. But how confident are we that the simulation-derived high-resolution dynamics will reflect those of teams working with live patients? We have developed neurodynamic models of a neurosurgery team while they performed a peroneal nerve decompression surgery on a patient to approach this question. The models were constructed from EEG-derived measures that provided second-by-second estimates of the neurodynamic responses of the team and team members to task uncertainty. The anesthesiologist and two neurosurgeons developed peaks, often coordinated, of elevated neurodynamic organization during the patient preparation and surgery which were similar to those seen during simulation training, and which occurred near important episodes of the patient preparation and surgery. As the analyses moved down the neurodynamic hierarchy, and the simulation and live patient neurodynamics occurring during the intubation procedure were compared at progressively smaller time scales, differences emerged across scalp locations and EEG frequencies. The most significant was the pronounced suppression of gamma rhythms detected by the frontal scalp sensors during the live patient intubation which was absent in simulation trials of the intubation procedure. These results indicate that while profiles of the second-by-second neurodynamics of teams were similar in both the simulation and live patient environments, a deeper analysis revealed differences in the EEG frequencies and scalp locations of the signals responsible for those team dynamics. As measures of individual and team performance become more micro-scale and dynamic, and simulations become extended into virtual environments, these results argue for the need for parallel studies in live environments to validate the dynamics of cognition being observed.
In embodied team learning activities, students are expected to learn to collaborate with others while freely moving in a physical learning space to complete a shared goal. Students can thus interact in various team configurations, resulting in increased complexity in their communication dynamics since unrelated dialogue segments can concurrently happen at different locations of the learning space. This can make it difficult to analyse students’ team dialogue solely using audio data. To address this problem, we present a study in a highly dynamic healthcare simulation setting to illustrate how spatial data can be combined with audio data to model embodied team communication. We used ordered network analysis (ONA) to model the co-occurrence and the order of coded co-located dialogue instances and identify key differences in the communication dynamics of high and low performing teams.KeywordsCollaborative LearningMultimodalityCommunication
Objective: This study evaluated the causal relationships among situation awareness (SA), cohesion, and autonomic synchrony (SE) within teams. SA is often a team effort and should be more accurate in better-functioning teams. Background: Cohesive teams perform better overall, although the relationship appears reciprocal; the relationship to SA has not been considered previously. SE is a collective neurocognitive activity that has been connected to team coordination, communication, and performance in some circumstances. Method: In this experiment, 71 undergraduates, organized into 16 teams, played two matches of a first-person shooter computer game and completed self-report measures of cohesion and SA. SE was determined through time series analysis of electrodermal responses using the driver-empath framework. Results: Empaths and those who came from more synchronized teams reported less cohesion in the team. Granger causality regression showed reciprocal relations among SA, SE, and cohesion that were both positive and negative after controlling for match difficulty. Conclusion: The cohesion-SA relationship is similar to the reciprocal cohesion-performance relationship. SE plays an important and independent role in both the social and cognitive aspects of team behavior. It is possible, furthermore, that individuals who are more attuned to their co-workers reported a more accurate, and less obliging, social situation. Application: Results are applicable to situations requiring teamwork in a dynamic environment.
Team neurodynamics is the study of the changing rhythms and organizations of teams from the perspective of neurophysiology. As a discipline, team neurodynamics is located at the intersection of collaborative learning, psychometrics, complexity theory, and neurobiology with the resulting principles and applications both drawing from and contributing to these specialties. This article describes the tools for studying team neurodynamics and illustrates the potential and the challenges these methods and models have for better understanding healthcare team training and performance. The fundamental metric is neurodynamic organization, which is the tendency of teams and its members to enter into prolonged metastable relationships when they experience and resolve uncertainty. The patterns of these relationships are resolved by symbolic modeling of electroencephalographic (EEG) power levels of the team members, and the information in these patterns are calculated using information theory tools. The topics discussed in this chapter anticipate the time when dynamic biometric data can contribute to our understanding of how to rapidly determine a team’s functional status, and how to use this information to optimize outcomes and training. The rapid, dynamic, and task neutral measures make the lessons learned in healthcare applicable to other complex group and team environments, and provide a foundation for incorporating these models into machines to support the training and performance of teams.
Conference Paper
The primary goal of this experiment is to examine and build dynamical systems models which specify the optimum coordination area that enables teams to perform better when they collaborate with an autonomous agent in the context of explicable behavior. In this preliminary study, we examine team coordination dynamics and explicable behavior by using Joint Recurrence Plot (RP) and Joint Recurrence Quantification Analysis (JRQA). In our example, visualizations of the interaction patterns show when explicable behavior happened, notably, during unexpected events, e.g., when there was a missing LEGO brick. Our preliminary data provides some initial findings about team interaction under dynamical changes along with content under uncertainty. Current and future work is focused on additional experimentation with three types of team configurations: all-human, human-agent, and human-multiagent. Through more experimentation, additional insights and examples of other unexpected events will be able to highlight any necessary additional requirements needed for effective teamwork.
This chapter presents factors affecting group work and group dynamics in PBL. Factors are grouped in three categories: resources (group size, individual differences and diversity), the learning task and group learning processes (group climate and teaching learning behaviors), and the learning context (discipline, culture, socialization and training, the role of the tutor or facilitator). The chapter concludes with an overview of structural losses (lack of elaboration, lack of cognitive diversity, unproductive brainstorm) and interpersonal losses (poor collaboration, social categorization, poor adjustment to PBL) and possible remedies.
Full-text available
When people coordinate as a team, they accomplish more than they would working alone. These team-coordination effects give new meaning to Aristotle’s famous phrase, “the whole is greater than the sum of its parts.” In this article, I consider two central issues confronting team-coordination research: Do the causes of team coordination reside within individual minds or between them, and at what levels of analysis (e.g., physiological, cognitive) do team-coordination effects occur? These issues are viewed in light of specific lines of coordination research, and some features of a general theory of team coordination are offered.
Full-text available
Studies indicate that novices are faster in manual tasks when performing with a partner ('intermanual') than with their own two hands ('bimanual'). The generality of this 'mode effect' was examined using a highly practised bimanual task, shoe tying, at which participants were experts. Speed-variability correlations confirmed participants were bimanually skilled but not intermanually skilled. Contrary to results using novices, intermanual was slower, such that prior skill reverses the effect. Analyses incorporating the similarity of shoe-tying strategies across dyads implicated a perceptual rather than shared knowledge/representation basis for intermanual performance. Practice effects indicated that intermanual performance built upon prior bimanual skill, such that novel relative timings between dyads' hands must be acquired. Motor transfer effects provided support for this conclusion. During shoe tying, hands were tightly coupled in the intermanual mode due to the perceptual coupling constraints of intermanual performance. Increased coupling was correlated with slower performance. Implications for real-world tasks (e.g. surgical knot tying) are described.
Full-text available
The quality of a team depends on its ability to deliver information through a hierarchy of team members and negotiate processes spanning different time scales. That structure and the behavior that results from it pose problems for researchers because multiply-nested interactions are not easily separated. We explored the behavior of a six-person team engaged in a Submarine Piloting and Navigation (SPAN) task using the tools of dynamical systems. The data were a single entropy time series that showed the distribution of activity across six team members, as recorded by nine-channel electroencephalography (EEG). A single team's data were analyzed for the purposes of illustrating the utility of multifractal analysis and allowing for in-depth exploratory analysis of temporal characteristics. Could the meaningful events experienced by one of these teams be captured using multifractal analysis, a dynamical systems tool that is specifically designed to extract patterns across levels of analysis? Results indicate that nested patterns of team activity can be identified from neural data streams, including both routine and novel events. The novelty of this tool is the ability to identify social patterns from the brain activity of individuals in the social interaction. Implications for application and future directions of this research are discussed.
The purpose of our project is to explore the measurement of cognitive skills in the domain of science through collaborative problem solving tasks, measure the collaborative skills, and gauge the potential feasibility of using game-like environments with avatar representation for the purposes of assessing the relevant skills. We are comparing students' performance in two conditions. In one condition, students work individually with two virtual agents in a game-like task. In the second condition, dyads of students work collaboratively with two virtual agents in the similar game-like task through a chat box. Our research is motivated by the distributed nature of cognition, extant research on computer-supported collaborative learning (CSCL) which has shown great value of collaborative activities for learning, and the framework for the Programme for International Student Assessment (PISA) framework. This chapter focuses on the development and implementation of a conceptual model to measure individuals' cognitive and social skills through collaborative activities.
The synchronization of autonomic arousal levels and other physiological responses between people is a potentially important component of work team performance, client-therapist relationships, and other types of human interaction. This study addressed several problems: What statistical models are viable for identijiing synchronization for loosely coupled human systems? How is the level of synchronization related to psychosocial variables such as empathy, subjective ratings of workload, and actual performance? Participants were 70 undergraduates who worked in pairs on a vigilance dual task in which they watched a virtual reality security camera, rang a bell when they saw the target intruder, and completed a jig-saw puzzle. Event rates either increased or decreased during the 90 mm work period. The average R-2 values for each person were .66, .66, .62, and .53 for the linear autoregressive model, linear autoregressive model with a synchronization component, the nonlinear autoregressive model, and the nonlinear autoregressive model with a synchronization component, respectively. All models were more accurate at a lag of 20 sec compared to 50 sec or customized lag lengths. Although the linear models were more accurate overall, the nonlinear synchronization parameters were more often related to psychological variables and performance. In particular, greater synchronization was observed with the nonlinear model when the target event rate increased, compared to when it decreased, which was expected from the general theory of synchronization. Nonlinear models were also more effective for uncovering inhibitory or dampening relationships between the co-workers as well as mutually excitatory relationships. Future research should explore the comparative model results for tasks that induce higher levels of synchronization and involve different types of internal group coordination.
Physiological synchronization of autonomic arousal between people is thought to be an important component of work team dynamics, therapist-client relationships, and other interpersonal dynamics. This article examines concepts and mathematical models of synchronization that could be relevant to work teams. Before it is possible to deploy nonlinear modeling, however, it is necessary to develop a strategy for determining appropriate lag lengths. If a measurement at time 2 is a function of itself at time 1 and a coupling effect from another source, what is the appropriate amount of real time that should be allowed to elapse between the two measurements in order to observe the coupling effect? This study examined four strategies for doing so. In the experiment, 78 undergraduates worked in pairs to perform a vigilance dual task for 90 min while galvanic skin responses (GSR) were recorded. Lags based on mutual entropy and the natural rate criteria produced corroborating results, whereas strategies based on a critical decline in the linear autocorrelation (max r/e) and Theiler's W did not produce usable results for this situation. Some connections were uncovered between linear autocorrelation strength and lag based on mutual entropy with performance on the tasks and subjective ratings of workload.
Advances in the assessment of submarine piloting and navigation teams have created opportunities for linking behavioral observations of team performances with neurodynamic measures of team organization, synchrony and change. Submarine navigation teams (n=12) were fitted with EEG headsets and recorded while conducting required navigation simulations. In parallel, their performances were assessed for team resilience by two evaluators using a team process rubric adopted by the Submarine Force. EEG models of team synchrony were created symbolically which identified times when there was increased across-team cognitive organization induced by the simulation and / or interactions with other crew members. One set of these organizations was observed in the 10 Hz EEG frequency band and coincided with the periodic activity of updating the ship’s position (e.g. Rounds). There were also periods of increased team synchrony between 25-40 Hz which were present during some Rounds events but were more prominent with task changes or when the team was stressed. More resilient teams had fewer periods of team synchrony and these were of smaller magnitude than those found in less resilient teams. These results indicate that both routine and unexpected activities trigger increased neurophysiologic synchrony / coherence in teams and that periods of persistent synchrony may signal a team being challenged.
Collaboration is generally recognized as a core competency of today's knowledge economy and has taken a central role in recent theoretical and technological developments in education research. Yet, the methodology for assessing the learning benefits of collaboration continues to rely on educational tests designed for isolated individuals. Thus, what counts as evidence of learning does not correspond to current best practices for teaching, and it does not reflect what students are ultimately expected to be able to do with their knowledge. The goals of this paper are to give an overview of the research conducted in several fields of work related to collaboration, propose a framework for the assessment of cognitive skills (such as science or math) through collaborative problem-solving tasks, and propose several statistical approaches to model the data collected from collaborative interactions. This research contributes to the knowledge needed to support a new generation of assessments based on collaboration.