ArticlePublisher preview available

Toward Automated Computer-Based Visualization and Assessment of Team-Based Performance

American Psychological Association
Journal of Educational Psychology
Authors:
Dirk Ifenthaler at University of Mannheim
Dirk Ifenthaler
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the author.

Abstract

A considerable amount of research has been undertaken to provide insights into the valid assessment of team performance. However, in many settings, manual and therefore labor-intensive assessment instruments for team performance have limitations. Therefore, automated assessment instruments enable more flexible and detailed insights into the complex processes influencing team performance. The central objective of this study was to advance knowledge in automated assessment of team-based performance using a language-oriented approach. Fifty-six teams of learners (N = 224) in 3 experimental conditions solved 2 tasks in an online learning environment. They were analyzed with the Automated Knowledge Visualization and Assessment (AKOVIA) methodology. AKOVIA integrates a natural language-oriented algorithm and enables a structural and semantic compression of individual-and team-based knowledge representations. Findings indicate initial evidence of the feasibility and validity of the fully automated methodology. A framework for integrating research and methodology development is suggested for improving educational technology innovations such as computer-based assessment environments in international large-scale assessments.
A preview of this full-text is provided by American Psychological Association.
Learn more
Content available from Journal of Educational Psychology 
This content is subject to copyright. Terms and conditions apply.
Toward Automated Computer-Based Visualization and Assessment of
Team-Based Performance
Dirk Ifenthaler
Deakin University
A considerable amount of research has been undertaken to provide insights into the valid assessment of
team performance. However, in many settings, manual and therefore labor-intensive assessment instru-
ments for team performance have limitations. Therefore, automated assessment instruments enable more
flexible and detailed insights into the complex processes influencing team performance. The central
objective of this study was to advance knowledge in automated assessment of team-based performance
using a language-oriented approach. Fifty-six teams of learners (N⫽224) in 3 experimental conditions
solved 2 tasks in an online learning environment. They were analyzed with the Automated Knowledge
Visualization and Assessment (AKOVIA) methodology. AKOVIA integrates a natural language-oriented
algorithm and enables a structural and semantic compression of individual- and team-based knowledge
representations. Findings indicate initial evidence of the feasibility and validity of the fully automated
methodology. A framework for integrating research and methodology development is suggested for
improving educational technology innovations such as computer-based assessment environments in
international large-scale assessments.
Keywords: team, shared mental model, automated assessment, natural language processing
Teams are a critical and essential part of most working envi-
ronments because they combine different views, multiple skills,
diverse experiences, analytical judgments, and rich knowledge.
Consequently, research in teams and their assessment has been a
continuous endeavor in various scientific areas for more than 30
years. Yet, there exist various definitions of team using different
perspectives. For example, Kanaga and Kossler (2011) defined a
team as “a specific kind of group whose members are collectively
accountable for achieving the team’s goals” (p. 4). A more detailed
definition is given by Katzenbach and Smith (2003), who de-
scribed a team as “a small number of people with complementary
skills who are committed to a common purpose, performance
goals, and approach for which they are mutually accountable” (p.
45). From an operational point of view, Cohen, Levesque, and
Smith (1997) defined a team as “a set of agents having a shared
objective and a shared mental state” (p. 95). Salas, Dickinson,
Converse, and Tannenbaum (1992) described a team as
a distinguishable set of two or more people who interact dynamically,
interdependently, and adaptively toward a common and valued goal,
who have each been assigned specific roles or functions to perform
and who have a limited life span of membership. (p. 4)
To sum up, common characteristics of definitions of a team in-
clude at least two individuals, common objectives, shared respon-
sibility and interdependence, as well as optimal performance.
Instruments for measuring team performance have been devel-
oped over the past decades; however, adequate computer-based
assessments of team-based performance are scarce (Fischer &
Mandl, 2005). The recent advancement of web-based technology
allowed widening the scope of computer-based assessments
(Csapó, Ainley, Bennett, Latour, & Law, 2012;Frey & Hartig,
2013). For example, international large-scale assessments such as
the Programme for International Student Assessment (PISA) or the
Programme for the International Assessment of Adult Competen-
cies (PIAAC) currently implement advanced computer-based as-
sessment environments (Organisation for Economic Co-operation
and Development [OECD], 2010,2013).
Previously, most of the team-based assessment instruments re-
quired a great deal of time and effort using highly trained research-
ers (e.g., think-aloud protocol analysis) and were mainly limited to
subjective self-reports (Wildman et al., 2012), and they as also
required labor-intense manual analysis of performance indicators
(Almond, Steinberg, & Mislevy, 2002). As a result, such assess-
ments have been limited to the scientific community and have had
only a minor impact on practical issues such as the design of
effective learning, teaching, and working environments. Motivated
by a desire to have practical assessment instruments that are useful
and valid has led researchers to uncover significant developments
in the last several years (Chung, O’Neil, & Herl, 1999;Mandl &
Fischer, 2000). Especially instruments using graphical representa-
tions for computer-based assessment have been successfully tested
and implemented, such as the DEEP methodology (Spector &
Koszalka, 2004), KU-Mapper (Taricani & Clariana, 2006), and
knowledge mapping tools (Herl, O’Neil, Chung, & Schacter, 1999;
O’Neil, Chuang, & Baker, 2010). However, only a few of these
instruments have been fully automated and tested for reliability
and validity. Furthermore, automated and language-oriented as-
sessment methodologies that enable a domain-independent analy-
This article was published Online First February 17, 2014.
Correspondence concerning this article should be addressed to Dirk
Ifenthaler, Centre for Research in Digital Learning, Deakin University,
Level 4, 550 Bourke Street, Melbourne VIC 3000, Australia. E-mail:
dirk@ifenthaler.info
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Journal of Educational Psychology © 2014 American Psychological Association
2014, Vol. 106, No. 3, 651–665 0022-0663/14/$12.00 DOI: 10.1037/a0035505
651
... Conceptual network analysis (aka knowledge structure analysis) has been used for decades in single document comprehension research (Clariana & Koul, 2004;Ifenthaler, 2014) and recent studies have demonstrated its potential to measure different aspects of multiple document reading (Lehmann et al., 2020;Wei et al., 2024). By converting students' learning products, such as summary essays, into networks with nodes representing the core concepts and edges (i.e., links) representing semantic relationships between them, researchers can better explore the properties of students' mental representations (Jonassen et al., 1993;Kintsch, 1988). ...
... These gaps will likely limit the accuracy of students' semantic statements even when the quantity is adequate (see Fig. 1). Thus, the semantic quality of integration can be measured as the similarity of integrative links between a student's concept network and the expert referent network (Follmer et al., 2018;Ifenthaler, 2014). ...
The influence of pre-reading purpose and extra-textual networks with summary writing on multiple document concept network integration: a replication of Wei et al. (2024)
Article
Full-text available
  • Jun 2025
  • ETR&D-EDUC TECH RES
Theories and practices to enhance multiple document comprehension and integration are crucial in both personal and work contexts, especially with the proliferation of printed and online sources. This experimental investigation replicates and extends (Wei et al., Educational Technology Research and Development 72:661–685, 2024) to examine how multiple documents integration is influenced by reading purpose, summary writing, and extra-textual networks (pre-reading, Study 1, and post-writing, Study 2). In Study 1 (N = 102), participants were randomly assigned to a pre-reading purpose set by a prompt (integrative or detailed) and by a network (an integrative or else an intra-text network) and then read three documents about Alzheimer’s disease to complete a writing task with revision (but no feedback). Three days later, they completed a delayed writing task and an inference verification test. In Study 2 (N = 90), the same procedure was used except that the network was used as feedback after writing to support revision. Results from the two studies agree with the previous research that the quantity and structural quality of integration can be improved by external cues and by delayed repeated writing. This research further confirms an innovative approach for evaluating different aspects of knowledge integration and contributes to the literature from the concept network perspective as a measure and an intervention of multiple-text reading.
View
... Concept network analysis was developed and widely applied in single document comprehension studies for the past two decades (Clariana & Koul, 2004;Ifenthaler, 2014;Kim & McCarthy, 2021) and has the potential to measure different aspects of integration. Although different theories have emphasized various dimensions of comprehension, most theories have assumed a connectionist representational mental architecture, with nodes representing concepts and edges representing the links among the concepts (McNamara & Magliano, 2009). ...
... Recent studies have shown that only a few novices elaborate on integrative statements in their summaries and even then, the extent of integration is limited (Du & List, 2020;List & Du, 2021). Hence, the semantic quality of integration can be measured by comparing the similarity of integrative links between a student's concept network and the expert network (Follmer et al., 2018;Ifenthaler, 2014). ...
The effects of reading prompts and of post-reading generative learning tasks on multiple document integration: evidence from concept network analysis
Article
Full-text available
  • Nov 2023
Learning from multiple documents is an essential ability in today’s society. This experimental study used concept network analysis to consider how reading prompts and post-reading generative learning tasks can alter students’ documents integration performance. Undergraduates (N = 119) read three documents about Alzheimer’s disease with one of two reading prompts (integrative prompts vs. detailed prompts) and then after reading completed a generative learning task (concept mapping vs. summary writing). Three days later they completed a delayed writing task and an inference verification test. Participants’ written texts were converted to concept networks to evaluate conceptual level integration, including the quantity of integration (measured by the proportion of integrative links), the semantic quality of integration (measured by the similarity of integrative links), and the structural quality of integration (measured by comparing network graph centrality). Results showed that the integrative prompts relative to the detailed prompts enhanced the quantity of integration but not the semantic and structural quality. Further, concept mapping relative to summary writing significantly improved the structural quality of integration. In summary, this study describes a new concept network approach for measuring different aspects of integration to advance theory and research in multiple document comprehension.
View
... By using these techniques, assessments were customized to fit the needs and backgrounds of the pupils. In the words of [14], such data sets and analytical procedures support policymakers in establishing the conditions for optimizing processes and outcomes as well as enable the provision of formative assistance and feedback for students and teachers. ...
Perspective Chapter: Innovation and Evolution in Tertiary Education – Assessment and Evaluation
Chapter
Full-text available
  • Oct 2024
The paper focused on innovation and evolution in tertiary education that undergo a significant transformation in assessment and evaluation practices in utilizing adaptive learning technologies into the system. The paper looked into the practice and roles of assessment and evaluation in tertiary education. It discussed the concept of innovative pedagogical approaches and evolution of assessment and evaluation in tertiary education that enhance students’ educational experience by considering their diverse learning preferences, needs, and access to advanced technology for active learning. It highlighted the need for evolution in changing educational systems and students’ needs, and the concept of innovations in assessment methods based on performance-based assessments. The paper investigated simulations and practical demonstrations in assessment methods, and the impact of innovative assessment and evaluation on students’ learning. Lastly, the study recommended that professional development is required to be implemented for the ongoing improvement of academic programs.
View
... A concept network constructed while reading is considered a type of knowledge visualization that represents an individual's conceptual knowledge structure using nodes displaying concepts (Engelmann & Hesse, 2010). Various approaches have been discussed for decades to measure and analyze concept networks (Clariana et al., 2013;Clariana & Koul, 2004;Follmer et al., 2018;Ifenthaler, 2014;Koul et al., 2005;Wei et al., 2022). Well-established methods include multi-decision approaches (e.g., pairwise similarity ratings, or list-wise similarity ratings) (Clariana, 2010b;Clariana & Wallace, 2007), sorting , summary writing (Clariana, 2010a) and concept mapping task (Wei et al., 2023). ...
Listening instead of reading: using network drawing task as a re-constructed method and measure of knowledge in mind
Article
Full-text available
The trend of podcast and audiobook listening is on the rise, thus paving the way for a new, popular learning environment. It is crucial to deliberate on the influence of multiple-text listening comprehension. This research aimed to explore the impact of conceptual network processing on multiple-text comprehension performance from the perspective of long-term working memory (LT-WM). Additionally, it examined the sensitivity of a concept network analysis measure in facilitating the construction of conceptual structures during text listening. To this end, 128 participants were randomly assigned to four different treatment groups. These groups varied in terms of text order (theoretical text followed by applied text or vice versa) and post-listening network drawing tasks. Participants either drew concept maps using key terms specific to each text or irrelevant terms as a control group. After completing both texts, all participants were required to complete a final multi-document network drawing task. Finally, a multiple-choice delayed test was administered three days later. The findings suggest that conceptual networks in LT-WM significantly influence multiple-text comprehension performance. Furthermore, the network drawing task appears to be an effective method for reconstructing knowledge structures in LT-WM, serving as a complementary measure to the MCT in assessing listening comprehension.
View
... The central focus of these strategies also includes ill-structured cases that are similar to the types of problems practitioners face. The complexity of these problems often consists of interconnected variables (latent, salient) and multiple perspectives, so there is rarely a single predetermined solution that satisfies all options (Ifenthaler, 2014). Additionally, these problems are challenging because they include multiple criteria for evaluation (Jonassen, 2011b;Ju & Choi, 2017), which makes it challenging to definitively determine when a 'right' answer has been achieved. ...
Decision Making and Problem-Solving: Implications for Learning Design
Article
  • Jan 2022
Educators are increasingly applying problem-solving through instructional strategies, such as inquiry-based learning. An important aspect of problem-solving includes the decision-making process and the rationale for learners’ choices. Although prior theories and models indeed yield important insight in other areas of problem-solving (e.g. - scaffolding, argumentation, reflection), the decision-making process has only been implicitly referenced within learning design. To better understand the role of decision-making and apply it towards design, the article reviews the theoretical basis of the following overarching frameworks: normative, descriptive, prescriptive, and case-based decision-making theory (CBDT). For each approach, an example is included that instantiates the theory within learning design. The article concludes with a discussion of how decision-making theory aligns with existing theories that are foundational to problem-solving, along with implications for future learning design.
View
... The central focus of these strategies also includes ill-structured cases that are similar to the types of problems practitioners face. The complexity of these problems often consists of interconnected variables (latent, salient) and multiple perspectives, so there is rarely a single predetermined solution that satisfies all options (Ifenthaler, 2014). Additionally, these problems are challenging because they include multiple criteria for evaluation (Jonassen, 2011b;Ju & Choi, 2017), which makes it challenging to definitively determine when a 'right' answer has been achieved. ...
View
... Virtual team learning introduced into the workplace and higher education institutes has a significant development in terms of asynchronous and synchronous methods (e.g., email, Wiki, forum, and social media) which are applied to team communication, taskwork, and teamwork. Task-and project-work focusses on the completion of a specific common goal by choosing team action methods or strategies, taking action, and getting feedback [3]. Teamwork emphasizes the role of team members during team activities [4]. ...
The Adoption of Intelligent and Virtual Teams in Online Entrepreneurship Education Courses
Conference Paper
Full-text available
  • Jul 2021
Educators adopt intelligent and virtual teams or elements of teams to online learning as well as teaching, especially in entrepreneurship education courses. This study aims to analyze the application and utilization of intelligent and virtual teams in online entrepreneurship education. As the limited open entrepreneurship courses provide the opportunities of virtual team activities, this study considers courses that adopt team elements as well. There were ten courses collected from massive open online course platforms and social media groups. According to the components of a virtual team, this research focused on communication, teamwork, taskwork, and use of technology. The communication of online courses applies multiply methods, e.g., social media and forums. Educators provide team tasks for online participants. With more communication and completing tasks together, distributed teammates have a closer relationship. The virtual teams are facilitated by technology. Accordingly, the provision of cooperative and collaborative opportunities can improve learning motivation which is key for learning effectiveness. However, the application of intelligent and virtual teams is still rare which may also be reflected in low attendance for online entrepreneurship courses. Keywords: adoption, intelligent and virtual team, entrepreneurship education, online entrepreneurship courses
View
... Assessment possibilities depend upon how team participation is captured, and what emphasis is placed on the required task in grading rubrics. Possible assessment includes instructor feedback of individual student contributions, instructor feedback on synchronous activities of a team captured via audio or video (Ifenthaler, 2014), written reasoning of team specific choices (Weinberger & Fischer, 2006), individual or team evaluations of other teams' specific choices, and peer evaluation of individual and team participation. With each of these opportunities, individual and team accountability becomes more robust, as does student participation (Thomas & Thorpe, 2019;Phirangee et al., 2016), and instructional oversight becomes more involved, F I G U R E 5 Collaborative learning technology support. ...
Application exercise design for team‐based learning in online courses
Article
Full-text available
  • Mar 2021
This chapter describes best practices for adapting traditional, face‐to‐face, team‐based learning principles to develop online application exercise design, support effective facilitation, and use appropriate technology to promote effective online team collaboration. The unique challenges of online TBL applications include maintaining effective team collaboration, discussion facilitation, and simultaneous reporting. A framework is proposed to guide practitioners to make appropriate, systematic choices in the development of online TBL applications.
View
The Effect of Social Closeness on Perceived Satisfaction of Collaborative Learning
Chapter
  • Dec 2022
Effective communication and coordination supported by well-established patterns of interactions are vital to collaborative learning. The quality of social relationships among group members can affect group dynamics and communication, as well as further influence students’ learning experiences and perceived learning outcomes. In this study, we use an ego network approach to measure an individual’s social closeness with group members and examine effects on the individual’s collaborative learning experience. The data was collected through a survey with N = 120 university students. Our findings indicate that social closeness among group members has a positive effect on the perceived satisfaction of collaborative learning, as students with stronger social closeness with group members tend to be more satisfied with their collaborative learning experience. These results provide advanced theoretical insights for understanding the factors associated with the success of collaborative learning. Based on our findings, appropriate activities for strengthening social closeness among group members are suggested. Implications include pedagogical approaches for facilitating students’ collaborative learning experience and outcomes.KeywordsCollaborative learningGroup collaborationSocial closenessTie strengthSocial network analysis
View
Data Mining and Analytics in the Context of Workplace Learning: Benefits and Affordances
Chapter
  • Aug 2022
Technology-based innovations in workplace learning have significantly altered both the scale and resolution of measurements for supporting learning processes in organisations. The increased availability of vast and highly varied amounts of data from settings in the context of workplace learning is overwhelming. This chapter outlines standards in data mining with a specific focus on data from workplace learning. Further, different data mining and analytics methodologies, such as Support Vector Machines or Decision Trees, are presented. An emphasis is shifted to the understanding of learning analytics which are a socio-technical data-mining and analytic practice in educational contexts. The chapter closes with an outlook on how data mining and analytics may provide benefits for future workplace learning scenarios.KeywordsData analyticsEducational data miningLearning analyticsWorkplace learningTechnology-supported learning
View
Self-Efficacy: Toward a Unifying Theory of Behavioral Change
Article
Full-text available
  • Mar 1977
Presents an integrative theoretical framework to explain and to predict psychological changes achieved by different modes of treatment. This theory states that psychological procedures, whatever their form, alter the level and strength of self-efficacy. It is hypothesized that expectations of personal efficacy determine whether coping behavior will be initiated, how much effort will be expended, and how long it will be sustained in the face of obstacles and aversive experiences. Persistence in activities that are subjectively threatening but in fact relatively safe produces, through experiences of mastery, further enhancement of self-efficacy and corresponding reductions in defensive behavior. In the proposed model, expectations of personal efficacy are derived from 4 principal sources of information: performance accomplishments, vicarious experience, verbal persuasion, and physiological states. Factors influencing the cognitive processing of efficacy information arise from enactive, vicarious, exhortative, and emotive sources. The differential power of diverse therapeutic procedures is analyzed in terms of the postulated cognitive mechanism of operation. Findings are reported from microanalyses of enactive, vicarious, and emotive modes of treatment that support the hypothesized relationship between perceived self-efficacy and behavioral changes. (21/2 p ref)
View
View
View
On Sentence Interpretation
Article
  • Jan 1999
At present there exists no empirically-motivated theory of how perceivers assign a grammatically-permissible interpretation to a sentence. Implicit in many investigations of language comprehension is the idea that each constituent of a sentence is interpreted by the perceiver at the earliest conceivable point, using all potentially relevant sources of information. A variety of counter examples are presented to argue against this implicit theory of sentence interpretation. It is argued that an explicit alternative theory is needed to specify which decisions are made at which points during interpretive processing and to spell out the principles governing the processor's preferred choice at points of ambiguity or uncertainty. Several specific issues are taken concerning how the processor assigns a focal structure to an input sentence, how it identifies the topic of the sentence, how implicit restrictors on the domain of quantification are interpreted and how the identification of the content of a restrictor may guide the processor's use of discourse information. Exploiting intuitions about preferred interpretations of ambiguous sentences as well as the results of both old and new experimental studies, a theory of the preferred interpretation of Determiner Phrases is presented. This work explores important, but overlooked questions in on-line sentence interpretation and attempts to erect some of the scaffolding for an eventual theory of sentence interpretation.
View
Perspectives on assessment of educational technologies for informal learning
Chapter
  • Jan 2014
Informal learning is a complex area, yet one that increasingly involves educational technologists in practice and research. Informal learning includes many different settings, contexts and goals, and draws upon many fields. Two fields that have yielded considerable research literature are museum/free-choice learning and informal science education. While some have defined informal learning as nonschool learning, the nature of informal learning includes dimensions such as location, timing, structure, control, pacing, regulation and content, and so informal learning is not necessarily tied to space. This means that informal learning can and does occur frequently in museums, after-school clubs, botanical gardens, zoos, science centers, and community centers, but can also be engaged in at home and also be integrated within formal school or college settings. Informal science education, museum learning and workplace learning are designed for varied outcomes and have different frameworks. Reasons for assessing for informal learning include to provide formative feedback to participants and to provide evaluation data to improve the organization's learning goals. Methods for assessment for informal learning encompass both quantitative and qualitative approaches, including traditional tests and measures, but primarily rely on such methods as surveys, group and individual interviews, observations, artifact and product analysis, and sometimes ethnographic or case studies. Emerging educational technologies, such as social media, Web 2.0 tools, eLearning and online learning, provide fertile ground for continuing developments in informal learning and assessment. © Springer Science+Business Media New York 2014. All rights reserved.
View
View
The relationship between student team learning outcomes and achievement, causal attributions, and affect.
Article
  • Mar 1991
This field investigation examined the relationship between prior achievement (high vs average vs low), individual outcome (success vs failure), team outcome (success vs failure) and students' achievement and academic perceptions. 190 students in 7 elementary school classes learned mathematics for 5 wks with the Teams-Games-Tournaments cooperative learning strategy. Team outcome was significantly related to achievement and academic perceptions and was independent of prior achievement and individual outcome. Members of successful teams attributed their team's performance more to ability and luck than did members of unsuccessful teams, believed themselves to be more successful, more deserving of reward, and happier about their team outcome. Alternatives such as criterion-referenced team goals and between-team cooperation are recommended.
View
View
Shared mental models in expert team decision making
Article
  • Jan 1993
Critical performance in many complex systems depends on the coordinated activity of a team of individuals. Cockpit crews, surgery teams, fire fighting teams, and military teams are all examples of teams who operate in situations where ineffective performance can have ...
View
View