MeetingCoach: An Intelligent Dashboard for Supporting Effective & Inclusive Meetings

Full text

MeetingCoach: An Intelligent Dashboard for Supporting
Eective & Inclusive Meetings
Samiha Samrose Daniel McDu Robert Sim
University of Rochester Microsoft Research Microsoft Research
ssamrose@cs.rochester.edu damcdu@microsoft.com rsim@microsoft.com
Jina Suh Kael Rowan Javier Hernandez
Microsoft Research Microsoft Research Microsoft Research
jinasuh@microsoft.com kael.rowan@microsoft.com javierh@microsoft.com
Sean Rintel Kevin Moynihan Mary Czerwinski
Microsoft Research Microsoft Research Microsoft Research
serintel@microsoft.com kevinmo@microsoft.com marycz@microsoft.com
ABSTRACT
Video-conferencing is essential for many companies, but its limi-
tations in conveying social cues can lead to ineective meetings.
We present MeetingCoach, an intelligent post-meeting feedback
dashboard that summarizes contextual and behavioral meeting
information. Through an exploratory survey (N=120), we identi-
ed important signals (e.g., turn taking, sentiment) and used these
insights to create a wireframe dashboard. The design was eval-
uated with in situ participants (N=16) who helped identify the
components they would prefer in a post-meeting dashboard. After
recording video-conferencing meetings of eight teams over four
weeks, we developed an AI system to quantify the meeting features
and created personalized dashboards for each participant. Through
interviews and surveys (N=23), we found that reviewing the dash-
board helped improve attendees’ awareness of meeting dynamics,
with implications for improved eectiveness and inclusivity. Based
on our ndings, we provide suggestions for future feedback system
designs of video-conferencing meetings.
CCS CONCEPTS
• Human-centered computing → Empirical studies in collab-
orative and social computing.
KEYWORDS
group, feedback, meeting, sensing, video-conferencing
ACM Reference Format:
Samiha Samrose, Daniel McDu, Robert Sim, Jina Suh, Kael Rowan, Javier
Hernandez, Sean Rintel, Kevin Moynihan, and Mary Czerwinski. 2021. Meet-
ingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive
Meetings. In CHI Conference on Human Factors in Computing Systems (CHI
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
’21), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 13 pages.
https://doi.org/10.1145/3411764.3445615
1 INTRODUCTION
Meetings are a primary mode of work [
58
], but many employ-
ees nd them frustrating and even counter-productive when good
meeting practices are lacking or violated [
1
,
46
]. The violations
of general meeting norms and disrespectful behaviors have been
shown to be negatively correlated with meeting eectiveness and
satisfaction [
45
]. In 2020, the “stay at home” orders and travel
restrictions of the COVID-19 pandemic dramatically accelerated
the use of the video-conferencing meetings for work. By March
2020, daily usage of video-conferencing services such as Zoom
and Microsoft Teams had increased by 300% and 775% respectively,
and video-conferencing apps jumped to the top of the Apple app
store.
1
Although video-conferencing has the potential to reduce
the cost and eort behind organizing travel, space, and scheduling
of in-person meetings
2
, the “fractured ecologies” [
37
] of video-
conferencing can aggravate negative outcomes and marginalize
some members of the meeting [
38
]. Video-conferencing has consis-
tently presented communicative challenges [
19
,
52
] and introduced
distractions [
61
] which can lead to ineective and non-inclusive
meetings [
23
,
30
]. A primary goal of remote collaboration tools
should be to support the most eective meetings possible for all
participants. Cutler et al. [
9
] conducted a large scale e-mail survey
on remote meeting eectiveness (N=4,425) at a technology com-
pany (pre-COVID-19) and showed that online meeting eectiveness
correlates with meeting inclusiveness, participation, and comfort
in contributing. They identify a large potential nancial benet
to companies that can achieve these goals, and an opportunity to
establish and maintain a workplace culture in which everyone feels
free to contribute. There are clearly opportunities for technological
solutions that assist attendees in feeling more included and improv-
ing meeting eectiveness by helping them understand their own
and others’ core meeting dynamics.
This paper reports on an exploratory study with in situ work
teams to identify the challenges they face in video-conferencing
CHI ’21, May 8–13, 2021, Yokohama, Japan
© 2021 Association for Computing Machinery. 1
https://www.marketwatch.com/story/zoom-microsoft-cloud-usage-are-
ACM ISBN 978-1-4503-8096-6/21/05.. . $15.00 rocketing-during-coronavirus-pandemic-new-data-show-2020-03-30
https://doi.org/10.1145/3411764.3445615 2https://nancesonline.com/video-web-conferencing-statistics/

CHI ’21, May 8–13, 2021, Yokohama, Japan
based meetings, and proposes a post-meeting feedback system to
address the issues. In particular, we aimed to provide insights on the
following research questions: (1) What aspects of meetings do video-
conferencing attendees need help with?; (2) How can we leverage AI
systems to make video-conferencing meetings more inclusive and
eective?; (3) How should AI-extracted meeting features (including
content, behavioral measurements, and sentiment) be categorized
and visualized in a feedback dashboard?; and (4) What concerns
exist regarding data privacy and accuracy for such systems?
Our work addresses these research questions through a series of
user studies and design iterations. Via an initial exploratory require-
ment analysis survey of 120 information workers, we identied the
communicative signals (e.g., participation, facial sentiment) which
are important in improving meeting eectiveness and inclusion.
We conducted a longitudinal user study to record in situ video-
conferencing meetings from eight teams over a four-week period.
We used the insights from the requirement analysis survey to create
a wireframe prototype of a post-meeting dashboard and 16 partici-
pants from the user study teams evaluated the design and helped us
further rene the components. Finally, we developed an AI sensing
system to quantify these meeting dynamics features and created
personalized, interactive, post-meeting dashboards for the partic-
ipants. Through surveys (N=23) and interviews (N=9), we found
that participants were able to become more aware of the group
dynamics by reviewing the dashboard. Our study shed light on the
privacy concerns participants had regarding such insights within
the dashboard. The dashboard also helped participants identify
and recollect important events of the past meetings. Our ndings
also showed that participants perceived that the dashboard would
improve meeting eectiveness and inclusivity. In addition, partici-
pants expressed that actionable suggestions were more helpful than
data visualizations alone. The main contributions of this work are
as follows:
•
We developed MeetingCoach, an AI-driven dashboard that
provides both contextual and actionable insights based on
meeting behaviors.
•
We implemented both behavioral (e.g., participation) and
topical (e.g., questions) meeting dynamics features in our
feedback system using state-of-the-art AI.
•
We identied and implemented shared and private design
approaches for dierent feature components based on users’
preferences from two design iterations and evaluations.
•
We demonstrated that MeetingCoach helped improve behav-
ioral awareness and recollection of past meeting events, and
bears the potential to improve perceived eectiveness and
inclusivity.
•
We proposed design guidelines explaining the need for ac-
tionable suggestions, reminders or highlights based on tim-
ing, and multi-modal feature implementations to be adopted
for future video-conferencing feedback systems.
2 RELATED WORK
2.1 Factors in Meeting Dynamics
Meeting eectiveness includes both task processing and interaction
eciency by a team [
10
,
14
,
31
,
47
,
53
]. Dickinson and McIntyre [
10
]
Samrose, et al.
emphasized the importance of goal specication, entailing identi-
cation and prioritization of tasks and sub-tasks, in agendas and
other meeting resources. Even with clear goals, though, interaction
eciency has a clear impact on both outcomes and satisfaction.
Balanced, active, and equal participation have been found to im-
prove team performance [
13
,
32
]. Depending on the type of meeting,
equal participation may not always be applicable or feasible, but in
a collaborative decision-making discussion, participation from all
members ensures at least the exchange of opinions and a sense of
“being heard”, which ultimately improves team satisfaction [
32
,
50
].
Turn-taking patterns also inuence team performance and satisfac-
tion, as some members may dominate the discussion without real-
izing they are doing so, reducing time for other members to voice
their opinion or expertise [
14
]. Lawford [
31
] found that rapport
building through verbal and non-verbal signals was an important
factor in eective and inclusive discussions. To ensure coordination
and rapport, aect management has been found to play an impor-
tant role in a team’s success [
4
,
8
]. Barsade [
4
] has shown that a
member’s positivity can improve the mood of the whole team, mak-
ing it more inclusive and improving the quality of decision-making.
Cannon-Bowers et al. [
8
] discussed the importance of eective strat-
egy formulation to consider alternative courses of action in case
of disagreement or task failure. Non-verbal gestures, through head
nodding and shaking, indicate signs of agreement or disagreement,
and levels of interest, acknowledgement, or understanding [
22
,
34
].
While the face-to-face views of video-conferencing intuitively
seem to support the above, they have been found to constrain atten-
tion to the non-verbal signals and the overall progress of the meet-
ing [
19
,
23
]. Sellen [
51
] showed that having video did not improve
the interruption or the turn-taking rate for video-conferencing
meeting participants compared to audio-only ones. This implies
that even though video is important in online meetings, it cannot
fully resemble in-person meeting dynamics. Especially during long
meetings, additional support for monitoring meeting progress and
participation may be needed. Taking into consideration these con-
cerns, we designed and developed an automated feedback system
to summarize meeting content and attendee behaviors, with the
goal toward improving meeting dynamics over time.
2.2 Feedback Systems for Videoconference
Meetings
Researchers demonstrated the impact of feedback systems on meet-
ing dynamics and discussion outcomes for in-person, text chat, and
video-conferencing meeting setups [
7
,
11
,
24
,
27
,
35
,
43
,
48
,
56
].
Feedback on participation [
48
], turn-taking [
13
], interruption [
49
],
agreement [
25
], and valence [
15
], have eectively improved group
discussion dynamics. These studies show that the timing (e.g., real-
time, post-meeting) and the design (e.g., number of features, vi-
sualization strategies) of the feedback are important in eectively
modulating collaboration behaviors in a group discussion.
Researchers explored feedback systems with aective, behav-
ioral and topical group discussion features. Dimicco et al. [
11
]
presented a real-time, shared-display feedback system measuring
speaking contributions from audio recordings, visualized as bar
graphs during a co-located meeting. They showed that eective
visualization can help improve the group discussion, even though

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings
shared visualization can also motivate behavioral changes due to
social pressure [
39
]. Nowak et al. [
44
] provided feedback on vocal
arousal and explored the impact on oneself and one’s partners be-
havior during a negotiation-based task conducted over the phone.
They found that real-time feedback can be dicult to process during
an ongoing task and can negatively impact user performance.
Therefore, even though real-time feedback has been found to be
eective in modulating behaviors during a discussion, it can also be
distracting and cognitively demanding [
54
,
56
]. Samrose et al. [
48
]
presented CoCo, an automated post-meeting feedback system pro-
viding summarized feedback on talk-time, turn-taking, speech over-
lap and sentiment through a chatbot for a video-conference group
discussion. They showed that post-meeting feedback can eectively
make successive discussion more balanced. Through a longitudinal
study in a video-conference learning environment, EMODASH [
15
],
an interactive dashboard providing feedback on aective meet-
ing features, improved behavioral awareness over time. Instead
of showing numeric or categorical feedback on linguistic features,
Tausczik and Pennebaker [
56
] provided real-time and individualized
actionable suggestions in text chat group discussions. Their nd-
ings showed that individualized suggestions helped teams shape
their behavior; however, too much information in the feedback
can be cognitively taxing. Suggestion-oriented feedback has been
found eective in behavior modulation [
54
], and could be useful
post-meeting. As we will later explain, our design incorporated an
individualized and suggestion-oriented approach to a post-meeting
feedback system. While identifying the feedback features for our
dashboard during our requirement analysis, we prioritized features
from these prior systems, such as talking time and turn-taking, but
with an eye toward reducing cognitive load.
Beyond the meeting context, researchers have developed a num-
ber of interfaces that allow users to view emotional or aective sig-
nals captured by self-report, diaries or sensor-based systems. These
have been used in several domains, such as self-reection [
21
,
36
,
41
], data exploration and information retrieval [
18
,
59
,
60
], stress
management [
2
,
20
], and studying interpersonal dynamics [
28
,
29
].
Data portraits can help people understand more about themselves
and other people [
12
]. However, there is still a lot left to be under-
stood about how to best represent complex and subjective data,
such as emotions or group dynamics. Aective data is often highly
dimensional, multi-modal, and continuous, all dicult when design-
ing useful visualizations. There are also important privacy concerns
raised by creating digital systems and artifacts that encode highly
personal information [12].
The feedback needs of organizational teams conducting video-
conferencing meetings require special attention, as these teams are
relatively stable over time and the members need additional support
in tracking the progress and the outcomes of their meetings [
8
,
26
,
33
]. As video-conferencing discussions can be prone to distraction
and multitasking [
30
], we hypothesize that a meeting feedback
system that helps members reect on meeting goals and progress
could be a useful tool. Feedback on the non-verbal behaviors can
also help with eective and inclusive videoconference meetings.
The Matrix to a Model of Coordinated Action (MoCA) presented
by Lee and Paine [
33
] is an elaborate framework that explains a
complex collaborative environment by seven dimensions. Within
the context of MoCA, the post-meeting feedback dashboard that
CHI ’21, May 8–13, 2021, Yokohama, Japan
we propose is characterized as a long-term asynchronous periodic
event for small teams placed in dierent locations.
In this study, we observe the meeting challenges and the needs
of several in situ work teams, and propose and test technologi-
cal solutions for them. Meetings have evolved from engaging in-
person, to fully computer-mediated (all members join remotely
via audio/video/chat), and hybrid (some members join remotely to
group/s who are together in person), and each have their distinct
character [
17
,
19
,
58
]. This study focuses on the fully-computer-
mediated meetings of teams in which all members were remote due
to COVID-19’s mandatory requirement to work from home. All
teams used the same video-conferencing system. We followed an
iterative, human-centered design process to address our research
questions. Our approach was comprised of two main phases. In the
rst phase, we performed a preliminary investigation via survey
to understand the current challenges and needs for online meet-
ings, and gathered design requirements for our technology probe.
Informed by our ndings from the preliminary study, in phase 2
we conducted two design iterations through a longitudinal study
of actual remote, recurring team meetings. In the following sec-
tions, we describe the details of the requirements analysis and the
longitudinal studies.
3 PHASE 1: REQUIREMENT ANALYSIS
SURVEY STUDY
In the rst phase of our work, we aimed to identify the challenges
and the expected solutions through a survey study. Our goal is to
understand how participation and inclusivity during meetings aect
meeting eectiveness, what social signals are relevant in the context
of meeting eectiveness, and what challenges people face in video-
conferences. The ndings from our survey were used to gather
requirements and inform the design of our AI-assisted feedback
system that we discuss later (Section 4). The survey was approved
by the Microsoft Research Institutional Review Board (IRB).
3.1 Survey Design
The topics of our survey questions spanned the frequency of meet-
ings, meeting eectiveness, challenges during meetings, and useful
information and signals from meetings. We asked the participants
to self-assess the meeting eectiveness, perceived inclusivity, and
participation in the most recent eective and ineective meeting
that they had. We asked the participants to provide quantitative
and qualitative feedback on the importance of a variety of infor-
mation for meeting eectiveness, such as social signals, meeting
summary, participation of the attendees (both time spent talking
and turn-taking), tone of the meeting, etc. Participants were asked
to reect on when this information would be useful and how it
could be presented to them (e.g., personalized view, highlight reel).
3.2 Participants
We recruited survey participants through an e-mail advertisement
at a large technology company. A total of 120 completed survey
responses were collected. While participating in the survey, partic-
ipants were working from home and joining meetings via video-
conferencing due to COVID-19 mandatory requirements. Partici-
pants reported as working in the roles of a program manager (25%),

CHI ’21, May 8–13, 2021, Yokohama, Japan Samrose, et al.
A
B
Figure 1: Relationship between eective & ineective meet-
ings with respect to participation & inclusivity. (A) Full
or equal participation prevailed more in eective meet-
ings (χ2(
1
, N =
231
) =
34
.
89
,p <
0
.
001
). (B) The feel-
ing of inclusivity was more prominent in eective meet-
ings (χ2(1, N = 240) = 21.78, p < 0.001).
developer (23%), manager (17%), researcher (3%), and administra-
tive assistant (1%). 58% of participants reported that they organized
1-5 meetings per week, and 40% of participants reported that they
attended more than 15 meetings per week.
3.3 Analysis & Findings
We used a chi-square test on quantitative responses to investi-
gate dierences between eective and ineective meetings. We
also performed a thematic analysis [
6
] on open-ended responses
to derive themes around topics of interest (e.g., challenges, infor-
mation needs). We then categorized the responses into various
themes and quantied their occurrence. Below, we highlight three
ndings from our analysis: (1) factors that inuence meeting eec-
tiveness, (2) challenges participants faced in online meetings, (3) so-
lutions that participants proposed or desired, and (4) privacy and
trust related concerns.
3.3.1
Meeting Eectiveness & Inclusivity
. From the meetings
that the participants reported as eective or ineective, we com-
pared the dierences in self-assessed participation or feeling of
being included. We found that full participation was signicantly
more prevalent in eective meetings in comparison with inef-
fective ones (
χ2(
1
, N =
231
) =
34
.
89
, p <
0
.
001
.
). We also
found that the feeling of being included was signicantly more
prevalent in eective meetings in comparison with ineective
69.3%
64.9%
46.5%
35.1%
32.5%
30.7%
29.0%
27.2%
14.9%
5.26%
Connectivity issues
Failure to stick to the agenda
Repetitious discussion of topics
Software issues
Hardware issues
Failure to reach the goals of the meeting
Difficulties in communicating - myself
Difficulties in communicating - other people
Negative tone to discussions
Other
Figure 2: Percentage of participants in our survey that re-
ported experiencing challenges in workplace meetings.
ones (
χ2(
1
, N =
240
) =
21
.
78
, p <
0
.
001). Fig. 1 shows the compar-
ison of eective and ineective meetings across participation and
inclusivity.
3.3.2
Challenges in Online Meetings
. The top two challenges
faced in online meetings that participants reported were related
to the connection issues (69.3%) and the meeting agenda (or lack
of) (64.91%), as can be seen in Fig 2. Excluding tech issues, responses
indicate that attendees faced diculties with maintaining agenda
items (64.91%), repetition (46.49%), reaching goals (30.7%), being
heard (self: 28.95%, others: 27.19%), negative tone (14.91%). A total of
78% of the participants reported understanding social signals to be
very important in meetings and 55% of the participants expressed
that social signals are more dicult to understand in online meet-
ings, even with audio/video, compared to face-to-face meetings.
3.3.3
Proposed Solutions
. Through thematic analysis on the re-
sponses, we identied ve main categories of solutions brought
up by the participants to address the aforementioned challenges.
(1) Preparing for the meeting: 37% of participants wanted to have a
clear agenda before the meeting and a summary of goal/outcome
after the meeting. (2) Ensuring productive meeting dynamics: 30% of
participants mentioned specic features that could assist in main-
taining productive meeting dynamics – 26% were participation or
turn-taking related (e.g., attendees speaking as per their role in the
agenda, who are participating and for how long, whether someone
is getting a chance to speak), and others included having enough
pauses for smoother turns, what questions are being asked, what
the agreement level is. (3) Improving the usability of the tools: 22% of
participants asked to improve general hardware/software/usability
related components of the video-conferencing platform. (4) Con-
veying social cues: 14% of participants suggested keeping attendees’
cameras on to understand each other’s social cues. (5) Moderating
the meeting: 8% of participants requested some form of eective
moderation whereas 4% brought up the need to have a dashboard
that summarizes the meeting events or notes, and the other 4%
suggested having a meeting moderator who can keep the discus-
sion on track and productive during the meeting. Nine out of ten
participants expressed interest in a post-meeting feedback tool with
56% asking for combined highlights of audio, video, and transcripts.
3.3.4
Privacy and Trust Concerns
. Participants brought up con-
cerns regarding with whom the meeting data would be shared. They
did not want sentiment as a behavioral feature to be included in
the post-meeting tool for four reasons: (1) trust of AI: suspicion

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings
regarding how sentiment computation works, its accuracy, what
the outcome would be if the computation is wrong, etc. (“I don’t
feel that AI is good enough at sentiment.”), (2) privacy concerns: par-
ticipants felt uncomfortable sharing their own or accessing others’
sentiment data (“I think it would be overly intrusive to have this data
from other participants”), (3) applicability: participants were not
certain how they would use sentiment information (“I’m not sure if
I’d nd that useful, or how.”), and (4) subjectivity: sentiment might
be context-dependent and negative/neutral tone might not make
a meeting ineective (“I have some doubts on whether this can be
done accurately, given how subjective some of this information would
be. I would have to see an example.”). People were intrigued to see
how the real data would look (“As long as the intelligence could truly
determine negativity or positivity it would be good documentation
and a way to think about how to ip a negative meeting to the positive
with the same group.” )
Participants expressed challenges and opportunities regard-
ing behavioral (e.g., participation, turn-taking) and contex-
tual (e.g., agreement, questions) issues faced in online meetings.
Their interest in a post-meeting tool with highlighted meeting
events was also prominent. Based on these, we identied the fea-
tures and designed a post-feedback tool, which we discuss in Sec-
tion 4.
4 PHASE 2: LONGITUDINAL USER STUDY
The second phase of our work involved a 4-week longitudinal user
study. We collected in situ meeting data to develop an AI system
that extracts salient and desired signals identied in our require-
ment study (Section 3). The collected data were used to conduct two
design iterations and evaluations of a meeting feedback system with
the participants, rst with mocked data (Section 4.2) and second
with real data from meetings captured during the user study (Sec-
tion 4.3).
4.1 Longitudinal Study Design
Our longitudinal user study involved capturing audio and video
streams from regularly scheduled weekly meetings of teams. We
targeted recurring meetings of teams rather than one-o meetings
because the members of recurring meetings have more opportuni-
ties to invest in improving their meeting eectiveness. We collected
separate audio-video stream from each participant of the meeting.
We captured the meetings of in situ work teams to ensure that
the data used for our system and the meeting feedback we pro-
vided were contextually appropriate and relevant. In addition, each
member of the participating teams provided feedback on our initial
wireframe design (Section 4.2) and evaluated their own, personal-
ized interactive dashboards (Section 4.3).
The participants were compensated $10 per meeting hour for tak-
ing part in the data collection, and an additional $10 for participating
in an interview after the study. We recruited a total of 8 teams from
the same company (49 participants, with an average of 6 participants
per team (
min =
3
,max =
10)), who consented for us to record their
weekly project meetings over 4 weeks. This second phrase study
was also approved by the IRB. All teams conducted their meetings
via the Microsoft Teams video-conferencing platform, closely mir-
roring their current workplace meeting practices: at the time of the
CHI ’21, May 8–13, 2021, Yokohama, Japan
study, all meetings were conducted remotely due to the COVID-19
work-from-home policy. We recorded a total of 28 meetings with
an average duration of 32 minutes (min = 11,max = 62).
4.2 Design and Evaluation of Meeting Feedback
Wireframe
Our ndings from Section 3 informed the initial design of the meet-
ing feedback system. We rst enumerated specic feedback features
that could support challenges and solutions expressed by our survey
participants. We then designed meeting feedback wireframe that in-
cluded those feedback features. Using the prototype wireframe, we
conducted a survey study with our longitudinal study participants
at the end of their 2
nd
week of the 4-week longitudinal study. The
goal of the survey was to evaluate the usability and eectiveness of
each of these features and to inform which features or components
should appear in the next design iteration. We collected a total of
16 completed evaluations from the participants. Here, we describe
the details of the design of the meeting feedback wireframe and the
results of the evaluation survey of our rst design.
4.2.1
Design Strategies & Feature Definitions
. As our previ-
ous requirement analysis survey study revealed the need for a post-
meeting feedback tool with meeting highlights, we focused on de-
signing a wireframe prototype with the meeting dynamics features.
First, we categorized the feedback features into two classes: (1) be-
havioral (e.g., participation, sentiment), and (2) topical (e.g., tran-
scription, questions about the meeting content). Second, we iden-
tied that participants felt more comfortable sharing the topical
contents with others, whereas they wanted most behavioral features
to be kept private. We incorporated those priorities in the wire-
frame dashboard design. Finally, participants expressed interest in
a combination of context-based summary and action items. As such,
we organized the feature visualization into three ways: (1) sum-
marized: showed a quick snippet of the average measurement of a
feature; (2) suggestive: provided actionable suggestions related to
the individual’s feature outcomes; and (3) temporal: presented meet-
ing events in a contextualized timeline manner. For all temporal
features, the idea was that upon clicking on any of the event com-
ponents, the corresponding video and transcript portions would be
played/highlighted. We organized the left panel to have the summa-
rized and suggestive features, whereas the right panel presented all
the temporal features in their timelines. The wireframe prototype
design is shown in Fig 3. We provide our primary feature denitions
and the corresponding constraints in Table 1.
4.2.2
Results & Next Steps
. The responses from the evaluation
survey showed that participants perceived the system prototype
to be important (Fig. 4A,
M =
4
.
5
, S D =
1
.
62) and capable of
impacting meeting inclusivity (Fig. 4B,
M =
4
.
63
, S D =
1
.
32). No-
tably, as the wireframe version was not interactive, the contextual
information tied to the temporal feedback could not be fully ex-
plored. Participants were interested in using the interactive version
of MeetingCoach and discussed the potential of the AI sensing
capabilities:
P2: “I like the dashboard and the insights it provides on
engagement and recommendation. The eectiveness

CHI ’21, May 8–13, 2021, Yokohama, Japan
and impact of the dashboard really depend on the de-
nition (AI algorithm) of the metrics on the dashboard.”
P8: “This would be great if you could hover over the
components to see what the questions, comments, and
topics were.”
Based on these responses, we modied the implementation of
the interactive dashboard features: (1) how each feature was de-
ned (e.g., hovering over the term consensus revealed that it was
captured from visual signals with a combination of head-nods
and head-shakes), and (2) what inherent information each feature
holds (e.g., hovering over each consensus event revealed the exact
distribution of head-nod/head-shake leading to the overall con-
sensus, and upon clicking on an event marker, the corresponding
portion of the meeting video was played).
Among the features, speaking turn (
M =
5
.
8
, S D =
0
.
98), engage-
ment (
M =
5
.
53
, S D =
1
.
32), and topic distribution (
M = 5.47,SD =
1
.
15) were perceived as the most useful features. The usefulness
of the wireframe feature components are shown in Fig. 7 (yellow
bars). Participants valued the idea of quickly traversing through
specic portions of the meeting and reviewing the discussion (“I
would personally see more use for it (speaking turn) to jump to specic
sections in a meeting recording where a specic speaker is talking”).
Some participants also asked for suggestions based on turn-taking
patterns, whereas some mentioned that it might be too context
dependent. Temporal sentiment modulation was rated the least
useful (
M =
3
.
73
, S D =
1
.
24). Participants found it interesting but
were unsure how to utilize the information (“This might be useful,
but not sure how to take action on it”). However, some also expressed
an interest in evaluating it with their own real data. Participants
also suggested some modications to the representations of the
components:
P7: “This (speech overlap) should be integrated into the
speaking turn graphic if it is meant to overlap.”
Samrose, et al.
P2: “Is it (average tone) accessible and how can I be sure
the information is private and condential to me?”
In the next step, we updated the design to integrate speaking
patterns and speech overlap into one representation. We updated
terms such as engagement to participation as used in some previous
literature, tone to your sentiment to make its scope unambiguous.
Accurately identifying dierent agendas of the meeting and pre-
cisely marking the timestamp for each of them were dicult to
implement. Another major challenge was dierentiating between
small talk and a short agenda discussion. Therefore, we exclude topic
distribution from the nal dashboard. The next section discusses
the interactive dashboard in detail.
4.3 Design and Evaluation of Interactive
Feedback Dashboard
Our evaluation of the prototype wireframe revealed the need for
several design changes. We carried out these changes in a second
design iteration phase, extracted the proposed features from actual
meeting data collected from the study, and implemented the inter-
active dashboard. At the end of the 4-week study, we evaluated the
interactive dashboard through a mix of semi-structured interviews
and a survey study with our study participants.
First, we conducted a 30-minute, semi-structured interview ses-
sion with nine randomly selected participants, one at a time, to
capture their interaction behaviors with the dashboard. We con-
ducted the interviews rst so that all participants were interviewed
about their rst impression of the dashboard. The interview adopted
‘think aloud’ or cognitive walk-through approach to understand the
interaction patterns while answering the interview questions. The
semi-structured questions asked the interviewees to nd out who
spoke most and least in the meeting, how meeting participation
changed from week to week, when meeting sentiment changed and
whether they agreed, when a question was asked by attendee-X
and whether it was answered, who interacted the most, how likely
A
B
C
D
E
F
G
H
I
J
K
L
M
Engagement Summary
Suggestion on Engagement
Tone Summary
Suggestion on Tone
History By Meeting
Transcript
Meeting Video
Question Event Markers
Agenda Distribution
Consensus Event Markers
Tone Modulation
Speech Overlap Event Marker
Speaking Pattern
Figure 3: Visualization of our wireframe dashboard design and its components.

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings CHI ’21, May 8–13, 2021, Yokohama, Japan
Feature Denition Type Viz. Privacy
(A) Engagement Summary:
The percentage of time one spoken during
Behavioral
Summarized
Shared
the whole meeting. It also showed the number of attendees who spoke at
least once. The intention was to help everyone feel included and heard.
(B) Suggestion on Engagement:
Tips on improving meeting engage-
ment, which were updated based on whether the “Engagement” metric
Behavioral Suggestive Private
was “too high”, “too low”, or “equal”. The intention was to provide specic
guidance to improve group engagement in future.
(C) Tone Summary:
The percentage of time one’s sentiment remained
Behavioral
Summarized
Private
positive/neutral/negative. The sentiment was measured from facial sig-
nals. The intention was to grow awareness about perceived tone.
(D) Suggestion on Tone:
Tips on improving one’s meeting sentiment re-
Behavioral Suggestive Private
lated to facial expression. The intention was to provide specic guidance
to improve individual sentiment in future.
(E) History by Meeting:
Average speaking time and sentiment in past
Behavioral
Summarized
Private
meetings. The intention was to track behavior modulation over the time.
(F) Transcript:
Timestamped text of what was spoken during the meet-
ing by which participant. The intention was to allow reading the script
Topical Temporal Shared
for connecting context to the other temporally visualized features.
(G) Meeting Video:
The recorded audio-video feed of the meeting. The
intention was to allow replaying the video for connecting context to the
Topical Temporal Shared
other temporally visualized features.
(H) Question Event Markers:
The moments when and what questions
were asked by which attendee. Topical Temporal Shared
(I) Agenda Distribution:
Identication of which agenda items were
discussed during the meeting and when. Topical Temporal Shared
(J) Consensus Event Markers:
The moments when attendees agreed
Topical Temporal Shared
or disagreed. It was measured by a combination of head-nods and head-
shakes. The number represented the count of members contributing to
that agreement/disagreement.
(K) Tone Modulation:
Facial sentiment of the group and the individual
throughout the meeting. The intention was to be able to connect aect-
Behavioral Temporal Private
rich moments with other meeting attributes, such as agenda.
(L) Speech Overlap Event Markers:
The moment when two or more
members spoke at the same time. The intention was to observe speaker’s
Behavioral Temporal Shared
oor transfer, interruption, etc.
(M) Speaking Pattern:
Showed which member spoke during what por-
Behavioral Temporal Shared
tion of the meeting. The intention was to review turn-taking, speaking
duration, agenda-wise speaker selection, etc.
Table 1: Denitions of the features included in our wireframe dashboard. Each feature was associated with behavioral or topical
information, some were summarized states, some temporal graphical visualizations, and others suggested actions. Those that
were only shown to the individual are labeled as private.
it was that they would continue using the system, what meeting
inclusivity meant to them and whether the dashboard supported it,
and what further modication the dashboard needed.
Second, the participants in the study received an individualized
feedback dashboard populated with their actual meeting data. Each
participant could see only their own participation history over mul-
tiple meetings, average and temporal sentiment, etc., and meeting
specic shared information, such as the meeting transcript, tempo-
ral group sentiment, etc. After interacting with the dashboard, the
participants lled out a survey evaluating the system’s ability to
improve meeting awareness, eectiveness, and inclusivity. Here, we
describe the details of the system design, feature implementation,
and the evaluation results of our interactive dashboard.
4.3.1
System Development
. We used Microsoft Teams meetings
for our data collection. A customized video recording bot
3
was
created to record the video and audio data from each participant in
the meeting separately. This bot was added to all the meetings in our
study and the recordings stored on a secure server. The recordings
were then processed with a series of software pipelines (shown in
Fig. 5) to automatically extract features related to the verbal and
non-verbal content of the meeting. After extracting the feature
signals and processing them into categorized feedback metrics, we
implemented a web-based interactive dashboard by using HTML,
jQuery, D3.js, and Google Charts Developer Tool. The dashboard
3
https://docs.microsoft.com/en-us/microsoftteams/platform/bots/calls-and-
meetings/real-time-media-concepts

CHI ’21, May 8–13, 2021, Yokohama, Japan Samrose, et al.
A
B
Figure 4: Survey responses evaluating the wireframe dash-
board. (A) Perceived agreement on the impact of the dash-
board on improving meeting eectiveness and inclusiv-
ity. (B) Rating of the potential usability of the wireframe.
was hosted in Microsoft Azure so that participants could interact
with it from anywhere. Participants received unique access IDs to
view their individualized dashboards. Fig 6 shows the interface of
the interactive dashboard. Right side of the corresponding gure
shows the hover functionalities. Upon clicking a temporal event,
the corresponding recorded meeting video was played.
4.3.2
Feature Implementation
. After collecting the recorded
meeting audio-video feed, we used a multi-modal sensing pipeline
(shown in Fig. 5) to extract the group dynamics related signals from
the collected feed and process them into feedback features:
Transcription and Questions Detection.
We used the Mi-
crosoft conversation transcription API
4
to extract the transcripts
of the meeting recordings. Given that the video and audio feeds
of each attendee were recorded separately, we used the speech-to-
text system to extract the transcript for each attendee and then
synchronized them with time stamps.
Participation & Turn-taking Detection.
Our participation
metric was based on the duration for which each meeting attendee
spoke during the meeting. The audio received from the recordings
was sampled at 16kHz and passed through the Microsoft Windows
Voice Activity Detector (VAD) [
55
] which provided a Boolean value
for every second of audio. For each attendee of a meeting, partic-
ipant feature was computed as the percent average talk-time. To
implement speaking-turn, the individually separated audio for each
attendee was analyzed and then the metrics were synchronized
using the video timestamps.
4
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-
service/conversation-transcription
Face and Facial Landmark Detection.
We used the Microsoft
Face API
5
to detect the faces in each of the video frames and applied
a landmark detector to identify the eyes, nose, and mouth. These
data are used for the downstream components of head gesture
detection and facial sentiment classication.
Agreement/Disagreement and Consensus Detection.
We
used a Hidden Markov Model (HMM) to calculate the probabil-
ities of the head nod and head shake gestures. The HMM used the
head yaw rotation value over time to detect head shakes, and the
head Y-position values over time to detect head nods. Consensus
was measured whenever two or more attendees asserted either
head nod or head shake signals in the synchronized timeline.
Sentiment Classication.
The faces were cropped from the
video frames using the bounding box information provided by the
face detector, and the resulting image-patches were sent to a fa-
cial expression detection algorithm. The facial expression detector
returned eight probabilities, one for each of the following basic
emotional expressions: anger, disgust, fear, happiness, sadness, sur-
prise and neutral. This is a frequently employed categorization of
facial expressions; however, it is not without critics, as displays of
emotion are not uni-modal or necessarily universal [
3
,
16
]. We used
the publicly available perceived EmotionAPI
6
, allowing other re-
searchers to replicate our method. The emotion detection algorithm
is a Convolutional Neural Network (CNN) based on VGG-13 [
5
]
which has been externally validated in prior work [40, 42].
Suggestion Construction.
We prepared a list of suggestions
to appear based on the dierent levels of participation and senti-
ment. Example of suggestions included “Before moving onto the
next agenda, consider checking whether everyone is on the same
page” (more than average participation), “While speaking is not
always needed, sharing your agreement or disagreement on the
agenda can make the meeting eective” (less than average partic-
ipation), “Continue cultivating a welcoming environment for the
team” (more positive sentiment), “Consider expressing any con-
cern you may have about the agenda” (more negative sentiment),
“Your webcam was o during the meeting. Turning on video can
help your team engage better” (no sentiment data due to no video
feed), etc.
4.3.3
Results
. The average responses of the survey questions
are presented in Fig 8. In general, on a scale of 1-7 in which
the higher the better, participants found the system to be impor-
tant (
M =
4
.
91
, S D =
1
.
41) and useful (
M =
5
.
23
, SD = 1.38
). They
agreed that the dashboard would improve their meeting aware-
ness (
M =
5
.
45
,SD = 0.86
). They perceived that the dashboard
would improve meeting eectiveness (
M =
4
.
45
, S D =
1
.
47) and in-
clusivity (
M = 5.23, SD = 1.23
). Five participants specically com-
mented that they found the system to be useful enough to utilize
across other meeting series (“I really like the types of data that are
presented in the dashboard, and I would be curious to use it for more
meetings to help drive inclusivity and engagement.” ).
After interacting with the dashboard and reviewing their own
meeting data, participants were able to draw insights from the
meeting (
M =
4
.
68
, S D =
1
.
43). However, the impact varied across
5https://azure.microsoft.com/en-us/services/cognitive-services/face/
6
https://docs.microsoft.com/en-us/xamarin/xamarin-forms/data-cloud/azure-
cognitive-services/emotion-recognition

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings CHI ’21, May 8–13, 2021, Yokohama, Japan
Figure 5: We developed a custom Microsoft Teams Bot to record and analyze the independent audio and video streams from
each member of an online meeting. Each video stream was processed to detect the face of the participant and then analyze
expressions and head gestures. Each audio stream was processed to detect voice activity of the participant and then determine
turn-taking. The audio was transcribed and questions detected from the resulting text.
Video
Transcript
Sentiment
Consensus
Questions
Speaking Turn
Your Participation History by Meeting
Participation
Your Sentiment
100%
75%
50%
25%
0%
08-11-2020 08-18-2020 08-25-2020 09-02-2020
positive negative
?
Sentiment
This graph shows the modulation
of your team's and your facial
emotion during the meetings.
Higher score means more positive
sentiment.
Example of hover-over explainer text:
?
?
?
?
?
?
You Rest of the Team
Tina
Sarah
Abdul
Tina
Sarah
Abdul
Tina
2
Sarah: I think the app you're
referring to was quick, right?
Agreement:
Disagreement:
2 Head Nod(s)
0 Head Shake(s)
Figure 6: A screenshot of our interactive web dashboard on the left. Hover-over functionalities are shown on the right.
dierent features. For example, participants could easily determine
at which point each attendee spoke and for how long from the
turn-taking feature (
M = 6.00, SD =
0
.
82), and could deduce from
the consensus feature who generally contributed to any decisions
made (
M = 3.45, SD = 1.22
). Although the consensus feature visu-
alized when an agreement/disagreement may have occurred, in-
spections of the meeting video and transcript were still needed
to understand exact details, such as whether the group made a
decision and precisely who contributed to it.
In terms of the usefulness of the feature components, video, speak-
ing turn, and participation were highly rated (shown in Fig 7 (blue
bars)). Participants appreciated having the meeting video to pro-
viding detailed context for the event highlights. During the semi-
structured interview, participants expressed that having the video
and the transcript helped them verify the accuracy of the features
and understand the context of that event snippet. Sentiment re-
lated features received lower ratings. The interview participants
expressed that because of the nature of any workplace meeting
the sentiment features remained mostly neutral, therefore most of
the time there was not any specic events to focus on. However,
participants also expressed that feedback might be only useful in
case of a major shift towards negative sentiment.

CHI ’21, May 8–13, 2021, Yokohama, Japan Samrose, et al.
Figure 7: Comparison of the usefulness ratings of the fea-
ture components in our iterative evaluation phases. Yellow
and blue bars represent the ratings participants provided to
each component after reviewing the wireframe and the in-
teractive dashboards, respectively. Notably, Topic Distribu-
tion and Speech Overlap ratings are null for interactive dash-
board as the features are excluded during iteration.
Q1: Improve my awareness of meeting behaviors.
Q2: Help identify important events in a meeting.
Q3: Help prepare for the next meeting in a recurring series.
Q4: Improve meeting effectiveness.
Q5: Improve meeting inclusivity.
Q6: I think the dashboard is important.
Q7: I think the dashboard is useful.
Q8: Satisfied with the dashboard.
Q9: Draw insights from my meetings.
Q10: Understand how my participation changed from week to week.
Q11: Determine when each attendee spoke and for how long.
Q12: Determine what each attendee said.
Q13: Determine WHAT questions were asked, if any.
Q14: Determine WHEN questions were asked, if any.
Q15: Determine WHO asked questions, if any.
Q16: Tell if consensus was reached.
Q17: Determine who contributed to decisions made, if any.
Q18: Determine if the overall sentiment in the meeting changed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Question
Average Score
1
2
3
4
5
6
7
Figure 8: User ratings evaluating the interactive dashboard
P13: “I’d be worried about the accuracy of the sentiment
measurements. Maybe a better understanding of what
“negative” sentiment means, together with some exam-
ples, would help alleviate my worry.”
P23: “People just might be serious in the meeting, which
might translate as negative, and that would not be ap-
propriate.”
Participants expressed how comfortable they were with the pri-
vacy strategy adopted in our system. On a scale of 1-5 (“Not at all”
to “A great deal”), participants felt comfortable knowing their par-
ticipation (
M =
4
.
41
, S D =
0
.
67) and turn-taking (
M =
3
.
59
, S D =
1
.
18) information were shared and sentiment information was pri-
vate (M = 3.36, SD = 1.26).
Participants also provided recommendations for improving the
dashboard. They suggested having fewer and/or customizable fea-
tures: “The dashboard should be customizable so that individuals
can see at a glance the things they most care about.” Participants
requested feature renements: “ It’s hard, for example, to identify
important events of a meeting because there are so many events. On
the questions view, for example, some events just say “Yes?” and some
are paragraphs of text long.” New features were also suggested: “I’d
love to see stats on interruptions as well. Maybe also stats on aggres-
sive behavior, e.g., raising my voice.” To improve trust in the system,
users mentioned that it could be more transparent: “I feel further
work is required to improve transparency, information design and
layout, and trust with the user using the system.”
For most feature components, participants expressed the need for
actionable suggestions that could help them eectively modify their
dynamics in future, if needed. In our current dashboard, only two
summarized features on the left panel (participation and sentiment)
had explicit suggestions. The participants thought that those brief
suggestions regarding how to improve meeting behaviors in the
future made the feedback more impactful.
P2: “I really enjoy the actionable recommendations. I
think it’s great to have these recommendations on the
screen. The recommendations were actually the rst
thing I looked at.”
P3:“At the end of a meeting you don’t want to sit and
study data. You want one nudge. Good nudges would be
... giving hints on what you could do next time to make
the meeting more inclusive.”
Our dashboard was a combination of event highlights (e.g., what
questions were asked) and behavior patterns (e.g., length of each
speaking turn). The ndings showed that based on when feedback
was being delivered, the information need might be dierent. Par-
ticipants mentioned that actionable suggestions would be useful
to have as a reminder right before the next meeting, whereas the
event highlights would be useful to review after the meeting.
P11: “I feel there’s lots of interesting ideas in this ‘one
dashboard’ that I’d rather see in dierent places rather
than all in one overview. I would rather receive an email
or a reminder just before a meeting about the informa-
tion in the left column to serve as a reminder on how
to improve my upcoming meeting. The right-column
looks very useful in Teams as maybe a tab inside the
meeting chat-channel? I’d personally split these two up
into separate components.”
P13: “I feel like this would be more valuable to managers
and new members on the team if they’re trying to gure
out how they t in. I would probably not use this for
every meeting, but I would use it to reect to meetings
that I want to reect on, either if I was feeling a meeting
didn’t go well that I wanted it to, or if I wanted to look
back and remember what was said during the meeting.”
5 DISCUSSION
Our study reveals a unanimous agreement that attendees of video-
conferencing meetings are in the need of feedback assistance to
make their meetings more eective and inclusive. The inability

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings
to understand the social cues, in association with hardware- and
software-related challenges, make online meetings dicult for
workplace meeting attendees. However, currently no major video-
conferencing platforms incorporate such elements. There is an
opportunity to help attendees improve their meeting dynamics by
leveraging the audio-video feed within such platforms.
The complexity of meeting dynamics makes it challenging to
generate contextually appropriate feedback for the attendees. We
explored the usefulness of various behavioral and topical features
and provided the context through a temporal visualization. Partici-
pants found it useful to have the recorded meeting video and the
transcript included with the dashboard, as these enabled them to
quickly review the context of dierent feature events.
Group-based features require data from participating team mem-
bers. If not everyone in the meeting agrees to be recorded and
included in the feedback tool, these features may not fully describe
the meeting dynamics. Although our work studied real workplace
meetings, we only recruited teams whose entire team agreed to
participate in the study. We found that potential participants carried
concerns regarding how the data would be captured and shared,
who would have access to the data and the dashboard, how would
sensitive meeting contents be excluded from external analysis, etc.
In addition to our potential participants’ concerns, team members
may feel coerced into participating in the group’s eort to improve
its meeting eectiveness and inclusivity, even when one is not com-
fortable with sharing their video and audio feeds. Another concern
is in using this data for performance evaluations, with or without
the employees’ direct knowledge. Making sure that the data capture
and the dashboard visualization addresses such concerns of all the
attendees is key to deploying such a system. Future research should
study how to appropriately capture consent and communicate the
role of the feedback tool.
5.1 Implications for Design
During the iterative process of the study, the participants men-
tioned suggestions, concerns, and expectations they had regarding
a post-meeting dashboard. Extrapolating from those suggestions,
we suggest some key design improvements and future possibilities.
5.1.1
Provide actionable suggestions
. Being able to dive into
the temporal features is valuable for examining specic events and
cross-matching those with the meeting context. However, analyzing
so many features is potentially time-consuming and overwhelming.
Based on our ndings, we propose that feedback systems, especially
with a high number of features, need to provide more actionable
suggestions that can help direct the behavior nudges.
From an implementation perspective, providing suggestions for
every meeting dynamic is challenging, as the suggestions will by
necessity be context dependent. For example, turn-taking patterns
will greatly vary based on the type of the meeting (scrum, planning
meeting, presentation, etc.). To understand the meeting context, AI
systems need more information about the meeting type and the
team culture, beyond simply capturing the meeting audio-video
signals.
5.1.2
Modify feedback design based on its delivery timing
.
Our iterative study revealed that meeting attendees have dierent
CHI ’21, May 8–13, 2021, Yokohama, Japan
needs before, after, and during the meeting. Therefore, feedback
should be modied based on the timing. We categorize the prospec-
tive delivery or timing of feedback information as below:
(a) Reminders before Meetings:
Right before attending a
meeting, quickly reviewing any actionable suggestions or feed-
back from the past meetings can help maintain eectiveness and
inclusivity. We nd that participants wanted actionable suggestion
as a reminder immediately before meetings. These notications can
help individual attendees remember behavioral goals. Therefore,
we propose that summarized suggestions from the last meeting
could be sent as a reminder before the next meeting of the series.
(b) Highlights after Meetings:
Participants mentioned that
the temporal information, especially regarding the topical fea-
tures (e.g., transcript, questions) would be more useful when some-
one missed the meeting, or needed to recollect particular informa-
tion from the previous meeting. They also mentioned that exploring
the temporal behavior features (e.g., consensus) after the meeting
would be useful to evaluate what actions needed to be taken in
the next meeting. Thus, feedback systems built with the intent to
help formulate goals for the next meeting should incorporate event
highlights from the previous meetings.
5.1.3
Provide training opportunities based on feedback
from a series of meetings
. The current implementation focuses
on per-meeting feedback, but participant interest in splitting feed-
back into pre-meeting and post-meeting time periods indicates that
there is the potential for further valuable longitudinal use of the
feedback. Aggregating personal and team feedback across multiple
meetings might enable deeper learning and foster good habits for
the long-term. Further, feedback from such a system could also
be used to personalize meeting training programs for both man-
agers and individuals who are seeking to measurably improve their
eectiveness and inclusion.
5.1.4
Incorporate multi-modal signals to provide rich feed-
back.
Our ndings show that to get deeper insights of a feature of
group discussion, analyzing a single modality might not be enough.
Multi-modal analysis of a feature may not only provide a more re-
ned and accurate metric, but also may achieve more trust from the
user. For example, in our implementation we measured consensus
by using head nod and shake visual signals. Combining this with
language properties to capture whether words related to agree-
ment or disagreement were used co-temporally with head nods and
shakes could improve the understanding of consensus. Previous
work [
56
] has also used agreement analysis from a single modality
by applying Language Style Matching. Based on our ndings, we
propose that future systems should incorporate multi-modal signals
to implement features for capturing group dynamics.
5.2 Limitations
The meetings we recorded, while diverse in terms of gender and job
role, were all from the employees of a single large technology com-
pany during COVID-19 mandatory working from home restrictions.
Future work will be needed to validate whether they generalize
beyond that sector and context. In addition, the team meetings were
recurring, which meant that the study participants all knew one an-
other to some degree. This may also have inuenced the perceived

CHI ’21, May 8–13, 2021, Yokohama, Japan
eectiveness and inclusivity of the meetings, but also the comfort
level with sharing the tone and participation information about the
meetings. All participants were remote in this study, but enabling
such a system for hybrid meetings will add a signicant level of
extra complexity, both technically and in the way that people can
use the system. Being recorded for a study might have caused a
Hawthorne eect impacting how participants behave. However, we
presume the Hawthorne eect to be minimal because our partici-
pants were already accustomed to being recorded during meetings
for documentation purposes, and the meetings we recorded over 4
weeks were recurrent meetings with already familiar colleagues. In
our study, the participants interacted with the dashboard once, but
frequent and repeated usage over time can provide better insights
on the impact of such a dashboard on meeting dynamics. In our
future work, we would explore the dashboard usage over a longer
period of time.
6 CONCLUSION
Video-conferencing meetings are a standard feature of modern
work, even though meeting eectiveness and inclusivity can be
diminished due to the constraints on the availability of social sig-
nals. We conducted an iterative study with two phases, including
a requirements analysis and interactive system design (wireframe
and interactive dashboards) to understand the eectiveness of these
tools for video-conferencing meeting attendees. Our requirement
analysis phase showed that, even though equal participation is
not always expected, participation to some degree can increase
the perceived eectiveness and inclusivity of a meeting. It also
revealed that, in online meetings, attendees are painfully aware of
not having access to each other’s social cues, and that there is a
need for a post-meeting dashboard to summarize meeting dynamic
highlights. While some video-conferencing platforms provide some
real-time features to make up for their constraints (e.g., hand raises,
thumbs up, clapping, etc.), no major commercial platform leverages
aective signals to provide actionable meeting metrics.
In our iterative system design, we included behavioral and con-
textual features in a summarized, temporal, and suggestive user
interface dashboard. Based on the privacy concerns expressed by
the participants, we kept sentiment features private to the individ-
ual, while including other features shared just among the meeting
attendees (exactly as would be true of a meeting recording). The
evaluation showed that reviewing behavior history over time can
improve attendee’s awareness to the meeting dynamics. Partici-
pants expressed an interest in comparing the meeting dynamics
over dierent meeting series and expected more actionable sugges-
tions that they could use in future meetings. Based on the feedback
from our participants, we proposed having meeting reminders with
suggestive feedback before a meeting, and event highlights with
detailed insights after the meeting.
The kind of sensing explored in this study looks forward to a
future of human-AI partnerships. We are at the point where we can
inuence the nature of those partnerships. On the one hand, such
systems can act as prosthetics, enabling people to do things they
otherwise could not but also setting up a relationship of dependency.
On the other hand, such systems could be empowering and change
the skills and capabilities of users over time [
57
]. Meetings, even
Samrose, et al.
when mediated, are valuable precisely because they leverage the
immediacy and intimacy of human social connection to achieve
what would be dicult to achieve by other means. We hope future
systems incorporating similar kinds of aective and actionable
highlights will enable people to be more inclusive and eective
during meetings and, in the long-run, improve their comfort and
productivity.
REFERENCES
[1]
Joseph A Allen, Nale Lehmann-Willenbrock, and Steven G Rogelberg. 2015. The
Cambridge handbook of meeting science. Cambridge University Press.
[2]
Yadid Ayzenberg, Javier Hernandez, and Rosalind W. Picard. 2012. FEEL: frequent
EDA and event logging - a mobile social interaction stress monitoring system. In
Extended Abstracts of Human Factors in Computing Systems. 2357–2362.
[3]
Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M Martinez, and
Seth D Pollak. 2019. Emotional expressions reconsidered: challenges to inferring
emotion from human facial movements. Psychological Science in the Public Interest
20, 1 (2019), 1–68.
[4]
Sigal G Barsade. 2002. The ripple eect: Emotional contagion and its inuence
on group behavior. Administrative science quarterly 47, 4 (2002), 644–675.
[5]
Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016.
Training deep networks for facial expression recognition with crowd-sourced
label distribution. In Proceedings of the 18th ACM International Conference on
Multimodal Interaction. 279–283.
[6]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[7]
Dan Calacci, Oren Lederman, David Shrier, and Alex ‘Sandy’ Pentland. 2016.
Breakout: An open measurement and intervention tool for distributed peer
learning groups. arXiv preprint arXiv:1607.01443 (2016).
[8]
Janis A Cannon-Bowers, Scott I Tannenbaum, Eduardo Salas, and Catherine E
Volpe. 1995. Dening competencies and establishing team training requirements.
Team eectiveness and decision making in organizations 333 (1995), 380.
[9]
Ross Cutler, Yasaman Hosseinkashi, Jamie Pool, Senja Filipi, Robert Aichner,
and Johannes Gehrke. 2020. Meeting Eectiveness and Inclusiveness in Remote
Collaboration. arXiv preprint arXiv: To Appear (2020).
[10]
Terry L Dickinson and Robert M McIntyre. 1997. A conceptual framework for
teamwork measurement. Team performance assessment and measurement (1997),
19–43.
[11]
Joan Morris DiMicco, Katherine J Hollenbach, Anna Pandolfo, and Walter Bender.
2007. The impact of increased awareness while face-to-face. Human–Computer
Interaction 22, 1-2 (2007), 47–96.
[12]
Judith Donath, Alex Dragulescu, Aaron Zinman, Fernanda Viégas, and Rebecca
Xiong. 2010. Data portraits. In ACM SIGGRAPH 2010 Art Gallery. 375–383.
[13]
Wen Dong, Bruno Lepri, Taemie Kim, Fabio Pianesi, and Alex Sandy Pentland.
2012. Modeling conversational dynamics and performance in a social dilemma
task. In 2012 5th International Symposium on Communications, Control and Signal
Processing. IEEE, 1–4.
[14]
Norah E Dunbar and Judee K Burgoon. 2005. Perceptions of power and interac-
tional dominance in interpersonal relationships. Journal of Social and Personal
Relationships 22, 2 (2005), 207–233.
[15]
Mohamed Ez-Zaouia, Aurélien Tabard, and Elise Lavoué. 2020. Emodash: A
dashboard supporting retrospective awareness of emotions in online learning.
International Journal of Human-Computer Studies 139 (2020), 102411.
[16]
José-Miguel Fernández-Dols and Maria-Angeles Ruiz-Belda. 1997. Spontaneous
facial behavior during intense emotional episodes: Artistic truth and optical
truth. In The psychology of facial expression, James A Russell and José-Miguel
Fernández-Dols (Eds.). Cambridge University Press, New York, NY, 255–274.
[17]
C Marlene Fiol and Edward J O’Connor. 2005. Identication in face-to-face,
hybrid, and pure virtual teams: Untangling the contradictions. Organization
science 16, 1 (2005), 19–32.
[18]
Michelle Gregory, Nancy Chinchor, Paul Whitney, Richard Carter, Elizabeth
Hetzler, and Alan Turner. 2006. User-directed sentiment analysis: Visualizing
the aective content of documents. In Proceedings of the Workshop on Sentiment
and Subjectivity in Text. 23–30.
[19]
Steve Harrison. 2009. Media space 20+ years of mediated life. Springer Science &
Business Media.
[20]
J. Hernandez, X. Benavides, P. Maes, D. McDu, J. Amores, and R.W. Picard. 2014.
AutoEmotive: Bringing empathy to the driving experience to manage stress. In
Proceedings of the Conference on Designing Interactive Systems: Processes, Practices,
Methods, and Techniques, DIS. https://doi.org/10.1145/2598784.2602780
[21]
J. Hernandez, D. McDu, R. Fletcher, and R.W. Picard. 2013. Inside-out: Reecting
on your inner state. In IEEE International Conference on Pervasive Computing and
Communications Workshops, PerCom Workshops 2013. https://doi.org/10.1109/
PerComW.2013.6529507

MeetingCoach: An Intelligent Dashboard for Supporting Eective & Inclusive Meetings
[22]
Dirk Heylen. 2005. Challenges ahead: Head movements and other social acts in
conversations. Virtual Social Agents (2005), 45–52.
[23]
Ellen A Isaacs and John C Tang. 1994. What video can and cannot do for collabo-
ration: a case study. Multimedia systems 2, 2 (1994), 63–73.
[24]
Alejandro Jaimes, Takeshi Nagamine, Jianyi Liu, Kengo Omura, and Nicu Sebe.
2005. Aective meeting video analysis. In 2005 IEEE International Conference on
Multimedia and Expo. IEEE, 1412–1415.
[25]
Jeroen Janssen, Gijsbert Erkens, and Gellof Kanselaar. 2007. Visualization of
agreement and discussion processes during computer-supported collaborative
learning. Computers in Human Behavior 23, 3 (2007), 1105–1125.
[26]
Florian Jentsch, John Barnett, Clint A Bowers, and Eduardo Salas. 1999. Who
is ying this plane anyway? What mishaps tell us about crew member role
assignment and air crew situation awareness. Human Factors 41, 1 (1999), 1–14.
[27]
Taemie Kim, Agnes Chang, Lindsey Holland, and Alex Sandy Pentland. 2008.
Meeting mediator: enhancing group collaborationusing sociometric feedback. In
Proceedings of the 2008 ACM conference on Computer supported cooperative work.
457–466.
[28]
Uroš Krčadinac, Jelena Jovanović, and Vladan Devedžić. 2012. Visualizing the
aective structure of students interaction. In International Conference on Hybrid
Learning. Springer, 23–34.
[29]
Kostiantyn Kucher, Daniel Cernea, and Andreas Kerren. 2016. Visualizing excite-
ment of individuals and groups. In Proceedings of the 2016 EmoVis Conference on
Emotion and Visualization. 15–22.
[30]
Anastasia Kuzminykh and Sean Rintel. 2020. Classication of Functional At-
tention in Video Meetings. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems. 1–13.
[31]
G Ross Lawford. 2003. Beyond success: Achieving synergy in teamwork. The
Journal for Quality and Participation 26, 3 (2003), 23.
[32]
Harold J Leavitt. 1951. Some eects of certain communication patterns on group
performance. The Journal of Abnormal and Social Psychology 46, 1 (1951), 38.
[33]
Charlotte P Lee and Drew Paine. 2015. From The Matrix to a Model of Coordinated
Action (MoCA) A Conceptual Framework of and for CSCW. In Proceedings of the
18th ACM conference on computer supported cooperative work & social computing.
179–194.
[34]
Jina Lee and Stacy C Marsella. 2010. Predicting speaker head nods and the eects
of aective information. IEEE Transactions on Multimedia 12, 6 (2010), 552–562.
[35]
Gilly Leshed, Diego Perez, Jerey T Hancock, Dan Cosley, Jeremy Birnholtz, Soy-
oung Lee, Poppy L McLeod, and Geri Gay. 2009. Visualizing real-time language-
based feedback on teamwork behavior in computer-mediated groups. In Proceed-
ings of the SIGCHI Conference on Human Factors in Computing Systems. 537–546.
[36] Madelene Lindström, Anna Ståhl, Kristina Höök, Petra Sundström, Jarmo Laak-
solathi, Marco Combetto, Alex Taylor, and Roberto Bresin. 2006. Aective diary:
designing for bodily expressiveness and self-reection. In CHI’06 extended ab-
stracts on Human factors in computing systems. 1037–1042.
[37]
Paul Lu, Christian Heath, Hideaki Kuzuoka, Jon Hindmarsh, Keiichi Yamazaki,
and Shinya Oyama. 2003. Fractured ecologies: creating environments for collab-
oration. Human–Computer Interaction 18, 1-2 (2003), 51–84.
[38]
Sandi Mann and Lynn Holdsworth. 2003. The psychological impact of telework-
ing: stress, emotions and health. New Technology, Work and Employment 18, 3
(2003), 196–211.
[39]
Rob McCarney, James Warner, Steve Ilie, Robbert Van Haselen, Mark Grin,
and Peter Fisher. 2007. The Hawthorne Eect: a randomised, controlled trial.
BMC medical research methodology 7, 1 (2007), 30.
[40]
Daniel McDu, Eunice Jun, Kael Rowan, and Mary Czerwinski. 2019. Longitudi-
nal Observational Evidence of the Impact of Emotion Regulation Strategies on
Aective Expression. IEEE Transactions on Aective Computing (2019).
[41]
Daniel McDu, Amy Karlson, Ashish Kapoor, Asta Roseway, and Mary Czerwin-
ski. 2012. AectAura: an intelligent system for emotional memory. In Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems. 849–858.
[42]
Daniel McDu, Kael Rowan, Piali Choudhury, Jessica Wolk, ThuVan Pham, and
Mary Czerwinski. 2019. A Multimodal Emotion Sensing Platform for Building
CHI ’21, May 8–13, 2021, Yokohama, Japan
Emotion-Aware Applications. arXiv preprint arXiv:1903.12133 (2019).
[43]
Poppy Lauretta McLeod, Jerey K Liker, and Sharon A Lobel. 1992. Process
feedback in task groups: An application of goal setting. The Journal of applied
behavioral science 28, 1 (1992), 15–41.
[44]
Michael Nowak, Juho Kim, Nam Wook Kim, and Cliord Nass. 2012. Social
Visualization and Negotiation: Eects of Feedback Conguration and Status. In
Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work
(Seattle, Washington, USA) (CSCW ’12). ACM, New York, NY, USA, 1081–1090.
https://doi.org/10.1145/2145204.2145365
[45]
Isabelle Odermatt, Cornelius J König, Martin Kleinmann, Maria Bachmann, Heiko
Röder, and Patricia Schmitz. 2018. Incivility in meetings: Predictors and outcomes.
Journal of Business and Psychology 33, 2 (2018), 263–282.
[46]
Steven G Rogelberg. 2018. The surprising science of meetings: How you can lead
your team to peak performance. Oxford University Press, USA.
[47]
Marcello Russo. 2012. Diversity in goal orientation, team performance, and
internal team environment. Equality, Diversity and Inclusion: An International
Journal (2012).
[48]
Samiha Samrose, Ru Zhao, Jeery White, Vivian Li, Luis Nova, Yichen Lu, Mo-
hammad Rafayet Ali, and Mohammed Ehsan Hoque. 2018. Coco: Collaboration
coach for understanding team dynamics during video conferencing. Proceedings
of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018),
1–24.
[49]
Sanat Sarda, Martin Constable, Justin Dauwels, Shoko Dauwels, Mohamed El-
gendi, Zhou Mengyu, Umer Rasheed, Yasir Tahir, Daniel Thalmann, and Nadia
Magnenat-Thalmann. 2014. Real-time feedback system for monitoring and facili-
tating discussions. In Natural Interaction with Robots, Knowbots and Smartphones.
Springer, 375–387.
[50]
Mark Schittekatte and Alain Van Hiel. 1996. Eects of partially shared information
and awareness of unshared information on information sampling. Small Group
Research 27, 3 (1996), 431–449.
[51]
Abigail J Sellen. 1995. Remote conversations: The eects of mediating talk with
technology. Human-computer interaction 10, 4 (1995), 401–444.
[52]
John Short, Ederyn Williams, and Bruce Christie. 1976. The social psychology of
telecommunications. John Wiley & Sons.
[53]
Tony L Simons and Randall S Peterson. 2000. Task conict and relationship
conict in top management teams: The pivotal role of intragroup trust. Journal
of applied psychology 85, 1 (2000), 102.
[54]
M.I. Tanveer, E. Lin, and M.E. Hoque. 2015. Rhema: A Real-time in-situ Intelli-
gent Interface to Help People with Public Speaking. In Proceedings of the 20th
International Conference on Intelligent User Interfaces. 286–295.
[55]
Ivan J Tashev. 2015. Oine voice activity detector using speech supergaussianity.
In 2015 Information Theory and Applications Workshop (ITA). IEEE, 214–219.
[56]
Yla R Tausczik and James W Pennebaker. 2013. Improving teamwork using
real-time language feedback. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. 459–468.
[57]
Anja Thieme, Ed Cutrell, Cecily Morrison, Alex Taylor, and Abigail Sellen. 2020.
Interpretability as a dynamic of human-AI interaction. Interactions 27, 5 (2020),
40–45.
[58]
Wilbert van Vree. 2019. Formalisation and Informalisation of Meeting Manners.
In Civilisation and Informalisation. Springer, 291–313.
[59]
Shiliang Zhang, Qingming Huang, Shuqiang Jiang, Wen Gao, and Qi Tian. 2010.
Aective visualization and retrieval for music video. IEEE Transactions on Multi-
media 12, 6 (2010), 510–522.
[60]
M. Zisook, J. Hernandez, M.S. Goodwin, and W. Picard R. 2013. Enabling Vi-
sual Exploration of Long-term Physiological Data. In IEEE Conference on Visual
Analytics Science and Technology.
[61]
Annuska Zolyomi, Andrew Begel, Jennifer Frances Waldern, John Tang, Michael
Barnett, Edward Cutrell, Daniel McDu, Sean Andrist, and Meredith Ringel
Morris. 2019. Managing Stress: The Needs of Autistic Adults in Video Calling.
Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–29.