PreprintPDF Available

The Quest for Omnioculars: Embedded Visualization for Augmenting Basketball Game Viewing Experiences

Authors:

Abstract and Figures

Sports game data is becoming increasingly complex, often consisting of multivariate data such as player performance stats, historical team records, and athletes' positional tracking information. While numerous visual analytics systems have been developed for sports analysts to derive insights, few tools target fans to improve their understanding and engagement of sports data during live games. By presenting extra data in the actual game views, embedded visualization has the potential to enhance fans' game-viewing experience. However, little is known about how to design such kinds of visualizations embedded into live games. In this work, we present a user-centered design study of developing interactive embedded visualizations for basketball fans to improve their live game-watching experiences. We first conducted a formative study to characterize basketball fans' in-game analysis behaviors and tasks. Based on our findings, we propose a design framework to inform the design of embedded visualizations based on specific data-seeking contexts. Following the design framework, we present five novel embedded visualization designs targeting five representative contexts identified by the fans, including shooting, offense, defense, player evaluation, and team comparison. We then developed Omnioculars, an interactive basketball game-viewing prototype that features the proposed embedded visualizations for fans' in-game data analysis. We evaluated Omnioculars in a simulated basketball game with basketball fans. The study results suggest that our design supports personalized in-game data analysis and enhances game understanding and engagement.
Content may be subject to copyright.
The Quest for : Embedded Visualization for
Augmenting Basketball Game Viewing Experiences
Tica Lin, Zhutian Chen, Yalong Yang, Daniele Chiappalupi, Johanna Beyer, Hanspeter Pfister
Fig. 1: Our Omnioculars prototype supports a personalized and interactive game viewing experience with embedded visualizations
for in-game analysis. We created a simulated basketball game environment to support our design probe of embedded visualizations.
Abstract
—Sports game data is becoming increasingly complex, often consisting of multivariate data such as player performance stats,
historical team records, and athletes’ positional tracking information. While numerous visual analytics systems have been developed for
sports analysts to derive insights, few tools target fans to improve their understanding and engagement of sports data during live games.
By presenting extra data in the actual game views, embedded visualization has the potential to enhance fans’ game-viewing experience.
However, little is known about how to design such kinds of visualizations embedded into live games. In this work, we present a
user-centered design study of developing interactive embedded visualizations for basketball fans to improve their live game-watching
experiences. We first conducted a formative study to characterize basketball fans’ in-game analysis behaviors and tasks. Based
on our findings, we propose a design framework to inform the design of embedded visualizations based on specific data-seeking
contexts. Following the design framework, we present five novel embedded visualization designs targeting five representative contexts
identified by the fans, including shooting, offense, defense, player evaluation, and team comparison. We then developed Omnioculars,
an interactive basketball game-viewing prototype that features the proposed embedded visualizations for fans’ in-game data analysis.
We evaluated Omnioculars in a simulated basketball game with basketball fans. The study results suggest that our design supports
personalized in-game data analysis and enhances game understanding and engagement.
Index Terms—Sports Analytics, Embedded Visualization, Data Visualization
1 INTRODUCTION
Sports broadcasting and streaming have seen exponential growth in
recent years. Basketball, for example, is one of the most popular team
sports worldwide. The NBA league alone averages about 1.4 million
viewers per game on ESPN [3]. With the advent of novel sensing and
image processing techniques, an increasing amount of live data can
now be collected in each NBA game, including scores, play-by-play
data, and even player trajectories. This data is released online in real-
time and is widely used by sports analysts and coaches to measure
the performance of teams, analyze the strengths and weaknesses of
players, and make in-game decisions. Overall, in-game sports data
has gradually become indispensable for professional basketball teams
to develop winning strategies [24,31, 50]. Several interactive sports
Tica Lin, Zhutian Chen, Johanna Beyer, and Hanspeter Pfister are with John
A. Paulson School of Engineering and Applied Sciences, Harvard University,
Cambridge, MA. E-mail: {mlin, ztchen, jbeyer, pfister}@g.harvard.edu
Yalong Yang is with the Department of Computer Science, Virginia Tech,
Blacksburg, VA. E-mail: yalongyang@vt.edu
Daniele Chiappalupi is with the Department of Computer Science, ETH
Z¨
urich, Switzerland. E-mail: daniele.chiappalupi@inf.ethz.ch. Work was
done while Daniele Chiappalupi was an intern at Harvard University.
analysis systems have been proposed to help experts consume and
analyze in-game data [28, 40, 55]. However, these systems are not yet
readily available for sports fans. Subsequently, the information needs
of a general audience during a game are largely unmet.
During a game, sports spectators and fans seek different data than
expert analysts to facilitate their own game understanding and engage-
ment. After decades of development, scoreboards or data panels shown
on TV are still the dominant method for spectators to obtain data during
live games. However, presenting data in this way often does not fulfill
the individual needs of the audience. Consequently, perhaps the most
common solution for spectators is to look up extra data on mobile
devices or separate screens. For example, basketball fans usually look
up live box scores and play-by-play data during a game on the official
NBA website [7], ESPN [3], or on the respective phone apps. Such
practices, however, distract from the game and limit the amount of
in-game data analysis that would increase game understanding and en-
gagement for fans and viewers. Hence, spectators would greatly benefit
from an effective way of analyzing in-game data and information that
does not distract from the actual game.
Embedded visualization provides a promising opportunity to satisfy
the in-game information needs of spectators, as it directly visualizes
data within its physical context [53]. Thereby, embedded visualizations
eliminate the need for context switching and reduce distractions when
looking up data. Recently, some commercial [10] and research [15] sys-
arXiv:2209.00202v1 [cs.HC] 1 Sep 2022
tems have utilized embedded visualizations to augment sports videos
(so-called augmented sports videos), providing extra information to-
gether with the video in a seamless and engaging manner. However,
all these systems are designed for experts to create augmented sports
videos post-game and do not support custom visualizations for general
audiences during a live game. In the live game, visualizations need to
be simple and allow viewers to consume and analyze data on the fly,
which can differ significantly from visualizations in post-game scenar-
ios. As such, it is still an open question on how to best use embedded
visualizations to satisfy the in-game data needs of a sports audience.
In this study, we aim to fill this gap by exploring the design space
of embedded visualizations for live sports in-game data analysis. We
adopted a user-centered design approach throughout the study to an-
swer three questions: First, we observe user needs and their workflow
through a formative user study to answer “What data do fans desire
in a live basketball game?”. Based on survey responses and in-depth
user interviews with active basketball fans, we identified user needs in
obtaining more advanced game data and control over their live game
viewing experience. Next, we tackle the question of “What are the
design considerations for embedded visualizations in live games, based
on user needs?”, by proposing a design framework for embedded
sports data visualization based on scenario,data,task, and embedded
vis. We further identify five representative game contexts, i.e., shooting,
offense, defense, player performance, and team comparison, and de-
velop embedded visualizations for them. To answer our final question
of “How well do embedded visualizations help users enhance game
understanding and engagement?”, we developed
Omnioculars
, an
embedded visualization prototype for live TV basketball game viewing
to support our design exploration. We took inspiration from Harry
Potter’s Omnioculars for Quidditch games [8], which gives fans the
ability to see everything during a game. We created a simulated game
environment in Unity3D [52] based on NBA player tracking data to em-
bed live visualizations on top of the simulated game. We conducted a
controlled user study with 16 active basketball fans using Omnioculars.
Users were able to generate distinct game insights with each embedded
visualization and to utilize Omnioculars to personalize their in-game
analysis in our simulated basketball game videos.
In summary, our contribution consists of the first design study on
embedded visualization for in-game data analysis of basketball fans.
Second, we propose a design framework to support embedded visualiza-
tion designs based on game contexts. Third, we built a simulated game-
watching environment and implemented Omnioculars, a novel design
prototype for showing five embedded visualizations in live basketball
game videos based on our design framework. Finally, we evaluate the
merits and limitations of embedded visualization for in-game analysis
with basketball fans in a user study.
2 RE LATE D WORK
Visual Analytics for Sports.
Advances in sensors and computer vision
techniques have led to the collection of more fine-grained, high-quality
sports data. Perin et al. [38] categorize sports data in visualizations
into box score data, tracking data, and metadata. Given the highly
competitive nature of sports, numerous visual analytics systems have
been developed to allow domain experts to analyze complex sports data
for discovering winning strategies. For example, SoccerStories [37]
employs a series of soccer-related visualizations, such as in-court trajec-
tories and heatmaps, to analyze spatio-temporal data collected in soccer
games. BKViz [28], an interactive data exploration tool for basketball
games, focuses on the analysis of play-by-play data. Fu and Stasko [22]
developed interactive visualization systems specifically targeting NBA
journalists. Some sports analytics approaches also leverage machine
learning models to predict and extract patterns from the data [31,34, 50].
Recently, with the proliferation of low-cost immersive devices such as
Virtual and Augmented Reality (VR/AR), immersive sports analytics
systems have been gaining traction. Compared to traditional desktop
systems, immersive sports analytics systems provide unique benefits,
including increased spatial understanding, rich embodied interaction,
peripheral awareness, and a large information space [27, 30]. Shut-
tleSpace [54], a VR-based visual analytics system for badminton data,
enables users to analyze shuttle trajectories from a player’s first-person
view. Following their work, TIVEE [17] allows experts to explore a
large scale of badminton trajectories from both first and third-person
views in VR. While these systems have demonstrated their effective-
ness in analyzing sports data, most of them are designed for post-game
analysis by domain experts, leading to a high entry barrier and steep
learning curve. On the other hand, compared to post-game analysis sys-
tems, in-game systems usually can be used by both domain experts and
fans to complete lightweight analysis. However, only a few approaches
focus on in-game analysis. GameViews [55], for example, supports
data-driven sports storytelling using in-game data for sportswriters and
fans. Yet, GameViews mainly uses simple visualizations separated from
the videos, forcing users to switch contexts when watching a game.
We draw on this line of research to develop an in-game analysis tool
for basketball fans, which features embedded visualizations to enable
lightweight analytics while retaining an engaging game experience.
Embedded Visualization in Sports Videos.
Sports data, generally
speaking, is inherently associated with its physical environment (e.g., a
basketball court). Visualizing sports data within this physical environ-
ment can significantly facilitate the understanding and analysis of the
data. Thus, numerous works have explored visualizing sports data in
static court diagrams. For example, CourtVision [23] uses point-based
visualizations to present the shot distribution and density for NBA data.
Sacha et al. [39] introduced a trajectory-based visualization system
that allows users to interactively explore the moving trajectories of
soccer players. To improve the analysis of soccer data, Stein et al. [47]
developed a region-based visualization that shows the interaction space
of each soccer player. In recent years, with the rapid development
of computer vision and image processing techniques, visualization re-
searchers have started to directly embed their visualizations into sports
videos. For instance, Stein et al. [48] developed a system to assist
soccer game analysis, which automatically extracts and visualizes data
from and in soccer videos. Stein et al. [46] then extended their work by
semi-automatically selecting the presented data based on the game’s se-
mantic context. However, these works mainly target experts for analytic
purposes and do not fulfill general spectators’ needs. More recently,
Chen et al. [15] explored the design space of augmented sports videos
and presented VisCommentator, a rapid prototyping tool for author-
ing augmented racket-based sports videos. In summary, current work
in augmented sports videos focuses on domain experts’ analytic and
authoring needs. Embedded visualizations designed for sports fans
are still under-explored. In this work, we aim to understand fans’ in-
game analysis needs and propose a structured design framework for
embedded visualization in basketball videos.
Personalized Game Viewing Systems.
When watching a sports game,
spectators often look up extra information to satisfy their curiosity,
deepen their understanding of the game, or generate game insights (as
found in our formative study in Sec 3). These information needs vary
from person to person, leading to a great demand for personalized game
viewing systems. The most common way fans access additional data is
using mobile devices to search the internet when watching the game.
However, this approach inevitably incurs context-switching costs and
significantly impedes an engaging watching experience. To overcome
this issue, HCI and Computer Graphics researchers have studied in-
teractive game viewing systems that enable spectators to look up data
while still focusing on the game. For instance, Gamebot [56] uses
a chatbot interface on a mobile app for fans to request and consume
game data visualization. ARSpectator [57] uses mobile AR devices to
overlay basic game data onto the scene for on-site spectators. Despite
the focus on interaction techniques to access game data, the existing
work did not evaluate the presentation of game data to facilitate game
understanding and engagement. On the other hand, E-Sports, such
as Defense of the Ancients2 [2] (Dota2) and League of Legends [4]
(LoL), are ahead of most traditional sports in providing personalized
game viewing experiences due to their interactive, online nature. In
Dota2, for example, the audience can interact with the viewing system
to inspect the gaming data (e.g., gold over time) of individual players or
the whole team. Nevertheless, most of these systems only present the
Fig. 2: We adopted a user-centered design process throughout the study.
We elicited user needs through a survey and interviews, and iterated
our designs based on user feedback.
data in side-by-side settings, in which the visualizations are displayed
in panels separated from the game view. We envision that embedded
visualizations have the potential to seamlessly blend the extra informa-
tion with the game view, providing a natural, intuitive, and engaging
watching experience. Our work takes a first step towards designing
and evaluating personalized game viewing systems with embedded
visualizations for traditional sports.
3 FORMATIVE STUDY W IT H BAS KE TBA LL FANS
Section 3, 4, and 5 present our user-centered design process towards
designing an interactive basketball game-viewing system with embed-
ded visualizations (Fig. 2). In this section, we detail a mixed-method
formative study using an online survey and interviews with active bas-
ketball fans to better understand the general workflow, pain points, and
best practices of fans to acquire desired data during a live game.
3.1 Procedures
We first used an online survey to collect initial feedback on basketball
fans’ data needs while watching live games. We analyzed survey
responses to understand fans’ in-game data analysis behaviors and to
inform the questions of the follow-up interviews (Fig. 2).
Online Survey.
The game-watching experiences survey (Appendix A)
asked about fandom levels (novice,casual,engaged, and die-hard fans),
game watching frequency, and in-game data analysis records. We asked
respondents to watch a live basketball game and record three moments
when they wanted to look up additional data. We advertised the survey
using university mailing lists and only collected responses from casual
fans and above. In total, we gathered 14 responses (R1-14; M = 10,
F = 4; Age: 18 - 45) and 40 records of their data-seeking moments
when watching a live game. All respondents are NBA fans (4 casual, 6
engaged, and 4 die-hard fans).
In-depth Interview.
We then interviewed six fans from the survey
respondents (I1-6; 4 die-hard and 2 engaged fans) who expressed
interest in the survey. Among them, five have been NBA basketball
fans for over ten years. We conducted the interviews remotely using
Zoom in two parts (Appendix B). We first asked respondents about
their overall game viewing experiences and data analysis behaviors in
recent games they watched, including occasions they looked at data
and whether the data was helpful and engaging. We then asked them to
watch three video clips selected from a 9-minute post-game highlight
video of an NBA regular-season game
1
and think aloud about what
game insights and data they observed and wanted to see. Each clip
consisted of two to four plays for about 30 seconds and collectively
covered a diverse range of gameplays, including offense strategy, a
player’s highlight (a layup with a foul), defense performance (a block),
or clutch time. The clips also included video effects to augment certain
gameplay, such as replays from different angles, showing a player’s
box score on the screen, etc. The interviewees contributed 63 in-game
data analysis records on the three clips. No further interviews were
conducted as saturation was met in the analysis.
Analysis.
We performed a reflexive thematic analysis [11] on the
survey and interview data. Three authors began coding separately on
the 40 survey records to create sets of plausible codes, and through
1Golden State Warriors vs. Cleveland Cavaliers, 2015
later discussion and affinity diagramming with another author, agreed
on a single coding scheme. Then the lead author finished the coding on
the remaining records collected from the interviews.
3.2 Findings and Discussions
We present findings on fans’ motivation, workflow, and pain points of
performing in-game data analysis.
3.2.1 Practices of Watching Live Basketball Games
Overall, most of the respondents watched NBA games at least once
per week (4 watched 3 games per week, 9 watched 1-3 games per
week, 1 watched less). 12 of them followed a team or a player in
the NBA league. All six interviewees watched 1-3 NBA games per
week, mainly to follow their favorite players. As for the game watching
media, the interviewees explained that while watching games on-site
is engaging and exciting for all fans, they mostly watch live streams
for convenience and economic reasons. When asked to compare their
game-viewing experience between live streams and highlight videos,
interviewees expressed that they preferred to watch live streams over
highlights. One key reason is that highlight videos do not present the
flow of the game. Although highlight videos usually provide additional
visual effects, such as play breakdown or embedded data overlays, fans
cannot observe how the game developed to the current moment and
feel “it shows no rhythm of the game” (I4). In summary, compared
to on-site watching and highlight videos, live streaming is the favorite
game watching medium among fans we surveyed.
3.2.2 Seeking Data in Various Contexts During Live Games
Seeking data while watching a live game was a common behavior
among fans we surveyed. While watching a game, all 14 respon-
dents in the survey expressed interest in looking up game data; all six
interviewed fans confirmed that they always look up game stats to com-
plement the viewing experience. Based on our analysis of the records,
a context in which a fan seeks data while watching a live game can be
characterized by a , , and . Specifically, in differ-
ent scenarios (When), fans seek different data (What) to accomplish
different tasks (Why):
Fans seek data in three typical (When)
. First, fans often
look for data at
specific game times
, such as before the game, during
halftime, or in the last quarter. These specific game times are defined
based on the rules of the sport, like basketball, and thus usually have
special meanings. Second, fans seek data when observing
a game
event
. These events include seeing a game stat, a game action, or the
appearance of a new player, “D. Rose made an and-1 play in the 2nd
quarter” (R8). Third, when
multiple repetitive game events
happen,
fans sometimes want to seek data to evaluate a player’s or team’s
performance related to this specific type of game event, “Thunder is
missing a lot of 3 pointers” (R5).
Fans seek two types of (What).
Based on the survey and in-
terviews, the data sought by the fans can be divided into two types,
inspired by Perin et al. [38]. First,
game data
are directly collected
from games, including box score, “foul number of Embiid” (R4), track-
ing data, “the usage of this particular play” (R9), and video data. The
game data sought by the fans can range from a single game, to multiple
games, or an entire career. Second,
metadata
includes data beyond
games, such as ranking, player background, and news.
Fans seek data to finish three (Why)
, ultimately generating in-
sights to deepen their understanding of the game. Inspired by Brehmer
and Munzner [12, 32], we summarized three typical tasks that the fans
perform with data during a live game. First, fans seek data to
identify
the performance of players or teams. For example, a respondent replied,
“I was wondering whether he was getting a triple-double” (R4). In addi-
tion to looking at the data of a specific player, fans are also interested in
the data of the whole team, such as the moving trajectories of a team to
understand the offensive and defensive strategies (I3). Second, fans
often need data to
compare
the in-game performance of a player or
team to others’ in-game performance or to their historical performance.
For example, a respondent replied, “I want to know how many assists
our team makes compared to other teams” (R5). Five interviewees also
provided that they wanted to compare a player’s in-game performance
to their average performance to evaluate the consistency. Third, fans
usually seek data to
summarize
the performance of players or teams.
Compared to the other two tasks, as pointed out by Brehmer and Mun-
zner [12,32], the summarize task often involves a deeper analysis of the
data, such as overviewing the game and observing patterns, deriving
the cause of the performance, and speculating on possible outcomes.
Representative responses include “to have a quick understanding of
both teams so I know what to expect in the match” (R10) (overview &
possible outcome) and “to understand what causes the score difference
between two teams” (R3) (cause of the performance).
3.2.3 Pain Points of Seeking Data in Watching Live Streams
While seeking data was common among basketball fans, the data ac-
quisition process in practice remains hindered by current technologies.
According to the survey, the main method of acquiring data is through
websites or mobile apps (30 out of 40). In the interviews, the fans
further detailed that they access websites (ESPN [3], NBA [7], or Red-
dit [9]) on the phone or on a separate screen to regularly track box score
data or team standing. Such practices reflect several issues in the live
game-watching experience:
Data provided by live streams lack diversity.
The interviewees
agreed that most information available on live streams, such as game
stats and timers, was useful. However, they also confirmed that the
information provided by the live streams is not diverse and sufficient
enough, e.g., “usually they are not sufficient, only showing a matrix
of all player names and scores (I2)”. Some information that the inter-
viewees would love to have is usually absent in live streams, such as
seeing all box scores and lineups information, or comparing shooting
performance by zones. This is the main reason that fans search websites
to obtain extra data while watching live games.
Data provided by live streams lack controllability.
Interestingly,
although live streams can provide some basic box score data (like
game stats and timers), the interviewees still will check this data on
the website regularly, “Right now, on a separate screen, I can look at
the full box scores to get an updated impression of the players” (I4).
This practice reflects that when watching live games, fans need not
only sufficient data but also the ability to control when and what data
to show. This finding also links to our characterization that users seek
different data in various contexts (Sec. 3.2.2).
Data provided by websites are separated from the game videos.
Unsurprisingly, the interviewees complained that looking up data on
the phone or a separate screen reduces the engagement of the game
watching experience as it introduces numerous context switches be-
tween the game videos and websites. For example, “in a live game
there’s a lot going on and likely I missed some events [when looking up
data]” (I1). Thus, interviewees would like a more intuitive method to
present the data directly within the game videos.
3.3 Summary
Our formative study found that live streams were the favorite watching
medium for basketball fans we interviewed. While watching live games,
fans seek different types of data in different scenarios for different tasks.
However, current live streams fall short in fulfilling fans’ diverse data
needs, forcing fans to look up data on separate screens (phones or
monitors), which reduces the engagement in watching games. To tackle
these issues, we propose to embed visualizations into live game videos
to present data in-situ and support in-game data analysis.
4 DESIGN SPACE EXPLORATION
To answer our second research question of how to design embedded
visualizations for fans’ in-game data analysis, we first developed a
design framework based on the formative study findings (Fig. 3). With
this framework, we then identified 20 contexts from the records of
fans’ in-game data seeking moments. By walking through these design
contexts using design mockups with the six interviewees, we prioritized
five contexts to be implemented in our prototype.
Fig. 3: Design framework for context-driven embedded visualizations.
4.1 Design Framework
Based on the data-seeking contexts we summarized in Sec. 3.2.2, we
propose a context-driven design framework for embedded visualiza-
tions for in-game data analysis (Fig. 3). The framework scaffolds the
relationship between four design aspects In a certain , the
user performs a on using ”:
describes the background of fans’ data analysis behav-
iors. Three design elements can be used to describe a scenario that
drives a fan’s data need: target (team or player), event (any
game event that is happening), and time (a specific moment or
a duration). For example, “When Joel Embiid fouled” describes
a player (target) with a foul (event) at the current time (time);
“When we have been making strides with offense for a streak of
time in the game” describes a team (target) make shots (event)
over a certain period (time).
that is of interest to fans can be collected in-game or beyond
games. For embedded visualizations, we focus on in-game data
and exclude metadata. Thus, the data types to be considered
include box scores (game summary scores), tracking data
(like player positions), and video. Different data can be further
combined to specify advanced stats, such as “the lineup’s plus-
minus” or “the shot percentage of #30 when there is no defender
in the front.
A is performed on game data by fans to generate insights
about players or teams, including identify (on a player or team),
compare (between players or teams), and summarize (among
players or teams).
can present in-game data directly in videos to
support fans performing in-game data analysis naturally. Prior re-
search [15] systematically studied the design space of augmented
sports videos and summarized the embedded visualizations used
in augmented videos to present sports data. We follow their de-
sign space with a focus on embedded visualizations for basketball,
and identified five relevant graphical marks and basic visual el-
ements: point,line,area,label, and side panel.
These basic visuals can be composited to construct embedded
visualizations in basketball game viewing.
This design framework helps designers identify fans’ in-game data
analysis contexts by specifying the , , and . Such
a context captures the fan’s specific requirements for in-game data
analysis at a given moment. Once a context is identified, the designer
can then focus on the design of embedded visualizations by composing
the five basic visuals, thereby ensuring the embedded visualizations are
developed based on and can satisfy the fan’s analysis needs.
4.2 Design Context Walkthrough
Following our design framework, we identified 20 design contexts from
103 records of in-game data seeking moments in the formative study
by grouping similar records and summarizing representative contexts.
We created 20 design mockups to demonstrate the contexts (Appendix
C) and walked through them with the six fans in follow-up interviews.
Based on their ratings of importance, we prioritized five contexts that
drove the design of five embedded visualizations in our prototype
(Fig. 4).
C1 Shooting
A player shoots ( ), and the fan wants to
identify and compare ( ) the player’s box score ( ).
Fig. 4: We developed five embedded visualization designs based on our
design framework and the five prioritized contexts in Sec. 4.2.
Fans found this context useful and suggested avoiding information
overload by only showing the most important stat based on player
and location.
C2 Offense
The offense team runs a play ( ), and the fan
wants to identify and compare ( ) the players’ movement
( ). Fans commented that this context is very useful to follow
the ball and understand the play better, especially for tracking
off-ball players and identifying open shot opportunities.
C3 Defense
The defense team defends well/poorly ( ), and
the fan wants to identify ( ) the defense scheme ( ).
Fans commented that this context helps identify the effective
defenders and track how defensive focus is changing.
C4 Player Performance
A player has made/missed shots
( ), and the fan wants to compare and summarize (
) the player’s shot performance by zones ( ). Fans found
this context helpful for evaluating a player’s game performance
in more detail with direct reference to their average performance.
C5 Team Comparison
At the clutch time ( ), the fan wants
to compare and summarize ( ) team stats ( ). Fans
commented instant access to this data is very convenient during
the game break.
5
Building upon our design space exploration in Sec. 4, we present Om-
nioculars, an interactive embedded visualization prototype for live game
data analysis for fans. Omnioculars consists of three main components:
a game simulator, embedded visualizations, and interactions.
5.1 Game Simulator
Ideally, Omnioculars should be built on top of live stream TVs as a
third-party game viewing system to enhance the real-world watching
experience. However, we decided to develop Omnioculars on top of a
game simulator instead of real-world broadcasting videos, considering
two aspects: On the one hand, implementing Omnioculars based on
publicly available broadcasting videos is extremely difficult. First,
the data that can be extracted from broadcasting videos is limited
due to the reduced frame rate and low resolution. Second, rendering
embedded visualizations into broadcasting videos is challenging as the
camera parameters of these videos are usually not made public. Third,
broadcasting videos often switch between cameras and insert replays,
reducing the controllability of Omnioculars. On the other hand, our
goal is not to develop Omnioculars as a fully functional system used in
real-world scenarios but as a design probe [25] to explore the notion of
using embedded visualizations to improve game viewing experiences.
To gain feedback on our design without being bogged down by the
aforementioned engineering challenges, we use a game simulator as the
system’s basis, thereby obtaining richer data, better rendering results,
and controllability. Please note that the aforementioned challenges can
be tackled once the source video streams and sensing data from the live
game collected by the TV companies and NBA league are available.
The game simulator is implemented using Unity3D [52], with a 3D
basketball arena and 10 player models (Fig. 1). To simulate a basketball
game, we adopt a data-driven strategy. Specifically, the simulator takes
a set of spatio-temporal data as the input and outputs a virtual basketball
game by moving the players and triggering events based on the data.
Table 1: The required data for the game simulator.
Type Description
Tracking Data The players and the ball’s position over time.
Event Data
The game events, each event is described by
its time, player, action, and outcome.
Shot Locations The position of each shot event.
Box Score Team and player statistics of the game.
Player Zoned
Shot %
Player’s detailed shot percentage by shot loca-
tions grouped into zones.
League Average
Zoned Shot %
The NBA league’s average shot percentage by
shot locations grouped into zones.
Table 1 shows the required data in our current implementation. Overall,
as suggested in the user study (Sec. 7.4), the simulated game was
perceived as comparable to an actual game in terms of acquiring game
insights by the users.
5.2 Embedded Visualization Designs
We iterated over each of our visualization designs with two fans we
interviewed. Both were die-hard fans with substantial basketball knowl-
edge and game viewing experiences. Therefore, we considered them to
be domain experts and asked for feedback on our visualization designs.
5.2.1 Shot Label
Fig. 5: Shot Label shows (a) static outcome and (b) dynamic percentage.
In
C1 Shooting
, we identified that fans want to see the shot perfor-
mance of the shooter before and after shooting to predict and evaluate
the shot outcome, and compare the shooter’s shot percentages to their
average or others’ performance. We propose a that consists
of two components: 1) a static label with the shooter’s shot outcome at
the shot location (Fig. 5a) and 2) a dynamic label showing the player’s
shot percentage based at their current location (Fig. 5b).
The static label is shown at the shot location after a player shoots
and is visible for 5 seconds and presents the shot outcome, the shooter’s
game, and the average shot percentage with bar charts. This allows fans
to get a quick comparison of the shooter’s own in-game performance
without interfering with the continuing game.
The dynamic label shows the shooter’s average shot percentage
based on the location above their head and moves with the player. This
supports comparing the player’s shot performance among their team
and instantly evaluating the player’s shot opportunity based on location.
The dynamic shot percentage is obtained from one of the seven shot
chart zones (see Fig. 8). To support direct visual comparison, the label
is color-coded to compare the player’s shot percentage to the league’s
average, colored in red (hot) if above and in blue (cold) if below. The
dynamic label supports fans to compare the shooter’s shot percentage
to other players, such as evaluating if the shooter is shooting from their
hot spot, and the quality of shot selection.
Alternatively, our original design had a label pop up right after a
shot and move with the shooter for a few seconds, showing the score
and shot percentage of the shooter. However, fans’ feedback suggested
that dynamic labels with dense information are difficult to read and
showing a player’s shot outcome as the player moves into the next play
does not help with game analysis. Thus, we abandoned this design.
5.2.2 Offense Trajectory
Fig. 6: Offense Trajectory
In
C2 Offense
, fans were eager to
use visualization to track the play-
set and spot open shot opportuni-
ties from the off-ball players. Our
design (Fig. 6)
shows the offense players’ trails and
highlights the open space around an
offense player with a circle under-
neath the player (area). The radius
of the open space is drawn based
on the player’s distance to the clos-
est defender and changes dynami-
cally to reflect the interaction be-
tween the offense and defense team. We also color the ball handler’s
open space in red to guide the viewer’s attention. Fans can identify
the play run by the offense team through the trails, and easily tell the
speed and rotation of the players. The open space circles reveal how
the defense team reacts to the play and help fans immediately spot open
shot opportunities (larger circles).
Before finalizing the design, we iterated on the design by encoding
other useful tracking data, such as speed measurement, player role,
and distance to other players. User feedback suggested that encoding
speed and player roles does not provide much value as a player’s speed
changes constantly and does not represent the actual moves of the
player. The differentiation of player roles also became less prominent
in modern basketball games. However, encoding the distance to the
defense player was deemed very useful for fans to track the execution
of the play. We thus kept this encoding.
5.2.3 Defense Form
Fig. 7: Defense Form shows (a) ball defender and (b) defensive focus.
In
C3 Defense
, fans wanted to identify the defense scheme and key de-
fenders, and track the change of defenders when a switch happens. The
highlights the key defenders by tracking the distance
between the defenders and the ball handler (Fig. 7a). When a defender
is within 6
ft
of the ball, that player is marked as the ball defender (dark
blue) and connected to the ball handler with a line. The defender is
marked as a helper (blue) if within 12
ft
of the ball. When multiple
defenders are identified as key defenders on the strong side (the ball
handler’s side of the court), the enclosed area is colored to highlight
the region of defensive focus (Fig. 7b). Fans can evaluate the defense
scheme from these visual marks without being overwhelmed.
An alternative design was to highlight the position of all defensive
players (point) and color the enclosed region (area) to achieve the goal
of tracking the defense scheme and transformation. However, this
design did not specify the key defenders, including the ball defender
and helpers on the strong side. Furthermore, the large enclosed area
can cause visual clutter, and does not represent the actual defense focus,
as the further away the defense players are from each other, the less
of a threat they are. Instead of highlighting all defensive players, our
final design focuses on drawing user attention to the effective defenders
and showing the defensive focus of the enclosed area between key
defenders.
5.2.4 Shot Chart
In
C4 Player Performance
, fans valued the benefit of using
embedded visualization to compare the shooter’s shot perfor-
mance in game to their own seasonal average in more detail.
Fig. 8: Shot Chart
We designed to have two
components, a heat map and a player
panel (Fig. 8). The heat map shows
the player’s seasonal shot percentage
by zones (detailed in Sec. 5.2.6). The
areas are color-coded by the compar-
ison to the league shot percentage
average, from dark blue (
10%),
blue (
5%), yellow (
±
5%), orange
(
5%), and red (
10%). We also
visualize shot attempts and locations
with 3D basketball icons on the court
to represent made (colored ball) and missed shots (black ball). The heat
map design allows fans to evaluate the shooter’s detailed game perfor-
mance directly within the game contexts, such as observing whether
the shooter made shots in their hot zone or comparing the shot number
in each zone. The player panel is shown on the courtside to show
player box score stats, providing a complete picture of the player’s
game performance besides shooting.
5.2.5 Team Panel
Fig. 9: Team panel
Based on
C5 Team Comparison
,
is designed to provide
an overview of team stats and com-
pare team performance. As shown
in Fig. 9, the panel is shown at the
courtside to allow instant look-ups
without losing sight of the game in
the front. Each stat label uses the
color of the team with the better per-
formance, such as blue for 3 Pointers
(Warriors have a higher shot percent-
age). The bars for each shooting stat
(Field Goals, 3 Pointers, and Free
Throws) are colored based on the shot percentage compared to the
league’s average using the same colors as Shot Chart. For example,
Cavaliers have a lower 3PT shot percentage than the league’s average
and are coded in dark blue. Showing a team stats comparison on a panel
is common in current TV streaming services during breaks, but it usu-
ally requires cutting away from the court view. Instead, our team panel
design is always-on, without interfering with the game and thereby
providing an effective direct comparison through colors.
5.2.6 Other Design Considerations
Color Usage.
Our color encoding of shot percentages uses blue to
red and is based on the convention of basketball analysis, where red
indicates a hot hand (good performance), and blue indicates cold (bad
performance). Similarly, Offense Trajectory uses a color scheme of
reddish tones to indicate offense strengths, while Defense Form uses
blue tones to show defense efforts.
Shot Percentage Threshold.
We use the NBA’s official shot charts [6]
and color palette to compare player and league shot percentages.
Defense Distance Threshold.
We define the effective defender as
being within 6
ft
of the offense player based on the definition of an open
shot [29]. The threshold decreases to 3
ft
if the offensive player has
passed the defender, as a threat from the back is low. We double the
threshold (12
ft
) for helpers as an approximation to indicate the strong
side range as there is no accurate definition for defensive helpers.
Shot Chart Zones.
We segmented the shot chart into seven zones,
including rim (within 8
ft
), mid-range (left/right-wing), corner three
(left/right), and 3-point range (left/right). We aggregated the 19 zones
in the NBA shot chart [5] to strike a balance betwen visual complexity
and usefulness.
5.3 Interactions
While the five embedded visualizations can be displayed based on
events, fans still desire control over when and what data should be
displayed. During the user study, we applied the Wizard-of-Oz (WoZ)
method [19, 20] to allow fans to control the visualizations with voice
commands, while the experimenter manually turned each visualization
on and off. The WoZ method is commonly used to evaluate complex,
intelligent systems while reducing implementation effort [13,14, 36].
Most importantly, the current speech recognition systems cannot meet
the speed and accuracy requirements for fast-paced use cases, like
sports. We envision with the advance in speech recognition and natural
language processing, a robust system will be available in the near
future. By using a WoZ method, we can ensure the consistency of
user experiences without being hindered by current speech recognition
systems. Nevertheless, while interaction techniques are not the focus of
our study, we identified insights regarding users’ interaction preferences
as implications for future work in Sec. 7.3.
6 USER ST UDY
We conducted a controlled user study with active basketball fans to
evaluate the usefulness and user engagement of Omnioculars.
6.1 Experiment Set-Up & Participants
To simulate the game contexts, we chose a famous game between
Golden State Warriors and Cleveland Cavaliers on Dec 25th, 2015. This
Christmas game featured the best teams from the previous season’s
final, who faced each other in the championship again, and was selected
as one of the best games by the NBA in the 2015-16 season. All the
required data, like event data, shot locations, and player zoned shot
percentage, is publicly available [1,5, 21,26], except for the tracking
data. We could only obtain 2D tracking data of the ball and players [43].
To preserve the authenticity of the game viewing experience in our
prototype, we manually annotated the vertical positions and polished
the player model animations to reflect the game events as closely as
possible.
For our study, we selected three video clips from the simulated game.
Each clip consists of about 30 seconds with two to three plays, focusing
on shooting, offense and defense performance, and the clutch time
situation. We used two video clips in the first part of the user study and
counter-balanced them for use as either training or study clip among
participants. We used the third clip, which contains the clutch time
situation, for the second part of personalized game analysis.
We recruited 16 participants through the university mailing list after
an initial screening of their fandom levels (P1-16; M = 13, F = 3; Age:
18 - 45). We targeted active fans who reported watching games at least
once a week during the game season and were interested in looking up
data during the game. Four identified as casual fans, seven as engaged
fans, and five as die-hard fans. We conducted the user study over Zoom,
which took one hour per participant to complete. We compensated each
participant with a $20 gift card.
6.2 Design & Procedures
The study had two parts. The first part focused on assessing the
understandability and usefulness of the five embedded visualiza-
tions ( , , , , and
). After being introduced to the study and filling out a
consent form, the participants went through five rounds, one for each
visualization condition. At the beginning of each round, we introduced
each visualization design and led participants through a training task to
think aloud their game insights from analyzing the embedded visual-
ization in a game clip. After they were familiar with the visualization
technique, they completed one study task of analyzing the game with
the embedded visualization on another game clip. The participants
were asked to rate and comment on the useful contexts supported by
each visualization. The second part focused on the usefulness and en-
gagement of the Omnioculars system as a whole. We asked participants
to freely explore different visualization combinations using voice com-
mands. Participants first completed a training task to analyze a game
clip with interactive visualizations using voice commands. They then
completed one study task of thinking aloud their game insights from
using interactive visualizations in another game clip. We asked them to
rate the usefulness and engagement of the overall Omnioculars system
and to comment on their strategy of using different visualizations to
Fig. 10: User study results. In Part 1, (a) participants rated all five
embedded visualizations to be easy to understand, helpful, fun to use,
and novel ways to present game data (Mdn
5). In Part 2, (b) partici-
pants confirmed the usefulness and engagement of Omnioculars, were
likely to use it in future games (Mdn=7), and perceived the simulated
game as comparable to an actual game (Mdn
=
6). (c) Participants used
different visualizations to perform game analysis.
generate game insights. At the end of the study, participants filled out a
post-study questionnaire.
6.3 Measures
Part 1.
For each embedded visualization condition, we collected the
subjective ratings on a 7-point Likert scale from low to high, including
understandability, usefulness, engagement, and novelty. We also col-
lected qualitative feedback, including the game insights generated by
the participants while watching the game clip with each visualization,
and the oral feedback about the contexts in which they found each
visualization helpful and interesting to use.
Part 2.
We collected subjective ratings on the usefulness and en-
gagement of Omnioculars, where we derived our questions from prior
work [35], including “It was helpful”,“It was fun”,“I felt in con-
trol”,“I felt encouraged”, and “I am likely” to use Omnioculars to
consume the desired game data in-game. We collected qualitative feed-
back of participants’ strategy of using different visualizations and the
strengths, weaknesses, and suggestions for Omnioculars in a post-study
questionnaire.
Qualitative Analysis.
Participants’ responses were analyzed by two
authors using affinity mapping. In Part 1, we derived primary use cases
for each visualization by grouping similar user insights. In Part 2, we
categorized user strategy in personalizing the visualizations and the
subsequent insights. The grouping results are shown in Appendix D.
7 RE SULTS & DISCUSSION
We present the user study results on how well embedded visualizations
help users enhance game understanding and engagement, and discuss
the design implications for future interactive game viewing.
7.1 What Game Insights Do Fans Generate with Each Em-
bedded Visualization?
In Part 1 of our study, participants provided their game insights while
using each embedded visualization, and commented on helpful use
cases. We discuss user subjective ratings and findings below. A detailed
categorization of user comments can be found in Appendix D.
Five embedded visualizations were perceived to be useful and
novel
. In Fig. 10a, participants rated each visualization from 1 (low)
to 7 (high) on Q1) easy to understand, Q2) helpful, Q3) fun to use,
and Q4) novel ways to present game data. The majority rated all five
embedded visualizations positively (
>
4) in usefulness and engagement,
reporting that they were easy to understand (
88%), helpful (
75%),
and fun (
69%). For novelty, Shot Label was rated novel by 94% of
the participants, Offense, Defense, and Shot Chart by 75%, and Team
Panel by 56%.
Participants used the to evaluate individual player per-
formance and player decisions
. Participants used color on the dy-
namic label to evaluate players’ shot performance, “Sean Livingston is
very low from the 3-point line (P2)”. Furthermore, some participants
contemplated the player’s decision-making, “When I see Lebron drives,
the color changed drastically, so I get that why he did not make the
three-point shots and tried to take the drive. (P1)”. Overall, partici-
pants found the Shot Label most helpful to focus on player and ball
movement, and after shooting.
Participants used the to spot the shot opportu-
nity and team strategy
. Participants used the circle underneath the
players to evaluate the shot opportunity, “They passed to people who
had more space, or a larger circle, at the time (P7)”, and the trails
to read the team strategy, “Like this run, [trails] help me understand
the team better, not just the player individually (P1)”. Participants
found the Offense Trajectory most useful when they focused on team
offense strategy and following offense during the transition, When
defense changes quickly, seeing opportunity arose when somebody was
breaking away to the basket or slipped away from the defenders is
helpful (P3)”.
Participants used the to track defense changes and
strategy
. Participants used the color of the circle underneath the players
to identify defenders. They also observed the defense strategy, “The
shaded area between the defenders shows how the defense blocked the
paint area to get the rebound (P6)”. Participants found the Defense
Form most useful when they wanted to track defense changes during a
switch or transition and to identify the defense strategy.
Participants used the to examine the overall shot perfor-
mance and shot decisions
. Participants used the heat map and virtual
balls on the court to evaluate the player’s shot performance. They also
derived insights on players’ shot decisions and on how players com-
pare between each other, “It’s interesting to see the chart differences
between shooters and role players. (P13)”. Participants found the Shot
Chart most useful to understand shot selection, “When there were lots
of shots from a particular area, to understand if they were high per-
centages shot or not (P11)”, and to evaluate players’ shot performance
immediately, “to understand someone’s performance and contribution,
especially with players I don’t know (P14)”.
Participants used the to compare team performance
.
Participants used the team panel to check on game stats. They also
derived game insights from the color encoding, “Interesting to see both
teams had lower shot percentages compared to the league average
(P10)”. Participants found the Team Panel most useful to follow the
team’s progress throughout the game and to analyze areas of focus,
“When the game score is close, I can see the key factors to evaluate and
predict the outcome (P8)”.
Overall, each embedded visualization provides helpful data on dif-
ferent game aspects for the participants
. The obtained insights and
identified use cases aligned with our design contexts. While individual
needs and knowledge levels vary, all participants were able to generate
different game insights from each visualization.
7.2 How Do Fans Personalize Omnioculars to Enhance
Game Understanding and Engagement?
In Part 2, participants interacted with Omnioculars to use different
visualizations to generate game insights. We were interested to see
how they combined or altered visualizations during the game, how their
individual preferences impacted their strategies, and how the selected
embedded visualizations helped users generate game insights.
Participants used various combinations of the different visualiza-
tions.
Fig. 10c shows the usage patterns of the five visualizations
for each participant (the amount of time a certain visualization was
displayed). On the individual level, each participant chose different vi-
sualization combinations and focused on different aspects of the game.
Six participants used all five visualizations, and all users used at least
Fig. 11: (a) and (b) show personalized visualizations configured by
different users during the user study.
three. Each visualization was used frequently (
>
33%) by several users
each, i.e., Shot Label (by 13 users), Offense (9), Defense (9), Shot
Chart (4), and Team Panel (7).
Participants developed different strategies to engage with the vi-
sualizations based on individual preferences and their game focus.
We observed two primary user strategies. The first strategy was
“con-
figure a fixed set”
, which was used by eight participants. These users
first identified their preferred combination of visualizations and used
this fixed set throughout the game clip. They hoped to access personal-
ized visualizations to facilitate game analysis without having to actively
control them. Among them, we identified two styles of rationales,
analytic-driven and experience-driven. Analytic-driven users decided
the visualization set based on their game analysis needs. For example,
P6 used Team Panel, Defense, Shot Label all together (Fig. 11a), and
explained that “these help me focus on why they are playing this way,
see if they switch or not, and track the shot percentage”. On the other
hand, P3 used Team Panel at first, and used Shot Label and Shot Chart
together for the rest of the game clip. P3 explained that “I wanted the
team panel first to get an idea of how things were going thus far. Shot
Label and Shot Chart are very useful for seeing current instant threats
by the ball handler. I also appreciated seeing whether a player was
hot compared to their average”. Experience-driven users chose the
visualizations based on engagement. For example, P2 wanted to start
with everything on because “it’s already such a fast-paced sport that
I liked just having them on. P15 used Shot Label, Defense, and Shot
Chart together because “they are very fun to use and watch” (Fig. 1).
The second strategy was
“alter based on context”
adopted by eight
users. Participants selected visualizations based on their needs, which
varied based on the current game context. For example, P5 explained
that “during the clutch time, I’ll look at team panel. When the game is
not as intense, I will look at defense and offense”. Some participants
selected visualizations based on the team they support, “If I’m watching
a game of a team I am rooting for, I focus on the visualization on that
(offense/defense vis when the team is attacking/defending)”. Other fans
showed more flexibility, “I first use Offense and Defense to understand
the team drills (Fig. 11b). When I am familiar with them, I’ll use Shot
Label to evaluate if the shot selection is reasonable (P9)”.
Embedded visualizations help participants derive insights.
When
asked about the game insights they obtained with Omnioculars, partic-
ipants focused on distinct aspects, such as identify player movement
and decision making, compare player’s in-game stats, or summarize
game performance. These goals align with the data and tasks we iden-
tified in Sec. 3.2.2. For example, P2 used the Offense Trajectory to
focus on player decisions, “when he just passed, I think it’s cool to
see that there’s not really open shots as there’s no big yellow circle
anywhere”. P3 evaluated player’s shooting with the Shot Label, stating
“it’s so interesting to see the shot percentage climb up as they drive to
the basket”. Additionally, P1 used the Team Panel to derive a game
summary, “We are doing really well in the assist and block, wow that’s
unexpected. The turnover is a problem, but if you have more assists,
you’re supposed to have more turnovers. I think that’s acceptable..
7.3 Implications for Future Interactive Game Viewing
Flexible embedded visualization design is important to enable a
customized experience.
A key finding is that individual needs for data
and desired interaction levels vary. Based on the user’s preferences and
basketball experience, some people felt a visualization was distracting
and overwhelming, while others felt the same visualization helped them
see the data more easily. For example, the trails in the Offense Trajec-
tory visualization were perceived useful to evaluate offense strategy
by some die-hard fans, “The trails are great to see the players’ strat-
egy when the players are closer, like pick-and-roll. It makes it easier
to see the tactics (P9)”. However, other fans felt they only focused
on the circles to identify the shot opportunity and did not utilize the
trails, “I just feel like it’s too much information to follow. (P2)”. A
recommended solution is to decouple the individual components of
a visualization, such as separating the trail and circle in the Offense
Trajectory visualization, or the heat map and player panel in the Shot
Chart visualization. As participants focus on a range of goals based on
their preferences, allowing them to select and mix from independent
visual components can best fulfill their individual needs.
Embedded sports visualizations are still underexplored.
We iter-
ated our designs with targeted users, and we believe they are the most
suitable in the scope of this project. However, we also believe there is a
need for more refined embedded visualizations targeting specific tasks
and contexts based on our proposed design framework. Additionally,
designing embedded sports visualizations not only requires considera-
tion for objective dimensions such as perceptual effectiveness, but also
for subjective dimensions, such as engagement, which plays a more
vital role compared to traditional visualization design.
Expressive interactions allow users to go beyond a passive viewing
experience.
We used voice commands as the interaction mechanism
for our study, as we believe, with overwhelming choices, it can best
articulate the user’s intention [41,44,45], especially in fast-paced sports
watching. Participants had different opinions about the provided inter-
actions: a few participants preferred to have a certain level of default
behaviors, “I wish Shot Chart would turn on automatically for a few
seconds after shots (P2)” and “I feel like it would be a hassle to be
turning it on and off and on and off (P3)”, while others enjoyed having
interaction, “I liked the voice activated part of it, very useful to follow
the game and still visualize what I wanted (P11)”. To this end, we
suggest an interaction approach that would allow users to configure
default settings based on their game focus on desired data. Future work
can explore interaction design space in the spectrum from manual to
automatic and train personalized AI models to support semi-automatic
interactions. In terms of functionality, we only allowed simple inter-
actions to toggle visualizations on and off, mainly considering the
fast-paced nature of sports. However, more complicated interactions
could be required (check the average stats of three players). Future
research can explore other interactions to support in-game data analysis.
A systematic evaluation of human perception and attention could
shape embedded visualization design.
Participants highlighted the
advantages of our embedded visualization design for game viewing,
including providing immediate feedback, being easily accessible while
watching the game video, and alleviating mental load. For example,
the Shot Label visualization provides “immediate visual feedback after
the pass. I’ve seen the color flip from red to blue to immediately know
that (P3)”. With Team Panel sitting next to the court, “I can look at
it, for as long as I need to and then switch over to looking at what’s
happening now (P2)”. The visualization can help fans focus on the
ball, “I could still be watching the ball, but out of the corner of my
eye could see whether or not they had options (P12)”. As the first
step towards understanding the use of embedded visualization in sports,
our work aims to provide proof-of-concept results for the design space
and its effectiveness. As a result, we did not systematically evaluate
these potential advantages in our study. We believe, as future work,
obtaining empirical knowledge of these effects can largely facilitate the
development of embedded sports visualization.
Future broadcasting with interactive embedded visualizations.
In
current live streams of games, camera angles may change and cut off
the scene. Without access to camera configurations and the complete
video footage, embedding visualizations into videos accurately can
be difficult. However, broadcasters already collect all the required
information, such as tracking data and the full video footage. Therefore,
we believe our design framework and visualizations can be adopted
and extended to augment actual live games by the broadcasters, or by
researchers once the data are made available. Similar to our envisioning
direction, a few personalizations are already offered by the NBA, such
as customized viewing angles with the NBA League Pass [33] and
augmented visual overlays on the live game view by CourtVision [18].
Beyond-the-screen game viewing.
Many people prefer watching
games in person to experience the live atmosphere. Unlike watch-
ing a game through digital devices with visual effects, it is challenging
to overlay extra information on a physical court. Fortunately, with
recent advances in augmented reality (AR), it is now possible to embed
digital graphics into the physical world. We believe that using AR to
embed visualizations into an in-person game will enhance the game
viewing experience, and help spectators make sense of the real-time
game data. We also believe our results can be largely generalized to an
AR in-person game viewing scenario, as we designed and implemented
our system in a 3D environment (Unity3D). However, using our designs
in AR still differs from a flat screen and we need to consider: the effect
of stereoscopy vision and depth perception; the limited field of view
(FoV) in current AR devices; and the ability to freely change viewing
direction so that physical and digital objects can be out-of-view. In the
future, we plan to adapt our current system to AR (by adding out-of-
view notifications and by simplifying visualization designs based on
the FoV) and to evaluate its effectiveness.
7.4 Simulating Basketball Games in a Virtual Environment
Our study shows that simulation can be a promising solution to ease
the development and evaluation of visualizations embedded in physical
contexts. 94% of all users considered game insights derived from the
simulated game comparable to an actual game (Fig. 10b). Compared to
traditional visualizations presented in digital environments, embedded
and situated visualizations presented in physical contexts are difficult to
develop and evaluate due to the limited accessibility and controllability
of the physical contexts. Moreover, unlike web-based visualizations,
embedded and situated visualizations are inherently challenging to
distribute, reproduce, and benchmark, hindering the research progress
in this emerging direction. By compromising a certain degree of fidelity,
simulation enables a highly accessible and controllable physical context
that can be shared, reproduced, and compared universally, significantly
lowering the barrier to designing embedded or situated visualizations.
In recent years, thanks to the proliferation of low-cost VR devices, many
researchers [16, 17, 42, 49,51, 54] have leveraged VR to simulate real-
world environments in user-centered design studies. We envision and
advocate a standard simulation protocol for developing visualizations
in physical contexts, which may eventually facilitate the research of
embedded and situated visualizations.
8 CONCLUSION & FUTURE WORK
This study explored the design space of embedded visualization for
basketball in-game data analysis. Through a user-centered design
process, we presented a formative study of basketball fans’ in-game
data needs and proposed a context-driven design framework of four
design elements Scenario,Data,Task, and Embedded Vis. We
designed five embedded visualizations for five primary game contexts
and developed Omnioculars, a prototype for personalized game viewing
with embedded visualizations. Our evaluation results suggest that each
embedded visualization provides helpful data on different game aspects,
and fans developed different strategies to engage with visualizations
based on their individual preferences and game focus with Omnioculars.
For future work, we identified the necessity to allow customizable
visual complexity and interaction with embedded sports visualizations
based on individual preferences and knowledge levels. We also hope to
build upon the current design framework to explore using embedded
visualizations for immersive and situated in-game data analysis.
ACK NOW LE DGMENTS
This work is supported by NSF grants III-2107328 and IIS-1901030. We thank
Hans H., Joan C., Harvey H., Ahmed S., Nhan H., and Max M. for their time.
REFERENCES
[1] Basketball-reference. https://www.basketball-reference.com/.
[2] Dota 2. https://www.dota2.com/.
[3] Espn - nba. https://www.espn.com/nba/.
[4] League of legends. https://www.leagueoflegends.com/.
[5] Nba shot charts. http://nbashotcharts.com/.
[6] Nba stats. https://www.nba.com/stats/.
[7] The offical site of nba. https://www.nba.com/.
[8]
Omnioculars harry potter wiki.
https://harrypotter.fandom.
com/wiki/Omnioculars/.
[9] Reddit - nba. https://www.reddit.com/r/nba/.
[10] Second spectrum. https://www.secondspectrum.com/.
[11] V. Braun and V. Clarke. Reflecting on reflexive thematic analysis. Quali-
tative research in sport, exercise and health, 11(4):589–597, 2019.
[12]
M. Brehmer and T. Munzner. A Multi-level Typology of Abstract Visual-
ization Tasks. IEEE TVCG, 19(12):2376–2385, 2013.
[13]
J. T. Browne. Wizard of oz prototyping for machine learning experiences.
In Extended Abstracts of the 2019 CHI Conference on Human Factors in
Computing Systems, pages 1–6, 2019.
[14]
P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, O. Ra-
madan, and M. Ga
ˇ
si
´
c. Multiwoz–a large-scale multi-domain wizard-
of-oz dataset for task-oriented dialogue modelling. arXiv preprint
arXiv:1810.00278, 2018.
[15]
Z. Chen, S. Ye, X. Chu, H. Xia, H. Zhang, H. Qu, and Y. Wu. Augmenting
Sports Videos with Viscommentator. IEEE TVCG, 28(1):824–834, 2021.
[16]
Y. Cheng, Y. Yan, X. Yi, Y. Shi, and D. Lindlbauer. SemanticAdapt:
Optimization-based Adaptation of Mixed Reality Layouts Leveraging
Virtual-Physical Semantic Connections. In Proc. of UIST, pages 282–297.
ACM, 2021.
[17]
X. Chu, X. Xie, S. Ye, H. Lu, H. Xiao, Z. Yuan, Z. Chen, H. Zhang, and
Y. Wu. TIVEE: Visual Exploration and Explanation of Badminton Tactics
in Immersive Visualizations. IEEE TVCG, 28(1):118–128, Jan. 2022.
[18]
CourtVision. Clippers courtvision.
https://www.
clipperscourtvision.com/.
[19]
N. Dahlb
¨
ack, A. J
¨
onsson, and L. Ahrenberg. Wizard of oz studies—why
and how. Knowledge-based systems, 6(4):258–266, 1993.
[20]
S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter, and M. Gandy. Wiz-
ard of oz support throughout an iterative design process. IEEE Pervasive
Computing, 4(4):18–26, 2005.
[21]
ESPN. Cavaliers vs. warriors - play-by-play - december 25, 2015.
https:
//www.espn.com/nba/playbyplay/_/gameId/400828325.
[22]
Y. Fu and J. Stasko. Supporting Data-Driven Basketball Journalism
through Interactive Visualization. In Proc. of CHI, 2022.
[23]
K. Goldsberry. Courtvision: New visual and spatial analytics for the nba,
in ‘2012 mit sloan sports analytics conference’. In MIT Sloan Sports
Analytics Conference, 2012.
[24]
M. Hayes. Analytics-driven sixers ride the numbers to nba playoffs, 2018.
[25]
H. B. Hutchinson, W. E. Mackay, B. Westerlund, B. B. Bederson, A. Druin,
C. Plaisant, M. Beaudouin-Lafon, S. Conversy, H. Evans, H. Hansen,
N. Roussel, and B. Eiderb
¨
ack. Technology probes: inspiring design for
and with families. In Proc. of CHI, pages 17–24. ACM, 2003.
[26]
Kaggle. Nba play-by-play data 2015-2021.
https://www.kaggle.com/
schmadam97/nba-playbyplay- data-20182019.
[27]
T. Lin, Y. Yang, J. Beyer, and H. Pfister. SportsXR Immersive Analytics
in Sports. In 4th Workshop on Immersive Analytics: Envisioning Future
Productivity for Immersive Analytics at ACM CHI, 2020.
[28]
A. G. Losada, R. Theron, and A. Benito. BKViz: A Basketball Visual
Analysis Tool. IEEE CG&A, 36(6):58–68, Nov. 2016.
[29]
P. Lucey, A. Bialkowski, P. Carr, Y. Yue, and I. Matthews. How to get an
open shot: Analyzing team movement in basketball using tracking data.
In Proceedings of the 8th annual MIT SLOAN sports analytics conference.
Citeseer, 2014.
[30]
K. Marriott, F. Schreiber, T. Dwyer, K. Klein, N. H. Riche, T. Itoh,
W. Stuerzlinger, and B. H. Thomas. Immersive analytics, volume 11190.
Springer, 2018.
[31]
A. McIntyre, J. Brooks, J. Guttag, and J. Wiens. Recognizing and analyz-
ing ball screen defense in the nba. In Proceedings of the MIT Sloan Sports
Analytics Conference, Boston, MA, USA, pages 11–12, 2016.
[32] T. Munzner. Visualization analysis and design. CRC press, 2014.
[33]
NBA. Nba league pass.
https://support.watch.nba.com/hc/
en-us/articles/115000585974- NBA-League- Pass.
[34]
A. Nistala. Using deep learning to understand patterns of player movement
in basketball. PhD thesis, Massachusetts Institute of Technology, 2018.
[35]
H. L. O’Brien and E. G. Toms. The development and evaluation of a
survey to measure user engagement. Journal of the American Society for
Information Science and Technology, 61(1):50–69, 2010.
[36]
A. R. Palmeiro, S. van der Kint, L. Vissers, H. Farah, J. C. de Winter,
and M. Hagenzieker. Interaction between pedestrians and automated
vehicles: A wizard of oz experiment. Transportation research part F:
traffic psychology and behaviour, 58:1005–1020, 2018.
[37]
C. Perin, R. Vuillemot, and J.-D. Fekete. SoccerStories: A Kick-off for
Visual Soccer Analysis. IEEE TVCG, 19(12):2506–2515, Dec. 2013.
[38]
C. Perin, R. Vuillemot, C. D. Stolper, J. T. Stasko, J. Wood, and S. Carpen-
dale. State of the Art of Sports Data Visualization. Computer Graphics
Forum, 37(3):663–686, June 2018.
[39]
D. Sacha, F. Al-Masoudi, M. Stein, T. Schreck, D. A. Keim, G. Andrienko,
and H. Janetzko. Dynamic Visual Abstraction of Soccer Movement. In
Computer Graphics Forum, volume 36, pages 305–315. Wiley Online
Library, 2017.
[40]
T. Seidl, A. Cherukumudi, A. Hartnett, P. Carr, and P. Lucey. Bhostgusters:
Realtime Interactive Play Sketching with Synthesized NBA Defenses.
page 13.
[41]
L. Shen, E. Shen, Y. Luo, X. Yang, X. Hu, X. Zhang, Z. Tai, and J. Wang.
Towards natural language interfaces for data visualization: A survey. IEEE
transactions on visualization and computer graphics, 2022.
[42]
S. Shimizu and K. Surni. Sports Training System for Visualizing Bird’s-
Eye View from First-Person View. In Proc. of VR, pages 1156–1158.
IEEE, 2019.
[43]
SportVU. Papers with code - nba sportvu dataset.
https://
paperswithcode.com/dataset/nba-sportvu.
[44]
A. Srinivasan, N. Nyapathy, B. Lee, S. M. Drucker, and J. Stasko. Collect-
ing and Characterizing Natural Language Utterances for Specifying Data
Visualizations. In Proc of CHI, pages 1–10, 2021.
[45]
A. Srinivasan and J. Stasko. How to ask what to say?: Strategies for evalu-
ating natural language interfaces for data visualization. IEEE Computer
Graphics and Applications, 40(4):96–103, 2020.
[46]
M. Stein, T. Breitkreutz, J. H
¨
aussler, D. Seebacher, C. Niederberger,
T. Schreck, M. Grossniklaus, D. A. Keim, and H. Janetzko. Revealing
the Invisible: Visual Analytics and Explanatory Storytelling for Advanced
Team Sport Analysis. In Proc. of BDVA, pages 1–9. IEEE, 2018.
[47]
M. Stein, H. Janetzko, T. Breitkreutz, D. Seebacher, T. Schreck, M. Gross-
niklaus, I. D. Couzin, and D. A. Keim. Director’s Cut: Analysis and
Annotation of Soccer Matches. IEEE CG&A, 36(5):50–60, 2016.
[48]
M. Stein, H. Janetzko, A. Lamprecht, T. Breitkreutz, P. Zimmermann,
B. Goldlucke, T. Schreck, G. Andrienko, M. Grossniklaus, and D. A.
Keim. Bring It to the Pitch: Combining Video and Movement Data to
Enhance Team Sport Analysis. IEEE TVCG, 24(1):13–22, Jan. 2018.
[49]
Y. Tanaka, T. Shiokawa, and M. Shiokawa. Scope of Manipulability
Sharing: A Case Study for Sports Training. In Proc. of VR, pages 701–702.
IEEE, 2018.
[50]
C. Tian, V. De Silva, M. Caine, and S. Swanson. Use of machine learning
to automate the identification of basketball strategies using whole team
player tracking data. Applied Sciences, 10(1):24, 2019.
[51]
W.-L. Tsai, T.-Y. Pan, and M.-C. Hu. Feasibility Study on Virtual Reality
Based Basketball Tactic Training. IEEE TVCG, 2020.
[52] Unity Technologies. https://unity.com/, 2022.
[53]
W. Willett, Y. Jansen, and P. Dragicevic. Embedded data representations.
IEEE transactions on visualization and computer graphics, 23(1):461–470,
2016.
[54]
S. Ye, Z. Chen, X. Chu, Y. Wang, S. Fu, L. Shen, K. Zhou, and Y. Wu.
Shuttlespace: Exploring and analyzing movement trajectory in immersive
visualization. IEEE TVCG, 27(2):860–869, 2020.
[55]
Q. Zhi, S. Lin, P. Talkad Sukumar, and R. Metoyer. GameViews: Under-
standing and Supporting Data-driven Sports Storytelling. In Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems -
CHI ’19, pages 1–13, Glasgow, Scotland Uk, 2019. ACM Press.
[56]
Q. Zhi and R. Metoyer. GameBot: A Visualization-augmented Chatbot
for Sports Game. In Extended Abstracts of the 2020 CHI Conference on
Human Factors in Computing Systems, pages 1–7, Honolulu HI USA, Apr.
2020. ACM.
[57]
S. Zollmann, T. Langlotz, M. Loos, W. H. Lo, and L. Baker. ARSpectator:
Exploring Augmented Reality for Sport Events. In SIGGRAPH Asia 2019
Technical Briefs on - SA ’19, pages 75–78, Brisbane, QLD, Australia,
2019. ACM Press.
... Secondly, we only evaluated users' performance on three fundamental visual tasks, while in-situ analytics with AR visualization can present more complex challenges. Finally, even though we followed exciting works [11,28,30,56], we acknowledge that simulating AR in VR may not fully represent real-world scenarios. These limitations suggest potential areas for future research and improvements to RL-LABEL. ...
Preprint
Full-text available
Labels are widely used in augmented reality (AR) to display digital information. Ensuring the readability of AR labels requires placing them occlusion-free while keeping visual linkings legible, especially when multiple labels exist in the scene. Although existing optimization-based methods, such as force-based methods, are effective in managing AR labels in static scenarios, they often struggle in dynamic scenarios with constantly moving objects. This is due to their focus on generating layouts optimal for the current moment, neglecting future moments and leading to sub-optimal or unstable layouts over time. In this work, we present RL-LABEL, a deep reinforcement learning-based method for managing the placement of AR labels in scenarios involving moving objects. RL-LABEL considers the current and predicted future states of objects and labels, such as positions and velocities, as well as the user's viewpoint, to make informed decisions about label placement. It balances the trade-offs between immediate and long-term objectives. Our experiments on two real-world datasets show that RL-LABEL effectively learns the decision-making process for long-term optimization, outperforming two baselines (i.e., no view management and a force-based method) by minimizing label occlusions, line intersections, and label movement distance. Additionally, a user study involving 18 participants indicates that RL-LABEL excels over the baselines in aiding users to identify, compare, and summarize data on AR labels within dynamic scenes.
Article
Full-text available
Utilizing Visualization-oriented Natural Language Interfaces (V-NLI) as a complementary input modality to direct manipulation for visual analytics can provide an engaging user experience. It enables users to focus on their tasks rather than having to worry about how to operate visualization tools on the interface. In the past two decades, leveraging advanced natural language processing technologies, numerous V-NLI systems have been developed in academic research and commercial software, especially in recent years. In this article, we conduct a comprehensive review of the existing V-NLIs. In order to classify each paper, we develop categorical dimensions based on a classic information visualization pipeline with the extension of a V-NLI layer. The following seven stages are used: query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation. Finally, we also shed light on several promising directions for future work in the V-NLI community.
Conference Paper
Full-text available
We present our initial investigation of key challenges and potentials of immersive analytics (IA) in sports, which we call SportsXR. Sports are usually highly dynamic and col-laborative by nature, which makes real-time decision making ubiquitous. However, there is limited support for athletes and coaches to make informed and clear-sighted decisions in real-time. SportsXR aims to support situational awareness for better and more agile decision making in sports. In this paper, we identify key challenges in SportsXR, including data collection, in-game decision making, situated sport-specific visualization design, and collaborating with domain experts. We then present potential user scenarios in training, coaching, and fan experiences. This position paper aims to inform and inspire future SportsXR research.
Article
Tactic analysis is a major issue in badminton as the effective usage of tactics is the key to win. The tactic in badminton is defined as a sequence of consecutive strokes. Most existing methods use statistical models to find sequential patterns of strokes and apply 2D visualizations such as glyphs and statistical charts to explore and analyze the discovered patterns. However, in badminton, spatial information like the shuttle trajectory, which is inherently 3D, is the core of a tactic. The lack of sufficient spatial awareness in 2D visualizations largely limited the tactic analysis of badminton. In this work, we collaborate with domain experts to study the tactic analysis of badminton in a 3D environment and propose an immersive visual analytics system, TIVEE, to assist users in exploring and explaining badminton tactics from multi-levels. Users can first explore various tactics from the third-person perspective using an unfolded visual presentation of stroke sequences. By selecting a tactic of interest, users can turn to the first-person perspective to perceive the detailed kinematic characteristics and explain its effects on the game result. The effectiveness and usefulness of TIVEE are demonstrated by case studies and an expert interview.
Article
Visualizing data in sports videos is gaining traction in sports analytics, given its ability to communicate insights and explicate player strategies engagingly. However, augmenting sports videos with such data visualizations is challenging, especially for sports analysts, as it requires considerable expertise in video editing. To ease the creation process, we present a design space that characterizes augmented sports videos at an element-level (what the constituents are) and clip-level (how those constituents are organized) . We do so by systematically reviewing 233 examples of augmented sports videos collected from TV channels, teams, and leagues. The design space guides selection of data insights and visualizations for various purposes. Informed by the design space and close collaboration with domain experts, we design VisCommentator, a fast prototyping tool, to eases the creation of augmented table tennis videos by leveraging machine learning-based data extractors and design space-based visualization recommendations. With VisCommentator, sports analysts can create an augmented video by selecting the data to visualize instead of manually drawing the graphical marks . Our system can be generalized to other racket sports (e.g ., tennis, badminton) once the underlying datasets and models are available. A user study with seven domain experts shows high satisfaction with our system, confirms that the participants can reproduce augmented sports videos in a short period, and provides insightful implications into future improvements and opportunities.
Article
In this work, a VR-based basketball training system comprising a standalone VR device and a tablet is proposed. The system is intended to improve the ability of players to understand offensive tactics and to enable them to perform these tactics correctly. We compare the training effectiveness of various degrees of immersion fidelity, including a conventional basketball tactic board, a 2D monitor, and virtual reality. A multi-camera-based human tracking system is developed and built around a real-world basketball court to record and analyze the running trajectory of each player during tactical execution. The accuracy of the running path and hesitation time at each tactical step is evaluated for each participant. Furthermore, we assess several subjective measurements, including simulator sickness, presence, and sport imagery ability, to carry out a more comprehensive exploration of the feasibility of the proposed VR framework for basketball tactics training. The results indicate that the proposed system is useful for learning complex tactics. Also, high VR immersion training benefits athletes in terms of abilities related to strategic imagery.
Article
We present ShuttleSpace, an immersive analytics system to assist experts in analyzing trajectory data in badminton. Trajectories in sports, such as the movement of players and balls, contain rich information on player behavior and thus have been widely analyzed by coaches and analysts to improve the players' performance. However, existing visual analytics systems often present the trajectories in court diagrams that are abstractions of reality, thereby causing difficulty for the experts to imagine the situation on the court and understand why the player acted in a certain way. With recent developments in immersive technologies, such as virtual reality (VR), experts gradually have the opportunity to see, feel, explore, and understand these 3D trajectories from the player's perspective. Yet, few research has studied how to support immersive analysis of sports data from such a perspective. Specific challenges are rooted in data presentation (e.g., how to seamlessly combine 2D and 3D visualizations) and interaction (e.g., how to naturally interact with data without keyboard and mouse) in VR. To address these challenges, we have worked closely with domain experts who have worked for a top national badminton team to design ShuttleSpace. Our system leverages 1) the peripheral vision to combine the 2D and 3D visualizations and 2) the VR controller to support natural interactions via a stroke metaphor. We demonstrate the effectiveness of ShuttleSpace through three case studies conducted by the experts with useful insights. We further conduct interviews with the experts whose feedback confirms that our first-person immersive analytics system is suitable and useful for analyzing badminton data.