Physical and digital media usage patterns on interactive tabletop surfaces.

Full text

Page 1

Physical and Digital Media Usage Patterns on
Interactive Tabletop Surfaces

Jürgen Steimle1, Mohammadreza Khalilbeigi1, Max Mühlhäuser1, James D. Hollan2

1 Technische Universität Darmstadt
64289 Darmstadt, Germany
{steimle, khalilbeigi, max}@tk.informatik.tu-
darmstadt.de

ABSTRACT
Concurrent interaction with physical and digital media is
ubiquitous in knowledge work. Although tabletop systems
increasingly support activities involving both physical and
digital media, patterns of use have not been systematically
assessed. This paper contributes the results of a study of
spatial usage patterns when physical and digital items are
grouped and sorted on a tabletop work surface. In addition,
analysis reveals a dual character of occlusion, involving
both inconvenient and desirable aspects. We conclude with
design implications for hybrid tabletop systems.
ACM Classification: H5.2 [Information interfaces and
presentation]: User Interfaces.
General terms: Design, Human Factors
2 University of California, San Diego
San Diego, CA 92093, USA
hollan@hci.ucsd.edu


interested in use cases in which physical and digital media
are both involved in an activity. A number of tabletop sys-
tems have been developed [38, 26, 16, 11, 7, 32, 13] that
support such hybrid use but myriad interaction design chal-
lenges remain. Prior work has not, for example, systemati-
cally studied the affordances and tradeoffs involved in how
users spatially arrange and group items or how they deal
with occlusion of screen contents by physical items.
This paper documents users’ spatial usage patterns while
grouping and sorting physical and digital items on an inter-
active surface. It also investigates the impact of physical
occlusion. The remainder of the paper is organized as fol-
lows: first, we discuss related work, next we describe the
methodology and the main results, and finally we discuss
key implications of our results for the design of hybrid tab-
letop user interfaces.
Keywords
Interactive surface, tabletop display, paper, physical media.
INTRODUCTION
The ways we interact with information continue to evolve
along with device form factors, computing speed, and ubiq-
uity of access to networked information. Digital information
can be displayed on the surface of desks, tables, or walls
and can be manipulated using multitouch interfaces [41, 12]
and tangible input devices [8, 34, 37]. Interactive surfaces
are becoming common components of information-based
activities [38, 11, 23]. Despite these changes and many oth-
er computing advances, use of paper in knowledge work, as
elegantly documented by Sellen and Harper [31], remains
pervasive. Paper provides key affordances not provided by
current computers. For example, most users prefer reading
printed documents to on-screen reading [1, 31]. As a con-
sequence of this and other affordances of paper (e.g. ease of
annotation and portability), paper is often involved concur-
rently with digital media. Similarly, while reading a printed
document, users increasingly access digital information.
Because of ubiquity of its occurrence, we are particularly
RELATED WORK
The present work is situated at the intersection of interac-
tive surfaces [25, 6, 41] and tangible user interfaces [8, 14].
Affordances of digital and paper media
Empirical research [31] shows that paper provides key af-
fordances for working with documents: it embodies and
makes information tangible, supports easy bimanual naviga-
tion and organization, facilitates communication and colla-
boration, provides high resolution and high contrast for
easy reading and a better viewing angle than horizontal
displays [22], and enables intuitive annotation with a pen
that is flexible and smoothly integrated with reading.
Multitouch interaction techniques on interactive displays
are also designed to exploit direct physical manipulation
[29, 33, 40]. A comprehensive study by Terrenghi et al.
[33] compared the affordances of interacting with digital
and physical media on tabletops. Analysis of basic tasks
with photos revealed distinctive differences in the ways
people interact with digital and physical photos. Most im-
portantly, they noted that one-handed interaction predomi-
nated with digital media even though multitouch interaction
is specially designed to support bimanual interactions. This
might be partly explained by the lack of haptic feedback
and by the restrictions of a two-dimensional surface. Piper
and Hollan [24] compared the affordances of digital educa-
tional material on tabletops with traditional paper handouts
in collaborative study practices. They found that the more
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that cop-
ies bear this notice and the full citation on the first page. To copy other-
wise, to republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee.
ITS’10, November 7–10, 2010, Saarbrücken, Germany.
Copyright 2010 ACM 978-1-4503-0399-6/10/11...$10.00.

Page 2

ephemeral character of digital ink annotations encouraged
students more than with paper to spontaneously create
drawings to support discussion.
In contrast to studies that compare the affordances of physi-
cal vs. digital media, our focus is to understand the com-
bined use of physical and digital media as a basis for de-
signing hybrid interactive surfaces.
Hybrid interactive surfaces
Interactive desks, tabletops and walls enable users to mani-
pulate physical and digital media on their surface [38, 26,
16, 11, 7, 32, 13]. The technical set-up typically consists of
a top- or rear-projected display of digital information. In-
formation from physical media can be captured with a cam-
era or digital pen. In a number of systems, information can
move between digital and physical environments [38, 11,
13]. Moreover, some systems allow users to create and fol-
low hyperlinks between information contained on physical
and digital media [26, 7, 32]. Prior work though has not
systematically assessed the patterns of media interaction
that emerge in hybrid physical-digital settings.
Managing space on the surface
Important and fundamental aspects of working on any sur-
face are how the space is used and how information items
can be arranged and manipulated. Scott et al. [30] per-
formed an in-depth analysis of territoriality in collaborative
tabletop workspaces. They found personal territories to be
different from shared territories. Areas for storing items are
integrated within these territories and change their positions
over time. Other studies analyzed the impact of table size
[27] and orientation of media [18].
Efficiently managing tabletop space and effectively assist-
ing semantic organization commonly involves creating
groups of items. Malone [20] in an early study examined
how people organize papers on their physical desktops.
Other studies [39] have examined how people manage pa-
per archives and the tradeoffs involved between piling and
filing. One consistent finding is that piling is a lightweight,
casual activity, involving less overhead than filing. Con-
cepts of piling on the desktop [21, 4, 2] and interactive tab-
letops [28, 3] have been presented. Examination of the lite-
rature reveals no studies of space management and group-
ing on hybrid interactive surfaces.
Physical occlusion of screen contents
Only a small number of studies have considered the prob-
lem of occlusions generated by physical objects on a dis-
play surface. Some work has dealt with occlusions generat-
ed by the user’s hand and forearm when working with a pen
on a tablet display [35]. Brandl et al. [5] proposed pie me-
nus that automatically adapt their orientation to minimize
occlusion created by the user’s hands. Other work deals
with occlusion generated by tangible objects [19]. It is im-
portant to note that all these approaches commonly consider
occlusion solely as a bothersome phenomenon. As we de-
scribe below, a detailed analysis of occlusion shows that
occlusion can also function to help users in expressing
meaning.
METHOD
Participants
We recruited 10 volunteer participants (5 female, 5 male; 8
right-handers, 2 left-handers). All were experienced know-
ledge workers, from both technical and non-technical back-
grounds (computer scientists, psychologists, jurists). All but
2 had little or no experience with tabletops. Their ages
ranged from 24 to 46 years with an average of 32. No com-
pensation was provided.
Technical Set-up
The study used an interactive tabletop of 130x105 cm size
with a display size of 100x60 cm. This is representative of
the space available on a typical desk. The rear-projection
was full HD resolution (1920x1080 pixels). Rear-projection
is currently the most common form of tabletop display.
Moreover, in contrast to top projection, this ensures that our
findings can be transferred to horizontal flat-panel displays.
The participants were seated. They could interact with one
digital document at a time using multitouch gestures for
moving (1 finger drag), enlarging/shrinking (2 finger pinch
gesture) and rotating (3 finger rotation). No physics simula-
tion was applied. Printed documents could be placed and
manipulated on the display surface and also on the sur-
rounding non-display areas of the table. The setting is
shown in Figure 1.
Materials
Participants were given 12 single-page documents. Each
document contained textual biographical information about
a popular musician. We used textual documents instead of
photos (as used in other studies) for two reasons. First,
working with documents is a more common and thus more
representative knowledge work task. Second, since docu-
ments are typically larger than photos this increased oppor-
tunities for occlusion. To create a hybrid media setting, six
of the documents were printed on paper (A4 size) and the

Figure 1. The study setting used a hybrid tabletop
including physical and digital media.

Page 3

remaining six were displayed on the tabletop display. In a
typical work setting, users might more commonly not have
a common set of information items arrayed across digital
and physical media but rather different types of items in
each (e.g., paper articles or notes used while consulting web
information). The motivation for splitting the set of docu-
ments between paper and digital sets was to simplify analy-
sis. We also wanted to ensure adequate space was available
for various arrangements. In each task, physical documents
and digital documents were initially presented as two sepa-
rate adjacent piles positioned at the center of the tabletop.
All participants used the same set of documents.
Procedure
Each user participated in a single-user session of 40 mi-
nutes average length (SD = 8 min.). The specific tasks we
asked participants to perform are common in knowledge
work with printed and digital documents: a grouping task
(inspired by [33]), a sorting task, and a search and compar-
ison task. These tasks enabled observation of component
activities such as selecting, reading, comparing, and moving
items.
First, gestures for interaction with digital documents (move,
enlarge/shrink, rotate) were demonstrated and participants
became familiar with them. They then performed the group-
ing task and were asked to browse the documents and
create three groups: those describing artists they like, don’t
like, or are unsure about. A sorting task followed in which
they sorted all documents in a linear order according to a
fact indicated in the document text, e.g the date of birth of
the artists or their respective numbers of albums. The final
search and comparison task required finding all albums
which were released in the same year as one specific album
by one artist. Therefore, participants had to compare all
documents with one specific paper document and note
down the names of the albums.
The grouping and sorting tasks were each performed first
using only the paper documents, then only the digital doc-
uments, and finally with both document types. To help
compensate for learning effects, different criteria for group-
ing and sorting were used in each iteration. The search and
comparison task was directly performed with both docu-
ment types. While performing the tasks, subjects were
asked to think aloud and after each task semi-structured
interviews were conducted. All sessions were videotaped
using a camera mounted above the table and field notes
were recorded.
Data Analysis
We coded 6.75 hours of video recordings for relevant be-
havior, iteratively refining the coding scheme. This led to
the set of categories used below to describe behavioral pat-
terns. Using the resulting scheme, we coded each instance
of selecting an item, grouping and sorting items, and zoom-
ing and moving items. We also coded the duration and spa-
tial placement patterns as participants worked with items.
The resulting data supports not only qualitative analyses but
also allowed us to derive statistical patterns. In describing
these patterns below, we augment them in some instances
with statements made in the interviews. While our primary
analysis focuses on hybrid use, i.e. combined use of printed
and digital documents, we also contrast this with patterns
found in tasks that involved using only physical or only
digital media.
RESULTS
We first describe the affordances of the combined use of
physical and digital media. This includes the general spatial
patterns of hybrid use and patterns of sorting and grouping.
We moreover detail users’ patterns of interacting with items
and provide an analysis of how physical occlusion affects
hybrid use.
Spatial Patterns of Hybrid Use
We are especially interested in how tabletop space is used
in a hybrid setting. Initial analysis of the video recordings
showed that participants distinguished between territories
for working with (e.g., reading, comparing) and for storing
items. As also reported by [30], the boundaries between
these areas were quite flexible. We coded usage locations
for each participant and aggregated this data. Figure 2
shows this aggregated data for the hybrid search and com-
parison task. Darker locations depict areas that were used
by more participants. The wooden frame of the table is in-
dicated by a gray border that encloses the white display
area. The dark gray circle indicates the position of the user.
Figure 2 (upper part) shows the areas used for working with
items. Note that physical and digital items tend to be used at
different tabletop locations (graphically summarized in Fig-
ure 2 a and b). Physical items were used nearer to the user
and more commonly to the left of the user. At times they
were not directly on top of the display but on the edge of
the table or even partially jutting off the table surface. In
contrast, digital items were more tightly clumped, primarily
positioned to the right and further away from the user than
physical items. Note also that the working area for physical
items involved a larger portion of the table, while the area
for digital items was almost exclusively confined to the
right of the table display. A reason might be that the haptic
feedback of physical documents affords cognitive offload-
ing, whereas the direct-touch manipulation of digital media
requires visual attention. This fact could explain why the
participants delegated macrometric tasks to the non-
dominant hand, and micrometric ones to the dominant hand
[10].

Page 4

Working Areas
a) Physical items b) Digital items c) Hybrid use
Storage Areas
h) Physical items i) Digital items k) Hybrid use
d) Physical (focus item) e) Physical (other items)
f) Physical (on the surface ) g) Physical (above the surface)
Figure 2. Activity maps depict areas where items were used on the hybrid surface in the hybrid search and compar-
ison task. Darker areas signify use by more participants.

Page 5

Figure 2 (lower part) depicts the areas used for storing
items (i.e. locations of temporarily placed items for later
use and of final placements). While the working areas were
centered more in the middle of the table, storage is situated
towards the outer left and right edges. Also, unlike working
areas, storage area placement patterns are similar across
physical, digital, and hybrid use. Note the clear difference
in use of the tabletop area between storage and working,
especially for physical and digital items. Again placement
of physical items extends beyond the table surface. It is also
important to notice that in contrast to working areas, physi-
cal and digital storage areas were co-located, with place-
ment of physical and digital items overlapping.
In addition to working and storage areas, we found a third
zone. In the search and comparison task one physical item
had to be compared with all other items. Figure 2 d) shows
that this item was used in a distinctive area, one differing
from the other physical items involved in the task.
Figure 2 f) and g) show that physical items were not only
used on the surface of the table but that there was also sig-
nificant manipulation of these paper items above the table.
We classified usage as being above-the-table if participants
held a physical item (or collection of physical items) above
the table for a period of time longer than was required to
pick it up and directly placing it elsewhere. The illustration
further shows that items in the air were held nearer the user
and more to the left than physical items on the table surface.
Interacting with Items
Physical Items. Terrenghi et al. [33] compared how users
interact with physical and digital media. They found that
users tend to pick up physical items to bring them closer to
their eyes. While their study did not involve piles, we ob-
served that users manipulated individual physical items as
well as piles of multiple items. When participants wanted to
interact with an element, in the vast majority of cases they
first selected a pile (45 cases) or an individual item (256
cases) by picking it up for holding it in their hands above
the table. Only then they engaged in further interactions,
such as reading or comparing information. Much less fre-
quently, participants did not pick up the item and interacted
directly on the surface with the pile (8 cases) or the individ-
ual item (20 cases).
The time span during which items were held in the hands
above the table varied considerably. Sometimes an item was
picked up, read and directly placed back on the table. But in
most cases, the participants kept items in their hands for
longer periods. In the grouping and sorting tasks, for exam-
ple, most participants initiated the task by picking up all
physical items and kept holding them while sequentially
placing one item after another on the table to create an ar-
rangement. Similarly in the search and comparison task, 7
of the 10 participants held the item to be compared conti-
nuously while comparing it with the other physical items
placed on the table.
Digital Items. When interacting with digital items, partici-
pants in almost all cases enlarged the item using the zoom-
ing pinch gesture. In order to display the text in a readable
size, digital documents needed to be enlarged, even though
the resolution of our tabletop was higher (50 dpi) than that
of most current tabletops. Enlarging was accomplished at
the initial place of the document; only 2 participants (14
instances) dragged digital items from their initial position to
a place nearer to themselves before enlarging, and only
rarely did they lean their whole body over the table.
Hybrid Interaction. Simultaneously interacting both with
physical and with digital items occurred in the search and
comparison task. Participants in this task had to compare
information contained on a physical item with information
from all other items. While all participants interacted with
digital items by enlarging them, strategies with physical
documents varied. Three participants permanently held the
paper item above the surface with the non-dominant hand.
Seven participants placed the physical item onto the table
surface near the digital items. This contrasts with how par-
ticipants compared only physical items, where the majority
held items in the air above the surface.
Hybrid Grouping
Unordered Group
We analyzed how participants interacted with groups that
contain both physical and digital items. We were particular-
ly interested in whether users prefer representing a hybrid
group as two separate, possibly adjacent groups, each con-
taining only physical or only digital media (spatially-
separated representation), or if they prefer arrangements

Figure 3. Examples of hybrid piles on the tabletop. Left: Neat pile and messy pile. Center: Spread-out representation,
which affords getting an overview and comparing pages. Right: Result of grouping task created by one participant.

Page 6

that integrate both types of items at the same place (spatial-
ly-integrated representation).
All participants grouped physical and digital items in a sin-
gle spatially-integrated representation, a layered arrange-
ment that we call a hybrid pile. This is an arrangement of
(largely or entirely) overlapping physical and digital docu-
ments. Figure 3 (right) shows an example of three hybrid
piles created by one participant in the grouping task.
The spatially-integrated hybrid representation enabled par-
ticipants to express a close relationship between physical
and digital documents. Comments during interviews also
underscored that participants preferred this spatially-
integrated hybrid representation to other representations.
For example, participant 9 explicitly mentioned that "hav-
ing all [digital and physical items] in a pile makes sense".
Participant 3 stated "I don’t want digital documents to be
[automatically] slid out" when placing physical documents
on them. Due to the high degree of spatial proximity of
overlapping physical and digital media, the overall gestalt
[17] of a hybrid pile clearly expresses that the elements of
the group belong to one conceptual group.
The creation and sequential browsing of hybrid piles ap-
peared easy, fluid and dynamic. All but one participant
created hybrid piles by first placing digital items on top of
each other and then placing physical items on top of the
digital items. To browse hybrid piles, all participants se-
quentially picked up the topmost item of the pile or dragged
it away, starting with the physical before going on to the
digital items.
Hybrid piles are more static than physical piles. Pure physi-
cal piles were frequently moved, whereas we observed no
instance of an entire hybrid pile being moved. In addition,
we observed frequent transformations of physical piles.
Different pile organizations simplified specific practices.
Placing all items neatly on top of each other saves screen
real estate and supports interacting with the group as a
whole, e.g. for moving and storing. In contrast, spreading
out the items affords looking at the contents of several doc-
uments in parallel, e.g. for getting an overview, comparing
and sorting items. In the pure physical condition, we fre-
quently observed participants performing these transitions
with one quick and intuitive bimanual movement. In con-
trast, in the hybrid case, we observed only two instances of
such transitions. Most likely this is due to the fact that each
digital item of the pile had to be displaced individually to
transform the arrangement.
Ordered Group
In the second task, the participants had to sort physical and
digital documents. This allowed us to analyze spatial ar-
rangements chosen for representing ordered hybrid groups.
In contrast to unordered groups, sorted hybrid groups could
not be represented with a single hybrid pile, as the sorting
could require placing digital items on top of physical ones.
Instead of using vertical piling, 9 participants represented
the ordered sequence by a horizontal arrangement. The up-
per left corner of a document indicated its position in the
sequence whereby each document was placed to the right of
its predecessors. In contrast, 1 participant expressed the
ordered sequence by a combination of several adjacent hy-
brid piles (Fig. 4). Each pile in itself was ordered. Each
time a digital item would have to be placed onto a physical
document, a new hybrid pile was created beneath or below
the preceding pile.
All participants viewed the hybrid sorting task as complex
and the result as unsatisfactory. Each approach has draw-
backs. The horizontal arrangement leads to a high degree of
occlusion, since the tabletop did not provide enough space
for placing all documents besides each other without over-
laps. Moreover, this arrangement does not aid rearrange-
ments. Inserting a document or changing its position within
the sequence typically required rearranging many docu-
ments or even reorganizing the entire structure. The other
approach, using adjacent hybrid piles, provided even less
flexibility for rearrangements. As a further drawback, this
representation conflicted with a conceptualization of one
single ordered group. The resulting gestalt of several adja-
cent piles indicates not one but several different groups.
Physical Occlusion
We have identified and described patterns of use in hybrid
tabletop settings. This provides a frame of reference for
discussing how occlusion influences the activities of select-
ing, zooming, moving, and grouping items.
Selecting Occluded Items
Occlusion was most salient in our data when users wanted
to access a digital item partly or entirely occluded by
physical items. In order to analyze strategies used for
accessing digital items, we characterized each selection

Figure 4. Representation of one single hybrid group
in a sorted arrangement.

Figure 5. Strategies used for selecting occluded
digital items.

Page 7

activity from the video recordings. We noted the strategy
used, characteristics of the occlusion (percentage of
occlusion, number of occluding items and whether these
items were part of a sorted arrangement), as well as features
of the surrounding tabletop surface (percentage covered by
physical and by digital items).
Figure 5 (left) depicts the four strategies we identified and
their frequencies. The most frequent strategy (42%) was
picking up the occluding physical item(s) so that the digital
item became visible. A second strategy (28%) involved
dragging the digital item to a non-occluded tabletop zone,
the physical item resting untouched. Moving the occluding
physical item(s) instead of the digital one was a third
strategy (27%). A final strategy (3%) consisted of moving
both the physical and the digital items concurrently to
remove occlusion.
Participants leveraged the affordance of physical items to
be moved not only on a flat surface but in all three
dimensions. In addition, they performed a considerable
number of bimanual hybrid interactions (28% of all
instances). These involve manipulating physical items with
one hand, while the other hand interacts with digital items.
While we observed symmetric bimanual interactions,
asymmetric interactions were more frequent. Participants
typically picked up the physical item(s) with one hand
(typically the non-dominant hand) and kept them in this
hand while using the other hand to interact with the digital
item (e.g. zooming). Finally, they placed the physical
item(s) back onto the surface.
The selection of one of the aforementioned strategies highly
depended on the degree of occlusion. If the amount of
occlusion was higher, users tended to pick up the physical
item(s), whereas in cases of less occlusion, users preferred
to move physical or digital items on the surface. This
correlation between degree of occlusion and picking up
physical item(s) was highly significant (r=.53, p < .001,
N=60). It was also highly significant for the degree to
which the area surrounding the digital item is occluded by
physical items (r=.48, p < .001, N=60).
We also analyzed how the selection affects the overall
hybrid arrangement of items. Groups should remain groups
and sorted sequences should not be altered by a selection
activity. We observed that the overall arrangement of items
remained the same before and after the selection if the
digital item was occluded by only one item or by several
unordered items. In these cases, the occluding items could
be easily placed back in the correct arrangement after
selection. In contrast, if the occlusion was generated by a
higher number of items that by their arrangement expressed
a sorting, it was much more complicated to place them back
in the correct arrangement. In this case the structure was
frequently destroyed by selecting the occluded item.
Zooming
As mentioned above, participants frequently enlarged digi-
tal items in order to read the contents. If one or more physi-
cal items are close by, enlarging can lead to occlusion. We
observed 23 instances of occlusion resulting from enlarg-
ing. On average 30% of the item’s surface was occluded
(SD = 15%), with a maximum occlusion of 70%.
Despite even considerable occlusion, participants did not
find it to be problematic. Since the task (for grouping, sort-
ing, comparing) required access to specific information that
was contained at similar positions on each document, par-
ticipants could perform the pinch gesture to enlarge the
document where the information was assumed to be lo-
cated. While the document was being enlarged, the outer
portions of the document expanded and moved under any
nearby physical items, but the position of interest remained
at the same non-occluded location and could easily be read.
If the task required skimming or reading the entire docu-
ment, occlusion would be more problematic.
Enlarging items that were partially occluded at their outer
edges resulted in a cluttered view. The physical arrange-
ment did not clearly convey whether the occlusion was ac-
cidental or the items were purposely arranged in an over-
lapping manner to express a relationship. Perhaps to avoid
this ambiguity, in 65% of all instances participants imme-
diately scaled down the item once they had read the infor-
mation. Often enlarging, reading and scaling were per-
formed in one integrated interaction using a single conti-
nuous pinch gesture. It is certain that zooming will be a less
frequent activity in future tabletops, which will offer a
higher resolution. Nevertheless, enlarging a view or an in-
dividual element is an important activity with many types of
documents and visualizations, independently of the resolu-
tion available.
Moving
Moving items is a core interaction for managing a work-
space. However, when using touch interaction to drag a
digital item, one cannot move it across positions occluded
by physical items. We were interested in how users would
deal with this situation.
Figure 6 (left) depicts the three different strategies we
observed and their frequencies. The most frequent strategy
(48%) consisted of moving the digital item around the phys-
ical item. If this move-around approach would be tedious or
even impossible, participants frequently leveraged hybrid
bimanual interaction. In order to remove any obstacles in
the motion path, they picked up the physical item with the
non-dominant hand, moved the digital item with the domi-

Figure 6. Strategies used for moving digital items
over occluded positions.

Page 8

nant hand to its target position, and then placed the physical
item back onto the table. This strategy was used not only
when a single item created the occlusion but also when en-
tire piles had to be picked up. A third strategy involved first
moving the physical item on the surface to free the path
before dragging the digital item to its target position.
Piling
Participants’ interactions with hybrid piles demonstrate that
they intentionally generate occlusion by placing physical
media items on top of digital media items. This spatial or-
ganization serves to represent and highlight a meaningful
relation between the elements of a pile and thus rather than
being problematic provides a visible representation of a
conceptual relationship. As a consequence, it seems appro-
priate to distinguish between semantically meaningful forms
of occlusion and forms that do not express semantic mean-
ing. Occlusion in a hybrid pile can be semantically mea-
ningful (and desired by the user when creating the pile).
In addition to this positive semantic aspect of occlusion,
problematic features were also evidenced. Once a hybrid
pile was created, physical items often entirely covered the
underlying digital items. As a consequence, there was no
indication that there were digital items hidden under physi-
cal ones. In interviews, five participants reported this to be
highly problematic. In order to indicate the presence of dig-
ital items, three participants slightly displaced the physical
items of a hybrid pile so that the digital items remained
visible.
DESIGN IMPLICATIONS AND GUIDELINES
Based on the above findings, we propose the following
guidelines for designing hybrid tabletop systems.
Support physical item use above and around the surface
Physical items were used not only on top of the display sur-
face but were frequently placed on the margins of the table
(even jutting off the surface) as well as being frequently
picked up and held in the air above the table. The physical
interaction space is larger than the digital interaction space,
extending in all three dimensions. As a consequence hybrid
tabletop systems that plan to support use of physical items
should track them not only on and directly above the sur-
face but also on and above the table edges and even in front
of the table. This is not possible in current tabletop systems
that track tangibles with fiducial markers using a camera
below the table. Tracking should cover areas beyond the
tabletop; for example, our findings suggest areas in front of
the display and to the side of the table associated with the
user’s non-dominant hand.
Support physical, digital and hybrid information groupings
Our study demonstrates once again that people can work
very flexibly with collections of physical documents. Users
easily interact with and move groups of physical items to
accomplish tasks and manage workspace organization. Hy-
brid piles without better technological support lack the ease
and the flexibility for moving and rearranging that tradi-
tional piles of paper documents afford. Therefore, it is par-
ticularly important that tabletop systems provide easy natu-
ral mechanisms for users to create groups of digital as well
as hybrid items and enable interaction with them as single
entities. Soap bubbles is one promising metaphor [36, 15].
For a sorted group of hybrid digital and physical items,
however, a hybrid pile is not currently a very effective re-
presentation. Future work needs to identify new representa-
tions for hybrid piles and mechanisms of interaction. In
particular, how users might easily transform items between
physical and digital representations should be explored so
the advantages of each representation might be exploited.
Support transitions between pile representations
We frequently observed that users transformed piles of
physical items to another arrangement to assist the task be-
ing performed. For example, a “tidy” pile affords interac-
tion with the group as whole, whereas a juxtapositioned or
partially overlapping arrangement affords an overview of
the items as well as reading or comparing them. With phys-
ical items, transformations were fluid and typically per-
formed with one bimanual movement. Similarly fluid transi-
tions between different representations should be supported
for digital and hybrid groups. The digital members of a hy-
brid pile could, for example, automatically imitate arrange-
ments that result from moving physical members [15].
If a hybrid pile is transformed to a spatially-separated re-
presentation (i.e. split into separate physical and digital
parts), our results suggest placing the digital portion to the
side associated with the user’s dominant hand. This enables
hybrid interactions using the non-dominant hand for mani-
pulating physical items and the dominant hand for interact-
ing with digital items.
Provide awareness on and support interaction with oc-
cluded digital items
In a hybrid setting, digital items can be occluded by physi-
cal items, losing any indication of their presence. Hybrid
tabletop systems should indicate the presence of occluded
items, for instance by displaying a halo around a physical
item that fully occludes one or more digital items.
When new digital items are added by the computer they
should be automatically displayed at a non-occluded posi-
tion on the screen if possible. However, if the user occludes
a digital item (e.g., by placing a physical item over it), it
should not be automatically repositioned to a non-occluded
area. Our findings show that frequently occlusion is desired
by users and can be semantically meaningful.
Nevertheless, selecting occluded items should be made eas-
ier to accomplish. Interfaces should permit users to tempo-
rarily display occluded items (perhaps as the result of ho-
vering) at non-occluded positions to enable easy selection.
As another example, a digital handle could be displayed at
the edge of a physical item to make it possible to easily pull
out the occluded digital item. Once the handle is released,
the digital item could automatically snap back to its original
position. Further, hybrid tabletop systems should support
moving digital items over occluded positions without the
need to remove the occlusion.

Page 9

Support occlusion-aware enlarging of digital items
The study has shown that due to the limited resolution of
current tabletop systems, users frequently enlarge textual
documents to be readable. If the surrounding area is oc-
cluded, enlarging can lead to partial or complete occlusions
as well as to a cluttered arrangement. Perhaps because of
this, in most cases users immediately scaled down the item
after reading its contents. Hybrid tabletop systems should
provide efficient support for interacting with digital items in
occluded contexts. A zoom lens could provide help for easi-
ly and quickly focusing on parts of an item without causing
the entire item to be enlarged and occluded. Fisheye views
[9] or automatically mirroring an item to a non-occluded
area in an enlarged view could also support access.
Limitations of the study
While most findings can be generalized to other cases of
hybrid tabletops, some limitations apply due to the setting
of the study. The initial presentation of documents as two
adjacent piles might influenced the grouping patterns cho-
sen by participants. While this bars us from deriving quan-
titative results, the study shows that the pattern of hybrid
piles exists and that it can be efficiently integrated into
work practice. This is inline with previous findings on pil-
ing [20, 21, 39]. As a second limitation, we assume that the
findings related to zooming are influenced by the still rather
low resolution of current tabletops. Future work should
examine how users focus on digital documents on higher-
resolution tabletops. Finally, the study only addressed rear-
projection setups. We leave to future work to examine hybr-
id document usage in front-projection setups.
CONCLUSIONS
This paper contributes to understanding activities in which
physical and digital media are both used on interactive tab-
letops. We explored the affordances and tradeoffs involved
in how users spatially arrange and group items and how
they deal with occlusion of screen contents by physical
items. The results revealed a spatially integrated use of
physical and digital items. While the areas for working with
both types of media differed (in general, physical items
were placed nearer the user than digital items and were less
clumped), the same areas are used for storing items. In par-
ticular, participants frequently stored groups of items in a
hybrid pile arrangement. Users are willing to physically
occlude digital contents in order to better manage their
workspace and to make meaningful collections. When cop-
ing with occlusion, participants had highly effective strate-
gies that rely on bimanual interaction.
Based on the study results, we proposed design recommen-
dations for future tabletop interfaces. The results highlight
the need to support physical interaction not only on, but
also above and around the surface. Moreover, future table-
top interfaces should enable the user to create and flexibly
manipulate hybrid groups that include both physical and
digital items. In addition, systems should appreciate the
positive semantic aspects of occlusion as well as trying to
confront the problems occlusion can engender. Finally,
there is a need to support better access to occluded items
and enable efficient interaction with them (e.g., in selection
and moving). This is an important aspect of bridging the
gap between the physical and the digital worlds.
We have explored a setting in which digital items could be
moved and arranged on the tabletop much like physical
items. In addition to suggestions enumerated above, future
studies should also assess the effects of occlusion with tab-
letop interfaces in which digital media cannot be freely ar-
ranged, such as a large map or graph.
ACKNOWLEDGMENTS
We thank the participants of the study and acknowledge
Erwin Aitenbichler for technical support. This work was
supported in part by the German Research Foundation
(DFG), SAP Research Germany, and NSF grant 0729013.
REFERENCES
1. Adler, A., Gujar, A., Harrison, B. L., O’Hara, K., and
Sellen, A. A diary study of work-related reading: design
implications for digital reading devices. In Proc. CHI
’98, 1998, pp. 241–248.
2. Agarawala, A., and Balakrishnan, R. Keepin’ it real:
pushing the desktop metaphor with physics, piles and
the pen. In Proc. CHI ’06, 2006, pp. 1283–1292.
3. Aliakseyeu, D., Subramanian, S., Lucero, A., and Gut-
win, C. Interacting with piles of artifacts on digital
tables. In Proc. AVI ’06, 2006, 159–162.
4. Bauer, D., Fastrez, P., and Hollan, J. Computationally-
enriched ’piles’ for managing digital photo collections.
In Proc. 2004 IEEE Symposium on Visual Languages -
Human Centric Computing, 2004, pp. 193–195.
5. Brandl, P., Leitner, J., Seifried, T., Haller, M., Doray,
B., and To, P. Occlusion-aware menu design for digital
tabletops. In CHI 2009 Extended Abstracts on Human
Factors in Computing Systems, 2009, pp. 3223–3228.
6. Dietz, P., and Leigh, D. Diamondtouch: a multi-user
touch technology. In Proc. UIST ’01, 2001, pp. 219–
226.
7. Everitt, K., Morris, M. R., Brush, A. B., and Wilson, A.
Docudesk: An interactive surface for creating and rehy-
drating many-to-many linkages among paper and digital
documents. In Proc. IEEE Tabletops and Interactive
Surfaces 2008, 2008, pp. 25–28.
8. Fitzmaurice, G. W., Ishii, H., and Buxton, W. A. S.
Bricks: laying the foundations for graspable user inter-
faces. In Proc. CHI ’95, 1995, pp. 442–449.
9. Furnas, G. W. Generalized fisheye views. ACM SIGCHI
Bull. 17, pp. 16–23.
10. Guiard, Y. Asymmetric division of labor in human
skilled bimanual action: The kinematic chain as a mod-
el. J. Motor Behavior 19, 4, 1987, 486–517.
11. Haller, M., Leithinger, D., Leitner, J., Seifried, T.,
Brandl, P., Zauner, J., and Billinghurst, M. The shared
design space. In Proc. ACM SIGGRAPH 2006 (2006).

Page 10

12. Han, J. Y. Low-cost multi-touch sensing through fru-
strated total internal reflection. In Proc. UIST ’05,
2005, pp. 115–118.
13. Hartmann, B., Morris, M. R., Benko, H., and Wilson,
A. D. Pictionaire: supporting collaborative design work
by integrating physical and digital artifacts. In Proc.
CSCW ’10, 2010, pp. 421–424.
14. Ishii, H., and Ullmer, B. Tangible bits: towards seam-
less interfaces between people, bits and atoms. In Proc.
CHI ’97, 1997, pp. 234–241.
15. Khalilbeigi, M., Steimle, J., and Mühlhäuser, M. Inte-
raction techniques for hybrid piles of documents on in-
teractive tabletops. In CHI 2010 Extended Abstracts on
Human Factors in Computing Systems, 2010,
pp. 3943–3948.
16. Klemmer, S. R., Newman, M. W., Farrell, R., Bilezik-
jian, M., and Landay, J. A. The designers’ outpost: a
tangible interface for collaborative web site design. In
Proc. UIST ’01, 2001, pp. 1–10.
17. Koffka, K. Principles of Gestalt Psychology. Harcourt-
Brace, 1935.
18. Kruger, R., Carpendale, S., Scott, S. D., and Greenberg,
S. How people use orientation on tables: comprehen-
sion, coordination and communication. In Proc.
GROUP ’03, 2003, pp. 369–378.
19. Leithinger, D., and Haller, M. Improving menu interac-
tion for cluttered tabletop setups with user-drawn path
menus. In Proc. IEEE TABLETOP ’07, 2007, pp. 121–
128.
20. Malone, T. W. How do people organize their desks?:
Implications for the design of office information sys-
tems. ACM Trans. Inf. Syst. 1, 1, 1983, 99–112.
21. Mander, R., Salomon, G., and Wong, Y. Y. A “pile”
metaphor for supporting casual organization of informa-
tion. In Proc. CHI ’92, 1992, pp. 627–634.
22. Morris, M. R., Brush, A. B., and Meyers, B. R. Reading
revisited: Evaluating the usability of digital display sur-
faces for active reading tasks. Proc. IEEE TABLETOP
’07, 2007, pp. 79–86.
23. Morris, M. R., Lombardo, J., and Wigdor, D. We-
search: supporting collaborative search and sensemak-
ing on a tabletop display. In Proc. CSCW ’10, 2010,
pp. 401–410.
24. Piper, A. M., and Hollan, J. D. Tabletop displays for
small group study: affordances of paper and digital ma-
terials. In Proc. CHI ’09, 2009, pp. 1227–1236.
25. Rekimoto, J., and Saitoh, M. Augmented surfaces: a
spatially continuous work space for hybrid computing
environments. In Proc. CHI ’99, 1999, pp. 378–385.
26. Robinson, P., Sheppard, D., Watts, R., Harding, R., and
Lay, S. Animated paper documents. In Proc. Proceed-
ings of the 7th International Conference on human-
computer interaction (HCI ’97), 1997, pp. 655–658.
27. Ryall, K., Forlines, C., Shen, C., and Morris, M. R. Ex-
ploring the effects of group size and table size on inte-
ractions with tabletop shared-display groupware. In
Proc. CSCW ’04, 2004, pp. 284–293.
28. Scott, S. D., Carpendale, M. S. T., and Habelski, S.
Storage bins: Mobile storage for collaborative tabletop
displays. IEEE Computer Graphics and Applications
25 (4), 2005, 58–65.
29. Scott, S. D., Grant, K. D., and Mandryk, R. L. System
guidelines for co-located, collaborative work on a table-
top display. In Proc. ECSCW’03, 2003, pp. 159–178.
30. Scott, S. D., Sheelagh, M., Carpendale, T., and Inkpen,
K. M. Territoriality in collaborative tabletop workspac-
es. In Proc. CSCW ’04, 2004, pp. 294–303.
31. Sellen, A. J., and Harper, R. H. The Myth of the Paper-
less Office. MIT Press, Cambridge, 2003.
32. Steimle, J., Brdiczka, O., and Mühlhäuser, M. Co-
Scribe: Integrating Paper and Digital Documents for
Collaborative Knowledge Work. IEEE Transactions on
Learning Technologies 2 (3), 2009, pp. 174–188.
33. Terrenghi, L., Kirk, D., Sellen, A., and Izadi, S. Affor-
dances for manipulation of physical versus digital me-
dia on interactive surfaces. In Proc. CHI ’07, 2007,
pp. 1157–1166.
34. Ullmer, B., and Ishii, H. The metadesk: models and
prototypes for tangible user interfaces. In Proc. UIST
’97, 1997, pp. 223–232.
35. Vogel, D., Cudmore, M., Casiez, G., Balakrishnan, R.,
and Keliher, L. Hand occlusion with tablet-sized direct
pen input. In Proc. CHI ’09, 2009, pp. 557–566.
36. Watanabe, N., Washida, M., and Igarashi, T. Bubble
clusters: an interface for manipulating spatial aggrega-
tion of graphical objects. In Proc. UIST ’07, 2007,
pp. 173–182.
37. Weiss, M., Wagner, J., Jansen, Y., Jennings, R., Kho-
shabeh, R., Hollan, J. D., and Borchers, J. Slap widgets:
bridging the gap between virtual and physical controls
on tabletops. In Proc. CHI ’09, 2009, pp. 481–490.
38. Wellner, P. Interacting with paper on the digitaldesk.
Communications of the ACM 36 (7), 1993, 87–96.
39. Whittaker, S., and Hirschberg, J. The character, value,
and management of personal paper archives. ACM
Trans. Comput.-Hum. Interact. 8 (2), 2001, 150–170.
40. Wobbrock, J. O., Morris, M. R., and Wilson, A. D. Us-
er-defined gestures for surface computing. In Proc. CHI
’09, 2009, pp. 1083–1092.
41. Wu, M., and Balakrishnan, R. Multi-finger and whole
hand gestural interaction techniques for multi-user tab-
letop displays. In Proc. UIST ’03, 2003, pp. 193–202.