Conference PaperPDF Available

PaperFold: Evaluating Shape Changes for Viewport Transformations in Foldable Thin-Film Display Devices


Abstract and Figures

In this paper, we investigate the use of shape changes in a multi-segmented mobile device for triggering viewport transformations in its graphical interface. We study PaperFold, a foldable device with reconfigurable thin-film electrophoretic display tiles. PaperFold enables users to attach, reorient and fold displays in a mobile form factor that is thin and lightweight even when fully collapsed. We discuss how our design was informed by a participatory study that resulted in 14 preferred shape changes. In a subsequent study, we asked users to rank the utility of shape changes for triggering common view operations in map and text editing applications. Results suggest participants were able to attribute specific view operations as automated responses to folding, attaching, reorienting or detaching displays. Collated or full screen views were preferred when users collocated two displays. When adding a third display, alternative views such as toolbars or a list of apps were suggested. Showing 3D views was strongly associated with folding PaperFold segments into a three dimensional structure.
Content may be subject to copyright.
PaperFold: Evaluating Shape Changes for Viewport
Transformations in Foldable Thin-Film Display Devices
Antonio Gomes
Human Media Lab
Queen’s University
Kingston, ON, Canada, K7L3N6
Roel Vertegaal
Human Media Lab
Queen’s University
Kingston, ON, Canada, K7L3N6
In this paper, we investigate the use of shape changes in a
multi-segmented mobile device for triggering viewport
transformations in its graphical interface. We study
PaperFold, a foldable device with reconfigurable thin-film
electrophoretic display tiles. PaperFold enables users to
attach, reorient and fold displays in a mobile form factor that
is thin and lightweight even when fully collapsed. We
discuss how our design was informed by a participatory
study that resulted in 14 preferred shape changes. In a
subsequent study, we asked users to rank the utility of shape
changes for triggering common view operations in map and
text editing applications. Results suggest participants were
able to attribute specific view operations as automated
responses to folding, attaching, reorienting or detaching
displays. Collated or full screen views were preferred when
users collocated two displays. When adding a third display,
alternative views such as toolbars or a list of apps were
suggested. Showing 3D views was strongly associated with
folding PaperFold segments into a three dimensional
ACM Classification Keywords
H.5.2. [Information interfaces and Presentation]: User
Interfaces - Interaction Styles, Evaluation/Methodology.
Author Keywords
Human Factors; Organic User Interfaces; Flexible Displays;
Foldable User Interfaces.
For reasons of portability, mobile devices have displays that
are limited in size and constrained by weight. Despite
increases in resolution, such size restrictions have resulted in
sequential interaction paradigms for mobile platforms, in
which apps are opened and closed one by one. By contrast,
the display size of paper documents can be modulated easily
through folding, tearing or combining multiple page
elements. Such properties allow paper documents to be
navigated and organized more efficiently, allowing
concurrent access to multiple documents [19]. Paper is also
very thin, durable and lightweight, perceived advantages in
the mobile design space. As such, the development of
electronic paper computers that adopt certain qualities or
metaphors of interacting with paper documents has been an
enduring research goal [9,19]. While initial research was
aimed at mimicking form factors or interaction techniques of
paper documents using rigid display devices [1,8], in recent
years, progress has been made towards developing
computers made of flexible E-Ink displays as thin and
lightweight as paper [4,22].
In this paper, we investigate how thin-film, paper-like
electrophoretic mobile devices [4,5,12,22] might adopt
dynamic modulation of screen real estate through folding and
tearing techniques in a prototype device called PaperFold1
(Figure 1) [5]. A number of paper metaphors served as
inspiration to the design of our prototype, which, like paper,
allows for multiple display segments to be folded into
smaller form factors. Books use folding as both a
navigational and space saving technique. Paper maps use
display size modulation via folding techniques of increasing
shape resolution [17]. PaperFold combines the benefits of
context-aware multi-display devices with paper-like thinness
and familiar interaction metaphors.
Why Thinness and Light Weight are Relevant
We believe thin, flexible displays are essential when trying
to mimic the kind of tactile-kinesthetic navigation,
Figure 1. PaperFold prototype folded into a 3D Hull.
Permission to make digital or hard copies of all or part of this work fo
ersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights fo
components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
ost on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from
TEI '15, January 16 - 19 2015, Stanford, CA, USA
Copyright 2015 ACM 978-1-4503-3305-4/15/01…$15.00
expandable screen real estate and lightweight multi-display
experiences that paper offers to users. The kind of folding
seen in today’s mobile devices [3,20] or previous
explorations [1,2,7,8] cannot be regarded as similar because
the thickness of individual display segments impacts the total
thickness of a folded device, while the weight of each display
segment impacts overall portability. Although none of the
known flexible display technologies are capable of
employing creases like paper, tessellation of flexible displays
allows for size modulations that approximate those of
foldable paper maps. While we aim to produce more
complex folds, we limited ourselves to what can be achieved
with current flexible technology.
In this paper, we evaluate the effectiveness of shape changes
as viewport transformations. We present PaperFold, a
prototype device consisting of 3 flexible display segments
that can be combined in arbitrary configurations. We discuss
interaction techniques that utilize shape changes to perform
view operations, e.g. adding a display segment extends a
document’s view; folding segments such that they can be
viewed by others produces shared views. We report on a
participatory design study that resulted in 14 shapes that can
be produced with PaperFold, and a subsequent study in
which participants ranked view operations to changes
between shapes. Results suggest participants were able to
attribute specific view operations to specific shape changes.
We conclude with design recommendations for matching
shape changes to view operations in reconfigurable mobile
While there has been research into folding displays into
various form factors, most of the display technologies in
prior studies were either simulations with paper mockups
[14], projection [9,10,11], or thick and rigid display devices
that are not particularly suitable for folding [1,2,7,8,15,18].
Other explorations focused on interactions with single
displays that were stacked or placed in close proximity
[4,22]. PaperFold bridges the tradeoff between portability
and increased screen real estate by introducing a
reconfigurable mobile display device with thin-film screens
that stay within acceptable boundaries of weight and
Slate Display Form Factors
There is a large body of work on book form factors involving
rigid dual-screen form factors that can orient around one
axis. Pocket Edge [3] combined a tablet computer and an
eBook reader in a dual-display interactive device. Sony
Tablet P [20] featured interoperating hinged displays. Chen
et al. [1] discussed an e-book reader featuring two displays
mounted on two separate slates that can be used in side-by-
side or detached configurations. Their design supported
embodied navigation techniques like folding, flipping, and
fanning. Their findings suggested that having additional
displays offers a better support for lightweight navigation.
Multi-display Form Factors
Chen et al. [2] evaluated a multi-slate reading system, aimed
at understanding how increased screen real estate across
multiple devices can enhance navigation and interaction
techniques in reading activities. Their results indicated that
multi-slate reading systems have the potential to recapture
many of the affordances associated with physical documents,
while providing functionality that surpasses that of paper.
Hinckley et al. [7] showed the use of tablet PCs for spanning
contiguous images across multiple display surfaces. The
system used simultaneous deceleration as a means for
detecting connections, and was capable of performing simple
view operations such as collating two displays, or spanning
an image across two displays upon connection. Hinckley et
al.’s [8] Codex, a dual-screen tablet computer, was an
exception to the above work in that its displays are
detachable and reconfigurable into various form factors.
Codex can be oriented in a variety of postures to support
individual work, ambient display, or collaboration with
another user. The authors demonstrate interaction techniques
supporting division of labor for tasks across displays. Their
findings identify several benefits for detachable displays
systems (i.e. Division of Tasks between Screens;
Collaboration between devices; Sidebars). Siftables [15]
demonstrated a series of networked compact devices with
embedded displays that could be physically manipulated as
a group to interact with digital information. Although it was
able to span visuals across multiple displays, Siftables lacked
connective hinges and featured thick screen objects. The
rigidity and thick form factor of the displays employed in the
above explorations posed real drawbacks in terms of
portability and physical effort.
Thin-film Display Form Factors
DisplayStacks [4] presented a system for physically
interacting with digital information via stacks of thin-film
displays. It introduced tools for navigating digital documents
using piles of physical display windows. DisplayStacks
combined affordances of paper documents with electronic
content. PaperTab [22] later featured an environment in
which multiple thin-film displays work together through
proximity sensing to mimic a physical paper desktop
Foldable Thin-film Designs
Lee et al. [13] explored image projection on foldable
materials to simulate flexible displays with variable form
factors. Their findings suggested that devices that can
incorporate large surfaces in a small collapsible volume offer
advantages for mobile contexts. Xpaaand [10] featured a
handheld device encapsulating a rollable display. The
authors explored dynamic resizing as a means of interacting
with digital contents. Their results indicated that having the
ability to physically manipulate screen real state may
effectively improve interaction with handheld devices.
FoldMe [11] simulated thin-film displays on the front and
back sides of a mockup device foldable along predefined
hinges. Their results suggested folding navigation techniques
could improve manipulation of information for portable
devices. Folds in this system were limited to a pre-defined
axis. Paddle [16] used projection to represent highly
deformable mobile devices that can be transformed into
various special-purpose controls. Their results suggested
benefits over touch interaction techniques commonly used
on mobile devices. We borrow from the aforementioned
explorations, in designing a multi-segmented reconfigurable
mobile device that can be attached and detached into
arbitrary configurations along multiple axis, noting that none
of the above systems used real flexible displays.
We believe that building a prototype using real flexible
displays is valuable due to their unique properties in terms of
thinness, lightweight and malleability. Current displays are
too thick and heavy to be able to achieve the kind of shape
transformations seen in our study without significant
physical effort by the user. Additionally, to our knowledge
there has not been a systematic exploration of three-way
folding in multi-display systems. To inform the design of a
multi-segmented reconfigurable mobile device, we used the
following design criteria:
Use of Folding to Modulate Screen Size
Our first goal was to allow users to modulate the size of
screen real estate of the device between folded conditions
and expanded shapes. While there have been examples of
rollable devices [10], we consider folding to be a more
natural metaphor as it allows the overall form factor to be
similar to that of current smartphones. We believed
unfolding would typically be used to expand the size of an
existing view.
Reconfigurability and Tile Geometry
Hinged displays can only be folded along one dimension. In
order to increase the number of form factors, we explored
attachment/detachment on multiple sides of the displays.
This allows users to go from, e.g. tablet form factors to paper
notebook while maintaining a simple folding interface.
Unlike [1,2,8], our work explores view-transformations
beyond 2 segments while preserving portability.
Detachability also allows users to increase or reduce the
number of displays dynamically, and implement concurrent
usage scenarios with multiple displays. In particular, our
design aimed to increase the subset of shapes demonstrated
in previous explorations.
The ability to alter the orientation has been used in
smartphones to switch between landscape and portrait
viewports. In PaperFold, orientation takes on a different role
as re-orienting the device may dramatically alter the
affordances of a shape, as well as visibility of information on
the displays. We hypothesized that orientation of the device
would therefore affect the function of the device.
Number and Angle of Tiles
The number of display segments creates a tradeoff between
shape resolution and portability of the device. More display
segments result in complex cable management, since
currently available flexible displays are not wireless. As we
will discuss in our participatory study, folding a four segment
device into a one segment device alters the shape geometry
of the device along two concurrent dimensions, making it
more difficult for users to (un)fold the device. Shapes created
by 3 segments are primarily determined by their geometry
and angle of displays towards one another.
3D Shapes
Using a minimum of three displays allows users to simulate
certain known display shapes through simple morphological
mimicry. A flat configuration resembles a book or map form
factor. Bending the displays inwards simulates a concave
display that might mimic gaming or virtual reality scenarios.
Bending the displays outwards, users can simulate a segment
of a spherical display. A 60-degree angle between 3 displays
creates a closed 3D hull. A 90-degree arrangement resembles
a portion of a cube [21]. One of the questions we were
interested in investigating was whether users would associate
such shapes with 3D objects.
Different orientations of display segments may cause them
to be visible to the user, other users, or hidden altogether.
According to Hinckley [8], configuration of tiles could be a
design parameter for determining whether a shape indicated
shared or private interactions.
Matching Shape Changes to Viewport Transformations
Segments of PaperFold form building blocks that can be
“shaped” into an application. We investigated the relative
transformation between shape states as an interaction
technique. If we generalize the above discussion, we note
that shape transitions in PaperFold typically appear to affect
its viewport. To further investigate this, we conducted a
participatory study.
Our methodologies were inspired by Hinckley et al., who
evaluated the affordances of pen-operated dual-display
devices [8], Chen et al. [1] who compared a dual-display
reader with single-display devices in typical reading tasks,
and Khalilbeigi et al. [11], who studied functions that can be
assigned to mockup double-sided displays with pre-defined
hinges. To investigate what would be the most functional
shapes of a foldable device, we conducted a participatory
design session in which subjects were presented with 4
10x15x0.15cm 3D printed tiles without displays. Each tile
contained magnets that allowed participants to interconnect
as well as hold shape at right angles.
Task & Participants
To determine what configurations users would find
appropriate for particular tasks, they were asked to imagine
a device in which the tiles featured displays showing apps
similar to those on their mobile phones. Users were told that
apps could span multiple displays. They were then asked to
create what they thought of as the most useful
transformations of shape for a 2 display hinged/detachable
device. We then repeated the exercise imagining 3 displays
and 4 displays. For each shape, we asked participants to
propose an associated functionality. 15 participants
volunteered for this study (6 females, 9 males; mean 23
years). Subjects received $10 for 1 hour of participation.
Results suggest that participants had difficulty relating to the
4 tile configurations. All participants preferred using a
maximum of 3 tiles. Most participants carried multiple
devices, such as smartphones, tablets and laptops on a
routine basis (Mean: 4.07, s.d.: 1.16). However, they
preferred to use a single device with different form factors,
if this was possible (Mean: 4.73, s.d.: 0.46). Figure 2 shows
the most common shape changes produced by participants
for 2 and 3 tiles configurations: Holding the display in
portrait (A,E,F,K,L), landscape (B,C,H,J) and 3D (M,N)
orientations; Attaching displays horizontally (A,I) and
vertically (B,F,J); Detaching displays (D); Bending Displays
Inwards (L) and Outwards (K); Folding the displays into
triangular (E,N), perpendicular (G,N) and cube-like
structures (M). When asked what functionality these shapes
or shape changes might have in a device, participants
comments almost exclusively referred to view operations
that are common in GUIs. We synthesized these comments
into a list of potential view operations, shown in Figure 3.
PaperFold is a foldable thin-film device with 3 detachable
flexible display tiles. The design of our prototype was
informed by the results of our participatory design study.
Each tile consists of a flexible thin-film E Ink display, and a
flexible 3D printed substrate with embedded sensors that
allows the system to determine the orientation and
connections of individual tiles (Figure 4).
Flexible Displays
Each PaperFold segment features a flexible 6.5” E-Ink
display. Magnetic hinges allow a continuous data connection
between displays. Each display is coupled to a driver board
connected to a computer controlling the logic and interface
3D Printed Tiles
PaperFold’s flexible 3D printed segments measure 15x10 cm
with a thickness of 1.25mm. Magnetic hinges allow
PaperFold tiles to be attached in a variety of configurations
(Figure 4). Note that the increased thickness at the
extremities of the tile is only a function of the required size
of the embedded magnets. Hall Effect sensors in the tiles
monitor the magnetic field in their vicinity, allowing our
system to detect distinct connections among various tiles. A
9DOF IMU calculates the absolute orientation of each tile.
Using this information, we are able to determine what view
operations should occur upon distinct shape configurations.
Sensing Touch
E-Ink displays pose challenges to sensing touch, as their
functioning requires pixels to be applied with positive or
negative charges to produce the desired graphics. Readily
available touch solutions have shown to interfere with the
screen refresh mechanism of e-ink displays. We designed
flexible printed circuit layers to sense touch input by
measuring changes in capacitance between direct touch and
the electrode pads.
In a subsequent experiment, we evaluated the use of our
PaperFold prototype for effecting view operations via shape
changes. We asked participants to rate the match between
each shape change and the list of view operations suggested
in our participatory study. Due to the nature of the
participatory design session, we used a different set of
participants to avoid bias towards choosing view operations
our first subset of users had suggested.
Shape Changes
Figure 2 shows the 14 shape transformations used in our
second study. Note that some are applied to different
orientations but topologically identical. Another defining
factor was the orientation of the hinge location: vertical or
horizontal. To illustrate, a vertically hinged portrait display
looks like a notebook. A horizontally hinged portrait display
looks like a book.
View Operations
In our study, we asked participants to match the list of 12
view operations in Figure 3 to the most frequently performed
shape changes derived from our participatory design session.
For each shape change, we asked participants to rank each of
the 12 view operations for appropriateness using
questionnaire items with a 5-point Likert scale (Strongly
Figure 2. The 14 participant defined shape configurations used in matching shape changes to view operations.
Agree-Strongly Disagree) that answered the following
question: “When I change the shape of this device in this
way, it [View Operation].”
Experiment Design and Participants
All participants experienced all shape changes in the same
order. To avoid bias, the list of view operations on the
questionnaire was presented in random order. 12 participants
took part in the experiment (6 Female; 6 Male; Mean Age:
21.3). All participants received $10 for 1 hour of
During training, each of the 12 view operations were
demonstrated in a GUI desktop application to avoid biasing
participants with PaperFold examples. Increasing the size of
the window and revealing more of an image demonstrated
Collate/Extend View. Show Full Screen was demonstrated by
showing a full screen window containing an entire photo.
Show Thumbnails was demonstrated by showing thumbnails
of the photo library; Zoom In/Out was demonstrated by
zooming in and out of a photo without changing the window
size. Duplicate View was demonstrated by showing two
windows with the same image. Show 3D View was
demonstrated by showing a 3D model of the object on the
photo. Other view operations were demonstrated by
showing: a keyboard; a toolbar; a phone keypad; a photo in
one window and a notepad in another; and an empty window.
Participants were then introduced to the prototype, instructed
on the task and given the questionnaire. A display showed a
list of image transformations from which each view
operation could be constructed on PaperFold. This list was
similar to the images shown in Figure 3. The experimenter
then loaded the first image onto the first display, after which
participants were asked to make the first shape change in
Figure 2: Attaching a second display horizontally.
Participants were asked to go through the questionnaire and
pick a favorite operation to perform upon this shape change,
if there was one. They were then asked to select the images
on the display that would best show this operation on the
prototype. They then rated the view operation on the
questionnaire, and were asked whether there were any
operations in the list that they thought would match this
shape change. If so, they were given the option to show these
on the prototype and rate them.
All shapes were repeated for each of two applications: 1) A
text editor showing pages of a scientific paper 2) A Google
map application showing a map of London. We based our
choice on applications described in previous work: [1]
focused on reading activities on dual-display devices; [8]
presented a pen-operated note taking application; and [11]
demonstrated interaction techniques for browsing maps
using mockup foldable displays. Additionally, the results of
our participatory design study showed a strong user
preference for these application scenarios.
Figure 5 shows the means and standard deviations for the
scores matching view operations with shape changes, for
portrait, landscape and 3D conditions in both application
scenarios. A non-parametric analysis of variance (Friedman
Test evaluated at α<.05) showed significant differences
between view operation scores for each shape change in the
text application (Friedman’s 2(11)>21.11, p<.05), with the
exception of Fold 3D Triangular Hull (Friedmans
2(11)=18.436, p=0.072). In the map application, results also
showed significant differences between view operation
scores for each shape change (Friedman’s 2(11)>23.9,
p<.05). Due to the exhaustive nature of possible
comparisons, and because we intended our results as design
heuristics, we did not perform further post-hoc tests. Instead,
we assigned the highest ranked viewport for any given shape
change as a best match, but only if it had a positive rating,
i.e., higher than 3.0. Note that by picking the top match, we
interpreted our data at the ordinal level of measurement.
Approximately half our top matches in the Text Editing
Scenario had a score above or equal to 4.0, suggesting strong
preferences for these particular view operations. Optional
alternative view operations, with scores between 3.0 and 4.0,
are also displayed. Finally, note that in the below section,
whenever we make a statement such as: “users preferred”,
we do not suggest such statement scientifically accurate, but
rather as a trend that informs the design process.
Figure 3. A list of view operations in GUIs suggested by
participatory design session.
Figure 4. Exploded view of a PaperFold segment.
Figure 5. Mean ratings and (standard deviations) for matching view operations to presented shape changes for both
User Agreement
Since we expected multiple view operations to match any
one particular shape, we decided against calculating a
measure of agreement. Instead, because the Likert scale
deployed does approximate an interval level of
measurement, we examine the standard deviations as
providing some indication of user agreement. In the text
application, standard deviations, as shown in Figure 5, were
low in most cases, averaging 1.33. In the map application,
standard deviations averaged 1.3. This suggests users
generally agreed well on rankings for both applications.
Matching View Operations to Shape Changes
Text Application
Extending the display, in both orientations, was strongly
associated with the Collate operation. In portrait conditions,
participants preferred Show Thumbnails as an alternative. In
landscape conditions, Full Screen or Show Keyboard were
suggested. Neither bending 3 displays inwards or outwards
had a strong user association in the case of the text
application. Bending inwards was associated with Collate,
while bending outwards was associated with Zoom In. Users
had a weak association with Show 3D View in the landscape
condition. In portrait mode, there was an equally weak
association with folding the display into a perpendicular
shape as pertaining to Collate. However, in the landscape
condition, there was a strong preference for Show Keyboard.
In portrait conditions, Folding Displays into a triangular
shape was associated with Duplicate Screen. Finally, for
Detach Displays, Duplicate Screen was preferred.
Map Application
Extending the displays in the portrait condition was highly
ranked for the Collate operation, in both horizontal and
vertical conditions. For the third display, Show Thumbnails,
Show Keyboard, Show Different App and Duplicate Screen
were also ranked positively. In the landscape condition,
second display, participants split their top ratings between
Collate an d Show Keyboard. For the landscape, third display,
there was a clearer preference for Collate, with Show
Thumbnails, Show Keyboard and Show Different App ranked
positively. Bending 3 Displays Inwards, Outwards and Fold
3D Hull (Triangle) were all associated with Show 3D View.
In the landscape condition, there was a strong preference of
Show 3D View for Fold 3D Hull (Cube), with Duplicate
Screen considered an alternative. In both c, there was a clear
association between folding the display into a perpendicular
shape and Show Keyboard. In both portrait and landscape
conditions, Folding Displays into a shape was associated
with Duplicate Screen. Finally, for Detach Displays Show
Different App had the strongest rating, with Duplicate Screen
and Show Phone considered weaker alternatives.
User Comments and Observations
Throughout the experiment, participants were encouraged to
verbalize their thought processes, which were noted by the
experimenters. Overall, participants liked having the ability
to arrange PaperFold in various configurations, thus being
able to dynamically alter viewports. “I would like to be able
to customize the functions of different shapes.” [P2]. There
were benefits identified for using PaperFold as a
multitasking system. “I enjoyed having the ability to
distribute different applications across displays and access
them concurrently.” [P5, P8]. While browsing through maps,
users were particularly impressed with the 3D View features
of our system. “3D shapes with multiple displays lead
themselves to viewing 3D models.” [P11]. Additional
benefits identified include adding peripherals [P4,P10];
thumbnails/toolbars [P3,P11]; or using PaperFold as means
for distributing information, by physically detaching and
sharing panels [P12].
For both application scenarios, results suggest that users
associate shape changes of PaperFold’s screens with specific
view operations. Some shape changes are, however, more
ambiguous than others. For example, extending PaperFold
wi th a 3rd horizontal display rates above neutral on 4 different
view operations. But even in those cases, generally only one
or two view operations rank highly for each shape change.
We will now discuss differences in trends between the two
application scenarios, trying to generalize our results into
design recommendations for viewport behaviors in folding
multi-display user interfaces.
Horizontal Extensions
When extending the displays horizontally, the most highly
ranked view operation was consistently Collate. In the case
of the text editor, where a contextual overview of data objects
is applicable, participants ranked Show Thumbnails higher
than a Full Screen operation. In the map browser, which has
no clear delineation of specific data objects, participants
were more inclined to try and fit the data on two screens.
Observations of image selections during the experiment
strongly suggests that participants preferred to extend a
thumbnail display with a second display showing a full
screen view of the first thumbnail. Since participants chose
to extend a full screen display with a collated view they
overwhelmingly chose Collated View as the highest rated
answer for both cases. Only when a third display was
attached, did participants widen their options, with scores for
collated views ranking lower and thumbnails higher.
Vertical Extensions
For vertical portrait extensions, Collate generally appears the
preferred option. Collate is also the highest ranking option
for vertical landscape extensions, where keyboard is also a
strong option.
3D Shapes
Bending three displays inwards or outwards was much more
highly ranked for Show 3D View operations in the map
application, where data elements were not inherently limited
to 2D. Observations confirmed that in the text editor
scenarios, participants had difficulty relating any 3D
operations to the inherent 2D nature of the data element. In
general, folding the displays into a 3D hull was the shape
change most strongly associated with Show 3D View. This
suggests that users associate the shape of the device with
known objects of similar form factor.
Detaching a screen was less easily generalizable: In the map
application detaching the displays was most strongly
associated with Show Different App. In the text editor
application Duplicate Screen was pr efe rred. V iew ope rat ions
that were not ranked positively for any shape change in either
application scenario included: Zooming; Show Toolbar; and
Show Phone.
We derived the following recommendations for the design of
segmented, foldable, multi-display devices:
1. Automated view operations appear to work well as
responses to folding, attaching or detaching displays.
2. Consider a Collate operation when users extend a first
display in Full Screen View with a second display.
3. Consider alternative views such as keyboards/toolbars or
different apps when extending a 2nd display with a 3rd
4. Provide mechanisms for users to choose view operation
functions, or allow them to set preferences.
5. Consider showing a keyboard on the lowest display when
the device is folded into a perpendicular shape.
6. Consider mirroring the displays when the device is folded
into a triangular shape.
7. Consider showing a 3D view of the data when users bend
displays inwards or outwards, or into a 3D hull shape.
We note that the above recommendations are based on
heuristics and can be considered valid only within the
limitations of the presented device and user study. We do
believe that results are sufficiently clear to generalize to
other application scenarios and device morphologies. We
recognize that our PaperFold prototype has shortcomings
that did not allow us to evaluate use by multiple users outside
a laboratory environment. In particular, an important future
direction is to remove the dependency on cables and
extraneous equipment for what is supposed to be a mobile
We evaluated PaperFold, a mobile device with flexible
display segments that can be folded into arbitrary
configurations. Our design was informed by a participatory
study that resulted in 14 preferred shape transformations. A
subsequent study investigated the effectiveness of shape
changes for viewport transformations. We evaluated how
participants ranked the utility of shape changes for triggering
common view operations in map and text editing
applications. Results indicate that participants were
generally able to select specific view operations as
automated responses to folding, attaching or detaching
displays. User feedback from our study indicates benefits
from having multiple detachable displays. Advantages
include better support for performing tasks that traditionally
require multiple devices, as well as physical manipulation
and sharing of views.
1. Chen, N., et al. Navigation Techniques for Dual-Display
E-Book Readers. Proc. CHI ’08, 1779-1788.
2. Chen, N., Guimbretière, F., Sellen, A. Graduate student
use of a multi-slate reading system. Proc. CHI’13, 1799-
3. Entourage Edge (2013) -
4. Girouard, A., Tarun, A., and Vertegaal, R.
DisplayStacks: Interaction Techniques for Stacks of
Flexible Thin-Film Displays. Proc CHI ’12, 2431-2440.
5. Gomes, A., Nesbitt, A., Vertegaal, R. MorePhone. Proc.
CHI ’13, 583-592.
6. Gomes, A., Vertegaal, R. PaperFold: A Shape Changing
Mobile Device with Multiple Reconfigurable
Electrophoretic Magnetic Display Tiles. Proc. CHI
EA’14, 535-538.
7. Hinckley, K. Synchronous gestures for multiple persons
and computers. Proc. UIST ‘03, 149-158.
8. Hinckley, K., Dixon, M., Sarin, R., Guimbretiere, F.,
and Balakrishnan, R. Codex. Proc. CHI ’99, 1933–1942.
9. Holman, D., et al. PaperWindows: interaction
techniques for digital paper. Proc. CHI '05, 591-599.
10. Khalilbeigi, M., Lissermann, R., Mühlhäuser, M., and
Steimle, J. Xpaaand. Proc. CHI ’11, 2729-2732.
11. Khalilbeigi, M., Lisserman, R., Kleine, W. FoldMe:
interacting with double-sided foldable displays. Proc.
TEI ’12, 33-40.
12. Lahey, B., et al. PaperPhone: Understanding the Use of
Bend Gestures in Mobile Devices with Flexible
Electronic Paper Displays. Proc. CHI ’11, 1303-1312.
13. Lee, J.C. Hudson, S.E., and Tse, E. Foldable interactive
displays. Proc. UIST ’08 (2008), 287-290.
14. Lee, S. et al. How users manipulate deformable displays
as input devices. Proc. CHI ’10, 1647-1656.
15. Merrill, D., Kalanithi, J., Maes, P. Siftables: towards
sensor network user interfaces. Proc. TEI ’07, 75-78.
16. Ramakers, R., Johannes, S. Luyten, K. Paddle. Proc.
CHI’14. 2569-2578.
17. Roudaut, A., et al. Morphees: toward high “shape
resolution” in self-actuated flexible mobile devices.
Proc. CHI ’13, 593-602.
18. Schwesig, C., Poupyrev, I., and Mori, E. Gummi: a
bendable computer. Proc. CHI '04, 263-270.
19. Sellen, A., Harper, R. The Myth of the Paperless Office,
MIT Press, Cambridge, MA, 2003.
20. Sony Corp -
21. Stavness, I., Lam, B., Fels, S. pCubee. Proc. CHI ’10,
22. Tarun, A., et al. PaperTab: an electronic paper computer
with multiple large flexible electrophoretic displays. EA
CHI ’13, 3131-3134.
... Deformable interfaces allow users to create shapes that complement the context of their use for both fully flexible devices, and those with rigid displays connected by hinges [7,8,22,26,57]. Most deformable prototypes use a flat, plane-like form factor [22,34,57,68]. The use of physical metaphors for input, like bend gestures, also offers potential benefits [28,31,34,42,57]. ...
... Deformable interfaces allow users to create shapes that complement the context of their use for both fully flexible devices, and those with rigid displays connected by hinges [7,8,22,26,57]. Most deformable prototypes use a flat, plane-like form factor [22,34,57,68]. The use of physical metaphors for input, like bend gestures, also offers potential benefits [28,31,34,42,57]. ...
... We used double-hinged connections allowing 360°rotation of each segment, which allows each segment to fold perfectly flat on top of its neighbour, offering more complex shape options than other hinged devices [22,46]. Figure 1 illustrates various shapes possible with Adaptic, such as flat objects or simple prisms. ...
Conference Paper
We present Adaptic, a novel "hybrid" active/passive haptic device that can change shape to act as a proxy for a range of virtual objects in VR.We use Adaptic with haptic retargeting to redirect the user’s hand to provide haptic feedback for several virtual objects in arm’s reach using only a single prop. To evaluate the effectiveness of Adaptic with haptic retargeting, we conducted a within-subjects experiment employing a docking task to compare Adaptic to nonmatching proxy objects (i.e., Styrofoam balls) and matching shape props. In our study, Adaptic sat on a desk in front of the user and changed shapes between grasps, to provide matching tactile feedback for various virtual objects placed in different virtual locations. Results indicate that the illusion was convincing: users felt they were manipulating several virtual objects in different virtual locations with a single Adaptic device. Docking performance (completion time and accuracy) with Adaptic was comparable to props without haptic retargeting.
... PaperFold [9,10] is a multi-segmented mobile device for triggering viewport transformations on the graphical interface. It enables users to attach, reorient and fold its touch displays.It defines folds based on attached displays and senses the movements on the hinges.WristOrigami [42] extends the interaction surface of a smartwatch by connecting its sides to foldable touch displays. ...
... high Flexkit [12] Hardware bendin click, touch desk medium med. PaperFold [9,10] Hardware fold, attach, reorient click, touch desk small to med. high ...
Full-text available
Whilst new patents and announcements advertise the technical availability of foldable displays, which are capable to be folded to some extent, there is still a lack of fundamental and applied understanding of how to model, to design, and to prototype graphical user interfaces for these devices before actually implementing them. Without waiting for their off-the-shelf availability and without being tied to any physical foldable mechanism, Flecto defines a model, an associated notation, and a supporting software for prototyping graphical user interfaces running on foldable displays, such as foldable smartphone or assemblies of foldable surfaces. For this purpose, we use an extended notation of the Yoshizawa-Randlett diagramming system, used to describe the folds of origami models, to characterize a foldable display and define possible interactive actions based on its folding operations. A guiding method for rapidly prototyping foldable user interfaces is devised and supported by Flecto, a design environment where foldable user interfaces are simulated in 3D environment instead of in physical reality. We report on a case study to demonstrate Flecto in action and we gather the feedback from users on Flecto, using Microsoft Product Reaction Cards.
... Recently, display technology has rapidly changed from conventional flat-type displays to flexible displays [1][2][3][4][5] such as bended, foldable [6][7][8][9][10][11][12][13], curved [14,15], and rollable [16][17][18][19][20] displays. These evolving display types require the appropriate display materials and process technologies. ...
Full-text available
Recent display technology has changed substantially from flat-type displays to bended displays. As a result, the lamination process for bonding the panel substrates and bended window glass has become difficult due to the changes in display shape, and the use of optically clear adhesive (OCA) makes it impossible to rework defective substrates due to residue problems. Therefore, it is necessary to research and develop a substrate-surface treatment that maintains the initial adhesion and is reusable via the complete removal of impurities during delamination in order to enable rework. In this paper, the possibility of maintaining adhesive force and reusing substrates was confirmed through the surface treatment of substrates and OCA using various materials. We found that a surface coating and a cooling treatment of additional substrates completely removed the impurities that remained on the substrates during reworking. These results could contribute to improving lamination-process technology and the productivity of the various forms of next-generation displays that are currently under development.
... As precursors, we can consider the 3 following works. PaperFold is a prototype device using detachable screens that cannot only be connected or broken apart to modify the screen footprint, but also dynamically folded along their bezels [5]. DisplayStacks are fully bendable screens whose screens can be stacked or slid over [4]. ...
Conference Paper
In this paper, we present BEXHI, a new mechanical structure for prototyping expandable and bendable handheld devices. Many research projects have pushed bendable surfaces from prototypes to commercially viable devices. In the meantime, expandable devices have become a topic of interest letting one foresee that such devices are on the horizon. With BEXHI, we provide a structure to explore the combined capabilities of these devices. The structure consists of multiple interweaved units allowing non-porous expandable surfaces to bend. Through an instanciation, we illustrate and discuss that the BEXHI structure allows for the exploration of the combination of both bend and expansion interaction spaces.
We explore the design of Hand Proximate User Interfaces (HPUIs) for head-mounted displays (HMDs) to facilitate near-body interactions with the display directly projected on, or around the user's hand. We focus on single-handed input, while taking into consideration the hand anatomy which distorts naturally when the user interacts with the display. Through two user studies, we explore the potential for discrete as well as continuous input. For discrete input, HPUIs favor targets that are directly on the fingers (as opposed to off-finger) as they offer tactile feedback. We demonstrate that continuous interaction is also possible, and is as effective on the fingers as in the off-finger space between the index finger and thumb. We also find that with continuous input, content is more easily controlled when the interaction occurs in the vertical or horizontal axes, and less with diagonal movements. We conclude with applications and recommendations for the design of future HPUIs.
Recent years have seen the development of display technology with the transformation from flat to flexible displays. This has led to changes in the typical shape of such displays, and curved displays with constant curvature have been developed recently. As the shape of the display changes, the sophistication of the lamination process for joining a flat and curved window cover glass with a panel substrate also increases, and continuous research and development is required. In this paper, we report the results of our study on the characteristics of resin spreading on flat and curved substrates using optical clear resin. We confirmed the consistency by comparing the results obtained from the theoretical analysis and experiments. These results are expected to be widely used in the lamination and manufacturing technologies for next-generation displays that will evolve in the future.
Large displays on smartphones accommodate tasks needing more screen real-estate, but at the expense of limiting the use of one-handed operations. In this paper we explore thumb reachability on an articulated dual-display smartphone. While mobile devices are featuring dual screens, no previous work has explored how each physical display’s positions impact one-handed reach and interaction. We explore 32 inter-display configurations resulting from the combination of orientations of the two displays along two axes. In a preliminary study we explore how users grasp such dual-display configurations. We then conduct a study to investigate the effects of the relative position of two displays on thumb reachability. Results provide a range of dual-display configurations that are ideal for one-handed use, enabling knowledge on how apps for such emerging devices can be optimized for thumb input.
Full-text available
We present the concept of highly deformable mobile devices that can be transformed into various special-purpose controls in order to bring physical controls to mobile devices. Physical controls have the advantage of exploiting people's innate abilities for manipulating physical objects in the real world. We designed and implemented a prototype, called Paddle, to demonstrate our concept. Additionally, we explore the interaction techniques enabled by this concept and conduct an in-depth study to evaluate our transformable physical controls. Our findings show that these physical controls provide several benefits over traditional touch interaction techniques commonly used on mobile devices.
Conference Paper
Full-text available
We present PaperFold, a novel shape changing mobile device with multiple reconfigurable touch sensitive thin-film electrophoretic magnetic display tiles. PaperFold explores the perceived benefits of having multiple computing devices combined into a single mobile device featuring multiple detachable displays. In PaperFold, each display tile can act independently or as part of a single system. Advantages include better support for performing tasks that traditionally require multiple devices, as well as physical manipulation and sharing of views. Touch and Inertial Measurement Unit (IMU) sensors embedded in each display tile allow users to dynamically manipulate content.
Conference Paper
Full-text available
We present Papertab, a paper computer with multiple 10.7" functional touch sensitive flexible electrophoretic displays. Papertab merges the benefits of working with electronic documents with the tangibility of paper documents. In Papertab, each document window is represented as a physical, functional, flexible e-paper screen called a displaywindow. Each displaywindow is an Android computer that can show documents at varying resolutions. The location of displaywindows is tracked on the desk using an electro-magnetic tracker. This allows for context-aware operations between displaywindows. Touch and bend sensors in each displaywindow allow users to navigate content.
Conference Paper
Full-text available
We present MorePhone, an actuated flexible smartphone with a thin-film E Ink display. MorePhone uses shape memory alloys to actuate the entire surface of the display as well as individual corners. We conducted a participatory study to determine how users associate urgency and notification type with full screen, 1 corner, 2 corner and 3 corner actuations of the smartphone. Results suggest that with the current prototype, actuated shape notifications are useful for visual feedback. Urgent notifications such as alarms and voice calls were best matched with actuation of the entire display surface, while less urgent notifications, such as software notifications were best matched to individual corner bends. While different corner actuations resulted in significantly different matches between notification types, medium urgency notification types were treated as similar, and best matched to a single corner bend. A follow-up study suggested that users prefer to dedicate each corner to a specific type of notification. Users would like to personalize the assignment of corners to notification type. Animation of shape actuation significantly increased the perceived urgency of any of the presented shapes.
Conference Paper
Full-text available
The Codex is a dual-screen tablet computer, about the size of a 4"x6" day planner, with a self-supporting binding and embedded sensors. The device can be oriented in a variety of postures to support different nuances of individual work, ambient display, or collaboration with another user. In the context of a pen-operated note taking application, we demonstrate interaction techniques that support a fluid division of labor for tasks and information across the two displays while minimizing disruption to the primary experience of authoring notes. Author Keywords dual screen, sensors, pen, tablet, mobility, collaboration
Conference Paper
Full-text available
Existing e-book readers do not do a good job supporting many reading tasks that people perform, as ethnographers report that when reading, people frequently read from multiple display surfaces. In this paper we present our design of a dual display e-book reader and explore how it can be used to interact with electronic documents. Our design supports embodied interactions like folding, flipping, and fanning for local/lightweight navigation. We also show how mechanisms like Space Filling Thumbnails can use the increased display space to aid global navigation. Lastly, the detachable faces in our design can facilitate inter-document operations and flexible layout of documents in the workspace. Semi-directed interviews with seven users found that dual-displays have the potential to improve the reading experience by supporting several local navigation tasks better than a single display device. Users also identified many reading tasks for which the device would be valuable. Users did not find the embodied interface particularly useful when reading in our controlled lab setting, however.
Conference Paper
In laboratory studies, multi-surface slate-based reading systems have shown great promise as platforms for active reading. However, the true utility of such a system can only be ascertained through the rigors of real world use. We conducted month-long deployments of a multi-slate reading system to support the active reading activities of graduate students in the humanities. During these deployments we documented how the added display area and increased micro-mobility of multiple devices enhanced navigation and reading comfort. We also noted the essential role of writing and annotation. Finally, we observed how electronic affordances like synchronization across devices helped provide functionality that would not have been possible with paper documents. This paper contributes new information about how electronic reading solutions fit into real world reading workflows.
Stacking physical documents is one of the main forms of spatio-temporal organization of information. We present DisplayStacks, a system that enables physical stacking of digital documents via piles of flexible E Ink displays. With a conductive dot pattern sensor attached to the flexible display, we dynamically track the position and orientation of these displays in relation to one another. We introduce mechanisms for interacting with these physical stacks for access and manipulation of information using asymmetric bi-manual interactions, such as providing contextual overviews. Initial user experiences indicate a preference for linear overlaps as a stacking configuration.
Conference Paper
We introduce the term shape resolution, which adds to the existing definitions of screen and touch resolution. We propose a framework, based on a geometric model (Non-Uniform Rational B-splines), which defines a metric for shape resolution in ten features. We illustrate it by comparing the current related work of shape changing devices. We then propose the concept of Morphees that are self-actuated flexible mobile devices adapting their shapes on their own to the context of use in order to offer better affordances. For instance, when a game is launched, the mobile device morphs into a console-like shape by curling two opposite edges to be better grasped with two hands. We then create preliminary prototypes of Morphees in order to explore six different building strategies using advanced shape changing materials (dielectric electro active polymers and shape memory alloys). By comparing the shape resolution of our prototypes, we generate insights to help designers toward creating high shape resolution Morphees.