Stroke-input methods for immersive styling environments
ABSTRACT This paper introduces an immersive styling environment, which aims at closing the gap between 2D drawing and 3D modeling in the design process. The main goal of the styling system is to provide the user an easy to use interface hiding the inherent mathematic nature of CAD. Creating rough 3D sketches should literally be as intuitive as 2D sketching with pen and paper. To achieve this, the tools developed in our system benefit from the stylists' skills, acquired through training over time. This paper focuses on the stroke-input methods of our styling system. We present different techniques for creating and modifying 3D curves: stroke splitting, oversketching and taping. In addition we report on the viability of using input constraints in immersive environments to overcome inherent weaknesses.
- SourceAvailable from: Michele Fiorentino
Conference Proceeding: Spacedesign: a mixed reality workspace for aesthetic industrial design[show abstract] [hide abstract]
ABSTRACT: Spacedesign is an innovative mixed reality (MR) application addressed to aesthetic design of free form curves and surfaces. It is a unique and comprehensive approach which uses task-specific configurations to support the design workflow from concept to mock-up evaluation and review. The first-phase conceptual design benefits from a workbench-like 3-D display for free hand sketching, surfacing and engineering visualization. Semitransparent stereo glasses augment the pre-production physical prototype by additional shapes, textures and annotations. Both workspaces share a common interface and allow collaboration and cooperation between different experts, who can configure the system for the specific task. A faster design workflow and CAD data consistency can be thus naturally achieved. Tests and collaborations with designers, mainly from automotive industry, are providing systematic feedback for this ongoing research. As far as the authors are concerned, there is no known similar approach that integrates the creation and editing phase of 3D curves and surfaces in virtual and augmented reality (VR/AR). Herein we see the major contribution of our new application.Mixed and Augmented Reality, 2002. ISMAR 2002. Proceedings. International Symposium on; 02/2002
- [show abstract] [hide abstract]
ABSTRACT: CALI is a fast, simple and compact online recognizer that identifies Scribbles (multi-stroke geometric shapes) drawn with a stylus on a digitizing tablet. Our method is able to identify shapes of different sizes and rotated at arbitrary angles, drawn with dashed, continuous strokes or overlapping lines. We use temporal adjacency to allow users to input the most common shapes in drawing such as triangles, lines, rectangles, circles, diamonds and ellipses, using multiple strokes.07/2002;
- [show abstract] [hide abstract]
ABSTRACT: This paper reports last developments and improvements of VISM (Virtual Integrated Surface Modeler) (1) running on I.R.R. (2) (Immersive Reconfigurable Room), installed at V- Lab ( http://v-lab.ingfo.unibo.it ) - University of Bologna - Forlì. VISM was originally developed for VRDD (Virtual Reality Design Desk) to explore a completely new concept of free-form surface modelling based on 3D tracking device and stereographic vision. More in detail, the use of an intuitive and tracking based surface modeller in a Immersive Reconfigurable Room (similar to CAVE Automatic Virtual Environment) gives the designer a definitely major advantage compared to a workstation user: he doesn't need any chalk full-scale model. Normally a designer, wishing to create or modify a project, follows these steps: - several hand-made colour sketches; - chalk or clay models construction; - 3D CAD modelling. These 3 activities may be collapsed in a single generative session in Virtual Reality. Furthermore VISM now includes a simple module to import a NURBS surface or group of surfaces in order to provide the designer a complete tool to create and modify a 3D geometry. Many benefits have been detected using the modified version of VISM, not only in terms of intuitive and easy-to-use interface, but much more in terms of time saving and 3D shape feeling.01/2001;
SMI 2004 – Paper #86
Abstract — This paper introduces an immersive styling
environment, which aims at closing the gap between 2D drawing
and 3D modeling in the design process. The main goal of the
styling system is to provide the user an easy to use interface
hiding the inherent mathematic nature of CAD. Creating rough
3D sketches should literally be as intuitive as 2D sketching with
pen and paper. To achieve this, the tools developed in our system
benefit from the stylists’ skills, acquired through training over
time. This paper focuses on the stroke-input methods of our
styling system. We present different techniques for creating and
modifying 3D curves: stroke splitting, oversketching and taping.
In addition we report on the viability of using input constraints
in immersive environments to overcome inherent weaknesses.
Index Terms — computer aided styling (CAS), immersive
environments, stroke input, interaction with shapes.
OWADAYS designers most preferred tool still is pen
and paper. Though a lot of mainly young designers
already started using computers, the interaction paradigm has
not changed. Whether pen or tablet, or paper or paint
software is used, the first sketches in the automobile design
process are always perspective sketches on a 2D medium.
Only after pre-selecting first concept drawings, a 3D model by
using computer aided styling tools is created. Current CAS
systems do not support direct 3D input. To use 2D input and
output devices to create 3D models, highly trained engineers
with good mathematical foundations are needed. The 3D
model evolves in collaboration between engineers and
designers, which is an immensely time consuming task.
Our aim is to reduce the gap between perspective 2D
drawings and 3D modeling by developing a 3D styling tool,
which can be used by designers as intuitively as pen and
paper. In general to reach this goal, there are two possible
approaches. One is to create 3D curves and surfaces using 2D
interfaces. The other is to create a model directly in 3D space
using an immersive environment. In this work we follow the
latter approach and do not consider the first.
Regardless of which approach, one main drawback of CAD
is the inherent mathematic nature of its operations and
representations. Making the system accessible to designers
means, that this characteristic has to be hidden by interaction
tools as much as possible, without loosing control over the
results. Though this seems contradictory, we focus on
presenting a viable solution to this challenge in our work.
Unlike paper or canvas, the media air, as free space, does
not create any feedback while sketching, which makes control
over a sketch more difficult. Although employment of haptic
devices could help to reduce this weakness, we do not
consider such devices here. This is mainly due to the fact, that
wide area haptic devices are still not commercially available.
Our approach to overcome this limitation is to constrain the
user input on the software side.
The presented immersive modeler is developed in close co-
operation with automotive
functionality tends to be tailored according to their specific
needs, meeting the criteria of their domain. Nevertheless, most
methods are very generic and results should partly be valid for
other design processes as well.
2. State of the Art
In the past few years design and modeling in virtual
environments has been more and more a subject of research.
Starting with simple visualization systems and point-grab-and-
move interaction, more complex applications are now ported
to and developed for VR.
One of the first modeling applications using immersive VR
technology was 3DM by Butterworth et al . Stereoscopic
output was realized by a head-mounted display (HMD). The
modeling functionality was limited to the creation of
compound objects from standard primitives like spheres and
cylinders. Free-form modeling functionality was not
supported. In non-immersive applications, Steed and Slater 
have evaluated different metaphors for 3D interaction with a
desktop bat (a 5 DOF device). They focused on picking and
moving objects as well as on navigation through space; object
creation and modification were not addressed.
designers. Therefore its
Stroke-Input Methods for
Immersive Styling Environments
Timo Fleisch, Gino Brunetti, Pedro Santos, André Stork
Fraunhofer IGD, Darmstadt, Germany
SMI 2004 – Paper #86
Chu et al.  are working on a multi-modal VR-CAD
system. This application can be used at a Virtual Table (VT)
driven by data gloves. Forsberg et al. have extended their
SKETCH  system towards 3D and use two magnetic
trackers at a VT for object transformation with the non-
dominant hand and 3D sketching with the dominant hand.
Fröhlich  et al. at Stanford University has developed an
architectural application that has assembly and disassembly
features and snapping capability.
An approach to define and alter free-form surfaces within
an immersive virtual environment by hand gestures was
described by Usoh et al . The deformations follow a
physical model, which results in an elastic behavior of
surfaces. The drawback of this approach is the complexity of
the calculations, which cannot be performed in real-time. This
heavily reduces the interactivity of the system.
Hummels et al.  examine the working situations of car
designers and suggest a gesture-based virtual environment.
The so-called “fish-tank” serves as desktop on which the
content of a monitor is mirrored. Most of the gestures should
be two-handed. No implementation details and results are
provided. In the same application area, i.e. car design, Fontana
 et al. suggest a classification scheme for so-called “detail
features” for an aesthetic and/or functional characterization of
predefined free-form features.
Further research and implementation was done by Dani et
al. . Their Conceptual Virtual Design System (COVIRDS)
explores the multi-modal use of different input streams, like
speech-input, gesture-recognition and 3D I/O devices for
modeling 3D objects. The proposed techniques for free-form
surface design are presented more detailed in Dani et al. .
They describe a creation and modification technique where
the user working at a projection wall is faced with a planar
default face. Indirect 3D interaction with a ray allows him to
select a set of control points, which can be a rectangular
subset of all control points, a row or a column. By moving the
set of control points the user alters the shape of the surface.
Liverani, G. Piraccini  reported on their development
on VISM (Virtual Surface Modeler). Addressing designers
their system allows working on full-scale models using 3 rear-
projected translucent screens (2.5mx1.9m each) to modify and
manipulate surfaces. The user uses a glove-like input device
moving over the surface to deform and shape it.
Wesche in  has described an interesting system for
modeling surfaces in a virtual environment. Using 3D input
devices his system allows to create and modify curves,
surfaces are derived from. The most interesting aspects are the
“smoothener” and “sharpener” operations, that provide local
modification of curves and – to a certain extent – also
surfaces. In the latter case, real-time behavior can only be
ensured for small patches. There is no immediate visual
feedback provided by the system when sketching surfaces.
In  Fiorentino et al. presented Spacedesign, which is the
predecessor of our system presented here. Spacedesign used a
similar setup to the one described here and offered functions
to create curves and surfaces in an immersive environment.
Some of the listed references already describe functionality
to create and modify geometry in an immersive environment.
However all of them merely investigated the technical aspect
of immersive modeling. Our system considers the stylist to be
at the center of the development process. Based on user tests
with partner from the car styling industry we studied the
typical skills of designers and developed our interaction
techniques to benefit from these skills. Our main conclusion is
that stylists work strongly stroke orientated. Though one-
gesture free form surface input methods, like the coon patch
function described by Fiorentino, create a 3D surface in a very
fast and convenient way, designers complain of not having
enough control over the results.
In our system we focus on methods to create and manipulate
strokes, which afterwards can be used to create surfaces. How
surfaces are created, is not subject of this paper.
3. System Overview
The immersive styling system setup basically consists of
The prototype can either run in virtual reality or in
augmented reality. In virtual reality a virtual table or
projection wall for augmented reality video or optical see-
through glasses is used.
To track the user’s head and hand positions we use an
optical tracking system from ART, which provides high
accuracy and low latency. To overcome the disadvantage of
keeping the camera’s line of sight clear, a redundant amount
of cameras and markers is used. In addition optical tracking
allows for untethered objects, such as a wireless pen.
Various tracked objects are used as input devices. They are
shown in Figure 1.
Depending on the setup, shutter glasses, linear polarization
glasses or a head mounted display is used. The virtual images
are rendered according to the user’s head position / viewpoint.
A wireless pen with 3 buttons is used as main input device.
SMI 2004 – Paper #86
Tape Finger L-Axis Tangible Plane
Figure 1: Input Devices
PIP-Sheet (Personal Interaction Panel)
The PIP (Personal Interaction Panel) is a transparent
plexiglas panel on which the application menu is projected.
The menus on the panel are operated with the pen.
Navigator axis (L shape, cube shape)
The navigator axis is used to navigate the model in 3DOF.
The user can choose between an “L”-shaped device and a
Tangible planes (mirror/projection)
These tracked artifacts are used by the mirror and
projection plane function. Both functions operate on a virtual
plane, which is moved according to the artifact.
This device is only used for the virtual taping function,
which is controlled by two input devices, the taping finger
representing the tape-fixing finger (see below) and the pen
applying a tangent to the curve.
Virtual Reality Setup
For styling, the output device should provide high quality
rendering, and immersion. We decided to use a semi-
immersive virtual table with a diagonal of 1.7 meters, which
allows creating e.g. parts of a car body in scale by appropriate
hand gestures and arm movements.
In stereoscopic mode the user wears a pair of tracked
shutter glasses, and the scene is rendered according to the
user’s point of view. In this way the virtual objects appear
floating in space above the table. The setup is displayed in
Figure 2: Virtual Reality Setup
A second scenario uses an upright projection wall to display
the picture. In this case the tracked artifacts are placed on a
table and the user operates the pen in front of the wall. The
disadvantage of this setup is that the tracked artifacts cannot
be placed anywhere conveniently.
4. Stroke input Methods
4.1. Stroke input
Our approach to stroke input processing is very straight
forward. While moving the 3D pen device in space the tracker
continuously delivers position events. These positions are
used as sample points for the stroke. The shape of the created
NURBS curve depends on the distances between the sample
points. The result is therefore always a trade off between
complexity of the curve and the distances between sample
points. Well-suited 3D curve approximation methods
including algorithmic details have been described by L. Piegl
and W. Tiller in .
Figure 3: Automatic Stroke Splitting
SMI 2004 – Paper #86
4.2. Automatic Stroke Splitting
High quality curves should have as few control points as
possible. But one problem with curves built of few control
points is, that it is not possible for them to have sharp edges.
We therefore developed a function that analyses the drawn
curve according to its curvature. It detects points of high
curvature and cuts the curve at these points. The resulting
curve then consists of partial curves with few control points
and possible sharp edges at the junctions. In
Figure 3 two curves are shown. The upper curve is drawn
with the conventional freehand spline function. It can clearly
be seen that the curve’s tip is rounded. The lower curve is
automatically split using our method, therefore the tip remains
as sharp as the user drew it.
The splitting algorithm uses the stroke input sample points.
These sample points can be regarded as a polyline. For each
point of the line the angles to neighbouring segments are
calculated. If these angles are small then the curvature of the
curve at this point is high. Along the list of per-point angles
the maxima are searched. If an angle maxima lies above a
defined curvature limit, the point becomes a splitting point.
Afterwards for each segment defined by the splitting points a
NURBS curve is generated according to the sample points and
the distance between sample points as described in the
4.3. Stroke Oversketching
In the task analysis phase we studied how stylists do their
pen and paper drawings. A common technique to create a
curve is to repetitively draw over – this technique is called
oversketching. To mimic this curve modification method, we
implemented virtual oversketching. With this function a curve
can be changed by sketching a new curve over the existing
The main tasks of the oversketching algorithm are:
• Find the part of the original stroke that has been
• Substitute the oversketched part with the new stroke.
• Smoothening the transition
Reshaping the original stroke according to the oversketch
can either be done on the base of control points or sample
Operation on control points
In this method the oversketch stroke is first converted to a
NURBS curve. In the following just the control points of the
original and the oversketch curve are considered. For the
oversketch curve’s first and last control point we search the
corresponding control point of the original curve. Then we
replace the control points in this interval with the
oversketched curve’s control points.
The disadvantage of this method is that the resulting control
points get unequally distributed over the curve, especially if
the operation is performed multiple times. Although the curve
itself still looks nice, surfaces created with these curves are of
Operation on sample points
As opposed to the above approach, not the control points
but the sample points are considered. Therefore the original
curve has to be re-sampled first, because it is internally stored
as NURBS curve. Then the interval of the original curve to be
replaced has to be found. Therefore the closest point to the
oversketched curve’s first and last point is identified. After
replacing this interval, the resulting curve is converted to a
NURBS curve again.
Because of the recalculation of the curve’s parameters, the
original curve’s character is lost. On the other hand the
recalculation always creates a good quality NURBS curve that
can be used to create surfaces from. Another disadvantage of
this method is that the transition area at the replace interval
start and end points appears to be harsher.
Both methods can be controlled by the user with two
parameters. One parameter defines the amount of influence of
the oversketched curve in the replacement interval. If the
value is 1, the curve is just replaced like described before. If
the value is below 1, the oversketched curve sample points or
control points are calculated as to lie on a perpendicular line
to the original curve’s sample point. If the scale value is 0 the
curve remains unchanged.
The second parameter defines a transition interval for the
replacement interval end and start points. If the transition
interval is zero the resulting curve might have harsh changes
at the replacement interval borders. This applies even more to
the method operating on sample points. Within the user
defined interval the replacement therefore is done according to
the above described factor of influence of the oversketched
curve. Also the scaling is defined similarly. However over the
transition interval scaling is a function from 0 to 1 to fade in
and 1 to 0 to fade out. For better results the transition is not
linear but according to a sine function.
Figure 4: Stroke Oversketching
SMI 2004 – Paper #86
In Figure 4 two samples of oversketching are shown. In a),
the scaling parameter of the oversketched curve is 0.5.
Therefore the resulting curve is about between the original
and the oversketched. In b) the influence parameter is 1.0,
therefore the result follows the oversketched curve.
Like in most CAD applications we also provide the
possibility to edit a curve by directly dragging its control
points. User tests showed that designers prefer our
oversketching method at early stages, where changes to apply
to curves are still quite big. For posterior refinement designers
still prefer control point modification, because it gives more
accurate control over the result.
We also extended our approach to surface oversketching,
where the shape of a surface is modified according to a curve
sketched close to it. (see
Figure 5: Surface Oversketching
Another one-stroke curve modeling method that has been
adapted from a traditional styling method is 3D tape drawing.
Tape drawing is a wide-spread modeling technique in
industrial design that is used to define characteristic lines,
typically on 1:1 sized perspective 2-dimenisional drawings of
the product model. In tape drawing instead of using pencils, a
special adhesive tape is used, which the stylist unrolls with
one hand (typically the right hand), sliding over the tape with
the other hand applying gentle pressure to fix it on the
drawing. In this way smooth 2-dimensional curves are
designed, where the first hand defines the tangent line with
respect to the curve point currently fixed by the other hand as
illustrated in Figure 6.
Experienced stylists define such taped curves with a single
two-handed stroke, where the tape easily allows undoing or
correcting the curve by un-taping it back to any point of the
curve, as required.
A first approach to develop a computer aided 2D tape
drawing has been presented by T. Grossman et al. ,
Figure 6: Traditional 2D-tape drawing
allowing to define a curve by using a 2D device (mouse) and
defining two subsequently clicked points on the projection
plane as a pair being the first point, a point on the curve and
the second point defining a tangent line (the vector between
the two points). To define the final 2D curve, the curve points
are interpolated applying a Hermite interpolation using the
tangent (direction and value of the first derivative)
information as boundary conditions.
The approach presented in this paper differs from previous
approaches, and actually extends digital tape drawing in four
1. It maps the 2-dimensional tape drawing into a
corresponding 3D metaphor as has been introduced by
De Amicis et al. ;
2. It extends 2D tape drawing to a 3D curve modeling
method, enhancing the modeling power of this styling
3. It provides direct manipulative two-handed input with
3D devices, which naturally builds on existing well
developed skills of stylists, allowing to define 3D curves
in a single stroke as it is the case in the traditional 2D
4. Finally, the approach integrates tape drawing into
allowing stylists to immediately inspect and validate
resulting curves from any viewpoint, with any scaling
factor, and related to the 3-dimensional presentation of
the product model, instead of a perspective 2D drawing,
The 3D devices applied for 3D tape drawing are shown in
Figure 7 and Figure 1. Those devices are a “tape finger”
device (left hand), which is tracked for the position of the
hand and finger, respectively, defining the curve points, and a
pen device (right hand) corresponding to the roll of tape. The
curve modeling starts when pushing the pen button and stops
when releasing it. All optically tracked positions of these two
devices are used to define the final shape of the curve. In
addition the head position and view orientation of the user as
well as the “navigator axis” are tracked to define the current
view with respect to the model.