ArticlePDF Available

CavePainting: A fully immersive 3D artistic medium and interactive experience


Abstract and Figures

Cavepainting is an artistic medium that uses a 3d analog of 2d brush strokes to create 3d works of art in a fully immersive Cave environment. Physical props and gestures are used to provide an intuitive interface for artists who may not be familiar with virtual reality. The system is designed to take advantage of the 8 ft. x 8 ft. x 8 ft. space in which the artist works. Cavepainting enables the artist to create a new type of art and provides a novel approach to viewing this art after it has been created. In this paper, we describe Cave-Painting'S 3d brush strokes, color pickers, artwork viewing mode, and interface. We also present several works of art created using the system along with feedback from artists. Artists are excited about this form of art and the gestural, full-Body experience of creating it.
Content may be subject to copyright.
CavePainting: A Fully Immersive 3D Artistic Medium and Interactive Experience
Daniel F. Keefe Daniel Acevedo Feliz Tomer Moscovich
David H. Laidlaw Joseph J. LaViola Jr.
Department of Computer Science
Brown University Providence, RI 02912
CavePainting is an artistic medium that uses a 3D analog of 2D
brush strokes to create 3D works of art in a fully immersive Cave
environment. Physical props and gestures are used to provide an
intuitive interface for artists who may not be familiar with virtual
reality. The system is designed to take advantage of the 8 ft. x 8 ft. x
8 ft. space in which the artist works. CavePainting enables the artist
to create a new type of art and provides a novel approach to viewing
this art after it has been created. In this paper, we describe Cave-
Paintings 3D brush strokes, color pickers, artwork viewing mode,
and interface. We also present several works of art created using the
system along with feedback from artists. Artists are excited about
this form of art and the gestural, full-body experience of creating it.
CR Categories and SubjectDescriptors: I.3.6 [Computer Graph-
ics]: Methodology and Techniques - Interaction Techniques; I.3.7
[Computer Graphics]: Three-Dimensional Graphics and Realism -
Virtual Reality; J.5 [Arts and Humanities]: Fine Arts
Additional Key Words: 3D painting, 3D modeling, gestures, tan-
gible user interface, Cave
1 Introduction
Paintings in which individual brush strokes are clearly visible are
often described as “loosely painted. Impressionistic paintings, for
example, often fit this description [26]. When we examine the brush
strokes in these paintings closely, we find both subtle and strik-
ing variations in color, size, shape, and texture. If we move close
enough, our eyes can only see the individual brush strokes. At this
level, the amazing variation in the type of stroke used, even within
a single painting, is apparent. The layering of strokes on top of each
other is also apparent at this close level. As we move away from the
painting, we stop seeing individual strokes. Our mind is able to fuse
the strokes together and comprehend a complex scene.
Our goal in this project was to create a system which uses 3D brush
strokes that function in the same way as the 2D brush strokes de-
scribed above. That is, we wanted our 3D brush strokes tobe clearly
visible when viewed up close, but we wanted a viewer to be able to
step back from a Cave-painting and see a recognizable scene. Since
the artwork is created entirely from these basic stroke elements, the
Figure 1 In CavePainting, the artist arranges and layers 3D brush
strokes in space to create a scene. Wedding Day - Daniel Keefe.
artist must be provided with a great deal of control over the type
of strokes used, variation within a stroke type, and 3D placement
of the strokes. The variation and placement of the brush strokes is
what allows us to discern a meaningful scene from a collection of
CavePaintingprovides the artistwith a natural interface for creating
a virtual 3D scene. The system runs in a four wall immersive vir-
tual reality system called a Cave [9]. The Cave-painter’s interaction
with the system is very similar to that of a traditional painter work-
ing on a large canvas. The artist is free to create long expressive
brush strokes and then step back to observe the work from different
angles. Interaction with the computer is accomplished through the
use of simple gestures and props that are commonly used in paint-
ing and positioned on a table inside the Cave. Scenes are created by
layering and arranging virtual 3D brush strokes in space.
We have invited many artists to try this new way of working. Ap-
proximately forty art students along with several instructors from
the Rhode Island School of Design and Brown University have used
Permission to make digital or hard copies of part or all of this
work or personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or
commercial advantage and that copies bear this notice and the
full citation on the first page. To copy otherwise, to republish,
to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee.
I3D '2001, Research Triangle Park, NC USA
© ACM 2001 1-58113-292-1/01/01 ...$5.00
the system in its current state. In addition, feedback from many
artists and art students who used the system in earlier stages helped
to guide the development of this application and its interface. Re-
cently, we have begun working with several artists who are inter-
ested in using the system for serious artistic compositions. Artists
have had an overwhelmingly positive reaction to CavePainting as a
new artistic medium.
In the remainder of this paper, we rst discuss related work. Then,
we introduce the different types of paint strokes and describe user
interaction in the system. Next, we present artwork and feedback
from artists who have used the system and conclude with a discus-
sion of the system and future work.
2 Related Work
In Surface Drawing [23] [24], Schkolne presented a free-form
3D surface construction tool. Surface Drawing is run on a Re-
sponsive Workbench. 3D shape is created by moving a tracked
on the users hand through the air. Schkolne’s sys-
tem has only one type of stroke, although the CyberGlove
allows the user to control this stroke to obtain interesting variations
as it is created. The focus of Schkolne’s work is on fusing these
strokes together automatically to form smooth surfaces.
Brody and Hartman presented a sketch at SIGGRAPH 2000 [5] on
a system called Body Language User Interface (BLUI). This is an-
other free-form modeling system that runs on a workbench. Lines,
point clouds, and extruded surfaces may be created by moving a
tracked wand through space. A menu is used to select drawing op-
erations. The distinctive feature of this system is that the geometry
created during this process is saved, imported into and rendered in
Maya, and then printed as a large panorama.
Other related projects such as 3DM [6], HoloSketch [10], and 3-
Draw [21] have used a tracked wand to create free-form lines or
geometry. HoloSketch, for example, was able to create free-form
“toothpaste” geometry, wire-frame lines, and clouds of triangle par-
CavePainting differs from these virtual reality free-form modeling
applications in several key ways. The first can be attributed to the
fact that our system runs in a fully immersive Cave environment.
The Cave provides the artist with enough space to stand up and
walk around while working. This directly affects the type of work
that the artist creates, as well as the way in which the artist works.
Additionally, since the user wears shutter glasses in the Cave, he or
she is able to see both the real world and the virtual world at the
same time. Thus, we are able use physical props and gestures that
a painter uses every day to interact with the system, eliminating the
need for a menu of drawing operations.
Second, CavePainting provides the artist with fine control over
color and a large, varied set of brush strokes with which to work.
CavePainting does not attempt to be a modeling system in a tradi-
tional sense, where the user is often concerned with exact coordi-
nate representations for the size or shape of objects. Rather, Cave-
Painting aspires to be an extension of painting to three dimensions.
Just as an oil painter builds up a painting with layers of varied brush
strokes, the Cave-painter creates many different 3D strokes to con-
vey the impression of a 3D scene. Finally, CavePainting promotes
the idea that art created by this dynamic 3D tool is meant to be
viewed in an interactive 3D display environment, since a static 2D
print, no matter how large, cannot truly convey the 3D nature of
this type of work. CavePainting presents a viewing mode of its own
which takes this notion a step further by providing the observer with
additional insight into the artistic process that produced each work.
Several others have worked on volumetric modeling using a tracker
as input [13] [25]. CavePainting is fundamentally different from
these systems because it creates varied 3D strokes of paint rather
than modifying voxel data which represents a solid, such as wood
or marble.
Additionally, there are several systems which use 2D input to gen-
erate 3D forms [8] [16] [28]. Though these can be used to obtain 3D
artistic results, they are limited in their ability to create dramatically
different 3D strokes and lack the fully immersive experience that is
so critical to CavePainting. Maya [2] is a more advanced artistic
package where 2D input can be used to generate 3D form with a
painterly quality.
Adding pigment or texture maps to a 3D surface [4] [14] is a tech-
nique often referred to as 3D painting. CavePainting defines 3D
painting differently. In our case, 3D painting is a process that gen-
erates 3D form.
3 Painting with 3D Strokes
A CavePainting is composed of many 3D paint strokes. These in-
dividual strokes are layered and arranged in space to produce a
scene. The artist can choose between several stroke types. The cur-
rent stroke types in the system are line, ribbon, tube, bumpy tube,
trail of any type of geometry, Jackson
Pollock++, splat, extrusion,
and bucket. The artist picks a stroke type to indicate the general
characteristics of the stroke. This is analogous to choosing to apply
oil paint with a large at brush, a small round brush, a sponge, or
a palette knife, since the artist can obtain considerable variation in
a stroke, even after a stroke type has been chosen. The artist actu-
ally applies the virtual paint, by moving a tracked paint brush prop
around in the cave. The virtual strokes respond to fine variations in
the position and orientation of the paint brush prop. The immediacy
of the response of the virtual paint to the artist’s movements is very
important to the artists that use CavePainting. The direct control
over the 3D paint is what allows them to create expressive varia-
tions in strokes. We found that it was a mistake to try to program
too much expressiveness into a stroke type. Rather, artists seem to
produce the most expressive strokes when given a simple stroke
type that they are able to easily control and immediately see the
results of their movements.
The following sections explain the way one artist used CavePaint-
ing’s 3D paint strokes to create the complex 3D painting shown in
Figure 9, in the Results section. This Cave-painting was inspired by
an oil pastel drawing the artist did while in a vineyard in Florence.
The original drawing is shown in Figure 2a.
As seen in Figure 2b, the artist started by defining a ground plane
and a wall rising out of it. This was done in a loose, abstract style
with the Jackson
Pollock++ and splat strokes. These strokes are an-
imated as they leave the artist’s brush. They fall in the direction
that the brush points until they reach one of the walls of the Cave,
where they splatter in the virtual world, as if they had actually hit
the physical wall of the Cave. Both strokes provide an interesting
link between the physical space the user occupies and the virtual
world in which he nds himself immersed. We find that this is im-
portant in CavePainting, since the artist is essentially defining the
space around him or her. The Jackson
Pollock++ stroke drips a line
of virtual paint, reminiscent of the drip paintings done by the great
expressionist artist [22]. In virtual reality, we are free from some of
the limitations imposed on Jackson Pollock. For example, our paint
does not need to drip according to gravity. We take advantage of
this, and are able to drip on all six sides of the cube defined by the
Cave. The same is true of the splat stroke. This stroke drips small
droplets of paint that create a random splatter shape when they hit
Figure 2 The progression of a Cave-painting.
In Figure 2c, the artist has started to define some of the structure
from which the grape vines will hang. A tube stroke was used to
create the posts coming out of the wall. The cross section of the
tube stroke follows the orientation of the brush as it is created. By
turning the brush while painting, the artist can create a flattenedthin
stroke or a tube with a circular cross section. The tube stroke is an
example of a relatively simple geometry that changes dynamically
as the artist moves and twists the brush.
Another simple stroke which has this quality and is used quite often
in this painting is the ribbon stroke. It was used to create the leaves
on the grape vine shown in Figures 2c and 2d. The actual vines were
created using the bumpy tube stroke. This is a tube stroke with a
radius that varies automatically. The radius changes based on time,
so a long drawn out bump can be created by painting quickly, as was
done for the vines, or short round bumps can be created by moving
the brush slowly. This was used to create the grapes in this scene.
All of these strokes are simple geometric shapes, but the interaction
in the system gives the artist the power to vary their appearance as
they are applied. This makes them very powerful in terms of their
expressive quality to the artist.
The extrusion stroke was used twice in this painting. The petals of
the red flower in the flower box and the flower pot on the right side
of the image in Figure 2e were created using the extrusion stroke.
Creating an extrusion stroke is a three step process. First, a free
form line is drawn using the brush. This will be the cross section
for the extruded surface. Next, the user positions the brush at an
anchor point somewhere near the line and presses the button on the
brush. This attaches the line to the brush so that it moves along
with the tracked brush that the user holds in his or her hand. The
final step is to paint normally with the cross section line attached to
the brush. The result is a smooth surface swept out along the path
that the user follows with the brush.
The cross section line that is swept out to form an extruded sur-
face is represented as a list of 3D points. We create a mesh by con-
necting these points from frame to frame as the user moves them
around bypainting with the brush. A challenge inimplementing this
stroke was to avoid creating prohibitively large meshes that would
grind rendering to a halt. To preserve the features of the cross sec-
tion while reducing the number of samples, we compute the Haar
wavelet transform of the list of points. We remove small coefficients
that have little effect on the curve, and then invert the transform to
produce a new list of points. Since we reduce the number of sam-
ples, the 3D mesh created by connecting these points together has
fewer polygons.
The final stroke illustrated in this artwork is the geometry stroke.
This is used to orient and place down a trail of any predefined geo-
metric model. In this example, the artist has imported a model of a
leaf and has placed colored instances of this model in several loca-
tions to complete the scene. They can be seen on the ground and on
the bench in Figure 2f.
4 Interaction
The first goal for the interaction in this system was to make it very
natural for an artist to use. We noticed early on that painting in 3D
in an area large enough to stand up and walk around was very sim-
ilar to how a painter works on a large canvas. For example, when
working on a large canvas, painters often make long gestural paint
strokes. After a few strokes are placed on the canvas, the painter
steps back to see the effect the entire painting. As the work pro-
gresses, there is a continued motion of the painter between the can-
vas and some area an optimal viewing distance away from the work.
This is, in fact, how many artists, especially painters, work in our
In addition to making the interaction natural for an artist, a sec-
ondary goal was to make the system intuitive for novices and artists
of all ages. We wanted CavePainting to scale to different ages and
levels of artistic ability just as real painting does.
We also wanted to take advantage of the physical space provided
to the user when working in a Cave. When the user rst enters this
cube, he or she knows that there are projection screen walls approx-
imately 4 feet away on all sides. Rather than trying to completely
hide these walls, we tried to take advantage of the fact that the user
Figure 3 The painting table interface.
knows roughly where these walls are. We do this with several of
our paint strokes, which interact with the walls as if they were part
of the virtual world. We also do this by placing physical props on a
real painting table [12] [15][17] (see Figure 3) that is located inside
the Cave along one of the walls, and by placing virtual controls in
the world so that they always appear to be attached to the physical
walls of the Cave.
4.1 Painting
Figure 4 An artist at work.
Strokes are created in 3D space by moving a tracked paint brush
through the air, as seen in Figure 4. A 3D path with orientations
within the Cave is sampled from the tracker on the brush. For most
strokes, the tracker is sampled once per frame. When the single
button on the brush is depressed, a stroke is interactively drawn
through the sampled 3D points. As the tracked physical paint brush
moves around, its position and orientation are reflected on a 3D vir-
tual brush icon. The size of the stroke being created is represented
by the size of the brush icon and is controlled by rotating a knob on
the painting table or by using a tracked pinch glove [1] [19] worn
on the non-dominant hand.
4.2 Assigning a Stroke Type to the Brush
To select the type of stroke to paint, the artist simply dips the phys-
ical paint brush into a cup (located on the painting table shown in
Figure 3) that “contains” the desired stroke. This interface is simi-
lar in style to the tool tray Fitzmaurice used in his GraspDraw ap-
plication [11]. When the brush touches the cup, the stroke type is
changed. Thisis implemented by placing conductive cloth on the tip
of the brush and along the inside of the paint cups. Audio feedback
is given to indicate that the brush’s stroke type has changed. This
is an interface which even young artists have been able to readily
4.3 Picking Colors
Figure 5 A 3D color picker represents the Hue, Lightness, and Sat-
uration color space. A large sphere is moved around inside the color
space to select a color. Hue is selected by moving aroundthe circum-
ference of the color space representation. Saturation is proportional
to the distance from the vertical axis, and the vertical position of the
sphere within the space controls the lightness of the color.
To select a specificcolor, the artist uses a 3D color picker that repre-
sents the Hue, Lightness, and Saturation color space. (see Figure 5)
This is similar Deering’s 3D RGB cube color picker [10]. There are
two ways to activate the color picker. The first is to make a circu-
lar gesture with the brush pointed straight up. This makes the color
picker appear at the center of the circle with the brush location con-
trolling the color selection. As the user moves the brush through the
color space, a large sphere moves with the brush and changes color
based on the location within the space. To select the color, dismiss
the color picker, and continue painting, the user tilts the brush so
that it no longer points up. In practice, we find that advanced users
want to switch colors quite often. These users often choose to wear
a tracked pinch glove on the non-dominant hand that gives them
quick access to the color picker and other options. When the glove
is worn, the color picker is activated by pinching and holding down
the thumb and index finger. This brings up the color space at the
location of the hand when the pinch was made. The position of the
non-dominant hand now controls the color selection, and the color
is updated immediately on the brush icon. To select a color and
continue painting, the user releases the thumb and index finger.
In addition to selecting a specific color for the stroke, the user can
select a color mapping to apply to the space of the Cave. In this
mode, the color of the stroke is determined by the position of the
brush within the physical space of the Cave. The artist can pick be-
tween several mappings of this space. The first is a straight mapping
of the red, green, and blue components of the color to the x, y,z co-
ordinates of the brush in the Cave. This mapping is primarily used
for abstract paintings. A more useful mapping for the artist is our
warm to cool color mapping. In this mapping we selected a range
of warm and cool earth tones. We interpolate between these colors
to assign a warm color to the brush while it is in the front of the
Cave and a progressively cooler color as it moves to the back. We
can also assign a “hot spot” in the cave and interpolate between
warm and cool colors based on the distance from this spot. We have
a large sphere hanging from the ceiling in our cave as part of our
Polhemus tracking system. We often use this physical object as the
“hot spot”.
These automatic color modes are used mostly for quick gestural
paintings that tend to explore the concepts of depth and spatial rela-
tions in three-dimensions. The way that artists tend to work within
these color modes is to do 3 or 4 quick paintings in a row. Each of
these will generally fill the whole space of the cave. After each one,
they translate the virtual world aside to clear the space of the cave.
Essentially, they turn to a blank canvas.
4.4 Painting with the Bucket
Figure 6 Virtual paint can be dripped out of a real bucket that has
a tracker attached to it. This paint can be thrown or dripped onto the
walls and floor of the Cave.
In addition to using the brush to paint, users can pick up a real
bucket and drip or throw paint out of it and onto the walls and floor
of the cave, as shown in Figure 6. When paint is thrown out of
the bucket, it flies out in the direction that the bucket is moving and
falls with an acceleration equal to gravity. When the bucket is tilted,
a stream of paint flows from the lip of the bucket. The size of the
flow is calculated based on the angle at which the bucket is held.
The ow changes dynamically as the user moves the bucket and
adjusts its tilt angle. Thus, this type of painting is another example,
of a simple stroke that is very responsive to the artists movements.
When not in use, the bucket is kept on two metal hooks on the side
of the painting table. When the metal handle of the bucket rests on
these two hooks, it creates a circuit. We detect that the user has
picked up the bucket and begin tracking it to determine its position
and orientation by checking to see whether this circuit is broken.
4.5 Two Handed Interaction
Advanced users have the option of wearing a tracked pinch glove
on the non-dominant hand to gain quick access to several features.
As mentioned previously, pinching the thumb and the index finger
activates the color picker. To set the brush size, the artist pinches
and holds down the middle finger and the thumb. The size of the
brush is proportional to the distance of the non-dominant hand from
the body. This method of setting the brush size offers much finer
control than using the knob on the painting table. Both of these
tasks can be carried out using the non-dominant hand while the
dominant hand (holding the brush prop) is used to paint. Thus, the
artist is able to change the width of a stroke or its color as it is being
Deering used a mouse and a tracker to explore a similar two-hand
interaction technique [10]. Others, have also explored two-handed
interaction and noted several advantages over one-handed input
when the two-handed interface is designed properly [7] [18] [27].
The most compelling advantage to using two hands in this case
comes when the glove and the brush are used simultaneously to
paint one stroke that contains a smooth transition between multi-
ple colors. Typically, artists use this technique by starting a stroke
with the non-dominant hand held steady at a certain color or size
value, and then adjusting this value while painting with the domi-
nant hand. Users typically focus on the brush rather than the color
picker and develop a feel for how moving the non-dominant hand
up or down causes a change in thelightness of the color and moving
the hand in other directions causes changes in hue or saturation.
Pinching the ring nger and the thumb activates the translation
mode, described in the navigation section below. This can also be
used while painting with the dominant hand.
Pinching the pinky and the thumb activates apainting scaling mode,
described below.
To help the user remember the function of each pinch, we present
3D text next to the appropriate finger on a 3D icon of the hand,
whenever the user turns his or her hand so that the palm faces up.
4.6 Navigation
For small translations, a tracked pinch glove is used on the non-
dominant hand. The user grabs the world by pinching together the
thumb and the ring finger, and then drags the world around by mov-
ing the hand [1] [19]. This is often used for quickly positioning an
object in the correct area to continue painting it, although, it can
also be used, as mentioned above, to move the world while painting
to create very long strokes.
For large translations, the user taps a foot pedal located on the floor
of the Cave near the painting table. This places the system in trans-
lation mode and causes a wire-frame floor plane to be displayed
in the virtual world. The artist then drips a trail of paint out of the
brush and onto the floor plane. After painting out the trail to follow,
the user controls the translation of the Cave forward and backward
along the path painted in the virtual world using a lever attached to
the painting table.
It is possible to change both the width and color of the stroke at the
same time while painting, but it is difficult to control both parameters at the
same time, with the same hand.
We have found that all of our users who choose to wear the pinch
glove prefer to use it for both short and long translations. The way
that our most comfortable artists work is to almost continuously
hold the world with the non-dominant hand. They tend to apply a
few strokes, step back and take a look at them, then grab the world
and move it a few feet. Then they take another look at their work
and keep proceeding in this fashion. The ability to quickly translate
the world as they work is essential to these artists. Those that do not
wear the pinch glove generally work on less serious compositions
and tend to fill the cave with paint and then want to move to a new
area to start something new. Dripping a path of paint to translate
along is appropriate for these artists.
Figure 7 To rotate a painting, the user makes a circular gesture with
the brush that activates this rotation widget.
To rotate the world, a simple gesture is used. The user points the
tip of the brush down and makes a circular gesture, as if mixing a
witch’s brew in a big cauldron. After one full rotation around in a
circle, a rotation widget (shown in Figure 7) pops up in the center
of the circle. The widget is a compass. Moving the brush around
the circumference of the compass causes both the compass and the
entire world to rotate as if it were attached to the brush and rotating
around the center of the compass.
4.7 Scaling
The user can activate a scaling mode by pinching together the
thumb and pinky. When in scaling mode, the brush icon changes
shape to indicate that the brush now controls the scale. To scale
up, the user points the brush up and holds down the button on the
brush. To scale down, the user does the same while pointing the
brush down. The entire painting is scaled around the location of the
brush. To exit scaling mode and continue painting, the user pinches
the thumb and pinky again.
4.8 Interactive Viewing Mode
CavePainting records the state of the artwork as it progresses. Once
the work is finished, an interactive viewing mode provides the ob-
server with an interface for accessing this data. When it is launched,
the interactive viewer first recreates the painting by moving the vir-
tual brush icon according to the saved data produced by the artist.
This creates an exact reproduction of the artist’s brush work, and
the final result is the painting that the artist produced. After this, the
observer is in control of the display. A timeline widget (shown in
Figure 8) is displayed onthe floor of the Cave. When the user stands
on the timeline, the position on the line determines the state of the
painting to display. The left end of the timeline is the initial state,
Figure 8 By moving around on a timeline widget placed on the
floor of the Cave, an observer controls the state of the 3D painting
currently displayed to see the artistic process that created the paint-
which is a blank canvas. The right end of the timeline is the finished
state of the painting. When the user steps off of the timeline wid-
get, the painting is held at the current state, so the the user can walk
around it in the Cave and examine the details of the brushwork.
Figure 9 Florentine Vineyard - Daniel Keefe.
Figures 9 through 12 are images of 3D scenes created with our sys-
tem. Note that these are snapshots of actual 3D models which we
prefer to present in an immersive viewing system.
6 Discussion
6.1 User Feedback
We have hosted two art classes at the Cave. One was a Brown Uni-
versity class, and one was a Rhode Island School of Design (RISD)
class. In addition, we have worked with several painters and art stu-
dents of varied backgrounds.
Painters, who used the system in its final state, expressed their sur-
prise at how easy the system was to understand and use. Two of
these artists had minimal experience with computers and had no
trouble learning the prop and gesture based interface. The over-
whelming comments from the artists were about their excitement
regarding the new things they could do with this medium. All of
them asked if they could come back again after their initial expe-
riences because they wanted to return with sketches of ideas for
serious compositions.
Figure 10 Number 3 - Eileen Vote.
Figure 11 Sailing - Daniel Keefe.
While working in the Cave, users said that they liked the control
they felt when picking colors and the responsiveness of the paint-
ing process. They also said that movement is essential in this new
medium, and they liked that the painting was created by moving the
entire body. They enjoyed watching the recreation of their work in
the interactive viewing mode and then walking through their paint-
ing. Many of the artists commented that virtual sculpture may be
a better metaphor for the experience, although they noted that the
looseness of the line quality is like drawing or painting.
The artists did find it hard to be as accurate as they wanted when
lining up the brush to paint fine details or paint on top of another
stroke, although theability to scale a painting helps withthese tasks.
Another comment was that the strokes are fairly final once they are
placed. The user can backup and undo strokes in order, but unlike
oil painting, there is no way to move paint around or scrape it off
the canvas once it has been placed. One artist commented that she
could not reproduce the earth tones with which she usually painted,
since the projectors in the Cave could not faithfully reproduce all of
these colors.
Figure 12 Madman in a car with really bad exhaust - Leora Maltz.
6.2 Young Artists
A group of 21 ninth-grade students also came to the Cave to try
the program. None of them had ever been in virtual reality before.
The interesting observation about this experience was that all of
these novice users immediately understood the paint brush inter-
face. Since there is only one button on the brush, there was no con-
fusion about how to begin drawing a stroke. Dipping the brush into
paint cups was also immediately understood as the way to change
brush strokes. In fact, these young artists found the props so con-
vincing that several of them took extra care moving the brush back
and forth in the bottom of cup, as if to make sure the brush was
fully loaded with paint.
6.3 Table Interface
We adopted a table interface for our props in the Cave for several
reasons. The rst is that if we place the table in one corner of the
Cave,we can light itindividually andcast a minimal amount of light
on the projection surfaces. This is important when working with
props in a Cave because the room lights must generally be kept very
low so that there is adequate contrast for the projector screens. In
a previous version of our interface, we used a different paint brush
for each stroke. The user carried these brushes around as he or she
moved in the Caveby attaching them with velcro to an apron. When
we brought art students who were unfamiliar with virtual reality to
the Cave to try our system, they thought holding a brush to create
the strokes felt very natural, but their biggest complaint was that
they had a hard time picking up the correct brush from the apron
since it can be very difficult to see the props in the Cave when the
lights are turned down and the user is wearing shutter glasses.
Another advantage to using our table based prop interface is that
it scales much better than our previous interface to the different
ages and abilities of our users. In targeting a range of users, we had
hoped for an interface in which the most inexperienced user could
walk into the Cave, pick up a paintbrush, and start using the system
right away. Our original interface required putting on two pinch
gloves and an apron. It was intimidating and restricting, especially
to users who had never experienced virtual reality before, to have
to put on these devices, all of which had wires coming out of them,
before even stepping intothe Cave.In the current interface, a novice
user begins by picking up the paintbrush and starting to paint with
whatever stroke is selected. As users become comfortable, we in-
troduce the table props, and then the color picker and navigation
We chose to put brush color and size controls on the pinch glove
because we noticed that artists use these options frequently. They
tend to adjust the color and size of a stroke much more than they
pick a different type of stroke. It would probably be distracting to
have to walk to the table to perform a task that occurs as often as
picking a color, but even artists working on serious compositions
did not mind taking a step or two to get to the painting table to
switch brush type. Given this, we chose to use gestures and the
pinch glove to activate controls that are used very frequently, and
the table for less frequent operations. We think that requiring the
user to walk over to a physical table to perform these less frequent
operations actually offers several enhancements to a user’s Cave
6.4 Using the Space of the Cave
For an application where the painting results are much more dra-
matic when the artist uses the entire space of the Cave, it was im-
portant to make even novice users understand that they could walk
around the Cave, and to have them feel comfortable doing so. Our
interface was designed to do both of these things. Users were re-
quired to interact with props placed along the walls of the Cave and
virtual controls that were stationed on the floor surface. One advan-
tage to doing this is that the user is able to take advantage of spatial
memory to remember where all his or her tools are in the world.
Just as when we enter a room, we remember where the light switch
is, when the user enters the Cave, it is easy to remember which
wall to walk to for a particular control panel. Additionally, if that
control panel happens to be a physical table or prop attached to a
wall, interacting with this physical object creates a link between the
physical space of the Cave and the virtual world. Retaining some
link to the real world while in the Cave seemed to have a positive
effect on our users, especially those that were new to virtual reality.
We think a large part of this could be attributed to their increased
comfort level while working in the Cave. For an artist who is not
too familiar with computers in general, walking into a dark room,
called the Cave, that is filled with computers, wires, and technology
you have never seen before, can be a rather intimidating experience
by itself. Something as simple as reducing the fear of bumping into
a wall and falling through an expensive projection surface by pro-
viding some indication of where the walls of the Cave are can make
a big difference in making a user comfortable enough in the Cave
to really walk around and use the whole space of the Cave.
6.5 Viewing Mode
We wereinterested in a special viewing mode for two reasons. First,
because our computer-based medium has great potential for archiv-
ing the artists process. Second, our observations and discussions
with artists lead us to consider the process of creating a CavePaint-
ing as part of the finished artistic result. That is, a CavePainting is
an artistic happening. Viewing a CavePainting without some sense
of the dynamic movement of the brush and body that created it is
only looking at part of the art.
The art world is often extremely interested in the process used to
create a work. Unfortunately, this is usually very difficult to deter-
mine by looking at a finished work. For example, a skilled painter
or art historian can often tell the order in which portions of a paint-
ing were created, but it is impossible to completely peel back each
brush stroke of a masterpiece to see what lies beneath. There are a
few examples of successful artworks in which the artist gives us a
rare glimpse into the artistic process. For example, Pablo Picasso’s
series of lithographs of a bull [3]. In this series, Picasso made 18
prints of 11 different states of his lithograph stone. The first states
are detailed realistic representations of a bull. As we look at the
next states in order, we seethe image on Picasso’s lithography stone
progress from an intricate, realistic image of a bull to a abstract line
representation. Printmakers nd this work particularly interesting
because each of the 18 prints is a record of the state of Picasso’s
stone. Using this record, printmakers can gain insight into Picasso’s
process from a technical and intellectual standpoint. In CavePaint-
ing we try to capture the same type of record of the artistic process
and then present it in a way that anyone who goes to look at the art
can understand.
One of the great artistic questions that surrounds working in vir-
tual reality as a medium is how to display one’s artwork. Some
artists choose to display their art in VRML files on the web. Others
prefer to capture 3D scenes at different angles and arrange these
snapshots into a print that is more appropriate for display in a tra-
ditional museum or exhibition. While it is possible to do either of
these with CavePainting, we realized while working with it, that a
great deal of the art is lost when it is viewed in this manner. Walk-
ing through an artist’s work, and watching the dance-like process
that the artist used to create it, is really essential to getting a feel
for the depth and dynamic nature of the scene and the new way that
art is created around an artist with this way of working. The inter-
active viewing mode attempts to provide some of this sense to the
viewer. Currently, we succeed in capturing and presenting the dy-
namic movements of the brush that were used to create a painting.
These movements by themselves are very dance-like, and of course,
they hint at the actual pose of the artist while painting. In the future,
we would like to be able to capture much more information about
the pose and movement of the artists entire body. Perhaps, with a
system similar to [20].
7 Conclusion and Future Work
Painting in its traditional form almost always produces a 2D result.
Paint strokes can have some 3D qualities since they may be laid
down with thick blobs of paint and layered on top of each other,
but traditionally, they are arranged on a 2D surface. Despite this,
we chose painting as a model for our new way of creating art in
the Cave. The reason is that virtual reality has given us the power
to create 3D strokes that mirror the function of a brush stroke in a
2D painting. A single brush stroke is a tiny element in a completed
painting. Yet, when we look at the painting, these strokes are able
to work together to form a coherent scene. One of the ways that
the painter produces a meaningful scene from a collection of brush
strokes is by varying the size, shape, color, and texture of the stroke.
Strokes with different characteristics evoke different responses, and
the skilled artist uses this knowledge to create a work which clearly
conveys a message to the observer.
Our 3D brush strokes function in the same way as 2D brush strokes.
By providing the artist with a wide range of choices in the type of
brush stroke to apply as well as complete control over the colors
and size of the stroke, we present the artist with numerous ways
to vary brush strokes within a work. The resulting strokes may be
used together and layered in three dimensions in a way which is
analogous to a 2D painting.
We are interested in continuing to advance the look and behavior
of our paint strokes by adding textures to them and creating strokes
that can produce animated paintings. Another area of future work is
developing an immersiveviewing environment that provides at least
some coarse tracking, but is still feasible for setup in a museum or
an exhibition. Perhaps the most interesting area to explore in the
future is to embrace the idea of CavePainting as a performance art.
We have started to experiment with painting to music and bringing
dancers into the Cave.
Our observations have shown that CavePainting provides an intu-
itive interface for the skilled artist working to create a meaningful
piece of art as well as for the novice user or young artist. In ad-
dition, we have found CavePainting to be a very successful Cave
application in terms of enticing a novice user to move around freely
and explore an environment in the Cave. As such, it is an excellent
application for introducing people to the world of virtual reality.
The brush strokes provided by the system are sufcient for expe-
rienced artists to create complex and meaningful 3D works using
the system. CavePainting allows these artists to create a type of art
that was not possible before and present this art in a novel viewing
8 Acknowledgments
Many thanks to the artists that have worked with us: Eileen Vote,
Leora Maltz, and students and professors at the Rhode Island
School of Design and Brown University. This work originated as
a final project for an Interactive Computer Graphics class taught by
John Hughes. We would like to thank him for his support. Also,
thanks to Robert Zeleznik for helpful discussions and comments
on the paper, and Andrew Forsberg for his invaluable help work-
ing in the Cave. This work was partially supported by NSF (CCR-
0086065, CCR-9996209).
[1] SmartScene
is a product of Multigen, Inc. More informa-
tion on SmartScene
is available from MultiGen’s website at
[2] Maya. AliasWavefront.
[3] Picasso the Printmaker: Graphics from the Marina Picasso Collec-
tion. Dallas Museum of Art, 1983.
[4] Maneesh Agrawala, Andrew C. Beers, and Marc Levoy. 3d painting
on scanned surfaces. 1995 Symposium on Interactive 3D Graphics,
pages 145–150, April 1995. ISBN 0-89791-736-7.
[5] Bill Brody and Chris Hartman. Painting space with BLUI. Conference
Abstracts and Applications of SIGGRAPH00, page 242, August 2000.
[6] Jeff Butterworth, Andrew Davidson, Stephen Hench, and T. Marc
Olano. 3dm: A three dimensional modeler using a head-mounted dis-
play. 1992 Symposium on Interactive 3D Graphics, 25(2):135–138,
March 1992. ISBN 0-89791-467-8.
[7] W. Buxton and B. A. Myers. A study in two-handed input. In Pro-
ceedings of ACM CHI 86 Conference on Human Factors in Comput-
ing Systems, pages 321–326, New York, NY, USA, April 1986. ACM
[8] Jonathan M. Cohen, John F. Hughes, and Robert C. Zeleznik. Harold:
A world made of drawings. In Proceedings of the First International
Symposium on Non-Photorealistic Animation and Rendering, pages
83–90, New York, NY, USA, May 2000. ACM Press.
[9] Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti.
Surround-screen projection-based virtual reality: The design and im-
plementation of the cave. Proceedings of SIGGRAPH 93, pages 135–
142, August 1993. ISBN 0-201-58889-7. Held in Anaheim, Califor-
[10] M. Deering. Holosketch: A virtual reality sketching/animation tool.
ACM Transactions on Computer-Human Interaction, 2(3):220–238,
[11] George W. Fitzmaurice. Graspable user interfaces. Ph.D. Thesis, Uni-
versity of Toronto, 1996.
[12] George W. Fitzmaurice, Hiroshi Ishii, and William Buxton. Bricks:
Laying the foundations for graspable user interfaces. In Irvin R. Katz,
Robert Mack, Linn Marks, Mary Beth Rosson, and Jakob Nielsen,
editors, Proceedings of ACM CHI 95 Conference on Human Factors
in Computing Systems, pages 442–449, New York, NY, USA, May
1995. ACM Press.
[13] Tinsley A. Galyean and John F. Hughes. Sculpting: An interac-
tive volumetric modeling technique. Proceedings of SIGGRAPH 91,
pages 267–274, July 1991. ISBN 0-201-56291-X. Held in Las Vegas,
[14] Pat Hanrahan and Paul E. Haeberli. Direct wysiwyg painting and tex-
turing on 3d shapes. Proceedings of SIGGRAPH 90, pages 215–223,
August 1990. ISBN 0-201-50933-4. Held in Dallas, Texas.
[15] Ken Hinckley, Randy Pausch, John C. Goble, and Neal F. Kassell.
Passive real-world interface props for neurosurgical visualization. In
Proceedings of ACM CHI 94 Conference on Human Factors in Com-
puting Systems, volume 2 of PAPER ABSTRACTS: Interacting in 3-D,
page 232, 1994.
[16] Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. Teddy:
A sketching interface for 3d freeform design. Proceedings of SIG-
GRAPH99,pages 409–416, August 1999. ISBN 0-20148-560-5. Held
in Los Angeles, California.
[17] Hiroshi Ishii and Brygg Ullmer. Tangible bits: Towards seamless in-
terfaces between people, bits and atoms. In Proceedings of ACM CHI
97 Conference on Human Factors in Computing Systems, volume 1 of
PAPERS: Beyond the Desktop, pages 234–241, 1997.
[18] Paul Kabbash, William Buxton, and Abigail Sellen. Two-handed in-
put in a compound task. In Beth Adelson, Susan Dumais, and Judith
Olson, editors, Proceedings of ACM CHI 94 Conference on Human
Factors in Computing Systems, pages 417–423, New York, NY, USA,
April 1994. ACM Press.
[19] D. P. Mapes and J. M. Moshell. A two-handed interface for object
manipulation in virtual environments. Presence, 4(4):403–416, 1995.
[20] Simon Penny, Jeffrey Smith, and Andre Bernhardt. Traces: Wireless
full body tracking in the cave. In ICAT 99 Conference Proceedings,
[21] Emanuel Sachs, Andrew Roberts, and David Stoops. 3-draw: A tool
for designing 3D shapes. IEEE Computer Graphics and Applications,
11(6):18–26, November 1991.
[22] Irving Sandler. Abstract Expressionism: The Triumph of American
Painting. Palll Mall P., London, 1970.
[23] Stephen Schkolne. Surface drawing: The perceptual construc-
tion of aesthetic form. Master’s Thesis, Caltech, May 1999.
[24] Steven Schkolne and Peter Schroder. Surface drawing. Technical Re-
port CS-TR-99-03, CalTech Department of Computer Science, 1999.
[25] Sidney W. Wang and Arie E. Kaufman. Volume sculpting. 1995 Sym-
posium on Interactive 3DGraphics, pages 151–156, April 1995. ISBN
[26] Barbara Ehrlich White. Impressionists side by side: their relation-
ships, rivalries, and artistic exchanges. A.A. Knopf, New York, 1996.
[27] Robert C. Zeleznik, Andrew S. Forsberg, and Paul S. Strauss. Two
pointer input for 3d interaction. 1997 Symposium on Interactive 3D
Graphics, pages 115–120, April 1997. ISBN 0-89791-884-3.
[28] Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. Sketch:
An interface for sketching 3d scenes. Proceedings of SIGGRAPH 96,
pages 163–170, August 1996. ISBN 0-201-94800-1. Held in New
Orleans, Louisiana.
... 2). Ribbon brushes [GravitySketch 2019; Keefe et al. 2007Keefe et al. , 2001; Mozilla 2021; TiltBrush 2019] enable users to draw 3D shapes using ruled-surface ribbon strokes. They define the ribbon geometry by sweeping a fixed-length straight-line ruling along the spatial trajectory, or path, traced by the tip of the user's handheld six degrees of freedom (6DOF) controller (Fig. 3). ...
... Existing ribbon brushes leverage the orientation of a fixed controller axis to compute the direction of the ribbon rulings. One family of methods [GravitySketch 2019; Keefe et al. 2001] directly uses the orientation of a specific controller axis as the direction of the rulings (Figs. 1b,3de), while the other [Keefe et al. 2007; Mozilla 2021; TiltBrush 2019] uses a specific controller axis as approximate ribbon normal and defines the ruling direction as a cross-product of the direction of this fixed axis and the tangent of the controller trajectory, resulting in rulings which are strictly orthogonal to this controller axis (Figs. ...
... VR Ribbon Brushes. Many research and commercial VR drawing tools allow users to draw ribbon, or ruled-surface brush strokes by moving a handheld controller in 3D space [GravitySketch 2019; Keefe et al. 2007Keefe et al. , 2001Oculus 2019;TiltBrush 2019]. In all cases, the constructed ribbons form ruled surfaces whose path tracks the trajectory of the tip of the handheld controller in space (Fig. 3), and have rulings whose length is pre-defined by the user. ...
Full-text available
Virtual reality drawing applications let users draw 3D shapes using brushes that form ribbon shaped, or ruled-surface, strokes. Each ribbon is uniquely defined by its user-specified ruling length, path, and the ruling directions at each point along this path. Existing brushes use the trajectory of a handheld controller in 3D space as the ribbon path, and compute the ruling directions using a fixed mapping from a specific controller coordinate-frame axis. This fixed mapping forces users to rotate the controller and thus their wrists to change ribbon normal or ruling directions, and requires substantial physical effort to draw even medium complexity ribbons. Since human ability to rotate their wrists continuously is heavily restricted, the space of ribbon geometries users can comfortably draw using these brushes is limited. These brushes can be unpredictable, producing ribbons with unexpectedly varying width or flipped and wobbly normals in response to seemingly natural hand gestures. Our AdaptiBrush ribbon brush system dramatically extends the space of ribbon geometries users can comfortably draw while enabling them to accurately predict the ribbon shape that a given hand motion produces. We achieve this by introducing a novel adaptive ruling direction computation method, enabling users to easily change ribbon ruling and normal orientation using predominantly translational controller, and thus wrist, motion. We facilitate ease-of-use by computing predictable ruling directions that smoothly change in both world and controller coordinate systems, and facilitate ease-of-learning by prioritizing ruling directions which are well-aligned with one of the controller coordinate system axes. Our comparative user studies confirm that our more general and predictable ruling computation leads to significant improvements in brush usability and effectiveness compared to all prior brushes; in a head to head comparison users preferred AdaptiBrush over the next-best brush by a margin of 2 to 1.
... As noted above, immersive 3D sketching was pioneered by the HoloSketch system [58], which used a 6-DoF wand as the input device for creating polyline sketches, 3D tubes, and primitives. In a similar vein, various subsequent systems have explored the creation of freeform 3D curves and swept surfaces [71,87,131,204]. While directly turning 3D input to creative output is acceptable for ideation, the inherent imprecision of 3D sketching is quickly apparent when more structured creation is desired. ...
... Since freehand sketching provides an intuitive method of conceptualizing ideas, there has been considerable research [26,57,107,205,267,274] into using freehand sketches to create three-dimensional artifacts by lifting 2D sketches into the third dimension. The HCI community has complemented this research by producing interfaces that allow sketching directly in 3D [92,131,232]. Prior works [110,232] indicate that professional designers are excited by the possibility of using direct 3D input for sketching, but find it difficult, and get frustrated by the lack of control over their strokes (Figure 3.1). While there have been some quantitative studies exploring how 3D sketching capabilities improve with learning [257] and how force feedback affects precision [130], there has been limited work that evaluates and quantifies the factors which influence drawing in unconstrained, mid-air environments. ...
... Combining 2D and 3D Sketching Sketching has been a design and conceptualization aid for centuries. While traditional 2D sketching is ingrained into our consciousness, abilities, and training from childhood [269], the growth of consumer grade augmented and virtual reality (VR/AR) devices has enabled us to transcend the limits of the sketch canvas digitally-equipping artists with the unprecedented ability to sketch 3D curves directly in the air [58,131]. However, the previous chapter described challenges in control and precision, perception, and ergonomics faced by the users of 3D sketching systems. ...
Full-text available
Recent developments in immersive technologies of virtual and augmented realities (VR/AR) have opened up the immersive space for 3D creation. By removing the crutch of two-dimensional proxies for input and output, the immersive medium presents a significant advance in the design of intuitive interfaces for 3D creative expression. However, immersion also presents its own unique challenges in precision, control, ergonomics, and perception. This thesis advocates for a human-centred approach to the design of creative tools for immersive environments. Conducting interviews, observational studies, and formal quantitative experiments reveals insights into the needs and aspirations of future users of immersive tools. These user studies build fundamental knowledge about the limitations of human perception and our musculoskeletal system. These insights are then utilized to build concrete design guidelines for immersive creation tools. We present four immersive tools, targeted at sketching, modelling, and animation tasks in VR/AR. These tools harness 3D mid-air interactions to improve the user experience for existing design tasks, and to enable novel design aesthetics.
... Some projects have used the CAVE system for 3D sketching in ways that might integrate well with our purpose. For example, CavePainting from Daniel Keefe [38] and Sketching in Space from Habakuk Israel [39] have been exploring CAVEs implementation within early conceptual design. These proposals explore a paperless sketching tool analysing the impact of drawing inside the 3D environment with ad-hoc tools. ...
... They are viewer-centred perspective artefacts [40, p. 65], that is, the pictures on the walls of the CAVE change with the position of the viewer so as to maintain an immersive anamorphosis at all times with regards to the position of the observer. In this case, the workflow goes directly from the designer's thoughts to the 3D digital model, capturing early conceptual design ideas and sculpting their digital model on-the-fly [38,39,41]. These artefacts consider 3D space as a different and more direct medium for externalising thoughts, a medium in which modelling happens automatically and there is no need for projecting anything, either on paper or at all [39, p. 4]. ...
We discuss the design requirements of a software platform for constructing immersive environments through handmade spherical perspective drawings in a performative setting, with concurrent interactive live feed of the spherical drawing’s VR visualization. We investigate current best practices and available software in order to extract functionalities, requirements, improvements, possible integrations and future developments. We map the base requirements of the software from three sources: the state of the art of drawing techniques for spherical perspectives (equirectangular, azimuthal equidistant and cubical), the available software for their practice and the experimentation with novel hybrid artefacts. For the latter, we use a node-based program that allows us to prototype the workflow before entering a pure coding stage. The desired software platform should integrate well within digital art practices, stimulate and facilitate the practice of anamorphic handmade spherical drawings, and expand spherical perspectives’ applications through the emerging media of Hybrid Immersive Models (HIMs).
... Drawing 3D curves freehand This is still a challenging task, and the main difficulty seems to be the compromise between intuitiveness [50,43,27,26,47] and precision [4,1], thus quality, of the line. We note that spline representations of curves are generally preferred in 3D modeling systems, when a direct control over the shape (curvature and torsion) is required [50,17,24]. ...
... We note that spline representations of curves are generally preferred in 3D modeling systems, when a direct control over the shape (curvature and torsion) is required [50,17,24]. In contrast, polylines are rather used in stroke sketching systems [27,20], appreciated by artists for loose and rough freehand drawings. As the quality of the line is a major point in our context, we choose to define the TL using a spline curve. ...
Aesthetic surface mesh modification, guided by a deforming feature line, is still a challenging task. Whereas most existing surface deformation methods work well when the new surface is stretched or isometrically deformed, in case of areas subjected to compression, unwanted bulging may appear. This configuration typically appears when a designer slides a feature line along the input surface. In this paper, we present a new approach for feature line modification solving the surface bulging problem. We consider physically-based deformation methods applied to surface meshes and characterize configurations that produce compression loading from a mechanical standpoint. Then, the deformation area is decomposed into two sub-domains with a boundary modification of the one subjected to compression loading. Thus, during a subsequent deformation, this sub-domain no longer undergoes a compression loading but rather a traction loading, which ultimately prevents bulging. This transformation looks like a retraction of this surface area and a corresponding processing pipeline is set up. Our feature line modification method is intuitive to use and runs interactively, making it attractive to be used during design reviews in immersive environments. Industrial examples and comparisons with state-of-the-art physically-based deformation methods validate the proposed approach.
... Immersive 3D sketching was pioneered by the HoloSketch system [Deering 1995], which used a 6-DoF wand as the input device for creating polyline sketches, 3D tubes, and primitives. In a similar vein, various subsequent systems have explored the creation of freeform 3D curves and swept surfaces [Google 2020; Keefe et al. 2001;Schkolne et al. 2001]. While directly turning 3D input to creative output is acceptable for ideation, the inherent imprecision of 3D sketching is quickly apparent when more structured creation is desired. ...
Full-text available
Complex 3D curves can be created by directly drawing mid-air in immersive environments (Augmented and Virtual Realities). Drawing mid-air strokes precisely on the surface of a 3D virtual object, however, is difficult, necessitating a projection of the mid-air stroke onto the user “intended” surface curve. We present the first detailed investigation of the fundamental problem of 3D stroke projection in VR. An assessment of the design requirements of real-time drawing of curves on 3D objects in VR is followed by the definition and classification of multiple techniques for 3D stroke projection. We analyze the advantages and shortcomings of these approaches both theoretically and via practical pilot testing. We then formally evaluate the two most promising techniques spraycan and mimicry with 20 users in VR. The study shows a strong qualitative and quantitative user preference for our novel stroke mimicry projection algorithm. We further illustrate the effectiveness and utility of stroke mimicry to draw complex 3D curves on surfaces for various artistic and functional design applications.
Full-text available
Volumetric representation is widely used in digital sculpting and terrain generation due to its highly modifiable characteristic. However, storing the entire raw voxel map requires a lot of memory, which limits its application on general machines. In this work, we propose to use a novel data structure based on the concept of layered depth-normal images to replace the commonly used scalar field for storing the volumetric information. The proposed data structure can be quickly edited, and the isosurface can be extracted by using general methods such as Marching Cubes and Dual Contouring. The proposed sculpting system can edit models with multiple resolution levels, and the surface between models of different resolutions can be connected seamlessly. Our method can achieve similar results as the existing volumetric methods while significantly reduces the memory usage and computation time. The model created by our system can be used in common game engines to make highly interactive games. Furthermore, all of our processes are parallel-friendly. We implemented a virtual reality digital content creation tool based on the proposed method to demonstrate the effectiveness and feasibility of our method.
Full-text available
Presented herein is a simple numerical method for determining the optimal location of internal restraints, such as braces and internal supports in beams so that their elastic lateraltorsional buckling capacities are maximized. The method consists of a direct search optimization technique and an automated Rayleigh-Ritz method for the buckling analysis. Some examples of braced and internally supportedI-beams are considered and their optimal restraint positions determined. The method can be used to solve similar problems of restraint locations optimization for any structural member against bending, buckling or vibration.
Conference Paper
Full-text available
Four techniques for performing a compound drawing/color selection task were studied: a unimanual technique, a bimanual technique where different hands controlled independent subtasks, and two other bimanual techniques in which the action of the right hand depended on that of the left hand. We call this latter class of two-handed technique "asymmetric dependent," and predict that because tasks of this sort most closely conform to bimanual tasks in the everyday world, they would give rise to the best performance. Results showed that one of the asymmetric bimauual techniques, called the Toolglass technique, did indeed give rise to the best overall performance. Reasons for the superiority of this technique are discussed in terms of their implications for design. These are contrasted with other kinds of two-handed techniques, and it is shown how, if designed inappropriately, two hands can be worse than one.
Two experiments were run to investigate two-handed input. The experimental tasks were representative of those found in CAD and office information systems. Experiment one involved the performance of a compound selection/positioning task. The two sub-tasks were performed by different hands using separate transducers. Without prompting, novice subjects adopted strategies that involved performing the two sub-tasks simultaneously. We interpret this as a demonstration that, in the appropriate context, users are capable of simultaneously providing continuous data from two hands without significant overhead. The results also show that the speed of performing the task was strongly correlated to the degree of parallelism employed. Experiment two involved the performance of a compound navigation/selection task. It compared a one-handed versus two-handed method for finding and selecting words in a document. The two-handed method significantly outperformed the commonly used one-handed method by a number of measures. Unlike experiment one, only two subjects adopted strategies that used both hands simultaneously. The benefits of the two-handed technique, therefore, are interpreted as being due to efficiency of hand motion. However, the two subjects who did use parallel strategies had the two fastest times of all subjects.
This paper describes a 3D object-space paint program. This program allows the user to directly manipulate the parameters used to shade the surface of the 3D shape by applying pigment to its surface. The pigment has all the properties normally associated with material shading models. This includes, but is not limited to, the diffuse color, the specular color, and the surface roughness. The pigment also can have thickness, which is modeled by simultaneously creating a bump map attached to the shape. The output of the paint program is a 3D model with associated texture maps. This information can be used with any rendering program with texture mapping capabilities. Almost all traditional techniques of 2D computer image painting have analogues in 3D object painting, but there are also many new techniques unique to 3D. One example is the use of solid textures to pattern the surface.
This dissertation defines and explores Graspable User Interfaces, an evolution of the input mechanisms used in graphical user interfaces (GUIs). A Graspable UI design provides users concurrent access to multiple, specialized input devices which can serve as dedicated physical interface widgets, affording physical manipulation and spatial arrangements. Like conventional GUIs, physical devices function as “handles” or manual controllers for logical functions on widgets in the interface. However, the notion of the Graspable UI builds on current practice in a number of ways. With conventional GUIs, there is typically only one graphical input device, such as a mouse. Hence, the physical handle is necessarily “time-multiplexed,” being repeatedly attached and unattached to the various logical functions of the GUI. A significant aspect of the Graspable UI is that there can be more than one input device. Hence input control can then be “space-multiplexed.” That is, different devices can be attached to different functions, each independently (but possibly simultaneously) accessible. This, then affords the capability to take advantage of the
Conference Paper
We present a modeling technique based on the metaphor of interactively sculpting complex 3D objects from a solid material, such as a block of wood or marble. The D model is represented in a 3D raster of voxels where each voxel stores local material property information such as color and texture. Sculpting is done by moving 3D voxel-based tools within the model. The affected regions are indicated directly on the 2D projected image of the 3D model. By reducing the complex operations between the 3D tool volume and the D model down to primitive voxel-by-voxel operations, coupled with the utilization of a localized ray casting for image updating, our sculpting tool achieves real-time interaction. Furthermore, volume sampling techniques and volume manipulations are employed to ensure that the process of sculpting does not introduce aliasing into the models.
Conference Paper
dm is a three dimensional (3D) surface modeling program that draws techniques of model manipulation from both CAD and drawing programs and applies them to modeling in an intuitive way. 3dm uses a head-mounted display (HMD) to simplify the problem of D model manipulation and understanding. A HMD places the user in the modeling space, making three dimensional relationships more understandable. As a result, 3dm is easy to learn how to use and encourages experimentation with model shapes.
Conference Paper
We present an intuitive interface for painting on unparameterized three-dimensional polygon meshes using a 6D Polhemus space tracker as an input device. Given a physical object we first acquire its surface geometry using a Cyberware scanner. We then treat the sensor of the space tracker as a paintbrush. As we move the sensor over the surface of the physcial object we color the corresponding locations on the scanned mesh. The physical object provides a natural force-feedback guide for painting on the mesh, making it intuitive and easy to accurately place color on the mesh.