ArticlePDF Available

Abstract and Figures

The geometry of biomolecules dictates their function, but reasoning about that structure is difficult because of their 3D complexity and the range of scales involved. The wooden or plastic ball-and-stick models that are common in high-school chemistry labs help people rea- son about these issues when the molecules involved are small, but they are useless in the study of large biomolecules. Largely for this reason, 3D computer visualization tools have become essential in this field. However, these tools are limited by their interfaces. Tra- ditional graphics workstations project a D model onto 2D screen, and interaction with the D model is indi- rect, using 2D mouse or pointing device. Immersive vi- sualization is a potential solution to this: it allows a user to visualize a biomolecule in 3D and interact with it di- rectly in 3-space. This paper reports upon a pilot study about the effects of immersive visualization upon an ex- pert's reasoning about the qualitative structure of these molecules. We ported a standard visualization applica- tion (PyMOL) to a CAVE-like immersive virtual envi- ronment (IVE), then invited three separate biochemistry research groups—people who use PyMOL routinely on desktop computers—to examine their favorite molecule in the IVE. Within ninety minutes of immersive inves- tigation, each group reported a new discovery about the qualitative structure of that molecule. We believe that the immersive environment facilitated these discoveries by supporting and facilitating the natural spatial reason- ing abilities of its users. An immersive virtual environment is a combination of hard- ware and software that provides a psychophysical experi- ence of being surrounded by a computer-generated scene (see Figure 1). Immersive virtual environments provide users with an egocentric three-dimensional perspective: users are immersed in a virtual world, where they can ex- plore complex spatial systems by looking through them, walking around them, and viewing them from different per- spectives. Immersive environments may help people see and understand the structure of complex three-dimensional datasets; in contrast to more traditional graphics worksta- tions, these environments allow one to visualize data us- ing the well-practiced, non-concious analysis that automati- cally accompanies an embodied, egocentric visual perspec- tive. There are several studies that have investigated the added value of immersive environments (Pausch, Proffitt, & Williams 1997; Ruddle, Payne, & Jones 1999; Arns, Cruz- Neira, & Cook 1999; Swan et al. 2003; Gruchalla 2004; Schulze et al. 2005; Demiralp et al. 2006). However, the results of these studies are mixed and the issue is somewhat controversial. There are few studies that clearly demonstrate the effectiveness of immersive environments for real-world problems, and none that approach this issue from the stand- point of qualitative reasoning. Our study does so, and our results indicate that experts understand more about the ge- ometry of biomolecules if they use an immersive environ- ment than if they use the same visualization tools on a stan- dard desktop. Within ninety minutes of immersive inves- tigation, each of the three groups in our study reported a new discovery about the qualitative structure of an important biomolecule—molecules that these groups had been study- ing for years in with the same software visualization tool on desktop environments. Immersive visualization has long been proposed as a means to analyze the complex three-dimensional structure of biological molecules (Ihlenfeldt 1997), and it is used by numerous investigators in basic research and industrial set- tings. Qualitative spatial analysis of the structure of these molecules at a range of scales is essential, because their overall three-dimensional configuration dictates the atomic interactions that are the basis of their function. Understand- ing the geometry of the building blocks of a biomolecule, and their relationships, is key to many of the grand-challenge problems in biochemistry: the rational design of drugs that enhance or inhibit molecular activity, the understanding of how steps in embryonic development normally proceed or go wrong in the presence of genetic mutations of molecular structure, and so on. This paper documents a pilot study in which three sepa- rate groups of biochemists visualized and interacted with in- dividual biological molecules in a CAVE-like IVE. In each case, the immersive working session yielded new insights that the same biochemists had not previously achieved with their extensive use of the same visualization package on standard desktop computer displays. Large-scale spatial features, such as pockets and ridges, were readily identi-
Content may be subject to copyright.
Immersive Examination of the Qualitative Structure of Biomolecules
Kenny Gruchalla
gruchall@colorado.edu
Department of Computer Science
University of Colorado at Boulder
Mark Dubin
dubin@colorado.edu
Department of Molecular, Cellular,
and Developmental Biology
University of Colorado at Boulder
Jonathan Marbach
marbach@colorado.edu
Department of Computer Science
University of Colorado at Boulder
Elizabeth Bradley
lizb@colorado.edu
Department of Computer Science
University of Colorado at Boulder
Abstract
The geometry of biomolecules dictates their function,
but reasoning about that structure is difficult because of
their 3D complexity and the range of scales involved.
The wooden or plastic ball-and-stick models that are
common in high-school chemistry labs help people rea-
son about these issues when the molecules involved
are small, but they are useless in the study of large
biomolecules. Largely for this reason, 3D computer
visualization tools have become essential in this field.
However, these tools are limited by their interfaces. Tra-
ditional graphics workstations project a 3D model onto
2D screen, and interaction with the 3D model is indi-
rect, using 2D mouse or pointing device. Immersive vi-
sualization is a potential solution to this: it allows a user
to visualize a biomolecule in 3D and interact with it di-
rectly in 3-space. This paper reports upon a pilot study
about the effects of immersive visualization upon an ex-
pert’s reasoning about the qualitative structure of these
molecules. We ported a standard visualization applica-
tion (PyMOL) to a CAVE-like immersive virtual envi-
ronment (IVE), then invited three separate biochemistry
research groups—people who use PyMOL routinely on
desktop computers—to examine their favorite molecule
in the IVE. Within ninety minutes of immersive inves-
tigation, each group reported a new discovery about the
qualitative structure of that molecule. We believe that
the immersive environment facilitated these discoveries
by supporting and facilitating the natural spatial reason-
ing abilities of its users.
An immersive virtual environment is a combination of hard-
ware and software that provides a psychophysical experi-
ence of being surrounded by a computer-generated scene
(see Figure 1). Immersive virtual environments provide
users with an egocentric three-dimensional perspective:
users are immersed in a virtual world, where they can ex-
plore complex spatial systems by looking through them,
walking around them, and viewing them from different per-
spectives. Immersive environments may help people see
and understand the structure of complex three-dimensional
datasets; in contrast to more traditional graphics worksta-
tions, these environments allow one to visualize data us-
ing the well-practiced, non-concious analysis that automati-
cally accompanies an embodied, egocentric visual perspec-
tive. There are several studies that have investigated the
added value of immersive environments (Pausch, Proffitt, &
Williams 1997; Ruddle, Payne, & Jones 1999; Arns, Cruz-
Neira, & Cook 1999; Swan et al. 2003; Gruchalla 2004;
Schulze et al. 2005; Demiralp et al. 2006). However, the
results of these studies are mixed and the issue is somewhat
controversial. There are few studies that clearly demonstrate
the effectiveness of immersive environments for real-world
problems, and none that approach this issue from the stand-
point of qualitative reasoning. Our study does so, and our
results indicate that experts understand more about the ge-
ometry of biomolecules if they use an immersive environ-
ment than if they use the same visualization tools on a stan-
dard desktop. Within ninety minutes of immersive inves-
tigation, each of the three groups in our study reported a
new discovery about the qualitative structure of an important
biomolecule—molecules that these groups had been study-
ing for years in with the same software visualization tool on
desktop environments.
Immersive visualization has long been proposed as a
means to analyze the complex three-dimensional structure
of biological molecules (Ihlenfeldt 1997), and it is used by
numerous investigators in basic research and industrial set-
tings. Qualitative spatial analysis of the structure of these
molecules at a range of scales is essential, because their
overall three-dimensional configuration dictates the atomic
interactions that are the basis of their function. Understand-
ing the geometry of the building blocks of a biomolecule,
and their relationships, is key to many of the grand-challenge
problems in biochemistry: the rational design of drugs that
enhance or inhibit molecular activity, the understanding of
how steps in embryonic development normally proceed or
go wrong in the presence of genetic mutations of molecular
structure, and so on.
This paper documents a pilot study in which three sepa-
rate groups of biochemists visualized and interacted with in-
dividual biological molecules in a CAVE-like IVE. In each
case, the immersive working session yielded new insights
that the same biochemists had not previously achieved with
their extensive use of the same visualization package on
standard desktop computer displays. Large-scale spatial
features, such as pockets and ridges, were readily identi-
Figure 1: A user interacting with a PyMOL visualization of
a molecular surface inside a CAVE-like immersive virtual
environment, which provides the opportunity to visualize the
molecule using normal, everyday-world perceptual abilities
that have been tuned and practiced from birth.
fied when walking around the molecule displayed at human
scale.
Methods
Three University of Colorado at Boulder (UCB) biochem-
istry research groups were invited to study a molecule of
their choice—one central to their current research—in a
FakeSpace Flex, a CAVE-like immersive virtual environ-
ment. The research groups had each intensively stud-
ied their chosen molecule using non-immersive visualiza-
tion techniques—the desktop version of PyMOL, a pop-
ular open-source molecular visualization system (DeLano
2002)—for at least a year prior to conducting their research
in the IVE. We ported this same tool to a stereoscopic, inter-
active IVE (Gruchalla, Marbach, & Dubin 2007) to provide
some informal control in our study.
The Flex is configurable large-screen projection-based
12’x12’x10’ theater, consisting of four walls: three rear-
projected screens measuring 12’x10’ that form the right
wall, back wall, and left wall of the IVE. The fourth wall
is the 12’x12’ floor that is projected from above. A three-
dimensional effect is created inside the IVE through ac-
tive stereo projection and motion parallax. Stereo projec-
tion is achieved by projecting two images in sequence on
each screen: an image for the viewer’s left eye, followed
by an image for the viewer’s right eye. Viewers wear active
stereo LCD shutter glasses to view the stereoscopic images.
Infrared emitters synchronize the glasses with the graphics
pipes. When the computer renders the image for the left
eye, the right eye shutter is closed. Similarly, when the com-
puter renders the image for the right eye, the left eye shut-
ter is closed. This shuttering action creates the illusion of
three-dimensional images. A motion parallax is supported
by tracking the position and orientation of the viewer’s head
and using this information to generate an egocentric per-
spective. Virtual objects can be manipulated inside the IVE
using a tracked wand.
PyMOL (DeLano 2002) is a powerful and versatile open-
source, cross-platform real-time molecular visualization
system that supports standard representations for molecu-
lar structures (e.g., wire bonds, cylinders, spheres, ball-and-
stick, dot surfaces, solid surfaces, wire meshes, backbone
ribbons, and cartoon ribbons). PyMOLs primary interface is
an embedded Python interpreter, which is the basis for its so-
phistication. Our immersive port of PyMOL allows users to
view PyMOL visualizations in a head-tracked IVE and ma-
nipulate molecular structures using a six-degree-of-freedom
input device. Only the visualization and 3D interaction ele-
ments of PyMOL were ported to the IVE; its python-based
command-line interface ran on a desktop computer. The vi-
sualization is composed (e.g., loading pdb files, choosing
representations, selecting colormaps, ...) using the PyMOL
command-line interface on this desktop, then viewed and
manipulated in the IVE. Clearly, an IVE is poorly suited
to support a command-line interface. Dividing the work-
flow between the two environments allows all the power and
sophistication of the command-line interface to be used to
construct the 3D model, while the visualization of the model
and the spatial reasoning about its nature can be done in the
3D space of the IVE.
In this environment, three biochemistry groups conducted
actual research about how the structure of their molecule re-
lates to its function:
The laboratory of Professor Arthur Pardi studying the
anti-VEGF aptamer (Ruckman et al. 1998)
The laboratory of Professor Natalie Ahn study-
ing the extracellular signal-regulated kinase ERK2
(1erk.pdb) (Zhang et al. 1994)
The laboratory of Professor Shelley Copley study-
ing the enzyme maleylacetoacetate isomerase
(1fw1.pdb) (Polekhina et al. 2001)
With one exception
1
, the participants had no previous ex-
perience in viewing or manipulating objects in the IVE.
Each group was given a brief introduction to the environ-
ment and how to manipulate molecular structures using the
wand. Each group worked for about 90 minutes, with three
of four members of the team working collaboratively inside
the IVE, while one team member controlled the content of
the visualization from a desktop computer using the PyMOL
command-line and desktop interfaces. This similar to a tra-
ditional team working session, in which one member group
would control the visualization from a desktop computer us-
ing the PyMOL command-line and desktop interface; how-
ever, in a traditional working session the rest of the team
would gather around the computer to view and try to under-
stand the resulting visualization.
1
Professor Pardi had toured the immersive facilities and seen
several immersive demos prior to the pilot study.
Results
Despite having a long and extensive research history with
their respective molecules, all three groups arrived at a new
insight from their 90-minute IVE research session. All of
these insights were similar, and all involved qualitative rea-
soning about geometry. Each group became newly aware
of a large spatial feature, such as an empty space or ridge,
that they had not noticed during their (considerable) previ-
ous PyMOL work with the molecule on desktop computer
monitors. In each case, the newly recognized feature led to
insights about the molecule’s function that follow directly
from geometry: how its pieces move, for instance, or how
they fit together. These are described in the following para-
graphs. Each group left with the intention of exploring a
new theoretical possibility based on these insights; the Ahn
group actually integrated a hypothesis concerning the struc-
ture into a new grant proposal.
The Copley group recognized an empty pocket indent-
ing from the surface in the enzyme maleylacetoacetate iso-
merase (MAAI) (see Figure 2). MAAI is normally a dimer,
in which the interface between the dimer molecules blocks
this pocket. However, the monomer of MAAI is similar to—
and is used by the Copley group as—a model for another
molecule, tetrachlorohydroquinone (TCHQ) dehalogenase,
which is a monomer. This group is studying how a key com-
ponent of the molecule’s active site, amino acid cysteine at
position 16 (cys16), interacts with substrate molecules that
must be able to diffuse into TCHQ dehalogenase in order
to reach cys16. The pocket represents a large enough open-
ing for such entry, with cys16 lying at its base (darkened
area in Figures 2c and 2d). When viewing this molecule on
workstations, the researchers had discounted this region as
a potential active site of TCHQ dehalogenase because they
did not judge it to be spacious enough for the substrate to
penetrate to cys16. The immersive visualization gave the re-
searchers the ability to stand inside the pocket, which gave
them enough information to reverse their decision.
The Ahn group was interested in a long “ridge” of poten-
tially interacting amino acids that link two important sites
of the enzyme ERK2 (see Figure 3). ERK2 is crucial com-
ponent of the machinery that underlies normal and malig-
nant cell production. Previous experiments had determined
that small conformational changes caused by mutations of
amino acids in the region shown in orange can cause a
change in shape all the way across the molecule, in the re-
gion shown in purple. Biochemists are interested in under-
standing how such conformational changes are transferred
across molecules, and the Ahn group used the IVE port of
PyMOL to investigate this issue in ERK2. During their im-
mersive investigation, they recognized the “ridge” between
these two regions (shown in green in figure 3) as a possible
physical linkage. This ridge could “transmit” changes in the
orange region across the molecule to the purple region.
The Pardi group was interested in understanding how the
complex surface regions of two molecules fit together in a
complementary way. They used the IVE’s six-degree-of-
freedom handheld wand to manipulate the positions and ori-
entations of the regulatory molecule VEGF and the anti-
VEGF aptamer. VEGF has been implicated in human mac-
Figure 2: A molecular “pocket” that was discovered in the
IVE: At top, the molecule (maleylacetoacetate isomerase)
is shown in stick-representation with the region of interest
shown with bright, non-dark-blue sticks; (a) is a view look-
ing down into the pocket, (b) is a side view of the molecule at
the same scale. The bottom two images show partial surface
views of the region of the molecule immediately surround-
ing the pocket, with the approximate inside “surface” of the
pocket in gold, and the amino acid cys16 in orange. (c) is a
view looking down into the pocket; the mouth of the pocket
corresponds to the region shown in the stick representation,
as indicated by the lines. For (d), the pocket was bisected by
the plane indicated by the line (x-axis) in (c), and rotated 90
degrees about the x-axis to yield a view of half of the pocket
seen in side view, corresponding to the portion of the pocket
at the top of (c). Panels (c) and (d) are to the same scale; the
width of visible pocket in (c) is approximately 10 angstroms.
ular degeneration, a progressive disease that causes loss of
high-acuity, central vision. The synthetic, anti-VEGF ap-
tamer has been shown to be effective in slowing the progres-
sion of macular degeneration; however, the ability of anti-
VEGF aptamer to inhibit the VEGF molecule is deterimined
by the quality of the fit between the two. Using the inter-
active capabilities of the IVE, the Pardi group discovered a
new possible fitting between the two molecules (see Figure
4).
Figure 3: A stick-figure model of ERK2 with regions of in-
terest shown in space-fill representation. Mutations in the
orange region are known to cause shape changes in the ma-
genta region. The “ridge” of green colored atoms, recog-
nized by the Ahn group in the IVE, is a possible linkage
between these regions.
Discussion
Reasoning about relationally generated space (such as a
pocket in a molecule) depends on the scale of presenta-
tion, varying points of view, and movement in and around
it. Thus, we suggest it is not surprising that the naturalis-
tic IVE display allowed each of the three research groups to
recognize important spatial features that they had previously
overlooked in small, flat-screen, computer displays that must
be indirectly manipulated with a mouse. The importance of
naturalistic viewing is supported by numerous elegant ex-
periments involving real-world, normal-size scenes (Purves
& Lotto 2003; Yang & Purves 2003). Probabilistic matching
of images like this provides a good explanation of numerous
visual phenomena. This is not surprising; perceptual experi-
ence has molded our genetic makeup and it is tuned by the
learning that each of us accumulates as we exist and develop
day-to-day (Geary & Huffman 2002). This kind of knowl-
edge is wired—and continuously rewired—in the neural cir-
cuitry of our brain, based on polysensory activities. That
is, cognitive structures are developed from perception and
action, and grounded in the physical interactions with the
Figure 4: The Pardi group used the 3D interactive wand
inside the IVE to investigate how the loop region of anti-
VEGF aptamer (cyan) fits to a site on the outside of VEGF
(magenta). The surfaces of both regions are show as meshes
surrounding stick-figure representations, with atoms consid-
ered important shown in space-fill view.
environment (Pecher & Zwaan 2005).
In this context, it makes complete sense that working in an
IVE allows people to reason more effectively about the ge-
ometry of biomolecules. Studies that demonstrate this effect
are surprisingly rare, though, and the added value of immer-
sion is controversial in the visualization community. This
study is part of a larger effort that addresses this broader is-
sue: a general definition of conditions under which the use of
fully interactive, three-dimensional, immersive visualization
adds value to research activities. As indicated in the previ-
ous section, our results suggest that short, intense sessions
of IVE viewing valuably augment more extensive, non-IVE
based research of molecular function. The underlying rea-
sons for this, we believe, are threefold:
First, spatial judgments are body-relative in everyday ac-
tivity (Hatfield 2003). It is easier, for example, to judge
whether you could crawl through a passageway in a cave
if you are in the cave and looking at the passageway than
if you are examining a five-inch-tall rendering of it on
a flat screen monitor. Our hypothesis is that examining
a molecule at a human scale made it easier for the bio-
chemists to reason about the different spatial structures.
Because the IVE presented the biomolecules at a natu-
ral and familiar scale—similar to the way many complex-
shaped, everyday objects appear in the world—it facili-
tated effective reasoning about there shape.
Second, the egocentric perspective improves spatial rea-
soning, object recognition, and stresses the role of action
in building knowledge. Much of this happens automati-
cally: people do not stop and think about how to move
their heads or bodies in order to get a better view of some-
thing. The IVE supports this very naturally. Its unique
features—natural body movements and well-practiced au-
tomatic brain function as the basis for examination of
the structure in question—is consistent with recent re-
search on embodied cognition, cognition that is based
on perceptual knowledge accumulated through what we
have encountered and manipulated with our bodies as
we move within and examine the world (Wilson 2002;
Wolputte 2002).
Finally, the collaborative nature of the environment facil-
itates collaborative reasoning about the data. The large
scale of the environment allowed multiple biochemistry
researchers to gather inside the environment simultane-
ously. All the groups commented that they found working
collaboratively in the IVE to be much easier than crowd-
ing around a small computer screen. The large scale made
it easy to see what atoms and regions another member of
the group was referring to. Often they used bodily refer-
ences to direct each other, such as, ”that group of bonds
near your left shoulder.
Reasoning about the geometry of objects has a long and
rich history in the qualitative reasoning field, and there are
interesting papers about ontologies, paradigms, techniques,
and applications for this in every QR workshop—beginning
with an augmented version of Hayes’s “pieces of stuff on-
tology that was presented at QR ’87 for reasoning about
collections of molecules (Collins 1987). A few QR sys-
tems have been built over the years specifically for rea-
soning about molecular structure (Bandini, Cattaneo, &
Stofella 1988). Most of the geometry-related work in the QR
community has involved mechanical devices, an application
(like biomolecules) where shape and function are intimately
inter-related. Iwasaki, Joskowicz, Nielsen, and Faltings
have made significant contributions to this over the years
(Joskowicz 1987; Iwasaki 1987; Nielsen 1987; 1988; Falt-
ings, Baechler, & Kun 1991; Tessler, Iwasaki, & Law 1993;
Faltings 1993; Sun & Faltings 1994; Joskowicz & Sacks
1997). There has also been some work in the QR commu-
nity that considers the cognitive science perspective along
with the representation and the geometry, notably from Ken
Forbus’s group (Ferguson & Forbus 1999; Forbus, Fergu-
son, & Usher 2000; Forbus, Tomai, & Usher 2003; 2005;
Lovett, Dehghani, & Forbus 2006).
The study reported here has a much more complicated
application area than most of these papers, and much less
lofty aims. We are not trying to simulate, design, or de-
duce anything. We rely on the human experts to figure out
what’s meaningful; we want simply to understand how im-
mersive environments support their reasoning about the ge-
ometry that factors into that determination. Because of the
comparative nature of our study, the ontology and the model
are pre-specified. Our goal is not to figure out whether a bet-
ter model or ontology exists for these purposes, as in many
interesting QR papers, e.g., (Pacheco, Escrig, & Toledo
2002) but rather to study how the presentation & interface
affects the spatial reasoning about the molecules. We are
not trying to generalize ideas about structure across applica-
tion domains, as in (Adorni et al. 1988) nor are we trying
build more-abstract modelling paradigms, as in the elegant
work of Escrig (which is concerned with many of the con-
cepts that arise here, like how things fit together) (Museros
& Escrig 2004). There are obviously many interesting prob-
lems to tackle involving the kinematics & dynamics of the
biomolecules in our study, as well as the role of geometry in
those processes, but these are “grand-challenge” problems
and outside our scope.
Conclusion
This pilot study suggests that immersive environments en-
hance the ability of human experts to reason about the ge-
ometry of complex biomolecules. It also contributes fur-
ther evidence to the general debate about the added value of
large-scale immersive environments in the investigation of
complex interactive spatial domains. The small sample size
and lack of formal controls, however, mean that the results
are only preliminary. The discoveries reported by the sci-
entists in the study may have been facilitated by the oppor-
tunity to use embodied perceptual mechanisms afforded by
the environment. Comments from the subjects suggest that
the environment may have also provided a much improved
collaborative atmosphere. Regardless of the specifics mech-
anisms, the results are very promising: all three user tests
in this study generated a new piece of science as a result of
their improved geometric reasoning about a complex prob-
lem.
Acknowledgements
We thank Geoffrey Dorn, Gwen Pech and Mick Coady of the
University of Colorado-Boulder, BP Center for Visualiza-
tion for their assistance, support and advice. We are grateful
to the members of the research groups who participated in
this study. Professor Pardi was especially helpful in defining
the early stages of this project and in choosing PyMOL for
this work. We thank Sara Klingenstein for assisting in the
initial research on theoretical considerations. This project
was supported by a University of Colorado Butcher Award
to Professors Dubin and Pardi and by equipment donations
from NVIDIA.
References
Adorni, G.; Burdese, M.; Del Grosso, A.; Loddo, R.; and
Zucchini, A. 1988. A qualitative approach to structural
mechanics. In Proceedings of the International Workshop
on Qualitative Reasoning about Physical Systems.
Arns, L.; Cruz-Neira, C.; and Cook, D. 1999. The ben-
efits of statistical visualization in an immersive environ-
ment. In VR ’99: Proceedings of the IEEE Virtual Reality
1999 (VR’99), 88–95.
Bandini, S.; Cattaneo, G.; and Stofella, P. 1988. A theory
for molecule structures: The molecular onthology theory.
In Proceedings of the International Workshop on Qualita-
tive Reasoning about Physical Systems.
Collins, J. W. 1987. Reasoning about fluids via molecular
collections. In Proceedings of the International Workshop
on Qualitative Reasoning about Physical Systems.
DeLano, W. 2002. The pymol molecular graphics system.
http://www.pymol.org.
Demiralp, C.; Jackson, C.; Karelitz, D.; Zhang, S.; and
Laidlaw, D. H. 2006. Cave and fishtank virtual-reality dis-
plays: A qualitative and quantitative comparison. IEEE
Transactions on Visualization and Computer Graphics
12(3):323–330.
Faltings, B.; Baechler, E.; and Kun, S. 1991. Efficient
qualitative kinematics. In Proceedings of the International
Workshop on Qualitative Reasoning about Physical Sys-
tems.
Faltings, B. 1993. Qualitative structural analysis using
diagrammatic reasoning. In Proceedings of the Interna-
tional Workshop on Qualitative Reasoning about Physical
Systems.
Ferguson, R., and Forbus, K. 1999. Georep: A flexible
tool for spatial representation of line drawings. In Proceed-
ings of the International Workshop on Qualitative Reason-
ing about Physical Systems.
Forbus, K. D.; Ferguson, R. W.; and Usher, J. M. 2000.
Towards a computational model of sketching. In Proceed-
ings of the International Workshop on Qualitative Reason-
ing about Physical Systems.
Forbus, K. D.; Tomai, E.; and Usher, J. 2003. Qualita-
tive spatial reasoning for visual grouping in sketches. In
Proceedings of the International Workshop on Qualitative
Reasoning about Physical Systems.
Forbus, K. D.; Tomai, E.; and Usher, J. 2005. Solving
everyday physical reasoning problems by analogy using
sketches. In Proceedings of the International Workshop
on Qualitative Reasoning about Physical Systems.
Geary, D., and Huffman, K. 2002. Brain and cognitive
evolution: forms of modularity and functions of mind. Psy-
chol. Bull. 128(5).
Gruchalla, K.; Marbach, J.; and Dubin, M. 2007. Port-
ing legacy applications to immersive virtual environments:
A case study. In Proceedings of 2007 International Con-
ference on Computer Graphics Theory and Applications,
179–184.
Gruchalla, K. 2004. Immersive well-path editing: Investi-
gating the added value of immersion. In VR ’04: Proceed-
ings of the IEEE Virtual Reality 2004 (VR’04), 157–164.
Hatfield, G. 2003. Representation and constraints: the
inverse problem and the structure of visual space. Acta
Psychol. (Amst.) 114:355–378.
Ihlenfeldt, W. 1997. Virtual reality in chemistry. Journal
of Molecular Modeling 3:368–402.
Iwasaki, Y. 1987. Generating behavior equations from ex-
plicit representation of mechanisms. In Proceedings of the
International Workshop on Qualitative Reasoning about
Physical Systems.
Joskowicz, L., and Sacks, E. 1997. Qualitative and quan-
titative mechanical assembly design. In Proceedings of the
International Workshop on Qualitative Reasoning about
Physical Systems.
Joskowicz, L. 1987. Shape and function in mechanical
devices. In Proceedings of the International Workshop on
Qualitative Reasoning about Physical Systems.
Lovett, A.; Dehghani, M.; and Forbus, K. 2006. Solv-
ing everyday physical reasoning problems by analogy us-
ing sketches. In Proceedings of the International Workshop
on Qualitative Reasoning about Physical Systems.
Museros, L., and Escrig, M. T. 2004. A qualitative the-
ory for shape representation and matching. In Proceed-
ings of the International Workshop on Qualitative Reason-
ing about Physical Systems.
Nielsen, P. 1987. A qualitative approach to mechanical
constraint. In Proceedings of the International Workshop
on Qualitative Reasoning about Physical Systems.
Nielsen, P. 1988. Qualitative mechanics: Envisioning the
clock. In Proceedings of the International Workshop on
Qualitative Reasoning about Physical Systems.
Pacheco, J.; Escrig, M. T.; and Toledo, F. 2002. Qualita-
tive spatial reasoning on three-dimensional orientation. In
Proceedings of the International Workshop on Qualitative
Reasoning about Physical Systems.
Pausch, R.; Proffitt, D.; and Williams, G. 1997. Quantify-
ing immersion in virtual reality. In Proceedings of the 24th
annual conference on Computer graphics and interactive
techniques, 13–18. ACM Press/Addison-Wesley Publish-
ing Co.
Pecher, D., and Zwaan, R. 2005. Introduction to grounding
cognition. In Pecher, D., and Zwaan, R., eds., Grounding
cognition, 1–7. Cambridge, England: Cambridge Univer-
sity Press.
Polekhina, G.; Board, P.; Blackburn, A.; and Parker,
M. 2001. Crystal structure of maleylacetoacetate iso-
merase/glutathione transferase zeta reveals the molecular
basis for its remarkable catalytic promiscuity. Biochem-
istry 40(6):567–576.
Purves, D., and Lotto, B. 2003. In Why We See What
We Do: An Empirical Theory of Vision. Sunderland, MA:
Sinauer.
Ruckman, J.; Green, L.; Beeson, J.; Waugh, S.; Gillette,
W.; Henninger, D.; Claesson-Welsh, L.; and Janjic, N.
1998. 2’-fluoropyrimidine rna-based aptamers to the
165-amino acid form of vascular endothelial growth fac-
tor(vegf(165)) - inhibition of receptor binding and vegf-
induced vascular permeability through interactions requir-
ing the exon 7-encoded domain. Journal of Biological
Chemistry 273:20556–20567.
Ruddle, R.; Payne, S.; and Jones, D. 1999. Navigating
large-scale virtual enviroments: What differences occur be-
tween helmet-mounted and desk-top displays? Presence:
Teleoperators and Virtual Environments 8:157–168.
Schulze, J.; Forsberg, A.; Kleppe; Zeleznik, R.; and Laid-
law, D. H. 2005. Characterizing the effect of level of im-
mersion on a 3D marking task. In Proceedings of HCI In-
ternational.
Sun, K., and Faltings, B. 1994. Supporting creative me-
chanical design. In Proceedings of the International Work-
shop on Qualitative Reasoning about Physical Systems.
Swan, J. E.; Gabbard, J. L.; Hix, D.; Schulman, R. S.; and
Kim, K. P. 2003. A comparative study of user performance
in a map-based virtual environment. In VR ’03: Proceed-
ings of IEEE Virtual Reality 2003 (VR’03), 259–266.
Tessler, S.; Iwasaki, Y.; and Law, K. 1993. Qualita-
tive structural analysis using diagrammatic reasoning. In
Proceedings of the International Workshop on Qualitative
Reasoning about Physical Systems.
Wilson, M. 2002. Six views of embodied cognition. Psy-
chon. Bull. Rev. 9(4):625–636.
Wolputte, S. V. 2002. Hang on to your self: of bodies,
embodiment, and selves. Ann. Rev. Anthropol. 33:251–269.
Yang, Z., and Purves, D. 2003. A statistical explanation of
visual spaces. Nat. Neurosci. 6(6):632–640.
Zhang, F.; Strand, A.; Robbins, D.; Cobb, M.; and Gold-
smith, E. 1994. Atomic structure of the map kinase erk2 at
2.3 a resolution. Nature 367:704–711.
... The three-dimensional stereo and motion parallaxes naturally increase the visibility of the separated lines, and users can directly interact with the visualization by brushing the coordinate planes with a virtual "brush" or "flashlight", easily highlighting the observations temporarily of interest. Additionally, it has been hypothesized [8] that the egocentric perspective may actually improve spatial reasoning through embodied cognition (i.e., the natural body movement and body-relative spatial judgments provided by an immersive environment engages well-practiced automatic brain function acquired by what we have encountered and manipulated with our bodies as we move within and examine the world.) Specific to the visualization of statistical graphics, there have been multiple controlled user studies [2,21,10] that have shown benefits to visualizing statistical graphics in immersive environments. ...
Conference Paper
Full-text available
We present a visualization-driven simulation system that tightly couples systems dynamics simulations with an immersive virtual environment to allow analysts to rapidly develop and test hypotheses in a high-dimensional parameter space. To accomplish this, we generalize the two-dimensional parallel-coordinates statistical graphic as an immersive " parallel-planes " visualization for multivariate time series emitted by simulations running in parallel with the visual-ization. In contrast to traditional parallel coordinate's mapping the multivariate dimensions onto coordinate axes represented by a series of parallel lines, we map pairs of the multivariate dimensions onto a series of parallel rectangles. As in the case of parallel coordinates, each individual observation in the dataset is mapped to a polyline whose vertices coincide with its coordinate values. Regions of the rectangles can be " brushed " to highlight and select observations of interest: a " slider " control allows the user to filter the observations by their time coordinate. In an immersive virtual environment, users interact with the parallel planes using a joystick that can select regions on the planes, manipulate selection, and filter time. The brushing and selection actions are used to both explore existing data as well as to launch additional simulations corresponding to the visually selected portions of the input parameter space. As soon as the new simulations complete, their resulting observations are displayed in the virtual environment. This tight feedback loop between simulation and immersive analytics accelerates users' realization of insights about the simulation and its output.
Article
Full-text available
Genetic and neurobiological research is reviewed as related to controversy over the extent to which neocortical organization and associated cognitive functions are genetically constrained or emerge through patterns of developmental experience. An evolutionary framework that accommodates genetic constraint and experiential modification of brain organization and cognitive function is then proposed. The authors argue that 4 forms of modularity and 3 forms of neural and cognitive plasticity define the relation between genetic constraint and the influence of developmental experience. For humans, the result is the ontogenetic emergence of functional modules in the domains of folk psychology, folk biology, and folk physics. The authors present a taxonomy of these modules and review associated research relating to brain and cognitive plasticity in these domains.
Article
Full-text available
Abstract We describe a unified approach to computer-aided mechanical assembly design in which all design tasks are performed within a single com-putational paradigm supported by integrated de-sign software. We have developed a prototype design environment for planar assemblies, called HIPAIR, that supports diverse design tasks. We organize the design tasks around the fundamen-tal task of contact, analysis, which we automate by configuration space computation. Configura-tion space is a complete, concise representation of rigid body interactions that contains the requisite qualitative and quantitative contact information for all design tasks. We describe practical algo-rithms for the key tasks of dynamical simulation and kinematic tolerancing. HIPAIR allows de-signers to perform computations that lie outside
Article
Full-text available
In thefieldof CivilEngineering,Knowledge Engineeringtechniqueshave been appliedto safetyevaluation,damage assessmentand designof structures, but applicationsare usuallylimitedto specificcases. In thispaper, a general approachtostructural mechanic problems,basedon QualitativePhysics,isdis- cussed,providingtheway tomodel and reasonabouta wide classofstructures as physicaldevices. Some aspectsof thetheoryproposed by De Kleerand Brown, recognizedto be particularly suitableforstructural mechanic problems, have been partially reformulated,takingintoaccountthevectorialnatureofthe problem. The resultsobtainedby theanalysisof some planeframe casespro- videsuggestionstoimprove thequalitative model ofthestructures and thepro- cessof findingtheirbehaviouraldescription .
Article
Full-text available
We present a user study comparing multiple immersive environments for the performance of a task. We expected them to perform differently but it was not obvious how they would differ. Our analysis aimed to discover the environment in which subjects performed best and the environment they preferred. The task, derived from developmental biology, requires a user interactively to label spheres distributed in a volume by marking the spheres in 3D. We tested three virtual reality displays: a CAVE-like environment, a single-wall display, and a desktop system (fish tank virtual reality). All displays supported head-tracked stereo viewing and a tracked wand device. The task conditions varied in field of view, apparent size of the spheres, and whether the user was standing or sitting. We also varied the number of targets that had already been marked when the user began a trial. Popular user interaction techniques allow the user to change viewpoint and do the marking. Data we collected led to four significant findings: a) in the Cave the subjects preferred medium sized or large spheres over small spheres, b) when only a few of the targets have been marked, larger spheres were marked faster than small spheres, c) large spheres are marked most accurately, and d) our wall display is not comparable to our fish tank virtual reality environment when the spheres are small. Additionally, occlusion and larger field of view inhibited performance in the Cave more than at the fish tank when the task was dominated by visual search. (a) (b) (c) Figure 1: Marking spheres in different virtual environments: (a) at the fish tank, (b) at the single wall, and (c) in the Cave.
Conference Paper
Knowledge-based CAD systems limit designers’ creativity by constraining them to work with the prototypes provided by the systems’ knowledge bases. We investigate knowledge-based CAD systems capable of supporting creative designs in the example domain of elementary mechanisms. We present a technique based on qualitative explanations which allows a designer to extend the knowledge base by demonstrating a structure which implements a function in a creative way. Structure is defined as the geometry of the parts, and function using a general logical language based on qualitative physics. We argue that the technique can accommodate any creative design in the example domain, and we demonstrate the technique using an example of a creative design. The use of qualitative physics as a tool for extensible knowledge-based systems points out a new and promising application area for qualitative physics.
Article
We believe that qualitative spatial reasoning provides a bridge between perception and cognition, by using visual computations to construct structural descriptions that have functional significance. We provide evidence for this hypothesis by describing how qualitative spatial reasoning can be used to model aspects of visual structure in sketches. We begin by outlining the nuSketch spatial reasoning architecture, including the representation of glyphs and sketches and the use of qualitative topology and Voronoi diagrams to construct spatial representations. We then describe our use of spatial analogies as a means for exploring the structure of visual representations. Three concepts of visual structure in sketches are introduced: connected glyph groups, contained glyph groups, and positional relations. We show that by using visual reasoning techniques to compute these qualitative descriptions, spatial analogies involving sketches are significantly improved.
Article
The intention of this textbook is not to dismantle a current conception of vision and replace it with something entirely different. Since there is at present no consensus about the link between visual physiology and perception, there is nothing to dismantle, even if that was the aim. As it is, the intention is simply to build upon present knowledge a framework for understanding vision that is consistent with the perceptual evidence. (PsycINFO Database Record (c) 2012 APA, all rights reserved)