Content uploaded by Ralf Schneider
Author content
All content in this area was uploaded by Ralf Schneider on May 22, 2019
Content may be subject to copyright.
Mixed Reality Redefines Spatial Visualization Opportunities
Ralf Schneider, Syracuse University
Introduction
As the digitally native generation enter college level design pro-
grams, their comfort level working with Mixed Reality (MR)
technology is becoming ever more innate.
CAD tools are increasingly integrated in high school curricula 1,
which changes the beginning design student’s expectations to
technology in the classroom. Playful use with smartphone tech-
nology is evident and recent development of augmented reality
(AR) capabilities in tablet computers and smart phones allow
the combination of real and virtual world imagery.
The physical context of the classroom will be enhanced by the
MR context. Augmented Reality (AR) allows partial immersion
while Virtual Reality (VR) is fully immersive. Transitioning be-
tween the physical and digital context will be a challenge when
a whole class experiments in a multi modal context. How
should instructors utilize the opportunity of immersive learning
through MR?
For design educators it is important to address this trend and
support students’ desire to work with digital content and tools
and playfully explore boundaries of what is now possible.
AR and VR on smart phones
The smart phone gaming app Pokémon 2 paved the way for the
wide spread use of augmented reality apps on smartphones.
The device video camera feed is overlayed with digital content
that is GPS location specific. This development is not limited to
gaming applications, but also spreads to practical applications.
For example, Students can use the IKEA Place app 3 to envision
furniture in their dorm room. The app Torch 4 enables the user
to create interactive AR scenes. The app Sketchfab 5 allows the
user to view 3D models either in VR using google cardboard or
in AR by scanning a flat surface and placing the object in space.
Fig. 1: Torch AR App
The smartphone screen becomes a portal to the augmented re-
ality. Since most student have smart phones, these application
could be used to explore various use cases for sharing three-di-
mensional objects with others.
Spatial visualization
“Spatial computing is digital technology that interacts with us in
the places we live, work and play.”6 Recent hardware develop-
ments bring high-resolution imagery and precise interaction to
the user. Spatial computing allows for powerful VR and AR expe-
riences when the user immerses themselves with a head-
mounted device and six degrees of freedom for human com-
puter interaction.
Spatial computing enables spatial visualization, which is integral
to the design process. In traditional design education, students
learn 3D visualization, Boolean form generation operations us-
ing materials such as paper, wood, foam or clay and composi-
tion.
Students need to be “able to see in the mind's eye how various
parts fit or work together and what objects might look like from
different vantage points.” 7 For some students it is easier to
think in 3D than for others. 8
Ralf Schneider
Moreover, spatial visualization “deficits have major implications
for retention and inclusion”. 9 If students practice spatial visuali-
zation skills, they can improve their ability to envision three-di-
mensional object and to understand spatial relationships.
MR tools that support visual and spatial thinking are a fitting ex-
ample to illustrate this trend. Three-dimensional VR sketching
programs (Google Tiltbrush, Gravity Sketch, Autodesk Sugarhill,
Microsoft Maquette) provide an opportunity to fill the gap be-
tween 2D sketching and 3D CAD modeling. They enable an im-
mediate understanding of an idea in space and make it easier to
explore iterations early in the design process.
A compelling use of VR sketching is exploring design principles of
dominant, subdominant, and subordinate forms based on the
rectilinear volumes exercise developed by Rowena Reed Kostel-
low.
Fig. 2: Axis of dominant, subdominant, and subordinate forms
In describing the exercise, Reed Kostellow encourages the
designer to “…always conceive a design from all positions. Work
on a sturdy turntable and continually rotate the sketch to make
sure it “reads” from all directions.” 10
Since VR sketching immerses the user in a three dimensional
environment, this approach is easy to act upon. Intersecting
volumes is much easier to achieve in VR compared to working
with clay, Bristol board, foam or wood.
Fig. 3: Gravity sketch
The designer can develop many iterations quickly and arrange
them in space. Computational power and memory has enabled
this technique in CAD modeling tools such as rhino 3D. While
developing the form, iterations can be copied and moved to the
side (Figure 4). This allows for documenting the thought process
while refining form, proportion and composition.
Fig. 4: iterations in CAD
When used in this capacity, VR supports the design process in
the iteration phase. Materially realizing the form is still im-
portant to assess its haptic qualities.
Another advantage in the application of VR sketching in founda-
tion studies is the ability to scale shapes. This allows the creator
to assess objects intended to be handheld. Then, after scaling
them up to an environmental scale, the designer can walk
around in space.
All shapes and sketches created in VR can be exported as an ob-
ject file and used for reference in CAD programs.
Mixed Reality Redefines Spatial Visualization Opportunities
At the University of Cincinnati, students have experimented in
Professor Ming Tangs’ class with AR supporting physical proto-
typing. 11
Fig. 5: DAAP AR based Digi_Fab
A digital model, projected as a hologram using the HoloLens
serves as a “3D template” for making models (Figure 5). Based
on the digital intersection points in space, the designer is adher-
ing glue sticks with a hot glue gun to resemble the hologram.
The designer can walk around while working on the prototype
to assess the design in progress and conduct a critical dialogue
with others.
Fig. 6: Storyboard sketching in VR at Syracuse University
Students at Syracuse University have been introduced to MR
and the power of spatial visualization in two ways. Second year
students were introduced to VR sketching as part of their first
visualization course. This experience allowed experimenting
sketching in three dimensions compared to setting up a two or
three point perspective drawing on marker paper. Once these
skills are introduced, they can be expected as part of mid level
courses and senior studio courses.
Fourth year students navigated the mixed reality topic for a full
semester. This project based learning course covered oppor-
tunity identification, concept generation and presentation of a
mixed reality enhanced scenario of the future.
Fig. 7: VR sketching in studio at Syracuse University
During development process, the analog and digital go hand in
hand. Creating videos to communicate the final concept was
very effective.
Conclusion
There are three levels of complexity when implementing MR in
a course to aid spatial visualization.
1. Low barrier: Sketching in VR
Only requires the VR hardware and software set up.
Can be used in many ways throughout the design
process.
2. Medium barrier: Looking at holograms
This approach requires a head mounted display
(HMD) such as the HoloLens, the ability to create as-
sets and some Unity skills to install the program on
the HMD.
3. High barrier: Creating interactive MR experience
Interactivity needs to be programmed in Unity writing
scripts in C#. Tutorials and script examples support
the effort, however plenty of time needs to be dedi-
cated to develop the necessary skills.
Ralf Schneider
Notes
1 Schneider, Ralf. “Mixed Reality and Its Future in Design Education.”
Cincinnati, 2018. https://journals.uc.edu/index.php/ncbds/arti-
cle/view/813.
2 Wingfield, Nick, and Mike Isaac. “Pok�mon Go Brings Augmented
Reality to a Mass Audience.” The New York Times, 2017, sec. Technol-
ogy. https://www.nytimes.com/2016/07/12/technology/pokemon-go-
brings-augmented-reality-to-a-mass-audience.html.
3 Pardes, Arielle. "Ikea's New App shows the Practical Promise of Aug-
mented Reality." Wired -09-20T16:18:55.922Z 2017Web. Feb 13,
2019 <https://www.wired.com/story/ikea-place-ar-kit-augmented-
reality/>.
4 “TORCH AR: Mobile Augmented Reality Prototyping and Design.
Code-Free Design.” Accessed February 13, 2019.
https://www.torch.app/.
5 “Sketchfab - Your 3D Content on Web, Mobile, AR, and VR.” Sketch-
fab. Accessed February 15, 2019. https://sketchfab.com.
6 Magic leap. “What Is Spatial Computing?,” July 17, 2018. https://cre-
ator.magicleap.com/learn/guides/design-spatial-computing.
7 "SHERYL SORBY." ASEE Prism 24, no. 4 (12, 2014): 42.
https://search.proquest.com/docview/1667166732?ac-
countid=14214. (accessed January 29, 2019).
8 Sorby, S. A., and B. J. Baartmans. “The Development and Assessment
of a Course for Enhancing the 3-D Spatial Visualization Skills of First
Year Engineering Students,” Journal of Engineering Education 89, 89,
no. 3 (2000): 301-307+387-392.
9 Lord, Mary. “The Mind’s Eye.” ASEE Prism. ASEE, October 2018.
http://www.asee-prism.org/the-minds-eye/.
10 The Rowena Reed Kostellow Fund. http://www.rowena-
fund.org/methodology/problem1-rectilinear.html
11 Tang, Ming. “Virtual and Augmented Reality in Architectural Design
and Education,” Vol. 34. National Conference on the Beginning Design
Student 34, 2018. https://journals.uc.edu/index.php/ncbds/arti-
cle/view/812.
Fig. 1: Torch AR App: “TORCH AR: Mobile Augmented Reality Prototyp-
ing and Design. Code-Free Design.” Accessed February 13, 2019.
https://www.torch.app/.
Fig. 2: http://www.rowenafund.org/methodology/problem1-rectilin-
ear.html
Fig. 3: Ralf Schneider, SU Gravity Sketch 2018, screenshot
Fig. 4: Simon Williamson, Behance.net
Fig. 5: Tang, Ming. “AR Based Digi_Fab.” Ming3D, 2019.
http://ming3d.com/new/2019/01/03/ar-based-digi_fab/.
Fig. 6: Sophia Jaberi, SU IID Studio 2018, screenshot
Fig. 7: Ralf Schneider, SU IID Studio 2018