SECRET WHISPERS & TRANSMOGRIFICATIONS:
A case study in online teaching of Augmented Reality technology for
collaborative design production.
GARVIN GOEPEL1and KRISTOF CROLLA2
1The Chinese University of Hong Kong
2The University of Hong Kong
Abstract. This paper focusses on teaching the integration of
Augmented (AR) and Mixed Reality (MR), combined referred to
as Extended-Reality (XR), and photogrammetry technology into
handicraft using an online-taught digital fabrication workshop as an
educational case study. Set up in response to restrictions from Covid-19,
workshop “Secret Whispers & Transmogrifications” had students and
instructors around the world participate in a course that challenged our
understanding of educating craft and technology without the necessity
of physical presence. The integration of AR into craftsmanship
enhances architectural design and fabrication processes as it overlays
computation-driven information onto the hands of the end user. These
computer-numerically-controlled workflows incorporate and rely on
manual actions as an integral part of a process that is typified by
inevitable, unpredictable, human error. In doing so, the workshop
questions common infatuation with precision in digital fabrication
and construction by striving for alternative approaches that embrace
the inaccuracies and imprecisions innate to technologically-augmented
human craftsmanship. Participants took part in a hands-on clay
modelling “secret whispers” experiment that was designed to introduce
theoretical concepts and applications of XR technology into the
production workflows. This paper concludes by highlighting that the
accessibility of today’s technology enables AR-enhanced craftsmanship
to be successfully taught remotely and online.
Keywords. Collaborative design; augmented-reality; mixed reality;
human-computer interaction; tolerances and error.
Workshop “Secret Whispers & Transmogrifications” had students and instructors
around the world participate in a course that challenged our understanding of
educating craft and technology without the necessity of physical presence as
being set up in response to restrictions from Covid-19. An online taught digital
PROJECTIONS, Proceedings of the 26th International Conference of the Association for Computer-Aided
Architectural Design Research in Asia (CAADRIA) 2021, Volume 2, 21-30. © 2021 and published by the
Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong.
22 G. GOEPEL AND K. CROLLA
fabrication workshop was used as an educational case study on teaching the
integration of Augmented (AR) and Mixed Reality (MR), combined referred to
as Extended-Reality (XR), and photogrammetry technology into handicraft.
1.1. COURSE FOCUS: INCREMENTAL SLIPPAGE
The workshop website describes how ”[...] Augmented Reality (AR)
integration into craftsmanship promises a radical overhaul of architecture
and design production as it brings computational power directly to
people’s fingertips. Yet, with the hand becoming a key component in
these computer-numerically-controlled workflows, innate and unpredictable
human imprecision, inaccuracy, and error become an inevitable part of the
equation.” (Crolla et al., 2020). By seeking beauty in incremental slippage
from technologically augmented human craftsmanship, ”Secret Whispers &
Transmogrifications” challenged the ”Digital’s” obsession with control and
precision. Participants were exposed to both theoretical concepts and practical
applications of AR technology integration in design and production workflows
by participating in a hands-on ”secret whispers” experiment. Positioning itself
in a ”Post-Digital” context, the work employs alternative notational systems in
implementation methods that aim at humanising digital technologies through
interplay between digital and analogue material systems (Crolla, 2018).
1.2. COURSE CONTEXT: AR AND CRAFTSMANSHIP
With the arrival of easily accessible AR/MR technology, opportunities present
themselves for an increased and productive dialogue between collaborating
designers and craftsmen, providing greater local agency and prospects for more
diverse design output (Goepel and Crolla, 2020). Architects and engineers
commonly use AR applications to facilitate information extraction from design
information models to improve the efficiency and effectiveness of workers‘ tasks
(Chi et al., 2013; Chu et al., 2018). These include onsite applications where AR
can be seen implemented in Smart Helmets and Tablets, primarily for helping
engineers to make more accurate and rapid judgments for construction review
tasks (Ren et Al., 2017). In industrial settings, case studies of AR systems’ user
experiences have demonstrated their potential to reduce errors in assembly and
improve the quality of maintenance work (Aromaa et al., 2018).
Showcases for the integration of AR into fabrication and design processes in
architecture and the arts include work from peers that used MR for tasks such as
bricklaying (Franco, 2019) plywood construction (Jahn, Wit and Pazzi, 2019) steel
artwork production (Jahn et al. 2018), bamboo construction (Goepel and Crolla,
2020), and many more, indicating that a paradigm shift in manual production
has been set in motion. Instead of surrendering human skill to automation in
manufacturing, AR enhances the human capacities to participate in complex
processes through simplified instructions (Goepel, 2019).
We foresee human-computer interaction as in AR/MR to become far more
effective in a “Post-Digital” context than e.g. robotics or other forms
of computer-numerically-controlled (CNC) production, because AR enables
SECRET WHISPERS & TRANSMOGRIFICATIONS: 23
augmentation of onsite skill through the direct visual overlay of specific
holographic instructions onto manual actions (Goepel and Crolla, 2020).
1.3. COURSE TASK: SCULPTING AND 3D REFERENCING
The workflow and methodology applied in this workshop relied on 3D referencing
and 3D replication. This study builds up on prior research in which a series of
AR-aided clay sculpting methods were developed and tested in a demonstrator
case study and adds to this more elaborate AR-aided sculpting method for remote
The workshop set out as an experiment in which a set of fourteen sculptures,
of which digital model files were sourced online, was altered through several
morphing cycles that oscillated between the analogue and the digital world. Each
iteration began by hand-modelling a sculpture based on a given digital file through
the aid of holographic instructions, displayed through an AR application on the
participants‘ mobile devices. The result was then captured through multiple
photographs, taken with these devices’ high-resolution cameras, that were then
processed in a photogrammetry software. The resulting digital 3D geometry model
files were then passed on to the next person for the following sculpting cycle until
four iterations were achieved (see Figure 1, 2 and 3).
Figure 1. Workshop concept, based on “Chinese Whispers” (image by Jean Julien).
Figure 2. Left: Original sculpture. Middle left: Iteration 1 by Student X. Middle right:
Iteration 2 by Student Y. Right: Iteration 3 by Student Z.
To speed up the start of the workshop, the instructors preselected fourteen
24 G. GOEPEL AND K. CROLLA
figures from free access online libraries such as 3D warehouse. These figures were
resized and trimmed to similar-scale sculptures using a Grasshopper® script in 3D
modelling software Rhinoceros®. Mesh resolutions were automatically optimised
to be suitable for seamless AR streaming, keeping sufficient detail for precise
modelling. These models were distributed to each student for the first iteration
of the secret whisper experiment. Throughout the process, these were gradually
transmogrified in three further steps until they reached their final form (see Figure
2 and 3).
Figure 3. Sculpture transmogrifications.
2.2. BESPOKE AR-APP
Fologram®, a Grasshopper® software add-on, was used to stream model data
through a custom developed AR application to the mobile handheld devices. The
app references itself and the clay block in the real-world environment through an
image target placed on the edge of the clay block (see Figure 4).
The custom AR application starts by using a digital bounding box which equals
the size of the physical clay block with the to-be-modelled digital sculpture in
its centre. The digital sculpture is intersected with several planes to identify and
highlight its contouring profiles. An interactive parametric slider determines the
spacing between each plane in X, Y, and Z direction. Intersection points are
connected with a curve, resulting in several silhouettes in each direction. Using
simple control buttons inside the app, users can switch between the X, Y, and Z
axis, and through sliders one can decide which silhouette is shown.
SECRET WHISPERS & TRANSMOGRIFICATIONS: 25
Figure 4. View through smartphone of customised AR application, showcasing the overlay of
holographic instructions on top of the physical clay model.
Figure 5. Analogue sculpting process informed by holographic instructions.
Augmented reality is then used to holographically overlay this digital
information directly on top of the analogue sculpture. App controls give users
the real-time ability to switch between the display of the predefined contours and
silhouettes, allowing them to decide, as they sculpt, which necessary guides to
access to inform their addition or removal of material. This process is repeated
until an analogue interpretation of the digital file is accomplished (see Figure 5
26 G. GOEPEL AND K. CROLLA
Figure 6. Top: View through mobile device with holographic instructions and view on clay
model where mass is removed or added accordingly. Bottom: Sculpting while viewing through
mobile device with holographic instructions. .
Photogrammetry software packages Meshroom® and ReCap® were then used to
capture the analogue clay sculpture and bring it back into a digital modelling
environment. A photo series taken by the participants was used as input to
regenerate a digital 3D model approximation of the analogue model (see Figure 7).
This digital model was then passed to the next participant for the next modelling
SECRET WHISPERS & TRANSMOGRIFICATIONS: 27
Figure 7. View of scanned clay model in Meshroom photogrammetry software.
Photogrammetry typically requires dozens of pictures of one object, a software
setup, processing time to convert the images into a 3D model, and post-processing
time for cleaning the mesh outcome in a 3D modelling software. This process can
take up to a few hours, depending on the used photogrammetry software.
Good photography skills are crucial to achieve successful photogrammetry
result. The object should be captured from all 360-degree angles and from the
top, middle and bottom. Image quality and resolution also affects the scan results,
with more recent handheld devices with better camera specifications typically
producing better results. Artificial lighting can possibly unfavourably affect the
scan outcome, whereas daylight conditions typically increase the quality of the
final scan. Used computer’s processing power also played an important role, as
one might only see an unsuccessful result after hours of processing, impacting
timelines. Clay properties also effected the scanning result: darker sculpting
clays resulted in less detailed scans than light-coloured material, as shadows and
highlights of the sculpted clay seemed to be less recognisable for the cameras on
Unique characteristics and qualities emerged as the transmogrifications by
multiple authors accumulated with each step. Three cycles were completed,
producing a total of 56 sculptures. Their digital models were rendered for display
in an online exhibition presenting the collection in a virtual space visible by means
of walkthroughs with 360-degree views (see Figure 8).
This exhibition and website was created with free online virtual tour
creator Theasys®, a tool which allows for a series of digital renderings to be
interconnected through a navigation system to create and publish a 360° Virtual
Tour. The exhibition can be either experienced with virtual reality goggles as an
immersive 3D environment or through web browsers where visitors can navigate
through the exhibition space by clicking on arrows and using the mouse or finger
to direct the view.
28 G. GOEPEL AND K. CROLLA
Figure 8. Virtual exhibition.
Exhibition visitors can observe and compare the sculpture transmogrifications
which are highly informed by the incremental slippage in the translation of the
sculptures. Rather than striving for accuracy in precise replication, the show
highlights the emergent characteristics in each iteration, leaving within each
sculpture a trace to the multiple co-authors’ hands, making the overall a collective
The photogrammetric scan and the human sculpting hand became the two
contradicting authors, placed in a bi-directional dialogue between accuracy and
transgression throughout a modelling process centred around the holographic
guidelines. A high-resolution scan, for example, will directly affect a following
iteration’s similarity far more than a low-resolution scan, and a precise replication
of the holographic overlay will sway the following scan more favourably than an
unprecise copy. The level of participants‘ prior sculpting skill also substantially
differed within the group, which can be observed in the execution of the models.
The participants’ learning curve for working with holographic guides also affected
the resolution of the outcomes. Each participant’s first iteration can therefore
be seen as a first prototype with the technique, rather than as a well-executed,
holographically guided model which can be found the last iterations.
4. Discussion and future opportunities
The scanning technology used in this workshop relied on photogrammetry, which
today can be accessed with free software and does not require additional hardware.
The integration in latest mobile devices of LiDAR hardware, a technology found
for example in the fourth generation Ipad Pro and the Iphone 12 Pro, presents an
increase in usability and quality of 3D scanning technology. LiDAR stands for
light detection and ranging. The LiDAR scanner measures how long it takes light
to reflect back from objects, so it can create a depth map of any space you’re
in (Apple, 2020). LiDAR has been used for several years now, for example in
SECRET WHISPERS & TRANSMOGRIFICATIONS: 29
self-driving cars, robotics and drones, but the integration into a mobile device
opens up a whole new world of possibilities for 3D scanning and the use of AR. It
allows devices to understand their surrounding space by mapping it in 3D, enabling
the accurate placement and interaction with virtual AR objects. It also allows for
the generation of 3D objects based on a quick scan with apps such as 3D Scanner
App. Within a few minutes one can create a meshed-out 3D object. The quality
of the 3D mesh scan is not as decent as the method presented in this paper yet,
but considering the acceleration of the workflow, we can see a potential use of
integration LiDAR mobile 3D scans for future applications.
XR integration today has become rather straightforward: workshop
participants were able to quickly grasp the setup and installation of the apps, and
an intuitive understanding of the holographic instructions could be observed. The
user-friendly interface of the Fologram app permitted participants to just follow
the instruction given by the bespoke guides. The Fologram app was streaming
information directly from a Grasshopper file, rather than being a standalone app,
so a prior knowledge of that platform was helpful, as this allowed users to adapt
and customise certain functionalities.
Future studies could benefit from incorporation of other softwares, such as
Unity Reflect, that allow sending data from third-party plugins such as Rhinoceros,
Revit, or Sketchup to a Unity Reflect cloud server or to a Reflect server on your
machine. This data is then pushed to the device of choice, such as IOS or Android
phones or the Unity Editor itself where the data can be enhanced. This improves
overall workflow, because a live data link can be set up between the base geometry
and the applications. This data change can be viewed simultaneously by users
across multiple applications.
Today, AR enhanced craftsmanship has the ability to be taught remotely and online
around the world by the aid of XR integration. This study demonstrates that
XR and photogrammetry technology have the ability to enhance clay modelling
craftsmanship, allowing for a technology-driven democratisation of skill. The
incremental slippage between sculpting iterations showcases how, as the hand
becomes a key component in these computer-numerically-controlled workflows,
inevitable innate and unpredictable human imprecision, inaccuracy, and error can
become a constructive, qualitative part of the creative process. In doing so,
this study proposes a counter-narrative to common research on robotic or CNC
fabrication aiming for high accuracy and precision.
This workshop was taught by Dr. Kristof Crolla (LEAD/HKU), Garvin Goepel
(CUHK), Keung Shing Fung, Adwin (CUHK), and Chan Tsz Sun, Ovan (CUHK)
as part of the Digital FUTURES World Workshops from 27 June to 03 July 2020,
with participants Akshata Ghuge, Duong Nguyen, Xu Jiaqi, Xu Ke, Li Yuan, Yao
Jiaxian, Hong Yuhan, Luisa Gutierrez, Zhuang Xinwei. Visualisations were made
by Julien Klisz (LEAD).
30 G. GOEPEL AND K. CROLLA
“Unity Reflect” : 2020. Available from <https://unity.com/> (accessed 4th February 2021).
“3D Scanner App” : 2020. Available from <https://www.3dscannerapp.com/> (accessed 4th
“Theasys” : 2020. Available from <https://www.theasys.io/> (accessed 4th February 2021).
“ReCap” : 2020. Available from <https://www.autodesk.com/products/recap/overview>
(accessed 4th February 2021).
“Meshroom” : 2020. Available from <https://alicevision.org/> (accessed 4th February 2021).
“Grasshopper3D” : 2020. Available from <https://www.grasshopper3d.com/> (accessed 4th
“Fologram” : 2020. Available from <https://fologram.com/> (accessed 4th February 2021).
“Rhinoceros3D” : 2020. Available from <https://www.rhino3d.com/> (accessed 4th February
“Apple Website: I-Phone 12 Pro and pro max” : 2020. Available from <https://www.appl
(accessed 10th December 2020).
Chi, H.L., Kang, S.-C. and Wang, X.: 2013, Research Trends and Opportunities of Augmented
Reality Applications in Architecture, Engineering, and Construction, Automation in
Chu, M., Matthews, J. and Love, P.E.D.: 2018, Integrating Mobile Building Information
Modelling and Augmented Reality Systems: An Experimental Study, Automation in
Crolla, K.: 2018, Building simplexity: the ‘more or less’ of post-digital architecture practice,
Ph.D. Thesis, RMIT University.
Crolla, K., Goepel, G., Keung, S.F.A. and Chan, T.A.O.: 2020, “Secret Whispers &
Transmogrifications Workshop Website” . Available from <https://secretwhispers2020.w
ixsite.com/secretwhispers> (accessed 10th December 2020).
Franco, J.T.: 2019, “This is How a Complex Brick Wall is Built Using Augmented Reality”
. Available from <https://www.archdaily.com/908618/this-is-how-a-complex-brick-wall-is
-built-using-augmented-reality.ISSN0719-8884> (accessed June 9, 2020).
Goepel, G.: 2019, Augmented Construction - Impact and opportunity of Mixed Reality
integration in Architectural Design Implementation, In Ubiquity and Autonomy,
Proceedings of the 39th Annual Conference of the Association for Computer Aided Design
in Architecture (ACADIA), Austin Texas, 430-437.
Goepel, G. and Crolla, K.: 2020, AUGMENTED REALITY-BASED COLLABORATION -
ARgan, a bamboo art installation case study, In RE: Anthropocene, Design in the Age of
Humans - Proceedings of the 25th CAADRIA Conference, Bangkok, 313-322.
Jahn, G., Newnham, C., van den Berg, N. and Beanland, M.: 2018, Making in Mixed
Reality: Holographic design, fabrication, assembly and analysis of woven steel struc-tures,
Recalibration: On Imprecision and Infidelity Structures, Proceedings of the 38th Annual
Conference of the Association for Computer Aided Design in Architecture 2018 (ACADIA),
Mexico City, 88-97.
Jahn, G., Newnham, C., van den Berg, N., Iraheta, M.I. and Wells, J.: 2019, Holographic
Construction“. In Impact: Design With All Senses, Design Modelling Symposium Berlin,
Jahn, G., Wit, A. and Pazzi, J.: 2019, “[BENT] Holographic handcraft in large-scale steam-bent
timber structures, Ubiquity and Autonomy, Proceedings of the 39th Annual Conference of the
Association for Computer Aided Design in Architecture (ACADIA), Austin Texas, 438-447.
Ren, J., Liu, Y. and Ruan, Z.: 2016, Architecture in an Age of Augmented Reality: Applications
and Practices for Mobile Intelligence BIM-based AR in the Entire Lifecycle, International
Conference on Electronic Information Technology and Intellectualization, Guangzhou,