Conference PaperPDF Available

Depth Camera Feedback for Guided Fabrication in Augmented Reality



Augmented reality environments have been demonstrated to assist with architectural fabrication tasks by displaying construction information at full scale and in context. However, this information typically needs to be sparse in order to prevent virtual models occluding a fabricators view of the physical environment, and this limits the application of augmented reality to tasks such as surface forming. To address this issue, we propose a method for guided fabrication in augmented reality using real time comparisons between depth scans of as built conditions and target conditions defined by design models. Through the design and fabrication of a small proof of concept prototype from paper strips, we demonstrate that guided fabrication is adequate for high speed, approximate and ad-hoc fabrication of complex surface geometries without the need for extensive rationalization for fabrication constraints or explicit documentation of parts. We further show how this method generalizes to other processes such as additive fabrication or part placement and speculate on the implications of accessible real time depth data from the HoloLens within Grasshopper.
Depth Camera Feedback for Guided
Fabrication in Augmented Reality
1 A proof of concept prototype
assembled from free-formed
paper strips using depth camera
feedback in augmented reality.
Gwyllim Jahn
Cameron Newnham
Nick van den Berg
Augmented reality environments have been demonstrated to assist with architectural fabrication
tasks by displaying construction information at full scale and in context. However, this informa-
tion typically needs to be sparse in order to prevent virtual models occluding a fabricators view
of the physical environment, and this limits the application of augmented reality to tasks such as
surface forming. To address this issue, we propose a method for guided fabrication in augmented
reality using real time comparisons between depth scans of as built conditions and target condi-
tions dened by design models. Through the design and fabrication of a small proof of concept
prototype from paper strips, we demonstrate that guided fabrication is adequate for high speed,
approximate and ad-hoc fabrication of complex surface geometries without the need for exten-
sive rationalization for fabrication constraints or explicit documentation of parts. We further show
how this method generalizes to other processes such as additive fabrication or part placement
and speculate on the implications of accessible real time depth data from the HoloLens within
Fabrication in augmented reality has proven to be an effective
and expedient method for forming and assembling complex
structures from steel (Jahn et al. 2018), bricks (Mitterberger et
al. 2020; Jahn et al. 2019) and steam bent timber (Jahn, Wit, and
Pazzi 2019). Recent work has also demonstrated augmented
reality applications working with exible materials such as bent
bamboo (Goepel and Crolla 2020) or non-uniform and unpro-
cessed timber (Lok 2022). However there remain few examples
of augmented reality fabrication approaches to forming
complex surface topologies due to the challenge of correctly
judging surface depth with current generation augmented reality
displays. This is a limitation on the design space afforded by
augmented reality fabrication, as surface forming is an ecient
way to describe geometry typically generated by CAD modeling
software. While there exists several computer aided manufac-
turing approaches to materializing surface geometry, each of
these approaches is not without limitations.
Subtractive forming complex surface models introduces the
challenge of avoiding undercuts, while additive manufacturing
is notoriously slow and introduces geometric constraints such
as draft angles and requirements for inll and support material.
Surfaces can be rationalized into smaller parts such as panels,
shingles, or strips, but this rationalization introduces design,
logistics and assembly complexity. By comparison, ad-hoc fabri-
cation of complex surface geometries has been demonstrated
to be possible by artists like Henrique Oliviera or Lucy Irvine,
though this negates the advantages of digital design to produc-
tion workows. This work is motivated by the desire to capitalize
on the eciency and formal possibilities of ad-hoc fabrication of
complex surface geometries while also facilitating the approxi-
mate adherence to digital design models. We are also motivated
by the desire to overcome the current limitations of augmented
reality approaches to surface fabrication by demonstrating a
generalizable approach to guiding fabricators using real time
comparisons between point cloud scans of as-built conditions
and target conditions dened by digital design models.
This research demonstrates a method for fast and approximate
forming of surfaces with arbitrarily complex topology and
geometry by hand in augmented reality. Within our approach,
the only documentation required is the design model of the
target surface, thereby simplifying or eliminating requirements
for rationalizing surfaces for fabrication or discretizing design
models to parts and joints (Figure 3). We aim to improve
the feasibility of fabricating complex surface geometry in
augmented reality compared to existing approaches, measured
by improvements in fabrication time and formal complexity, and
provide evidence of this method in a proof-of-concept prototype
(Figure 1) completed over the course of a single day by a team
of 3 inexperienced fabricators. The proof-of-concept project is
2 A detail view of the paper
3 Hologram of the design model at
1:1. Mesh triangles are rendered
as occlusion geometry with lines
representing the mesh quads on
3TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
used to explore how real time feedback can assist fabricators
in forming complex surface structures from exible, lightweight
materials by improving legibility of digital models viewed in
augmented reality and enabling the improvisation of ad-hoc
strategies for their materialization.
Formwork-free Approaches to Fabricating Surfaces from
Fabrication of curved surfaces typically involves generating a
supporting substructure that is then clad to form the surface.
These tectonic systems constrain the design space of fabricate-
able geometry by limiting design topology to that which can
be derived from sub structures (such as wae grids or other
contoured supports) and cladding geometries (such as panels,
strips or shingles). More complex double curved topologies
can be fabricated by developing approaches that do not rely
on permanent formwork or substructure. For example, work by
Mark Fornes shows that complex, branching mesh topologies
can be assembled from thin aluminium strips by using the
inherent stiffness of the surface to support the form during
assembly (Fornes 2014). Ayers, Martin and Zwierzycki work
with the hand craft of Kagami Weaving illustrates how lattice
structures can be fabricated by working from a modication of
the edge topology of arbitrary mesh geometries (Ayres, Martin,
and Zwierzycki 2018). Extensive examples can be found in the
literature of fabricating simpler surface topologies from elasti-
cally bent timber lamellas, such as work by Djordje Stanojevic
(Quartara and Stanojevic 2019) or Oliver David Krieg (Meyboom,
Correa Zuluaga, and Krieg 2019). Each of these examples
requires exactly determining the geometry of parts to fabricate
a desired surface.
Simple Approaches to Fabricating Curved Surfaces in
Augmented Reality
Despite the recent proliferation of examples of augmented
reality fabrication in academic literature (Song, Koeck, and
Luo 2021), there are relatively few demonstrated examples
of methods for fabricating curved surfaces. Artillion Studio
utilized augmented reality to form a complex double curved
sculpture from hand-bent pieces of stainless-steel at bar
(“Inferno Redux/ Wind Stone” 2018). Kristof Crolla and Garvin
Goepel have demonstrated the use of augmented reality to form
double curved surfaces from interwoven strips of bent bamboo
(Goepel and Crolla 2020). Yang Song completed a project called
BloomShell as part of Soomeen Hahm’s research lab at the
Bartlett that explored the use of thermoplastic sheets to form
and assemble double curved surfaces from augmented reality
guides (Song 2020). In each of these examples, surfaces are
rst discretised into explicitly modelled parts which are used
to guide simple forming processes. Parts are then assembled
together to form larger curving surfaces.
Guided Fabrication in Augmented Reality
Many technical approaches exist to taking dimensional
measurements from physical objects (Peggs et al. 2009).
The comparison of metrology data from physical objects to
target dimensions dened by digital models can be used to
guide fabrication processes in augmented reality. By mapping
the distance between current and target vertex positions of
a design to a colour gradient and projecting these colours
directly on the workpiece, Skeels and Rehg demonstrated that
guided augmented reality could enable fabricators to sculpt
accurate recreations of digital models by hand (Skeels and
Rehg 2007). Yoshida et al demonstrated that similar real-time
projection-based methods could be used to assist in addi-
tive fabrication of building scale structures from chopsticks
(Yoshida et al. 2015). However, projection based methods rely
on the work surface being in the line of sight of the projector,
and limit the capacity to work with surfaces with large numbers
of undercuts or other complex topologies. Chun et al demon-
strated a method for carving arbitrary designs from clay in
augmented reality that utilized photogrammetry to create 3D
models of as-built objects and indicating differences to target
geometry by colourising mesh vertices. However, the authors
note that this process is time consuming and could be improved
with real-time methods for scanning the as-built object (Chun
et al. 2020). Object detection and tracking algorithms have also
been shown to assist fabricators with positioning parts in arbi-
trary orientations (T. Sandy and J. Buchli 2018), eliminating the
need to calibrate a 3D scan with the workpiece. The limitation
of object tracking methods is that the position of parts in the
digital design must be dened prior to fabrication, preventing
the possibility of working with subtractive or deposition-based
Paper Title Author last names, separated by commas42022
manufacturing processes, or assembly from ad-hoc distribution
of parts.
Simple Approaches to Fabricating Curved Surfaces in
Augmented Reality
We have previously attempted several approaches to fabricating
complex double curved surfaces without explicitly designing
parts and instead using only the surface model viewed in
augmented reality on the HoloLens. The HoloLens is an
augmented reality display produced by Microsoft that renders
virtual objects called “holograms” by projecting a stereoscopic
image of the object onto a transparent lens located a few
centimetres in front of the wearer's eyes. Because the light
representing a hologram originates from this 2D display and
not from the hologram’s true position in physical space, phys-
ical objects that are between the hologram and the display are
occluded by the additive light of the hologram and disappear
behind them. Thus, trying to use a hologram for fabrication
of surface models is challenging because it is very dicult to
correctly judge the precise depth of the surface of a hologram
without the ordinary reference points of knowing if something
(like a part, construction material or your own hand) is in front or
behind it.
We have previously attempted to work around this limitation by
constructing curved structures in layers. The surface model is
rst contoured or sliced into layers that are then displayed to the
user one at a time to reliably view the correct shape of the form
at any given height without creating large areas of holographic
surface that would occlude the physical structure being built.
However, this results in design detail being directly correlated
4 Aruco markers stenciled on the
surface of an acrylic sheet to
approximately track changes in
the geometry of the surface during
5 Simultaneous fabrication by two
fabricators. Colour indicates devi-
ation of physical material from the
target surface.
to the number of layers (as in additive manufacturing) and
introduces additional labor if small details or accurate curvature
are required. Furthermore, the representation of double curved
surfaces as planar layers prevents fabricators from observing
and understanding changes in the geometry and curvature of
the surface that would be useful to eciently plan fabrication
tasks and improve workmanship.
We have also explored simple approaches to digitizing existing
parts and structures during fabrication using ducial markers
directly embedded in construction materials (Figure 4). Fiducial
markers such as printed Aruco or QR codes provide a low reso-
lution and approximate indication of the geometry described as
planes within Grasshopper. The main limitation of this approach
is that to increase the resolution of the digitized geometry,
markers must decrease in size thereby reducing precision.
Digitizing geometry with printed ducial markers also doesn’t
generalize very well to double curved surfaces as markers
deform out of plane when xed to twisting strips and surface
curvature results in most markers appearing at oblique angles
to the tracking camera on the HoloLens, all of which result in
suboptimal tracking precision. The methods outlined in this
paper instead make use of the long-throw depth camera on the
HoloLens to provide fabricators with real time visual feedback
on the current position of construction material relative to the
desired geometry of the surface.
Mixed Reality Application
An augmented reality application was developed by the authors
in Unity for viewing models on the HoloLens 2. The unity
5TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
application includes a model loader for downloading GLTF
models hosted remotely on the cloud, utilities for locating
models in physical space and thereby enabling shared experi-
ence of virtual content by viewing the same model in the same
location on multiple HoloLens 2 headsets, rendering models
as surfaces or outlines and displaying depth stream data as a
coloured point cloud. The HoloLens 2 provides access to the
long throw and short throw depth stream data behind a C# API
available when Research Mode is enabled. Our method utilizes
long-throw depth data at a maximum resolution of 1024x1024
depth points updated at 5 frames per second (Bamji et al. 2018).
Rendering Point Clouds on the HoloLens
The long throw point cloud was clipped to 350x410 to approxi-
mately match the 52-degree eld of view provided by the display.
These points were then further ltered to remove background
features by culling points further than 150mm from the target
geometry. Due to the performance cost of exact mesh proximity
calculations, we developed a method of approximate mesh
proximity testing by raycasting from the depth camera location
through the geometry, followed by a second raycast from the
detected point position in the inverse direction of the mesh
normal (Figure 6).
This reduced the inaccuracy of distance measurements on
curved surfaces or those at oblique angles to the camera
position. The point cloud is rendered as a gradient through red,
green and blue, signifying in front of, equal to, and behind the
surface of the digital mesh model. The rendered opacity of each
point decreases with proximity to the surface, and a completely
transparent point indicates that the point is within 3 mm of the
digital mesh model.
Proof of Concept Surface Design
A proof-of-concept prototype was designed by lling a volume
of 900x600x1250 mm with 38 randomly distributed points that
were then isosurfaced to generate a uniformly curving mesh
with an arbitrarily complex topology. The design of the surface
aimed to balance estimated structural stiffness due to double
curvature with the size of strips required to match this curva-
ture, however there were otherwise very few intended design
constraints. The mesh of the surface was then offset to create
geometry for ray casting point clouds against, and the edges of
a low-resolution quad mesh approximation of the design model
were extracted to provide fabricators with an approximate guide
to the design intent during fabrication.
The prototype was fabricated by three people working simul-
taneously from the same digital model viewed in augmented
reality on the HoloLens 2. Each fabricator began to construct
a part of the model by locating 50x300mm strips of 300gsm
paper such that they approximated the geometry of the surface
indicated in augmented reality. Each strip would typically need
to bend and twist into arbitrary curves to match the geometry
of the surface. While the strip of paper was visible to the depth
camera on the HoloLens and within a distance threshold of
150mm to the mesh, any points that intersected the strip would
be displayed on the HoloLens. As the fabricator moved the strip
closer to the surface the colour of these points would change
to indicate a change in the proximity of the physical strip to the
surface (Figure 4).
In practice no strip perfectly matched the geometry of the
design model due to limitations of forming strips by hand, the
exibility of the material and the work time of the adhesive.
Instead fabricators would optimize for fabrication speed rather
than fabrication precision. Once a strip was within acceptable
tolerances of the surface it would be xed in place with minor
imprecision occasionally overcome by inserting subsequent
strips. The visual feedback on the precision of the strip was
therefore a means to understand the desired geometry of the
design model rather than a means to improve precision over
unguided processes. Acceptable tolerances varied over the
course of fabricating the prototype as it became necessary to
work with large variations due to deformation from self-weight.
Fabricators would typically test multiple strip orientations and
shapes using this visual feedback to balance several fabrica-
tion goals. These included maximizing covered surface area,
6 Diagram illustrating raycasting algorithm. Note the distance along the
second raycast to the mesh (green) is less than the distance along the
rst raycast (blue).
Paper Title Author last names, separated by commas62022
strip and surface curvature, connections to existing strips
for stiffness and accessibility for the purposes of applying
adhesive. Because of the inherent exibility of the material,
fabricators would attempt to avoid adding strips that could not
be supported by existing structure or that would add cantilevers
causing the structure to sag. The geometry of strips was not
dened a priori by the design model and therefore the patterns
produced by the strips were determined on the y by each
fabricator. Adding strips to the model following the heuristics
previously described required skill, and each fabricator devel-
oped a unique approach to inserting strips into the structure
with a corresponding and recognisable aesthetic (Figure 5).
Once a satisfactory geometry for a strip was found, one end
of the strip was pinned in place using hot glue. The remainder
of the strip was then formed into the planned geometry and
tacked in place with additional adhesive. This allowed some
workability of the strip and prevented accidentally xing material
in unplanned locations due to the very short work time of the
adhesive. Because the visual feedback ran as a background
process and was “always on”, any part of the as-built structure
could easily be compared to the design model for precision and
small changes could be made such as adding more strips to
reinforce slumping areas, or to work on a completely different
part of the model (Figure 5).
Because no parts were dened by the design model there was
also no predetermined and explicit sequence of assembly,
and fabrication could occur at any part of the design model
at any time. A sparse distribution of strips was produced that
7 Image of the point cloud scan and capture locations. Colours indicate deviation of points from the target digital model.
approximated the form of the prototype while holding its form
as much as possible under self-weight. This sparse distribution
was then reinforced with additional strips that could be added
without using an augmented reality guide.
The prototype was completed in 6 hours by the team of three
fabricators using approximately 1000 strips of 50x300 mm
300gsm paper (Figure 2). The average deviation of the fabri-
cated surface was 27 mm. This deviation was calculated by
recording 24 point clouds from 8 vantage points and performing
a closest point analysis for the approximately 200,000 points
(Figure 7). Because this deviation is distributed over large
surfaces rather than local to the geometry of individual strips,
we expect deviation to be due to deformation of low-curvature
areas of the surface under self-weight rather than any technical
limitations to the method described for guided fabrication.
This deviation could be reduced by increasing the stiffness of
construction material, increasing the curvature of the design or
avoiding cantilevering and unsupported geometry. However, we
expect that deviation and construction time could be reduced
further simply by increasing the skill of the fabrication team.
Our fabrication team had no prior experience constructing such
complex surfaces from paper, and different ad-hoc methods
for locating strips following the surface of the design model
were invented and rened while fabricating the prototype. These
included attempting to follow the edges of the quad mesh,
following estimations of minimum and maximum curvature,
following and reinforcing topological loops or simply optimizing
7TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
for simplicity of installation. Providing fabricators with some
additional guidance on optimal strip orientation; for instance, by
displaying principal curvature directions of the mesh or esti-
mating geodesic curves for a given starting vector of a strip;
would accelerate the acquisition of skills that would assist in
fabricating a maximally stiff representation of arbitrary curving
surfaces. However, this would also introduce additional design
complexity and necessitate an understanding of these computa-
tional tools and approaches.
We contribute a novel method for approximately fabricating
arbitrarily curving surfaces in augmented reality that does not
require an underlying substructure or rationalization to explicit
parts. Because the method does not require the assembly of
sub structures, 2D documentation or any form of digital design
rationalization we further contribute a fast, low cost, accessible
and unconstrained design to production workow compared
to additive manufacturing or other computer aided manufac-
turing processes such as assembly from CNC cut strips. Our
method affords an intuitive approach to fabricating surfaces
from exible and lightweight materials in the same fashion
as artists such as Henrique Oliviera, while also affording the
possibility of relative precision, analysis and iteration provided
by digital design workows. Mixed reality environments provide
an additional benet and contribution to these types of proj-
ects whereby bottlenecks of design expertise are eliminated by
simultaneous and unambiguous access to design intent. This
enables complex structures to be fabricated eciently in parallel
and suggests that this would enable even larger and more
complex structures to be completed simply by increasing the
size of the fabrication team.
To evidence that the method described in this paper is gener-
alizable to other fabrication approaches we have explored two
other small proof of concepts including additive fabrication
from river pebbles (Figure 10) and arbitrary part placement with
6 degrees of freedom (Figure 11). In both cases providing fabri-
cators with visual feedback on the delta between the location of
physical material and the target geometry improves the ability
of fabricators to complete tasks. We speculate that reducing
the complexity of these fabrication tasks in augmented reality
will impact the discourse on digital fabrication by broadening
the design space of fabricatable structures in augmented reality
without introducing the need to rationalize surfaces to parts or
substructures. This in turn will have follow on implications by
improving the feasibility of constructing geometry from arbitrary,
heterogenous and non-standard materials such as those in the
Stik Pavilion (Yoshida et al. 2015), as the digital model doesn’t
need to explicitly model material behaviours and constraints for
this material to be guided into a target form.
Our guidance system enabled the fabrication team to approx-
imately complete the prototype in a very short span of
fabrication time compared to alternative methods. The speed
of the method is achieved by enabling a theoretically unlimited
number of fabricators to work on construction simultaneously.
As evidence of the ecacy of the guidance system, the fabrica-
tion team was not able to construct any part of the prototype by
working with the hologram of the design model alone. This was
attributed to the diculty of matching strip geometry to surface
geometry in augmented reality. Fabricating the prototype would
therefore ordinarily require explicitly dening parts making up
the surface and fabricating each part from rigid material to
9 A prototype fabricated with our method from long strips of timber veneer.8 A simple double-curved surface model for fabrication from timber veneer.
Paper Title Author last names, separated by commas82022
within tolerances of the digital model as has been demonstrated
by other approaches to realizing complex surface models in
augmented reality.
While completing the prototype we learned that forming double
curved surfaces from exible strips of paper requires craft skill
that needed to be acquired from practice. This resulted in initially
poor workmanship and precision that improved gradually over
time. Poor decisions in forming parts of the structure introduced
areas of low curvature that then deformed under self-weight and
could not be reinforced with additional material. The primary
failure in our method as demonstrated is in the appropriateness
of the design language of the proof-of-concept model. Large
areas of the model were without sucient double curvature or
topological support to hold their form when constructed from
paper, introducing deviation from the digital model that had little
to do with the precision of the guidance system or the accuracy
of individual strips. The marching cubes algorithm used to
create the mesh of the proof-of-concept model also contained
many details too small to form with the 50 mm wide strips,
leading to further deviations that could easily have been avoided.
This proof-of-concept project represents the very rst exper-
iment with real-time guided fabrication of surface models in
augmented reality without discretization to explicitly dened
parts. A fundamental cause of error and limitation of our
prototype was deformation of the structure under self-weight,
and in future work we aim to address this limitation by working
with more rigid materials and design models. However, we
also recognise the opportunities afforded by light weight and
readily available materials such as paper, and the possibility of
exploring ad-hoc design processes by colouring or patterning
the paper strips as an additional expressive device during
We have conducted additional experiments forming curved
surfaces from strips of timber veneer (Figure 8, 9). Our method
should generalize to pliable construction scale materials such
as bent timber or steel, as well as to sculpting materials such
as clay and plaster. Guided feedback in augmented reality
may also address limitations observed by other fabrication
approaches within the discourse. For instance, bricklaying with
simple augmented reality guides describing courses of bricks
in a design requires maintaining a top-down view to accurately
position each brick. This in turn necessitates frequently moving
scaffolding to construct larger walls, and this is unfamiliar to
bricklayers used to working up to and beyond eye height. Guided
feedback should enable the accurate placement of bricks in
arbitrary orientation and eliminate the need for top-down views.
Future work intends to explore these opportunities.
While we have aimed to implement and demonstrate a
generalisable and accessible method for guided fabrication
in augmented reality, future work will rene this approach to
address other observed limitations. By performing analysis
on design models, it may be possible to reduce the risk of
poor structural performance by displaying structural analysis,
geodesic curves or principal stress directions in real time based
11 Guidance system for positioning a part with 6 degrees of freedom.10 Guidance system for assembly of curved surfaces from pebbles.
9TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
on the proposed position of a strip of material by a fabricator.
Similarly, providing designers with access to the HoloLens 2
depth stream within parametric modelling software such as
Grasshopper would enable the design of customisable guidance
algorithms. One could imagine tting parts to point clouds,
displaying correctional information with arrows, or colourising
mesh surfaces with various proximity information, and these
will be explored in future work and collaborations.
Ayres, Phil, Alison Martin, and Mateusz Zwierzycki. 2018. Beyond the
Basket Case: A Principled Approach to the Modelling of Kagome Weave
Patterns for the Fabrication of Interlaced Lattice Structures Using
Straight Strips. Advances in Architectural Geometry 2018, 72–91. Vienna:
Chalmers University of Technology
Bamji, Cyrus S., Swati Mehta, Barry Thompson, Tamer Elkhatib, Stefan
Wurster, Onur Akkaya, Andrew Payne, et al. 2018. “IMpixel 65nm
BSI 320MHz Demodulated TOF Image Sensor with 3μm Global
Shutter Pixels and Analog Binning.” In 2018 IEEE International Solid
- State Circuits Conference - (ISSCC), 94–96.
Chun, Jacky, Adabelle Long, Wing Sze, Chung Hei, Garvin Goepel, and
Kristof Crolla. 2020. “Augmenting Craft with Mixed Reality: A Case
Study Project of AR-Driven Analogue Clay Modelling.” In ACADIA 2020:
Distributed Proximities [Proceedings of the 40th Annual Conference of the
Association of Computer Aided Design in Architecture (ACADIA).
Fornes, Marc. 2014. “Double Agent White.” In ACADIA 14: Design Agency
[Projects of the 34th Annual Conference of the Association for Computer
Aided Design in Architecture (ACADIA), 157-160. Los Angeles, USA.
Goepel, Garvin, and Kristof Crolla. 2020. “AUGMENTED REALITY-BASED
COLLABORATION - ARgan, a Bamboo Art Installation Case Study.” In
RE: Anthropocene, Design in the Age of Humans - Proceedings of the 25th
CAADRIA Conference, 313–22. University, Bangkok, Thailand.
“Inferno Redux/ Wind Stone.” 2018. Inferno Redux/ Wind Stone. 2018.
Jahn, Gwyllim, Cameron Newnham, Nick Berg, and Matthew Beanland.
2018. “Making in Mixed Reality.” In Proceedings of the 38th Annual
Conference of the Association for Computer Aided Design in Architecture
(ACADIA), 88–97. Mexico City, Mexico.
Jahn, Gwyllim, Cameron Newnham, Nick Berg, Melissa Iraheta, and
Jackson Wells. 2019. “Holographic Construction.” In Impact: Design With
All The Senses, 314–24. Berlin.
Jahn, Gwyllim, Cameron Newnham, and Nick van den Berg. 2022.
"Augmented Reality For Construction From Steam Bent Timber". In
Proceedings of the 27th International Conference of the Association for
Computer-Aided Architectural Design Research in Asia. Sydney, Australia.
Jahn, Gwyllim, Andrew Wit, and James Pazzi. 2019. “[BENT] Holographic
Handcraft in Large-Scale Steam-Bent Timber Structures.” In Proceedings
of the 39th Annual Conference of the Association for Computer Aided
Design in Architecture (ACADIA), 438–47. The University of Texas at
Austin School of Architecture, Austin, Texas.
Lok, Leslie. 2022. “Timber De-Standardized 2.0 : Mixed Reality
Visualizations And User Interface For Processing Irregular Timber.” In
Proceedings of the 27th International Conference of the Association for
Computer-Aided Architectural Design Research in Asia. Sydney, Australia.
Meyboom, Annalisa, David Correa Zuluaga, and Oliver Krieg. 2019.
Stressed Skin Wood Surface Structures: Potential Applications in
Architecture. In ACADIA 19:UBIQUITY AND AUTONOMY [Proceedings of the
39th Annual Conference of the Association for Computer Aided Design in
Architecture (ACADIA), 470-477. The University of Texas at Austin School
of Architecture, Austin, Texas.
Mitterberger, Daniela, Kathrin Dörer, Timothy Sandy, Foteini
Salveridou, Marco Hutter, Fabio Gramazio, and Matthias Kohler.
2020. “Augmented Bricklaying: Human–Machine Interaction for in
Situ Assembly of Complex Brickwork Using Object-Aware Augmented
Reality. Construction Robotics 4 (December).
Peggs, G.N., Paul Maropoulos, Ben Hughes, Alistair Forbes,
Stuart Robson, Marek Ziebart, and Bala Muralikrishnan. 2009.
“Recent Developments in Large-Scale Dimensional Metrology.
Proceedings of the Institution of Mechanical Engineers - Part B:
Journal of Engineering Manufacture 223 (June): 571–95. https://doi.
Quartara, Quartara, and Djordje Stanojevic. 2019. Computational
and Manufacturing Strategies. Springer.
Skeels, Christopher, and James M Rehg. 2007. “ShapeShift: A Projector-
Guided Sculpture System.” Proc. UIST 7: 2.
Song, Yang. 2020. “BloomShell: Augmented Reality for the Assembly and
Real-Time Modication of Complex Curved Structure.” In Anthropologic:
Architecture and Fabrication in the Cognitive Age - Proceedings of the 38th
ECAADe Conference, 345–54. Berlin.
Song, Yang, Richard Koeck, and Shan Luo. 2021. “Review and Analysis
of Augmented Reality (AR) Literature for Digital Fabrication in
Architecture.Automation in Construction 128 (August): 103762. https://
Paper Title Author last names, separated by commas10 2022
T. Sandy, and J. Buchli. 2018. “Object-Based Visual-Inertial Tracking for
Additive Fabrication. IEEE Robotics and Automation Letters PP (99): 1–1.
Yoshida, Hironori, Takeo Igarashi, Yusuke Obuchi, Yosuke Takami, Jun
Sato, Mika Araki, Masaaki Miki, Kosuke Nagata, Kazuhide Sakai, and
Syunsuke Igarashi. 2015. “Architecture-Scale Human-Assisted Additive
Manufacturing.” ACM Transactions on Graphics 34 (4): 88:1-88:8. https://
All drawings and images by the authors.
Gwyllim Jahn Gwyllim Jahn is a co-founder of Fologram and a Lecturer
in Architecture at RMIT University in Melbourne. His work focuses on
designing for mixed reality fabrication, most notably in the design and
construction of the 2019 Tallinn Architecture Biennale Pavilion. Gwyllim’s
research has been published in leading computational design confer-
ences and journals including IJAC, ACADIA and RobArch and he has
given talks, presentations and workshops at international institutions
including MIT, Stuttgart ICD, UCL, SciArc, Tongji and Tsinghua University.
Cameron Newnham is the co-founder and CTO of Fologram where he
leads the technical development of mixed reality software for the design
and construction industry. His experience lies in the creation of novel
tools for designing and fabricating complex geometric systems, ranging
from code libraries to mixed reality interfaces and extending to machine
design and robotic fabrication. Cameron has experience as a computa-
tional designer in internationally renowned architectural practices, and
academic experience as an Associate Lecturer – Industry Fellow at RMIT
University. Cameron has led numerous international design and build
workshops in Shanghai, New York, Paris, Boston, Sydney, and Melbourne.
Nick van den Berg is the co-founder and CEO of Fologram, a design
research practice and technology startup building a platform for
designing and making in mixed reality. Fologram’s platform is being used
by world leading architecture practices, product design houses, manu-
facturers and design schools internationally. Nick is especially interested
in building solutions that are utilised by a large user base around the
world and has assisted with workshops focusing on utilising augmented
and mixed reality as a design and fabrication tool at DMS, CAADRIA,
Cooper Union, McNeel Europe, UDK & TU Berlin.
11TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
... Other research projects such as Augmented Feedback, Timber De-Standardized 1.0, and Augmented Vision use various methods of AruCo markers for tracking, physics simulation, and real-time scanning to create an active responsive environment between digital and physical objects (11)(12)(13). In Augmented Feedback, AruCo makers were placed at nodal intersections of a bending-active bamboo gridshell structure (11). ...
... Timber De-Standardized 1.0 utilized a physics simulation for fabricators to visualize and virtually "drop" irregular scanned meshes of logs till they found their resting point, which allowed for a precise alignment with its associated physical log (12). Finally, Augmented Vision uses the Hololens 2 to track and scan the user's environment then display such information to inform the progress of constructing a minimal surface with strips of paper and/or bark (13). These projects have demonstrated the capabilities of feedback-based MR using additional systems such as AruCo markers, scanned meshes, and simulation. ...
Conference Paper
Full-text available
Mixed Reality (MR) platforms enable users to interact with three-dimensional holographic instructions during the assembly and fabrication of highly custom and parametric architectural constructions without the necessity of two-dimensional drawings. Previous MR fabrication projects have primarily relied on digital menus and custom buttons as the interface for user interaction with the MR environment. Despite this approach being widely adopted, it is limited in its ability to allow for direct human interaction with physical objects to modify fabrication instructions within the MR environment. This research integrates user interactions with physical objects through real-time gesture recognition as input to modify, update or generate new digital information enabling reciprocal stimuli between the physical and the virtual environment. Consequently, the digital environment is generative of the user's provided interaction with physical objects to allow seamless feedback in the fabrication process. This research investigates gesture recognition for feedback-based MR workflows for robotic fabrication, human assembly, and quality control in the construction of the UnLog Tower.
... Larsson and Yoshida (209) who propose an audio-visual tracking system based on fiducial markers able to guide users through the machining of moderately sized irregular tree branches with a 6-axis router. Jahn et al. Jahn et al. (2022) showcase how introducing a feedback loop logic based on depth sensing streams can successfully assist a user during a sculpting operation. ...
Full-text available
Augmented reality in additive fabrication is predominantly applied to the manufacturing of structures from regularly shaped materials. Our work however embraces natural heterogeneity, by focusing on the assembly of irregularly shaped elements such as mineral scraps. We introduce a computer-aided framework with a 3D stacking engine and an augmented reality interface capable of guiding users through the balanced positioning of discrete and highly nonuniform objects according to an on-the-fly computed model and without prior trimming of the building unit. The pipeline is tested by the construction of two dry-stone structures (i.e., lacking any adhesives), and its accuracy is validated with a comparative study between the point cloud of the as-built digitized artifacts and the generated execution model, which shows an average 2.9 ± 1.8 cm error between the stones of the two models. We finally show how the proposed framework could be improved both in terms of software and hardware. In the interests of reproducibility, all the methods are shared as open source with the community. Graphical abstract
Conference Paper
Full-text available
Digital models viewed in augmented reality can serve as guides to form and assemble parts during construction and reduce the need to build temporary formwork or sub structures. However, static digital models are often inadequate for describing the behaviour of material that is dynamically formed over time, leading to breakages and difficulty following augmented reality guides during assembly. To address this issue, we propose a method for fast and approximate simulation of material behaviour using a goal-based physics solver, enabling the design and fabrication of steam bent timber parts using an adaptable system of sparse formwork. Through the design and construction of a pavilion from steam bent timber we demonstrate that approximate simulation of material behaviour is adequate for wide tolerance construction by hand and eye in augmented reality, avoiding part breakages and accumulative error.
Conference Paper
Full-text available
This paper discusses novel methods for and advantages of integrating augmented reality (AR) and photogrammetry in hand clay-sculpting workflows. These techniques permit nontrained users to achieve higher precision during the sculpting process by holographically overlaying instructions from digital 3D source geometry on top of the sculpting material. By employing alternative notational systems in design implementation methods, the research positions itself in a postdigital context aimed at humanizing digital technologies. Throughout history, devices have been developed to increase production, such as Henry Dexter’s 1842 “Apparatus for Sculptors” for marble sculpting. Extrapolating from this, the workflow presented in this paper uses AR to overlay extracted information from 3D models directly onto the sculptor’s field of vision. This information can then become an AR-driven guidance system that assists the sculptor. Using the Microsoft HoloLens, holographic instructions are introduced in the production sequence, connecting the analog sculpture fabrication directly with a digital environment, thus augmenting the craftspeople’s agency. A series of AR-aided sculpting methods were developed and tested in a demonstrator case study project that created a small-scale clay copy of Henry Moore’s Sheep Piece (1971–1972). This paper demonstrates how user-friendly software and hardware tools have lowered the threshold for end users to develop new methods that straightforwardly facilitate and improve their crafts’ effectiveness and agency. This shows that the fusion of computational design technology and AR visualization technology can innovate a specific craft’s design and production workflow, opening the door for further application developments in more architecture-specific fabrication contexts.
Conference Paper
Full-text available
Augmented Reality (AR) as a new technical tool has developed rapidly in the last few years and has now the potential of bridging the gap between holographic drawings and the real world. This paper addresses whether AR can guide unskilled labour on complex structure assembly and fabrication process. It contains three experiments developed with AR. The research aims to prove that with intuitive holographic instructions, AR helps to reduce the time spent in comparing 2D drawings to the real site during the assembly process, and therefore offers possibilities to improve the construction efficiency significantly. The research also paves the way for shell structures, considering the latest technology such as AR and AI, and gives emphasis on the communication between computer and human during the fabrication process through the physical model. It is an exploration of how people might change their mind or decisions can be changed in a real-time manner harmoniously using AI through AR.
Full-text available
Augmented bricklaying explores the manual construction of intricate brickwork through visual augmentation, and applies and validates the concept in a real-scale building project—a fair-faced brickwork facade for a winery in Greece. As shown in previous research, robotic systems have proven to be very suitable to achieve various differentiated brickwork designs with high efficiency but show certain limitations, for example, in regard to spatial freedom or the usage of mortar on site. Hence, this research aims to show that through the use of a craft-specific augmented reality system, the same geometric complexity and precision seen in robotic fabrication can be achieved with an augmented manual process. Towards this aim, a custom-built augmented reality system for in situ construction was established. This process allows bricklayers to not depend on physical templates, and it enables enhanced spatial freedom, preserving and capitalizing on the bricklayer’s craft of mortar handling. In extension to conventional holographic representations seen in current augmented reality fabrication processes that have limited context-awareness and insufficient geometric feedback capabilities, this system is based on an object-based visual-inertial tracking method to achieve dynamic optical guidance for bricklayers with real-time tracking and highly precise 3D registration features in on-site conditions. By integrating findings from the field of human-computer interfaces and human-machine communication, this research establishes, explores, and validates a human-computer interactive fabrication system, in which explicit machine operations and implicit craftsmanship knowledge are combined. In addition to the overall concept, the method of implementation, and the description of the project application, this paper also quantifies process parameters of the applied augmented reality assembly method concerning building accuracy and assembly speed. In the outlook, this paper aims to outline future directions and potential application areas of object-aware augmented reality systems and their implications for architecture and digital fabrication.
Conference Paper
Full-text available
ARgan is a geometrically complex bamboo sculpture that relied on Mixed Reality (MR) for its joint creation by multiple sculptors and used latest Augmented Reality (AR) technology to guide manual fabrication actions. It was built at the Chinese University of Hong Kong in the fall of 2019 by thirty participants of a design-and-build workshop on the integration of AR in construction. As part of its construction workflow, holographic setups were created on multiple devices, including a series of Microsoft HoloLenses and several handheld Smartphones, all linked simultaneously to a single digital base model to interactively guide the manufacturing process. This paper critically evaluates the experience of extending recent AR and MR tool developments towards applications that centre on creative collaborative production. Using ARgan as a demonstrator project, its developed workflow is assessed on its ability to transform a geometrically complex digitally drafted design to its final physically built form, highlighting the necessary strategic integration of variability as an opportunity to relax notions on design precision and exact control. The paper concludes with a plea for digital technology’s ability to stimulate dialogue and collaboration in creative production and augment craftsmanship, thus providing greater agency and more diverse design output.
Conference Paper
Full-text available
Innovation in parametric design and robotic fabrication is in reciprocal relationship with the investigation of new structural types that facilitated by this technology. The stressed skin structure has historically been used to create lightweight curved structures, mainly in engineering applications such as naval vessels, aircraft, and space shuttles. Stressed skin structures were first referred to by Fairbairn in 1849. In England, the first use of the structure was in the Mosquito night bomber of World War II. In the United States, stressed skin structures were used at the same time, when the Wright Patterson Air Force Base designed and fabricated the Vultee BT-15 fuselage using fiberglass-reinforced polyester as the face material and both glass-fabric honeycomb and balsa wood core. With the renewed interest in wood as a structural building material, due to its sustainable characteristics, new potentials for the use of stressed skin structures made from wood on building scales are emerging. The authors present a material informed system that is characterized by its adaptability to freeform curvature on exterior surfaces. A stressed skin system can employ thinner materials that can be bent in their elastic bending range and then fixed into place, leading to the ability to be architecturally malleable, structurally highly efficient, as well as easily buildable. The interstitial space can also be used for services. Advanced digital fabrication and robotic manufacturing methods further enhance this capability by enabling precisely fabricated tolerances and embedded assembly instructions; these are essential to fabricate complex, multi-component forms. Through a prototypical installation, the authors demonstrate and discuss the technology of the stressed skin structure in wood considering current digital design and fabrication technologies.
Conference Paper
Full-text available
We present a method for generating holographic construction information from parametric models. Holographic models replace 2D drawings and templates with unambiguous, contextual, shared and interactive design information. We show that our method enabled a team of expert bricklayers to complete a section of wall in a fraction of expected construction time and within typical tolerances, measured through comparative analysis of digital models to 3D point cloud scans of as built conditions.
Conference Paper
Full-text available
Over the past two decades, advances in computation, digital fabrication, and robotics have opened up new avenues for the design and production of complex forms, emergent processes, as well as new levels of efficiency. Many of these methods, however, tend to focus on a specific tool, such as the industrial robotic arm. Due to their initial costs and space/power/safety requirements, difficulties associated in creating automated work-flows and custom tooling, as well as the need for reliable/repeatable procedures, these tools are often out of reach for the average designer or design institution. Additionally, these tools are typically treated as methods of production rather than collaborators, leaving outcomes that can feel void of craft, with the appearance of a typical CNC-machined object. Rather than focusing on a specific production tool for manufacturing, this paper investigates a novel method for holographic handcraft-based production. This holographic augmentation-of simple and easily attainable analog tool sets-allows for the creation of extremely complex forms with high levels of precision in extremely short time frames. Through the lens of the recently completed steam-bent timber installation [BENT] produced at the Tyler School of Art, this paper discusses how Microsoft HoloLens in conjunction with the Fologram software plug-in can be integrated into the entirety of design and production processes as a means of producing a new typology of digital craft.
The use of Augmented Reality (AR) technologies has increased recently, due to the equipment update and the mature technology. For architectural design, especially in digital fabrication projects, more designers begin to integrate AR methods to achieve the visualization in the process. To help unskilled labors for holographic on-site previewing and instruction training, experimental and practice-based studies in AR for the architectural digital fabrication have emerged in recent years. Now, it is a great opportunity to discuss the topic of AR in architectural digital fabrication. By presenting a statistical review of AR technology in architecture projects, this literature review aims to review ongoing research and provide pathways for further research in architectural digital fabrication. This review article is based on information found in journal publications and conference papers in the fields of architecture, engineering, robotics, and digital fabrication, published to date (from 2010 to 2020). The review narrows the literature within these papers by filtering 84 articles through the keyword “Augmented Reality”, “Digital Fabrication” and “Assembly”. The selected articles can be categorized based on the most use of AR function in architectural digital fabrication into an order of the following three classifications with the most significant growth in the last years: (A) AR 3D holographic instruction, (B) AR data sharing, (C) AR for Human-Computer interaction. The information collected from these articles within their classifications is meant to give insight into the current state-of-the-art of AR in the architectural digital fabrication area, as well as to summarize how the topic has matured and developed over time in the research and industry literature. This article has not only analyzed the existing literature but also highlighted new emerging fields in AR research and the future trends of AR function in architectural digital fabrication.