Content uploaded by Gwyllim Jahn
Author content
All content in this area was uploaded by Gwyllim Jahn on Nov 28, 2022
Content may be subject to copyright.
Depth Camera Feedback for Guided
Fabrication in Augmented Reality
1 A proof of concept prototype
assembled from free-formed
paper strips using depth camera
feedback in augmented reality.
Gwyllim Jahn
Fologram
Cameron Newnham
Fologram
Nick van den Berg
Fologram
ABSTRACT
Augmented reality environments have been demonstrated to assist with architectural fabrication
tasks by displaying construction information at full scale and in context. However, this informa-
tion typically needs to be sparse in order to prevent virtual models occluding a fabricators view
of the physical environment, and this limits the application of augmented reality to tasks such as
surface forming. To address this issue, we propose a method for guided fabrication in augmented
reality using real time comparisons between depth scans of as built conditions and target condi-
tions dened by design models. Through the design and fabrication of a small proof of concept
prototype from paper strips, we demonstrate that guided fabrication is adequate for high speed,
approximate and ad-hoc fabrication of complex surface geometries without the need for exten-
sive rationalization for fabrication constraints or explicit documentation of parts. We further show
how this method generalizes to other processes such as additive fabrication or part placement
and speculate on the implications of accessible real time depth data from the HoloLens within
Grasshopper.
22022
ACADIA
INTRODUCTION
Fabrication in augmented reality has proven to be an effective
and expedient method for forming and assembling complex
structures from steel (Jahn et al. 2018), bricks (Mitterberger et
al. 2020; Jahn et al. 2019) and steam bent timber (Jahn, Wit, and
Pazzi 2019). Recent work has also demonstrated augmented
reality applications working with exible materials such as bent
bamboo (Goepel and Crolla 2020) or non-uniform and unpro-
cessed timber (Lok 2022). However there remain few examples
of augmented reality fabrication approaches to forming
complex surface topologies due to the challenge of correctly
judging surface depth with current generation augmented reality
displays. This is a limitation on the design space afforded by
augmented reality fabrication, as surface forming is an ecient
way to describe geometry typically generated by CAD modeling
software. While there exists several computer aided manufac-
turing approaches to materializing surface geometry, each of
these approaches is not without limitations.
Subtractive forming complex surface models introduces the
challenge of avoiding undercuts, while additive manufacturing
is notoriously slow and introduces geometric constraints such
as draft angles and requirements for inll and support material.
Surfaces can be rationalized into smaller parts such as panels,
shingles, or strips, but this rationalization introduces design,
logistics and assembly complexity. By comparison, ad-hoc fabri-
cation of complex surface geometries has been demonstrated
to be possible by artists like Henrique Oliviera or Lucy Irvine,
though this negates the advantages of digital design to produc-
tion workows. This work is motivated by the desire to capitalize
on the eciency and formal possibilities of ad-hoc fabrication of
complex surface geometries while also facilitating the approxi-
mate adherence to digital design models. We are also motivated
by the desire to overcome the current limitations of augmented
reality approaches to surface fabrication by demonstrating a
generalizable approach to guiding fabricators using real time
comparisons between point cloud scans of as-built conditions
and target conditions dened by digital design models.
This research demonstrates a method for fast and approximate
forming of surfaces with arbitrarily complex topology and
geometry by hand in augmented reality. Within our approach,
the only documentation required is the design model of the
target surface, thereby simplifying or eliminating requirements
for rationalizing surfaces for fabrication or discretizing design
models to parts and joints (Figure 3). We aim to improve
the feasibility of fabricating complex surface geometry in
augmented reality compared to existing approaches, measured
by improvements in fabrication time and formal complexity, and
provide evidence of this method in a proof-of-concept prototype
(Figure 1) completed over the course of a single day by a team
of 3 inexperienced fabricators. The proof-of-concept project is
2 A detail view of the paper
prototype.
3 Hologram of the design model at
1:1. Mesh triangles are rendered
as occlusion geometry with lines
representing the mesh quads on
top.
3TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
2
3
used to explore how real time feedback can assist fabricators
in forming complex surface structures from exible, lightweight
materials by improving legibility of digital models viewed in
augmented reality and enabling the improvisation of ad-hoc
strategies for their materialization.
BACKGROUND
Formwork-free Approaches to Fabricating Surfaces from
Strips
Fabrication of curved surfaces typically involves generating a
supporting substructure that is then clad to form the surface.
These tectonic systems constrain the design space of fabricate-
able geometry by limiting design topology to that which can
be derived from sub structures (such as wae grids or other
contoured supports) and cladding geometries (such as panels,
strips or shingles). More complex double curved topologies
can be fabricated by developing approaches that do not rely
on permanent formwork or substructure. For example, work by
Mark Fornes shows that complex, branching mesh topologies
can be assembled from thin aluminium strips by using the
inherent stiffness of the surface to support the form during
assembly (Fornes 2014). Ayers, Martin and Zwierzycki work
with the hand craft of Kagami Weaving illustrates how lattice
structures can be fabricated by working from a modication of
the edge topology of arbitrary mesh geometries (Ayres, Martin,
and Zwierzycki 2018). Extensive examples can be found in the
literature of fabricating simpler surface topologies from elasti-
cally bent timber lamellas, such as work by Djordje Stanojevic
(Quartara and Stanojevic 2019) or Oliver David Krieg (Meyboom,
Correa Zuluaga, and Krieg 2019). Each of these examples
requires exactly determining the geometry of parts to fabricate
a desired surface.
Simple Approaches to Fabricating Curved Surfaces in
Augmented Reality
Despite the recent proliferation of examples of augmented
reality fabrication in academic literature (Song, Koeck, and
Luo 2021), there are relatively few demonstrated examples
of methods for fabricating curved surfaces. Artillion Studio
utilized augmented reality to form a complex double curved
sculpture from hand-bent pieces of stainless-steel at bar
(“Inferno Redux/ Wind Stone” 2018). Kristof Crolla and Garvin
Goepel have demonstrated the use of augmented reality to form
double curved surfaces from interwoven strips of bent bamboo
(Goepel and Crolla 2020). Yang Song completed a project called
BloomShell as part of Soomeen Hahm’s research lab at the
Bartlett that explored the use of thermoplastic sheets to form
and assemble double curved surfaces from augmented reality
guides (Song 2020). In each of these examples, surfaces are
rst discretised into explicitly modelled parts which are used
to guide simple forming processes. Parts are then assembled
together to form larger curving surfaces.
Guided Fabrication in Augmented Reality
Many technical approaches exist to taking dimensional
measurements from physical objects (Peggs et al. 2009).
The comparison of metrology data from physical objects to
target dimensions dened by digital models can be used to
guide fabrication processes in augmented reality. By mapping
the distance between current and target vertex positions of
a design to a colour gradient and projecting these colours
directly on the workpiece, Skeels and Rehg demonstrated that
guided augmented reality could enable fabricators to sculpt
accurate recreations of digital models by hand (Skeels and
Rehg 2007). Yoshida et al demonstrated that similar real-time
projection-based methods could be used to assist in addi-
tive fabrication of building scale structures from chopsticks
(Yoshida et al. 2015). However, projection based methods rely
on the work surface being in the line of sight of the projector,
and limit the capacity to work with surfaces with large numbers
of undercuts or other complex topologies. Chun et al demon-
strated a method for carving arbitrary designs from clay in
augmented reality that utilized photogrammetry to create 3D
models of as-built objects and indicating differences to target
geometry by colourising mesh vertices. However, the authors
note that this process is time consuming and could be improved
with real-time methods for scanning the as-built object (Chun
et al. 2020). Object detection and tracking algorithms have also
been shown to assist fabricators with positioning parts in arbi-
trary orientations (T. Sandy and J. Buchli 2018), eliminating the
need to calibrate a 3D scan with the workpiece. The limitation
of object tracking methods is that the position of parts in the
digital design must be dened prior to fabrication, preventing
the possibility of working with subtractive or deposition-based
Paper Title Author last names, separated by commas42022
ACADIA
4
manufacturing processes, or assembly from ad-hoc distribution
of parts.
Simple Approaches to Fabricating Curved Surfaces in
Augmented Reality
We have previously attempted several approaches to fabricating
complex double curved surfaces without explicitly designing
parts and instead using only the surface model viewed in
augmented reality on the HoloLens. The HoloLens is an
augmented reality display produced by Microsoft that renders
virtual objects called “holograms” by projecting a stereoscopic
image of the object onto a transparent lens located a few
centimetres in front of the wearer's eyes. Because the light
representing a hologram originates from this 2D display and
not from the hologram’s true position in physical space, phys-
ical objects that are between the hologram and the display are
occluded by the additive light of the hologram and disappear
behind them. Thus, trying to use a hologram for fabrication
of surface models is challenging because it is very dicult to
correctly judge the precise depth of the surface of a hologram
without the ordinary reference points of knowing if something
(like a part, construction material or your own hand) is in front or
behind it.
We have previously attempted to work around this limitation by
constructing curved structures in layers. The surface model is
rst contoured or sliced into layers that are then displayed to the
user one at a time to reliably view the correct shape of the form
at any given height without creating large areas of holographic
surface that would occlude the physical structure being built.
However, this results in design detail being directly correlated
4 Aruco markers stenciled on the
surface of an acrylic sheet to
approximately track changes in
the geometry of the surface during
fabrication.
5 Simultaneous fabrication by two
fabricators. Colour indicates devi-
ation of physical material from the
target surface.
to the number of layers (as in additive manufacturing) and
introduces additional labor if small details or accurate curvature
are required. Furthermore, the representation of double curved
surfaces as planar layers prevents fabricators from observing
and understanding changes in the geometry and curvature of
the surface that would be useful to eciently plan fabrication
tasks and improve workmanship.
We have also explored simple approaches to digitizing existing
parts and structures during fabrication using ducial markers
directly embedded in construction materials (Figure 4). Fiducial
markers such as printed Aruco or QR codes provide a low reso-
lution and approximate indication of the geometry described as
planes within Grasshopper. The main limitation of this approach
is that to increase the resolution of the digitized geometry,
markers must decrease in size thereby reducing precision.
Digitizing geometry with printed ducial markers also doesn’t
generalize very well to double curved surfaces as markers
deform out of plane when xed to twisting strips and surface
curvature results in most markers appearing at oblique angles
to the tracking camera on the HoloLens, all of which result in
suboptimal tracking precision. The methods outlined in this
paper instead make use of the long-throw depth camera on the
HoloLens to provide fabricators with real time visual feedback
on the current position of construction material relative to the
desired geometry of the surface.
METHODS
Mixed Reality Application
An augmented reality application was developed by the authors
in Unity for viewing models on the HoloLens 2. The unity
5TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
5
application includes a model loader for downloading GLTF
models hosted remotely on the cloud, utilities for locating
models in physical space and thereby enabling shared experi-
ence of virtual content by viewing the same model in the same
location on multiple HoloLens 2 headsets, rendering models
as surfaces or outlines and displaying depth stream data as a
coloured point cloud. The HoloLens 2 provides access to the
long throw and short throw depth stream data behind a C# API
available when Research Mode is enabled. Our method utilizes
long-throw depth data at a maximum resolution of 1024x1024
depth points updated at 5 frames per second (Bamji et al. 2018).
Rendering Point Clouds on the HoloLens
The long throw point cloud was clipped to 350x410 to approxi-
mately match the 52-degree eld of view provided by the display.
These points were then further ltered to remove background
features by culling points further than 150mm from the target
geometry. Due to the performance cost of exact mesh proximity
calculations, we developed a method of approximate mesh
proximity testing by raycasting from the depth camera location
through the geometry, followed by a second raycast from the
detected point position in the inverse direction of the mesh
normal (Figure 6).
This reduced the inaccuracy of distance measurements on
curved surfaces or those at oblique angles to the camera
position. The point cloud is rendered as a gradient through red,
green and blue, signifying in front of, equal to, and behind the
surface of the digital mesh model. The rendered opacity of each
point decreases with proximity to the surface, and a completely
transparent point indicates that the point is within 3 mm of the
digital mesh model.
Proof of Concept Surface Design
A proof-of-concept prototype was designed by lling a volume
of 900x600x1250 mm with 38 randomly distributed points that
were then isosurfaced to generate a uniformly curving mesh
with an arbitrarily complex topology. The design of the surface
aimed to balance estimated structural stiffness due to double
curvature with the size of strips required to match this curva-
ture, however there were otherwise very few intended design
constraints. The mesh of the surface was then offset to create
geometry for ray casting point clouds against, and the edges of
a low-resolution quad mesh approximation of the design model
were extracted to provide fabricators with an approximate guide
to the design intent during fabrication.
Fabrication
The prototype was fabricated by three people working simul-
taneously from the same digital model viewed in augmented
reality on the HoloLens 2. Each fabricator began to construct
a part of the model by locating 50x300mm strips of 300gsm
paper such that they approximated the geometry of the surface
indicated in augmented reality. Each strip would typically need
to bend and twist into arbitrary curves to match the geometry
of the surface. While the strip of paper was visible to the depth
camera on the HoloLens and within a distance threshold of
150mm to the mesh, any points that intersected the strip would
be displayed on the HoloLens. As the fabricator moved the strip
closer to the surface the colour of these points would change
to indicate a change in the proximity of the physical strip to the
surface (Figure 4).
In practice no strip perfectly matched the geometry of the
design model due to limitations of forming strips by hand, the
exibility of the material and the work time of the adhesive.
Instead fabricators would optimize for fabrication speed rather
than fabrication precision. Once a strip was within acceptable
tolerances of the surface it would be xed in place with minor
imprecision occasionally overcome by inserting subsequent
strips. The visual feedback on the precision of the strip was
therefore a means to understand the desired geometry of the
design model rather than a means to improve precision over
unguided processes. Acceptable tolerances varied over the
course of fabricating the prototype as it became necessary to
work with large variations due to deformation from self-weight.
Fabricators would typically test multiple strip orientations and
shapes using this visual feedback to balance several fabrica-
tion goals. These included maximizing covered surface area,
6 Diagram illustrating raycasting algorithm. Note the distance along the
second raycast to the mesh (green) is less than the distance along the
rst raycast (blue).
Paper Title Author last names, separated by commas62022
ACADIA
6
strip and surface curvature, connections to existing strips
for stiffness and accessibility for the purposes of applying
adhesive. Because of the inherent exibility of the material,
fabricators would attempt to avoid adding strips that could not
be supported by existing structure or that would add cantilevers
causing the structure to sag. The geometry of strips was not
dened a priori by the design model and therefore the patterns
produced by the strips were determined on the y by each
fabricator. Adding strips to the model following the heuristics
previously described required skill, and each fabricator devel-
oped a unique approach to inserting strips into the structure
with a corresponding and recognisable aesthetic (Figure 5).
Once a satisfactory geometry for a strip was found, one end
of the strip was pinned in place using hot glue. The remainder
of the strip was then formed into the planned geometry and
tacked in place with additional adhesive. This allowed some
workability of the strip and prevented accidentally xing material
in unplanned locations due to the very short work time of the
adhesive. Because the visual feedback ran as a background
process and was “always on”, any part of the as-built structure
could easily be compared to the design model for precision and
small changes could be made such as adding more strips to
reinforce slumping areas, or to work on a completely different
part of the model (Figure 5).
Because no parts were dened by the design model there was
also no predetermined and explicit sequence of assembly,
and fabrication could occur at any part of the design model
at any time. A sparse distribution of strips was produced that
7 Image of the point cloud scan and capture locations. Colours indicate deviation of points from the target digital model.
approximated the form of the prototype while holding its form
as much as possible under self-weight. This sparse distribution
was then reinforced with additional strips that could be added
without using an augmented reality guide.
RESULTS AND DISCUSSION
The prototype was completed in 6 hours by the team of three
fabricators using approximately 1000 strips of 50x300 mm
300gsm paper (Figure 2). The average deviation of the fabri-
cated surface was 27 mm. This deviation was calculated by
recording 24 point clouds from 8 vantage points and performing
a closest point analysis for the approximately 200,000 points
(Figure 7). Because this deviation is distributed over large
surfaces rather than local to the geometry of individual strips,
we expect deviation to be due to deformation of low-curvature
areas of the surface under self-weight rather than any technical
limitations to the method described for guided fabrication.
This deviation could be reduced by increasing the stiffness of
construction material, increasing the curvature of the design or
avoiding cantilevering and unsupported geometry. However, we
expect that deviation and construction time could be reduced
further simply by increasing the skill of the fabrication team.
Our fabrication team had no prior experience constructing such
complex surfaces from paper, and different ad-hoc methods
for locating strips following the surface of the design model
were invented and rened while fabricating the prototype. These
included attempting to follow the edges of the quad mesh,
following estimations of minimum and maximum curvature,
following and reinforcing topological loops or simply optimizing
7TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
7
for simplicity of installation. Providing fabricators with some
additional guidance on optimal strip orientation; for instance, by
displaying principal curvature directions of the mesh or esti-
mating geodesic curves for a given starting vector of a strip;
would accelerate the acquisition of skills that would assist in
fabricating a maximally stiff representation of arbitrary curving
surfaces. However, this would also introduce additional design
complexity and necessitate an understanding of these computa-
tional tools and approaches.
We contribute a novel method for approximately fabricating
arbitrarily curving surfaces in augmented reality that does not
require an underlying substructure or rationalization to explicit
parts. Because the method does not require the assembly of
sub structures, 2D documentation or any form of digital design
rationalization we further contribute a fast, low cost, accessible
and unconstrained design to production workow compared
to additive manufacturing or other computer aided manufac-
turing processes such as assembly from CNC cut strips. Our
method affords an intuitive approach to fabricating surfaces
from exible and lightweight materials in the same fashion
as artists such as Henrique Oliviera, while also affording the
possibility of relative precision, analysis and iteration provided
by digital design workows. Mixed reality environments provide
an additional benet and contribution to these types of proj-
ects whereby bottlenecks of design expertise are eliminated by
simultaneous and unambiguous access to design intent. This
enables complex structures to be fabricated eciently in parallel
and suggests that this would enable even larger and more
complex structures to be completed simply by increasing the
size of the fabrication team.
To evidence that the method described in this paper is gener-
alizable to other fabrication approaches we have explored two
other small proof of concepts including additive fabrication
from river pebbles (Figure 10) and arbitrary part placement with
6 degrees of freedom (Figure 11). In both cases providing fabri-
cators with visual feedback on the delta between the location of
physical material and the target geometry improves the ability
of fabricators to complete tasks. We speculate that reducing
the complexity of these fabrication tasks in augmented reality
will impact the discourse on digital fabrication by broadening
the design space of fabricatable structures in augmented reality
without introducing the need to rationalize surfaces to parts or
substructures. This in turn will have follow on implications by
improving the feasibility of constructing geometry from arbitrary,
heterogenous and non-standard materials such as those in the
Stik Pavilion (Yoshida et al. 2015), as the digital model doesn’t
need to explicitly model material behaviours and constraints for
this material to be guided into a target form.
Our guidance system enabled the fabrication team to approx-
imately complete the prototype in a very short span of
fabrication time compared to alternative methods. The speed
of the method is achieved by enabling a theoretically unlimited
number of fabricators to work on construction simultaneously.
As evidence of the ecacy of the guidance system, the fabrica-
tion team was not able to construct any part of the prototype by
working with the hologram of the design model alone. This was
attributed to the diculty of matching strip geometry to surface
geometry in augmented reality. Fabricating the prototype would
therefore ordinarily require explicitly dening parts making up
the surface and fabricating each part from rigid material to
9 A prototype fabricated with our method from long strips of timber veneer.8 A simple double-curved surface model for fabrication from timber veneer.
Paper Title Author last names, separated by commas82022
ACADIA
98
within tolerances of the digital model as has been demonstrated
by other approaches to realizing complex surface models in
augmented reality.
While completing the prototype we learned that forming double
curved surfaces from exible strips of paper requires craft skill
that needed to be acquired from practice. This resulted in initially
poor workmanship and precision that improved gradually over
time. Poor decisions in forming parts of the structure introduced
areas of low curvature that then deformed under self-weight and
could not be reinforced with additional material. The primary
failure in our method as demonstrated is in the appropriateness
of the design language of the proof-of-concept model. Large
areas of the model were without sucient double curvature or
topological support to hold their form when constructed from
paper, introducing deviation from the digital model that had little
to do with the precision of the guidance system or the accuracy
of individual strips. The marching cubes algorithm used to
create the mesh of the proof-of-concept model also contained
many details too small to form with the 50 mm wide strips,
leading to further deviations that could easily have been avoided.
CONCLUSION
This proof-of-concept project represents the very rst exper-
iment with real-time guided fabrication of surface models in
augmented reality without discretization to explicitly dened
parts. A fundamental cause of error and limitation of our
prototype was deformation of the structure under self-weight,
and in future work we aim to address this limitation by working
with more rigid materials and design models. However, we
also recognise the opportunities afforded by light weight and
readily available materials such as paper, and the possibility of
exploring ad-hoc design processes by colouring or patterning
the paper strips as an additional expressive device during
fabrication.
We have conducted additional experiments forming curved
surfaces from strips of timber veneer (Figure 8, 9). Our method
should generalize to pliable construction scale materials such
as bent timber or steel, as well as to sculpting materials such
as clay and plaster. Guided feedback in augmented reality
may also address limitations observed by other fabrication
approaches within the discourse. For instance, bricklaying with
simple augmented reality guides describing courses of bricks
in a design requires maintaining a top-down view to accurately
position each brick. This in turn necessitates frequently moving
scaffolding to construct larger walls, and this is unfamiliar to
bricklayers used to working up to and beyond eye height. Guided
feedback should enable the accurate placement of bricks in
arbitrary orientation and eliminate the need for top-down views.
Future work intends to explore these opportunities.
While we have aimed to implement and demonstrate a
generalisable and accessible method for guided fabrication
in augmented reality, future work will rene this approach to
address other observed limitations. By performing analysis
on design models, it may be possible to reduce the risk of
poor structural performance by displaying structural analysis,
geodesic curves or principal stress directions in real time based
11 Guidance system for positioning a part with 6 degrees of freedom.10 Guidance system for assembly of curved surfaces from pebbles.
9TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
1110
on the proposed position of a strip of material by a fabricator.
Similarly, providing designers with access to the HoloLens 2
depth stream within parametric modelling software such as
Grasshopper would enable the design of customisable guidance
algorithms. One could imagine tting parts to point clouds,
displaying correctional information with arrows, or colourising
mesh surfaces with various proximity information, and these
will be explored in future work and collaborations.
REFERENCES
Ayres, Phil, Alison Martin, and Mateusz Zwierzycki. 2018. Beyond the
Basket Case: A Principled Approach to the Modelling of Kagome Weave
Patterns for the Fabrication of Interlaced Lattice Structures Using
Straight Strips. Advances in Architectural Geometry 2018, 72–91. Vienna:
Chalmers University of Technology
Bamji, Cyrus S., Swati Mehta, Barry Thompson, Tamer Elkhatib, Stefan
Wurster, Onur Akkaya, Andrew Payne, et al. 2018. “IMpixel 65nm
BSI 320MHz Demodulated TOF Image Sensor with 3μm Global
Shutter Pixels and Analog Binning.” In 2018 IEEE International Solid
- State Circuits Conference - (ISSCC), 94–96. https://doi.org/10.1109/
ISSCC.2018.8310200.
Chun, Jacky, Adabelle Long, Wing Sze, Chung Hei, Garvin Goepel, and
Kristof Crolla. 2020. “Augmenting Craft with Mixed Reality: A Case
Study Project of AR-Driven Analogue Clay Modelling.” In ACADIA 2020:
Distributed Proximities [Proceedings of the 40th Annual Conference of the
Association of Computer Aided Design in Architecture (ACADIA).
Fornes, Marc. 2014. “Double Agent White.” In ACADIA 14: Design Agency
[Projects of the 34th Annual Conference of the Association for Computer
Aided Design in Architecture (ACADIA), 157-160. Los Angeles, USA.
Goepel, Garvin, and Kristof Crolla. 2020. “AUGMENTED REALITY-BASED
COLLABORATION - ARgan, a Bamboo Art Installation Case Study.” In
RE: Anthropocene, Design in the Age of Humans - Proceedings of the 25th
CAADRIA Conference, 313–22. University, Bangkok, Thailand.
“Inferno Redux/ Wind Stone.” 2018. Inferno Redux/ Wind Stone. 2018.
https://geofffarquharstill.com/exhibitions/inferno-redux-wind-stone.
Jahn, Gwyllim, Cameron Newnham, Nick Berg, and Matthew Beanland.
2018. “Making in Mixed Reality.” In Proceedings of the 38th Annual
Conference of the Association for Computer Aided Design in Architecture
(ACADIA), 88–97. Mexico City, Mexico.
Jahn, Gwyllim, Cameron Newnham, Nick Berg, Melissa Iraheta, and
Jackson Wells. 2019. “Holographic Construction.” In Impact: Design With
All The Senses, 314–24. Berlin.
Jahn, Gwyllim, Cameron Newnham, and Nick van den Berg. 2022.
"Augmented Reality For Construction From Steam Bent Timber". In
Proceedings of the 27th International Conference of the Association for
Computer-Aided Architectural Design Research in Asia. Sydney, Australia.
Jahn, Gwyllim, Andrew Wit, and James Pazzi. 2019. “[BENT] Holographic
Handcraft in Large-Scale Steam-Bent Timber Structures.” In Proceedings
of the 39th Annual Conference of the Association for Computer Aided
Design in Architecture (ACADIA), 438–47. The University of Texas at
Austin School of Architecture, Austin, Texas.
Lok, Leslie. 2022. “Timber De-Standardized 2.0 : Mixed Reality
Visualizations And User Interface For Processing Irregular Timber.” In
Proceedings of the 27th International Conference of the Association for
Computer-Aided Architectural Design Research in Asia. Sydney, Australia.
Meyboom, Annalisa, David Correa Zuluaga, and Oliver Krieg. 2019.
Stressed Skin Wood Surface Structures: Potential Applications in
Architecture. In ACADIA 19:UBIQUITY AND AUTONOMY [Proceedings of the
39th Annual Conference of the Association for Computer Aided Design in
Architecture (ACADIA), 470-477. The University of Texas at Austin School
of Architecture, Austin, Texas.
Mitterberger, Daniela, Kathrin Dörer, Timothy Sandy, Foteini
Salveridou, Marco Hutter, Fabio Gramazio, and Matthias Kohler.
2020. “Augmented Bricklaying: Human–Machine Interaction for in
Situ Assembly of Complex Brickwork Using Object-Aware Augmented
Reality.” Construction Robotics 4 (December). https://doi.org/10.1007/
s41693-020-00035-8.
Peggs, G.N., Paul Maropoulos, Ben Hughes, Alistair Forbes,
Stuart Robson, Marek Ziebart, and Bala Muralikrishnan. 2009.
“Recent Developments in Large-Scale Dimensional Metrology.”
Proceedings of the Institution of Mechanical Engineers - Part B:
Journal of Engineering Manufacture 223 (June): 571–95. https://doi.
org/10.1243/09544054JEM1284.
Quartara, Quartara, and Djordje Stanojevic. 2019. Computational
and Manufacturing Strategies. Springer. https://link.springer.com/
book/10.1007/978-981-10-8830-8.
Skeels, Christopher, and James M Rehg. 2007. “ShapeShift: A Projector-
Guided Sculpture System.” Proc. UIST 7: 2.
Song, Yang. 2020. “BloomShell: Augmented Reality for the Assembly and
Real-Time Modication of Complex Curved Structure.” In Anthropologic:
Architecture and Fabrication in the Cognitive Age - Proceedings of the 38th
ECAADe Conference, 345–54. Berlin.
Song, Yang, Richard Koeck, and Shan Luo. 2021. “Review and Analysis
of Augmented Reality (AR) Literature for Digital Fabrication in
Architecture.” Automation in Construction 128 (August): 103762. https://
doi.org/10.1016/j.autcon.2021.103762.
Paper Title Author last names, separated by commas10 2022
ACADIA
T. Sandy, and J. Buchli. 2018. “Object-Based Visual-Inertial Tracking for
Additive Fabrication.” IEEE Robotics and Automation Letters PP (99): 1–1.
https://doi.org/10.1109/LRA.2018.2798700.
Yoshida, Hironori, Takeo Igarashi, Yusuke Obuchi, Yosuke Takami, Jun
Sato, Mika Araki, Masaaki Miki, Kosuke Nagata, Kazuhide Sakai, and
Syunsuke Igarashi. 2015. “Architecture-Scale Human-Assisted Additive
Manufacturing.” ACM Transactions on Graphics 34 (4): 88:1-88:8. https://
doi.org/10.1145/2766951.
IMAGE CREDITS
All drawings and images by the authors.
Gwyllim Jahn Gwyllim Jahn is a co-founder of Fologram and a Lecturer
in Architecture at RMIT University in Melbourne. His work focuses on
designing for mixed reality fabrication, most notably in the design and
construction of the 2019 Tallinn Architecture Biennale Pavilion. Gwyllim’s
research has been published in leading computational design confer-
ences and journals including IJAC, ACADIA and RobArch and he has
given talks, presentations and workshops at international institutions
including MIT, Stuttgart ICD, UCL, SciArc, Tongji and Tsinghua University.
Cameron Newnham is the co-founder and CTO of Fologram where he
leads the technical development of mixed reality software for the design
and construction industry. His experience lies in the creation of novel
tools for designing and fabricating complex geometric systems, ranging
from code libraries to mixed reality interfaces and extending to machine
design and robotic fabrication. Cameron has experience as a computa-
tional designer in internationally renowned architectural practices, and
academic experience as an Associate Lecturer – Industry Fellow at RMIT
University. Cameron has led numerous international design and build
workshops in Shanghai, New York, Paris, Boston, Sydney, and Melbourne.
Nick van den Berg is the co-founder and CEO of Fologram, a design
research practice and technology startup building a platform for
designing and making in mixed reality. Fologram’s platform is being used
by world leading architecture practices, product design houses, manu-
facturers and design schools internationally. Nick is especially interested
in building solutions that are utilised by a large user base around the
world and has assisted with workshops focusing on utilising augmented
and mixed reality as a design and fabrication tool at DMS, CAADRIA,
Cooper Union, McNeel Europe, UDK & TU Berlin.
11TOPIC (ACADIA team will ll in)
Hybrids & Haecceities
