Content uploaded by Lawson Spencer
Author content
All content in this area was uploaded by Lawson Spencer on Oct 30, 2023
Content may be subject to copyright.
ACADIA 2023
318
318
Extended Reality (XR) Workows
for Multi-Material Assemblies
1 Eye-level perspective of the
Unlog Tower
Lawson Spencer
Cornell University
Alexander Htet Kyaw
Cornell University
Sasa Zivkovic
Cornell University
Leslie Lok
Cornell University
1
TIER 4
11’-8”
TIER 3
8’-10”
TIER 2
7’-11”
TIER 1
6’-8”
FOUNDATION STEEL
1’-2”
ABSTRACT
The architecture and construction industries have been developing methods to integrate
Augmented Reality (AR) and Mixed Reality (MR) workows into the building industry for
1-to-1 scale design, visualization, and paperless fabrication. While these AR workows
have been primarily focused on mono-material assemblies, this paper investigates the
potential of AR and MR for multi-material fabrication, combining various materials and
structural components throughout each phase of the construction of the Unlog Tower.
The installation uses infested and dying ash trees to construct a 36-foot-tall triangular,
lightweight timber structure. The Unlog Tower leverages bending active elastic kinematics
to stretch robotically kerfed logs braced by threaded rods and tube steel. Three extended
reality (XR) workows were explored for the construction of this bespoke timber struc-
ture: (1) ducial marker coordinated AR instruction, (2) multiple QR code AR instruction,
and (3) gesture-based MR instruction. These XR workows incorporate feedback-based
construction notation and animation for the assembly of non-standard natural materials
and standardized parts through three construction phases: materials to parts, parts to
prefab modules, and onsite assembly. The research highlights the potential of AR and MR
workows for human-machine interaction in robotic fabrication, analog means of making,
prefabrication, onsite construction, and coordination. The result of this investigation has
demonstrated many advantages and disadvantages of varying AR/MR workows in facili-
tating the construction of multi-material and multi-phase structural assemblies.
TOPIC (ACADIA team will ll in) 319
HABITS OF THE ANTHROPOCENE
TOPIC (ACADIA team will ll in) 319
(optional)
(optional)
(optional)
FIDUCIAL MARKER AR (FM AR) WORKFLOW GESTURE BA SED MR (GBMR) WORKFLOWMULTI-QR CODE AR (MQAR) WORKFLOW
2
INTRODUCTION
Easily deployed and assembled, the Unlog Tower (Figure
1) stretches several logs into a triangular, lightweight
timber tower through robotic kerng and bending-ac-
tive kinematics (Schleicher et al. 2011). The research
utilizes robotic kerng techniques and extended reality
(XR) instruction to transform Emerald Ash Borer (EAB)
infected logs (Flower, Knight, and Gonzalez-Meler 2012)
into a materially ecient and locally available timber
resource. As a bending-active timber structure, the Unlog
Tower is comprised of several material types ranging from
recently harvested ash logs to various hardware (such as
nuts, bolts, washers, and threaded rods) and extruded/
rolled steel proles. The structure also utilized an existing
concrete foundation at the installation site. The fabrication
and assembly of the tower are broken down into three
construction phases: materials to parts, parts to prefab
modules, and onsite assembly. These three construction
phases utilize the three AR and MR workows for the
fabrication methods, ranging from robotic fabrication,
analog making, prefab assembly, and onsite construction
coordination.
These materials were processed through computer-aided
manufacturing techniques such as computer numerical
control (CNC), plasma cutting, and robotics along with
analog techniques such as welding, drilling, and cutting.
Throughout the design and construction of the Unlog
Tower, a set of three distinct augmented reality (AR) and
mixed reality (MR) workows were developed to address
the diverse range of materials and processes involved.
These include (1) ducial marker AR (FMAR) instruction, (2)
multiple QR code AR (MQAR) instruction, and (3) gesture-
based MR (GBMR) instruction, each tailored to facilitate the
fabrication and assembly stages of the installation. Within
architecture and fabrication, current research on AR and
MR technology has showcased the potential of AR in aiding
1:1 scale design, visualization, and paperless fabrication.
While previous approaches have primarily focused on AR
workows for mono-material fabrication, this paper intro-
duces workows for multi-material assemblies.
STATE OF THE ART
The following precedents demonstrate that dierent XR
methods of instruction and ducial marker placement have
varying degrees of physical and digital interactivity, 3D
user interface (3DUI) customization, and accuracy for digi-
tal-physical object alignment for both mobile devices and
head-mounted displays (HMD). Most architectural fabrica-
tion research projects, such as Holographic Construction,
Code-Bothy, Woven Steel, and Augmented Feedback have
developed interactive 3DUIs to instruct the fabrication
of complex assemblies using the FMAR workow (Jahn et
al. 2020; Lee 2022; Jahn, Newnham, and Beanland 2018;
Goepel and Crolla 2022; Lok and Bae 2022). This workow
uses a single QR code to superimpose digital geometry
into or onto the physical environment (Figure 2). Further
interplay between digital and physical objects can be
conducted through the tracking and use of ArUco markers
(Figure 2). In Holographic Construction and Code-Bothy, a
single QR code is used to superimpose digital “buttons” to
toggle between rows of bricks as they are laid (Lee 2022;
Jahn et al. 2020). HoloWall superimposed an interactive
MR work surface to process salvaged lumber into boards
for a hollow-core, cross-laminated timber (CLT) panel (Lok
and Bae 2022). Woven Steel uses interactive menus to
instruct the bending of tube steel, which is then checked
by the placement of ArUco markers (Jahn, Newnham,
and Beanland 2018). Augmented Feedback places ArUco
markers at each joint location on a bending-active, bamboo,
gridshell structure to track and check the elastic displace-
ment of each joint until the structure nds a natural stable
position (Goepel and Crolla 2022). ArUco markers have
also been used to index and instruct the assembly of a
2 Dierent AR and MR methods of interaction with dierent QR code types.
EXTENDING REALITIES
ACADIA 2023
320
320
curvilinear wall surface made of discrete recycled wood
boards (Parry and Guy 2021). Each of these research
investigations relies on the placement of a single QR code
for digital geometry localization and orientation with the
optional added interactive instruction through custom
digital menus and the use of ArUco markers.
The MQAR workow uses multiple ducial markers for the
accurate superimposition of digital geometry on phys-
ical objects (Figure 2). This workow has been commonly
tested and employed in both large-scale construction
projects using building information modeling (BIM) and
high-precision fabrication projects. One such research
project investigated the accuracy of using AR with
precision marker registration to instruct contractors on
the installation and inspection of outtting parts of large-
scale, oshore ship building without the use of paper
drawings (Choi and Park 2021). The research demon-
strated a method in which ducial markers were placed in
measurable locations relative to existing physical objects,
the markers then tracked through a sh-eye camera using
simultaneous localizing and mapping (SLAM) technology to
reduce pose drift despite the large scale of the BIM model
(Choi and Park 2021). Also using SLAM technology, the
authors of this paper have previously employed a multiple
non-rotational, QR code workow for accurate post-pro-
cessing of glulam beams (Kyaw, Xu, et al. 2023). Still, others
have tested the digital-to-physical accuracy of object
fiducial marker AR (FMAR) workflow multi-QR code AR (MQAR) workflow gesture based MR (GBMR) workflow R = See Results
angle steel
industrial stock manufactured natural
materials [ A ] part s [ C ] on-site construc tion[ B ] prefab modules
wide flange
hardware
logs half logs
A1 A2 R R A3
or
or
[ [] or or ]+
B1 B2
+
R
or
+
A5
or
or
[ ]
A4 R
wall panels
C1
C2 R
tower assembly
foundation coordination
threaded rods w/ hex nuts
custom slip washers
bracing frames
foundation steel
threaded rod
plate steel
tube steel
3
3 Workow Diagram
TOPIC (ACADIA team will ll in) 321
HABITS OF THE ANTHROPOCENE
TOPIC (ACADIA team will ll in) 321
overlay using only a global positioning system (GPS) and the
internal measuring unit (IMU) on a mobile device (Ashour
and Yan 2020). Though the method did employ several QR
codes, the drift error was on average two meters between
tests (Ashour and Yan 2020). This issue has also been
observed by the authors in previous research when using
the FMAR workow with SLAM technology (Kyaw, Xu, et al.
2023).
The last workow utilizes gesture recognition for interac-
tive MR instruction. The GBMR workow method records
the user’s ‘pinching’ motion, and saves it as a point in
space (Figure 2) (Kyaw, Spencer, et al. 2023). This method
can be employed to place points in digital space relative to
physical objects to enable interactive assembly instruc-
tions that correspond to the user’s actions (Kyaw, Spencer,
et al. 2023). This novel interaction enables a seamless way
to integrate analog habits of making with MR assembly
instructions.
METHODS
This research employs the three above-mentioned XR
workows to instruct the users throughout each phase
of the construction process (Figure 3). The Unlog Tower
encompasses a multi-material assembly of logs, threaded
rods, standard hardware, custom steel slip washers, and
extruded/rolled steel proles. The project uses a 6-axis
robotic arm on an external track with a 5-HP bandsaw end
eector to kerf 6 logs into 12 operable leaf-spring half-logs,
which are stretched along threaded rods to form individual
panels. The stretched half-log panels are then assembled
into three prefab panels connected by six triangular steel
tube frames to resist lateral buckling. Each prefab panel
is composed of four tiers (Figure 1). Finally, the connected
panels were lifted to the existing foundation pad. Each
of the three construction phases: (A) parts, (B) prefab
modules, and (C) onsite assembly, uses several of the XR
workows that are further elaborated upon in the subse-
quent sections for this multi-material assembly.
Materials to Parts
The rst phase, materials to parts, includes (A1) log
indexing and mounting for robotic fabrication, (A2) robotic
log kerng, (A3) half-log nger jointing, (A4) hex nut place-
ment on threaded rods, and (A5) tube steel coordination,
as illustrated in Figure 3. The materials to parts phase
involves converting raw materials, custom hardware, and
standard hardware into specic parts for the Unlog Tower.
Log Indexing and Mounting for Robotic Fabrication
Logs are organic material that naturally come with many
defects and irregularities. The presence of defects, such
as knots and signicant checking or splitting, aect the
structural integrity of the material. On the other hand,
the curvature of logs can also introduce signicant
irregularity. To index the curvature and irregularities of
log geometries for robotic fabrication, the FMAR method
was employed. First, ArUco markers were placed on the
log at points of irregularity (knots, splits, et cetera), then
two ArUco markers were placed at either end of the log
(Figure 4). By using an HMD, the location of the ArUco
markers could be registered as either points or planes
within the digital environment. After the ArUco markers
were placed, the log was scanned, creating a digital mesh
of the physical object. Once the digital mesh was saved, the
cut geometries were planned based on the irregularities
and curvature of the scanned mesh. After the log indexing
process was complete, the log was mounted in the robot
cell.
For the robotic fabrication procedure, the physical location
and orientation of the mounted log in the robot cell are
required (Figure 5). The rst robotic procedure involves
cutting the log into halves. A reference plane of the
4 Log Indexing and Planning. 4
0
1
2
3
LOG LOG SCANNING AND INDEXED
W/ ARUCO MARKERS
CUT GEOMETRY PL ANNING AND LOCATING
irregularities
indexed cut geometry for robot
scan in order
0 → 1 → 2 → 3
pre-attached
ArUco marker
marks of
irregularity
scanned log
physical log
EXTENDING REALITIES
ACADIA 2023
322
322
01
32
SCANNING ARUCO MARKER S RE-ORIENT DIGITAL
TO PHYSICAL
scan ArUco Markers
with smart phone
cut geometry
screw position
ArUco makers
digital log
mobile device
cut geometry for robot
mounted log
FMAR workflow
digital planning
physical
5
mounted log in the robot cell is constructed by registering
the coordinates of the four physical ArUco markers placed
at the log’s ends using an HMD (Figure 6a). This coordi-
nating plane is used to reorient the scanned log mesh and
the associated cut geometry to the actual position of the
physical log in the robot cell (Figure 6b). The digital log,
along with the associated cut geometry, is then displayed
back to the user as feedback in MR (Figure 6c). Based on
this visual feedback, the fabricator can re-orient the log on
the robot mounts, if necessary, or adjust the cut geometry.
Another procedure using the GBMR workow was devel-
oped for straight log mounting without ArUco markers.
By using the “pinch” gesture, the user can register three
points at each end of the mounted log to generate two
circles that represent the log ends (Figure 7). These two
circles are lofted to create a digital log mesh to orient
and locate the physical mounted log in the robot room.
This method is more ecient, as it only takes one step to
generate and locate the digital geometry relative to the
physical log in the robot cell, thus, reducing the steps, such
as 3D scanning and additional re-orientation adjustment.
Robotic Log Kerfing
For the coordination of the second robotic procedure, the
FMAR method was utilized to situate the robotic kerng
cut geometry. The users can place an ArUco marker at
the desired location they want to situate the cut geometry
(Figure 8). The cut geometry and fabrication notation are
oriented by scanning the ArUco marker via one’s mobile
device or HMD. In the AR 3DUI, the digital geometry is
displayed in two dierent cut directions, (1) in cyan, the
bandsaw will enter the half-log from the left, and (2) in
magenta, the bandsaw will enter the log from the right.
Additionally, a digital twin of the robot cell is modeled to
a b c
6
a b c
7
8
5 Digital log planning and re-orientation in robot cell using ArUco markers.
6 Orientation of digital geometry to match physical: (a) ArUco markers
are used to determine the physical log orientation; (b) Overlay of digital
geometry; (c) Pink surface illustrates rst cut for robot.
7 GBMR generates the location of a cylinder according to the diameter(s) of
the log which digitalizes the location of the log for robotic fabrication.
8 Robot Cut Geometry: in cyan: bandsaw end eector cuts left to right; in
magenta: bandsaw end eector cuts right to left; in orange: potential
collisions; and in green: added instruction through notation.
TOPIC (ACADIA team will ll in) 323
HABITS OF THE ANTHROPOCENE
TOPIC (ACADIA team will ll in) 323
provide collision feedback to the user. In AR 3DUI, the
probable locations of the mounting screws are indicated
in orange notation. This provides valuable guidance to the
fabricator in assessing potential intersections between the
cut geometry and the screws. Lastly, the green notation
provides the location in which to place a stop block, which
helps the bandsaw blade from being pinched by the wood
as it backs out of the log.
Half-Log Finger Jointing
In order to create panel components that can be
assembled into prefab modules (Figure 9a), nger joint
connections are fabricated at either end of each half-log
(Figure 9b). The location of the nger joint is often stag-
gered between layers of boards within the half-logs (Figure
9c). The GBMR method was utilized to display the location of
the nger joint template corresponding to the board layer
of the user’s recorded gesture. By using gestural recog-
nition, the user can pinch their index nger to their thumb
to register a digital point at the corner of the board in the
MR environment. The recorded "z" value of this digital point
is used as the parameter to locate the template per each
board layer (Figure 10). The template illustrates the nger
joint outline and the location of the hole for the threaded
rod to pass through. Once the template was correctly
placed, it was marked then the nger joints were cut with a
handheld oscillating saw.
Hex-Nut Placement on Threaded Rods
As mentioned previously, each half-log part was stretched
along two threaded rods (Figures 9a and 9b). The threaded
rods have pre-located hex nuts to determine the kerf
spacing; therefore, the location of the hex nuts required
a high degree of accuracy to ensure that the kerfed logs
would spread correctly. A jig was designed to secure the
hex nut holder in place using the MQAR method. Each hex
nut holder was placed as instructed by the digital notation
and screwed into the mounting board (Figure 11a). Then,
the threaded rod was screwed through the hex nuts using
a drill. The notation provides the user with information
about each threaded rod (green), a notation about the hex
nut locations (blue), and an overlay of the digital threaded
rod and hex nuts (purple) for visual quality control (Figure
11b).
Tube Steel Frame Coordination
To brace each of the prefab wall panels from lateral
buckling, several tube steel frames were designed. Three
unique frames were designed to be placed in between
the top three tiers of the installation. Each frame was
comprised of three dierent tube steel lengths with three
dierent cuts on the end of the tube steel, resulting in nine
unique types and lengths of tube steel. Using the GBMR
method, the tube steel length was used as a parameter to
distinguish between each member. The user can “pinch” at
the ends of a tube steel component; the recorded distance
was matched to the nearest length value of the steel tubes
to correctly identify each member through visual feedback
(Figure 12). The feedback displayed a notation for the tube
steel type as well as a 1:1 AR model of the member within
the tube steel frame and a 1:10 coordination model of the
frame within the whole tower. These virtual coordination
models were used to instruct the welding of each tube
steel frame. Here, the GBMR method is a more streamlined
method of interaction that doesn’t require the arduous
placement of ArUco markers.
a b c
a b
a b c
a b c
12
11
9
10
9 Finger joints and threaded rod holes: (a) two connected kerfed panels;
(b) kerfed log spread out along threaded rod with temporary plywood
slip washers; (c) nger joints and threaded rod holes staggered through
robotically kerfed half-log.
10 Gesture Recognition is used to identify physical components and instruct
the user on where to place the nger joint template per board layer.
11 Pre-locating hex nuts on threaded rod: (a) 3D printed jig pieces placed
using AR 3DUI; (b) notation on threaded rod.
12 GBMR is used to catalog and distinguish between tube steel lengths.
EXTENDING REALITIES
ACADIA 2023
324
selected
item
1A
tube steelused
1B
1C
2A
2B
2C
3A
3B
3C
coordination
check list
coordination
model
tube steel
FMAR workflow
digital planning
physical
The FMAR method was also tested using ArUco markers to
identify and distinguish between the 9-tube steel lengths
which were pre-grouped. Each unique set of tube steel
geometry was assigned an ArUco marker with an identi-
cation number (Figure 13). When the fabricator taps the
ArUco marker, a tube steel checklist would highlight the
name of the item, along with the two types of coordination
models listed above. The GBMR method did not require
individual members to be pre-grouped according to length;
individual members could be distinguished within an
unsorted pile.
Parts to Pre-fab Modules
The third phase, components to prefab modules,
includes (B1) finger joint connection and (B2) panel
assembly. This phase involves connecting the panel
components into prefabricated modules that can then
be assembled onsite.
Finger Joint Connection
The panel components are connected to each other via
a splice connection through the nger joints. The splice
connection includes four 0.375-inch bolts that x the
0.25-inch-thick plasma cut slip washer into place. The
custom slip washers are designed to slide onto the
threaded rods with pre-located hex nuts. The connection
is made rigid by four 0.375-inch bolts. To communicate
this complex joint to the assembly team, an AR assembly
instruction is animated by a slider within the 3DUI using
the FMAR workow (Figure 14).
Panel Assembly
A wall module is comprised of four panels joined through
28 splice connections. The Unlog Tower consists of three
total prefab modules. Each module is comprised of four
tiers resulting in 84 total splice connections. To coordinate
fabrication between each prefab module, the assembly
team used the GBMR workow to coordinate the joining
of each kerfed timber half-log, and ensure the correct
spacing of each splice connection (Figure 15). By using
gestural tracking, the user can place a digital point at the
center of the nger joint after the splice connection is
fastened. The distance between this digital point and the
center point of the closest digital board is calculated. If
the deviation is less than 0.125-inch, a green notation is
displayed. If the deviation is more than 0.125-inch, a red
notation is displayed, indicating that the splice connection
is not connected or properly calibrated. The notations
can also be seen by all other users in the fabrication team
using a mobile device or an HMD. Therefore, the assembly
team can also use this step for quality control to ensure
that all splice connections are properly spread and spaced
along the threaded rod.
On-site Assembly
The last phase, onsite assembly, includes (C1) panel bracing
and (C2) foundation coordination. In this section, the three
prefabricated modules are joined and braced together
before connecting them to the foundation.
13
a b
a b
14
15
16
13 Notation and coordination of steel tube placement for welding.
14 Assembly sequence of nger joint connection between kerfed timber
components with interactive slider.
15 Assembly of components into prefab elements with detail notation.
16 Quality Control – overlay of AR on physical construction.
TOPIC (ACADIA team will ll in) 325
HABITS OF THE ANTHROPOCENE
Panel Bracing
The prefabricated panel modules and the steel frames
were transferred to the site and assembled horizontally.
The three prefabricated panel modules were assembled
into a four-tier triangular structure. The bracing frames
were attached to the panel lying on the ground, then the
other two prefab panels were attached to the bracing
frames. Using the FMAR workow, the notations display
the associated detail drawing, as well as any misalignment,
between digital geometry and the physical object(s) (Figure
16).
Foundation Coordination
Lastly, the Unlog Tower is sited on an existing foundation
from a previous installation (Chong 2015). In the draw-
ings provided by the previous engineer, the reinforced
concrete pad measures 9’9” x 9’9” x 2’10” thick, and sits 12”
below grade. The previously used foundation pad contains
exhibits several threaded rods. To keep the timber for the
Unlog Tower from absorbing water from the ground, the
base of the tower sits two inches above the ground on top
of three W14x30 wide ange steel beams. The W14x30 has
welded steel plates on angle steel to fasten the base of the
timber to the foundation steel. Each wide ange member
was connected to the foundation with eight 0.5-inch-diam-
eter threaded rods. The FMAR workow was employed to
detect potential overlaps between new foundation beam
locations and the embedded thread rods. The QR code
was placed at one corner of the slab, once two edges of
the existing slab aligned with the digital geometry. AR was
used to instruct the placement of the foundation steel.
Here, the digital model was adjusted so that none of the
new holes for the threaded rods would overlap with the
embedded rods from the previous installation (Figure 17a).
Once the digital model was revised to an acceptable cong-
uration, the foundation steel was then placed into location
ab c
17
(Figure 17b), and then checked with measuring tapes for
inaccuracies that might occur due to pose drift (Figure
17c). The resulting misalignment was no greater than 0.125
inches.
RESULTS
After the foundation steel was installed, the 36-foot-tall
Unlog Tower was lifted into place with a boom forklift. The
tower stretches, both literally as an assembly and gura-
tively as a locally available natural resource devastated
by the EAB epidemic (Figure 18). Through structural
simulation and XR workows, the Unlog Tower questions
our habits in the Anthropocene, by stretching a material
resource that would otherwise be discarded or burned
for rewood. This paper studies the application of three
dierent XR workows, according to various degrees of
interactivity, accuracy, and set-up diculty during each of
the nine multi-material construction steps.
In the log indexing and planning step (A1 in Figure 3), the
FMAR workow was used to place ArUco markers at key
locations along the log, then the log was scanned. The
resulting digital mesh was used to plan the cut geometry
for the log. The advantage of scanning the log with the
ArUco markers already on the log is that the user could
re-orient the log in physical space as many times as neces-
sary and then just re-scan the ArUco markers to orient
the digital geometry to the physical, which was used in the
log-mounting step. A second test also showed the validity of
using the GBMR workow to locate a straight physical log
quickly and accurately. Further investigations will improve
upon this method to account for log curvature.
In the half-log cutting step (A2 in Figure 3), the FMAR
workow was employed to align the ArUco marker on
the half-log to the physical space in the robot cell, which
17 Placement of foundation steel with AR; cyan illustrates notation including dimension and corner of steel for alignment, green illustrates construction lines
for corner of steel placement, magenta illustrates location of threaded rod holes to be drilled, and white lines are for physical objects: (a) location of new
threaded rod (magenta) adjacent to threaded rod from previous project (photo taken with HoloLens 2); (b and c) placement of foundation steel with AR
overlay (photos taken with iPad Pro).
EXTENDING REALITIES
ACADIA 2023
326
326
1918
18 View stretched half-logs as Tier 2 and Tier 3 connection
19 Perspective of steel frames coordinated through the GBMR workow.
was important to visualize the location of the mounting
screws relative to the cut geometry. Because of the drift
issue, and the diculty of correctly orienting the ArUco
marker, this step would have beneted from a more
rened GBMR workow. However, the visibility of the cut
geometry and the added notation were helpful in ensuring
that the half-log was large enough for the board width for
each kerfed layer. Future investigations will test the MQAR
workow for improved accuracy, assuming it is required
for the fabrication.
The half-log nger jointing step (A3 in Figure 3) employed
the GBMR workow. Though this step could have also been
completed with the FMAR workow, the GBMR workow
was faster to work with. This was due to the fact that the
location of each template was based upon the distance of
the gesture to ground as opposed to the indexing of many
ArUco markers to many locations. The use of gestural
recognition allows the user to display digital geometry
through gestures, as opposed to ArUco markers.
The hex nut placement (A4 in Figure 3) used the MQAR
workow to accurately place the hex nut holder so that
hex nuts would be correctly spaced along the threaded rod.
This workow also used graphic notation to call out which
threaded rod was being used, as well as how many nuts
would be used, along with location of the hex nut holders.
In continued research, the GBMR workow has proven to
be useful in this step (Kyaw, Spencer, et al. 2023).
The tube steel coordination (A5 in Figure 3) was conducted
using the GBMR workow with the length of each tube
as the variable to dierentiate each unique part (Figure
19). Although it was faster to set up nine ArUco markers
and use the FMAR workow, an advantage of the GBMR
workow is that it allows the user to index individual
members from an unsorted pile. An improvement would be
to integrate a checklist to track progress similar to the one
developed for FMAR workow.
The prefab module instruction and assembly (B in Figure 3)
had two parts: rst, the interactive detail animation, and
second, the quality control check. The FMAR workow was
particularly useful to instruct the assembly of a complex
joint. The GBMR workow was employed to precisely adjust
and locate the kerfed boards at each nger joint, and to
ensure overall geometric accuracy.
The on-site tower assembly (C1 in Figure 3) step used the
FMAR workow to coordinate the on-site assembly of the
prefab panels as they were xed into location with the steel
tube frames. Because the tower was assembled horizon-
tally on a sloped ground surface, it was dicult to align the
digital geometry to the physical using the FMAR workow.
Future investigations could use GBMR workow to register
precise points on the physical structure as a means to
better align and orient the digital geometry.
Finally, the FMAR workow was quite valuable for the
on-site foundation steel coordination (C2 in Figure 3). As
there are no as-built drawings of the precise location of
the existing embedded threaded rod locations, it would
have been a time-consuming process to identify new drill
locations for the foundation steel without an XR work-
ow. A future iteration of this step might also investigate
the use of the GBMR workow, as long as the accuracy is
comparable to what was measured in the eld (0.125-inch
tolerance).
CONCLUSION
Through the construction of the Unlog Tower, the inte-
gration of the three XR workows provided an in-depth
study and evaluation of the various AR/MR workows for
fabrication instruction and coordination. In the installa-
tion's three construction phases, there were pros and cons
for each specic workow that involved varying complexity
of 3DUI development, degree of interactivity, and object
alignment in relation to scale and unique material types,
such as salvaged timber. The FMAR workow allows both
users and designers to develop custom 3DUIs for digital
interaction that can inform and instruct the fabrication
process. However, this workow does require a tedious
precise orientation of the base QR code to accurately
overlay digital geometry with physical objects. The MQAR
workow allows users to be less precise with the rotation
of the QR code(s) placed, yet the corner location of the
QR code is imperative for an accurate overlay between
digital geometry and physical objects. The advantage
of this workow is its capacity for a highly precise AR
superimposition; however, it has yet to be employed for
more interactive instruction, such as custom menus,
assembly animations, and object indexing. For the GBMR
TOPIC (ACADIA team will ll in) 327
HABITS OF THE ANTHROPOCENE 327
workow, whose precision is limited to the accuracy of
the hand-based gesture, it is more compatible with human
scale procedures and objects. The other advantage is
a highly interactive and direct association with physical
objects. However, the GBMR workow is highly specic
and customized per task, thus, it is more time-consuming
to develop the 3DUI for each task. Like the Unlog Tower,
building construction is a multi-material assembly; it is
imperative to integrate multiple XR workows for various
degrees of interactivity, accuracy, and scale for the
instruction between digital geometries and non-standard
and standard/manufactured materials.
ACKNOWLEDGEMENTS
The Unlog Tower was exhibited at the 2022 Cornell Biennial, and
was partially funded by the Cornell Council of the Arts.
REFERENCES
Ashour, Ziad, and Wei Yan. 2020. “BIM-Powered Augmented
Reality for Advancing Human-Building Interaction.” In
Proceedings of the 38th eCAADe Conference, 1:169–78. Berlin,
Germany.
Choi, Sungin, and Jung-Seo Park. 2021. “Development of
Augmented Reality System for Productivity Enhancement in
Oshore Plant Construction.” Journal of Marine Science and
Engineering 9 (2).
Chong, Jaeho. 2015. “A Needle Woman: Galaxy Was a Memory,
Earth Is a Souvenir.” Space, 114–18.
Flower, Charles E., Kathleen S. Knight, and Miquel A. Gonzalez-
Meler. 2012. “Impacts of the Emerald Ash Borer (Agrilus
Planipennis Fairmaire) Induced Ash (Fraxinus Spp.) Mortality
on Forest Carbon Cycling and Successional Dynamics in the
Eastern United States.” Biological Invasions 15 (4): 931–44.
Goepel, Garvin, and Kristof Crolla. 2022. “Augmented Feedback:
A Case Study in Mixed-Reality as a Tool for Assembly and
Real-Time Feedback in Bamboo Construction.” In ACADIA 2021:
Toward Critical Computation - Proceedings of the 41st ACADIA
Conference., edited by Kathrin Dörer, Stefana Parasho, Jane
Scott, Biayna Bogosian, Behnaz Farahi, Jose Luis García del
Castillo y López, June A. Grant, and Vernell A. A. Noel, 232–37.
Jahn, Gwyllim, Cameron Newnham, and Matthew Beanland. 2018.
“Making in Mixed Reality. Holographic Design, Fabrication,
Assembly and Analysis of Woven Steel Structures.” In
Proceedings of the 38th ACADIA Conference., 88–97. Mexico
City, Mexico.
Jahn, Gwyllim, Cameron Newnham, Nick van den Berg, Melissa
Iraheta, and Jackson Wells. 2020. “Holographic Construction.”
In Impact: Design With All Senses, edited by Christoph
Gengnagel, Olivier Baverel, Jane Burry, Mette Ramsgaard
Thomsen, and Stefan Weinzierl, 314–24. Cham: Springer
International Publishing.
Kyaw, Alexander Htet, Lawson Spencer, Sasa Zivkovic, and Leslie
Lok. 2023. “Gesture Recognition for Feedback Based Mixed
Reality and Robotic Fabrication: A Case Study of the UnLog
Tower.” In Proceedings of DigitalFUTURES—The 5th International
CDRF Conference. Shanghai, China: Springer.
Kyaw, Alexander Htet, Arvin HaoCheng Xu, Gwyllim Jahn, Nick
van den Berg, Cameron Newnham, and Sasa Zivkovic. 2023.
“Augmented Reality for High Precision Fabrication of Glued
Laminated Timber Beams.” Automation in Construction 152.
Lee, Guan. 2022. “Code-Bothy: Mixed Reality and Craft
Sustainability.” Frontiers of Architectural Research, June.
Lok, Leslie, and Jiyoon Bae. 2022. “HoloWall: A Mixed Reality-
Informed Hollow Core Wall Assembly.” In ACADIA 2022: Hybrids
& Haecceities - Proceedings of the 42nd ACADIA Conference,
edited by Dr. Masound Akbarzadeh, Hina Jamelle, Dr. Dorit Aviv,
and Robert Stuart-Smith. Philadelphia.
Parry, Caitlyn, and Sean Guy. 2021. “Recycling Construction
Waste Material with the Use of AR.” In Proceedings of the 2020
DigitalFUTURES, edited by Philip F. Yuan, Jiawei Yao, Chao
Yan, Xiang Wang, and Neil Leach, 57–67. Singapore: Springer
Singapore.
Schleicher, Simon, Julian Lienhard, Simon Poppinga, Tom
Masselter, Thomas Speck, and Jan Knippers. 2011. “Adaptive
Façade Shading Systems Inspired by Natural Elastic
Kinematics.” In Proceedings of the International Adaptive
Architecture Conference IAAC, 2545–54. London.
IMAGE CREDITS
Figures 1, 9a, 18 and 19: by Cynthia Kuo 2022.
All other drawings and images by the authors.
Lawson Spencer is a research associate at the Cornell Robotic
Construction Laboratory (RCL) at Cornell College of Architecture,
Art, and Planning and project lead at HANNAH. Lawson’s research
broadly investigates computational methods to structurally simu-
late and robotically fabricate with timber.
Alexander Htet Kyaw is a research associate at the Rural Urban
Building Innovation Lab (RUBI) at Cornell AAP, with a background
EXTENDING REALITIES
ACADIA 2023
328
328
in mixed reality, digital fabrication, robotics, biomaterial, digital
twins, simulation, machine learning, and computer vision for
human-machine collaboration in design, visualization, and
fabrication.
Sasa Zivkovic is an Assistant Professor at Cornell University
AAP, the Director of the RCL at Cornell AAP, and a Co-Principal
at HANNAH. Zivkovic’s research focuses on the development of
sustainable robotic construction technologies, material systems,
and fabrication processes.
Leslie Lok is an Assistant Professor at Cornell University AAP
where she directs RUBI Lab. She is also a co-founder at HANNAH.
Working with non-standardized material and biomaterial, her
research explores computational, robotic, and mixed reality tech-
nologies to customize design and fabrication workows.
TOPIC (ACADIA team will ll in) 329
HABITS OF THE ANTHROPOCENE 329EXTENDING REALITIES
