Conference PaperPDF Available

Drawing Recognition - Integrating Machine Learning Systems into Architectural Design Workflows

Authors:

Abstract and Figures

Machine Learning (ML) has valuable applications that are yet to be proliferated in the AEC industry. Yet, ML offers arguably significant new ways to produce and assist design. However, ML tools are too often out of the reach of designers, severely limiting opportunities to improve the methods by which designers design. To address this and to optimise the practices of designers, the research aims to create a ML tool that can be integrated into architectural design workflows. Thus, this research investigates how ML can be used to universally move BIM data across various design platforms through the development of a convolutional neural network (CNN) for the recognition and labelling of rooms within floor plan images of multi-residential apartments. The effects of this computation and thinking shift will have meaningful impacts on future practices enveloping all major aspects of our built environment from designing, to construction to management.
Content may be subject to copyright.
Drawing Recognition
Integrating Machine Learning Systems into Architectural Design Workflows
Lachlan Brown1, Michael Yip2, Nicole Gardner3, M. Hank Haeusler4,
Nariddh Khean5, Yannis Zavoleas6, Cristina Ramos7
1,3,4UNSW / Computational Design 2PTW Sydney 5Austrian Institute of Technology
6,7UNSW / Computational Design
1lachlan.brown@student.unsw.edu.au 2m.yip@ptw.com.au 3,4,6,7{n.gardner|m.
haeusler|y.zavoleas|c.ramos}@unsw.edu.au
5Nariddh.Khean@ait.ac.at
Machine Learning (ML) has valuable applications that are yet to be proliferated
in the AEC industry. Yet, ML offers arguably significant new ways to produce and
assist design. However, ML tools are too often out of the reach of designers,
severely limiting opportunities to improve the methods by which designers design.
To address this and to optimise the practices of designers, the research aims to
create a ML tool that can be integrated into architectural design workflows.
Thus, this research investigates how ML can be used to universally move BIM
data across various design platforms through the development of a convolutional
neural network (CNN) for the recognition and labelling of rooms within floor
plan images of multi-residential apartments. The effects of this computation and
thinking shift will have meaningful impacts on future practices enveloping all
major aspects of our built environment from designing, to construction to
management.
Keywords: machine learning, convolutional neural networks, labelling and
classification, design recognition
INTRODUCTION, CONTEXT, MOTIVATION
If you can understand a floor plan, you can design a
floor plan, and to understand you must learn. While
this statement might simplify learning one can argue
that with Machine Learning (ML), provided enough
training, a program can currently become the mas-
ter of an increased number of things. Popular exam-
ples might include strategy such as DeepMind’s Al-
phaGo Zero which, after three days of self-teaching
defeated world no.1 Go player Ke Jie [1], or creative
endeavours observed in The Next Rembrandt project,
that generated a new Rembrandt artwork after study-
ing the artist’s body of work [2]. Considering the ver-
satility, and rampant accessibility of modern ML al-
gorithms, this research is provoked into exploring ML
within architecture, a field with an increasing number
of meaningful applied results and attempts (Khean,
2017, 2019).
D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2 - eCAADe 38 |289
Yet there is evidence that the Architecture, En-
gineering and Construction (AEC) industry often re-
sists the integration of modern technologies into
their workflows [3]. As outlined by McKinsey in [3]
there currently exists a visible reluctance to adopt
emerging technologies, which only serves to hinder
the means for architects to improve their designs to-
wards more economic, efficient, environmental and
aesthetic structures. Consequently these issues drain
energy from core design work and compound in a
compromised end product resulting in the end users
experiencing the built environment negatively and
compromising the productivity of the AEC sector [4].
When analysing the productivity gain in other sectors
enabled through the introduction of ML [5] (McAfee,
Brynjolfsson, 2017) one can clearly argue for the need
to introduce ML into the AEC sector as it serves the
creation of a nation’s critical infrastructure and build-
ing stock with currently AU$200B in the pipeline [6]
needed to keep pace with the nations population
growth [7]. It is an industry crucial to the national
economy responsible for 8% of GDP or $134B P.A.
[8]. It has, however, been slow to adopt new tech-
nology (Deutsch 2019), digitally enabled workflow
processes that unlock productivity [4] (Susskind &
Susskind 2016; McAfee et al. 2017). We argue there-
fore for research and investigations into ML’s ability
to automate ‘mundane’ tasks that are essential but
not ‘very sexy’ in order to increase the productivity
of the AEC industry. Items that, amongst others can
fall under the category of mundane tasks are com-
pliance checks and the question of how to perform
compliance checks (Knodel, Naab, 2016). Specifi-
cally, for [NAME WITHHOLD] Architects whom this
research was conduct with identified a need to im-
prove solar analysis compliance checking according
to the local building code. The code required that
only living rooms and balconies needed to be as-
sessed, however the BIM data defining these rooms
failed to be transferred from Revit to Rhino. Con-
sequently the [NAME WITHHOLD] Architects ques-
tioned if ML could extract relevant information from
one place and interpolate it in another.
RESEARCH AIMS, OBJECTIVES AND OUT-
COMES
Accordingly, it is the aim of this research to combat
this issue of interoperability and introduce future-
proof practices into design workflows through the
development of a ML algorithm in the form of a Con-
volutional Neural Network (CNN) that recognises ar-
chitectural drawings and is thus able to automatically
classify and label elements within floor plans such
as individual room properties. It will be the role of
the CNN to be given a floor plan as a visual input
and output an instance segmentation function on
said image, where each room, is classified and the re-
gion it occupies is masked (i.e. the pixels inside the
region are labelled corresponding to the classifica-
tion). The CNN accomplishing this aim by automat-
ically producing universal BIM data capable of being
delineated across various design platforms. This re-
search additionally contributes knowledge towards
architectural understandings of ML, the data created
through this research can be extrapolated to assist fu-
ture machine generation of floor plans.
With this aim in mind the research has the follow-
ing objectives:
Despite the nearly ubiquitous presence of ML in
research fields and even our popular zeitgeist,
there continues to be an underwhelming rep-
resentation of ML systems within architectural
practices. This research’s objective is to con-
tribute to how ML can be used in the AEC sec-
tor using a pragmatist approach (Gardner et al
2020)
It appears theoretical developments produced
in research scenarios fail to be converted into
tools implemented in practical scenarios. The
result of this is somewhat of a frustrating situa-
tion, where there exists technology that can im-
prove not only the practices but the architecture
in which we inhabit, yet it lies dormant and un-
used. A further objec tiveof this research is using
the industry partnership and its applied context
as vehicle to identify how practice might use ML.
290 |eCAADe 38 - D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2
Considering this context, the outcomes based on the
aims and objectives of this research are twofold, to in-
vestigate how ML systems can be implemented into
architectural design workflows, and if ML systems can
develop some form of understanding of design to
draw nearer to a goal of machine designed architec-
ture.
Hence as an outcome this research details the de-
velopment of a CNN that can interpret architectural
drawings to classify and label the rooms of residen-
tial floor plan images. The CNN can inform the de-
velopment of an automated room labelling applica-
tion that can be implemented into architectural de-
sign workflows and contribute knowledge to ML un-
derstandings.
RESEARCH QUESTIONS
From these aims, objectives and outcomes the re-
search is provoked into exploring the following key
questions:
How can convolutional neuralnet works be utilised
within AEC design workflows to optimise or auto-
mate the recognition, classification and extraction
of architectural elements such as rooms?
How can machine learning algorithms synthesise
the elements, topologies, values, etc. of floor plans
to identify, distinguish and separate between the
spaces within?
To what extent can machine learning algorithms
understand and interpolate architectural design?
And to what extent of sophistication must a ML al-
gorithm have to generate design?
METHODOLOGY
This project adopts the design action research
methodology to inform and guide the course of work
undertaken. Action design research as proposed by
Maung K. Sein (2011), is an approach whereby the
generation of knowledge is motivated by the discov-
ery of problem situations in a specific organisational
setting and through an iterative cycle produces an
evaluated ‘IT’ artifact that addresses said problem.
Likewise, this research is motivated by present issues
observed within AEC design workflows (through the
research partnership with [NAME WITHHOLD] Archi-
tects, and through the construction of a ML algorithm
produces knowledge which will be used to inform an
‘IT’ artifact that will automate, and thus solve/allevi-
ate the issue. Following the structure of action design
research this research is divided into three separate
stages which operate in this manner: investigate, act,
observe.
In the first stage (problem formulation), archi-
tectural design workflows are observed and investi-
gated and after exploring ML a specific problem is
formulated and the strategy by which to solve it. In
the second stage (building intervention and evalu-
ation), the Neural Network (NN) is constructed and
the accuracy of its predictions and the robustness of
the system is assessed, since this is an iterative ap-
proach this step may repeated a number of times be-
fore a satisfactory result is attained. Finally, in the
third stage (reflection and learning), moves thinking
into a conceptual frame where the knowledge, ap-
plications and implications obtained from producing
the NN is ascertained.
BACKGROUND RESEARCH AND LITERA-
TURE REVIEW
ML is by no means a recent technology, examples
of applied ML can be traced back to Arthur Samuel’s
checker player for the IBM 701 in 1952 (Barto, Sut-
ton, 1992). However, today with access to unprece-
dented computational powers and open source tools
such as TensorFlow and PyTorch have enabled us to
openly explore ML outside of traditional computer
science fields. Mario Carpo (2017) predicts that the
effects of this computational shift will see design in-
formed by the mass retrieval of data and information,
an environment where ML technologies may thrive
and potentially create a new form of artificial intelli-
gence. Cer tain types of ML enables a program to per-
form given tasks without explicit instruction through
a process of training. Tom M. Mitchel of Carnegie
Mellon University (1997) states that “Aprogram is said
D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2 - eCAADe 38 |291
to learn from experience with respect to some class of
tasks, and performance measure, if its performance at
tasks, as measured by performance measure, improves
with experience.
A particular field of ML comes under the um-
brella of deep learning, commonly associated with
the implementation of artificial neural network (NN)
architectures. NNs probe mass quantities of data
to find statistical correlations, patterns, trends etc.
and extract this metadata to inform an output such
as some form of prediction, classification or sugges-
tion. As their name suggests NN draw loose inspira-
tion from biological models of neural networks like
the brain, a NN is a series of interconnected neurons
that receive, affect and send data [9]. The process
by which a NN learns is surprisingly brute force by
nature, using forward and backpropagation and NN
receives an input and is told to produce an output,
the data it is fed has labels associated with it (su-
pervised learning) that inform the NN of the desired
(correct) outputs. The NN iteratively receives a piece
of training data, produces and output, finds out its
wrong, fine tunes itself, repeats, and now is slightly
less wrong. The is a process referred to as gradient
decent and can be likened to climbing down a moun-
tain blindfolded, and after a long enough cycle the
NN has a series of values and hyperparameters that
consistently produce (mostly) correct outputs when
it receives a certain class of input data.
Of particular interest are prior mentioned Con-
volutional Neural Networks (CNN), they possess the
ability to extract patterns from spatial data types and
are thus frequently used in image recognition tasks.
With accurate training methods a CNN architecture
have achieved high accuracy rates in image classifica-
tion tasks (Hinton, et al, 2017). CNNs operate in two
stages, firstly they detect features by applying convo-
lutional operations over images on a pixel level, and
secondly, they classify features in later layers by de-
veloping understandings of detected features. But
what is the current state of play in the AEC industry?
There currently exists only a handful but grow-
ing number of research examples that explore the ca-
pacity of ML system to be applied to architectural en-
deavours, leaving many questions concerning the le-
gitimacy of ML in this field open. To provide a few
examples; Ng (2018) employed CNNs to distinguish
between sections and plans (Ng, 2018). Chaillou’s
(2019) AI + Architecture demonstrates ML’s ability
to classify, validate and generate architectural draw-
ings. Yoshimura, Cai, Wang & Ratti’s (2018) research
suggests that ML can comprehend design to the ex-
tent that researchers from MIT managed to differen-
tiate designs between architects using a NN [10].
Through the literature review in the above, spe-
cific research findings were discovered that helped
inform the course of this research. In one case Huang,
Zheng (2018) used a generative adversarial network
for the recognition and generation of floor plan im-
ages, and achieving interesting results. However, the
decision to generate a training dataset from a single
data source (that being marketing floor plans from a
Chinese website) resulted in an overfitted NN which
was unable to correctly classify non-orthogonal plans
or plans that did not pertain to the specific design
style the NN was trained on, thus rendering itself use-
less in all other situations. This example highlights
the importance of the data to determine the success
of ML applications.
Ferrando et al. (2019) investigated the CNNs
ability to distinguish between building typologies
of monasteries and mosques, training their CNN on
floor plans of these typologies. Again, it was the
process of data collection and pre-processing that
proved most crucial and laborious/time consuming.
This research proved to be quite insular in its final ap-
plication in practice as it concentrated on monaster-
ies and mosques whilst training of the CNN was, in
most respects successful.
Equipped with these insights, the research team
started its collaboration with [NAME WITH HOLD] Ar-
chitects with the aim to investigate ML technology
towards the architecturally oriented goal of recog-
nising architectural drawings and thus being able to
automatically classify and label elements within floor
plans such as individual room properties.
292 |eCAADe 38 - D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2
CASE STUDY
The existing research above demonstrates that ML
can be used to classify architectural topologies, as-
sess designs through a certain metric such as compli-
ance regulations, make predictions and assessments
on projects or even directly influence design deci-
sions. But this research project specifically aims to ex-
plore drawing recognition and classification of spa-
tial types in multi-scale residential floor plans to in-
form the development of an automated room la-
belling workflow for use on residential architecture
projects. The project was divided into four key stages
that involved ML processes research, data collection
and processing, script development, testing, reflec-
tion and the production of a ML floor plan recogni-
tion model.
Researching Machine Learning and existing
applications in the AEC industry.
The first stage of the project involved researching
machine learning and existing applications in the
AEC industry. In combination with the above listed
literature review these findings informed the appro-
priate ML methodology to adopt, the requirements
of this methodology and the feasibility to perform all
requirements to contribute new findings. This stage
necessitate the creation of a CNN, trains it on labelled
floor plan images, and generate a program that takes
its predications and produce classification instance
segmentation labels. When completed successfully
and all components work harmoniously with each
other one has produced a ML algorithm whose out-
puts will inform an automated room labelling work-
flow and ML understandings towards design.
Research specific to this stage concludes that
using the Python programming language with the
open source ML library TensorFlow and the Keras API
in the Visual Studio Code IDE is the optimal approach
to create the CNN. This step is based on existence
of the extensive documentation, learning resources,
compatible tools and community support provided
by users of these popular tools. Consequently it en-
ables a higher chance of successfully producing an
optimal CNN that will accurately perform its desired
task. The development of a CNN can be synthesised
into three main components: (a) assembling layers,
(b) compiling model and (c) fitting model. The pro-
cess of ML in CNNs has previously been delineated in
the background research section of this paper, there-
fore the paper will describe these three components
in detail.
Assembling layers defines the structure of the
CNN; the number of layers and the purpose they have
(whether they perform a convolutional or pooling
operation for instance), the number of neurons and
their activations (Sigmoid, Relu, Softmax, etc.) and,
what amount of potential outputs it can produce
(size of the output layer). Essentially, one defines how
data is fed into the CNN, interpreted and converted
into an output. Compiling the model defines the ML
metrics for the processes of learning in forward and
back propagation, the loss and optimiser functions
that are used to analyse the performance of the CNN
during training, and inform how the weights and bi-
ases are altered to improve output accuracy. Fitting
the model segments the dataset into training, and
validating and giving them to the CNN, the training
set is the largest (approximately 90% of the images)
and is used to inform the learning. The images are ac-
companied by labels which are the image’s desired
outputs if it were to be fed to the CNN. Within the
training cycle the CNN wants to produces outputs
that match these labels to the greatest degree of ac-
curacy possible. The validation dataset are used al-
most as a test; it withholds a small section (˜10%) of
the data to show after training to assess the model’s
performance and ensure that the CNN has not been
overfitted with the training data (when the CNN is too
closely trained on the specific examples in the train-
ing data and thus becomes useless in any practical
scenario).
D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2 - eCAADe 38 |293
Gathering floor plan images, labelling im-
ages and pre-processing images into a
dataset to feed the CNN
The second stage involved gathering floor plan im-
ages, labelling images and pre-processing images
into a dataset to feed the CNN to enable training to
commence. Typically, ML algorithms require mas-
sive quantities of varied data which posed a prob-
lem to this research in collecting enough to form a
substantial dataset to enable any meaningful ML de-
velopments. Initially it was thought that marketing
floor plans could easily be sourced from real-estate
companies, this proved false. Fortunately, [NAME
WITH HOLD] Architects provided a dataset of 454
floor plan images from their previous projects. These
consisted of CAD drawings, marketing images, and
hand-drawn floor plans from a variety of multi-scale
residential buildings. Despite, still the limited size of
this dataset, it was believed that quality of this data
would make up for this limitation, furthermore, the
feasibility of manually labelling 454 images within
the span of this research made it quickly apparent
that whether the dataset size was satisfactory or not
was of lesser importance to the feasibility of compil-
ing and processing data.
As argued earlier, ML is mostly about the data.
Consequently this stage of the project proofed to be
by far the most extensive and time consuming. This
is logical, ML is entirely dependent on the data it
is provided to learn, if there exists flaws within this
data the ML product is made redundant. The super-
vised learning approach employed by this research
required all the images collected to be accompanied
by a label in the form of a .JSON file that defined
the regions in which certain room classes were occu-
pying. To simplify this task and focus the efforts of
the ML algorithm, it was decided to make six room
classes for the CNN to search for, these being:
• Bedroom
Bathroom (includes ensuite)
Living room
Dining room (if connected to the living room,
counted as part of the living room)
• Kitchen
Balcony (includes terraces, patios, etc.)
The images were labelled using the VGG image anno-
tator (Dutta, Zisserman, 2019), this involved drawing
the regions that rooms occupied and defining said
region as whatever room it was. The output of this
process was a large .JSON file that contained every
label for all the images and would be referenced for
the training cycle.
Develop program that could perform in-
stance segmentation on test data
The third stage involved developing a program that
could perform instance segmentation on test data.
Instance segmentation is the classification of multi-
ple class regions within a single image and produce
a mask that defines the space the classes occupy
within the image. This task proved itself to be too
computational advanced for this research to build
from the ground up. Consequently the Mask R-CNN
framework developed by the Facebook AI Research
Group was employed [11]. Using the pre-defined
functions from Mask R-CNN and altering them to fit
our data and project’s desired outcomes within our
own scripts enabled us to create a CNN capable of
meeting the aims of this research. With a CNN, a
dataset and a framework for instance segmentation,
training commenced over a period of 1000 epochs.
Initial training cycles failed due to various memory
and iterative errors, after a process of debugging and
re-writing to more efficiently pass data (alleviating
the stresses on the system), a full training cycle was
completed. The finalised CNN model was then ex-
ported as a .h5 file containing the finalised hyper-
parameters to an inference script where testing was
conducted.
Testing and evaluation of the CNN
The fourth and final stage involved the testing and
evaluation of the CNN. With a completed CNN run-
ning instance segmentation on floor plan images,
new test images were inputted. Overall the results
were satisfactory and demonstrated that CNN was
294 |eCAADe 38 - D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2
beginning to develop a fundamental understanding
of design. The test images were purposely varied in
design typologies, visual style, and size, they even in-
cluded floor plans from projects outside of Australia
and plans where features such as furnishing were ab-
sent, through this the CNN was validated as tool of
wide capabilities. The CNN began with randomised
values and through the training cycle developed its
own methods to detect and interpret floor plans.
Through testing it is evident that a multitude of fac-
tors informs its decision-making processes these are:
(1) visual features including symbols, icons, line work,
patterns, etc and (2) the shape and form of elements
within the image and relational data, for instance the
layout of floors and general design principals that
dictate how we design the spaces we inhabit. The
ability for the CNN to rely on multiple features is good
because it means it makes it a more robust system
capable of dealing with more floor plans that may be
unique in their features and styles. However, the CNN
is far from perfect, it is evident that from viewing its
outputs that there is yet to a be a single ‘perfect’ out-
put where rooms are labelled exactly and correctly,
these issues will be further discussed in the next sec-
tion of this paper.
DISCUSSION, EVALUATION AND SIGNIFI-
CANCE
The aim of this research was to develop a ML algo-
rithm that could understand designs of floor plans
and subsequently be used to inform the creation of
an automated room labelling application. This re-
search documented the development of the CNN for
this purpose, it has shown an initial success in reading
and labelling architectural drawings, that with fur-
ther refinement and training with additional data will
be a sophisticated, reliable tool. Hence the research
was able to demonstrate how ML systems and think-
ing may be applied and integrated into architectural
design workflows to optimise practices within design
workflows.
CNNs do possess the ability to understand and
interpret design within an architectural framework
if directed and trained correctly. A CNN possesses
an elementary ability to classify rooms and an ad-
vanced ability to distinguish between spaces. Due
to only being able to train the CNN with a dataset
of 454 floor plan images we believe that the results
with additional training data sets will improve. The
CNN initially randomises the variables by which it de-
tects features and only through trial and error dur-
ing training discovers what is of significance to com-
plete its task successfully. From analysing several of
the CNNs outputs it is evident that a number of fac-
tors informs its predictions. These factors include
features such as the addition of text, symbols and
icons (e.g. the word bedroom and a bed symbol), line
work such as room boundaries (walls) and patterns
(floors), and most interesting to this research, rela-
tional data. It is possible that the CNN is unintention-
ally learning design rules and compliances through
its brute force training approach to learning, such as
room sizes (e.g. the largest room is the living room),
room layout (e.g. the balcony is separated from the
rest of the floor plan, often accessible only through
another room) and room shapes (e.g. the bedrooms
often have wardrobes which extrude out from their
otherwise thick rectangular typology).
To elaborate on this point further. In an exam-
ple for instance, the CNN may find that bathrooms
typically contain a number of visually distinct graph-
ics (e.g. the symbol for a toilet, shower, bath and
sink) within a narrow much smaller space in rela-
tion to the rest of the floor plan, confined by thick
lines and chequered pattern within the bounds of
said lines and contextually close to bedroom regions.
So, when the CNN is viewing an image containing a
bathroom/s it may rely on some or all these things
and potentially more to formulate a bathroom pre-
diction. There are, however, aspects that continue
to confound the CNN’s prediction making process,
larger images with too many class instances appear
to overwhelm the program, this however may just be
a limitation of the hardware (standard university lap-
D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2 - eCAADe 38 |295
Figure 1
Various Room
Labelling Results
top - Microsoft Surface Pro) available to this research.
More serious issues pertain to aspects of classifying
and distinguishing. Popular trends in modern design
facilitate open space living, thus blurring the sepa-
ration between the kitchen, living room, and dining
room. In many tests cases the CNN’s distinction be-
tween between rooms connected in open space liv-
ing were unsatisfactory, and often displayed an un-
certainty. Rooms that are visually similar on floor
plans such as a laundry and bathroom occasionally
result in incorrect classifications. With more exten-
sive training these nuanced problems can be over-
come. Furthermore, it is clear that these problems are
not fundamental issues within CNN’s but are a result
of research limitations.
A further point of contention comes from the
selection of training data. Here the issue of data
size (454 floor plan images) is an obvious one, ide-
ally the size of the dataset would have been in the
thousands. Despite this limitation, this research has
already proven that within this limited dataset a
CNN can understand design. Consequently a larger
dataset would simply make it more sophisticated.
However, a more important, less visible limitation
was data variety availability, this research used ex-
clusively floor plans that [WITH HOLD] Architects de-
signed. This is significant because it may mean that
the CNN is learning the biases and design prefer-
ences of [WITH HOLD] Architects, resulting in a layer
of subjectivity in a tool that is desired to be objec-
tive. Another fear sparked by this revelation is that
of overfitting the CNN, where the data is too special-
ized to a specific context, that being [WITH HOLD]
Architects’s work that the CNN fails to be a universal
tool that other firms can benefit from. It is unclear
whether simply brute forcing this issue with more
296 |eCAADe 38 - D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2
training data will overcome these concerns and is a
more philosophical question that will be investigated
with further inquiry beyond this research.
CONCLUSION
Currently we stand at a crucial time for architec-
ture (Deutsch, 2019) [12] [13], the decisions made
by firms now, will dictate their prosperity for the
next decade. It is clear that it is a necessity for
the AEC sector to adapt and open up to emerging
technologies. Proponents of technology have iden-
tified a lack of automation and adoption of technol-
ogy as the primary reasons for the AEC industry’s
poor performance (Chapman 2005; Deutsch 2019;
Khean et al. 2019). We argue amongst others that
the AEC industry needs reform if it is to sustain and
grow in the new economy (Brynjolfsson et al. 2014;
Susskind, Susskind 2015; McAfee 2017; Parker et al.
2016, van Rijmenam 2019). Automation technolo-
gies have the potential to help AEC firms sustain or
even grow as they experience inevitable digital trans-
formation. That is why investing in further ML inves-
tigations and applications in architecture is of signif-
icance, the knowledge collated in this research has
proved that even with a several limitations a CNN can
be produced to optimise, automate and improve sev-
eral processes that previously hindered design work-
flows.
This research is definitely preliminary, address-
ing the statement made in the introduction alluding
to floor plans designed by machines and the tech-
nology produced by this research is not capable of
such a feet, the level to which the CNN understands
design is yet to surpass or even attain to a human’s
cognitive ability. In the spectrum of design oriented
applications, however, initial applications where the
ML algorithms work in tandem with architects to sug-
gest and inform design decisions are within reach.
The applications that can be produced as a direct
consequence are quite staggering, the results of this
research possess a multitude of direct and indirect
implications. Directly, we know that an application
where floor plans are read and automatically labelled
can be produced after another, more in depth train-
ing cycle and indirectly we can see that ML opens the
door to the concept of automatically generated, uni-
versal data, where this thinking can be applied to a
plethora of other scenarios.
With further developments of ML in architec-
ture, and if we divulge more of our data and gen-
eral architectural information to ML algorithms, then
the concept of a Synthetic Design method com-
bining ML with computational design for an opti-
mised design workflow consumable for humans via
a web-browser in and for Architecture, Engineering
and Construction (AEC) disciplines becomes feasible
(Haeusler, 2019). The effects of this kind of computa-
tion shift will be staggering, ML will divulge informa-
tion concerning design that humans are cognitively
incapable of perceiving, informing better, optimised
and automated design practices. The consequence
of this computational shift will ultimately augment
the perceptions we as designers have towards our
built environment and the means by which we con-
ceptualise, design and create it.
ACKNOWLEDGEMENT
The authors would like to acknowledge the contribu-
tion of [NAME WITH HOLD] Architects for providing
us with the research challenge, the data sets and an
ongoing conversation about improving the research
during the duration of this project. The project is part
of the ongoing research in AI Enabled Workflow Au-
tomation for the AEC Industry an [NAME WITH HOLD]
research group to grow and improve competitive-
ness in the Australian AEC industry by accelerating
AI-enabled workflow automation.
REFERENCES
Barto, A and Sutton, R 1992, Reinforcement Learning: An
Introduction, The MIT Press, London
Brynjolfson, E and McAfee, A 2014, The second machine
age: Work, progress, and prosperity in a time of bril-
liant technologies, W. W. Nor ton&Company
Carpo, M 2017, The Second Digital Turn: Design Beyond In-
telligence, The MIT Press, London
D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2 - eCAADe 38 |297
Chaillou, S 2019, AI + Architecture | Towards a New Ap-
proach, Master’s Thesis, Harvard University
Chapman, R 2005 ’Inadequate Interoperability: A Closer
Look at the Costs’, 22nd International Symposium on
Automation and Robotics in Construction ISARC 2005
- September 11-14, Ferrara, Italy, pp. 1-6
Deutsch, R 2019, Superusers: Design Technology Special-
ists and the Future of Practice, Routledge
Dutta, A and Zisserman, A 2019 ’The VIA Annotation
Software for Images, Audio and Video’, 27th ACM In-
ternational Conference on Multimedia, Nice, France,
pp. 2276-2279
Ferrando, C, Dalmasso, N, Mai, J and Llach, D 2019 ’Archi-
tectural Distant Reading, Using Machine Learning to
Identify Typological Traits Across Multiple Building’,
18th International Conference, CAAD Futures 2019,
Proceedings, Daejeon, Korea, pp. 114-127
Haeusler, M H 2019, ’Theory (Methods) and Design in the
Second Machine Age’, in Gardner, N, Haeusler, M H
and Zavoleas, Y (eds) 2019, Computational Design:
From Promise to Practice, av edition , Ludwigsburg,
Germany, pp. 56-67
Hinton, G, Krizhevsky, A and Sutskever, I 2012, ’ImageNet
Classification with Deep Convolutional Neural Net-
works’, Advances in neural information processing sys-
tems, 25(2), pp. 1-9
Huang, W and Zheng, H 2018 ’Architectural Draw-
ings Recognition and Generation through Machine
Learning’, Proceedings of ACADIA 2018, Mexico City,
Mexico, pp. 156-165
Khean, N, Gerber, D, Fabbri, A and Haeusler, M H 2019
’Examining Potential Socio-economic Factors that
Affect Machine Learning Research in the AEC In-
dustry’, 18th International Conference, CAAD Futures
2019, Proceedings, Daejeon, Korea, p. 254
Khean, N, Kim, L, Martinez, J, Doherty, B, Fabbri, A, Gard-
ner, N and Haeusler, M H 2018 ’The Introspection
of Deep Neural Networks - Towards Illuminating the
Black Box’, Proceedings of CAADRIA 2018, Beijing, pp.
237-246
Knodel, J and Naab, M 2016, ’How to Perform the Archi-
tecture Compliance Check (ACC)?’, in The Fraunhofer
IESE Series on Software, and Systems Engineering
(eds) 2016, Pragmatic Evaluation of Software Archi-
tectures., Springer, pp. 83-94
McAfee, A and Brynolfsson, E 2017, Machine, Platform,
Crowd: Harnessing Our Digital Future’;, W. W. Nor-
ton&Company
Mitchell, T M 1997, Machine Learning, McGraw Hill
Ng, J M Y, Khean, N, Madden, D, Fabbri, A, Gardner,
N, Haeusler, M H and Zavoleas, Y 2019 ’Optimising
Image Classification - Implementation of Convolu-
tional Neural Network Algorithms to Distinguish Be-
tween Plans and Sections within the Architectural,
Engineering and Construction (AEC) Industry’, Intel-
ligent & Informed - Proceedings of the 24th CAADRIA
Conference - Volume2,, Wellington, New Zealand, pp.
795-804
Parker, G G, Van Alstyne, M W and Sangeet, P 2016, Plat-
form Revolution: How Networked Markets Are Trans-
forming the Economy - and How to Make Them Work
for You, Harvard Business Review
van Rijmenam, M 2019, The Organisation of Tomorrow –
How AI, blockchain and analytics turn your business
into a data organisation, Routledge
Susskind, R and Susskind, D 2016, The Future of the Pro-
fessions – How technology will transform the work of
human experts, Oxford University Press
[1] https://deepmind.com/blog/article/alphago-zero-st
arting-scratch
[2] https://www.adweek.com/brand-marketing/inside-
next-rembrandt-how-jwt-got-computer-paint-old-mast
er-172257/
[3] https://www.mckinsey.com/industries/capital-proje
cts-and-infrastructure/our-insights/decoding-digital-tra
nsformation-in-construction
[4] https://nvlpubs.nist.gov/nistpubs/gcr/2004/NIST.GC
R.04-867.pdf
[5] https://www.mckinsey.com/industries/capital-proje
cts-and-infrastructure/our-insights/reinventing-constru
ction-through-a-productivity-revolution
[6] https://www.infrastructureaustralia.gov.au/publicat
ions/australian-infrastructure-audit-2019
[7] https://population.gov.au/
[8] https://www.buildsoft.com.au/blog/10-statistics-def
ining-the-australian-construction-industry
[9] https://stevenmiller888.github.io/mind-how-to-buil
d-a-neural-network/
[10] https://www.researchgate.net/publication/329441
652_Deep_Learning_Architect_Classification_for_Archi
tectural_Design_through_the_Eye_of_Artificial_Intellig
ence/stats
[11] https://arxiv.org/pdf/1703.06870.pdf
[12] https://www.danieldavis.com/future-of-the-profes
sions/
[13] https://www.weforum.org/agenda/2018/06/constr
uction-industry-future-scenarios-labour-technology/
298 |eCAADe 38 - D2.T8.S1. THE COGNITIVE CITY (AI) - Volume 2
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Conference Paper
With the development of information technology, the ideas of programming and mass calculating were introduced into the design field, resulting in the upcoming of computer-aided design. With the idea of designing by data, we began to manipulate data directly, and interpret data through design works. Machine Learning, as a decision making tool, has been widely used in many fields. It can be used to analyze large amount of data, and predict the future changes. Generative Adversarial Network (GAN) is a model frame in machine learning. It’s specially designed to learn and generate output data with similar or identical characteristics. Pix2pixHD is a modified version of GAN that learns image data in pairs and generate new images based on the input. The author applied pix2pixHD in recognizing and generating architectural drawings, marking rooms with different colors and then generating apartment plans through two convolutional neural networks. Next, in order to understand how these networks work, the author analyzed the frame of them, and provide an explanation of the three working principles of the networks, convolution layer, residual network layer and deconvolution layer. Lastly, in order to visualize the networks in architectural drawings, the author derived data from different layer and different training epochs, and visualized as gray scale images. It was found that the features of the architectural plan drawings have been gradually learned and stored as parameters in the networks. As the networks get deeper and the training epoch increases, the features in the graph become more concise and clearer. This phenomenon may be inspiring in understanding the designing behavior of human.
Conference Paper
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called dropout that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry
Chapter
The main goal of the Architecture Compliance Check (ACC) is to check whether the implementation is consistent with the architecture as intended: only then do the architectural solutions provide any value. Nevertheless, implementation often drifts away from the intended architecture and in particular from the one that was documented. We will show typical architectural solutions that are well suited to being checked for compliance. Compliance checking has to deal with large amounts of code and thus benefits from automation with tools. Not all violations of architecture concepts have the same weight: we provide guidance for the interpretation of compliance checking results.
The second machine age: Work, progress, and prosperity in a time of brilliant technologies
  • Barto
  • Sutton
  • London Brynjolfson
  • Mcafee
Barto, A and Sutton, R 1992, Reinforcement Learning: An Introduction, The MIT Press, London Brynjolfson, E and McAfee, A 2014, The second machine age: Work, progress, and prosperity in a time of brilliant technologies, W. W. Norton&Company Carpo, M 2017, The Second Digital Turn: Design Beyond Intelligence, The MIT Press, London
Inadequate Interoperability: A Closer Look at the Costs
  • S Chaillou
Chaillou, S 2019, AI + Architecture | Towards a New Approach, Master's Thesis, Harvard University Chapman, R 2005 'Inadequate Interoperability: A Closer Look at the Costs', 22nd International Symposium on Automation and Robotics in Construction ISARC 2005 -September 11-14, Ferrara, Italy, pp. 1-6
Superusers: Design Technology Specialists and the Future of Practice, Routledge Dutta, A and Zisserman, A 2019 'The VIA Annotation Software for Images
  • R Deutsch
Deutsch, R 2019, Superusers: Design Technology Specialists and the Future of Practice, Routledge Dutta, A and Zisserman, A 2019 'The VIA Annotation Software for Images, Audio and Video', 27th ACM International Conference on Multimedia, Nice, France, pp. 2276-2279
Architectural Distant Reading, Using Machine Learning to Identify Typological Traits Across Multiple Building
  • C Ferrando
  • N Dalmasso
  • Mai
  • D Llach
Ferrando, C, Dalmasso, N, Mai, J and Llach, D 2019 'Architectural Distant Reading, Using Machine Learning to Identify Typological Traits Across Multiple Building', 18th International Conference, CAAD Futures 2019, Proceedings, Daejeon, Korea, pp. 114-127
Theory (Methods) and Design in the Second Machine Age
  • M Haeusler
Haeusler, M H 2019, 'Theory (Methods) and Design in the Second Machine Age', in Gardner, N, Haeusler, M H and Zavoleas, Y (eds) 2019, Computational Design: From Promise to Practice, av edition, Ludwigsburg, Germany, pp. 56-67
Examining Potential Socio-economic Factors that Affect Machine Learning Research in the AEC Industry
  • N Khean
  • D Gerber
  • Fabbri
  • M Haeusler
Khean, N, Gerber, D, Fabbri, A and Haeusler, M H 2019 'Examining Potential Socio-economic Factors that Affect Machine Learning Research in the AEC Industry', 18th International Conference, CAAD Futures 2019, Proceedings, Daejeon, Korea, p. 254
The Introspection of Deep Neural Networks -Towards Illuminating the Black Box
  • N Khean
  • L Kim
  • J Martinez
  • B Doherty
  • A Fabbri
  • Gardner
  • M Haeusler
Khean, N, Kim, L, Martinez, J, Doherty, B, Fabbri, A, Gardner, N and Haeusler, M H 2018 'The Introspection of Deep Neural Networks -Towards Illuminating the Black Box', Proceedings of CAADRIA 2018, Beijing, pp. 237-246