Conference PaperPDF Available

Drawing with Bots: Human-computer Collaborative Drawing Experiments

Authors:
Hao Zheng at City University of Hong Kong
Hao Zheng
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

When drawing architectural images like plan drawings, designers should always define every details, so the images can contain enough information to support a design. However, the core information in a complex design is usually very simple. For example, in a plan drawing of an office building, what we really care is the boundary and function of each room, rather than the exact location of furniture should be. So here, we propose a method to help designers automatically generate the predicted details of architectural drawings, based on Conditional Generative Adversarial Network (CGAN). Through machine learning of hundreds of image pairs, the learning program will build a model to find out the connections between two given images, then the evaluation program will generate an output image according to the new input image we provide. Four experiments about generating architectural plan images and city satellite images will be introduced in this article. The final goal for this method is to assist designers to simplify drawing process, and eventually to replace designers to self-draw architectural and city images.
Figures - uploaded by Hao Zheng
Author content
All figure content in this area was uploaded by Hao Zheng
Content may be subject to copyright.
Content uploaded by Hao Zheng
Author content
All content in this area was uploaded by Hao Zheng on Jan 28, 2021
Content may be subject to copyright.
W. Huang, M. Williams, D. Luo, Y. Wu and Y. Lin (eds.), Learning, Prototyping and Adapting, Short
Paper Proceedings of the 23rd International Conference on Computer-Aided Architectural Design Research
in Asia (CAADRIA) 2018. © 2018, The Association for Computer-Aided Architectural Design Research in
Asia (CAADRIA), Hong Kong.
DRAWING WITH BOTS: HUMAN-COMPUTER COL-
LABORATIVE DRAWING EXPERIMENTS
HAO ZHENG
University of California, Berkeley, USA
pt990platina@berkeley.edu
Abstract. When drawing architectural images like plan drawings, de-
signers should always define every details, so the images can contain
enough information to support a design. However, the core information
in a complex design is usually very simple. For example, in a plan draw-
ing of an office building, what we really care is the boundary and func-
tion of each room, rather than the exact location of furniture should be.
So here, we propose a method to help designers automatically generate
the predicted details of architectural drawings, based on Conditional
Generative Adversarial Network (CGAN). Through machine learning
of hundreds of image pairs, the learning program will build a model to
find out the connections between two given images, then the evaluation
program will generate an output image according to the new input im-
age we provide. Four experiments about generating architectural plan
images and city satellite images will be introduced in this article. The
final goal for this method is to assist designers to simplify drawing pro-
cess, and eventually to replace designers to self-draw architectural and
city images.
Keywords. machine learning; image generating; self-drawing; archi-
tectural drawing; CGAN.
1. Introduction
1.1. BACKGROUND AND PROBLEM DESCRIPTION
When drawing architectural images like plan drawings, designers should al-
ways define every details, so the images can contain enough information to
support a design. However, the core information in a complex design is usu-
ally very simple. For example, in a plan drawing of an office building, what
128 H. ZHENG
we really care is the boundary and function of each room, rather than the exact
location of furniture should be.
However, recently, machine learning, as a decision making tool, is being
developed in many fields. It can be used to analyse large amount of data, and
predict the future changes.
So, this article discusses one of the machine learning algorithms, Condi-
tional Generative Adversarial Network (CGAN), helping designers simplify
drawing process at the beginning stage of a design.
1.2. CGAN INTRODUCTION
Goodfellow et al. (2014) proposed a new framework (GAN) in machine learn-
ing, and trained model G for generating data and model D for estimating the
possibility that the output data comes from original training data. To maximize
this possibility, the program trained model G, and finally G reached a situation
that, D can’t distinguish the data generated by G from the original data.
Soon, Mirza and Osindero (2014) refined the framework, and proposed
Conditional Generative Adversarial Network (CGAN), which introduced ex-
tra information y as additional input layer. The objective of a CGAN can be
expressed as
LcGAN(G,D) = Ex,y[log D(x,y)] + Ex,z[log(1-D(x,G(x,z))] (1)
Recently, Isola et al. (2016) applied CGAN to image-to-image translation,
proposed an efficient method to build connection between image pairs and
predict possible corresponding output image of a given input image (Figure
1).
Figure 1. Example results by Isola et al. (2016).
So, based on the work done by Isola et al. (2016), we propose a new appli-
cation of CGAN in generating architectural drawings and city satellite images
in this article.
2. Generating images of architectural plan drawings
The first thing we tried is to build a connection model between architectural
plan drawings and its core information.
DRAWING WITH BOTS: HUMAN-COMPUTER COLLABORATIVE DRAWING
EXPERIMENTS 129
2.1. TRAINING BY BOUNDARY
We got 800 floor plans from the Columbia University database, which were
student assignments or exhibition works at the Department of Architecture in
the 1960s. The corresponding input image is the boundary of each plan (Figure
2).
Figure 2. Training data – plan drawing (left) and boundary block (right) [Data from Artstor
collection (http://www.artstor.org/)].
Then we generated random block area images as input images to test the
performance of this connection model (Figure 3). It recognizes the outer walls
perfectly but the inner area still seems blurry.
Figure 3. Output data – predicted architectural plan drawing (left) and input boundary block
image (right).
2.2. TRAINING BY FUNCTION
However, boundary information is not enough to record a plan drawing. So
we used different colours to represent different areas as input images (Figure
4). Here, Black represents the indoor boundary, yellow represents the interior
area, cyan represents the furniture area, and red represents the architectural
axis.
130 H. ZHENG
Figure 4. Training data – original plan drawing (left) and corresponding coloured image
(right).
Then we drew coloured images of new plan drawings, and used the con-
nection model we built to predict the architectural plans (Figure 5). As the
figure shows, with more information, the new connection model can make
more detailed prediction. This model performs better than the former model.
So by giving simplified images, designers can easily generate complex archi-
tectural plan drawings.
Figure 5. Output data – coloured image (left), predicted image (middle), and original draw-
ing (right).
3. Generating images of city satellite photos
Next, we tried to use same algorithm to test its performance with larger data.
While architectural plan drawings are very hard to collect, city data, such as
satellite images and street view images, are easier to grab from the Internet.
3.1. TRAINING BY BOUNDARY
So first, we grabbed the vector information and satellite photos in a city of
Florida from Google Map. A program loaded the data, and found out all closed
curves and their corresponding satellite images. After cleaning useless infor-
mation, the training set contains only boundary blocks and satellite images
(Figure 6).
DRAWING WITH BOTS: HUMAN-COMPUTER COLLABORATIVE DRAWING
EXPERIMENTS 131
Figure 6. Training data – satellite image (left) and city block boundary (right).
We then gave the program boundary images in another city, and asked for
a prediction image (Figure 7). It shows that the connection model well pre-
dicted the common features of city in Florida, and it’s totally different from
those in Barcelona City.
Figure 7. Output data – boundary image (left), predicted image for Florida (middle), and
original image in Barcelona City (right).
3.2. TRAINING BY FUNCTION
Next, we added further information about a city to the input images. We
distinguished different blocks by their characteristics with different colours
(Figure 8). Here, red means buildings, green means green lands, blue means
water or pool, and white means others.
Figure 8. Training data – satellite image (left) and city information image (right).
Based on the training data from three cities, Barcelona, Manhattan, and
London, we built three models to predict the most possible satellite images of
a given coloured map, which was drawn by hand (Figure 9). So also, by giving
sketches, designers can get the predicted satellite photos as a preview of city
images.
132 H. ZHENG
Figure 9. Output data – sketch image (left), predicted image of Barcelona (middle left), pre-
dicted image of Manhattan (middle right), and predicted image of London (right).
4. Conclusion
As described above, Conditional Generative Adversarial Network (CGAN)
performs well in predicting architectural images. Designers can cooperate
with computer programs, by providing this system with simplified infor-
mation or sketches, get complex drawings. It liberates designers from heavy
drawing work at the beginning step of designing, and has potential to become
a more powerful tool, not only to support designers, but also to change the
design process.
Acknowledgements
I'd like to show my gratitude to Prof. Kyle Steinfeld from UC Berkeley, who supervised me in
this research, and Prof. Weixin Huang from Tsinghua University, who gave me a lot of advice.
References
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio,
Y. (2014). Generative adversarial nets. In Advances in neural information processing sys-
tems (pp. 2672-2680).
Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1784.
Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2016). Image-to-image translation with condi-
tional adversarial networks. arXiv preprint arXiv:1611.07004.
... Early research mainly focused on indoor design in architecture. Huang [7] and Zheng [8] explored the layout of single-story indoor space floor plans by generating design plan from design layout (colour block diagrams) using pix2pix, establishing the primary research paradigm in this field and laying the foundation for subsequent research. Subsequently, Yang [9] and others initially expanded upon this research paradigm by dividing the generation process into two steps: 'site conditions-functional zones-design schemes', to construct a framework for the automatic generation of design schemes for youth apartments. ...
... Early research focused on interior design [7][8][9][10][11], and in recent years there has been a gradual emergence of automatic generation design for outdoor spaces, but mainly focuses on functional areas with regular layout, such as residential areas [14]. To be more specific, Liu et al. [17] studied generative design, focusing on the selection of research objects with strong regularity and enclosed space in Jiangnan private gardens. ...
Generative Design of Outdoor Green Spaces Based on Generative Adversarial Networks
Article
Full-text available
  • Apr 2023
Generative Adversarial Networks (GANs) possess a significant ability to generate novel images that adhere to specific guidelines across multiple domains. GAN-assisted generative design is a design method that can automatically generate design schemes without the constraints of human conditions. However, more research on complex objects with weak regularity, such as parks, is required. In this study, parks were selected as the research object, and we conducted our experiment as follows: (1) data preparation and collection; (2) pre-train the two neural network, then create the design layout generation system and the design plan generation system; (3) realize the data augmentation and enhanced hundred level dataset to thousand level dataset; (4) optimized training; (5) test the optimized training model. Experimental results show that (1) the machine learning model can acquire specific park layout patterns, quickly generate well-laid-out plan layout plans, and create innovative designs that differ from the human designer’s style within reasonable limits; (2) GAN-driven data augmentation methods can significantly improve the generative ability of algorithms, reduce generative pressure, and achieve better generative results; (3) pix2pix is prone to mode collapse, and CycleGAN has fixed rule errors in expressing certain design elements; and (4) GAN has the ability to mine design rules in the same way as humans.
View
... The results show that the method can effectively support the decision-making process of urban graphic scheme designs. Huang W et al. used the cGAN model to effectively control the results to generate satellite images, hand-drawn architectural sketches, and architectural plan function zoning maps based on the boundary and functional training [32]. Shen et al. extracted urban information and simplified it into a floor plan and used cGAN to generate architectural layouts based on site conditions to achieve a rapid urban design method [33]. ...
M-StruGAN: An Automatic 2D-Plan Generation System under Mixed Structural Constraints for Homestays
Article
Full-text available
  • Apr 2023
Existing methods for generating 2D plans based on intelligent systems usually require human-defined rules, and their operations are complex. GANs can solve these problems through independent research and learning. However, they only have generative design research based on a single constraint condition, and whether they can generate a qualified design scheme under many constraints is still unclear. Therefore, this paper develops the M-StruGAN generative model based on the structural design framework of a GAN. Its application research is extended to the 2D-plan layout generation of homestay based on the constraints of hybrid structures, and the feasibility of the method is comprehensively verified through three aspects: image synthesis quality assessment, scheme rationality assessment, and scheme design quality assessment. Experimental results show that the quality of the drawings generated by M-StruGAN is qualified, designers have a high degree of acceptance of the design results of M-StruGAN, and M-StruGAN completed the learning of the critical points of the 2D layout. Finally, through the human–computer interaction application of M-StruGAN, it can be found that compared with traditional design methods, M-StruGAN based on pix2pixHD has high-definition image quality, higher design efficiency, lower design cost, and more stable design quality.
View
... However, precedent studies that use pix2pix models for the generation of floor plans also use different evaluation criteria; for example, Peters (2018) and Zheng (2018) compared the results of models trained on datasets with different graphical representations, such as labels (colored diagrams) or linear axis (just wall lines). Rahbar et al. (2019) used questionnaires to compare human-designed plans to machine-designed plans using topological and geometrical criteria such as orthogonal design, space dimensions, entrance recognition, and space logics. ...
Mapping Space Allocation with Artificial Intelligence - An approach towards mass customized housing units
Conference Paper
Full-text available
  • Sep 2022
Artificial Intelligence represents a substantial part of the available tools on architectural design, especially for Space Layout Planning (SLP). At the same time, the challenge of Mass Customization (MC) is to increase the product variety while maintaining a good cost-benefit ratio. Thus, this research aims to identify new, valid, and easily understandable data patterns through human-machine interaction in an attempt to deal with the challenges of MC during the early phases of SLP. The Design Science Research method was adopted to develop a digital artifact based on deep generative models and a reverse image search engine. The results indicate that the artifact can deliver a series of design alternatives and enhance the navigation process in the solution space, besides giving key insights on dataset design for further research.
View
... Therefore, after five rounds of iteration, we obtained a waterfront map with clear street networks and regularly distributed land-use types (Figure 8 middle). We further implemented another image-to-image machine learning model (Zheng, 2018, Huang andZheng, 2018) to generate the satellite image from the colourcoded map, the output image (Figure 8 right) shows a better visualization of the generated result. To conclude, the research explores the potentiality of GAN to learn and apply the morphological rules of industrial waterfront. ...
Morphological Regeneration of the Industrial Waterfront Based on Machine Learning
Conference Paper
Full-text available
  • Apr 2022
The regeneration of the industrial waterfront is a global issue, and its significance lies in transforming the waterfront brownfield into an eco-friendly, hospitable, and vibrant urban space. However, the industrial waterfront naturally has comparatively unmanageable morphological features, including linear shape, irregular waterfront boundary, and separation with urban networks. Therefore, how to subdivide the vacant land and determine the land-use type for each subdivision becomes a challenging problem. Accordingly, this study proposes an application of machine learning models. It allows the generation of morphological elements of the vacant industrial waterfront by comparing the before-and-after scenarios of successful regeneration projects. The data collected from New York City is used as a showcase of this method.
View
... Hao Zheng is one of the early researchers in this scope. In 2018, he pplied a conditional GAN, Pix2Pix to prove building plan, urban plan, and satellite images of city could be generated by given conditional input, such as footprints or color pattern images [12]. In the following research, he successfully generated plausible apartment plans, and explained the working principles [13][14][15]. ...
Exploring in the Latent Space of Design: A Method of Plausible Building Facades Images Generation, Properties Control and Model Explanation Base on StyleGAN2
Chapter
Full-text available
  • Jan 2022
GAN has been widely applied in the research of architectural image generation. However, the quality and controllability of generated images, and the interpretability of model are still potential to be improved. In this paper, by implementing StyleGAN2 model, plausible building façade images could be generated without conditional input. In addition, by applying GANSpace to analysis the latent space, high-level properties could be controlled for both generated images and novel images outside of training set. At last, the generating and controlling process could be visualized with image embedding and PCA projection method, which could achieve unsupervised classification of generated images, and help to understand the correlation between the images and their latent vectors.
View
... The input parameters of the generator are typically a random vector used to promote variations on the synthesized data, but with the extensions of the model (Hindupur, 2017), conditional information or even user input can be used for additional control. Recently, GANs have been widely applied to architectural design to analyze and generate building representations, such as floor-plan layouts (Huang & Zheng, 2018;Zheng, 2018;Chaillou, 2019;Newton, 2019;Nauata et al., 2020). ...
View
... Huang and Zheng (2018) applied pix2pix to generate 2-D images of floor plans for a single-family house. Zheng (2018) also explored pix2pix to generate aerial images of cities from sketches. Chaillou (n.d.) used pix2pix in a nested fashion to create multiunit residential plans nested within site boundaries. ...
Generative Deep Learning in Architectural Design
Article
  • Jul 2019
Generative Adversarial Networks (GANs) are an emerging research area in deep learning that have demonstrated impressive abilities to synthesize designs, however, their application in architectural design has been limited. This research provides a survey of GAN technologies and contributes new knowledge on their application in select architectural design tasks involving the creation and analysis of 2D and 3D designs from specific architectural styles. Experimental results demonstrate how the curation of training data can be used to control the fidelity and diversity of generated designs. Techniques for working with small training sets are introduced and shown to improve the visual quality of synthesized designs. Lastly, experiments demonstrate how GANs might be used analytically to gain insight into specific architectural oeuvres.
View
Proceedings of the 2021 DigitalFUTURES The 3rd International Conference on Computational Design and Robotic Fabrication (CDRF 2021): The 3rd International Conference on Computational Design and Robotic Fabrication (CDRF 2021)
Book
Full-text available
  • Jan 2022
This open access book is a compilation of selected papers from 2021 DigitalFUTURES—The 3rd International Conference on Computational Design and Robotic Fabrication (CDRF 2021). The work focuses on novel techniques for computational design and robotic fabrication. The contents make valuable contributions to academic researchers, designers, and engineers in the industry. As well, readers encounter new ideas about understanding material intelligence in architecture.
View
A generative architectural and urban design method through artificial neural networks
Article
Full-text available
  • Jul 2021
  • BUILD ENVIRON
Machine learning, as a computational tool for finding mappings between the input and output data, has been widely used in engineering fields. Researchers have applied machine learning models to generate 2D drawings with pixels or 3D models with voxels, but the pixelization reduces the precision of the geometries. Therefore, in order to learn and generate 3D geometries as vectorized models with higher precision and faster computation speed, we develop a specific artificial neural network, learning and generating design features for the forms of buildings. A customized data structure with feature parameters is constructed, meeting the requirements of the neural network by rebuilding surfaces with controlling points and appending additional input neurons as quantified vectors to describe the properties of the design. The neural network is first trained with generated design data and then tested by adjusting the feature parameters. The prediction of the generated data shows the basic generative ability of the neural network. Furthermore, trained with design data collected from existing buildings, the neural network learns and infers the geometric design features of architectural design with different feature parameters, providing a data-driven method for designers to generate and analyze architectural forms.
View
View
Image-to-Image Translation with Conditional Adversarial Networks
Article
Full-text available
  • Nov 2016
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.
View
Generative Adversarial Nets
Article
  • Jun 2014
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
View
  • Jan 2014
  • M Mirza
  • S Osindero
Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784.