Conference PaperPDF Available

Exploring the Feature Space to Aid Learning in Design Space Exploration

Design Computing and Cognition DCC’18. J.S. Gero (ed),
pp. xx-yy. © Springer 2018
Exploring the Feature Space to Aid Learning in Design
Space Exploration
In this paper, we introduce the concept of exploring the feature space to aid
learning in the context of design space exploration (a.k.a. tradespace explo-
ration). The feature space is defined as a possible set of features (specific
values of design decisions or attributes), mapped in a 2D plane with each
axis representing different interestingness measures. Similar to how a de-
signer explores the design space, one can explore the feature space by ob-
serving how different features vary in their usefulness in explaining a group
of design solutions. This could aid in the designers better understanding of
the design space. To test the effectiveness of this process as a learning tool,
we conduct a controlled experiment with human subjects. The result shows
that feature space exploration has potential to enhance the designer’s ability
to identify important features and predict the performance of a design.
1. Introduction
Over the last two decades, “design by shopping” paradigm [1] has become
a popular approach to tackle early-phase (conceptual design or system ar-
chitecting) engineering design problems. An important step in this approach
is called design space exploration (a.k.a. tradespace exploration), where the
designer analyzes the structure of the design space and learns about the
trade-offs in the system, sensitivities of design criteria to design decisions,
couplings between design decisions, etc. For the remainder of this paper,
“learning” in tradespace exploration refers to gaining knowledge about these
parameters, and more generally about the mapping between design decisions
and design criteria. Through the process of design space exploration, the
designer can make a more informed decision for the selection of the final
However, design space exploration presents us with the challenge of in-
formation overload. The problem of information overload becomes more
prominent in design tasks involving many design decisions, multiple objec-
tives, and intricate couplings between them. It has been shown that as the
A.N. Author
design problem gets more complex, the designers are overwhelmed by the
size and the complexity of the data, thus leading to the degradation of their
ability to understand the relationships between different variables [2][4].
To address this issue, various data visualization methods and tools have
been developed [5][11]. These visualizations are useful, as they provide a
quick and intuitive sense of the structure of the design space. However, they
also have limitations, as the knowledge learned through visualizations can
be ambiguous.
Another complementary approach to learn about the design space is to
extract knowledge using data mining algorithms that mine knowledge ex-
plicitly in the form of logical if-then rules [12][15]. These methods can be
used to extract driving features, i.e., the common features (specific values
of design decisions, attributes, or combinations there-of) that are shared by
a group of designs that exhibit similar objective values [16]. For example,
Watanabe et al. use association rule mining to analyze hybrid rocket engine
designs, and find that 83% of all non-dominated (Pareto optimal) solutions
had a similar initial port radius [17]. The major advantage of such
knowledge is that it can be expressed relatively concisely and unambigu-
ously through a formal representation [18].
While having been used successfully in the past to analyze design spaces,
these methods are not without limitations. One of the limitations of the cur-
rent methods is that they impose a rigid structure in the mined features (the
conditional “if” parts of the rules); indeed, all features are represented as
predicates (i.e., binary features) joined by logical conjunctions (i.e., “and”
operator). From a mathematical point of view, this does not reduce expres-
sivity, as any logical formula can be converted into a Disjunctive Normal
Form or DNF (i.e., a disjunction OR of conjunctive clauses) [19]. There-
fore, any Boolean concept (a concept whose membership is determined by
a combination of binary features [20]) can be represented using a set of rules
(disjunction of rules). However, from a human learning point of view, the
conversion to DNF often results in longer features, and thus harder to un-
derstand by humans.
Another limitation of the data mining methods is that they generate a large
set of features without an easy way to identify the most useful and informa-
tive one [21]. One approach to select a single feature is to sort all features
using one measure such as confidence or lift [22]. Intuitively, these interest-
ingness measures provide a quantitative metric of the predictive power of
the feature. However, selecting a single metric from a large list of alterna-
tives can be arbitrary, and may not necessarily be the right measure for the
given design problem [23], [24].
Exploring the Feature Space to Aid Learning in Design Space Exploration 3
In this paper, we present a new method, feature space exploration, to aid
human learning in design space exploration and a tool to use over the
method. The aim of this method is to improve the designer’s ability to iden-
tify important features and use that information to predict the performance
of a design. In order to foster learning, we enable designers to explore vari-
ous forms of features and get immediate feedback on how well these features
explain a certain region of the design space (e.g., a cluster, or the Pareto
front). This is done by defining a space of all possible features (called the
feature space), visualized on a 2D plane. Each axis in the plane represents
one of the interestingness measures of features used in classification (e.g.
precision and recall [24]) or association analysis (e.g. confidence, lift, and
Gini index [23]). If one selects conflicting goodness measures such as pre-
cision and recall [25], the Pareto front of the feature space can also be de-
fined. The designer can then use the visualization to observe how the good-
ness measures change in response to a change in the feature, and elicit his
or her preferences among those two importance measures. Due to its simi-
larity to how a designer explores the design space, we refer to this process
as “exploring the feature space”. Exploring the feature space helps the de-
signer identify the driving features that shape the structure of the design
space, as it allows selecting or synthesizing a feature that best fits the pur-
pose of the current analysis. The process takes advantage of the intuitive and
fast nature of exploring options through visualization, as well as the ease of
learning through formal representations that are clear and concise.
To demonstrate the effectiveness of this new method to improve learning,
we conduct a controlled experiment with human subjects. The experiment
tests whether exploring the feature space improves the designer’s ability to
predict whether a given design will exhibit desirable performance and cost.
The result shows that exploring the feature space may indeed improve learn-
ing about the design space but only under certain conditions for subjects
who have received some formal training in design space exploration.
The remainder of the paper is organized as follows. Section 2 provides a
background and literature review on design space exploration methods. Sec-
tion 3 introduces the example design problem that is used in this paper. Sec-
tion 4 explains the proposed method of exploring the feature space. Section
5 covers the setup for a human-subject experiment. Section 6 shows the re-
sult from the experiment. Section 7 discusses the experiment result, and Sec-
tion 8 discusses the conclusions and future work.
A.N. Author
2. Related Work
Various knowledge discovery and decision support tools have been de-
veloped to support design space exploration for solving complex design
problems. Bandaru et al. classifies these methods based on whether the
knowledge is extracted in an implicit or in an explicit form [18]. Implicit
knowledge is defined as having no formal notation, thus making it difficult
to transfer information unambiguously.
The typical way of representing implicit knowledge is through visualiza-
tions. Various data visualization methods have been applied to design space
exploration, including cloud visualization [5], self-organizing maps [7], in-
teractive multiscale-nested clustering and aggregation (iMSNCA) [8], city-
plot [10], isomap [26], kernel PCA [27], Hyper-Space Diagonal Counting
(HSDC) [28], and Hyper-Radial Visualization (HRV) [29]. Most of these
tools focus on providing different views of designs defined in a multi-di-
mensional space, in some cases coupled with unsupervised machine learn-
ing methods such as clustering, feature selection, and manifold learning.
However, due to the knowledge being implicit in visualization, these meth-
ods require an additional step for the humans to visually inspect the result
and make interpretations. Therefore, the knowledge obtained through visu-
alization can be ambiguous and subjective. Moreover, visually inspecting
and finding patterns may be challenging without sophisticated re-arranging
strategies [30], [31].
There also exist knowledge discovery methods that generate knowledge
in an explicit form, i.e., using formal representations such as logical rules.
Data mining methods that extract knowledge in the form of logical if-then
rules include decision trees [32][34], Algorithm Quasi-optimal (AQ) learn-
ing [14], [35], and association rule mining [16], [17], [36]. The advantages
of extracting knowledge in the form of logical rules is that they are compact
and unambiguous. However, these rule mining methods also have limita-
tions. They can only extract rules with a rigid, pre-determined form such as
DNF. This imposes a significant limitation in extract knowledge in learnable
format. Another limitation often found in rule mining methods is that while
they generate a large number of alternative rules, it is not easy for the user
to grasp what are the differences in performances of those rules relative to
each other.
Exploring the Feature Space to Aid Learning in Design Space Exploration 5
3. Example Design Problem: Architecting Earth Ob-
serving Satellite System
Before explaining how the proposed method works, we first introduce an
example design problem to help explain the methodology in the remainder
of the paper. It should be noted that the proposed method is not specific to a
type of a design problem. However, there are some implementation details
that are tailored to the structure of this problem. This point will be elaborated
on after the design problem is outlined.
The design problem is a real-world system architecting problem previ-
ously studied in [37]. The goal of the design task is to architect a constella-
tion of satellites to provide operational observation of the Earth’s climate.
There are two objectives: maximizing the scientific benefit and minimizing
the lifecycle cost. Scientific benefit is a function of an architecture’s satis-
faction of 371 climate-related measurement objectives, generated based on
the World Meteorological Organization’s OSCAR (Observing Systems Ca-
pability Analysis and Review Tool) database1. The level of satisfaction of
each measurement objective is quantified based on the capabilities of each
design, and then aggregated to obtain a number that represents how much
scientific benefit each design brings to the climate scientific community.
The design problem has been formulated as an assignment problem be-
tween a set of candidate measurement instruments (remote sensors related
to climate monitoring) and a set of candidate orbits (defined by orbital pa-
rameters such as altitude and inclination). Given a set P of candidate instru-
ments and a set O of candidate orbits, the design space is defined as a set of
all binary relations from P to O. Each instrument in P can be assigned to any
subset of orbit O, including the empty set. Therefore, the size of the design
space is !" # , where $ is the number of candidate instruments and % is
the number of candidate orbits. In this work, we considered 12 candidate
instruments and 5 candidate orbits, making a total of !&' possible designs.
Each design is represented by a Boolean matrix M of size ()*!, where
+ ,- . / * if instrument p is assigned to orbit o, and + ,- . / 0 other-
wise. Graphically, this can be displayed by a figure similar to Fig. 1. Here,
each row represents a mission that will fly in each orbit, and the columns
represent the assignment of different instruments. Note that in the examples
that will follow throughout this paper, we replace the name of the actual
orbits and instruments with numbers (e.g., 1000, 2000) and alphabetical let-
ters (e.g., A, B, C) to simplify the presentation of the examples.
A.N. Author
Fig. 1 An example architecture representation. Each row represents a single spacecraft flying
in a certain orbit. For example, a spacecraft carrying Cloud and Precipitation Radar
(CPR_RAD) and UV/VIS Limb Spectrometer (CHEM_UVSPEC) will fly in a sun-synchro-
nous orbit at altitude of 800km, and the local time at the ascending node in the afternoon.
Once the design decisions and the corresponding objective values are pro-
vided in a structured format, the proposed method mostly considers the de-
sign problem as a black box. At the implementation level, however, there is
one critical step necessary in order to run data mining, which is formulating
the base features. The base features are predicates used to construct more
sophisticated Boolean concepts related to the design space. In its simplest
form, a base feature can be a single design decision set to 0 or 1. However,
we introduce more complex base features to pre-specify the structure of the
patterns to be searched, thus biasing the search towards more promising re-
gions. The base features used for the current system architecting problem
are shown in Table 1. The formulation of the base features requires some
domain-specific knowledge and insights obtained by observing the structure
of the design problem.
Table 1 Base features
Name of the
Instrument 12 is present in at least one of the orbits
Instrument 12 is absent in all the orbits
%2- 1
3- 415- 167
Instrument 1
3 (and 15, 16) is/are present in orbit %2
%2- 1
3- 415- 167
Instrument 1
3 (and 15, 16) is/are not present in orbit %2
12- 1
3- 4157
Instruments 12, 1
3 (and 15) are present together in any orbit
12- 1
3- 4157
Instruments 12, 1
3 (and 15) are not present together in any
single orbit
No instrument is present in orbit %2
The number of orbits that have at least one instrument as-
signed is n
For example, Present is a base feature that describes whether an instru-
ment 9 is used in at least one of the orbits. This feature is equivalent to a
Exploring the Feature Space to Aid Learning in Design Space Exploration 7
disjunction of 5 base features (instrument 9 being assigned to each one of
the orbits). We know that Present may potentially speed up the search, since
the decision whether to use an instrument or not has a bigger influence in
the objective value compared to the decision of which orbit it should be as-
signed to. In the remaining sections of this paper, we will use these pre-
defined set of base features to build more complex features.
4. Exploring the Feature Space
In this paper, we propose exploring the feature space as a learning aid for
design space exploration. We define the feature space as a set of all possible
features, visualized by mapping features in a coordinate system where each
axis represents a different measure of the goodness of a feature (e.g. preci-
sion, recall, F score, confidence, lift, and mutual information [23], [24]). In
the following sections, we introduce the graphical user interface that enables
visualizing and exploring the feature space, and explain how a designer can
use it for insight generation and learning.
4.1. iFEED
The capability to explore the feature space is built as an extension to the
interactive knowledge discovery tool called iFEED [16]. Its goal is to help
engineering designers learn interesting features that drive designs towards a
particular region of the objective space as they interact with the tool. A user
of iFEED can select a group of target designs, and run data mining algo-
rithms to extract the common features that are shared by those designs. The
main interface of iFEED is shown in Fig. 2. It consists of an interactive
scatter plot, which shows the design space populated by thousands of alter-
native designs. When the user hovers his or her mouse over one of the points
in the scatter plot, the information about that design is displayed below the
scatter plot. The displayed information includes values of the objectives and
design decisions of a design.
The scatter plot can help the user select a region of interest in the objective
space. When the user drags the mouse over the scatter plot, designs in the
selected region are highlighted, and they are considered as target solutions
when running the data mining process.
The data mining process is based on the Apriori algorithm, which is one
of the earliest and most popular algorithms developed for association rule
A.N. Author
mining [38]. The algorithm has been extended to mine classification associ-
ation rules, which follow the structure : ; <. Here, X is a feature that de-
scribes a design, and C is a class label that indicates whether a certain design
belongs to the target region or not. The data mining returns a list of features
that are shared by the target designs is returned. For more details on the data
mining algorithm, readers are referred to [16].
Fig. 2 The main graphical user interface of iFEED, which consists of a scatter plot showing
the objective space and a display of one design that is currently viewed. Dots highlighted in
cyan represent the target region selected by the user.
4.2. Visualization of Feature Space
The features extracted by running the data mining algorithm have varying
level of “goodnessin explaining the target designs. Such measures can be
defined using various metrics used in binary classification and association
rule mining [23], [24]. In this work, we use two measures of confidence de-
fined as follows.
=,8> ? ; @ / AB.. C D E
AB.. C
Exploring the Feature Space to Aid Learning in Design Space Exploration 9
=,8> @ ; ? / AB.. C D E
AB.. E
Here, S is the set of all designs that are in the target region, and F is the set
of all designs that have the particular feature that is being considered. supp
stands for support, which is defined as:
AB.. F / G :
where U is the set of all designs in the database and I indicates the cardi-
nality of the set. Confidence is often used in association rule mining to rep-
resent the strength of a rule [38]. =,8> ? ; @ represents how complete the
feature is in terms of the fraction of the target region that exhibits the feature,
while =,8> @ ; ? represents how consistent or specific the feature is in
explaining only the target region (fraction of designs with the feature that
are in the target region). In fact, because we extract only binary classification
rules, =,8> ? ; @ and =,8> @ ; ? are equivalent to recall and precision,
After we calculate both confidence measures for the extracted features,
we can map them in a two-dimensional plane with each axis representing
one of the confidence measures, as shown in Fig. 3. This visualizes the
feature space as we defined at the beginning of this section. In the figure,
each triangle is a feature obtained from the data mining algorithm. The gen-
eral trend in the mined features shows that there is a trade-off between the
two confidences, consistent with relationship often seen between recall and
The scatter plot displaying the feature space is also implemented as an
interactive plot. When the user hovers the mouse over a feature in Fig. 3,
the designs that have the feature are highlighted in the scatter plot as shown
in Fig. 4. From these figures, the user can get a quick and intuitive sense of
how the feature is distributed within the design space. For example, Fig. 4a
shows a design space, and it highlights a feature whose =,8> C ; E is
high andG=,8> E ; C is low. This feature explains most of the target de-
signs, but it is too general, such that it also covers many other designs that
are not in the target region. In contrast, the feature highlighted in Fig. 4b has
low =,8> C ; E and high =,8> E ; C . The designs that have this fea-
ture fall mostly inside the target region, but only a small portion of it is ex-
plained by this feature.
A.N. Author
Fig. 3 Feature space plot, where each axis is one of the confidence measures. Each triangle
represents one feature. The red star represents the utopia point of the feature space.
Fig. 4 Design space highlighting different features. (a) Designs that have the feature with
high =,8> C ; E , and (b) another feature with high =,8> E ; C are highlighted. The
cyan dots are the target designs. The pink dots are the designs that have the feature. The
purple dots are the overlap of those two sets of designs.
Exploring the Feature Space to Aid Learning in Design Space Exploration 11
4.3. Representing and Modifying Features
When the user hovers a mouse over a feature in the feature space plot, a
tooltip appears with the name of the feature. For example, the following text
represents a feature that consists of two base features linked with a conjunc-
absent(I) AND present(K)”
In natural language, this can be interpreted as, “Instrument I is not used in
any orbit, and instrument K is used in at least one of the orbits.” However,
in our tool, such representation can only be used to view the extracted fea-
tures and cannot be used to modify the given feature or input a new one.
Fig. 5 Representation of a feature using a graph. The displayed feature consists of 5 base
features linked using both conjunctions and disjunctions. The feature can be interpreted in
text as “absent(I) AND present(K) AND notInOrbit(4000,K,G,B) AND (inOrbit(1000,L) OR
In order to enable the user to modify and explore other features, we imple-
mented a graphical representation of the feature as shown in Fig. 5. This
representation uses a tree structure, consisting of two types of nodes. A leaf
node represents a base feature, and a logical connective node represents a
logical connective (logical conjunction or disjunction) that links all its chil-
dren nodes. Therefore, the feature shown in Fig. 5 can also be written in text
A.N. Author
as: “absent(I) AND present(K) AND notInOrbit(4000,K,G,B) AND (inOr-
bit(1000,L) OR inOrbit(1000,G,A)).” This graphical representation allows
the user to easily see the hierarchical structure within a logical expression
when both conjunctions and disjunctions are used. Moreover, the user can
modify the structure of a feature by changing the location of nodes through
a simple drag and drop. Being able to modify and test different features are
important in order to quickly explore the feature space and gather infor-
4.4. Search in Feature Space
While the user can explore the feature space by modifying and testing indi-
vidual features, we also implement a local search method to speed up the
exploration process. The local search extends a given feature by adding an
additional base feature either using a conjunction (AND) or a disjunction
(OR). The possible set of base features is set by the user during the problem
formulation step (see section 3), and its size is limited to a small number.
Therefore, the system can test the addition of all possible base features, and
return the new set of features that improve one of the goodness metrics.
To run the local search, the user has to select a feature from Fig. 3 by
clicking on one of the features. Then the user can choose to use either a
conjunction or a disjunction in linking the new base feature to the selected
feature. When a conjunction is used, the feature becomes more specific (the
feature covers less designs), most likely leading to an increase in
=,8> E ; C . On the other hand, if a disjunction is used instead, the feature
becomes more general (the feature covers more designs), thus increasing
=,8> C ; E . The newly generated features are compared with the existing
set of features and only the non-dominated ones are added to the visualiza-
tion. This provides a quick and easy way for the user to explore the feature
space effectively, advancing the Pareto front of the feature space.
5. Evaluation
To test the efficacy of exploring the feature space as a way to improve the
users learning, we conduct a controlled experiment with human participants.
Exploring the Feature Space to Aid Learning in Design Space Exploration 13
5.1. Hypothesis and Experiment Conditions
The aim of the experiment is to examine whether exploring the feature space
improves learning, compared to when the user interacts only with the design
space. Learning is defined here as learning the mapping between design de-
cisions and objective values. Therefore, we set our hypothesis as the follow-
- H1: Exploring the feature space improves a designer’s ability to predict
the performance of a design.
To test this hypothesis, we use a within-subject experiment design and com-
pare the learning in two different conditions: design space vs feature space
exploration. The capabilities of these two conditions are summarized in Ta-
ble 2.
In the first condition, called the design space exploration condition, we
provide only the parts in the graphical user interface that are related to the
design space. For example, the user can inspect each design shown in the
design space (see Fig. 2) and observe the values of design decisions and
objectives. The user can also modify each design by adding/deleting/moving
instruments through drag-and-drop. After modifying the design, the new de-
sign can be evaluated to get the corresponding objective values. In addition,
a local search in design space has been implemented to mimic the local
search in feature space (see section 4.4). The local search is done by ran-
domly sampling four neighboring designs from the currently selected design,
evaluating them, and displaying the newly added designs to the scatter plot.
A neighboring design is defined as a design that can be reached by changing
a single design decision from the currently selected design.
The second condition is called the feature space exploration condition.
Here, the user is still able to inspect individual designs in the design space.
However, other interactions in the design space are not allowed. Instead, the
user can run data mining to obtain an initial set of features visualized in a
similar manner to Fig. 3. Modifying, evaluating, and inspecting each feature
is also enabled through the interface shown in Fig. 4. Moreover, the user can
run a local search to quickly explore the feature space.
The conditions are designed to make the types of interactions as similar
as possible in both conditions. The user can modify, evaluate, and inspect
designs/features, and run local searches in the respective spaces.
A.N. Author
Table 2. The capabilities provided in each condition
Design Space
Feature Space
5.2. Experiment Protocol
Participants are first provided with an interactive tutorial that explains the
design problem as well as all the capabilities of the tool. After the tutorial,
each participant is given two different tasks with different design spaces.
Each task has a pre-specified target region in the design space, and the par-
ticipants are asked to take notes of the features that would be useful to iden-
tify whether an arbitrary design will be in the target region or not. During
the task, the two treatment conditions are presented in random order. A 10-
minute time limit is applied to each task.
After each task is finished, the participants are given a short quiz to meas-
ure how much they have learned during the interaction. All the problems in
the quiz are YES/NO questions, asking the user to predict whether a given
design will be located inside the target region or not. A total of 25 questions
are given.
5.3. Participants
We recruited 38 participants, all of whom are university students. The aver-
age age of the participants is 23.0, with a standard deviation of 4.05. There
were 21 male participants and 17 female participants. 26 students identified
themselves as majoring in the STEM field, and 12 students identified them-
selves as having majors other than STEM.
Exploring the Feature Space to Aid Learning in Design Space Exploration 15
The recruitment was done through two different channels. First, we re-
cruited from the general student population on campus and offered $15 Am-
azon gift cards as compensation.
Second, we recruited students who are taking a graduate-level course on
Systems Architecture. These students were offered a small amount of extra
credit for the class as compensation. The reason for recruiting from this se-
cond group of students was our previous experiences in running a pilot ex-
periment and other experiments with similar interfaces. We have observed
in the past that participants who have not been exposed before to some basic
concepts in design space exploration such as design decisions, objectives,
features, recall and precision often struggled to understand the task they
were asked to perform and did not utilize all the capabilities of the tool that
was provided. In addition to our main hypothesis, we also wanted to test if
the participants’ formal training in some of the important concepts has any
interaction effect with their performance in each condition.
6. Result
The test scores of all participants are summarized in Table 3. The average
scores shown in the table represents the percentage of questions that were
answered correctly out of 25 questions asked in each problem set. It shows
that the average scores for both conditions are effectively the same. Running
a paired samples one-tailed t-test gives a p-value of 0.209.
Table 3. Descriptives: All subjects. The Mean score shows the percentage of questions
answered correctly out of 25 questions in each test.
Design Space Exploration
Feature Space Exploration
A more interesting result is observed when the participants are grouped
based on whether they had the formal training (a first-year graduate course
on system architecture) or not. We ran two-way repeated measures ANOVA
to compare the difference in the mean scores of the two conditions while
also considering the effect of the formal training of the subjects. Table 4
shows the within-subject effects, and Table 5 shows the between-subject
effects. The result shows that there is no statistical significance when we
only consider either the experiment condition or the formal training (taking
A.N. Author
the System Architecture class or not) separately. However, there is a signif-
icant interaction effect between the two factors (the p-value is 0.002).
Table 4. Within-subject effects
Sum of
Exploration Strategy
Exploration Strategy * Formal Training
Table 5. Between-subject effects
Sum of
Formal Training
Fig. 6 The test scores for each exploration strategy, factored by whether participants
received formal training in system architecture design. The error bar shows the standard
Fig. 6 shows the average test scores after the participants have been divided
into two groups (received formal training or not). These two groups of par-
ticipants exhibit opposite trends in the scores. Those who have not received
any formal training scored better in the quiz (one-tailed paired samples t-
test: t=1.261, p=0.890), when they explored the design space (M=74.09,
SD=12.41) than when whey explored the feature space (M=70.26,
SD=11.11). On the other hand, those who received formal training per-
formed better (one-tailed paired samples t-test: t=3.759, p<0.001), when
Exploring the Feature Space to Aid Learning in Design Space Exploration 17
they explored the feature space (M=82.13, SD=6.906), compared to when
they explored the design space (M=71.20, SD=8.029).
7. Discussion
From the experiment, we find that there is an interaction between the explo-
ration strategy and the formal training. When we compare the average scores
within the participants who received formal training, it is significantly
higher in the feature space exploration condition than in the design space
exploration condition.
The interaction effect between the exploration strategies and the formal
training suggests that those who have been exposed to the basic concepts of
design space exploration find the feature space exploration more useful. In
the Systems Architecture class, the lectures cover a wide range of topics
related to tradespace analysis including decision space, objective space, Pa-
reto dominance, data mining, and sensitivity analysis among others. While
this does not ensure a student’s understanding of these subjects, we can as-
sume that they have been exposed to, and thus familiar with, these topics.
The current experiment result partially supports our hypothesis that explor-
ing the feature space improves learning, with a condition that the user has to
be familiar with the key concepts of design space exploration and have been
trained to reason in an abstract space. This is a promising result, since the
proposed method is mainly intended for professional engineering designers
who are familiar with the concept of design space exploration.
8. Conclusion
This paper introduced the new concept of exploring the feature space,
where the feature space is defined as a set of possible features (combinations
of values for various design decisions). The feature space is visualized in a
2D plane, with each axis representing =,8> ? ; @ and =,8> @ ; ?
two measures that are equivalent to recall and precision, respectively. The
designer can explore the feature space by modifying and testing different
features and receiving immediate feedback on how the values of the good-
ness of features change in response. Such interaction provides a chance to
learn the how well different features explain a selected region of the design
space. As a result, the designer can easily identify what the important fea-
tures are in driving the performance of design.
A.N. Author
To test the effectiveness of this process as a learning tool, we conducted
a human subject experiment. The result showed that the participants who
received formal training in the key concepts of design space exploration per-
formed better when they had a chance to explore the feature space, as op-
posed to when they explored only in the design space. This shows that fea-
ture space exploration has potential to enhance designer’s learning about the
important features and thus help predicting the behavior of a design.
9. References
[1] R. Balling, “Design by shopping: A new paradigm?,” in Proceedings of
the Third World Congress of structural and multidisciplinary optimization
(WCSMO-3), 1999, vol. 1, pp. 295297.
[2] N. W. Hirschi and D. D. Frey, “Cognition and complexity: An experiment
on the effect of coupling in parameter design,” Res. Eng. Des., vol. 13, pp.
123131, 2002.
[3] F. Flager, D. J. Gerber, and B. Kallman, “Measuring the impact of scale
and coupling on solution quality for building design problems,” Des.
Stud., vol. 35, no. 2, pp. 180199, 2014.
[4] P. T. Grogan and O. L. de Weck, “Collaboration and complexity: an
experiment on the effect of multi-actor coupled design,” Res. Eng. Des.,
vol. 27, no. 3, pp. 221235, 2016.
[5] J. Eddy and K. E. Lewis, “Visualization of Multidimensional Design and
Optimization using Cloud Visualization,” in Proceedings of DETC’02,
2002, pp. 899908.
[6] G. M. Stump, M. Yukish, T. W. Simpson, and E. N. Harris, “Design
Space Visualization and Its Application To a Design By Shopping
Paradigm,” in Proceedings of DETC’03, 2003.
[7] S. Obayashi and D. Sasaki, “Visualization and Data Mining of Pareto
Solutions Using Self-Organizing Map,” Evol. Multi-Criterion Optim., vol.
LNCS 2632, pp. 796809, 2003.
[8] X. (Luke) Zhang, T. Simpson, M. Frecker, and G. Lesieutre, “Supporting
knowledge exploration and discovery in multi-dimensional data with
interactive multiscale visualisation,” J. Eng. Des., vol. 23, no. 1, pp. 23
47, 2012.
[9] B. J. German, K. M. Feigh, and M. Daskilewicz, “An Experimental Study
of Continuous and Discrete Visualization Paradigms for Interactive Trade
Space Exploration,” J. Comput. Inf. Sci. Eng., vol. 13, no. June 2013, pp.
1–12, 2013.
[10] N. Knerr and D. Selva, “Cityplot: Visualization of High-Dimensional
Design Spaces with Multiple Criteria,” J. Mech. Des., vol. 138, no. 9, p.
91403, 2016.
[11] M. J. Diaz, D. D. Fullmer, S. I. Briceno, and D. N. Mavris, “A
Exploring the Feature Space to Aid Learning in Design Space Exploration 19
Collaborative Approach to Complex Systems Engineering using a Web-
based Visual Analytics Framework,” in 55th AIAA Aerospace Sciences
Meeting, 2017, no. January.
[12] R. Agrawal, T. Imielinski, and A. Swami, “Mining Association Rules
between Sets of Items in Large Databases,” ACM SIGMOD Rec., vol. 22,
no. 2, pp. 207216, 1993.
[13] J. R. Quinlan, C4. 5: programs for machine learning. Elsevier, 1993.
[14] G. Cervone, P. Franzese, and A. P. K. Keesee, “Algorithm quasi-optimal
(AQ) learning,” Wiley Interdiscip. Rev. Comput. Stat., vol. 2, no. 2, pp.
218236, 2010.
[15] J. C. A. Santos, H. Nassif, D. Page, S. H. Muggleton, and M. J. E.
Sternberg, “Automated identification of protein-ligand interaction fea-
tures using Inductive Logic Programming: A hexose binding case study,”
BMC Bioinformatics, vol. 13, no. 1, p. 162, 2012.
[16] H. Bang and D. Selva, “iFEED: Interactive Feature Extraction for
Engineering Design,” in Proceedings of the ASME 2016 International
Design Engineering Technical Conferences & Computers and Information
in Engineering Conference IDETC/CIE 2016, 2016, pp. 111.
[17] S. Watanabe, Y. Chiba, and M. Kanazaki, “A proposal on analysis support
system based on association rule analysis for non-dominated solutions,” in
Proceedings of the 2014 IEEE Congress on Evolutionary Computation,
CEC 2014, 2014, pp. 880887.
[18] S. Bandaru, A. H. C. Ng, and K. Deb, “Data mining methods for
knowledge discovery in multi-objective optimization: Part A - Survey,”
Expert Syst. Appl., vol. 70, pp. 139159, 2017.
[19] R. Brachman and H. Levesque, Knowledge Representation and
Reasoning. Morgan Kaufmann, 2004.
[20] J. Feldman, “The Simplicity Principle Concept Learning in Human,”
Psychol. Sci., vol. 12, no. 6, pp. 227–232, 2010.
[21] W. L. W. Li, J. H. J. Han, and J. P. J. Pei, “CMAR: accurate and efficient
classification based on multiple\nclass-association rules,” Proc. 2001
IEEE Int. Conf. Data Min., pp. 369376, 2001.
[22] B. Liu, W. Hsu, Y. Ma, and B. Ma, “Integrating Classification and
Association Rule Mining,” in Knowledge Discovery and Data Mining,
1998, pp. 8086.
[23] P.-N. Tan, V. Kumar, and J. Srivastava, “Selecting the right objective
measure for association analysis,” Inf. Syst., vol. 29, no. 4, pp. 293313,
[24] M. Sokolova and G. Lapalme, “A systematic analysis of performance
measures for classification tasks,” Inf. Process. Manag., vol. 45, no. 4, pp.
427437, 2009.
[25] M. Buckland and F. Gey, “The Relationship Between Recall and
Precision,” J. Am. Soc. Inf. Sci., vol. 45, no. 1, pp. 1219, 1994.
[26] F. Kudo and T. Yoshikawa, “Knowledge extraction in multi-objective
optimization problem based on visualization of Pareto solutions,” in 2012
IEEE Congress on Evolutionary Computation, CEC 2012, 2012.
A.N. Author
[27] W. Chen, J. Chazan, and M. Fuge, “How Designs Differ: Non-linear
Embeddings Illuminate Intrinsic Design Complexity,” in Proceedings of
the ASME 2016 International Design Engineering Technical Conferences
& Computers and Information in Engineering Conference IDETC/CIE
2016, 2016, pp. 112.
[28] G. Agrawal, C. L. Bloebaum, and K. E. Lewis, “Intuitive Design Selection
Using Visualized n-Dimensional Pareto Frontier,” in 46th
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics &
Materials Conference, AIAA 2005-1813, 2005, no. April, pp. 114.
[29] P.-W. Chiu and C. Bloebaum, “Hyper-Radial Visualization (HRV) for
Decision-Making in Multi-Objective Optimization,” in 46th AIAA
Aerospace Sciences Meeting and Exhibit, AIAA 2008-907, 2008, no.
[30] D. J. Walker, R. M. Everson, and J. E. Fieldsend, “Visualizing mutually
nondominating solution sets in many-objective optimization,” IEEE
Trans. Evol. Comput., vol. 17, no. 2, pp. 165184, 2013.
[31] W. Peng, M. O. Ward, and E. A. Rundensteiner, “Clutter reduction in
multi-dimensional data visualization using dimension reordering,” in
Proceedings of IEEE Symposium on Information Visualization, INFO VIS,
2004, pp. 8996.
[32] A. H. C. Ng, C. Dudas, H. Bostrom, and K. Deb, “Interleaving
Innovization with Evolutionary Multi-objective Optimization in
Production System Simulation for Faster Convergence,” in Learning and
Intelligent Optimization, Springer Berlin Heidelberg, 2013, pp. 118.
[33] R. Jahr, H. Calborean, L. Vintan, and T. Ungerer, “Boosting design space
explorations with existing or automatically learned knowledge,” Proc.
16th Int. GI/ITG Conf. Meas. Model. Eval. Comput. Syst. Dependability
Fault Toler., pp. 221235, 2012.
[34] X. Yan, Q. Mu, T. W. Simpson, J. Li, and X. Zhang, “Work-Centered
Visual Analytics to Support Multidisciplinary Design Analysis and
Optimization,” in 12th AIAA Aviation Technology, Integration, and
Operations (ATIO) Conference and 14th AIAA/ISSM, 2012, pp. 112.
[35] R. S. Michalski, J. Wojtusiak, and K. A. Kaufman, “Intelligent
optimization via learnable evolution model,” Proc. - Int. Conf. Tools with
Artif. Intell. ICTAI, no. 3, pp. 332335, 2006.
[36] K. Sugimura, S. Jeong, S. Obayashi, and T. Kimura, “Kriging-model-
based multi-objective robust optimization and trade-off-rule mining using
association rule with aspiration vector,” 2009 IEEE Congr. Evol. Comput.
CEC 2009, pp. 522529, 2009.
[37] D. Selva, “Knowledge-intensive global optimization of Earth observing
system architectures: a climate-centric case study,” SPIE Remote Sens.,
vol. 9241, p. 92411S92411S, 2014.
[38] R. Agrawal and R. Srikant, “Fast algorithms for mining association rules,
Proceeding VLDB ’94 Proc. 20th Int. Conf. Very Large Data Bases, vol.
1215, pp. 487499, 1994.
... In this paper, we highlight that bringing a human designer into the design space exploration (DSE) can potentially better balance a designer's learning and design perfor-mance compared to a "black-box" only optimization [14,15]. Existing approaches to human-in-the-loop design space exploration take designer inputs on design-and feature selection [16,17,18] and present feedback on performance metrics and the diversity of generated methods [3,19]. The interaction between a human designer and the computer includes visual graphical user interfaces [16], natural language processing interfaces for question answering and textual explanations [20], and tangible physical interfaces [14]. ...
... The input type pertains to how user feedback is incorporated. A user might steer design exploration by choosing a desired set of designs or design variable values [34], select design parameters or features [35,17], or set a range for desired objective values [15,19]. Using highlevel rather than low-level features can reduce the number of input commands. ...
... Any desired DSE outcomes must be represented as a particular loss function. Existing generative design methods can optimize predefined performance metrics like structural compliance [36,5,4], maximize the diversity of generated designs [3,7,8], or learn driving features behind selected designs [17,35]. Finally, the mode of interaction between a user and the underlying tool can be a graphical user interface, natural language interface, or tangible interface. ...
Full-text available
Deep generative models have shown significant promise in improving performance in design space exploration. But there is limited understanding of their interpretability, a necessity when model explanations are desired and problems are ill-defined. Interpretability involves learning design features behind design performance, called designer learning. This study explores human-machine collaboration's effects on designer learning and design performance. We conduct an experiment (N=42) designing mechanical metamaterials using a conditional variational auto-encoder. The independent variables are: (i) the level of automation of design synthesis, e.g., manual (where the user manually manipulates design variables), manual feature-based (where the user manipulates the weights of the features learned by the encoder), and semi-automated feature-based (where the agent generates a local design based on a start design and user-selected step size); and (ii) feature semanticity, e.g., meaningful versus abstract features. We assess feature-specific learning using Item Response Theory and design performance using utopia distance and hypervolume improvement. The results suggest design performance depends on the subjects' feature-specific knowledge, emphasizing the precursory role of learning. The semi-automated synthesis locally improves the utopia distance. Still, it does not result in higher global hypervolume improvement compared to manual design synthesis, and reduced designer learning compared to manual feature-based synthesis. The subjects learn semantic features better than abstract features only when design performance is sensitive to them. Potential cognitive constructs influencing learning in human-AI collaborative settings are discussed, such as cognitive load and recognition heuristics.
... In this paper, we highlight that bringing a human designer into the design space exploration (DSE) can potentially better balance a designer's learning and design perfor-mance compared to a "black-box" only optimization [14,15]. Existing approaches to human-in-the-loop design space exploration take designer inputs on design-and feature selection [16,17,18] and present feedback on performance metrics and the diversity of generated methods [3,19]. The interaction between a human designer and the computer includes visual graphical user interfaces [16], natural language processing interfaces for question answering and textual explanations [20], and tangible physical interfaces [14]. ...
... The input type pertains to how user feedback is incorporated. A user might steer design exploration by choosing a desired set of designs or design variable values [34], select design parameters or features [35,17], or set a range for desired objective values [15,19]. Using highlevel rather than low-level features can reduce the number of input commands. ...
... Any desired DSE outcomes must be represented as a particular loss function. Existing generative design methods can optimize predefined performance metrics like structural compliance [36,5,4], maximize the diversity of generated designs [3,7,8], or learn driving features behind selected designs [17,35]. Finally, the mode of interaction between a user and the underlying tool can be a graphical user interface, natural language interface, or tangible interface. ...
Conference Paper
Full-text available
Deep generative models have shown significant promise to improve performance in design space exploration (DSE), but they lack interpretability. A component of interpretability in DSE is helping designers learn how input design decisions influence multi-objective performance. This experimental study explores how human-machine collaboration influences both designer learning and design performance in deep learning-based DSE. A within-subject experiment is implemented with 42 subjects involving mechanical metamaterial design using a conditional variational auto-encoder. The independent variables in the experiment are two interactivity factors: (i) simulatability, e.g., manual design generation (high simulatability), manual feature-based design synthesis, and semi-automated feature-based synthesis (low simulatibility); and (ii) semanticity of features, e.g., meaningful versus abstract latent features. We perform assessment of designer learning using item response theory and design performance using metrics such as distance to utopia point and hypervolume improvement. The findings highlights a highly intertwined relationship between designer learning and design performance. Compared to manual design generation, the semi-automated synthesis generates designs closer to the utopia point. Still, it does not result in greater hyper-volume improvement. Further, the subjects learn the effects of semantic features better than abstract features, but only when the design performance is sensitive to those semantic features. Potential cognitive constructs, such as cognitive load and recognition heuristic, that may influence the interpretability of deep generative models are discussed.
Full-text available
This article describes Daphne, a virtual assistant for designing Earth observation distributed spacecraft missions. It is, to the best of our knowledge, the first virtual assistant for such application. The article provides a thorough description of Daphne, including its question answering system and the main features we have implemented to help system engineers design distributed spacecraft missions. In addition, the article describes a study performed at NASA's Jet Propulsion Laboratory (JPL) to assess the usefulness of Daphne in this use case. The study was conducted with N = 9 subjects from JPL, who were asked to work on a mission design task with two versions of Daphne, one that was fully featured implementing the cognitive assistance functions, and one that only had the features one would find in a traditional design space exploration tool. After the task, they filled out a standard user experience survey, completed a test to assess how much they learned about the task, and were asked a number of questions in a semi-structured exit interview. Results of the study suggest that Daphne can help improve performance during system design tasks compared to traditional tools, while keeping the system usable. However, the study also raises some concerns with respect to a potential reduction in human learning due to the use of the cognitive assistant. The article ends with a list of suggestions for future development of virtual assistants for space mission design.
Conference Paper
Full-text available
One of the major challenges faced by the decision maker in the design of complex engineering systems is information overload. When the size and dimensionality of the data exceeds a certain level, a designer may become overwhelmed and no longer be able to perceive and analyze the underlying dynamics of the design problem at hand, which can result in premature or poor design selection. There exist various knowledge discovery and visual analytic tools designed to relieve the information overload, such as BrickViz, Cloud Visualization, ATSV, and LIVE, to name a few. However, most of them do not explicitly support the discovery of key knowledge about the mapping between the design space and the objective space, such as the set of high-level design features that drive most of the trade-offs between objectives. In this paper, we introduce a new interactive method, called iFEED, that supports the designer in the process of high-level knowledge discovery in a large, multiobjective design space. The primary goal of the method is to iteratively mine the design space dataset for driving features, i.e., combinations of design variables that appear to consistently drive designs towards specific target regions in the design space set by the user. This is implemented using a data mining algorithm that mines interesting patterns in the form of association rules. The extracted patterns are then used to build a surrogate classification model based on a decision tree that predicts whether a design is likely to be located in the target region of the tradespace or not. Higher level features will generate more compact classification trees while improving classification accuracy. If the mined features are not satisfactory, the user can go back to the first step and extract higher level features. Such iterative process helps the user to gain insights and build a mental model of how design variables are mapped into objective values. A controlled experiment with human subjects is designed to test the effectiveness of the proposed method. A preliminary result from the pilot experiment is presented. Copyright © 2016 by ASME Country-Specific Mortality and Growth Failure in Infancy and Yound Children and Association With Material Stature Use interactive graphics and maps to view and sort country-specific infant and early dhildhood mortality and growth failure data and their association with maternal
Conference Paper
Full-text available
This paper shows how to measure the complexity and reduce the dimensionality of a geometric design space. It assumes that high-dimensional design parameters actually lie in a much lower-dimensional space that represents semantic attributes. Past work has shown how to embed designs using techniques like autoencoders; in contrast, this paper quantifies when and how various embeddings are better than others. It captures the intrinsic dimensionality of a design space, the performance of recreating new designs for an embedding, and the preservation of topology of the original design space. We demonstrate this with both synthetic superformula shapes of varying non-linearity and real glassware designs. We evaluate multiple embeddings by measuring shape reconstruction error, topology preservation, and required semantic space dimensionality. Our work generates fundamental knowledge about the inherent complexity of a design space and how designs differ from one another. This deepens our understanding of design complexity in general.
Full-text available
Design of complex systems requires collaborative teams to overcome limitations of individuals; however, teamwork contributes new sources of complexity related to information exchange among members. This paper formulates a human subjects experiment to quantify the relative contribution of technical and social sources of complexity to design effort using a surrogate task based on a parameter design problem. Ten groups of 3 subjects each perform 42 design tasks with variable problem size and coupling (technical complexity) and team size (social complexity) to measure completion time (design effort). Results of a two-level regression model replicate past work to show completion time grows geometrically with problem size for highly coupled tasks. New findings show the effect of team size is independent from problem size for both coupled and uncoupled tasks considered in this study. Collaboration contributes a large fraction of total effort, and it increases with team size: about 50–60 % of time and 70–80 % of cost for pairs and 60–80 % of time and 90 % of cost for triads. Conclusions identify a role for improved design methods and tools to anticipate and overcome the high cost of collaboration.
Real-world optimization problems typically involve multiple objectives to be optimized simultaneously under multiple constraints and with respect to several variables. While multi-objective optimization itself can be a challenging task, equally difficult is the ability to make sense of the obtained solutions. In this two-part paper, we deal with data mining methods that can be applied to extract knowledge about multi-objective optimization problems from the solutions generated during optimization. This knowledge is expected to provide deeper insights about the problem to the decision maker, in addition to assisting the optimization process in future design iterations through an expert system. The current paper surveys several existing data mining methods and classifies them by methodology and type of knowledge discovered. Most of these methods come from the domain of exploratory data analysis and can be applied to any multivariate data. We specifically look at methods that can generate explicit knowledge in a machine-usable form. A framework for knowledge-driven optimization is proposed, which involves both online and offline elements of knowledge discovery. One of the conclusions of this survey is that while there are a number of data mining methods that can deal with data involving continuous variables, only a few ad hoc methods exist that can provide explicit knowledge when the variables involved are of a discrete nature. Part B of this paper proposes new techniques that can be used with such datasets and applies them to discrete variable multi-objective problems related to production systems.
In the early-phase design of complex systems, a model of design performance is coupled with visualizations of competing designs and used to aid human decision-makers in finding and understanding an optimal design. This consists of understanding the tradeoffs among multiple criteria of a "good" design and the features of good designs. Current visualization techniques are limited when visualizing many performance criteria and/or do not explicitly relate the mapping between the design space and the objective space. We present a new technique called Cityplot, which can visualize a sample of an arbitrary (continuous or combinatorial) design space and the corresponding single or multidimensional objective space simultaneously. Essentially a superposition of a dimensionally reduced representation of the design decisions and bar plots representing the multiple criteria of the objective space, Cityplot can provide explicit information on the relationships between the design decisions and the design criteria. Cityplot can present decision settings in different parts of the space and reveal information on the decision †' criteria mapping, such as sensitivity, smoothness, and key decisions that result in particular criteria values. By focusing the Cityplot on the Pareto frontier from the criteria, Cityplot can reveal tradeoffs and Pareto optimal design families without prior assumptions on the structure of either. The method is demonstrated on two toy problems and two real engineered systems, namely, the NASA earth observing system (EOS) and a guidance, navigation and control (GNC) system.
Software tools that enable interactive data visualization are now commonly available for engineering design. These tools allow engineers to inspect, filter, and select promising alternatives from large multivariate design spaces based upon an examination of the trade-offs between multiple objectives. There are two general approaches for visually representing data: (1) discretely, by plotting a sample of designs as distinct points; and (2) continuously, by plotting the functional relationships between design variables and design metrics as curves or surfaces. In this paper, we examine these two approaches through a human subjects experiment. Participants were asked to complete two design tasks with an interactive visualization tool: one by using a sample of discrete designs and one by using a continuous representation of the design space. Metrics describing the optimality of the design outcomes, the usage of different graphics, and the task workload were quantified by mouse tracking, user process descriptions, and analysis of the selected designs. The results indicate that users had more difficultly in selecting multiobjective optimal designs with common continuous graphics than with discrete graphics. The findings suggest that innovative features and additional usability studies are required in order for continuous trade space visualization tools to achieve their full potential.
Conference Paper
A visualization methodology is developed in which a Hyperspace Pareto Frontier (HPF) can be visualized in an intuitive way for an n-dimensional performance space. The new approach is termed the Hyper-Radial Visualization (HRV) method. Three approaches for preference incorporation are developed that enable designers to quickly identify better regions in the performance space for multi-objective optimization problems. Visualizing Pareto solutions that have more than three objective functions is a significant challenge in the multi-objective optimization community. The HRV method described here provides a lossless visualization tool to represent an HPF and is applied using a Direct Sorting Method (DSM) to design concept selection in multi-objective optimization problems in this paper. The viability and desirability of using the HRV for visualizing an HPF is explored here. First, this paper presents an approach to generate a lossless and intuitive representation of HPF by using the HRV method. Second, three different color-coded preference schemes are developed and presented to enable easy sorting of results in the HRV. Finally, the HRV-based visualization method is validated using three multi-objective optimization problems.
Conference Paper
*† ‡ § ¶ Information overload has become a notable problem in many multidisciplinary design optimization analyses. When dealing with large design datasets, engineering designers can quickly become overwhelmed by the data, solutions, and their relationship. This paper presents a work-centered visual analytics framework to address such challenges. The proposed framework integrates user-centered interactive visualization and data-oriented computational algorithms into two analytical loops to help designers perform in-depth analysis on a trade space. An application system prototype, LIVE, has been developed to support multidisciplinary design optimization. The proposed system allows designers to analyze data, discover patterns, and formalize preferences in a uniform and integrated software platform by combining visualization and data mining. This approach is expected to help designers efficiently making sense of complicated multi-dimensional design data sets. I. Background uring the process of analysis and optimization of multidisciplinary problems, information overload has become a notable problem. A posteriori design paradigm 1 allows engineer designers to view the visualization of the entire design space before choosing optimal designs based on their preferences. Comparing to traditional a priori approaches, a posteriori approaches have the advantage of giving designers more control in the design exploration and selection process; however, the amount of computer-simulated design alternatives and solutions can easily overwhelm designers and go beyond human's analysis capability. Selecting interesting design candidates from such large-scale data sets becomes especially difficult for designers even when aided by optimization algorithms. Therefore, effective tools to support exploring and analyzing such massive datasets are needed in the process of multidisciplinary design optimization. Information visualization has been used to support trade space exploration during design analysis and optimization because it can utilize human’s visual perception capabilities to identify patterns and features within data sets. However, traditional visualization techniques cannot handle large, multi-dimensional data sets well due to the limitations of display devices and human’s cognitive bandwidth. Recently, the new discipline of visual analytics 2