Exploring the Feature Space to Aid Learning in Design Space Exploration

Full text

Design Computing and Cognition DCC’18. J.S. Gero (ed),
pp. xx-yy. © Springer 2018
1
Exploring the Feature Space to Aid Learning in Design
Space Exploration
In this paper, we introduce the concept of exploring the feature space to aid
learning in the context of design space exploration (a.k.a. tradespace explo-
ration). The feature space is defined as a possible set of features (specific
values of design decisions or attributes), mapped in a 2D plane with each
axis representing different interestingness measures. Similar to how a de-
signer explores the design space, one can explore the feature space by ob-
serving how different features vary in their usefulness in explaining a group
of design solutions. This could aid in the designer’s better understanding of
the design space. To test the effectiveness of this process as a learning tool,
we conduct a controlled experiment with human subjects. The result shows
that feature space exploration has potential to enhance the designer’s ability
to identify important features and predict the performance of a design.
1. Introduction
Over the last two decades, “design by shopping” paradigm [1] has become
a popular approach to tackle early-phase (conceptual design or system ar-
chitecting) engineering design problems. An important step in this approach
is called design space exploration (a.k.a. tradespace exploration), where the
designer analyzes the structure of the design space and learns about the
trade-offs in the system, sensitivities of design criteria to design decisions,
couplings between design decisions, etc. For the remainder of this paper,
“learning” in tradespace exploration refers to gaining knowledge about these
parameters, and more generally about the mapping between design decisions
and design criteria. Through the process of design space exploration, the
designer can make a more informed decision for the selection of the final
design.
However, design space exploration presents us with the challenge of in-
formation overload. The problem of information overload becomes more
prominent in design tasks involving many design decisions, multiple objec-
tives, and intricate couplings between them. It has been shown that as the

A.N. Author
2
design problem gets more complex, the designers are overwhelmed by the
size and the complexity of the data, thus leading to the degradation of their
ability to understand the relationships between different variables [2]–[4].
To address this issue, various data visualization methods and tools have
been developed [5]–[11]. These visualizations are useful, as they provide a
quick and intuitive sense of the structure of the design space. However, they
also have limitations, as the knowledge learned through visualizations can
be ambiguous.
Another complementary approach to learn about the design space is to
extract knowledge using data mining algorithms that mine knowledge ex-
plicitly in the form of logical if-then rules [12]–[15]. These methods can be
used to extract driving features, i.e., the common features (specific values
of design decisions, attributes, or combinations there-of) that are shared by
a group of designs that exhibit similar objective values [16]. For example,
Watanabe et al. use association rule mining to analyze hybrid rocket engine
designs, and find that 83% of all non-dominated (Pareto optimal) solutions
had a similar initial port radius [17]. The major advantage of such
knowledge is that it can be expressed relatively concisely and unambigu-
ously through a formal representation [18].
While having been used successfully in the past to analyze design spaces,
these methods are not without limitations. One of the limitations of the cur-
rent methods is that they impose a rigid structure in the mined features (the
conditional “if” parts of the rules); indeed, all features are represented as
predicates (i.e., binary features) joined by logical conjunctions (i.e., “and”
operator). From a mathematical point of view, this does not reduce expres-
sivity, as any logical formula can be converted into a Disjunctive Normal
Form or DNF (i.e., a disjunction – OR – of conjunctive clauses) [19]. There-
fore, any Boolean concept (a concept whose membership is determined by
a combination of binary features [20]) can be represented using a set of rules
(disjunction of rules). However, from a human learning point of view, the
conversion to DNF often results in longer features, and thus harder to un-
derstand by humans.
Another limitation of the data mining methods is that they generate a large
set of features without an easy way to identify the most useful and informa-
tive one [21]. One approach to select a single feature is to sort all features
using one measure such as confidence or lift [22]. Intuitively, these interest-
ingness measures provide a quantitative metric of the predictive power of
the feature. However, selecting a single metric from a large list of alterna-
tives can be arbitrary, and may not necessarily be the right measure for the
given design problem [23], [24].

Exploring the Feature Space to Aid Learning in Design Space Exploration 3
In this paper, we present a new method, feature space exploration, to aid
human learning in design space exploration and a tool to use over the
method. The aim of this method is to improve the designer’s ability to iden-
tify important features and use that information to predict the performance
of a design. In order to foster learning, we enable designers to explore vari-
ous forms of features and get immediate feedback on how well these features
explain a certain region of the design space (e.g., a cluster, or the Pareto
front). This is done by defining a space of all possible features (called the
feature space), visualized on a 2D plane. Each axis in the plane represents
one of the interestingness measures of features used in classification (e.g.
precision and recall [24]) or association analysis (e.g. confidence, lift, and
Gini index [23]). If one selects conflicting goodness measures such as pre-
cision and recall [25], the Pareto front of the feature space can also be de-
fined. The designer can then use the visualization to observe how the good-
ness measures change in response to a change in the feature, and elicit his
or her preferences among those two importance measures. Due to its simi-
larity to how a designer explores the design space, we refer to this process
as “exploring the feature space”. Exploring the feature space helps the de-
signer identify the driving features that shape the structure of the design
space, as it allows selecting or synthesizing a feature that best fits the pur-
pose of the current analysis. The process takes advantage of the intuitive and
fast nature of exploring options through visualization, as well as the ease of
learning through formal representations that are clear and concise.
To demonstrate the effectiveness of this new method to improve learning,
we conduct a controlled experiment with human subjects. The experiment
tests whether exploring the feature space improves the designer’s ability to
predict whether a given design will exhibit desirable performance and cost.
The result shows that exploring the feature space may indeed improve learn-
ing about the design space but only under certain conditions – for subjects
who have received some formal training in design space exploration.
The remainder of the paper is organized as follows. Section 2 provides a
background and literature review on design space exploration methods. Sec-
tion 3 introduces the example design problem that is used in this paper. Sec-
tion 4 explains the proposed method of exploring the feature space. Section
5 covers the setup for a human-subject experiment. Section 6 shows the re-
sult from the experiment. Section 7 discusses the experiment result, and Sec-
tion 8 discusses the conclusions and future work.

A.N. Author
4
2. Related Work
Various knowledge discovery and decision support tools have been de-
veloped to support design space exploration for solving complex design
problems. Bandaru et al. classifies these methods based on whether the
knowledge is extracted in an implicit or in an explicit form [18]. Implicit
knowledge is defined as having no formal notation, thus making it difficult
to transfer information unambiguously.
The typical way of representing implicit knowledge is through visualiza-
tions. Various data visualization methods have been applied to design space
exploration, including cloud visualization [5], self-organizing maps [7], in-
teractive multiscale-nested clustering and aggregation (iMSNCA) [8], city-
plot [10], isomap [26], kernel PCA [27], Hyper-Space Diagonal Counting
(HSDC) [28], and Hyper-Radial Visualization (HRV) [29]. Most of these
tools focus on providing different views of designs defined in a multi-di-
mensional space, in some cases coupled with unsupervised machine learn-
ing methods such as clustering, feature selection, and manifold learning.
However, due to the knowledge being implicit in visualization, these meth-
ods require an additional step for the humans to visually inspect the result
and make interpretations. Therefore, the knowledge obtained through visu-
alization can be ambiguous and subjective. Moreover, visually inspecting
and finding patterns may be challenging without sophisticated re-arranging
strategies [30], [31].
There also exist knowledge discovery methods that generate knowledge
in an explicit form, i.e., using formal representations such as logical rules.
Data mining methods that extract knowledge in the form of logical if-then
rules include decision trees [32]–[34], Algorithm Quasi-optimal (AQ) learn-
ing [14], [35], and association rule mining [16], [17], [36]. The advantages
of extracting knowledge in the form of logical rules is that they are compact
and unambiguous. However, these rule mining methods also have limita-
tions. They can only extract rules with a rigid, pre-determined form such as
DNF. This imposes a significant limitation in extract knowledge in learnable
format. Another limitation often found in rule mining methods is that while
they generate a large number of alternative rules, it is not easy for the user
to grasp what are the differences in performances of those rules relative to
each other.

Exploring the Feature Space to Aid Learning in Design Space Exploration 5
3. Example Design Problem: Architecting Earth Ob-
serving Satellite System
Before explaining how the proposed method works, we first introduce an
example design problem to help explain the methodology in the remainder
of the paper. It should be noted that the proposed method is not specific to a
type of a design problem. However, there are some implementation details
that are tailored to the structure of this problem. This point will be elaborated
on after the design problem is outlined.
The design problem is a real-world system architecting problem previ-
ously studied in [37]. The goal of the design task is to architect a constella-
tion of satellites to provide operational observation of the Earth’s climate.
There are two objectives: maximizing the scientific benefit and minimizing
the lifecycle cost. Scientific benefit is a function of an architecture’s satis-
faction of 371 climate-related measurement objectives, generated based on
the World Meteorological Organization’s OSCAR (Observing Systems Ca-
pability Analysis and Review Tool) database1. The level of satisfaction of
each measurement objective is quantified based on the capabilities of each
design, and then aggregated to obtain a number that represents how much
scientific benefit each design brings to the climate scientific community.
The design problem has been formulated as an assignment problem be-
tween a set of candidate measurement instruments (remote sensors related
to climate monitoring) and a set of candidate orbits (defined by orbital pa-
rameters such as altitude and inclination). Given a set P of candidate instru-
ments and a set O of candidate orbits, the design space is defined as a set of
all binary relations from P to O. Each instrument in P can be assigned to any
subset of orbit O, including the empty set. Therefore, the size of the design
space is !" # , where $ is the number of candidate instruments and % is
the number of candidate orbits. In this work, we considered 12 candidate
instruments and 5 candidate orbits, making a total of !&' possible designs.
Each design is represented by a Boolean matrix M of size ()*!, where
+ ,- . / * if instrument p is assigned to orbit o, and + ,- . / 0 other-
wise. Graphically, this can be displayed by a figure similar to Fig. 1. Here,
each row represents a mission that will fly in each orbit, and the columns
represent the assignment of different instruments. Note that in the examples
that will follow throughout this paper, we replace the name of the actual
orbits and instruments with numbers (e.g., 1000, 2000) and alphabetical let-
ters (e.g., A, B, C) to simplify the presentation of the examples.
1 http://www.wmo-sat.info/oscar/

A.N. Author
6
Fig. 1 An example architecture representation. Each row represents a single spacecraft flying
in a certain orbit. For example, a spacecraft carrying Cloud and Precipitation Radar
(CPR_RAD) and UV/VIS Limb Spectrometer (CHEM_UVSPEC) will fly in a sun-synchro-
nous orbit at altitude of 800km, and the local time at the ascending node in the afternoon.
Once the design decisions and the corresponding objective values are pro-
vided in a structured format, the proposed method mostly considers the de-
sign problem as a black box. At the implementation level, however, there is
one critical step necessary in order to run data mining, which is formulating
the base features. The base features are predicates used to construct more
sophisticated Boolean concepts related to the design space. In its simplest
form, a base feature can be a single design decision set to 0 or 1. However,
we introduce more complex base features to pre-specify the structure of the
patterns to be searched, thus biasing the search towards more promising re-
gions. The base features used for the current system architecting problem
are shown in Table 1. The formulation of the base features requires some
domain-specific knowledge and insights obtained by observing the structure
of the design problem.
Table 1 Base features
Name of the
feature
Arguments
Description
Present
12
Instrument 12 is present in at least one of the orbits
Absent
12
Instrument 12 is absent in all the orbits
InOrbit
%2- 1
3- 415- 167
Instrument 1
3 (and 15, 16) is/are present in orbit %2
NotInOrbit
%2- 1
3- 415- 167
Instrument 1
3 (and 15, 16) is/are not present in orbit %2
Together
12- 1
3- 4157
Instruments 12, 1
3 (and 15) are present together in any orbit
Separate
12- 1
3- 4157
Instruments 12, 1
3 (and 15) are not present together in any
single orbit
emptyOrbit
%2
No instrument is present in orbit %2
numOrbits
8
The number of orbits that have at least one instrument as-
signed is n
For example, Present is a base feature that describes whether an instru-
ment 9 is used in at least one of the orbits. This feature is equivalent to a

Exploring the Feature Space to Aid Learning in Design Space Exploration 7
disjunction of 5 base features (instrument 9 being assigned to each one of
the orbits). We know that Present may potentially speed up the search, since
the decision whether to use an instrument or not has a bigger influence in
the objective value compared to the decision of which orbit it should be as-
signed to. In the remaining sections of this paper, we will use these pre-
defined set of base features to build more complex features.
4. Exploring the Feature Space
In this paper, we propose exploring the feature space as a learning aid for
design space exploration. We define the feature space as a set of all possible
features, visualized by mapping features in a coordinate system where each
axis represents a different measure of the goodness of a feature (e.g. preci-
sion, recall, F score, confidence, lift, and mutual information [23], [24]). In
the following sections, we introduce the graphical user interface that enables
visualizing and exploring the feature space, and explain how a designer can
use it for insight generation and learning.
4.1. iFEED
The capability to explore the feature space is built as an extension to the
interactive knowledge discovery tool called iFEED [16]. Its goal is to help
engineering designers learn interesting features that drive designs towards a
particular region of the objective space as they interact with the tool. A user
of iFEED can select a group of target designs, and run data mining algo-
rithms to extract the common features that are shared by those designs. The
main interface of iFEED is shown in Fig. 2. It consists of an interactive
scatter plot, which shows the design space populated by thousands of alter-
native designs. When the user hovers his or her mouse over one of the points
in the scatter plot, the information about that design is displayed below the
scatter plot. The displayed information includes values of the objectives and
design decisions of a design.
The scatter plot can help the user select a region of interest in the objective
space. When the user drags the mouse over the scatter plot, designs in the
selected region are highlighted, and they are considered as target solutions
when running the data mining process.
The data mining process is based on the Apriori algorithm, which is one
of the earliest and most popular algorithms developed for association rule

A.N. Author
8
mining [38]. The algorithm has been extended to mine classification associ-
ation rules, which follow the structure : ; <. Here, X is a feature that de-
scribes a design, and C is a class label that indicates whether a certain design
belongs to the target region or not. The data mining returns a list of features
that are shared by the target designs is returned. For more details on the data
mining algorithm, readers are referred to [16].
Fig. 2 The main graphical user interface of iFEED, which consists of a scatter plot showing
the objective space and a display of one design that is currently viewed. Dots highlighted in
cyan represent the target region selected by the user.
4.2. Visualization of Feature Space
The features extracted by running the data mining algorithm have varying
level of “goodness” in explaining the target designs. Such measures can be
defined using various metrics used in binary classification and association
rule mining [23], [24]. In this work, we use two measures of confidence de-
fined as follows.
=,8> ? ; @ / AB.. C D E
AB.. C

Exploring the Feature Space to Aid Learning in Design Space Exploration 9
=,8> @ ; ? / AB.. C D E
AB.. E
Here, S is the set of all designs that are in the target region, and F is the set
of all designs that have the particular feature that is being considered. supp
stands for support, which is defined as:
AB.. F / G :
H
where U is the set of all designs in the database and I indicates the cardi-
nality of the set. Confidence is often used in association rule mining to rep-
resent the strength of a rule [38]. =,8> ? ; @ represents how complete the
feature is in terms of the fraction of the target region that exhibits the feature,
while =,8> @ ; ? represents how consistent or specific the feature is in
explaining only the target region (fraction of designs with the feature that
are in the target region). In fact, because we extract only binary classification
rules, =,8> ? ; @ and =,8> @ ; ? are equivalent to recall and precision,
respectively.
After we calculate both confidence measures for the extracted features,
we can map them in a two-dimensional plane with each axis representing
one of the confidence measures, as shown in Fig. 3. This visualizes the
feature space as we defined at the beginning of this section. In the figure,
each triangle is a feature obtained from the data mining algorithm. The gen-
eral trend in the mined features shows that there is a trade-off between the
two confidences, consistent with relationship often seen between recall and
precision.
The scatter plot displaying the feature space is also implemented as an
interactive plot. When the user hovers the mouse over a feature in Fig. 3,
the designs that have the feature are highlighted in the scatter plot as shown
in Fig. 4. From these figures, the user can get a quick and intuitive sense of
how the feature is distributed within the design space. For example, Fig. 4a
shows a design space, and it highlights a feature whose =,8> C ; E is
high andG=,8> E ; C is low. This feature explains most of the target de-
signs, but it is too general, such that it also covers many other designs that
are not in the target region. In contrast, the feature highlighted in Fig. 4b has
low =,8> C ; E and high =,8> E ; C . The designs that have this fea-
ture fall mostly inside the target region, but only a small portion of it is ex-
plained by this feature.

A.N. Author
10
Fig. 3 Feature space plot, where each axis is one of the confidence measures. Each triangle
represents one feature. The red star represents the utopia point of the feature space.
Fig. 4 Design space highlighting different features. (a) Designs that have the feature with
high =,8> C ; E , and (b) another feature with high =,8> E ; C are highlighted. The
cyan dots are the target designs. The pink dots are the designs that have the feature. The
purple dots are the overlap of those two sets of designs.

Exploring the Feature Space to Aid Learning in Design Space Exploration 11
4.3. Representing and Modifying Features
When the user hovers a mouse over a feature in the feature space plot, a
tooltip appears with the name of the feature. For example, the following text
represents a feature that consists of two base features linked with a conjunc-
tion.
“absent(I) AND present(K)”
In natural language, this can be interpreted as, “Instrument I is not used in
any orbit, and instrument K is used in at least one of the orbits.” However,
in our tool, such representation can only be used to view the extracted fea-
tures and cannot be used to modify the given feature or input a new one.
Fig. 5 Representation of a feature using a graph. The displayed feature consists of 5 base
features linked using both conjunctions and disjunctions. The feature can be interpreted in
text as “absent(I) AND present(K) AND notInOrbit(4000,K,G,B) AND (inOrbit(1000,L) OR
inOrbit(1000,G,A)).”
In order to enable the user to modify and explore other features, we imple-
mented a graphical representation of the feature as shown in Fig. 5. This
representation uses a tree structure, consisting of two types of nodes. A leaf
node represents a base feature, and a logical connective node represents a
logical connective (logical conjunction or disjunction) that links all its chil-
dren nodes. Therefore, the feature shown in Fig. 5 can also be written in text

A.N. Author
12
as: “absent(I) AND present(K) AND notInOrbit(4000,K,G,B) AND (inOr-
bit(1000,L) OR inOrbit(1000,G,A)).” This graphical representation allows
the user to easily see the hierarchical structure within a logical expression
when both conjunctions and disjunctions are used. Moreover, the user can
modify the structure of a feature by changing the location of nodes through
a simple drag and drop. Being able to modify and test different features are
important in order to quickly explore the feature space and gather infor-
mation.
4.4. Search in Feature Space
While the user can explore the feature space by modifying and testing indi-
vidual features, we also implement a local search method to speed up the
exploration process. The local search extends a given feature by adding an
additional base feature either using a conjunction (AND) or a disjunction
(OR). The possible set of base features is set by the user during the problem
formulation step (see section 3), and its size is limited to a small number.
Therefore, the system can test the addition of all possible base features, and
return the new set of features that improve one of the goodness metrics.
To run the local search, the user has to select a feature from Fig. 3 by
clicking on one of the features. Then the user can choose to use either a
conjunction or a disjunction in linking the new base feature to the selected
feature. When a conjunction is used, the feature becomes more specific (the
feature covers less designs), most likely leading to an increase in
=,8> E ; C . On the other hand, if a disjunction is used instead, the feature
becomes more general (the feature covers more designs), thus increasing
=,8> C ; E . The newly generated features are compared with the existing
set of features and only the non-dominated ones are added to the visualiza-
tion. This provides a quick and easy way for the user to explore the feature
space effectively, advancing the Pareto front of the feature space.
5. Evaluation
To test the efficacy of exploring the feature space as a way to improve the
user’s learning, we conduct a controlled experiment with human participants.

Exploring the Feature Space to Aid Learning in Design Space Exploration 13
5.1. Hypothesis and Experiment Conditions
The aim of the experiment is to examine whether exploring the feature space
improves learning, compared to when the user interacts only with the design
space. Learning is defined here as learning the mapping between design de-
cisions and objective values. Therefore, we set our hypothesis as the follow-
ing:
- H1: Exploring the feature space improves a designer’s ability to predict
the performance of a design.
To test this hypothesis, we use a within-subject experiment design and com-
pare the learning in two different conditions: design space vs feature space
exploration. The capabilities of these two conditions are summarized in Ta-
ble 2.
In the first condition, called the design space exploration condition, we
provide only the parts in the graphical user interface that are related to the
design space. For example, the user can inspect each design shown in the
design space (see Fig. 2) and observe the values of design decisions and
objectives. The user can also modify each design by adding/deleting/moving
instruments through drag-and-drop. After modifying the design, the new de-
sign can be evaluated to get the corresponding objective values. In addition,
a local search in design space has been implemented to mimic the local
search in feature space (see section 4.4). The local search is done by ran-
domly sampling four neighboring designs from the currently selected design,
evaluating them, and displaying the newly added designs to the scatter plot.
A neighboring design is defined as a design that can be reached by changing
a single design decision from the currently selected design.
The second condition is called the feature space exploration condition.
Here, the user is still able to inspect individual designs in the design space.
However, other interactions in the design space are not allowed. Instead, the
user can run data mining to obtain an initial set of features visualized in a
similar manner to Fig. 3. Modifying, evaluating, and inspecting each feature
is also enabled through the interface shown in Fig. 4. Moreover, the user can
run a local search to quickly explore the feature space.
The conditions are designed to make the types of interactions as similar
as possible in both conditions. The user can modify, evaluate, and inspect
designs/features, and run local searches in the respective spaces.

A.N. Author
14
Table 2. The capabilities provided in each condition
Capabilities
Design Space
Exploration
Feature Space
Exploration
Inspect designs
Ö
Ö
Modify and evaluate designs
Ö
Local search in the design space
Ö
Run data mining and inspect features
Ö
Modify and evaluate features
Ö
Local search in the feature space
Ö
5.2. Experiment Protocol
Participants are first provided with an interactive tutorial that explains the
design problem as well as all the capabilities of the tool. After the tutorial,
each participant is given two different tasks with different design spaces.
Each task has a pre-specified target region in the design space, and the par-
ticipants are asked to take notes of the features that would be useful to iden-
tify whether an arbitrary design will be in the target region or not. During
the task, the two treatment conditions are presented in random order. A 10-
minute time limit is applied to each task.
After each task is finished, the participants are given a short quiz to meas-
ure how much they have learned during the interaction. All the problems in
the quiz are YES/NO questions, asking the user to predict whether a given
design will be located inside the target region or not. A total of 25 questions
are given.
5.3. Participants
We recruited 38 participants, all of whom are university students. The aver-
age age of the participants is 23.0, with a standard deviation of 4.05. There
were 21 male participants and 17 female participants. 26 students identified
themselves as majoring in the STEM field, and 12 students identified them-
selves as having majors other than STEM.

Exploring the Feature Space to Aid Learning in Design Space Exploration 15
The recruitment was done through two different channels. First, we re-
cruited from the general student population on campus and offered $15 Am-
azon gift cards as compensation.
Second, we recruited students who are taking a graduate-level course on
Systems Architecture. These students were offered a small amount of extra
credit for the class as compensation. The reason for recruiting from this se-
cond group of students was our previous experiences in running a pilot ex-
periment and other experiments with similar interfaces. We have observed
in the past that participants who have not been exposed before to some basic
concepts in design space exploration – such as design decisions, objectives,
features, recall and precision – often struggled to understand the task they
were asked to perform and did not utilize all the capabilities of the tool that
was provided. In addition to our main hypothesis, we also wanted to test if
the participants’ formal training in some of the important concepts has any
interaction effect with their performance in each condition.
6. Result
The test scores of all participants are summarized in Table 3. The average
scores shown in the table represents the percentage of questions that were
answered correctly out of 25 questions asked in each problem set. It shows
that the average scores for both conditions are effectively the same. Running
a paired samples one-tailed t-test gives a p-value of 0.209.
Table 3. Descriptives: All subjects. The Mean score shows the percentage of questions
answered correctly out of 25 questions in each test.
N
Mean
SD
SE
Design Space Exploration
38
72.95
10.86
1.762
Feature Space Exploration
38
74.95
11.22
1.821
A more interesting result is observed when the participants are grouped
based on whether they had the formal training (a first-year graduate course
on system architecture) or not. We ran two-way repeated measures ANOVA
to compare the difference in the mean scores of the two conditions while
also considering the effect of the formal training of the subjects. Table 4
shows the within-subject effects, and Table 5 shows the between-subject
effects. The result shows that there is no statistical significance when we
only consider either the experiment condition or the formal training (taking

A.N. Author
16
the System Architecture class or not) separately. However, there is a signif-
icant interaction effect between the two factors (the p-value is 0.002).
Table 4. Within-subject effects
Sum of
Squares
df
Mean
Square
F
P
Exploration Strategy
14.33
1
14.331
2.580
0.117
Exploration Strategy * Formal Training
61.81
1
61.805
11.128
0.002
Residual
199.94
36
5.554
Table 5. Between-subject effects
Sum of
Squares
df
Mean
Square
F
P
Formal Training
22.91
1
22.907
2.950
0.094
Residual
279.58
36
7.766
Fig. 6 The test scores for each exploration strategy, factored by whether participants
received formal training in system architecture design. The error bar shows the standard
error.
Fig. 6 shows the average test scores after the participants have been divided
into two groups (received formal training or not). These two groups of par-
ticipants exhibit opposite trends in the scores. Those who have not received
any formal training scored better in the quiz (one-tailed paired samples t-
test: t=1.261, p=0.890), when they explored the design space (M=74.09,
SD=12.41) than when whey explored the feature space (M=70.26,
SD=11.11). On the other hand, those who received formal training per-
formed better (one-tailed paired samples t-test: t=3.759, p<0.001), when

Exploring the Feature Space to Aid Learning in Design Space Exploration 17
they explored the feature space (M=82.13, SD=6.906), compared to when
they explored the design space (M=71.20, SD=8.029).
7. Discussion
From the experiment, we find that there is an interaction between the explo-
ration strategy and the formal training. When we compare the average scores
within the participants who received formal training, it is significantly
higher in the feature space exploration condition than in the design space
exploration condition.
The interaction effect between the exploration strategies and the formal
training suggests that those who have been exposed to the basic concepts of
design space exploration find the feature space exploration more useful. In
the Systems Architecture class, the lectures cover a wide range of topics
related to tradespace analysis including decision space, objective space, Pa-
reto dominance, data mining, and sensitivity analysis among others. While
this does not ensure a student’s understanding of these subjects, we can as-
sume that they have been exposed to, and thus familiar with, these topics.
The current experiment result partially supports our hypothesis that explor-
ing the feature space improves learning, with a condition that the user has to
be familiar with the key concepts of design space exploration and have been
trained to reason in an abstract space. This is a promising result, since the
proposed method is mainly intended for professional engineering designers
who are familiar with the concept of design space exploration.
8. Conclusion
This paper introduced the new concept of exploring the feature space,
where the feature space is defined as a set of possible features (combinations
of values for various design decisions). The feature space is visualized in a
2D plane, with each axis representing =,8> ? ; @ and =,8> @ ; ? –
two measures that are equivalent to recall and precision, respectively. The
designer can explore the feature space by modifying and testing different
features and receiving immediate feedback on how the values of the good-
ness of features change in response. Such interaction provides a chance to
learn the how well different features explain a selected region of the design
space. As a result, the designer can easily identify what the important fea-
tures are in driving the performance of design.

A.N. Author
18
To test the effectiveness of this process as a learning tool, we conducted
a human subject experiment. The result showed that the participants who
received formal training in the key concepts of design space exploration per-
formed better when they had a chance to explore the feature space, as op-
posed to when they explored only in the design space. This shows that fea-
ture space exploration has potential to enhance designer’s learning about the
important features and thus help predicting the behavior of a design.
9. References
[1] R. Balling, “Design by shopping: A new paradigm?,” in Proceedings of
the Third World Congress of structural and multidisciplinary optimization
(WCSMO-3), 1999, vol. 1, pp. 295–297.
[2] N. W. Hirschi and D. D. Frey, “Cognition and complexity: An experiment
on the effect of coupling in parameter design,” Res. Eng. Des., vol. 13, pp.
123–131, 2002.
[3] F. Flager, D. J. Gerber, and B. Kallman, “Measuring the impact of scale
and coupling on solution quality for building design problems,” Des.
Stud., vol. 35, no. 2, pp. 180–199, 2014.
[4] P. T. Grogan and O. L. de Weck, “Collaboration and complexity: an
experiment on the effect of multi-actor coupled design,” Res. Eng. Des.,
vol. 27, no. 3, pp. 221–235, 2016.
[5] J. Eddy and K. E. Lewis, “Visualization of Multidimensional Design and
Optimization using Cloud Visualization,” in Proceedings of DETC’02,
2002, pp. 899–908.
[6] G. M. Stump, M. Yukish, T. W. Simpson, and E. N. Harris, “Design
Space Visualization and Its Application To a Design By Shopping
Paradigm,” in Proceedings of DETC’03, 2003.
[7] S. Obayashi and D. Sasaki, “Visualization and Data Mining of Pareto
Solutions Using Self-Organizing Map,” Evol. Multi-Criterion Optim., vol.
LNCS 2632, pp. 796–809, 2003.
[8] X. (Luke) Zhang, T. Simpson, M. Frecker, and G. Lesieutre, “Supporting
knowledge exploration and discovery in multi-dimensional data with
interactive multiscale visualisation,” J. Eng. Des., vol. 23, no. 1, pp. 23–
47, 2012.
[9] B. J. German, K. M. Feigh, and M. Daskilewicz, “An Experimental Study
of Continuous and Discrete Visualization Paradigms for Interactive Trade
Space Exploration,” J. Comput. Inf. Sci. Eng., vol. 13, no. June 2013, pp.
1–12, 2013.
[10] N. Knerr and D. Selva, “Cityplot: Visualization of High-Dimensional
Design Spaces with Multiple Criteria,” J. Mech. Des., vol. 138, no. 9, p.
91403, 2016.
[11] M. J. Diaz, D. D. Fullmer, S. I. Briceno, and D. N. Mavris, “A

Exploring the Feature Space to Aid Learning in Design Space Exploration 19
Collaborative Approach to Complex Systems Engineering using a Web-
based Visual Analytics Framework,” in 55th AIAA Aerospace Sciences
Meeting, 2017, no. January.
[12] R. Agrawal, T. Imielinski, and A. Swami, “Mining Association Rules
between Sets of Items in Large Databases,” ACM SIGMOD Rec., vol. 22,
no. 2, pp. 207–216, 1993.
[13] J. R. Quinlan, C4. 5: programs for machine learning. Elsevier, 1993.
[14] G. Cervone, P. Franzese, and A. P. K. Keesee, “Algorithm quasi-optimal
(AQ) learning,” Wiley Interdiscip. Rev. Comput. Stat., vol. 2, no. 2, pp.
218–236, 2010.
[15] J. C. A. Santos, H. Nassif, D. Page, S. H. Muggleton, and M. J. E.
Sternberg, “Automated identification of protein-ligand interaction fea-
tures using Inductive Logic Programming : A hexose binding case study,”
BMC Bioinformatics, vol. 13, no. 1, p. 162, 2012.
[16] H. Bang and D. Selva, “iFEED: Interactive Feature Extraction for
Engineering Design,” in Proceedings of the ASME 2016 International
Design Engineering Technical Conferences & Computers and Information
in Engineering Conference IDETC/CIE 2016, 2016, pp. 1–11.
[17] S. Watanabe, Y. Chiba, and M. Kanazaki, “A proposal on analysis support
system based on association rule analysis for non-dominated solutions,” in
Proceedings of the 2014 IEEE Congress on Evolutionary Computation,
CEC 2014, 2014, pp. 880–887.
[18] S. Bandaru, A. H. C. Ng, and K. Deb, “Data mining methods for
knowledge discovery in multi-objective optimization: Part A - Survey,”
Expert Syst. Appl., vol. 70, pp. 139–159, 2017.
[19] R. Brachman and H. Levesque, Knowledge Representation and
Reasoning. Morgan Kaufmann, 2004.
[20] J. Feldman, “The Simplicity Principle Concept Learning in Human,”
Psychol. Sci., vol. 12, no. 6, pp. 227–232, 2010.
[21] W. L. W. Li, J. H. J. Han, and J. P. J. Pei, “CMAR: accurate and efficient
classification based on multiple\nclass-association rules,” Proc. 2001
IEEE Int. Conf. Data Min., pp. 369–376, 2001.
[22] B. Liu, W. Hsu, Y. Ma, and B. Ma, “Integrating Classification and
Association Rule Mining,” in Knowledge Discovery and Data Mining,
1998, pp. 80–86.
[23] P.-N. Tan, V. Kumar, and J. Srivastava, “Selecting the right objective
measure for association analysis,” Inf. Syst., vol. 29, no. 4, pp. 293–313,
2004.
[24] M. Sokolova and G. Lapalme, “A systematic analysis of performance
measures for classification tasks,” Inf. Process. Manag., vol. 45, no. 4, pp.
427–437, 2009.
[25] M. Buckland and F. Gey, “The Relationship Between Recall and
Precision,” J. Am. Soc. Inf. Sci., vol. 45, no. 1, pp. 12–19, 1994.
[26] F. Kudo and T. Yoshikawa, “Knowledge extraction in multi-objective
optimization problem based on visualization of Pareto solutions,” in 2012
IEEE Congress on Evolutionary Computation, CEC 2012, 2012.

A.N. Author
20
[27] W. Chen, J. Chazan, and M. Fuge, “How Designs Differ: Non-linear
Embeddings Illuminate Intrinsic Design Complexity,” in Proceedings of
the ASME 2016 International Design Engineering Technical Conferences
& Computers and Information in Engineering Conference IDETC/CIE
2016, 2016, pp. 1–12.
[28] G. Agrawal, C. L. Bloebaum, and K. E. Lewis, “Intuitive Design Selection
Using Visualized n-Dimensional Pareto Frontier,” in 46th
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics &
Materials Conference, AIAA 2005-1813, 2005, no. April, pp. 1–14.
[29] P.-W. Chiu and C. Bloebaum, “Hyper-Radial Visualization (HRV) for
Decision-Making in Multi-Objective Optimization,” in 46th AIAA
Aerospace Sciences Meeting and Exhibit, AIAA 2008-907, 2008, no.
January.
[30] D. J. Walker, R. M. Everson, and J. E. Fieldsend, “Visualizing mutually
nondominating solution sets in many-objective optimization,” IEEE
Trans. Evol. Comput., vol. 17, no. 2, pp. 165–184, 2013.
[31] W. Peng, M. O. Ward, and E. A. Rundensteiner, “Clutter reduction in
multi-dimensional data visualization using dimension reordering,” in
Proceedings of IEEE Symposium on Information Visualization, INFO VIS,
2004, pp. 89–96.
[32] A. H. C. Ng, C. Dudas, H. Bostrom, and K. Deb, “Interleaving
Innovization with Evolutionary Multi-objective Optimization in
Production System Simulation for Faster Convergence,” in Learning and
Intelligent Optimization, Springer Berlin Heidelberg, 2013, pp. 1–18.
[33] R. Jahr, H. Calborean, L. Vintan, and T. Ungerer, “Boosting design space
explorations with existing or automatically learned knowledge,” Proc.
16th Int. GI/ITG Conf. Meas. Model. Eval. Comput. Syst. Dependability
Fault Toler., pp. 221–235, 2012.
[34] X. Yan, Q. Mu, T. W. Simpson, J. Li, and X. Zhang, “Work-Centered
Visual Analytics to Support Multidisciplinary Design Analysis and
Optimization,” in 12th AIAA Aviation Technology, Integration, and
Operations (ATIO) Conference and 14th AIAA/ISSM, 2012, pp. 1–12.
[35] R. S. Michalski, J. Wojtusiak, and K. A. Kaufman, “Intelligent
optimization via learnable evolution model,” Proc. - Int. Conf. Tools with
Artif. Intell. ICTAI, no. 3, pp. 332–335, 2006.
[36] K. Sugimura, S. Jeong, S. Obayashi, and T. Kimura, “Kriging-model-
based multi-objective robust optimization and trade-off-rule mining using
association rule with aspiration vector,” 2009 IEEE Congr. Evol. Comput.
CEC 2009, pp. 522–529, 2009.
[37] D. Selva, “Knowledge-intensive global optimization of Earth observing
system architectures: a climate-centric case study,” SPIE Remote Sens.,
vol. 9241, p. 92411S–92411S, 2014.
[38] R. Agrawal and R. Srikant, “Fast algorithms for mining association rules,”
Proceeding VLDB ’94 Proc. 20th Int. Conf. Very Large Data Bases, vol.
1215, pp. 487–499, 1994.