An Investigation of Engineering Design Cognition and Achievement in Primary School

Abstract

This study examined the design cognition and achievement results of both kindergarten and fourth grade students engaged in engineering design-based instructional activities. Relationships between design cognition and student grade level, as well as quality of student work, were investigated. 30 concurrent think-aloud protocols were collected from individual primary students as they worked in groups to design and make a solution to a design task. The concurrent think-aloud protocols were examined and coded to determine the duration of time the participants devoted to a pre-established set of mental processes for technological problem solving. Significant differences between kindergarten and fourth grade participants were found in the amount of time various cognitive processes were employed. Fourth grade students dedicated significantly more time to the mental processes of Creating, Defining Problems, Measuring, and Testing than kindergarten students. In addition, when examining the think-aloud protocols along with the evaluations of the participant’s design work, it was found that more time devoted to the cognitive process of Managing could be a significant predictor of lower design achievement. These findings can highlight potential areas for improving educational practice based on the cognitive abilities of students at different grade levels and the quality of their design work. As engineering design-based activities become more prevalent for the teaching of STEM-related content and practices, the results of this research, and the employed methodology, may demonstrate a promising practice for better understanding and assessing such education efforts.

Full text

An Investigation of Engineering Design Cognition
and Achievement in Primary School
Greg J. Strimel
1
&Scott R. Bartholomew
1
&Eunhye Kim
1
&Liwei Zhang
1
#Springer Nature Switzerland AG 2018
Abstract
This study examined the design cognition and achievement results of both kindergarten
and fourth grade students engaged in engineering design-based instructional activities.
Relationships between design cognition and student grade level, as well as quality of
student work, were investigated. 30 concurrent think-aloud protocols were collected from
individual primary students as they worked in groups to design and make a solution to a
design task. The concurrent think-aloud protocols were examined and coded to determine
the duration of time the participants devoted to a pre-established set of mental processes for
technological problem solving. Significant differences between kindergarten and fourth
grade participants were found in the amount of time various cognitive processes were
employed. Fourth grade students dedicated significantly more time to the mental processes
of Creating,Defining Problems,Measuring,andTe sting than kindergarten students. In
addition, when examining the think-aloud protocols along with the evaluations of the
participant’s design work, it was found that more time devoted to the cognitive process of
Managing could be a significant predictor of lower design achievement. These findings can
highlight potential areas for improving educational practice based on the cognitive abilities
of students at different grade levels and the quality of their design work. As engineering
design-based activities become more prevalent for the teaching of STEM-related content
and practices, the results of this research, and the employed methodology, may demonstrate
a promising practice for better understanding and assessing such education efforts.
Journal for STEM Education Research
https://doi.org/10.1007/s41979-018-0008-0
*Greg J. Strimel
gstrimel@purdue.edu
Scott R. Bartholomew
sbartho@purdue.edu
Eunhye Kim
kim1906@purdue.edu
Liwei Zhang
zhan1128@purdue.edu
1
Purdue University, West Lafayette, IN, USA

Keywords Primary school.En gineering education .Design cognition .Design assessment
.Concurrent think-aloud protocols
Introduction
Engineering design activities, which require students to work in groups to devise, test,
optimize, and communicate a solution to an open-ended problem, have been a hallmark of
engineering education for decades (Cunningham 2009;Dutsonetal.1997; Engineering
Accreditation Commission 2016; National Academy of Engineering and National
Research Council 2009). More recently, the interest in engineering education for P-12
students has been on the rise (Carr et al. 2012; Strimel et al. 2016), and the presence of
engineering design activities in primary schools has increased (Hegedus 2014). At the
primary level, engineering design-based learning is believed to help young students expand
their often-limited perceptions of engineering. According to Cunningham and Hester
(2007), young students often perceive engineering as just using, fixing, and/or improving
thingswhichisvoidofthepracticesrelatedtoinformeddesign(GrubbsandStrimel2015).
Moreover, engineering design activities are oftenusedtohelpstudentsfostertwenty-first
century skills, such as teamwork and collaboration, as they work together in open-ended
design environments (Hammack et al. 2015). These activities are also believed to support
students in learning the practices necessary to frame problems, plan and develop solutions,
and share ideas with others (McCullar 2015). Studies have indicated that engineering
activities for children can indirectly influence their learning and achievement in science and
mathematics (National Academy of Engineering and National Research Council 2009). In
the long term, exposure to engineering education has been found to be influential to
students in developing career aspirations for engineering and other STEM-related occupa-
tions (Capobianco et al. 2012; Hegedus 2014; National Academy of Engineering and
National Research Council 2009). As a result, multiple attempts have been made to
integrate engineering education into primary school curricula. For example, curriculum
programs such as Engineering is Elementary from the Boston Museum of Science, Launch
from Project Lead the Way,andtheTechnology, Engineering, Environment, Mathematics,
and Science (TEEMS) by Engineering byDesign™can be found in schools across the
country. In addition, engineering design has become a core component of the standards for
both technology education (ITEA/ITEEA, 2000/2002/2007) and science education (NGSS
Lead States 2013) in the United States.
In this context, it becomes important to assist educators in using developmentally
appropriate curriculum and pedagogical strategies to ensure students are able to build
capabilities to employ engineering design practices to produce viable solutions to
authentic problems. However, engineering design can be challenging to understand,
teach, and evaluate as many efforts to infuse engineering are void of empirical
foundations around how students think as they engage in engineering design (Dym
et al. 2005). Therefore, examining engineering design cognition at various levels of
education has become a topic of interest to STEM education practitioners and re-
searchers in hopes to provide recommendations for effective teaching, the suitable
scaffolding of engineering design experiences, and ultimately, enhancing student
design abilities (Lammi and Becker 2013). However, minimal agreement exists on
how primary students design and limited examinations have explored more effective
G. J. Strimel et al.

ways to bridge design cognition research with teaching and learning strategies (Grubbs
2013). Moreover, there has been limited research investigating design cognition at the
primary level (Kelley et al. 2015). These concerns may be further exacerbated, as most
research does not examine design cognition with respect to the evaluation of the
outcomes of both designing and making. Much of design cognition research is limited
to only identifying the cognitive processes involved with developing an initial solution
concept without the creation of a prototype. While previous design cognition studies
have been beneficial for identifying procedures in which participants employ to
develop design ideas, the lack of research relating their cognitive activity to the tangible
outcomes of their design process may (a) lead researchers to overlook the mental
strategies involved in making and optimizing prototypes and (b) limit opportunities
to uncover potential design heuristics for improved design achievement.
As Atman and Bursic (1998) explain, examining both design process and product can
enable the uncovering of potential relationships between a student’s thinking and the
outcomes of their process. They further describe that understanding this potential rela-
tionship can help to recognize successful or unsuccessful practices in engineering design.
As described by Wilson et al. (2013), this type of research may provide useful heuristics
for teachers who seek to incorporate engineering practices within their classrooms.
Therefore, related research efforts could support the development of instructional inter-
ventions aimed toward improving the teaching of engineering design and developing
effective problem solvers. As a result, the authors conducted a study to investigate the
cognitive activities of both kindergarten and fourth grade students as they designed and
prototyped a solution to an engineering design task. The research was conducted with the
intent of comparing student’s design processes with the evaluation of their design work to
highlight potential indicators for improving design practice. In addition, the research
design facilitated the comparison of these results with students’grade levels. In doing
so, the findings presented in this article can potentially help identify ways in which to
better scaffold the teaching and learning of engineering design at the primary level.
Design Cognition Research Review
In tandem with the increased emphasis on engineering design, at all levels of education,
attempts to establish effective ways of teaching the practices of engineering design have
similarly increased (Dym et al. 2005). As part of these efforts, engineering design
cognition studies have been conducted across multiple disciplines and professions
(Grubbs and Strimel 2016; Lammi and Gero 2011). Design cognition studies com-
monly employ a concurrent think-aloud protocol procedure which is used to capture a
participant’s actions, along with their own verbal interpretation of their thought pro-
cesses as they perform those actions, while solving a predetermined design task (Kelley
et al. 2015). The resulting verbal interpretations are then analyzed using a verbal
protocol analysis technique. This technique involves segmenting the collected design
protocol into individual cognitive tasks and then applying a previously derived coding
scheme over each segment of either a video recording or transcription of a design
session (Purcell et al. 1996). The coded data can then be used to interpret the processes
and procedures a participant employs to develop a solution to the design task. However,
much of this type of research may provide an incomplete picture of design cognition, as
limited studies examine participants beyond the development of a design concept,
An Investigation of Engineering Design Cognition and Achievement in...

through the making of a prototype. Therefore, an incomplete understanding of the
cognitive processes involved in both designing and making may exist which can also
limit the means for comparing a participant’s design process with the assessment of
their process. Further, many of these studies collect student data from group-settings;
which, although beneficial, captures group discussions and limits collecting an
individual’s thought process(es) while designing. While both settings (i.e., indi-
vidual and group) are worthy of study, this effort focuses on individual student
cognitive processes while engaged in a design task. Therefore, the participants
were individually equipped with point-of-view cameras and trained to think aloud
while working within their groups to complete the design tasks. As such, only the
individual thought processes of the participants wearing the cameras were collect-
ed and then coded solely from their own perspective.
In addition to the previously highlighted research gaps, design cognition studies
have traditionally focused on how engineers, architects, or post-secondary engineering
students solve an engineering design problem (Cross 2001; Grubbs 2013; Lammi and
Becker 2013). However, as the emphasis on P-12 engineering education continues to
increase (Grubbs and Strimel 2016), the body of P-12 engineering design cognition
studies has been growing with the purpose of understanding and improving pupils’
engineering design thinking capabilities. Through a review of literature, Grubbs et al.
(2018) identified 15 published design cognition studies involving unique data collected
from P-12 students between 1995 and 2016. Of these studies, 14 investigated the
design cognition of secondary students (grades 6–12) while only one study (Kelley
et al. 2015) examined how primary students (grades K-5) distributed their cognitive
processes when solving engineering design problems.
Kelley et al. (2015) studied the design thinking of students in grades five and six while
they engaged in engineering design tasks. As the researchers examined the collected think-
aloud protocols, they determined that students in these grades were able to transfer the
learning of the systematic approaches to engineering design—such as problem identification,
analysis, and refinement of design solutions—to subsequent design contexts. However, their
analysis indicated that fifth and sixth grade students might not fully understand, appreciate,
or value the iterative nature of design, as several students did not conduct multiple iterations.
These researchers also found that participants tended to emphasize the brainstorming of
design solutions. Therefore, they posited that teaching students how to frame design
problems and manage brainstorming sessions are critical to students’success as designers.
Also, Kelley et al. (2015) suggested that more research using the concurrent think-aloud
protocol procedure should be conducted to further investigate how primary students learn
and how teachers use design experiences to enhance learning.
Statement of Problem
While the teaching of engineering design continues to increase in primary schools,
uncertainty exists as to how students, at various grade levels, navigate a multi-
faceted engineering design problem. Specifically, limited research is available that
examines how students at the primary education level distribute their cognitive
efforts while completing an engineering design task. Additionally, little research
exists that examines cognition during both the designing and making of a solution
for primary students. With greater insight to design cognition, educators may be
G. J. Strimel et al.

better equipped to manage the difficulties in planning and assessing student
abilities for producing viable solutions to engineering design problems and
researchers can examine the relationships between student design thinking
across grade levels. Hence, the purpose of this study was to identify the
cognitive processes employed by kindergarten and fourth grade students to
solve engineering design problems in an effort to expand the understanding
of how these pupils cognitively navigate an engineering design process from
design conception through the production of a prototype. In addition, this
research was intentionally designed and conducted in such a manner to compare
a student’s thinking process with the evaluation of their design work to enable
the investigation of potential significant cognitive predictors of success in
designing and making.
Addressing this research gap can be of specific importance as engineering
design activities have now become widespread within primary school
coursework—specifically following the introduction of engineering design with-
in the Next Generation Science Standards (NGSS Lead States 2013) and along
with the continued implementation of national engineering curriculum programs
within primary schools. While theories about cognitive development, such as
Piaget’s Cognitive Development Theory, can suggest that children might be
unable to operate in an ill-structured design space, others (Cohen 2002;
Crossland 2015; Sutherland 1992;Weiten1992) have contended that these
theories miscomprehend the development of children as their learning capabil-
ities and biological maturation can vary widely. According to these incongrui-
ties, the authors believe that exploring how kindergarteners and fourth grade
students cognitively navigate open-ended design tasks may shed light on pri-
mary student design thinking and assist in identifying potentially-useful peda-
gogical approaches for improving student achievement through the better scaf-
folding of design activities.
Research Objectives and Questions
This study was driven by the overarching objectives of (1) identifying ways in
which primary students cognitively navigate an engineering design task from
design conception through prototype production and (2) determining ways in
which to inform P-12 engineering education by examining design cognition in
respect of student grade level and design achievement. To achieve these objectives
the following research questions were generated:
Research Objective 1: identify ways in which primary students cognitively
navigate an engineering design task from design conception through prototype
production.
RQ1a: What cognitive processes do kindergarten students employ to design and
make solutions to engineering design problems?
RQ1b:What cognitive processes do fourth grade students employ to design and
make solutions to engineering design problems?
An Investigation of Engineering Design Cognition and Achievement in...

Research Objective 2: determine ways in which to inform P-12 engineering
education by examining design cognition in respect of student grade level and
design achievement.
RQ2a:What differences, if any, exist between the design cognition of kindergar-
ten and fourth grade students?
RQ2b:What are the potential cognitive indicators, if any, of design achievement
for kindergarten and fourth grade students?
Methods
Participants and Design Tasks
Data from 30 concurrent think-aloud protocols (18 kindergarten participants, ages 5–7,
and 12 fourth grade participants, ages 8–10) were collected across four classrooms (2
kindergarten, 2 fourth grade) while students engaged in one of six design tasks—that
were similarly structured but contextually different. The data collection took place in
one small suburban school district located in the Midwestern United States during the
spring of 2017. At the time, the district was composed of a mainly middle-class
population and served approximately 10,000 students with only a small percentage
(22%) of those receiving free/reduced meals. Following the receipt of Institutional
Review Board approval for research with minors, four teachers (two fourth grade
teachers, two kindergarten teachers) from one primary school in this district were
recruited for participation based on their STEM integration efforts involving engineer-
ing design activities. Each teacher was recommended by the school instructional
excellence coach and had similar years of experience, licensure qualifications, and
experience implementing engineering design activities in their classroom. All teachers
were Caucasian, had been teaching for more than 5 yrs, and had only just begun
integrating engineering design tasks into their classrooms. Prior to the study, each
teacher was trained on the various design activities and a member of the research team
was present during each class to ensure fidelity of implementation throughout the study.
Each teacher in this study led their class through three open-ended engineering
design activities (see Table 1), which involved students working in groups to employ an
Belementary level^design process (steps including; Ask, Explore, Model, Evaluate,
and Explain) to resolve the posed problems. The students, in each grade level, were
presented with a design problem from a book currently being read and discussed in
class [e.g., Pink and Say by Patricia Polacco (1994) for fourth grade students and My
Bug Box by Pat Blanchard and Suhr (1999) for kindergarten students] and asked how
they might solve the problem highlighted in the literature [e.g., How can we create a
device to help carry a wounded soldier (Pink and Say) or How can we create a box to
keep bugs safe from the toad (My Bug Box)?]. Each task required the students to design
and make a physical prototype/model while also completing a design worksheet to
document their process. Students worked in teacher-assigned groups of 2–3toidentify
the criteria and constraints for the presented problem from the in-class reading, explore
G. J. Strimel et al.

important questions, brainstorm ideas, develop possible solutions, and evaluate the
effectiveness of their solution. As each group attempted to solve the problem, they
completed the design worksheet (see Fig. 1), which was developed collaboratively by
the teachers and researchers involved in this project. To complete the challenge, the
students were provided with several low-fidelity prototyping materials such as craft
sticks, straws, paper, masking tape, etc. to produce their solutions. Each design session
centered on one problem and lasted 31 min on average. While working on the problem,
students were allowed to move freely about the classroom, solicit help from peers or the
teacher, and obtain additional supplies, if needed, with permission from the teacher. A
sample of the student prototypes and completed design worksheets are shown in Fig. 2.
Concurrent Think-Aloud Protocol Collection and Analysis
This study employed a multiple exploratory case study approach using concurrent
think-aloud protocol analysis to identify the cognitive processes used by both
kindergarten and fourth grade participants as they worked to develop solutions to
engineering design problems. Atman and Bursic (1998), when studying under-
graduate students, posited that using concurrent think-aloud protocol analysis was
a valuable process for understanding the cognitive processes students use when
developing a design solution. Although different in age, and development, the
authors postulate that younger students will be similarly capable of using concur-
rent think-aloud protocol. This position is based upon the rationale for the use of
think-aloud methods with triads of primary students that was developed by Kelley
et al. (2015). They state that concurrent think-aloud approaches can serve as an
easier method for engaging young participants in eliciting their thought processes
Table 1 Design challenges
Grade Design Task In Class Reading
Kindergarten How can we keep the bugs safe from the toad?
•Design and build a bug box that does not allow
frogs in but allows bug in/out
My Bug Box by Pat Blanchard
and Suhr (1999)
Kindergarten How can we help the dragonfly’s legs that were
unable to catch prey?
•Design and build a device to help the dragonfly
catch prey
Dragonflies by Margaret Hall (2006)
Kindergarten How can we help toad come out during the day?
•Design and build a device to help the toad come
out into the sunlight without getting too hot
Toad s by Alyse Sweeney (2010)
4th Grade How can we help Pink to carry Say?
•Design and build a device to help carry a
wounded soldier
Pink and Say by Patricia
Polacco (1994)
4th Grade How can we conceal a secret message?
•Design and build a device to conceal and carry a
secret message for the army
Great Women of the Civil War
by Molly Kolpin (2014)
4th Grade How can we make Civil War soldiers stay healthy
and strong?
•Design and build a device to help protect soldiers
The Terrible, Awful Civil War
by Kay Mechisedech (2011)
An Investigation of Engineering Design Cognition and Achievement in...

as it enables them to use their own language within their own dialog, communi-
cate their design thoughts in a genuine way rather than through a more structured
elicitation technique, removes the risk of the researcher imposing upon their
natural thought processes, and allows them to express their design thinking while
engaged in the act of designing.
Fig. 1 Example design worksheet for one of the assigned design tasks
Fig. 2 Student design worksheet and prototype examples (Left: 4th grade, Design and build a device to help
protect soldiers; Right: 4th grade, Design and build a device to help carry a wounded soldier)
G. J. Strimel et al.

Accordingly, the participants in this study were asked to verbalize their thoughts
while engaged in an engineering design task and, to facilitate data collection,
they were equipped with point-of-view cameras that enabled the recording of verbal
protocols as well as their non-verbal cues—known as observational protocol. The
combination of think-aloud and observational protocols in addition to the participants’
design artifacts were used in data triangulation during the data analyses (Cross 2004).
While students in these classes were required to work in groups for each design task,
the concurrent think-aloud protocols were collected from one participant per group.
These participants were selected by the teachers based on their comfort with wearing
the point-of-view cameras—which is recognized as a limitation to this study. Once the
participants were selected, following the approved Institutional Review Board proto-
col, and equipped with the cameras, they were tasked to think aloud while completing
the design task.
Following data collection, the recorded think-aloud protocols were segmented
into individual utterances and coded using the 17 mental processes for technolog-
ical problem solving definedandvalidatedbyHalfin(1973). This coding scheme
was selected as it was developed through a content analysis of practicing de-
signers, innovators, technologists, and engineers and validated through a Delphi
studybyHalfin(1973)andrevalidatedbyWickleinandRojewski(1999). The
operational definitions of these 17 mental processes and sample utterances are
provided in Table 2. However, based on a review of literature, the mental process
of Modeling was determined by the researchers to be too similar to the other codes
of Model/Prototype Constructing and Designing. The inability to differentiate
between these codes was stated in the original work by Halfin (1973). As a result,
Modeling was not used and the actions that could be considered Modeling were
coded as either Designing or Model/Prototype Constructing. Also, it is important
to note that the mental processes identified by Halfin (1973)maynotallbe
represented in the thought processes of young learners as they may beyond their
level of development. While this can be a limitation to this study, the results of
this research can highlight the mental processes that may not be developmentally
appropriate to expect with the different ages of the learners.
Additionally, to further clarify coding decisions, enhance transparency, and support
replication, this study organized a hierarchy (see Fig. 3) of the16 cognitive processes
based on a study by Strimel (2014) which highlighted patterns of cognitive processes
within a conceptual engineering design model. According to his study, the processes of
Communicating and Managing are often employed throughout an entire engineering
design task to control and converse the actions and outcomes of the problem-solving
episode. Next, the processes of Analyzing, Designing, Creating, Model/Prototype
Constructing,andTe s t ing are predominately employed during a particular phase of
the design process. Lastly, the mental processes of Experimenting, Observing, Com-
puting, Measuring, Interpreting Data, Questioning/Hypothesizing, Predicting, Defining
Problems, and Visualizing are often employed parsimoniously throughout an engineer-
ing design task. Therefore, this study assigned Communicating and Managing as level
1 codes; Analyzing, Designing, Creating, Model/Prototype Constructing,andTest ing as
level 2 codes; and Experimenting, Observing, Computing, Measuring, Interpreting
Data, Questioning/Hypothesizing, Predicting, Defining Problems, and Visualizing as
level 3 codes. These codes were layered to assist with coding specifically when
An Investigation of Engineering Design Cognition and Achievement in...

Table 2 Halfin’s 17 mental processes for technological problem solving
Cognitive Process Definition Sample Utterance Indicating the Cognitive Process
Analyzing This is the process of identifying, isolating, taking apart, breaking down, or performing similar
actions for the purpose of setting forth or clarifying the basic components of a phenomenon,
problem, opportunity, object, system, or point of view.
[While reviewing prototype testing data BI believe
I have a design flaw which is this right here^
Communicating This is the process of conveying information (or ideas) from one source (sender) to another
(receiver) through a media using various modes (The modes may be oral or written or
pictures or symbols, or any combination of these.).
[While recording testing information]BLet’s write
down the original sample number.^
Computing This is the process of selecting and applying mathematical symbols, operations, and
processes to describe, estimate, calculate, quantify, relate, and/or evaluate in the
real or abstract numerical sense.
[While adding dimensions to a design sketch]B
At 14 in. intervals, I will need 2 of them.^
Creating This is the process of combining the basic components or ideas of phenomena, objects, events,
systems, or points of view in a unique manner that will better satisfy a need, either for the
individual or for the outside world.
[While looking at multiple design sketches]B
I should combine both ideas.^
Defining problem(s) This is the process of stating or defining a problem, which will then enhance the investigation
leading to an optimal solution. It is transforming one state of affairs to another desired state.
[While reading a design brief]BWhat does the
device need to do?^
Designing This is the process of conceiving, creating, inventing, contriving, sketching, or planning by
which some practical ends may be affected, or proposing a goal to meet the societal needs,
desires, problems, or opportunities and do things better. Design is a cyclic or iterative process
of continuous refinement or improvement.
[While drawing in a notebook]BLet’sjust
create a sketch now.^
Experimenting This is the process of determining the effects of something previously untried in order to test the
validity of a hypothesis, to demonstrate a known (or unknown) truth, or to try out various
factors relating to a particular phenomenon problem, opportunity element, object,
event, system, or point of view.
[While creating a physical prototype]BLet’s
see what works better for the base other
then the foam I have.^
Interpreting data This is the process of clarifying, evaluating, explaining, and translating to provide
(or communicate) the meaning of particular data.
[While collecting prototype test results]BIcan
deduct that this way of sampling is not working^
Managing The process of planning, organizing, directing, coordinating, and controlling the inputs
and outputs of the system.
[While preparing to produce a prototype]BIwill
move all of our stuff back to the table.^
Measuring This is the process of describing characteristics (by the use of numbers) of a phenomenon, problem,
opportunity, element, object, event, system, or point of view in terms that are transferable.
[While preparing materials for their prototype]B
I know it needs to be at least this big.^
G. J. Strimel et al.

Tab l e 2 (continued)
Cognitive Process Definition Sample Utterance Indicating the Cognitive Process
Modeling* This is the process of producing or reducing an act or condition to a generalized construct that
may then be presented graphically in the form of a sketch, diagram, or equation; physically in
the form of a scale model or prototype; or in the form of a written generalization.
NA
Model/ Prototype
Constructing
This is the process of forming, making, building, fabricating, creating, or combining parts to
produce a scale model or prototype.
[While manipulating materials]BIneedapair
of scissors to cut a hole in the bottom.^
Observing This is the process of interacting with the environment through one or more of the senses (seeing,
hearing, touching, smelling, or tasting). The senses are utilized to determine the characteristics
of a phenomenon, problem, opportunity, element, object, event, system, or point of view.
[While looking at their prototype when testing] B
I can tell I’m not doing any better.^
Predicting This is the process of prophesying or foretelling something in advance, anticipating the future
based on special knowledge.
[While looking at a design sketch]BThat’s
not going to work.^
Questioning/
Hypothesizing
Questioning is the process of asking, interrogating, challenging, or seeking answers related
to a phenomenon, problem, opportunity, element, object, event, system, or point of view.
[While posing a question to themself]BWhat
materials will work best?^
Testing This is the process of determining the workability of a model, component, system, product,
or point of view in a real or simulated environment to obtain information for clarifying
or modifying design specifications.
[While preparing to test their prototype] B
Okay let’s see how this works!^
Visualizing This is the process of perceiving a phenomenon, problem, opportunity, element, object, event,
or system in the form of a mental image based on the experience of the perceiver.
[While examining a design sketch]BIf I
poke holes in the cup here, the water
will run into there.^
Note. Definitions for each mental process were derived from the works of Halfin (1973), Hill and Wicklein (1999), and Wicklein and Rojewski (1999).
*Modeling was not used as a code when analyzing the data as it was deemed to be too similar to other codes
An Investigation of Engineering Design Cognition and Achievement in...

multiple cognitive processes were thought to be occurring simultaneously. These levels
do not indicate complexity as each process can potentially have its own levels of
complexity. However, based on the analysis of previous design protocols and design
cognition research, the order of these codes indicates what cognitive process could be
considered at the forefront of the participants’thoughts. Therefore, in the hierarchy of
the 16 codes, priority for a coding decision, where multiple processes could be
considered as happening simultaneously, was given to the higher level codes as they
were seen as the most relevant to the actions of the participant.
To enable the coding process, the researchers used NVivo software, which permits a
researcher to segment and code the recorded protocols simultaneously while observing
the video recordings (Mentzer et al. 2015). Also, to ensure the reliability of this
procedure, two independent coders were involved in the coding process (Mentzer
2014). After each coder coded each segment of every protocol, agreement rates and
Cohen’s Kappa values were calculated between the two coders’results (Cohen 2007).
Across all codes for the three design projects for each grade level, the average of
agreement rates was 97.67%, and the average of Cohen’s Kappa values was 0.72,
which indicates substantial agreement of the analysis (McHugh 2012).
Design Achievement Analysis
The design worksheets and prototypes were evaluated by the teachers using the
Adaptive Comparative Judgment (ACJ) assessment technique—a method that has
recently gained attention as a reliable and valid method of evaluating open-ended
design problems (Bartholomew 2017; Bartholomew et al. 2017;Kimbell2012; Seery
et al. 2016; Bartholomew et al. 2018). Through the ACJ software CompareAssess,
teachers assessed students design portfolios at the conclusion of the assignment (the
portfolios consisted of the design worksheet and an image of their prototype). This ACJ
Fig. 3 Hierarchy of Halfin’s mental processes for the purpose of guiding the coding of design protocols
G. J. Strimel et al.

process involved teachers viewing two student portfolios and, rather than assigning a
grade through micro-scoring using a rubric, they were asked to bear in mind the assign-
ment criteria and holistically chose which portfolio was better. The specific assignment
criteria, which was provided to teachers and students from the onset of the assignment and
which was used by teachers in making their paired comparative judgments, was (1)
demonstration of collaboration, (2) design process, and (3) the final prototype creation.
Recognizing that this approach to assessment was novel to these teachers, a training
session was held prior to the research to acquaint them with the method. Further, post-
study interviews with teachers revealed that they were comfortable with the approach
and felt confident in the results produced. ACJ, as an approach to assessment has
repeatedly demonstrated an improved ability to achieve very high reliability levels over
traditionally used approaches, such as rubrics, as the process centers on comparative
decisions rather than a series of subjective decisions (e.g., based on multiple criteria
from a rubric). In this study, as the teachers progressed through a series of paired
comparisons, choosing the better design portfolio each time, an embedded algorithm
using rash-model statistics produced a rank order of all included student work. The rank
order provides a potentially better means for analyzing achievement over scores
collected through a rubric. When using rubrics, teachers can assign multiple students
similar scores while their design actions/outcomes may be different. However, a rank
order provides a defined sequence of achievement from the best to the worst based on
the collective consensus of those judging the work. Therefore, the resulting rank order
from the ACJ process was used to represent participants’design achievement and
applied in the subsequent data analysis related to the research questions.
Findings
Research Objective 1: Identify how Primary Students Cognitively Navigate
an Engineering Design Task
RQ1a: What Cognitive Processes do Kindergarten Students Employ to Design and Make
Solutions to Engineering Design Problems? This question was answered by coding
video recordings of participants thinking aloud during engineering design tasks. On
average, kindergarten participants completed the tasks within 30 min and 45 s.
Throughout the engineering design activities, the top three most employed mental
processes were Model/Prototype Constructing,Managing,andObserving.Model/Pro-
totype Constructing consumed 34.4% of the participant’s time on average, which
mostly consisted of physically manipulating materials for making prototypes. Next,
Managing consumed 20.7% of the participant’s time on average and consisted mostly
of finding/contemplating materials and tools, planning next steps, coordinating prior-
ities, and directing a teammate to do something. Lastly, Observing consumed 13.3% of
the participant’s time on average and consisted mostly of examining/handling materials
and watching the actions of a teammate or a situation to understand the phenomenon.
After coding the design protocols, it was determined that the least used mental
processes for the kindergarten participants were Testing (0.1%), Creating (0.3%),
Measuring (0.4%), Defining Problems (0.6%), Questioning (1.2%), and Predicting
(1.2%); each of these mental processes consumed less than 2% of the participant’time
An Investigation of Engineering Design Cognition and Achievement in...

on average. Of note, the mental processes of Computing and Experimenting were not
observed with the kindergarten participants. The average time dedicated to each
cognitive process throughout the design task and standard deviation can be found in
Tab le 3.Figure4provides a graphical representation of the division of time dedicated
to each cognitive process, on average, throughout the design tasks.
RQ1b: What Cognitive Processes do Fourth Grade Students Employ to Design and Make
Solutions to Engineering Design Problems? This question was answered by coding
video recordings of participants thinking aloud during engineering design tasks.
On average, fourth grade participants completed the tasks within 27 min and 32 s.
Throughout the design activities, the top three most used mental processes were
Model/Prototype Constructing,Managing,andAnalyzing.Model/Prototype Con-
structing consumed 26.3% of the fourth-grade participant’sdesigntimeonaver-
age, followed by Managing, which consumed 18.0% of their time, and Analyzing,
which consumed 17.9% of the participant’s time on average. The Analyzing time
consisted mostly of gathering and examining information related to the design
problem, listening to peers and teachers to understand design decisions, and
clarifying possible explanations of a phenomenon.
After coding the design protocols, it was determined that the least used mental
processes for the fourth grade participants were Creating (1.0%), Visualizing (1.3%),
Table 3 Mean cognitive process distribution for the engineering design activities
Kindergarten Participants
(NK=18)
4th Grade Participants
(N4th =12)
Total Participants
(N=30)
Mean
(Sec.)
Standard
Deviation
(Sec.)
Mean
(Sec.)
Standard
Deviation (Sec.)
Mean
(Sec.)
Standard
Deviation
(Sec.)
Analyzing 183.73 104.45 259.93 200.03 214.21 151.70
Communicating 116.68 64.39 109.69 42.71 113.89 55.99
Computing 0.00 0.00 0.00 0.00 0.00 0.00
Creating 4.36 6.45 14.36 11.57 8.36 10.00
Designing 66.36 53.52 87.33 51.62 74.74 52.91
Defining Problems 9.72 11.35 40.98 35.25 22.22 28.10
Experimenting 0.00 0.00 0.00 0.00 0.00 0.00
Interpreting Data 0.38 1.63 0.00 0.00 0.23 1.26
Managing 315.28 145.40 261.01 95.17 293.57 128.68
Measuring 6.35 10.63 19.57 16.26 11.64 14.49
Model/Prototype 523.22 218.62 382.33 223.22 466.87 227.70
Observing 202.14 146.57 155.58 98.66 183.52 129.70
Predicting 18.91 30.25 21.50 22.89 19.95 27.15
Questioning 17.79 17.99 36.92 37.04 25.44 28.30
Testing 1.16 4.93 45.56 64.89 18.92 45.83
Visualizing 53.85 54.95 19.29 15.87 40.03 46.50
Total Time 1844.23 532.27 1651.08 402.51 1766.97 486.61
G. J. Strimel et al.

Measuring (1.3%), and Predicting (1.5%). Each of these mental processes consumed
less than 2% of the participant’time on average. Of note, similar to the kindergarten
students, the mental processes of Computing and Experimenting were not observed
with the fourth grade participants. The mean time dedicated to each cognitive process
throughout the design task and standard deviation can also be found in Table 3. Figure 4
includes a graphical representation of the division of time dedicated to each cognitive
process, on average, throughout the design tasks.
Research Objective 2: Examine Design Cognition in Respect of Student Grade Level
and Design Achievement
RQ2a: What Differences, if any, Exist between the Design Cognition of Kindergarten
and Fourth Grade Students? To address this question, the researchers performed a
comparison between kindergarten and fourth grade participants’design cognition.
Before choosing a method for the comparison, the researchers tested normality and
the homogeneity of variance assumptions of the data. The test results demonstrated
violation of these assumptions. Therefore, recognizing that the Mann-Whitney test is
less sensitive to these violations, the researchers applied the non-parametric test for
comparing the two groups. The Mann-Whitney test indicated that there was no
significant difference in the total time used for designing between the kindergarten
and fourth grade participants. However, the test results did indicate that fourth grade
participants devoted significantly more time to Creating (Kindergarten M=4.36 s,
Fourth grade M=14.36 s, U=50.00, p= .011), Defining Problems (Kindergarten =
9.72 s, Fourth grade = 40.98 s, U=51.00, p=.015), Measuring (Kindergarten M=
6.35 s, Fourth grade M=19.57s,U=52.50,p=.012), andTe st ing (Kindergarten M=
1.16 s, Fourth grade M=45.56 s, U=49.50, p= .001) than the kindergarten
Mean Division of Time Per Cognitive Process
for the Engineering Design Activities
Kindergarten Participants
Total Time: 30 mins 44.2 secs
4th Grade Participants
Total Time 27 mins 31.1 secs
Model/Prototype
34.4%
Managing
20.7%
Observing
13.3%
Analyzing
12.1%
Communicating
7.7%
Designing
4.4%
Visualizing
3.5%
Predicting
1.2%
Questioning
1.2%
Defining
Proble ms
0.6%
Measuring
0.4%
Creating
0.3%
Testing
0.1%
Model/Prototype
26.3%
Managing
18.0%
Analyzing
17.9%
Observing
10.7%
Communic ating
7.5%
Designing
6.0%
Testing
3.1%
Defining
Proble ms
2.8%
Questioning
2.5%
Predicting
1.5%
Measuring
1.3% Visualizing
1.3% Creating
1.0%
Fig. 4 Mean division of time dedicated to each cognitive process, on average, throughout the design tasks
An Investigation of Engineering Design Cognition and Achievement in...

participants. The greatest difference in cognitive processing time between the two
groups was found in the mental process of Tes tin g (Kindergarten M= 1.16 s; Fourth
grade M= 45.56 s). Although most kindergarten participants ended their design session
once a prototype was made, the fourth grade participants revealed the design practices
of Testing, which was demonstrated by examining the workability and/or durability of
their prototypes and then attempting to improve the prototypes. The complete Mann-
Whitney analysis results are presented in Table 4.
RQ2b: What Are the Potential Cognitive Indicators, if any, of Design Achievement for
Kindergarten and Fourth Grade Students? To address this question, the researchers
examined the correlations between the achievement of the 30 participants and the
cognitive processes they revealed during the design session. The achievement was
determined by the teachers’evaluations of the participants’project deliverables (pro-
totypes and design worksheets) using the ACJ assessment technique, which resulted in
a rank order of student projects. The analysis between the rank orders and cognitive
processing times of the primary students revealed that project rankings were signifi-
cantly correlated with the amount of time dedicated to the cognitive process of
Managing (Spearman’srho=.442, p= .014). This finding indicates that a project’s
rank tended to increase (meaning the project’s quality decreased) as participants
Table 4 Mann -Whitney analysis between kindergarten and fourth grade participants
Mean Time Dedicated to Each Cognitive
Process (sec.)
Mann-Whitney U Asymp. Sig.
(2-tailed)
Kindergarten
(N= 18)
4th Grade
(N= 12)
Analyzing 183.73 259.93 88.00 0.397
Communicating 116.68 109.69 104.00 0.866
Computing 0.00 0.00 N/A N/A
Creating 4.36 14.36 50.00 0.011*
Designing 66.36 87.33 79.00 0.220
Defining Problems 9.72 40.98 51.00 0.015*
Experimenting 0.00 0.00 N/A N/A
Interpreting Data 0.38 0.00 102.00 0.414
Managing 315.28 261.01 83.00 0.290
Measuring 6.35 19.57 52.50 0.012*
Model/Prototype 523.22 382.33 71.00 0.117
Observing 202.14 155.58 94.00 0.553
Predicting 18.91 21.50 91.00 0.466
Questioning 17.79 36.92 64.00 0.063
Testing 1.16 45.56 49.50 0.001**
Visualizing 53.85 19.29 66.00 0.075
Total Time 1844.23 1651.08 93.00 0.525
**Significant at the 0.01 level (2-tailed)
*Significant at the 0.05 level (2-tailed)
G. J. Strimel et al.

dedicated more time to the process of Managing. During the design session, partici-
pants had to allocate their given time to multiple cognitive strategies, so if they spent a
certain amount of time on a cognitive process, then they may lose the amount of time
for other, maybe more critical cognitive processes. Thus, the results may suggest that
students need to use their time Managing more efficiently and distribute their cognitive
efforts to other metal strategies to achieve better project results. Based on the observa-
tions of participants and the limited time they devoted to the process of Defining
Problems, the authors found that a lack of problem framing and scoping may have
resulted in the increased time needed for Managing their design efforts. As such, efforts
to assist students in effectively Defining Problems may be beneficial.
Besides Managing, there were no statistically significant correlations between
participants’assessment results and any other cognitive processes employed
during the design session. However, Creating, Defining Problems, Interpreting
Data, Measuring, Observing, Predicting, Testing, and Visualizing were all corre-
lated with better ranks, although these relationships were not statistically signif-
icant. It should be noted that a negative correlation suggests a positive relationship
between cognitive process and achievement as a lower rank signifies a better
achievement level. Table 5presents the correlation analysis results between
cognitive processes and participant achievement.
Table 5 Correlation between participants’cognitive processes and design achievement
ACJ Rank
Spearman’s rho Sig. (2-tailed)
p
Analyzing .083 .661
Communicating .045 .813
Computing N/A N/A
Creating −.108 .572
Designing .109 .566
Defining Problems −.161 .397
Experimenting N/A N/A
Interpreting Data −.163 .391
Managing .442*.014
Measuring −.239 .203
Model/Prototype .127 .504
Observing −.168 .374
Predicting −.144 .449
Questioning .074 .697
Testing −.016 .934
Visualizing −.277 .138
Tot al Tim e .3 16 .0 8 8
The codes of Computing and Experimenting were not used by the participants
*Significant at the 0.05 level (2-tailed)
An Investigation of Engineering Design Cognition and Achievement in...

Furthermore, to identify potential significant cognitive predictors of design success
in terms of project rank, ordinal regression analyses of the design cognition data were
calculated between the cognitive process times and the final project ranks. Prior to
pursuing the ordinal regression analyses, regression diagnostics were conducted to test
statistical assumptions of linearity, homoscedasticity, normality of residuals, mean
independence, and non-linear relationships. These assumptions were met with the
exception of an issue with multicollinearity related to several of the predictors (cogni-
tive processes) being highly correlated with one another. To account for this issue, the
number of collinear predictors was reduced by combining highly correlated cognitive
processes into aggregate processes. Based on the results of these statistical diagnostics
and the work by Strimel et al. (2018) the cognitive processes of Computing and
Interpreting Data were combined to form Quantitative Reasoning;Experimenting,
Testing,Questioning/Hypothesizing,andObserving were combined to form Scientific
Inquiring;andDesigning and Creating were combined to form Designing/Ideating (see
Fig. 5). Following the creation of the new cognitive process codes, the data were again
examined with the ordinal regression procedure to investigate potentially significant
cognitive predictors for design achievement (rank). Based on the correlations between a
total of 11 variables, two-way interaction terms significantly correlated with one
another were included in the analysis, but there was no significant interaction effect.
The results indicated that Managing was a significant predictor of project rank. An
increase in time spent on Managing was associated with a poorer design ranking, with
an odds ratio of .007 (95% CI, .002 to .013), Wald χ2(1) = 6.239, p= .012. Thus, in this
sample a participant’s rank would worsen by .007 of a rank for each second they
devoted to the process of Managing. This result can be aligned with the result of
Fig. 5 Proposed revised list of the 17 mental processes for technological problem solving based on the
combination of highly correlated processes. Note: Modeling was removed from the list based on Halfin’s
(1973) original work which highlighted that the process was too similar to other mental processes (Model/
Prototype Constructing and Designing)
G. J. Strimel et al.

correlation analysis, which suggests that it could be important for students to develop
abilities to effectively spend their time Managing by better framing design problems.
These changes may provide more time for students to devote to other cognitive tasks
that could potentially help them achieve better achievement results.
Discussions and Recommendations
It is important to note that this study was exploratory in nature and limited to a sample
of 30 participants. Therefore, the results of this study are not generalizable to all
primary school engineering programs. Although limitations exist, the findings can
highlight elements of design cognition that warrant further investigation and provide
discussion points/recommendations for pedagogical practice within the STEM educa-
tion community. Accordingly, the following sections will discuss this study’sfindings
and present recommendations for future research and educational practice.
Design Cognition Research Recommendations
Typicallythe rationale for conducting design cognition research is to better understand
how different groups of people solve design problems. However, researchers should
be purposeful in their research design to help bridge the gap between design cognition
research and educational practice in order to better inform teaching and learning.
Therefore, this study attempted to discover the relationships, if any, between students’
grade level, cognition, and success in engineering design through concurrent think-
aloud protocol, correlational, and ordinal regression analyses. Specific attention was
focused on identifying potential cognitive predictors of achievement in design to
provide a foundation for recommendations toward improving student design abilities.
The analysis results showed that Managing was the only cognitive process correlated
with the ranking of student design success. However, this relationship implied that the
more time students devoted to Managing, the poorer their design portfolio was ranked.
The authors believe that during the design session, participants had to allocate their
given time to multiple cognitive strategies; and consequently, the more time they spent
on Managing, the less time they were able to engage in other, potentially more critical,
cognitive processes. These other cognitive processes may have positively impacted
their design achievement. For example, while not significant, the mental processes of
Creating, Defining Problems, Interpreting Data, Measuring, Observing, Predicting,
Testing , and Visualizing were all related to a better design ranking. However, these
were some of the least used cognitive processes by the participants. Thus, the results
may imply that interventions could be developed, such as improved problem framing
practices or interim goal setting, in an attempt to help primary students use their
Managing time more efficiently by better defining design problems and goals. As
seen in this example, this type of research can help identify potential pain points in the
design practices of students. Therefore, the authors recommend that the research
methods presented in this study be enacted with larger sample sizes of students at
various ages and experience lev els within a variety of settings to further investigate the
development of design practices and design-based educational interventions.
An Investigation of Engineering Design Cognition and Achievement in...

Also, to help facilitate and provide consistency in the design protocol coding
process, it is recommended for future researchers using the Halfin (1973) coding
scheme to consider the hierarchical organization of the cognitive processes proposed
by the authors in Fig. 3. This organization assisted the research team in making more
consistent coding decisions throughout the data analysis process when more than one
mental process seemed to be occurring at the same time. The organization enabled the
selection of the code that was perceived to be the most relevant to the actions of the
participant. Further, as several cognitive processes were highly correlated with one
another in this sample and also in the study conducted by Strimel et al. (2018), there is
support for potentially refining the Halfin (1973) coding scheme by combining highly
correlated cognitive processes into aggregate processes (see Fig. 5). While a revised set
of codes may not be suitable for all settings, future research focused on identifying
cognitive predictors of success through an ordinal regression analysis, should consider
combining the cognitive processes of Computing and Interpreting data to form Quan-
titative Reasoning;Experimenting, Testing, Questioning/Hypothesizing,andObserving
to form Scientific Inquiring;andDesigning and Creating to form Designing/Ideating.
Lastly, as stated by Kelley and Sung (2017), it is recommended that concurrent
think-aloud protocol analysis be considered as a form of design assessment. In their
study, they found that students were able to achieve success on traditional pre/post
multiple-choice assessments developed to measure mathematical and scientific
knowledge but the same students demonstrated flawed conceptual understandings of
the topics when completing a design task. Therefore, Kelley and Sung (2017) believe
that the concurrent think-aloud protocol analysis method may be an effective process
for assessing a student’s ability to transfer learning.
Discussions for Informing Educational Practice
Problem Framing and Scoping In this study, the analysis of 30 design protocols
revealed a tendency for primary students to allocate the largest portion of the cognitive
activity toward manipulating materials to produce a physical prototype rather than
problem framing and scoping. While fourth grade students in this study dedicated
significantly more cognitive effort to Defining Problems than kindergarten students, the
two groups, overall, devoted limited time to analyzing and defining the given problems.
In particular, the observations of kindergarten students suggested a shortcoming in
ability to apply design criteria and constraints to their design. These findings can be of
importance to educational practice as Strimel et al. (2018) found that higher design
rubric scores on a design project were significantly correlated with the total time
participants dedicated toward the design phase of their problem-solving process, which
includes the practice of problem scoping. Additionally, Kelley et al. (2015)speculated
that student teams who dedicated more time to defining problems could complete their
problem-solving task more efficiently. Other studies have also indicated that the lack of
effort in identifying design criteria and constraints can lead students to impulsively
determine feasibilities of their solutions or to delay the investigation of alternative solutions
later in their design process (Mentzer 2014; Mentzer et al. 2015;Wilsonetal.2013).
Strimel (2014) suggested that the lack of students’efforts to define problems may be
aresultofstudents’familiarity with solution-driven design approaches rather than
problem-driven ones. Kruger and Cross (2006) explain that problem-driven design
G. J. Strimel et al.

involves the designer focusing on understanding the given problem by carefully
examining the design task to either (1) frame a well-defined problem to limit potential
distractions from solution alternatives or (2) abstractly define a problem to provide
opportunities for alternative solutions. In either case, they believe the problem-driven
approach strongly focuses or concentrates the process of generating solutions. In
contrast to this approach, Kruger and Cross (2006) describe that solution-driven design
involves the designer keeping the problem ill-defined and focusing most of their efforts
on generating a large number of potential solutions. When Kruger and Cross (2006)
studied designers to compare these different design strategies, they found that those
using a solution-driven strategy tended to have lower overall solution quality scores but
higher creativity scores. However, they found that those using a problem-driven design
strategy tended to produce the best results in terms of the balance of both overall
solution quality and creativity. Accordingly, it may be necessary for primary students to
have more opportunities to learn problem-driven approaches and practice transferring
them to a variety of engineering design contexts.
Further emphasizing a need to examine the practices involved with problem framing,
the primary students in this study were observed focusing attention on ancillary and
irrelevant components of the design scenario. For example, kindergarten students tended
to spend more time on making items, such as model insects for the project that required
the creation of a device to help an injured dragonfly catch prey, instead of the actual
prototype of the device to solve the problem. This was also observed with one of the
fourth grade participants that spent most of their design time making model soldiers
instead of a prototype shelter for the soldiers, which was the actual focus of the design
problem. When associating these observations with the low percentage of cognitive
processing time dedicated to Defining Problems (Kindergarten, 0.6%; Fourth Grade,
2.8%) when compared to Managing (Kindergarten, 20.7%; Fourth Grade, 18.0%) and
Model/Prototype Constructing (Kindergarten, 34.4%; Fourth Grade, 26.3%), it appears
that problem framing and scoping is a challenge for primary students and may not be
developmentally appropriate for kindergarten students. As the findings suggest, more
time associated with the cognitive process of Managing was a significant predictor of a
poorer ranking and this process was often seen linked with the inability to understand or
frame the problem. These findings also align with other reports (Berland 2013;Berland
and Busch 2012; Goldstone and Sakamoto 2003; Kaminski et al. 2009; National
Academy of Engineering and National Research Council 2014; Sloutsky et al. 2005)that
describe how the rich perceptual information associated with, and irrelevant features of,
design experiences can potentially distract students from identifying and learning the
difficult to understand theoretical discipline-specific concepts. Therefore, when
implementing engineering design tasks with young children, one should take caution—
based on their purpose for the design-based learning initiatives—as engineering design
tasks are not proven to be the best method for developing transferrable knowledge in all
situations and learning could be limited based on a student’s problem framing abilities.
Furthermore, the findings related to problem framing and scoping also align with the
results of a study by Welch and Lim (2000), which examined a total of 28 middle
school students’design cognition. Their study determined that the majority of middle
school students’cognitive efforts were used for modeling a possible solution idea and
building a physical prototype. Additionally, the findings align with the studies
conducted by Grubbs (2016) and Wilson et al. (2013) which indicate that secondary
An Investigation of Engineering Design Cognition and Achievement in...

students tend to focus more on making physical solutions rather than clearly defining
the design problems and analytically developing design concepts prior to prototyping.
Moreover, Strimel et al. (2018) reported observing first-year engineering students with
no prior engineering education experiences struggling to determine the necessary tasks
to begin the design process when required to solve an ill-structured problem. Therefore,
as the practices of (a) clearly defining goals associated with problem-solving efforts,(b)
determining key information and procedures required to solve a problem, and (c)
recognizing the constraints that limit potential solutions are necessary for achieving
true engineering design capabilities (Cross 2001), these findings may provide support
for establishing a coherent and better scaffolded study of engineering across primary,
secondary, and post-secondary education. If a goal of education is to promote the
development of more innovative and effective problem-solvers, a coherent study of
engineering across P-12 education seems compulsory.
In light of this discussion, the authors believe that scaffolding the instruction and
practice of problem framing is necessary. To do so, one recommendation may be to begin
first with the problem-driven design approaches of framing a well-defined problem in
order to limit potential distractions from solution. In terms of learning outcomes, this can
support achieving the Next Generation Science Standard’s(NGSS) (2013) engineering
design performance expectation that states students, grades K-2, should be able to:
Ask questions, make observations, and gather information about a situation
people want to change to define a simple problem that can be solved through
the development of a new or improved object or tool.
Then, as students progress toward the end of primary school, they can practice defining
problems in a more abstract manner to help meet the NGSS performance expectation
for grades 3 though 5 which states that students should be able to:
Generate and compare multiple possible solutions to a problem based on how
well each is likely to meet the criteria and constraints of the problem.
Also, as most of the participants’efforts were devoted to just making a prototype
without much forethought, it may be beneficial to require approval of a student’sdesign
concept prior to making. If all of these practices are maintained, then students may be
able to develop the abilities to identify design goals, criteria, and constraints to
formulate problem statements that first, help them frame the design scenario as a means
to guide their problem solving process and then, remove assumptions limiting solution
opportunities by rephrasing a problem statement from multiple perspectives. However,
further research into these possibilities is necessary.
Examining the Scaffolding of Engineering Design Experiences The cognitive processing
times in this study suggest that between kindergarten and fourth grade, students may
develop the cognitive abilities to tackle slightly more complex design problems. However,
without the proper scaffolding of design activities from structured to ill-structured prob-
lems, students, such as the one fourth grade student discussed earlier, may continue to be
distracted by the irrelevant items related to the contextually-rich problem scenarios and
thus, be ineffective in identifying and solving the actual problem presented in a design
G. J. Strimel et al.

activity. These findings suggest the need for further research into students’abilities to
transfer ideas, articulate concepts, and progress through design challenges across grade
levels. It is possible that the findings from this study may not be replicated with different
students, age groups, and participant backgrounds—thus, additional efforts to understand
these concepts are encouraged. However, without the presence of a formalized P-
12 engineering curriculum, and prepared teachers throughout primary schools, students
may continue to hold a limited view of engineering (e.g., engineering is simply a crafts-
oriented subject). As observed in the settings in which this study took place, teachers did
not seemed equipped with the knowledge and tools for discussing problem framing and
scoping with students and provided little interventions to direct students toward under-
standing the problem, which may have been evidenced by the limited cognitive effort
placed by the participants on Defining Problems. Therefore, it seems that tested and
validated guidelines for appropriately scaffolding design experiences may mitigate these
concerns with the implementation of engineering design tasks.
Accordingly, the authors created Fig. 6to provide a sample conceptual model for
scaffolding design experiences across the grade levels to potentially help students
develop effective design abilities and achieve more Bengineering^design capabilities.
As the figure depicts, starting in the early grades, students could be provided with
structured design problems, that will inherently be inauthentic, to allow them to build
upon playful and experimental approaches to design and making. The structured
problems can provide experiences for students to achieve some success to begin
building their design confidence and making abilities. However, as students develop
and their knowledge deepens, the authors posit that they should be provided with more
authentic, and less-structured problems, which may provide them with opportunities to
learn from failure while also continuing to build design confidence. As students
continue to grow and develop more analytic thinking abilities, they could then move
from these trial-and-error problem solving approaches to more informed design that
includes more calculated engineering design practices—which also necessitates the
developmentally appropriate applications of STEM knowledge. In doing such, the
authors propose that students can begin to Btake the chance out^of designing by
narrowing down and refining ideas through informed engineering design practices as
opposed to a simple Bguess-and-check^or Btrial-and-error^approach.
Fig. 6 Conceptual model of scaffolding design experiences across the grade levels to potentially support the
development of authentic engineering design capabilities
An Investigation of Engineering Design Cognition and Achievement in...

Computational Thinking and Scientific Inquiry in Design Another identified area of
weakness in the teaching of engineering design at the P-12 level is the appropriate
application of computational thinking and scientific inquiry to support the practices of
design, particularly at the high school level (National Academy of Engineering and
National Research Council 2009). Merrill et al. (2009) posit that the necessary analyt-
ical tools for engineering design are not frequently used in classrooms to inform design
ideas and as such, they stress the importance of teaching secondary engineering
students how to be predictive in designing. While computational and scientific tools
and practices may be beyond the capabilities of primary students, it is important to
build their foundation for future success in design. In spite of these calls for increased
emphasis in this area, studies continue to highlight students’lack of computational,
mathematical, and scientific thinking necessary for authentic engineering design
(Kelley 2008; Kelley et al. 2010; Kelley and Sung 2017; Wilson et al. 2013). Relatedly,
this study found that primary school students rarely engaged in the cognitive strategies
for Computing, Experimenting, and Interpreting Data.
Appropriately, Kelley et al. (2010) emphasized the necessity to provide students
with more learning opportunities to perform computational/mathematical and scientific
inquiry activities within the engineering design process. Although primary students
may not be equipped with mathematical or scientific principles for solving authentic
problems, it can be helpful for children to experience activities which push them toward
the application of fundamental mathematical principles they have learned in class, the
use of scientific tools, an emphasis on recording experiment results, and a purposeful
effort toward using the resulting data of these actions to inform solutions. A study of
fifth grade students conducted by Kelley and Sung (2017) found that when participants
were engaged in an engineering design task that was developed to necessitate the
application of the mathematical and scientific understandings of the concept of mass,
they failed to recall the conservation of mass, compute the mass of their solution, and
understand the difference between the concept of mass and volume. However, their
study did show that primary students were able to increase the amount of time they
dedicated to computational thinking when given additional Btransfer^design tasks with
embedded mathematics/science activities. Therefore, Kelley and Sung (2017)recom-
mend that primary teachers should provide additional opportunities for students to
transfer their knowledge beyond an initial design task. What this effort involves, is
developing a Btransfer^design task, which is a design problem that is structurally
similar to the initial design task but requires students to transfer their learning from the
initial task to a novel context. This effort can potentially provide students an opportu-
nity beyond the initial design task, to enhance their use of the computational and
scientific mental processes of Computing, Experimenting, and Interpreting Data.
Conclusion
As the teaching of engineering continues to gain attention in the primary school setting,
it becomes fundamental to comprehend the ways in which students cognitively process
engineering design tasks to provide effective engineering instruction, establish the
adequate scaffolding of engineering design experiences, and enact interventions to
G. J. Strimel et al.

enhance student design abilities. However, there has been minimal research to provide
insight into the range of cognitive processes employed by primary students while
designing and making a product, system, or device and limited discussion on effective
ways to link design cognition research with practice in engineering education. More-
over, if content standards for engineering are ever to be developed, then use-inspired
design cognition research is a necessity (National Academy of Engineering and
National Research Council 2010). Consequently, the authors enacted a multiple ex-
ploratory case study approach employing the concurrent think-aloud protocol analysis
technique to describe the cognitive activity of both kindergarten and fourth grade
students as they worked to complete engineering design tasks. This study was driven
by the overarching research objectives of (1) identifying ways in which primary
students cognitively navigate an engineering design task from design conception
through prototype production and (2) determining ways in which to inform P-12
engineering education by examining design cognition in respect of student grade level
and design achievement.
To achieve the first research objective, the authors sought to determine what
cognitive processes kindergarten and fourth grade students employ to design and make
solutions to engineering design problems. To do so, the authors collected and coded
video recordings of participants thinking aloud during engineering design tasks. The
analysis of the data revealed that the top three most employed mental processes—of the
17 mental processes for technological problem solving (Halfin 1973)—for kindergarten
students were Model/Prototype Constructing,Managing,andObserving and the least
used mental processes (each consuming less than 2% of the students design time on
average) were Test i n g ,Creating,Measuring,Defining Problems,Questioning, and
Predicting. In addition, the mental processes of Computing and Experimenting were
not observed in the kindergarten participants. As for the fourth grade participants, the
top three most used mental processes were Model/Prototype Constructing,Managing,
and Analyzing and least used mental processes (each consuming less than 2% of the
students design time on average) were Creating,Visualizing,Measuring, and
Predicting. Also, similar to the kindergarten students, the mental processes of Com-
puting and Experimenting were not observed being used by the fourth grade partici-
pants. Lastly, the analysis revealed a tendency for both groups of students to allocate
the largest portion of their cognitive activity toward manipulating materials to produce
a physical prototype rather than problem framing, information gathering, and generat-
ing design concepts. While fourth grade students in this study dedicated significantly
more cognitive effort to Defining Problems than kindergarten students, the two groups,
overall, devoted limited time to analyzing and defining the given problems to generate
refined solution ideas.
To achieve the second research objective, the authors sought to examine potential
relationships between the design cognition of kindergarten and fourth grade students as
well as identify any potential cognitive indicators for design achievement. To do so, the
design cognition data of the two participant groups were compared along with the
evaluations of their design work. The analysis revealed significant differences between
kindergarten and fourth grade participants with regard to the amount of time various
cognitive processes were employed to complete a design task. Fourth grade students
dedicated significantly more time to the mental processes of Creating,Defining
Problems,Measuring,andTe st ing than kindergarten students. The greatest difference
An Investigation of Engineering Design Cognition and Achievement in...

in cognitive processing time between the two groups was found in the mental process
of Testing. Accordingly, the observations of these students showed that most of the
kindergarten participants ended their design session once a prototype was made and
essentially no prototype revisions were enacted. However, the fourth grade participants
did showcase the iterative design practices related to Tes t ing, which was demonstrated
by them examining the workability and/or durability of their prototypes and then
attempting to improve their results. In addition, when examining the design proto-
cols along with the evaluations of the student design portfolios, it was found that
more time devoted to the cognitive process of Managing could be a significant
predictor of a poorer design ranking. This result was seen as a potential effect of
limitedstudentabilitiesinregardtoproblem framing and scoping. However, the
cognitive processes of Creating, Defining Problems, Interpreting Data, Measuring,
Observing, Predicting, Testing, and Visualizing, while not statistically significant,
were related to a better design ranking.
Based on this analysis, several recommendations for educational practice and future
research were discussed. First, to help aid in improving student design abilities, a
recommendation could be to provide students with developmentally appropriate in-
struction related to problem framing and scoping. Initially, engineering design tasks
may be too perceptually rich for kindergarten students and distract them from the
learning objective of the assignment. Also, their level of experience may limit them
from understanding the criteria for the actual design task and result in them physically
making objects that are irrelevant to solving the problem. Perhaps the results may even
indicate that there is an age where open-ended design tasks are not cognitively
appropriate for learners; specifically, if these tasks are used for learning specific STEM
concepts rather than just a fun reprieve from the Bnormal^classroom environment.
However, if appropriate attention is given to teaching young children problem-driven
design approaches and the practices of defining the problem, determining the criteria
and constraints, and identifying clear goals for developing a solution, rather then
immediately making prototypes, then students may be able to better manage their
design process and ultimately, produce enhanced solutions. Also, the scaffolding of
instruction from well-defined problems to more authentic, ill-defined problems could
be recommended as a means to help transition students to more informed engineering
design practices. Lastly, the participants in this study did not engage in the cognitive
strategies for Computing, Experimenting, and Interpreting Data, which can be critical
for advancing a student design capabilities as well as enabling their learning of science
and mathematics. Therefore, a recommendation from Kelley and Sung (2017)of
providing students with math/science embedded Btransfer^design tasks that require
them to translate their learning from an initial design task to novel contexts seems
beneficial for increasing a their use of these cognitive processes.
It is important to note that this study was exploratory in nature and as such, has
limitations. Therefore, the authors recommend that the research methods presented in
this study be enacted with larger sample sizes of groups of various ages and experience
levels. These further investigations can help identify potential cognitive pain points in
the design practices of students in an effort to provide support for informing the
implementation of engineering at the P-12 level and developing strategies to help
improve student design achievement. As engineering design-based activities continue
to be touted as a preferred approach to teaching STEM-related content and practices, it
G. J. Strimel et al.

is important that further attention is given to how students develop and learn design as
well as function within design environments.
Compliance with Ethical Standards
Conflict of Interest On behalf of all authors, the corresponding author states that there is no conflict of
interest.
References
Atman, C. J., & Bursic, K. M. (1998). Verbal protocol analysis as a method to document engineering student
design processes. Journal of Engineering Education, 87(2), 121–132.
Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education
Teac h e r, 76 (6), 13–17.
Bartholomew, S. R., Strimel, G. J., & Jackson, A. (2017). A comparison of traditional and adaptive
comparative judgment assessment techniques for freshman engineering design projects. International
Journal of Engineering Education, 34(1), 20–33.
Bartholomew, S. R., Strimel, G. J., & Yoshikowa, E. (2018). Using adaptive comparative judgment for student
formative feedback and learning during a middle school open-ended design challenge. International
Journal of Design & Technology Education.
Berland, L. K. (2013). Designing for STEM integration. Journal of Pre-College Engineering Education
Research, 3(1), 22–31.
Berland, L. K, & Busch, K. (2012). Negotiating STEM epistemic commitments for engineering design
challenges. In American Society for Engineering Education 2012 Annual Conference and Exposition,
Conference Proceedings.
Blanchard, P., & Suhr, J. (1999). My bug box. Katonah: Richard C. Owen Publishers, Inc.
Capobianco, B. M., French, B. F., & Diefes-Dux, H. A. (2012). Engineering identity development among pre-
adolescent learners. Journal of Engineering Education, 101(4), 698–716.
Carr, R. L., Bennett IV, L. D., & Strobel, J. (2012). Engineering in the K- 12 STEM standards of the 50 U.S.
states: An analysis of presence and extent. Journal of Engineering Education, 101(3), 539–564.
Cohen, D. (2002). How the child’s mind develops? London: Routledge.
Cohen, B. H. (2007). Explaining psychological statistics (3rd ed.). New York: John Wiley and Sons.
Cross, N. (2001). Designerly ways of knowing: Design discipline versus design science. Design Issues, 17(3),
49–56.
Cross, N. (2004). Expertise in design: An overview. Design Studies, 25(5), 427–441.
Crossland, J. (2015). Is Piaget wrong? Primary Science, 137,30–32.
Cunningham, C. M. (2009). Engineering is elementary. The Bridge, 30(3), 11–17.
Cunningham, C. M., & Hester, K. (2007). Engineering is Elementary: An Engineering and Technology
Curriculum for Children.Paper presented at ASSE Annual Conference and Exposition, Honolulu, HI.
Retrived from http://www.eie.org/sites/default/files/research_article/research_file/ac2007full8.pdf
Dutson, A., Todd, R., Magleby, S., & Sorensen, C. (1997). A review of literature on teaching engineering
design through project-oriented capstone courses. Journal of Engineering Education, 86(1), 17–28.
Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking,
teaching, and learning. Journa l of Engineering Education, 94(1), 103–120.
Engineering Accreditation Commission. (2016). Criteria for accrediting engineering programs. Baltimore:
Accreditation Board for Engineering and Technology Retrieved from http://www.abet.org/wp-
content/uploads/2016/12/T001-17-18-ETAC-Criteria-10-29-16.
Goldstone, R. L., & Sakamoto, Y. (2003). The transfer of abstract principles governing complex adaptive
systems. Cognitive Psychology, 46(4), 414–466.
Grubbs, M. E. (2013). Bridging design cognition research and theory with teaching and learning.New
Zealand: Published Proceedings of the Pupil's Attitudes Toward Technology International Conference,
Christ Church.
Grubbs,M.E.(2016).Further characterization of high school pre- and non-engineering students’cognitive
activity during engineering design (Doctoral dissertation). Retrieved from ProQuest Dissertations and
Theses database. (UMI No. 3662376).
An Investigation of Engineering Design Cognition and Achievement in...

Grubbs, M. E., & Strimel, G. (2015). Engineering design: The great integrator. Journal of STEM Teacher
Education, 50(1), 77–90.
Grubbs, M. E., & Strimel, G. J. (2016). Cognitive research: Transferring theories and findings to k-12
engineering educational practice, American Society for Engineering Education 103rd Annual
Conference and Exposition. New Orleans, LA, June 26, 2016.
Halfin, H. H. (1973) Technology: A process approach. (Doctoral dissertation, West Virginia University, 1973)
Dissertati on Ab str act s Intern ati onal, (1) 1111A.
Hall, M. (2006). Dragonflies. Mankato: Capstone Press.
Hammack, R., Ivey, T. A., Utley, J., & High, K. A. (2015). Effect of an engineering camp on students’
perceptions of engineering and technology. Journal of Pre-College Engineering Education Research,
5(2), 10–21. https://doi.org/10.7771/2157-9288.1102.
Hegedus, T. (2014). Engineering education for youth: Diverse elementary school students' experiences with
engineering design (Doctoral dissertation). Retrieved from http://eie.org/sites/default/files/research_
article/research_file/hegedus-dissertation-final-dec2014.pdf.
Hill, R. B., & Wicklein, R. C. (1999). A factor analysis of primary mental processes for technological
problem-solving. Journal of Industrial Teacher Education, 36(2), 83–100.
International Technology and Engineering Educators Association (ITEA/ITEEA). (2000/2002/2007).
Standards for technological literacy: Content for the study of technology.Reston,VA:Author.
Kaminski, J. A., Sloutsky, V. M., & Heckler, A. (2009). Transfer of mathematical knowledge: The portability
of generic instantiations. Child Development Perspectives, 3(3), 151–155.
Kelley, T. R. (2008). Cognitive processes of students participating in engineering-focused design instruction.
Journal of Technology Education, 19(2), 50–64.
Kelley, T., & Sung, E. (2017). Examining elementary school students’transfer of learning through engineering
design using think-aloud protocol analysis. Journal of Technology Education, 28(2), 83–108.
Kelley, T. R., Brenner, D. C., & Pieper, J. T. (2010). Two approaches to engineering design: Observations in
stem education. Journal of stem Teacher Education, 47(2), 5–40.
Kelley, T. R., Capobianco, B. M., & Kaluf, K. J. (2015). Concurrent think-aloud protocols to assess
elementary design students. International Journal of Technology and Design Education, 25(4), 521–540.
Kimbell, R. (2012). Evolving project e-scape for national assessment. International Journal of Technology &
Design Education, 22,135–155.
Kolpin,M.(2014).Great women of the civil war. Mankato: Capstone Press.
Kruger, C., & Cross, N. (2006). Solution driven versus problem driven design: Strategies and outcomes.
Design Studies, 27(5), 527–548.
Lammi, M. D., & Becker, K. (2013). Engineering design thinking. Journal of Technology Education, 24(2),
55–77.
Lammi,M.D.&Gero,J.S.(2011).Comparing design cognition of undergraduate engineering students and
high school pre-engineering students. Paper presented at the 2011 Frontiers in Education Conference,
Rapid City, SD.
Lead States, N. G. S. S. (2013). Next generation science standards: For states, by states. Washington, DC:
National Academies Press.
McCullar, H. (2015). Am I really teaching engineering to elementary students? Science & Children,
52(7), 80–84.
McHugh, M. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.
Mechisedech, K. (2011). The terrible, awful civil war.Mankato:CapstonePress.
Mentzer, N. (2014). Team based engineering design thinking. Journal of Technology Education, 25(2), 52–72.
Mentzer, N., Becker, K., & Sutton, M. (2015). Engineering design thinking: High school students’perfor-
mance and knowledge. Journal of Engineering Education, 104(4), 417–432.
Merrill, C., Custer, R. L., Daugherty, J., Westrick, M., & Zeng, Y. (2009). Delivering core engineering
concepts to secondary level students. Journal of Technology Education, 20(1), 48–64.
National Academy of Engineering & National Research Council. (2009). Engineering in K-12 education:
Understanding the status and improving the prospects. Washington, DC: The National Academies Press.
National Academy of Engineering & National Research Council. (2010). Standards for K-12 engineering
education? Washington, DC: The National Academies Press.
National Academy of Engineering & National Research Council. (2014). STEM integration in K–12 educa-
tion: Status, prospects, and an agenda for research. Washington, DC: National Academies Press.
NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC:
National Academies Press.
Polacco, P. (1994). Pink and say. New York: Babushka Press.
G. J. Strimel et al.

Purcell, A. T., Gero, J. S., Edwards, H., & McNeill, T. (1996). The data in design protocols: The issue of data
coding, data analysis in the development of models of the design process. In N. Cross, H. Christiaans, &
K. Dorst (Eds.), Analysing design activity (pp. 225–252). Chichester: John Wiley.
Seery, N., Buckley, J., Doyle, A., & Canty, D. (2016). The validity and reliability of Adaptive Comparative
Judgements in the assessment of graphical capability. In Proceedings of the 71st Mid-Year Conference of
the Engineering Design Graphics Division (pp. 104–109).
Sloutsky, V. M., Kaminski, J. A., & Heckler, A. F. (2005). The advantage of simple symbols for learning and
transfer. Psychonomic Bulletin & Review, 12(3), 508–513.
Strimel, G. J. (2014). Engineering design: A cognitive process approach (Doctoral dissertation). Retrieved
from ProQuest Dissertations and Theses database. (UMI No. 3662376).
Strimel, G. J., Grubbs, M. E, & Wells, J. G. (2016). Engineering education: A clear decision. Technology and
Engineering Teacher, 76(4), 18–24.
Strimel, G. J., Bartholomew, S. R., Kim, E., & Cantu, D. V. (2018). Examining engineering design cognition
with respect to student performance. International Journal of Engineering Education. 34(6), 1–20.
Sutherland, P. (1992). Cognitive development today : Piaget and his critics. London: P. Chapman Pub.
Sweeney, A. (2010). Toads. Mankato: Capstone Press.
Weiten, W. (1992). Psychology: Themes and variations (2nd ed). California: Brooks Cole Publishing
Company.
Welch, M., & Lim, H. S. (2000). The strategic thinking of novice designers: Discontinuity between theory and
practice. Journal of Technology Studies, 26(2), 34–44.
Wicklein, R. C., & Rojewski, J. W. (1999). Toward a Bunified curriculum framework^for technology
education. Journal of Industrial Teacher Education, 36(4), 38–56.
Wilson, A. A., Smith, E. R., & Householder, D. L. (2013). High school students’cognitive activity while
solving authentic problems through engineering design processes. Atlanta: Published Proceedings of the
American Society of Engineering Education.
An Investigation of Engineering Design Cognition and Achievement in...