Which Divergent Thinking Index is More Associated with Problem Construction Ability? The Role of Flexibility and Task Nature

Full text

Which Divergent Thinking Index is More Associated with Problem
Construction Ability? The Role of Flexibility and Task Nature
Authors: Ahmed M. Abdulla Alabbasi, Roni Reiter-Palmon, Zainab M. Sultan
Affiliations:
Arabian Gulf University
University of Nebraska at Omaha
Correspondence concerning this article should be addressed to Ahmed M. Abdulla Alabbasi,
PhD, Department of Gifted Education, Arabian Gulf University, Email: ahmedmda@agu.edu.bh
Draft version: October 1st, 2020. This paper is under review
Abstract
Problem construction and divergent thinking (DT) are considered to be indicators of creative
potential. Previous studies, with different goals, suggest a positive correlation between problem
construction and DT. However, none of these works have explicitly examined which index of
DT is more associated with problem construction. The current investigation examined the
association between problem construction and three main indexes of DT: fluency, flexibility, and
originality. It also tested whether such a relation differs based on task nature (verbal versus
figural). The sample consisted of 90 sixth graders who completed three tests: (a) a verbal DT
test, (b) a figural DT test, and (c) a problem construction test. Correlational analysis showed that
flexibility was highly correlated with problem construction in the verbal DT test, whereas
originality was highly correlated with problem construction in the figural test. The association
between problem construction and verbal versus figural DT significantly differed in all DT
indexes. This finding suggests that figural and verbal tasks assess DT in a different way, which
was confirmed by a canonical correlation analysis. Finally, results of a multiple regression
analysis showed that verbal DT significantly explained 59% of the variance in fluency scores in
problem construction and 60% in originality scores in problem construction. Meanwhile, figural
DT explained 8% to 9% of the variability in fluency and originality scores in problem
construction. As suggested by experts in the field of problem construction, the role of flexibility
in problem construction is a fertile area to be considered in future studies.
Keywords: problem construction, divergent thinking, task nature, ill-defined problems

1. Introduction
The significance of problem construction is widely recognized, and it is considered to be
the first step in any creative problem-solving effort (e.g., Mumford et al., 1991; Osborn, 1953;
Treffinger & Isaksen, 2005; Wallas, 1926). Some scholars have considered problem construction
to be more important than problem solving (e.g., Csikszentmihalyi, 1988; Einstein & Infeld,
1938; Mackworth, 1965; Wertheimer, 1945). A recent meta-analysis showed that problem
construction and creativity are significantly positively correlated (Abdulla et al., 2020), and the
effect size is the nearly same as that reported in a meta-analysis (Kim, 2008) on the correlation
between divergent thinking (DT) and creative achievement.
The history of studying problem construction started with the seminal work of
Csikszentmihalyi and Getzels (1971) that aimed to assess problem construction in art students.
Since then, the field of creativity research has shown a growing interest in studying problem
construction. However, this important cognitive process has not receiving as much attention from
researchers in the creativity field relative to DT or ideation. To date, there is only one edited
book on problem construction (Runco, 1994a). Figures 1 and 2 show a comparison between the
number of hits when three terms (problem solving, DT, and problem construction) were searched
from 1960 to 2020 in the following databases: ERIC, Academic Search Complete, Academic
Search Premium, and ProQuest Central. Advanced searched was used, and the following
combinations of keywords were used: (a) problem solving AND creativity OR creative, (b) DT
AND creativity OR creative, and (c) all terminologies of problem construction reported in
Abdulla and Cramond (2018; see page 202) AND creativity OR creative. The search included
titles only, and redundant works were eliminated.

___________________
Figures 1 and 2 around here
___________________
While the results indicate that there is indeed growth for research on problem
construction, and it is increasing at a higher rate relative to other research, the sheer number of
papers is still relatively small. This is problem for a number of reasons. First, problem
construction is an essential element in all models of creative process. Wallas (1926), Osborn
(1953), and Merrifield et al. (1962) called such a step preparation, whereas others used different
terminologies, such as problem finding (PF), problem identification, and idea finding (Amabile,
1988; Mumford et al., 1991; Reiter-Palmon et al., 1997; Runco, 1994a; Silverman, 1985).
Regardless of the terms used, they all refer to a situation in which the problem is ill-defined. In
addition, some definitions of creativity explicitly include problem construction (e.g., Cropley,
2011; Runco, 2014; Torrance, 1966). Second, advancing innovation and creativity for the benefit
of business and society is depends on those who can sense gaps in knowledge and discover ill-
defined problems to solve them in a novel way. A meta-analysis by Ma (2009) indicated that
problem construction was the strongest predictor of creativity providing support for this
assertion.
However, this is not to say that other cognitive processes are less important than problem
construction for creativity and creative thinking. The argument here is that problem construction
is an essential cognitive ability that deserves more attention from researchers in the creativity
field.
Nonetheless, after six decades of systematic study of problem construction, we have a
better understanding of the relation between problem construction and some cognitive processes,

such as inter and intrapersonal evaluation (Basadur, 1995; Mumford et al., 1991; Runco, 1994b),
problem solving (Arlin, 1975; Bouchard & Drauden, 1976; Carson & Runco, 1999), selective
encoding and combination, selective comparison (Brugman, 1995), and DT (e.g., Arlin, 1975;
Artley et al., 1980; Chand & Runco, 1993; Csikszentmihalyi & Getzels, 1971; Hoover, 1994;
Wakefield, 1985).
Among these cognitive processes, creativity literature suggests DT to be associated with
problem construction. Many researchers have sought to explore such an association (Ambrosio,
1993; Arlin, 1975; Chand & Runco, 1993; Hoover, 1994; Hoover & Feldhusen, 1990; Reiter-
Palmon et al., 1997; Runco & Acar, 2010; Wakefield, 1985); however, these studies have
yielded conflicting findings (see Table 1). Such conflict might be due to (a) the DT and problem
construction assessments used in the studies, (b) the DT skills targeted (i.e., fluency, flexibility,
and originality), and (c) the task nature (verbal vs. figural tasks).
___________________
Table 1 around here
___________________
For example, Ambrosio (1993) used the figural form of the Torrance Test of Creative
Thinking (TTCT) and reported a negative relation between DT and problem construction quality
(r= -.08), and a significant positive relation between DT and problem construction quantity (r =
.41). Arlin (1975) reported a positive relation between problem construction quality and adaptive
flexibility (r = .26), and a negative relation between problem construction quality and
spontaneous flexibility (r = -.09) and fluency (r= -.18). Runco and Acar (2010) showed a
significant positive relation between verbal DT test and problem construction (r ranged from .29
to .44).

As shown in Table 1, only one study looked at the relation between problem construction
and elaboration (i.e., Arlin, 1975), and only two studies examined the relation between problem
construction and flexibility (Arlin, 1975; Hoover, 1994). It has been suggested that the specific
DT measure and index used (verbal or figural; fluency, originality or flexibility) can affect the
results (Hornberg & Reiter-Palmon,2017; Reiter-Palmon et al., 2019). Further, much of the
research utilizing DT tasks tends to use fluency. Therefore, one of the contributions of this study
is to evaluate flexibility and originality in addition to fluency. Flexibility in particular may be
critical for problem construction. According to Mumford et al. (1994), “Flexibility in applying
selection and screening strategies may contribute to individual differences in problem finding
skills.” In addition to evaluating the relationships with different indices of DT, our study also
determined whether the relationship between DT and problem construction will differ based on
the nature of the DT task, verbal or figural.
2. Methods
2.1. Participants and Procedures
The participants included 90 students in the sixth grade (42 boys and 48 girls). We
randomly collected data from two public schools after receiving the official approval from the
Directorate for Scientific Research at the Ministry of Education. Prior to data collection, each
participant was asked to read and sign a consent form prepared by the authors of this study.
Before the tests were administered, the first author visited the two schools and spent one
class (40 minutes) with the students to create familiarity with them (no DT or problem
construction activities were discussed). By the end of this meeting, the author informed the study
participants of a subsequent visit in the next week to administer some activities that could show
the students’ creativity. The test was administered in the second class/session (8:45 am), as

recommended by the teachers. The participants were informed that they will obtain the results of
their “creative activity score” after the authors had scored them. The tests were conducted in a
game-like condition (untimed), following the results of a recent meta-analysis that showed that
the untimed condition significantly improves students’ performance in DT tests (Said-Metwaly
et al., 2020).
2.2. Instruments
2.2.1. Uses Test
We administered three tasks from the Uses test (Wallach & Kogan, 1965): (a) uses for a
spoon, (b) uses for a wheel, and (c) uses for a toothbrush. The directions for the task are as
follows:
In this game, I am going to name an object—any kind of object, like a light bulb
or the floor—and it will be your job to tell lots of different ways that the object
could be used. Any object can be used in a lot of different ways. Remember, think
of all the different ways you could use the object that I mentioned (Wallach &
Kogan, 1965, p. 31).
Responses to the Uses test were scored for fluency, flexibility, and originality. A total of
199 different ideas were generated for the spoon task, 124 different ideas for the wheel task, and
181 different ideas for the toothbrush task. Fluency was defined as the number of different ideas
related to a given stimulus. Flexibility was defined as the number of different categories or shifts
between ideas. Based on participants’ responses, 12 categories were created for the spoon task,
11 for the wheel task, and eight for the toothbrush task (see Table 2). Finally, originality was
scored based on a 3% cutoff criterion.

2.2.2. Figural Test
We administered the Figural test from the Runco Creativity Assessment Battery, which
consists of three tasks, similar to the Wallach and Kogan Line Meaning test (Wallach & Kogan,
1965, p. 36). The Figural test includes three shapes: Spiral, Blocks, and Lines. The directions are
as follows:
Look at the figure below. What do you see? List as many things as you can that
this figure might be or represent. This is NOT a test. Think of this as a game and
have fun with it! The more ideas you list, the better (Runco et al., 2016, p. 6).
The Figural test was also scored for fluency, flexibility, and originality. A total of
105 different ideas were generated for the first task (Spiral), 111 for the second task
(Blocks), and 53 for the third task (Lines). For flexibility, 10 categories were created in
the Spiral task, 11 categories in the Blocks task, and 11 categories in the Lines task.
Originality was scored using the same method as the Uses test (i.e., 3% cutoff).
___________________
Table 2 around here
___________________
2.2.3. Problem Generation Test
We used the Problem Generation (PG) test to assess participants’ problem construction
ability (Okuda et al., 1991; www.creativitytestingservices.com). The PG test consists of three
open-ended tasks that require participants to list as many problems as they can. This kind of
assessment can be considered as a problem discovery test given than participants are asked to
generate as many problems as they can (real or hypothetical) without any instruction to evaluate
any of their ideas, or that their ideas will be evaluated. The three PG tasks are related to (a) home

and school, (b) life situations, and (c) health and well-being. An example of a PG task is as
follows:
List problems along with your friends, peers, or schoolmates (any individual who
is approximately the same age as yourself). These problems can be real, or they
can be hypothetical and imaginary. Do not limit yourself; the more problems you
can list, the better. You can think of problems that exist now or those that might
exist in future.
The PG test was scored for fluency and originality. The reason for not scoring for
flexibility was that the nature of the tasks was extremely diverse. PG tasks asked participants to
list real and hypothetical problems, which made it difficult to score for flexibility. The
participants generated 235, 253, and 269 different ideas in the first, second, and third tasks,
respectively. The same method was utilized for scoring fluency and originality in the PG test.
Finally, the students were asked to fill out a demographic questionnaire with items on
their age and sex.
3. Results
3.1. Reliability
Both inter-class correlation (ICC) and internal consistency reliability were calculated for
the three tests used. As mentioned above, DT verbal and figural tests were scored for fluency,
flexibility, and originality, whereas problem construction was scored for fluency and originality.
Table (3) shows the reliability coefficients for the three tests. For the ICC, a two-way mixed
model absolute agreement was calculated.

___________________
Table 3 around here
___________________
As moderate reliability coefficients were observed in the Figural test, we looked at each
task of the Figural test. The Lines task showed lowered reliability coefficients in the Figural test.
After removing the Lines test, the reliability coefficients in the Figural test improved. Therefore,
the Lines task scores were eliminated in all subsequent analyses.
3.2. Correlation Analysis
As shown in Table 4, the pattern of correlations between DT and problem construction
differed in the verbal versus figural tests. In the verbal DT, the highest correlation was found
between flexibility and (a) problem construction fluency (r = .75, p <.001) and (b) problem
construction originality (r = .74, p <.001), followed by the correlation between (c) verbal fluency
and problem construction fluency and originality (r = .73, p <.001). Originality in DT was also
significantly correlated with (a) fluency in problem construction (r = .67, p <.001) and (b)
originality in problem construction (r = .68, p <.001).
Meanwhile, the magnitude of the correlation between figural DT and problem
construction was much lower compared with verbal DT, which ranged from .22 to .30. We tested
whether the relation between problem construction and DT was statistically different based on
the task nature (i.e., problem construction and verbal DT vs. problem construction and figural
DT; Lee & Preacher, 2013; Steiger, 1980, pp. 237–248). In transforming the Pearson correlation
coefficients to Fisher’s z, the results showed that the relation between fluency in problem
construction and DT (verbal vs. figural) was significantly different in terms of fluency (z = 4.48,
p <.001), flexibility (z = 4.47, p <.001), and originality (z = 3.31, p <.001); and originality in

problem construction and DT (verbal vs. figural) was significantly different in terms of fluency
(z = 4.26, p <.001), flexibility (z = 4.18, p <.001), and originality (z = 3.28, p <.001). Thus, it can
be concluded that the task nature significantly influenced the relationship between problem
construction and DT such that verbal tasks were more strongly related to problem construction.
A canonical correlation analysis (CCA) was conducted to examine the relation between
the verbal and figural DT tests. The results showed no significant correlation between the verbal
and figural DT in all DT indexes: (a) fluency [Wilks’ Lambda = .850, Rc = .35, p = .13], (b)
flexibility [Wilks’ Lambda = 970, Rc = .17, p = .60], and originality [Wilks’ Lambda = 998, Rc =
.04, p = .70].
___________________
Table 4 around here
___________________
3.3. Regression Analysis
Multiple regression analysis was conducted to evaluate the prediction of the dependent
variables (i.e., figural and verbal DT) on problem construction scores (see Table 5). Verbal DT
indices were significantly related to problem construction fluency [F(3,86) = 41.94, p <.001, R2
= .59] and problem construction originality [F(3,86) = 42.21, p < .001, R2 = .60]. Meanwhile,
figural DT was not statistically related to problem construction fluency [F(3,86) = 2.34, p = .079,
R2 = .08] but significantly related to problem construction originality [F(3,86) = 2.78, p = .046,
R2 = .09].
___________________
Table 5 around here
___________________

3.4. CCA
We performed a CCA for each DT index and problem construction. The first set included
fluency in the Uses and Figural tests, and the second set included fluency and originality in
problem construction. The results showed that fluency in the Uses test significantly correlated
with problem construction fluency and originality [Rc = .74, p <.001], whereas fluency in the
Figural test was not significantly correlated with problem construction fluency and originality
[Rc = .16, p = .13].
In the second CCA, flexibility in the Uses and Figural tests was entered as set 1. The
results showed that flexibility in the Uses test significantly correlated with problem construction
fluency and originality [Rc = .75, p <.001], whereas flexibility in the Figural test did not
significantly correlate with problem construction fluency and originality [Rc = .14, p = .20].
Finally, in the third CCA, originality scores in the Uses and Figural tests were entered as
set 1. The results indicated a significant correlation between originality in the Uses test and
problem construction [Rc = .69, p <.001]. We found a non-significant relation between
originality scores in the Figural test and between problem construction fluency and originality
[Rc = .12, p = .25].
4. Discussion
Studies on problem construction have suggested that some cognitive processing skills are
related to problem construction, and have also paid attention to DT along with problem
construction. Previous research has examined the association between problem construction and
DT, however, none of these studies have explicitly aimed to examine multiple DT indices and
tasks and the relative association with problem construction. The results of our study showed that
flexibility, which was neglected in previous works on problem construction and DT, was highly

correlated with problem construction compared with fluency and originality, especially when
verbal tasks were administered. Arlin (1975) described problem finders as “flexible thinkers.”
Our findings support this claim and suggest that good problem finders are characterized by an
ability to shift between ideas and keep formulating different ideas to find a novel problem that
needs to be solved. However, more studies are needed to elucidate fully the role of flexibility in
problem construction.
The other important finding in the present study is the difference in the relation between
problem construction and DT based on the task nature. We found that problem construction can
be better assessed using verbal tasks than figural ones. This finding is partly supported by
Abdulla et al. (2020), in which problem construction in the writing domains is highly correlated
with creativity (r = .36) compared with art (r = .20). One issue that future studies might consider
is whether highly capable problem finders are overrepresented in writing versus in other
domains. It is possible that the strong correlations between problem construction and DT in the
writing domain is in part a result of both requiring writing as the mode of generated responses.
Previous research suggests that the task and index of measurement is important in understanding
these types of relationships (Reiter-Palmon et al., 2019; Reiter-Palmon & Schoenbeck, in press).
Additional research evaluating whether these relationships hold with other verbal DT measures
such as consequences as well as other figural DT measures would help clarify this issue.
It is also important to note problem construction is not a single process. Many scholars
have differentiated known problem construction skills, such as problem definition, problem
identification, and problem discovery (e.g., Getzels, 1982; Mumford et al., 1994; Runco, 1994b).
The differences between these and other problem construction-related skills lie in (a) how well-
or ill-defined a problem is and (b) the degree to which ideation and evaluation are required

(Abdulla & Cramond, 2018). In our study, problem construction was operationalized as problem
discovery, which has been considered as the highest level of PF hierarchy (Abdulla & Cramond,
2018). We asked the participants to produce as many ideas as they could think of for different
and novel problems related to home and school, life situations, and health and well-being. They
were explicitly instructed to think of real or hypothetical problems, and to think of problems that
exist or those that might exist in the future. Previous work on instructions indicates that the focus
of instructions can have a significant impact on the outcomes associated with DT measures
(Acar, Runco, & Park, 2020; Nusbaum, Silvia, & Beaty, 2014; Runco, Illies, & Reiter-Palmon,
2005). The focus of the instructions is on both different and novel ideas may be one reason why
flexibility was such an effective predictor. This point is crucial because different results might be
attributed to the level of ideation and evaluation required in the problem construction task (Lee
& Cho, 2007). Thus, our findings might not be generalized to all problem construction tasks,
except those that require problem discovery where there is limited role for evaluation.

References
Abdulla, A. M., & Cramond, B. (2018). The creative problem finding hierarchy: A suggested
model for understanding problem finding. Creativity. Theories-Research-Applications, 5,
197-229. https://doi.org/10.1515/ctra-2018-0019
Abdulla, A. M., Paek, S. H., Cramond, B., & Runco, M. A. (2020). Problem finding and
creativity: A meta-analytic review. Psychology of Aesthetics, Creativity, and the Arts, 14,
3–14. https://doi.org/10.1037/aca0000194
Acar, S., Runco, M. A., & Park, H. (2020). What should people be told when they take a
divergent thinking test? A meta-analytic review of explicit instructions for divergent
thinking. Psychology of Aesthetics, Creativity, and the Arts, 14, 39–
49. https://doi.org/10.1037/aca0000256
Amabile, T. (1988). A model of creativity and innovation in organizations. Research in
Organizational Behavior, 10, 123–169.
Ambrosio, A. L. (1994). Social problem finding: A cognitive-developmental perspective.
Dissertation Abstracts International Section A, 54, 3700.
Arlin, P. K. (1975). A Cognitive process model of problem finding. Educational Horizons,
Winter, 99-106.
Artley, N. L., Van Horn, R., Friedrich, D. D., & Carroll, J. L. (1980). The relationship between
problem finding, creativity and cognitive style. Creative Child & Adult Quarterly, 5, 20–
26.
Basadur, M. (1995). Optimal ideation-evaluation ratios. Creativity Research Journal, 8, 63–
75. https://doi.org/10.1207/s15326934crj0801_5

Bouchard, T. J., & Drauden, G. (1976). Discovery-oriented behavior and problem solving. The
Journal of Psychology: Interdisciplinary and Applied, 92(2), 243–
248. https://doi.org/10.1080/00223980.1976.9921362
Brugman, G. (1995). The discovery and formulation of problems. European Education, 27(1),
38-57. doi.org/10.2753/EUE1056-4934270138
Carson, D. K., & Runco, M. A. (1999). Creative problem solving and problem finding in young
adults: Interconnections with stress, hassles, and coping abilities. The Journal of Creative
Behavior, 33, 167–190. https://doi.org/10.1002/j.2162-6057.1999.tb01195.x
Chand, I., & Runco, M. A. (1993). Problem finding skills as components in the creative
process. Personality and Individual Differences, 14(1), 155–
162. https://doi.org/10.1016/0191-8869(93)90185-6
Cropley, A. J. (2011). Definitions of creativity. In M. A. Runco & S. R. Pritzker (Eds.),
Encyclopedia of creativity (pp. 511-524). San Diego, CA: Academic Press.
Csikszentmihalyi, M. (1988). Solving a problem is not finding a new one: A reply to Simon. New
Ideas in Psychology, 6, 183-186. doi.org/10.1016/0732-118X(88)90003-7
Csikszentmihalyi, M., & Getzels, J. W. (1971). Discovery-oriented behavior and the originality
of creative products: A study with artists. Journal of Personality and Social Psychology,
19, 47–52. https://doi.org/10.1037/h0031106
Einstein, A., & Infeld, L. (1938). The evolution of physics. New York: Simon & Schuster.
Frederiksen, N., & Evans, F. R. (1974). Effects of models of creative performance on ability to
formulate hypotheses. Journal of Educational Psychology, 66, 67–
82. https://doi.org/10.1037/h0035808

Getzels, J. W. (1982). The problem of the problem. In R. Hogarth (Ed.), New directions for
methodology of social and behavioral science: Question framing (pp. 37-49). San
Francisco, CA: Jossey-Bass.
Hoover, S. M. (1994). Scientific problem finding in gifted fifth-grade students. Roeper Review,
16, 156–159. https://doi.org/10.1080/02783199409553563
Hoover, S. M., & Feldhusen, J. F. (1990). The scientific hypothesis formulation ability of gifted
ninth-grade students. Journal of Educational Psychology, 82(4), 838–
848. https://doi.org/10.1037/0022-0663.82.4.838
Jaarsveld, S., Lachmann, T., & van Leeuwen, C. (2012). Creative reasoning across
developmental levels: Convergence and divergence in problem creation. Intelligence, 40,
172–188. doi.org/10.1016/j.intell.2012.01.002
Kim, K. H. (2008). Meta-analyses of the relationship of creative achievement to both IQ and
divergent thinking test scores. The Journal of Creative Behavior, 42, 106–
130. https://doi.org/10.1002/j.2162-6057.2008.tb01290.x
Lee, H., & Cho, Y. (2007). Factors affecting problem finding depending on degree of structure
of problem situation. Journal of Educational Research, 101(2), 113–123.
doi.org/10.3200/JOER.101.2.113-125
Lee, I. A., & Preacher, K. J. (2013, September). Calculation for the test of the difference between
two dependent correlations with one variable in common [Computer software]. Available
from http://quantpsy.org.
Ma, H.-H. (2009). The effect size of variables associated with creativity: A meta-
analysis. Creativity Research Journal, 21, 30–
42. https://doi.org/10.1080/10400410802633400

Mackworth, N. H. (1965). Originality. American Psychologist, 20(1), 51–
66. https://doi.org/10.1037/h0021900
Merrifield, P. R, Guilford, J. P., Christensen, P. R., & Frick, J. W. (1962). The role of intellectual
factors in problem solving. Psychological Monographs: General and Applied, 76, 1–
21. https://doi.org/10.1037/h0093850
Mumford, M. D., Mobley, M. I., Uhlman, C. E., Reiter-Palmon, R., & Doares, L. M. (1991).
Process analytic models of creative capacities. Creativity Research Journal, 4, 91–
122. https://doi.org/10.1080/10400419109534380
Mumford, M. D., Reiter-Palmon, R., & Redmond, M. R. (1994). Problem construction and
cognition: Applying problem representations in ill-defined domains. In M. A. Runco
(Ed.), Problem finding, problem solving, and creativity (pp. 3–39). Ablex Publishing.
Nusbaum, E. C., Silvia, P. J., & Beaty, R. E. (2014). Ready, set, create: What instructing people
to “be creative” reveals about the meaning and mechanisms of divergent
thinking. Psychology of Aesthetics, Creativity, and the Arts, 8, 423–
432. https://doi.org/10.1037/a0036549
Okuda, S. M., Runco, M. A., & Berger, D. E. (1991). Creativity and the finding and solving of
real-world problems. Journal of Psychoeducational Assessment, 9, 45–
53. https://doi.org/10.1177/073428299100900104
Osborn, A. F. (1953). Applied imagination. Scribner's.
Reiter-Palmon, R., Forthmann, B., & Barbot, B. (2019). Scoring divergent thinking tests: A
review and systematic framework. Psychology of Aesthetics, Creativity, and the Arts, 13,
144–152. https://doi.org/10.1037/aca0000227

Reiter-Palmon, R., Mumford, M. D., O'Connor Boes, J., & Runco, M. A. (1997). Problem
construction and creativity: The role of ability, cue consistency and active
processing. Creativity Research Journal, 10, 9-23. doi.org/10.1207/s15326934crj1001_2
Runco Creativity Assessment Battery (2020, April). Creativity Testing Services. Retrieved from
https://www.creativitytestingservices.com/products
Runco, M. A. (1994b). Conclusions concerning problem finding, problem solving, and creativity.
In M. A. Runco (Ed.), Problem finding, problem solving, and creativity (pp. 272–290).
Ablex Publishing.
Runco, M. A. (2014). Creativity: Theories and themes: Research, development, and practice.
San Diego, CA: Academic Press.
Runco, M. A., & Acar, S. (2010). Do tests of divergent thinking have an experiential
bias? Psychology of Aesthetics, Creativity, and the Arts, 4, 144–
148. doi.org/10.1037/a0018969
Runco, M. A., Abdulla, A. M., Peak, S., Aljasim, F. A., & AlSuwaidi, H. N. (2016). Which test
of divergent thinking is best? Creativity. Theories-Research-Applications, 3, 4-18.
10.1515/ctra-2016-0001
Runco, M. A., Illies, J. J., & Reiter-Palmon, R. (2005). Explicit instructions to be creative and
original: A comparison of strategies and criteria as targets with three types of divergent
thinking tests. Korean Journal of Thinking & Problem Solving, 15, 5–15.

Said-Metwaly, S., Fernández-Castilla, B., Kyndt, E., & Van den Noortgate, W. (2020). Testing
conditions and creative performance: Meta-analyses of the impact of time limits and
instructions. Psychology of Aesthetics, Creativity, and the Arts, 14, 15–
38. doi.org/10.1037/aca0000244
Silverman, B.G. (1985). The use of analogs in the innovation process: A software engineering
protocol analysis. IEEE Transactions on Systems, Man, & Cybernetics, 15, 30-44.
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological
Bulletin, 87, 245–251. https://doi.org/10.1037/0033-2909.87.2.245
Torrance, E. P. (1966). Torrance Tests of Creative Thinking: Norms technical manual (Research
Edition). Princeton, NJ: Personnel Press.
Treffinger, D. J., & Isaksen, S. G. (2005). Creative Problem Solving: The history, development,
and implications for gifted education and talent development. Gifted Child Quarterly, 49,
342-353. doi.org/10.1177/001698620504900407
Wakefield, J. F. (1985). Towards creativity: Problem finding in a divergent-thinking
exercise. Child Study Journal, 15, 265–270.
Wallach, M. A., & Kogan, N. (1965). Modes of thinking in young children: A study of the
creativity-intelligence distinction. Holt, Rinehart & Winston.
Wallas, G. (1926). The art of thought. New York: Harcourt Brace and World.
Wertheimer, M. (1945). Productive thinking. Harper.

Table 1
Selected Studies on the Relation Between DT and Problem Construction
Authors
DT Test
N
DT Skills
Task
Nature
Main Findings
Frederiksen & Evans
(1974)
• Consequences Test
• 395
• Fluency
• Verbal
• The correlation between fluency and five
Formulating Hypotheses (FH) tests ranged
from .05 to .34.
Arlin (1975)
• TTCT
• Puzzles Test
• 60
• Expressional Fluency
• Associational Fluency
• Ideational Fluency
• Adaptive Flexibility
• Spontaneous
Flexibility
• Elaboration
• Verbal
• Figural
• A positive correlation was found between PF
quality and (a) elaboration, (b) adaptive
flexibility, (c) and expressional fluency; a
negative correlation was found between PF
quality and (a) spontaneous flexibility, (b)
ideational fluency, and (c) associational
fluency.
• A positive relation was found between PF
quantity and (a) spontaneous flexibility, (b)
ideational fluency, and (c) associational
fluency. PF quantity was negatively associated
with (a) elaboration, (b) spontaneous
flexibility, and (c) expressional fluency.
Artley et al. (1980)
• TTCT
• 84
• Composite
• Verbal
• TTCT positively correlated with PF quantity,
and negatively correlated with PF quality.
Wakefield (1985)
• DT Test developed by
the author
• 23
• Composite
• Figural
• DT and PF were positively correlated.
Ambrosio (1993)
• TTCT
• 60
• Composite
• Verbal
• DT and PF quality were negatively correlated,
whereas DT and PF and PF quantity were
positively correlated.
Chand & Runco
(1993)
• Presented Problems
Test (PP)
• Problem Generation
Test (PG)
• Discovered DT Test
(DP)
• 80
• Fluency
• Originality
• Verbal
• PG Test was positively correlated with fluency
and originality with both PP and DP (ranged
from r = .31 to .51).

Table 1 Continued
Selected Studies on the Relation Between DT and Problem Construction
Authors
DT Test
N
DT Skills
Task
Nature
Main Findings
Hoover (1994)
• TTCT
• 40
• Fluency
• Flexibility
• Originality
• Verbal
• TTCT fluency was positively correlated with
fluency in the FH test.
• TTCT flexibility was positively correlated
with fluency and originality in the FH test. No
significant relation was found between TTCT
originality and FH fluency and originality.
Reiter-Palmon et al.
(1997)
• Consequences Test
(Guilford, 1965)
• 195
• Ideational Fluency
• Category
Combination
• Verbal
• A negative correlation was found between
problem construction and DT.
Lee & Cho (2007)
• TTCT
• 115
• Composite
• Verbal
• Ill-structured PF task was negatively
correlated with DT, whereas PF and DT were
positively correlated with the moderate
problems.
Runco & Acar
(2010)
• Uses Test
• 81
• Fluency
• Originality
• Verbal
• DT fluency and originality were positively
correlated with PF fluency and originality.
Jaarsveld et al.
(2012)
• Test for Creative
Thinking-Drawing
Production (TCT-DP)
• 205
• Composite
• Figural
• DT and PF were not significantly associated.
Note: TTCT = Torrance Test of Creative Thinking; DT = Divergent Thinking; PF = Problem Finding

Table 2
Flexibility Categories for Verbal and Figural Tests
Task Nature
Categories
Verbal
Spoon
(1) carry or move things, (2) care and beauty, (3) design and decoration, (4) artistic uses, (5) scientific uses, (6) recycling, (7)
food and kitchen uses, (8) clean/cleaning, (9) stationery material, (10) game/use it in a game, (11) school/study uses, and (12)
other uses*
Wheel
(1) carry or move things, (2) recycling and manufacturing, (3) decoration, (4) blocks, (5) shelter for pets, (6) container, (7) sport
uses, (8) artistic uses, (9) musical instrument, (10) play and games, and (11) other uses*
Toothbrush
(1) cleaning, (2) care and beauty, (3) recycling, (4) artistic uses, (5) food and kitchen uses, (6) stationery material, (7) hanger,
and (8) other uses*
Figural
Spiral
(1) ideas/thoughts, (2) ropes, (3) nature (e.g., air, steam, smoke), (4) path/road, (5) food, (6) letters/numbers, (7) games, (8)
tools, (9) trees, and (10) others*
Blocks
(1) places, (2) games, (3) geometric figures, (4) mathematics, (5) roads/crossroads, (6) machines, (7) part of the human body,
(8) tools, (9) phone/tablets, (10) letter or words, and (11) others*
Lines
(1) road, (2) sorting things, (3) architecture, (4) stick/cane, (5) artistic and design uses, (6) furniture, (8) games, (9) musical
instruments, (10) levels, and (11) others*
Note: *Others = all responses that did not fit any of the mentioned categories.

Table 3
Interclass and Internal Consistency Reliability Coefficients of Study Instruments
ICC
Internal Consistency
Fluency - Uses
.81
.93
Flexibility - Uses
.84
.85
Originality - Uses
.83
.84
Fluency - Figural
.53 (.71)*
.61 (.73)*
Flexibility - Figural
.50 (.68)*
.64 (.69)*
Originality - Figural
.51 (.62)*
.55 (.62)*
Problem Generation - Fluency
.87
.88
Problem Generation - Originality
.87
.87
Note: * = After removing the Lines task; ICC = Interclass Correlation

Table 4
Correlations Between Study Variables (n= 90)
Fluency
(Uses)
Flexibility
(Uses)
Originality
(Uses)
Fluency
(Figures)
Flexibility
(Figures)
Originality
(Figures)
Fluency
(PG)
Originality
(PG)
Fluency
(Uses)
Flexibility
(Uses)
.87**
Originality
(Uses)
.93**
.72**
Fluency
(Figures)
.25*
.20
.26*
Flexibility
(Figures)
.29**
.24*
.30**
.89**
Originality
(Figures)
.27**
.27*
.30**
.80**
.73*
Fluency
(PG)
.73**
.75**
.67**
.22*
.26*
.28**
Originality
(PG)
.73**
.74**
.68**
.25*
.28**
.30**
.98**
Note: Uses = Uses Test; PG = Problem Generation Test
*p <.05 **p <.01.

Table 5
Regression Analysis of Divergent Thinking Skills with Problem Construction
Variable
B
SE
t
p
95% CI
Problem Construction (Fluency)
DT Verbal
Fluency
.03
.43
.08
.94
[-3.96–2.61]
Flexibility
1.17
.49
3.5
.001
[.741–2.69]
Originality
.52
.43
1.22
.227
[-.330–1.371]
R2 = .59
DT Figural
Fluency
-.88
1.13
-.78
.435
[-3.14–1.36]
Flexibility
1.18
1.23
.957
.341
[-1.27–3.63]
Originality
1.13
.72
1.57
.121
[-.31–2.58]
R2 = .08
Problem Construction (Originality)
DT Verbal
Fluency
-.07
.36
-.19
.848
[-.787 - .648]
Flexibility
1.47
.41
3.6
.001
[.650–2.29]
Originality
.57
.36
1.6
.117
[-.146–1.29]
R2 = .60
DT Figural
Fluency
-.66
.95
-.69
.49
[-2.55–1.23]
Flexibility
.94
1.03
.90
.37
[-1.12–2.99]
Originality
1.02
.61
1.67
.098
[-.19–2.23]
R2 = .09
Note: DT = Divergent Thinking

Figure 1. Line Chart of the Comparison Between the Number of Publications on Problem Construction, DT, and
Problem Solving in the Past Five Decades.

Figure 2. Histogram of the Comparison Between the Number of Publications on Problem Construction, DT, and
Problem Solving in the Past Five Decades.