Content uploaded by Gabe Avakian Orona
Author content
All content in this area was uploaded by Gabe Avakian Orona on Dec 02, 2024
Content may be subject to copyright.
The structure of adult thinking: A network approach to (meta)
cognitive processing
Gabe Avakian Orona
a,*
, Jacquelynne S. Eccles
b
, Sabrina Solanki
b
, David A. Copp
b
,
Quoc-Viet Dang
b
, Richard Arum
b
a
Hector Research Institute of Education Sciences and Psychology, University of Tübingen, Germany
b
University of California, Irvine, United States of America
ABSTRACT
Complex cognitive processes have been broadly categorized into three general domains: rst-order cognition (i.e., thinking directed to solve problems), metacog-
nition (i.e., thinking about one’s thinking during problem-solving), and epistemic cognition (i.e., thinking about the epistemic nature of problems and beliefs about
criteria for knowledge justication). Few, if any studies, have empirically examined the conditional dependencies between a large inventory of components
simultaneously. This paper aims to contribute the rst set of preliminary explorations into the interrelationships between different thinking and reasoning com-
ponents that represent key aspects of emerging adult cognitive processing using a psychological network approach. In two cross-sectional studies (combined N =
1496), data was collected from undergraduate students enrolled at a large public university. Scrutiny of the networks suggests that thinking dispositions and
competency with probability are key bridges between metacognitive abilities and epistemic beliefs. Implications for instruction are discussed.
Educational relevance statement: It remains a perennial aim of all education systems to improve the thinking and reasoning of students. But which complex cognitive
processes are worthwhile targets, and how do they t among the plethora of metacognitive, self-regulatory, and epistemological belief aspects of students? The
present set of studies is the rst to apply a network approach to a broad array of cognitive components to uncover the central student-level variables that can be
targeted with instruction. Based on the ndings of the two studies presented, instruction aimed at epistemic dispositions could potentially assist in the development of
complex cognition because of their centrality to networks of effective reasoning processes.
1. Introduction
The modern adult confronts a plethora of complexities that require
careful reasoning and reection on a daily basis. Many occupations have
discarded routine procedures, popular news outlets mislead, nancial
scams are increasingly savvy, and social media platforms promulgate
misinformation (Auxier & Anderson, 2021; Metts & Bressler, 2023; Shin
et al., 2018). Having the ability to make accurate inferences, monitor
what one knows and needs to know, and evaluate the verisimilitude of
conicting knowledge claims appear to be essential tools for navigating
the epistemic terrain of the 21st century (Greene & Yu, 2016; Orona
et al., 2023a).
Cognitive psychologists have described these reasoning abilities at
three distinct levels of processing: (rst-order) cognition, metacognition,
and epistemic cognition (Kitchner, 1983). However, while much research
has been conducted within each cognitive domain, little is known about
how the reasoning components from different theories and across
cognitive levels are related. In this manuscript, we present two explor-
atory studies utilizing a psychological network approach to shed light on
the conditional dependencies between cognitive components across
each level in attempt to gain a preliminary picture of the structure of
emerging adult thinking and reasoning processes. A vast array of com-
ponents are included to ascertain the central aspects linking diverse
forms of reasoning in college-aged students, providing generative ideas
for learning and instruction.
2. Conceptual framework
Kitchner (1983) introduced a three-level model of cognitive pro-
cessing, arguing that different types of situations and problems call for
different modes of reasoning. The crux of her argument is situated in the
distinction between well-structured and ill-structured problems. The
former set is dened by issues and tasks—dubbed “puzzles”—for which
there are clear rules and algorithms one can apply to generate a correct
solution. These problems have right and wrong answers (e.g., solving an
algebra equation). Conversely, the latter set is dened by issues and
tasks for which there are no clear algorithms leading to a correct solu-
tion; there are no right or wrong answers, though there may be solutions
* Corresponding author at: Europastraße 6, 72072 Tübingen, Germany.
E-mail address: gabriel.orona@uni-tuebingen.de (G.A. Orona).
Contents lists available at ScienceDirect
Learning and Individual Differences
journal homepage: www.elsevier.com/locate/lindif
https://doi.org/10.1016/j.lindif.2024.102584
Received 27 April 2024; Received in revised form 4 November 2024; Accepted 8 November 2024
Learning and Individual Dierences 117 (2025) 102584
Available online 30 November 2024
1041-6080/© 2024 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (
http://creativecommons.org/licenses/by/4.0/ ).
that are better than others according to some prescribed criteria or
dened set of goals (e.g., evaluating proposals to address the combined
issues of ination and unemployment).
These different types of problems require different types of cognitive
processes (Kitchner, 1983). Cognition, according to Kitchner (1983),
deals with well-structured problems; metacognition deals with moni-
toring the progress and strategies to solve those problems; and epistemic
cognition is involved to recognize, represent, and successfully navigate
ill-structured problems (Schraw et al., 1995). A useful view of the
components constituting each level, and their potential relationship, is
given by the understanding that metacognition is a subset of rst-order
cognition and epistemic cognition is a subset of metacognition
(Moshman, 2013). Fig. 1 depicts the three levels and some example
concepts and constructs. In the sub-sections below, we succinctly review
each cognitive levels’ dening features and general origins, current
expansions, and some specic constructs, with a view towards emerging
adult thinking. Additionally, we briey discuss an overarching frame-
work that integrates the three cognitive levels, as well as empirical
research relating different components to each other, before presenting
the current set of studies.
2.1. Cognition
First-order cognition, or simply “cognition”, is unequivocally the
widest set of the three domains, with skills and abilities ranging from
perception, (working) memory, visual/spatial processing, speed pro-
cessing, and any form of knowledge, thinking and reasoning (Moshman,
2020). However, with respect to cognition in late adolescence/emerging
adulthood, and within educational and cognitive psychology, scholarly
attention is directed towards advanced forms of reasoning and knowl-
edge strategies. Some of the more discussed constructs include cognitive
abilities such as critical thinking and reective thinking or rationality
(Alexander, 2023; Ganuthula & Dyaram, 2016; Halpern & Dunn, 2021).
Critical thinking involves “interpretation, analysis, evaluation, and
inference, as well as explanations of the evidential, conceptual, meth-
odological, criteriological or contextual considerations that judgment is
based upon” (Facione, 1990, p. 2). Assessments of critical thinking
therefore typically involve tasks requiring students to discern relevant
facts and evaluate the deductive and inductive soundness of arguments
(Liu et al., 2014). As a mainstay outcome of colleges and universities,
critical thinking has been examined in numerous studies throughout the
decades (e.g., Arum & Roksa, 2011; Orona et al., 2023b), and has
signicantly predicted real-world outcomes as measured by avoidable
negative life events (Butler et al., 2017), over and above general intel-
ligence (Ren et al., 2020).
Rationality (sometimes referred to as reective thinking; Ivory et al.,
2023), on the other hand, involves reective operations that assist in
resisting the cognitive biases which obscure sound reasoning processes
(Grimm & Richter, 2024). The heuristic-and-bias research base has
introduced a set of tasks assessing such processes, such as when prior
beliefs interfere with evaluating the logic of arguments (belief bias syl-
logisms), when individuals seek and validate information that conrms
their existing beliefs while ignoring opposing data (conrmation bias),
and forms of probabilistic reasoning that–while simple–present common
pitfalls (conjunction fallacy; Stanovich, 2016). Empirically, latent vari-
able approaches assessing scores on biases and intuitive responding
tasks have shown moderately positive, though non-overlapping, re-
lations with general intelligence, suggesting rationality as a distinct
mental capacity (Burgoyne et al., 2023). Moreover, recent research from
Grimm and Richter (2024) show that rational thinking is predictive of
academic success at university, over and above intelligence.
Fig. 1. The structure of adult cognition with example concepts and constructs. Epistemic cognition is a subset of metacognition, which is a subset of cognition
(Moshman, 2020). Key concepts and constructs are informed by contemporary models of higher-order cognitive abilities, such as Kuhn (2022), Halpern and Dunn
(2021), and Sinnott (2021), as well as others (e.g., Sinnott, 1998).
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
2
2.2. Metacognition
Originally introduced by Flavell (1976), metacognition was
described as “one’s knowledge concerning one’s own cognitive process
and products, or anything related to them, e.g., the learning-relevant
properties of information and data” (p. 232). Under this original
formulation, metacognition entails a set of self-monitoring tools in-
dividuals employ when engaging in problem-solving and/or learning
tasks. In essence, it is an evaluation of one’s thought processes. Thus, it
has received the intuitive–yet deceptively simplistic–description of
“thinking about thinking” (Livingston, 2003).
Flavell’s (1979) early studies showed that adolescents and older
children are more capable of reporting their memory of facts than
younger children. That is, while young children display self-monitoring
capacity early on, the accuracy in gauging one’s state of knowledge
appeared to clearly progress with age. This developmental pattern was
also identied years later by Vukman (2005), who examined accuracy in
strategy detection up to mature adulthood.
Seminal work by Schraw et al. (1995) later solidied two dening
attributes of metacognition: knowledge of cognition and regulation of
cognition. Both aspects are vital for learning and are therefore particu-
larly salient in educational settings (Fernandez-Duque et al., 2000;
Swanson, 1990; van der Wel, 2015). But after nearly 50 years of
research, the understanding of metacognition has become so wide and
encompassing as to include many different constructs that are generally
related to regulating and directing one’s thoughts, actions and behav-
iors. Expanded facets of metacognition now include: executive function,
self-regulation, and even self-control and conscientiousness (Inzlicht et al.,
2021; Kuhn, 2022; Moshman, 2020). Part of this wider understanding of
metacognition involves directing reection beyond an individual’s own
knowledge/strategies for knowing to the knowledge acquisition pro-
cesses itself. This aspect of metacognition is termed epistemic cognition
and is the third level of complex cognitive processing.
2.3. Epistemic cognition
Epistemic cognition involves awareness of and sensitivity to the
epistemological issues involved in a problem. Despite a close connection
between the two, a key distinction between metacognition and
epistemic cognition concerns the object of deliberation. The questions
shift from, “What do I know?/What strategy should I employ?” to “How
can I know?/What are the relevant criteria to judge knowledge claims?”
Epistemic cognition involves coordinating the subjective and objective
elements in context and is described as the most distinctively adult form
of thinking of the three cognitive levels (Kallio, 2020).
While the term was introduced by Kitchner (1983), Perry’s (1970)
scheme of intellectual development is widely recognized as the pio-
neering work in epistemic cognition, being among the rst to reveal how
traditional college-age learners’ thinking evolved from a dualistic/
objective worldview to one where context and contingencies highlight
the complexities with knowledge claims. Studies have largely validated
Perry’s (1970) developmental positions (Erwin, 1983; Fago, 1995), and
the scheme has been indirectly conrmed with a plethora of model
variants in the neo-Piagetian tradition (e.g., Belenky et al., 1986; King &
Kitchener, 1994; Kuhn et al., 2000; Sinnott, 1998).These models have
accompanying instruments that measure epistemic cognition as a gen-
eral cognitive structure developing largely in emerging adulthood and
operating in a wide variety of contexts and situations (Kallio, 2020;
Mason, 2016). For example, in Kuhn et al.’s (2000) prominent model of
epistemological understanding, individuals develop along a continuum
that bespeaks coordinating subjectivity and objectivity. Initially, people
begin with strong objective views across domain areas (e.g., values,
personal taste, science, etc.) that correspond to beliefs that knowledge is
certain, and assertions bespeak reality—the Absolutist stage. As their
thinking matures, they recognize the subjectivity in knowledge claims
giving way to a multitude of interpretations, rendering the belief that
knowledge is generated by human minds, which leads to the notion that
assertions are mostly objects of opinion—the Multiplist stage. As devel-
opment continues further still, these two extremes are coordinated and
reconciled in viewing knowledge as generated by human minds and
uncertain, yet subject to critical evaluation against external criteria to
make (better) judgements—the Evaluativist stage. The levels of epistemic
understanding thereby extend beyond beliefs but have implications for
the use and perceived relevance of critical thinking: Multiplists’ may
view critical thinking as irrelevant because all evaluations are equally
valid, whereas Absolutists may be overly condent in right/wrong an-
swers, or too narrow in their response to complex problems (Kuhn et al.,
2000).
However, thinking about knowledge criteria and the process of
knowing no longer singly denes this cognitive level (Chinn et al.,
2011). Just as with metacognition–and more than likely due to its close
connection with it–recent years have witnessed an explosion of
epistemic cognition scholarship beyond the classical formulation
(Greene et al., 2016). That is, the developmental frameworks mentioned
above have been supplemented with multidimensional approaches that
posit several independent aspects (Hofer & Pintrich, 1997). Such di-
mensions refer to: (a) the certainty of knowledge, (b) complexity of
knowledge, (c) sources of knowledge, and (d) the justication of
knowledge (Hofer & Pintrich, 1997). Still, further expansions now
incorporate components such as theory of mind, perspective-taking
(Kindlinger, 2021), disciplinary and topic specic beliefs (Barger et al.,
2018; Merk et al., 2018), epistemic metacognition (Barzilai & Zohar,
2014), situated resources (Mason, 2016), and thinking dispositions such
as intellectual curiosity and open-mindedness which are known as
epistemic virtues (Chinn et al., 2011).
Theoretically, this expansion is made most apparent in the AIR
(Aims, Ideals, and Reliable processes of knowing) model of epistemic
cognition (Chinn et al., 2011; Chinn & Rinehart, 2016), which argues
that epistemic cognition is a complex, multifaceted, contextual set of
processes related to achieving epistemic goods, such as knowledge,
truth, and accuracy, as opposed to a single construct. Eschewing the
characterization of mature epistemic thinking as restricted to uncer-
tainty in knowledge claims, Chinn and colleagues argued for a situated
analysis of learning that takes inventory of how individuals pursue and
value their epistemic activities (Aims), what criteria they select and why
(Ideals), and the strategies they employ to achieve those aims and
compare the outputs of their thinking activities against ideals (Reliable
processes of knowing). Differences in both aims and ideals across tasks
and persons can therefore explain why some people support one
explanation over another, or why some choose to engage in certain
knowing strategies (e.g., checking the trustworthiness of references on a
website) and others do not (e.g., if the aim is to merely win an argument
over knowing the truth; Chinn & Rinehart, 2016). On this account,
motivational thinking dispositions, otherwise known as epistemic vir-
tues, play a role in tying aims to processes because they represent stable
tendencies to inquire, persist, and pursue epistemic goods (Chinn &
Rinehart, 2016; Orona et al., 2023a).
While the AIR model and its implications for educational practice
continues to gain inuence in the literature (Chinn et al., 2021), most
empirical scholarship in epistemic cognition continues to leverage sur-
vey measures corresponding to either developmental or multidimen-
sional models (Greene et al., 2018), but with the addition of other
epistemic components, such as dispositions, emotions, and more (Muis
et al., 2021; Thacker & Sinatra, 2022). This has led to an expansive
understanding of what constitutes epistemic cognition, with beliefs
about knowledge and knowing being one—albeit central—piece in a
larger system of concepts and constructs related to how and why people
interact with information in the ways they do. Thus, to adequately de-
pict this system, one must take inventory of a broad range of components
that represent these diverse views. Therefore, in this manuscript, we
intentionally incorporate constructs—across a variety of theories and
models—that instantiate a variety of facets of epistemic cognition in
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
3
hopes to reect this diversity.
2.4. Theoretical perspectives tying the cognitive levels together
When interpreted in the abovementioned ways, the three cognitive
levels have clear distinctions, with each one demarcated by various
objects of deliberation. Such distinctions are an oversimplication:
many tasks involve processing at multiple levels. For instance, the
famous bat-and-ball problem
1
from the cognitive reection test can be
viewed as rst-order cognition whereby a correct solution depends on
basic reading, algebra, and analytic skills (Frederick, 2005). But the
extended deliberation that solution is posited to rely on might involve
one’s metacognitive skill (e.g., thinking about their strategies to solve
the problem). Still, the tendency to deliberate more generally and apply
effortful thinking can be viewed as an epistemic virtue, recruiting as-
pects of epistemic cognition. That is, in one single task, multiple facets
from each cognitive level may be at play (Greene et al., 2024; Kuhn,
2022).
Thus, several frameworks and models have built upon Kitchener’s
classication towards an integration of the three levels, specifying the
dynamic role different cognitive components contribute to thinking,
reasoning, and knowing. For instance, Kuhn (2001, 2022) introduced a
cognitive map that details how metacognition and epistemic thinking
determine the use of reliable processes of knowing, such as inference,
analysis, argument, and inquiry (e.g., critical thinking abilities). Ac-
cording to Kuhn (2001, 2022), meta-level knowing of two varieties can
inuence effective thinking. First, meta-level knowledge of how (pro-
cedural) to use and regulate critical thinking skills and strategies de-
termines the extent to which they are efciently applied, entailing an
individual’s competence in using them. However, successful reasoning
isn’t solely dependent on “how-to” knowledge of strategies, but also
involves what (declarative) individuals’ think about knowledge and
knowing, and whether such strategies are worthwhile. This second type
of meta-level knowing involves wrestling with the epistemological un-
derpinnings of problems, entailing one’s epistemological beliefs/un-
derstanding (e.g., believing knowledge is certain or relative, etc.).
Epistemological beliefs/understanding inuences intellectual values (e.
g., “Can analysis be worthwhile?”, “Is there a point to arguing?”), which
in turn shapes the disposition(s) to enact reliable processes of knowing
(e.g., inference, analysis, argument, inquiry).
With her cognitive map, Kuhn (2022) has not only integrated several
key dimensions of cognition in a coherent system but has also suggested
the simultaneous relevance of various dispositional factors, meta-
cognitive strategies, and epistemic processing components as they shape
reasoning performance. That is, someone may have the metacognitive
skills to reect on their use of knowing strategies, but not the epistemic
values and dispositions to deploy them. As such, researchers have
examined how constructs representing different levels of cognition
interact with and/or inuence one another in an array of contexts. Some
of the empirical literature that has examined cross-level constructs in
emerging adult age groups will now be reviewed.
2.5. Empirical research examining the cognitive levels together
2.5.1. Cognition and metacognition
A vast body of research details how different metacognitive pro-
cesses relate to cognitive performance. For example, experimental work
shows that university students randomly assigned to a metacognitive
instructional treatment signicantly outperformed the control condition
in mathematical reasoning, as well as in self-reported metacognitive
knowledge. (Mevarech & Fridkin, 2006). In a more recent study, Rivas
et al. (2022) introduced an intervention aimed at enhancing critical
thinking skills via metacognitive instruction. Their ndings showed that
participants signicantly increased on several dimensions of critical
thinking from pre to post test. And very recently, a causal effect in the
opposite direction was observed: a critical thinking training intervention
was shown to enhance metacognitive sensitivity (Simonovic et al.,
2023).
Non-intervention research complements the positive experimental
effects found between metacognition and cognition. In a longitudinal
design, Magno (2010) found that a latent variable composed of items
from the Metacognitive Assessment Inventory self-report instrument
positively and signicantly predicted university students logical
reasoning scores (measured via the Wason card task) across different
model specications. Kozikoglu (2019) found a bivariate relationship
between metacognitive skills and critical thinking tendencies of r =
0.62. Other sub-variants of metacognition, such as executive function,
have been directly linked to critical thinking. In a study by Li et al.
(2021), two key aspects of executive function, updating (measured via
the running memory task) and inhibition (measured via the antisaccade
task), signicantly predicted critical thinking in a university sample over
and above intelligence and thinking dispositions, with standardized
coefcients of 0.16 and 0.15, respectively.
2.5.2. Cognition and epistemic cognition
Previous research has often related epistemic cognition to argu-
mentation aspects of critical thinking. For instance, Mason and Scirica
(2006) utilized Kuhn et al.’s (2000) scale to obtain epistemological
understanding scores, which were subsequently used to predict argu-
mentation abilities. Their ndings suggest that an evaluativist (e.g.,
viewing knowledge as subject to evaluative criteria) perspective
strongly predicts the formation of arguments, counterarguments, and
rebuttals, over-and-above knowledge of and interest in the provided
topic. Weinstock et al. (2006) used the same assessment to examine the
identication of informal reasoning fallacies across epistemological
levels. Similarly, Baytelman et al. (2020) utilized the Dimensions of
Epistemological Beliefs towards Science instrument in predicting how
sophisticated epistemic beliefs relate to argument quality and diversity.
Aside from argumentation variables, constructs such as reective
judgement have been positively correlated with classical critical
thinking assessments (Jensen, 1998). Muis et al. (2021) showed that
epistemic cognition (beliefs about the complexity of knowledge) pre-
dicts critical thinking about socio-scientic issues, as well as more
curious emotions when evaluating complex text. Dyer and Hall (2019)
used critical thinking to predict epistemically unwarranted beliefs,
showing a negative relationship (i.e., higher critical thinking scores
were related to lower levels of endorsing paranormal and conspiracy
beliefs). And Thacker and Sinatra’s (2022) study showed how epistemic
cognition contributes to undergraduates’ ability to learn from data.
Moreover, their ndings suggest that actively open-minded thinking (a
key epistemic virtue) moderated the inuence of their epistemic
cognition intervention on conceptual change.
2.5.3. Metacognition and epistemic cognition
The conceptual proximity between metacognition and epistemic
cognition as two forms of meta-level knowing (Kuhn, 2022) has led to
many studies examining them together. For instance, adding meta-
cognitive enriching exercises within college classes increased epistemic
cognition measured via self-report (Wyre, 2007). Muis and Franco
(2010) showed that students who exhibit both rational and empirical
epistemic proles utilized greater metacognitive strategies and problem-
solving skills. Iordanou et al.’s (2019) results showed that participants
who have prior knowledge of a topic, and who view knowledge as
culminating and requiring justication (as opposed to either fail-proof
evidence or personal justication alone), utilized scientic standards
of evaluation to a greater extent than those with different epistemic
orientations. And recent neurological evidence revealed a nuanced
nding simultaneously showing the connection and dissociation
1
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.
How much does the ball cost?
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
4
between two levels of perspective-taking (Healey & Grossman, 2018).
While cognitive perspective-taking (e.g., ability to infer thoughts/be-
liefs) is dependent on executive function, affective perspective-taking (e.
g., ability to infer feelings and emotions) is to a much lesser extent
(Healey & Grossman, 2018).
Furthermore, the close connection between metacognition and
epistemic cognition is most prominent in the relatively newer distinction
of epistemic metacognition (Barzilai & Zohar, 2014). Essentially,
epistemic metacognition is metacognitive thinking applied to epistemic
problems. Important studies have shown the positive relationship be-
tween epistemic cognition and epistemic metacognition, particularly in
evaluating divergent sources of information (Barzilai & Ka’adan, 2017)
and developing scientic explanations (Tang, 2020). More work is
needed to fully tease these concepts apart, as epistemic understanding is
a type of meta-level knowing which is intrinsic to metacognition (as
Kuhn, 2022, p. 82, notes, “The foundation for this work [epistemic
cognition] is metacognitive”).
2.6. The present set of studies
As the review indicates, there are many different aspects involved in
complex cognitive processing, and cross-domain constructs have been
researched together in a variety of ways. With so many diverse in-
stantiations of cognition, metacognition, and epistemic cognition, it is
imperative to understand how different characteristics of each domain
relate to others within and across cognitive levels. Thus, many measures
and constructs from different research traditions and elds are required
to obtain a coherent understanding of complex reasoning. The impor-
tance of potential inquiries of this sort can be afrmed by recognizing
the educational value of conveying comprehensive information that
generates ideas relevant to cultivating individual attributes that foster
critical, reective thinking habits and learning skills in students. Un-
fortunately, however, previous research has yet to simultaneously
examine how a diverse array of components representing all three cognitive
levels give way to a broader picture of emerging adult thinking. There are
several plausible explanations for this, as noted by Tamura et al. (2022).
First, many of the constructs listed above belong to scholarly sub-elds
unto themselves (e.g., heuristic-and-bias literature). Relatedly, theories
within each eld prescribe testing nuanced relations between specic
constructs. And nally, until recently, the disciplinary perspectives
studying these constructs, such as (educational/cognitive) psychology,
has only recently introduced a statistical framework–psychometric
network analysis–that can accommodate a systems approach to incor-
porating diverse measures of complex cognition.
This paper intends to contribute the rst set of preliminary explo-
rations of the interrelationships between different thinking and
reasoning components that represent key aspects of college student
cognitive processing. Two cross-sectional studies are presented. In both
studies the aim is to explore the structure of emerging adult thinking in
undergraduate students, examining how components between and
within cognitive domains relate to one another. The strength of a multi-
study approach is that a larger set of components can be examined, and
similarities and differences in results across studies can be compared.
The selection of components is largely based upon the different aspects
related to cognition, metacognition, and epistemic cognition informed
by Kuhn (2022), which locates a variety of critical thinking operations
(e.g., analysis, inference, deduction) as related to meta-level strategies,
skills, dispositions, and epistemologies. While the general selection is
informed by Kuhn (2022), additional theoretical perspectives are
considered to secure a broad, diverse picture of how various constructs
instantiate the general concepts featured as relevant to the three-level
model of cognitive processing in emerging adults (e.g., Chinn et al.,
2011; Greene et al., 2016; Kitchner, 1983; Sinnott, 2021). As an example
of our process for node incorporation, while Kuhn (2022) centered
metacognition mostly on the classic notion of “thinking about thinking”
and inhibitory control, other scholars view it as intertwined and
inseparable from self-regulatory behaviors and strategies (e.g., Frazier
et al., 2021; Inzlicht et al., 2021)—we sought to include both under the
metacognitive purview to reect the diversity in conceptualizations.
Therefore, the list of variables used is neither strictly informed by one
theory nor exhaustive of all components under each cognitive level, but
rather can be viewed as an initial attempt to understand the conditional
dependencies between key components listed in contemporary theo-
retical perspectives. The broad perspective for variable incorporation
was applied to the other cognitive levels (rst-order and epistemic), too.
Thus, the overarching research question (RQ) across both studies is the
following:
•RQ: How are the components associated with various cognitive do-
mains (rst-order cognition, metacognition, and epistemic cogni-
tion) connected when simultaneously considered, and which are
central to this system?
To obtain a holistic description of the general structure of cognitive
processing across these domains, and to ascertain the important con-
nective nodes in a system, it is recommended to use a psychological
network approach (Borsboom et al., 2021). In contrast to traditional
psychometric techniques, such as latent variable modeling, which
emphasize the covariation between a set of items, network modeling
enables one to empirically depict adult cognition as a dynamic
ecosystem of interacting processes (Waldorp et al., 2022). This approach
does not emphasize constructs or factors to be extracted and interpreted;
rather, it focuses on the patterns in multivariate data that give way to
how a system operates. Because relations between individual nodes
(variables) are estimated while simultaneously controlling for all other
components in the system, networks offer generative clues for future
hypothesis testing (Borsboom et al., 2021). In the present case, specif-
ically, our aim is consistent with what network analysis can offer by
focusing on a large swathe of components relevant to the system of
emerging adult cognition, attempting to explore the relations—not be-
tween individual items and a latent factor—but to other salient features
of the system.
3. Study 1: cognition networks using performance assessments
In this study, as with study 2, we utilize cross-sectional data to depict
a broad picture of adult cognition. The aim of study 1 is to examine the
interrelationships between different thinking and reasoning components
that represent key aspects of emerging adult cognitive processing using a
psychological network approach. To accomplish this, we instantiate
concepts under each domain with various scales and assessments. Sta-
bility tests of the results using bootstrapping and available items are
provided in the supplementary material.
3.1. Methods
3.1.1. Participants
Participants were taken from the UCI-MUST (University of Califor-
nia, Irvine — Measuring Undergraduate Success Trajectories) study
(Arum et al., 2021a; Arum et al., 2021b) at a highly diverse public
university in southern California. Two samples of undergraduate stu-
dents (incoming freshman and returning juniors) enrolled in fall 2019
and fall 2021 were asked to participate via an email sent directly to their
university email address; the samples were combined for the present
study. Before completing any tasks, students signed a consent form
outlining the procedure and general aims of the study, which also stated
that none of their identifying information would be made available. This
took place at the start of the two terms, respectively. A total of 2546
students enrolled in the study. However, of these, 1180 decided to
continue with taking the performance assessments. Of these, 97 %
completed all assessments. Thus, the combined sample for this study is n
=1164. Demographics of the sample followed closely that of the
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
5
university population (although male students were underrepresented;
UCI Demographics, 2022–23; Orona et al., 2023b), with: 68 % female,
48 % rst-generation college students, 30 % underrepresented minority
students (Hispanic, Black, and Native American), and approximately 39
% indicated as low-income students in the university records.
3.1.2. Measures
A total of 10 components were assessed. Because measures are not
positioned on the same scale, all variables were min/max normalized to
have a minimum of 0 and a maximum of 1. This kind of normalization
enables interpretable and comparable means (Moeller, 2015). The
measures used in this study can be viewed in Table 1. Each measure is at
the subscale level (unless where tests rely on a single score, like the
perspective-taking assessment). Where appropriate, the internal con-
sistency estimates (Cronbach’s
α
or Kappa in Table 1) are reported.
3.1.2.1. Cognition measures. As indications of the cognitive level,
participant scores on the HEighten Critical Thinking (HCT) exam and a set
of conrmation bias tasks were utilized. The HCT has been conceptually
and empirically developed to assess two dimensions: Analytic Critical
Thinking and Synthetic Critical Thinking (Liu et al., 2016). The former
dimension emphasizes deconstructing arguments and evaluating their
validity; the latter on generating valid inferences by recognizing rele-
vant information. Individual items were not available to the researchers
due to propriety reasons; however, the instrument has been validated in
numerous studies in the United States (Liu et al., 2016), China (Liu et al.,
2018), Russia (Shaw et al., 2020), and Ireland (O’Leary et al., 2020).
Additionally, data from novel conrmation bias tasks developed by ETS
(Educational Testing Service) and the UCI-MUST team were collected.
Each participant was presented with three tasks and were asked to select
information that either supported or opposed their initial, primed
choice. Each conrmation bias score ranged from 0 to 6 (before min/
max transformation). The composite CB score was reverse-scored such
that higher scores represent lower instances of selecting information
consistent with an initial choice (e.g., greater resistance to bias).
3.1.2.2. Metacognition measures. As indications of the metacognitive
level, three calibration scores were used. Participants were asked to rate
their general level of critical thinking, perspective-taking, and ability to
evaluate false/epistemically suspect information relative to their peers.
The standardized scores from the self-reports were subtracted from
standardized scores of participants actual performance on these assess-
ments. Since knowing what one knows is a dening characteristic of
metacognition, we utilized the absolute (ignoring whether participants
were under or over-condent) score in the network model(s). Addi-
tionally, a composite of 8 items corresponding to self-regulation and
planning (SRP) from the academic self-efcacy scale was incorporated
(Chemers et al., 2001). An example item is as follows, “How good are you
at: Scheduling your time to accomplish your tasks?” All items are shown in
the supplemental material.
3.1.2.3. Epistemic cognition measures. As indications of the epistemic
cognitive level, the civic online reasoning (COR) assessment and a
perspective-taking assessment were implemented. The COR was devel-
oped by McGrew et al. (2018, 2017), Wineburg et al. (2016) and their
colleagues. For the COR, participants must respond to two prompts. The
prompts ask participants to juxtapose news headlines and accompanying
text taken from the articles. Participants must read the information and
provide a rational for and judgement on the verisimilitude of the claims
and the trustworthiness of the news source. The responses are scored on
a three-point scale, with 0 =Needing improvement, 1 =Intermediate,
and 2 =Mastery based upon: (a) selection of the correct choice and (b)
explanation for why one source is more trustworthy than the other.
We conceptualize the COR under epistemic cognition for several
reasons. First, it corresponds with the processes involved in multidi-
mensional perspectives of epistemic cognition, specically that relating
to source justication and credibility. A students’ thinking–should they
earn a high score–must attend to epistemic criteria, such as the trust-
worthiness of information (e.g., Barzilai & Zohar, 2014 examined this
process with a think-aloud protocol). This is quite distinct from the
classical CT assessment also employed in this study, where the correct
choices are not only visible (multiple choice, etc.), but where the skills
involved are localized to argument structure, deduction, and induction.
The perspective-taking (PT) test is a novel assessment developed in
collaboration with ETS and the UCI-MUST team (Arum et al., 2021a;
Arum et al., 2021b; Orona et al., 2023b). Respondents are presented
with a vignette describing two ctional characters who nd themselves
in a mildly confrontational situation. Details on the characters’ goals,
personalities, and general ambitions are provided. Respondents are then
tasked with representing the goals and emotions of each character and
are subsequently required to offer a solution to the confrontation that’s
sensitive to the ambitions and emotions/personality of each character.
Participants were scored on how well they articulate the perspectives of
both characters and whether their suggestions correspond to the facts of
the scenario and ambitions of the two characters. The responses were
evaluated on a ve-point scale, with 1 =Insufcient, 0 =Limited, and 3
=Adequate, 4 =Strong, and 5 =Outstanding. Two raters scored the
responses, generating an interrater reliability estimate of 0.69.
We conceptualize the PT assessment under epistemic cognition for
several reasons. While the PT may initially appear to not involve
epistemic awareness in the contemporary sense, it corresponds directly
with the original formulation of epistemic cognition introduced by
Kitchner (1983) as dealing with ill-structured problems that require
balancing multiple perspectives. Second, PT is a construct relevant to
one’s epistemic development when viewed as a mature extension of
theory of mind, which has a chapter dedicated to the topic in the
Handbook of Epistemic Cognition (Greene et al., 2016). The chapter by
Sodian and Kristen (2016, p. 79) states that, “Higher-order perspective
taking is related to a deepened metaconceptual understanding of the
knowing process”. And, therefore, perspective-taking (and theory of
mind) “addresses the developmental origins of epistemic cognition”
(2016, p. 69). Third, the PT assessment requires test-takers to coordinate
the subjective and objective elements in context, thereby balancing
abstract notions of fairness, equity, and fact to present a novel solution
that may, or may not, be the best way forward, but must be argued for.
Such an integration bespeaks the kind of higher-level processing char-
acteristic of epistemic cognition (King & Kitchener, 1994).
As mentioned in the conceptual framework, epistemic virtues are
dispositions that assist in apt epistemic performance (Chinn et al., 2021,
Table 1
Descriptive statistics of cognitive measures.
M SD Reliability
(
α
| Kappa)
Cognition
Analytic: heighten critical thinking (CTA) 0.399 0.245 0.97 |
Synthetic: heighten critical thinking (CTS) 0.401 0.244 0.97 |
Conrmation bias 0.413 0.21 .
Metacognition
Calibration for CT total (CCT) 0.275 0.197 .
Calibration for PT (CPT) 0.196 0.155 .
Calibration for COR total (CCOR) 0.214 0.153 .
Self-regulation and planning (SRP) 0.686 0.163 0.86 |
Epistemic cognition
Civic online reasoning (COR) 0.312 0.309 0.76 |
Perspective-taking (PT) 0.539 0.196 | 0.69
Need for cognition (NFC) 0.660 0.179 0.76 |
Note. Reliability with “.” =not applicable. The CTA and CTS reliability are not
generated for the current sample; these are reported from the test development
study conducted by Liu et al., 2016. The COR reliability is based upon four
scores: two items by two raters. The Kappa scores for each COR item were 0.782
and 0.672, respectively.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
6
2011). Thus, as another indication of constructs at the epistemic
cognitive level, the six-item Need for Cognition (NFC) scale validated by
Lins de Holanda Coelho et al. (2020) was administered. An example item
is: “I really enjoy a task that involves coming up with new solutions to
problems.” We conceptualize NFC as under the epistemic cognition
purview because it embodies a tendency to enact reliable processes of
knowing, i.e., engaging analytic thinking and extending deliberation.
While we recognize that NFC is not localized to epistemic assumptions,
beliefs, or criteria, these are not the dening attributes of epistemic
dispositions. Rather, a disposition characterized by a fundamental
motivation to love truth and hence enjoy the (thinking) process that
secures epistemic goods, like knowledge and accuracy, is widely un-
derstood as the dening feature(s) of a thinking virtue (Chinn et al.,
2011; Baehr, 2015; Zagzebski, 1996). Moreover, NFC is closely identi-
ed with intellectual curiosity, which is considered a foundational
epistemic virtue (Orona, 2021a; Orona & Pritchard, 2022; Ross, 2020).
And nally, Sinatra (2016, p. 486)—in the Handbook of Epistemic
Cognition—explicitly connects NFC to epistemic thinking: “Thinking
dispositions such as whether one is impulsive or reective, or high or low in
need for cognition, do matter and do relate to how one thinks about and with
knowledge.” Thus, we include NFC
2
in our network as an epistemic
cognition component.
3.1.3. Data analysis
A Gaussian graphical network model (GGM)–which is a class of
pairwise Markov random eld (PRMF) models–is applied to the cogni-
tive, metacognitive, and epistemic cognitive measures to ascertain the
psychometric structure of adult thinking. GGMs are undirected,
weighted network models that employ partial correlations. What this
means is that, as nodes (variables) are entered into the network, the
connections between them do not have arrowheads (that is, the causal
direction is not assumed). Moreover, these models allow edges (con-
nections) to be interpreted as partial correlations between any two
nodes (variables) in the network, holding constant every other node
(Borsboom et al., 2021). Thus, such models are networks of conditional
associations (e.g., conditional on every other node in the network), and
two nodes can be interpreted as independent of one another when no
edge is estimated (Epskamp et al., 2018).
The GGM network model was chosen not only because it permits the
system perspective in contrast to traditional latent variable models, but
because our data are non-binary (e.g., 0/1 response categories), in
which case, an Ising network model would have been appropriate.
What’s more, GGMs are commonly used in cross-sectional psychological
research (Epskamp et al., 2018). And to this point, importantly, the
phenomena studied in the present investigation(s), are not theorized to
uctuate dramatically within individuals; indeed, constructs such as
critical thinking, need for cognition, and so on are stable attributes that
require time, education, and tailored intervention to boost. Thus, a GGM
applied to cross-sectional data in this kind of situation—in contrast to if
we were investigation feelings, emotions, or daily habits—is likely to
avoid confusing within and between-subjects variation, permitting a
between-subject interpretation of the results (Epskamp et al., 2018).
As with any multivariate statistical model, a parsimonious structure
is desired that is neither too unconstrained (e.g., not enough edges
estimated) nor saturated (e.g., every edge estimated). One common
approach to regularizing GGMs, which is leveraged in this study, is the
graphical least absolute shrinkage and selection operator (GLASSO),
which penalizes complexity by estimating some edge weights to exactly
zero (Friedman et al., 2008). Penalization is dictated by a tuning
parameter, λ. Based on its success with PRMF structures (Chen & Chen,
2008; Van Borkulo et al., 2014), the value of λ—and henceforth the
amount of penalization—is determined by optimizing the extended
Bayesian Information Criterion (EBIC). The EBIC in turn is regulated by
a hypertuningparameter, γ; when high specicity is valued, γ is typically
set to 0.5, as is done in the current study. This parameter controls the
extent to which simpler models are preferred (Blanken et al., 2022). The
EBIC is therefore a metric which buffers model complexity, nding the
best tting or most parsimonious network structure is one that avoids
estimating (all or most) spurious connections (Chen & Chen, 2008).
Because of both its conservative effects on complexity and its success
with GGMs (Epskamp et al., 2012), we leverage EBIC regularization in
this study.
Employed in an exploratory manner, the current GGM can provide
insights for hypothesis generation and subsequent testing, while offering
a preliminary picture of emerging adult cognition. Such network models
have been used in a similar fashion to study IQ (McGrew et al., 2023)
and motivation (Tamura et al., 2022), respectively. Importantly, we
follow the reporting standards for psychological networks applied to
cross-sectional data outlined by Burger et al. (2023). All analyses were
conducted with R version 4.1.2. The bootnet package (Epskamp et al.,
2018) was used to estimate the GGM; and the igraph package was used to
visualize it (Epskamp et al., 2012).
4. Results
Table 1 showcases the descriptive statistics for the variables in the
analysis after normalizing each variable to have a minimum of 0 and a
maximum of 1. Thus, the means and variances are comparable across
measures. On average, participants scored lower on the COR than on the
other assessments. However, the COR also had the most variance.
Table 2 presents the bivariate correlations between the subscales. The
largest absolute correlation was between CTA and CTS, (r =0.58, p <
.001), followed by the correlation between CTS and COR (r =0.31, p <
.001).
Fig. 2 presents the GGM of cognition using EBIC regularization. The
nodes are colored based upon the classication of variables as belonging
to one of the three cognitive levels (i.e., colors do not represent clusters).
Darker, more visible edges indicate stronger relationships (red indicates
a negative relationship). The sparsity of the network as indicated by the
number of non-zero edges was 25/45. Via visual inspection, it can be
viewed that CTA and CTS are closely associated with both of the
epistemic cognition measures. Conrmation bias correlated very weakly
with CTA, moderately with COR, and negatively and moderately with
SRP. Miscalibration scores in one domain are associated with mis-
calibration in another; however, miscalibration of one’s abilities is
related to higher scores on the COR and PT, respectively. All indications
of epistemic cognition were positively correlated with one another.
Interestingly, SRP was negatively related to nearly every performance
assessment.
Fig. 3 reports the direct (strength) and indirect (closeness and
betweenness) centrality indices of each component. The node with the
highest strength centrality is the CTS score, meaning it has the greatest
sum of absolute partial correlations; it is followed by the CTA dimension.
COR displays the highest closeness and betweenness, indicating its sig-
nicance in transporting information from one node to another (Deserno
et al., 2022).
5. Discussion
This study utilized a psychometric network approach to explore the
conditional dependencies between measures of complex cognition. The
results showed that evaluating epistemically suspect information and
the ability to take someone else’s perspective are closely related to each
other, as well as with critical thinking. A plausible conjecture is that
advanced forms of reasoning in ill-structured problems (epistemic
cognition) may depend upon having a foundation in advanced inference
and deductive reasoning (Dwyer et al., 2014). Conrmation bias stood
2
The justication for NFC is also applied to the Preference for Effortful
Thinking (PET) used in study 2, which is another validated short form of the
longer NFC scale.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
7
apart from the other cognitive measures, having low direct and indirect
strength in the network. Interestingly, being miscalibrated on COR and
PT is positively associated with COR and PT, respectively. These mis-
calibration nodes may be tracking a form of intellectual humility, such
that people who possess high(er) civic online reasoning and perspective-
taking emphasize their limitations to a greater extent than those who are
low(er) performers (e.g., an indication of a Dunning-Kruger effect
[Dunning, 2011]).
A possible limitation of this study is the limited number of variables
available for analysis. More variables related to cognitive biases and
explicit epistemic cognition constructs would greatly strengthen the
interpretation of the network; both issues are addressed in study 2.
6. Study 2: cognition networks using heuristic-and-bias tasks
In study 1, 10 nodes in total were entered in the analysis. In study 2,
many more nodes (28) are entered in the network. Moreover, scores
from explicit epistemic cognition instruments are used in the analysis.
Aside from using a different sample and different measures, the design
and aim of study 2 is akin to study 1. Therefore, the aim of study 2 is to
examine the interrelationships between different thinking and reasoning
components that represent key aspects of emerging adult cognitive
processing using a psychological network approach. To accomplish this,
we instantiate concepts under each domain with various scales and as-
sessments. Additionally, stability tests of the results using bootstrapping
can be found in the supplementary material, alongside items from each
Table 2
Correlations.
123456789
1. CTA
2. CTS 0.58***
3. CB 0.09** 0.09**
4. CCT −0.03 −0.06*0
5. CPT 0.07*0.03 0.01 0.14***
6. CCOR 0.06*0.05 −0.02 0.18*** 0.10***
7. SRP −0.16*** −0.17*** −0.10*** −0.03 −0.11*** −0.01
8. COR 0.28*** 0.31*** 0.12*** −0.03 0.03 0.23*** −0.08**
9. PT 0.27*** 0.23*** 0.04 −0.05 0.13*** 0−0.02 0.23***
10. NFC 0.08** 0.13*** 0.04 −0.05 0.04 0.04 0.28*** 0.14*** 0.12***
Note. CB =conrmation bias; CTA =analytic: heighten critical thinking; CTS =synthetic: heighten critical thinking; CCT =calibration for CT total; CPT =calibration
for PT; CCOR =calibration for COR total; PT =perspective-taking; COR =civic online reasoning; SRP =self-regulation and planning; NFC =need for cognition.
*
p <.05.
**
p <.01.
***
p <.001.
Fig. 2. Gaussian graphical model using EBICglasso regularization (coloring based on a priori classication) for network specied in study 1. CB =conrmation bias;
CTA =analytic: heighten critical thinking; CTS =synthetic: heighten critical thinking; CCT =calibration for CT total; CPT =calibration for PT; CCOR =calibration
for COR total; PT =perspective-taking; COR =civic online reasoning; SRP =self-regulation and planning; NFC =need for cognition.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
8
scale. Because of the large number of nodes, we also present two addi-
tional networks aggregating the variables more broadly for robustness
and interpretability.
3
6.1. Methods
6.1.1. Participants
After obtaining approval from the university’s institutional review
board, participants were recruited from chemistry, engineering, and
philosophy courses at the same large public research university, located
in southern California, as study 1. To boost participation, instructors
were asked via email (through convenience) if they were willing to
provide extra credit to their students who chose to participate. In-
structors who agreed to provide extra credit to participants who chose to
complete the study advertised a link to the study information sheet via
email. The consent form relayed that no personal identifying informa-
tion would be made available, and that the aim of the study was to
understand intellectual development and capacities among college
learners. Once students signed the consent form, they were redirected to
a survey that contained cognitive, metacognitive, and epistemic
cognitive items. Of those who consented to participate (N =364), 91 %
had full data (n =332). Approximately 42 % of the sample was female;
43 % rst-generation; 28 % underrepresented minority students (His-
panic, Black, and Native American), and 29 % low-income students as
indicated in the in the university records. Compared to the population of
the university, similar proportions of underrepresented minority and
rst-generation students were observed. There was, however, an un-
derrepresentation of females in the sample, as well as low-income
students.
6.1.2. Measures
A total of 28 components were included in the GGM. Because mea-
sures are not positioned on the same scale, all variables were min/max
normalized to have a minimum of 0 and a maximum of 1. This kind of
normalization enables interpretable and comparable means. The mea-
sures used in this study can be viewed in Table 3. Each measure is at the
total scale or subscale level; some (sub)scales are indicated by single
items. Where appropriate, the internal consistency estimates (e.g.,
Cronbach’s
α
) are reported. All individual items in this study are shown
in the supplemental material.
6.1.2.1. Cognition measures. As indications of the cognitive level, a va-
riety of reective thinking tasks assessing fallacies, intuitive responding,
probabilistic reasoning, and syllogistic reasoning are used (Erceg et al.,
Fig. 3. Centrality indices for network displayed in Fig. 2 for study 1. CB =conrmation bias; CTA =analytic: heighten critical thinking; CTS =synthetic: heighten
critical thinking; CCT =calibration for CT total; CPT =calibration for PT; CCOR =calibration for COR Total; PT =perspective-taking; COR =civic online reasoning;
SRP =self-regulation and planning; NFC =need for cognition.
3
We would like to thank Reviewer #1 for suggesting that we perform these
additional networks.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
9
2020; Stanovich, 2016). Two tasks were used to measure base-rate
neglect, including the famous hospital and Helen problem (e.g.,
“Among the 1000 people that participated in the study, there were 50 16-
year-olds and 950 50-year-olds. Helen is randomly chosen participant in
this research. Helen listens to hip hop and rap music. She likes to wear tight T-
shirts and jeans. She loves to dance and has a small nose piercing. Which is
more likely? a. Helen is 16 years old; b. Helen is 50 years old). Intuitive
responding was measured using the three-item cognitive reection test
(CRT; Frederick, 2005). An example item is, “A bat and a ball cost $1.10
in total. The bat costs $1.00 more than the ball. How much does the ball
cost?” Belief bias syllogisms were measured with 8 correct/incorrect
statements regarding the deductive validity of statements. An example
item is: “Premise 1: All things that are smoked are good for the health.
Premise 2: Cigarettes are smoked. Conclusion: Cigarettes are good for the
health” (respondents answer True or False if the conclusion logically
follows the premises). Logical reasoning/conrmation bias was
measured with two Wason card tasks (see supplemental material, as
items use visuals). And probabilistic reasoning (solving problems
involving probabilistic knowledge) was measured with 6 correct/
incorrect items; 5 were taken from the Probabilistic Reasoning Scale
(Primi et al., 2017); 1 item was a Bayesian probability question
(example: “A village has 1000 inhabitants. 600 people own a pet, and
amongst pet owners 1 in 3 own more than 1 pet. If we select one person from
this village at random, what is the probability that they own more than one
pet?”). Before normalization, all items were scored correct (1) or
incorrect (0) except the causal base-rate neglect question (involving the
test-taker to rate the likelihood of choosing a textbook based upon one
trusted expert or many informed strangers), which was positioned on a
4-point scale. All items are shown in the supplemental material.
6.1.2.2. Metacognition measures. As indications of the metacognitive
level, we utilized a variety of self-report questionnaires. To measure the
two staples of metacognition, Knowledge of Cognition and Regulation of
Cognition, the validated dimensions from the Metacognitive Awareness
Inventory scale (Harrison & Vallin, 2018) was employed. An example
item is: “I think about what I really need to learn before I begin a task.” To
measure metacognition as a disposition, the short conscientiousness
scale (Soto & John, 2017) was administered, with the subscales Orga-
nization, Productiveness, and Organization (example item: “I am some-
one who: Is persistent, works until the task is nished”). Additionally,
participants completed the Baars et al. (2015) adapted executive func-
tioning scale with sub-dimensions: Attention, Planning, and Self-control
and self-monitoring. A reverse-scored example item is: “I am not able to
focus on the same topic for a long period of time.” Finally, we administered
a four-item scale regarding the classical notion of metacognition as
“thinking about thinking.” All items are shown in the supplemental
material.
6.1.2.3. Epistemic cognition measures. As indications of the epistemic
cognitive level, three general instruments were employed. First, we
leveraged ˙
Zyluk et al.’s (2022) Standardized Epistemological Under-
standing Assessment-Abstract (SEUA), which is a measure of epistemo-
logical understanding in ve domains: Physical Science, Social Science,
Morality, Personal Taste, and Esthetics. The SEUA is a validated adap-
tation of the instrument based upon Kuhn et al.’s (2000) model of
epistemological understanding whereby higher scores reect a sophis-
ticated orientation towards knowledge justication: understanding
knowledge as generated by human minds, as remaining uncertain, yet
available for evaluation via critical thinking (i.e., an Evaluativist
perspective). For each domain, participants were presented with two
characters, and asked whether both have some “rightness” or if only one
view can be right. (Example item: Robin believes that the well-being of a
certain part of society is more important than the well-being of others. Chris
believes that the well-being of another part of society is more important than
the well-being of others). If participants indicated that both could have
some “rightness”, they were given a score of 2 (1 otherwise); for those
given a score of 2, a second question was posed: Do you think one view
could be better or more right than the other? If participants selected that
one could be more right, they were given a score of 3 (otherwise they
remained at a score of 2). Five items were summed within each of the
ve domains, with higher scores indicating greater levels of epistemo-
logical understanding by possessing an Evaluativist perspective (Kuhn
et al., 2000; ˙
Zyluk et al., 2022). All items are shown in the supplemental
material.
Second, we leveraged the Postformal Thought Questionnaire, which is
based on Sinnott’s (1998) theory of postformal thinking, dened as “a
more complex logic” that goes beyond formal operations and involves
acknowledging and navigating the subjectivity accompanying ill-
structured problems and situations. The Postformal Though Question-
naire (PFT) has three subscales: Multiple Elements, Subjective Choice,
and Underlying Complexities (Cartwright et al., 2009). The PFT has
been psychometrically validated and successfully utilized in subsequent
studies predicting metacognition (Contreras et al., 2023; Sinnott et al.,
2020). An example item is: “I see that a given dilemma always has several
good solutions.” All items are shown in the supplemental material.
As mentioned above, epistemic virtues are relevant to epistemic
performance, and constitute a key set of constructs related to reliable
Table 3
Descriptive statistics of cognitive measures.
M SD Reliability
(
α
| Spearman-
Brown)
Cognition
Probabilistic reasoning (PR) 0.683 0.230 0.67 |
Base-rate neglect (BRN) 0.389 0.358 .
Causal base-rate neglect: McGraw problem
(CBR)
0.592 0.244 .
Probabilistic fallacy (FAL) 0.625 0.278 .
Belief-bias syllogism (BBS) 0.534 0.299 0.77 |
Wason card task (WCT) 0.140 0.314 .
Cognitive reection test (CRT) 0.652 0.357 0.61 |
Metacognition
Conscientiousness: organization (CNO) 0.617 0.237 .
Conscientiousness: productivity (CNP) 0.563 0.211 .
Conscientiousness: responsibility (CNR) 0.569 0.221 .
Metacognition awareness inventory:
knowledge of cognition (MKC)
0.623 0.153 0.79 |
Metacognition awareness inventory:
regulation of cognition (MRC)
0.595 0.153 0.81 |
Metacognition general (MG) 0.659 0.208 0.83 |
Executive function: attention (EFA) 0.422 0.301 0.82 |
Executive function: planning (EFP) 0.617 0.256 0.62 |
Executive function: self-control and self-
monitoring (EFS)
0.661 0.275 0.75 |
Epistemic cognition
Postformal thought: underlying
complexities (PUC)
0.744 0.203 0.83 |
Postformal thought: subjective choice
(PSC)
0.664 0.191 0.61|
Postformal thought: multiple elements
(PME)
0.537 0.19 0.61 |
Epistemological understanding: esthetics
(EUE)
0.667 0.206 0.74 |
Epistemological understanding: morality
(EUM)
0.693 0.224 0.76 |
Epistemological understanding: personal
taste (EUT)
0.626 0.219 0.76 |
Epistemological understanding: physical
science (EUP)
0.573 0.325 0.87 |
Epistemological understanding: social
science (EUS)
0.748 0.226 0.78 |
Actively open-minded thinking (AOT) 0.522 0.19 0.84 |
Close-minded thinking (CMT) 0.680 0.207 0.83 |
Preference for intuitive thinking (PIT) 0.435 0.175 0.89 |
Preference for effortful thinking (PET) 0.665 0.211 0.90 |
Note. Reliability with “.” =not applicable. The low reliability for CNP and CNR
could be attributed to the fact that these subscales only involve 2 items.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
10
processes of knowing (Chinn et al., 2021, 2011). In this study, we uti-
lized the 4-Component Thinking Styles Questionnaire (4-CTSQ) recently
curated by Newton et al. (2024). The 4-CTSQ subscales include: Actively
Open-minded Thinking (AOT), Close-Minded Thinking (CMT), Prefer-
ence for Intuitive Thinking (PIT), and Preference for Effortful Thinking
(PET). Each subscale has 6 items. Items were originally positioned such
that higher scores reect a less virtuous disposition; thus, values were
reversed (e.g., higher scores on PIT indicate less preference for intuitive
thinking) to easily interpret connections with other nodes. An example
item is: “Even if there is concrete evidence against what you believe to be true,
it is OK to maintain cherished beliefs.” All items are shown in the sup-
plemental material.
6.1.3. Data analysis
The same GGM model used in study 1 is applied in study 2. Regu-
larization using the GLASSO and EBIC are again employed. The hyper-
tuningparameter, γ, was set to 0.5 for study 2. Thus, the approach and
general aim is identical to study 1.
6.2. Results
Table 3 showcases the descriptive statistics for the variables in the
analysis after normalizing each variable to have a minimum of 0 and a
maximum of 1. Thus the means and standard deviations are comparable
across measures. In general, the variable with the greatest level of
endorsement was Epistemological Understanding: Social Science (EUS).
The most difcult/least endorsed index was the Wason card task (WCT).
The Base-Rate Neglect index had the greatest amount of variance.
Table 4 presents the bivariate correlations between the cognition vari-
ables. The largest absolute correlation was between EUE (Esthetics) and
EUT (Personal Taste), (r =0.75, p <.001), which makes sense as the two
subscales touch on similar domains. This was followed by the correlation
between EUT and EUM (Morality; r =0.68, p <.001).
Fig. 4 presents the GGM of adult cognition using EBIC regularization.
The nodes are colored based upon the classication of variables as
belonging to one of the three cognitive levels (i.e., colors do not repre-
sent clusters). Darker, more visible edges indicate stronger relationships
(red indicates a negative relationship). The sparsity of the network as
indicated by the number of non-zero edges was 91/378. Via visual in-
spection, it can be viewed that, generally, nodes conceptually belonging
to their respective cognitive level are more aligned empirically with
each other. The most notable exception is between measures of episte-
mological understanding and postformal thinking (epistemic cognition
level), with postformal thinking more associated with metacognitive
nodes and epistemological understanding more associated with cogni-
tive nodes, and only a weak link between PUC and EUS. Moreover, the
network contains mostly positive edges (positive partial correlations),
with notable exceptions between two metacognition variables (EFP and
CNR) and WCT, and between two postformal thinking (PSC and PME)
measures and PIT.
Fig. 5 reports the direct (strength) and indirect (closeness and
betweenness) centrality indices of each component. The node with the
highest strength centrality is the EUM score, meaning it has the greatest
sum of absolute partial correlations; it is followed by the MRC dimen-
sion. PET, BBS, and CRT have the highest closeness. PET and BBS also
have the highest betweenness, indicating they are central in transporting
information from one node to another (Deserno et al., 2022).
6.2.1. Robustness checks: additional forms of aggregation
Researchers can generate variables at various levels of aggregation
(item-level, subscale level, total score level, etc.). Our emphasis in the
networks has been at the subscale level, to strike a balance between
meaningfulness and interpretability, while prioritizing the breadth of
constructs covered. However, as this study still contains many
nodes—though within the general limits for the sample size (Constantin,
2018)—we also examine how the results might change when aggre-
gating and/or dropping the variables in different ways to achieve a
much sparser, mor palatable depiction. Thus, we (1) Combine all SEUA
subscales; (2) Combine all epistemic disposition scales, (3) Combine
some of the cognition measures in ways that make sense (those that
employ heuristics and biases—leaving out strictly probabilistic
reasoning and/or Wason card task), (4) Exclude Postformal Thought
(while historically epistemic cognition, it is not as frequently employed
as other scales), and, nally, we (5) Exclude the conscientiousness
Table 4
Correlations.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
11
subscales (which is arguably overly emphasizing behavioral control as
opposed to directed cognition).
In Fig. 6, two additional networks are generated after the above-
mentioned changes. On the left, three nodes per cognitive level were
entered, whereas on the right, only two were entered (whereby Post-
formal Thought was excluded). Examining both networks and the
accompanying centrality indices, we see remarkable overlap between
both additional networks and the main network presented in Fig. 4: (a)
Epistemic virtue measures (EV) and rationality/reective thinking (RT)
are central to the network, serving as key carrier nodes, (b) Epistemic
understanding/beliefs (EU) are moderately-to-strongly related to dis-
positions and to rationality/reective thinking measures, (c) Postformal
Thought (PFT) is more closely linked to metacognition than epistemic
cognition, with a weak link to probabilistic reasoning (PR), and (d)
similar correlations are found for metacognitive nodes as the original
study 2 network.
6.3. Discussion
This study utilized a psychometric network approach to explore the
conditional dependencies between complex cognition. In general, each
node corresponded with its conceptual level of cognitive processing.
Akin to Grimm and Richter’s (2024) ndings, the results of the ratio-
nality tasks displayed high levels of inter-correlations, suggesting a
general latent factor of rationality. A preference for effortful thinking
(PET), belief bias syllogisms (BBS), and cognitive reection (CRT)
appear to be highly important variables in the network of adult thinking,
lying in the shortest path between other nodes in the network. This may
be unsurprising, as most of these variables are multi-faceted constructs
with variance attributed to individual differences in deliberation and
self-monitoring, numeracy, and preferences for complex thinking
(Campitelli & Gerrans, 2014). Another one of the more notable ndings
is that an evaluativist perspective towards morality (EUM) and social
science (EUS) had direct positive associations with several cognitive
nodes, including CRT and PR. This suggests that, at least in part, pos-
sessing a deliberate, non-intuitive thinking style, alongside probabilistic
reasoning skills, may assist in understanding the complexities with
multi-faceted problems. Also, the ability to suspend one’s beliefs and
evaluate the validity of argument structure (BBS) was directly linked to
EUM, suggesting a link between sidelining one’s appraisal of plausibility
and possessing a dynamic perspective on morality. The results showed
that executive function and conscientiousness are highly correlated with
one another, suggesting substantial overlap between the two constructs.
Finally, the general pattern of results did not change in additional net-
works whereby we aggregated various subscales, and excluded others (e.
g., conscientiousness).
A possible limitation of this study is that, aside from probabilistic
reasoning, all other cognitive nodes required reective/rational
thinking; that is, the study could be improved by introducing non-
heuristic-and-bias critical thinking variables. Additionally, while the
sample size is adequate—especially after condensing the variables in the
additional analyses—and the estimates stable (see supplementary
Fig. 4. Gaussian graphical model using EBICglasso regularization (coloring based on a priori classication) for network specied in study 2. PR =probabilistic
reasoning; BRN =base-rate neglect; CBR =causal base-rate neglect: McGraw problem; FAL =probabilistic fallacy; BBS =belief-bias syllogism; WCT =Wason card
task; CRT =cognitive reection test; CNO =conscientiousness: organization; CNP =conscientiousness: productivity; CNR =conscientiousness: responsibility; MKC
=metacognition awareness inventory: knowledge of cognition; MRC =metacognition awareness inventory: regulation of cognition; MG =metacognition general;
EFA =executive function: attention; EFP =executive function: planning; EFS =executive function: self-control and self-monitoring; PUC =postformal thought:
underlying complexities; PSC =postformal thought: subjective choice; PME =postformal thought: multiple elements; EUE =epistemological understanding: es-
thetics; EUM =epistemological understanding: morality; EUT =epistemological understanding: personal taste; EUP =epistemological understanding: physical
science; EUS =epistemological understanding: social science; AOT =actively open minded thinking; CMT =close-minded thinking; PIT =preference for intuitive
thinking; PET =preference for effortful thinking.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
12
material), ideally many more observations would strengthen the
generalizability of the network.
7. General discussion
This research offers rich insight into the interrelationships between
adult thinking components at various levels of cognitive processing
across two studies. While this research is exploratory, the ndings relate
back to Kuhn’s (2022) thinking and reasoning framework, which posits
a diverse array of knowledge strategies and meta-level competencies
and dispositions as central to the complex reasoning process.
In general, across studies 1 and 2, the cognitive and the epistemic
cognitive levels appear to be clearly linked to one another, but not so
related that they overlap. The role of metacognition across the two
studies showed some consistency. In study 1, miscalibration of one
ability appeared to be related to miscalibration on another; however,
these absolute miscalibration scores were positively related with per-
formance (e.g., a possible indication of Dunning-Kruger effect [Dunning,
2011]). The self-monitoring aspect of executive function (EFS) in study 2
was consistent with this nding as it showed positive partial correlations
with probabilistic reasoning and an evaluativist perspective on social
science. However, it’s unclear why the responsibility and planning as-
pects in both studies (study 1: SRP; study 2: EFP) would negatively
correlate with the objective cognitive measures (study 1: CTA and CTS;
study 2: WCT). A mere speculation is that people who are responsible
planners may be lower in cognitive exibility, which is involved in
successfully responding to Wason task(s).
Another key nding across the two studies was that epistemic vir-
tues, or thinking dispositions, were fairly central to both networks. They
were also related to cognitive and epistemic cognitive tasks, suggesting
that epistemic virtues play a role in navigating advanced forms of ill-
structured dilemmas involving real-world complexities. Moreover, we
see that such epistemic virtues/dispositions are not highly correlated
with metacognitive dispositions (study 2). In fact, only a limited number
of weak connections existed between the two sets of dispositions. This
nding ts well with an emerging research base that distinguishes these
two concepts at the state level. That is, theoretical and empirical evi-
dence is disentangling metacognitive feelings (feelings that arise during
thinking activities as one reects on their cognitive processing, such as
feelings of familiarity or difculty) from epistemic emotions (emotions
arising in the face of new or conicting knowledge claims, such as sur-
prise and curiosity; Nerantzaki et al., 2021; Puente-Díaz et al., 2021;
Fig. 5. Centrality indices for network displayed in Fig. 4 for study 2. PR =probabilistic reasoning; BRN =base-rate neglect; CBR =causal base-rate neglect: McGraw
problem; FAL =probabilistic fallacy; BBS =belief-bias syllogism; WCT =Wason card task; CRT =cognitive reection test; CNO =conscientiousness: organization;
CNP =conscientiousness: productivity; CNR =conscientiousness: responsibility; MKC =metacognition awareness inventory: knowledge of cognition; MRC =
metacognition awareness inventory: regulation of cognition; MG =metacognition general; EFA =executive function: attention; EFP =executive function: planning;
EFS =executive function: self-control and self-monitoring; PUC =postformal thought: underlying complexities; PSC =postformal thought: subjective choice; PME =
postformal thought: multiple elements; EUE =epistemological understanding: esthetics; EUM =epistemological understanding: morality; EUT =epistemological
understanding: personal taste; EUP =epistemological understanding: physical science; EUS =epistemological understanding: social science; AOT =actively open-
minded thinking; CMT =close-minded thinking; PIT =preference for intuitive thinking; PET =preference for effortful thinking.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
13
Vogl et al., 2021). In the network models examined here, we see that this
distinction holds for trait-level aspects of these concepts.
Another salient notion surfacing from these studies is how the results
relate back to different theoretical perspectives of epistemic cognition.
While the data/method do not lend themselves to a test of different
theories—nor was that an aim of the studies—we do see support for a
broad view of epistemic thinking in both networks. In study 1, reliable
processes embedded in assessments such as considering and mapping
both perspectives of individuals in a story (PT assessment) and evalu-
ating source trustworthiness of online content (COR assessment) have
close ties with, not only more classical CT assessments (CTS and CTA),
but also dispositional attributes signifying the enjoyment of complex
thinking (NFC). Such a nding is more consistent with a holistic view of
epistemic thinking that considers the aims, motivations, and criteria
individuals pursue as they are faced with knowledge tasks, such as in the
AIR model (Chinn & Rinehart, 2016). In study 2, similarly, we see that
epistemic beliefs (e.g., beliefs about the nature of knowledge) are related
to reective thinking, but also to thinking dispositions, which were
shown to be important carrier nodes of information in the network. The
conditional dependencies between beliefs, character, and reliable pro-
cesses of knowing can be viewed as consistent with the AIR model and
Kuhn’s (2001) map of complex thinking, pointing towards a systems
perspective where the dynamics of human development are captured as
a complex web across relevant features of cognition (Fischer et al., 2003;
Fischer & Bidell, 2006; Fischer & Pruyne, 2003; Yan & Fischer, 2002). Of
course, cross-sectional research of the likes of the present set of studies is
ultimately inadequate to fully adjudicate between theories or
perspectives.
In addition to similarities, we also observed discrepancies between
studies 1 and 2. In study 1, conrmation bias—which is in line with
reective thinking/rational choice behavior—was minimally associated
with epistemic cognition measures. However, in study 2, the reective
thinking/rational choice behavior measures (blue) were, generally,
clearly linked to epistemic cognition. This is a discrepancy but could be a
feature of the specic type of reasoning task: for instance, among the
weaker nodes in study 2 was the Wason card task, which is a form of
logical reasoning inuenced by conrmation bias (though there is some
debate on this, e.g., Evans, 2016). It is unclear how the inclusion of more
rationality tasks in study 1’s network would change this relation; simi-
larly, it’s unclear how the addition of traditional critical thinking mea-
sures to study 2 might alter the structure of the networks. Ultimately,
both types of measures should be included in one network to clarify this.
Second, epistemic cognition appears to be more intertwined with the
other forms of reasoning in study 1 than in study 2. For example,
epistemic belief measures in study 2 stood apart from metacognitive
measures, being connected mostly through dispositional attributes,
whereas in study 1 the non-dispositional epistemic cognitive measures
(PT and COR) contained direct links to metacognition. Again, to settle
these discrepancies requires that both types of instantiations are
included in one network.
7.1. Implications
The present research offers several practical implications for learning
and instruction. First, the components incorporated in the networks
emphasize teachable habits of mind and critical thinking skills. For
instance, the current emphasis on cognitive components such as
perspective-taking, rating the trustworthiness of information, intuitive
responding, probabilistic thinking, conrmation bias, and self-
regulatory abilities is in harmony with scholarship emphasizing
malleable attributes (e.g., Bråten, 2016; Erceg et al., 2023; Wrzus &
Roberts, 2017).
Second, since CRT is an important carrier node, and since recent
work has highlighted its developmental signicance for other reasoning
abilities (Shtulman & Young, 2023), instruction targeting intuitive
responding may provide support for a host of thinking skills. Work on
this front is underway, as illustrated by Orona and Trautwein (2024),
who utilized an experimental protocol to evaluate the effectiveness of an
Fig. 6. Gaussian graphical model using EBICglasso regularization (coloring based on a priori classication) and centrality indices for additional networks specied in
study 2 without conscientiousness and greater aggregation of nodes. On the left, three nodes per cognitive level were entered, whereas on the right, two were entered.
Different aggregations were used. RT =rationality/reective thinking; PR =probabilistic reasoning; WCT =Wason card task; MRC =metacognition awareness
inventory: regulation of cognition; MC =metacognition general and knowledge of cognition; EXF =executive function: overall; PFT =postformal thought: overall;
EU =epistemological understanding: overall; EV =epistemic virtue: overall; MET =metacognition: overall.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
14
epistemic virtue intervention on CRT, nding robust growth for partic-
ipants in the treatment condition. Simonovic et al. (2023) also con-
ducted a critical thinking intervention that boosted CRT from pretest-to-
posttest.
Third, and related to the second point, it supports calls for nuanced
instructional design and practice geared towards apt epistemic perfor-
mance (Chinn et al., 2021), given that seemingly related concepts
(critical thinking and civic online reasoning) share a meaningful rela-
tionship but are non-overlapping. Educators thus have key areas that can
be targeted based on the results of the present studies: epistemological
beliefs, probabilistic reasoning, and epistemic dispositions/virtues. For
example, in study 2, an evaluativist orientation towards morality was
clearly linked to bracketing prior beliefs and successfully evaluating
argument structure (BBS). The moderate relationship suggests the sig-
nicance of prior beliefs in the development of epistemic thinking, as
has been revealed in previous research (Kardash & Scholes, 1996). The
ndings also suggest that instruction aimed at enhancing thinking dis-
positions may contribute to success in a variety of epistemic tasks;
therefore, it may be worthwhile to investigate instructional in-
terventions developing epistemic virtue (Orona et al., 2024; Tarchi,
2024). Thus, unique and creative forms of instruction to improve
thinking and reasoning from multiple angles could be constructed by
targeting resistance to cognitive biases, character, and epistemic beliefs.
And nally, while the relationships presented are not causal, they
offer a generative set of nuanced observations that can be subsequently
tested in future studies (see Greene, 2022 for the theory-building cycle).
For instance, solid foundations in argument structure can enhance an
individual’s capacity for sifting through epistemically suspect informa-
tion sources and understanding another person’s perspective (links
found in study 1), and the observed link that probabilistic reasoning
predicts a judgement-based approach to understanding social science
(link found in study 2) represent propositions that can be explicitly
tested in laboratory and eld settings (Orona, 2021b). Furthermore, by
examining the information in the networks, one might longitudinally
test which clusters of cognitive skills co-occur as a dynamic develop-
mental trajectory (Fischer et al., 2003; Fischer & Pruyne, 2003).
7.2. Limitations and future directions
Despite diverse (across ethnicity, sex, and academic status) samples
with a multi-study design, which buffers against transient patterns in
data, there are several limitations of the present research. First, while a
large inventory of cognitive components was implemented (especially in
study 2), there are important extant measures across all three domains of
cognition that were not examined. At the cognitive level, these studies
could be enhanced by incorporating more cognitive biases (e.g., hind-
sight bias, myside bias, etc.), as well as measures of logical reasoning (e.
g., Hao et al., 2023) together in the same network. Incorporating mea-
sures of the Dunning-Kruger Effect (Dunning, 2011) for assessments not
included in the network would enhance the measures of metacognition.
And at the epistemic cognition level, expanded facets from multidi-
mensional frameworks, such as epistemic emotions, epistemic beliefs,
and epistemic metacognition, would provide a more detailed and ho-
listic depiction of cognitive processing (Danielson et al., 2023; Chevrier
et al., 2020).
Second, similar to using more variables, incorporating measures used
in study 1 and study 2 together in the same network would provide a
fuller picture of emerging adult complex cognitive processing. Third,
these studies utilized undergraduate, emerging adult samples. While this
is not uncommon and may be even more advantageous from an educa-
tional perspective, future research could extend this to middle-age and
older adults, and/or examine within-person networks over the course of
the college years, circumventing the limitations of cross-sectional
research.
Fourth, it’s important to note that the current set of analyses were
conducted at the subscale level (except when subscales were not
available). Variables could be entered in a variety of ways, and this
researcher degree of freedom should be acknowledged. That is, total
scores or items could have been used. The choice for using subscales was
a balance between theoretical and practical concerns. For example,
making claims about associations between individual items may not be
very elucidating, especially when the number of items is large (as in the
case of study 2). Alternatively, using total scores might obscure mean-
ingful distinctions associated with facets of a larger construct. While in
study 2 we examined alternate node aggregation, as well as excluded
those that are traditionally non-cognitive, such as conscientiousness
(Fagioli et al., 2020), and found strong similarity between the networks,
larger samples with a larger inventory of cognitive measures could
explore these issues in more detail.
Fifth, the granularity of our understanding could be greatly
enhanced by considering cognitive processes within individual tasks,
rather than across them, including the cognitive load involved with the
problem-solving activities introduced here (Sweller, 1988). Under-
standing what various tasks demand of individuals’ working memory as
opposed to long-term schemas would provide richer insight into the
determinants of successful problem-solving (Sweller et al., 1998). Along
these lines, a rich task analysis would assist in the specic cognitive
operations individuals must control, as opposed to standings on global
constructs (Fischer, 1980). For example, focusing on incidental exposure
to COVID-19 information, Greene et al. (2024) coded cognitive, meta-
cognitive, and epistemic cognitive mental processes via think-aloud
protocols. A network model to such data could provide a much more
ne-grained understanding of the dynamics of interacting components
in specic situations than our broad-based approach.
Finally, greater generalizability can also be secured. While across
both studies the proportion of ethnic minorities and other demographic
groups followed closely to the university population, in study 1 males
were underrepresented and in study 2 females were underrepresented.
This could be because in sample 1, whereby the study was advertised to
a wider set of university undergraduates, more females participated as
emerging evidence shows women tend to participate more in research
studies (Nuzzo & Deaner, 2023). Whereas in study 2, students were
elicited from various classroom contexts, several of which were engi-
neering courses which tend to be traditionally male dominant. Inter-
estingly, however, the under/overrepresentation of males across the two
studies did not result in widely different correlations between the nodes
at the domain level—that is, similar patterns were observed for cogni-
tive, metacognitive, and epistemic cognitive components across the two
studies, as noted above. Where there were differences, it’s worth
exploring if these were due to differences in sample proportions or in the
selection of components themselves.
8. Conclusion
Adult cognition encompasses a wide variety of intellectual opera-
tions which interact with each other across a plethora of contexts and
settings. The current set of studies is–to the authors’ knowledge–the rst
attempt to explore the relationships between and dynamics of a rela-
tively large set of complex cognitive reasoning components simulta-
neously. The innovative aspects of this investigation are the explicit
incorporation of concepts corresponding to key domains of emerging
adult cognition analyzed as a network system across two separate
samples, informed by theoretical traditions of development and per-
formance. Along these lines, we moved beyond narrow emphasis on any
one test, survey, or construct, opting instead for a perspective that is
representative of the multifarious range of dispositions, skills and abil-
ities relevant to mature cognitive processing. In general, though with
some exceptions, cognitive, metacognitive, and epistemic cognitive
components are intertwined yet largely distinct forms of complex
cognition, with teachable habits and knowledge mechanisms linking
their interconnectedness. Hopefully, the results of this exploratory
research will serve as a basis for more intensive investigations into the
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
15
dynamic system of complex reasoning among college-going students.
Ethics approval
University of California, Irvine Internal Review Board approved data
collection for both studies presented in this manuscript.
Declaration competing interest
There are no nancial or personal competing interests to report.
Funding
John Templeton Foundation: Embedding the Development of Intel-
lectual Character within a University Curriculum (#62330). This
research was also supported by the Postdoctoral Academy of Education
Sciences and Psychology of the Hector Research Institute of Education
Sciences and Psychology, Tübingen, funded by the Baden-Württemberg
Ministry of Science, Research, and the Arts. This article was funded by
the Open Access Publication Fund from the University of Tübingen.
CRediT authorship contribution statement
Gabe Avakian Orona: Writing – review & editing, Writing – original
draft, Visualization, Methodology, Formal analysis, Data curation,
Conceptualization. Jacquelynne S. Eccles: Supervision. Sabrina Sol-
anki: Resources, Data curation. David A. Copp: Resources, Data cura-
tion. Quoc-Viet Dang: Resources, Data curation. Richard Arum:
Supervision, Resources, Project administration.
Acknowledgements
The authors would like to thank Andres S. Bustamante for supporting
this project with data collection. The authors would also like to thank
Sarah Grace Orona for early feedback on the conceptual idea and pre-
sentation of the results.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.lindif.2024.102584.
Data availability
No.
References
Alexander, P. A. (2023). “Here be dragons!” Mapping the realm of higher-order, critical,
and critical-analytic thinking. Educational Psychology Review, 35(2), 42.
Arum, R., Eccles, J. S., Heckhausen, J., Orona, G. A., von Keyserlingk, L.,
Wegemer, C. M., et al. (2021a). A framework for measuring undergraduate learning
and growth. Change: The Magazine of Higher Learning, 53(6), 51–59.
Arum, R., Eccles, J. S., Heckhausen, J., Orona, G. A., von Keyserlingk, L.,
Wegemer, C. M., et al. (2021b). Ensuring a more equitable future: Assessing student
learning and growth in higher education. Postsecondary Value Commission.
Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college campuses.
University of Chicago Press.
Auxier, B., & Anderson, M. (2021). Social media use in 2021. Pew Research Center:
Internet, Science & Tech. https://www.pewresearch.org/internet/2021/04/07/s
ocial-media-use-in2021/.
Baars, M. A., Nije Bijvank, M., Tonnaer, G. H., & Jolles, J. (2015). Self-report measures of
executive functioning are a determinant of academic performance in rst-year
students at a university of applied sciences. Frontiers in Psychology, 6, 1131.
Baehr, J. (2015). The four dimensions of an intellectual virtue. In Slote, & E. Sosa (Eds.),
Moral and intellectual virtues in western and chinese philosophy (pp. 86–98). Routledge.
Chienkuo Mi Michael.
Barger, M. M., Perez, T., Canelas, D. A., & Linnenbrink-Garcia, L. (2018). Constructivism
and personal epistemology development in undergraduate chemistry students.
Learning and Individual Differences, 63, 89–101.
Barzilai, S., & Ka’adan, I. (2017). Learning to integrate divergent information sources:
The interplay of epistemic cognition and epistemic metacognition. Metacognition and
Learning, 12(2), 193–232.
Barzilai, S., & Zohar, A. (2014). Reconsidering personal epistemology as metacognition:
A multifaceted approach to the analysis of epistemic thinking. Educational
Psychologist, 49(1), 13–35.
Baytelman, A., Iordanou, K., & Constantinou, C. P. (2020). Epistemic beliefs and prior
knowledge as predictors of the construction of different types of arguments on
socioscientic issues. Journal of Research in Science Teaching, 57(8), 1199–1227.
Belenky, M. F., Clinchy, B. M., Goldberger, N. R., & Tarule, J. M. (1986). Women’s ways of
knowing: The development of self, voice, and mind (Vol. 15). New York: Basic Books.
Blanken, T. F., Isvoranu, A.-M., & Epskamp, S. (2022). Estimating network structures
using model selection. In Network psychometrics with r (pp. 111–132). Routledge.
Borsboom, D., Deserno, M. K., Rhemtulla, M., Epskamp, S., Fried, E. I., McNally, R. J.,
et al. (2021). Network analysis of multivariate data in psychological science. Nature
Reviews Methods Primers, 1(1), 58.
Bråten, I. (2016). Epistemic cognition interventions: Issues, challenges, and directions. In
J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp.
360–372).
Burger, J., Isvoranu, A.-M., Lunansky, G., Haslbeck, J., Epskamp, S., Hoekstra, R. H.,
et al. (2023). Reporting standards for psychological network analyses in cross-
sectional data. Psychological Methods, 28(4), 806.
Burgoyne, A. P., Mashburn, C. A., Tsukahara, J. S., Hambrick, D. Z., & Engle, R. W.
(2023). Understanding the relationship between rationality and intelligence: A
latent-variable approach. Thinking & Reasoning, 29(1), 1–42.
Butler, H. A., Pentoney, C., & Bong, M. P. (2017). Predicting real-world outcomes:
Critical thinking ability is a better predictor of life decisions than intelligence.
Thinking Skills and Creativity, 25, 38–46.
Campitelli, G., & Gerrans, P. (2014). Does the cognitive reection test measure cognitive
reection? A mathematical modeling approach. Memory & Cognition, 42, 434–447.
Cartwright, K. B., Galupo, M. P., Tyree, S. D., & Jennings, J. G. (2009). Reliability and
validity of the complex postformal thought questionnaire: Assessing adults’
cognitive development. Journal of Adult Development, 16, 183–189.
Chemers, M. M., Hu, L., & Garcia, B. F. (2001). Academic self-efcacy and rst year
college student performance and adjustment. Journal of Educational Psychology, 93
(1), 55.
Chen, J., & Chen, Z. (2008). Extended bayesian information criteria for model selection
with large model spaces. Biometrika, 95(3), 759–771.
Chevrier, M., Muis, K. R., & Di Leo, I. (2020). Calibration to task complexity: The role of
epistemic cognition. The Journal of Experimental Education, 88(1), 1–26.
Chinn, C. A., Barzilai, S., & Duncan, R. G. (2021). Education for a “post-truth” world:
New directions for research and practice. Educational Researcher, 50(1), 51–60.
Chinn, C. A., Buckland, L. A., & Samarapungavan, A. (2011). Expanding the dimensions
of epistemic cognition: Arguments from philosophy and psychology. Educational
Psychologist, 46(3), 141–167.
Chinn, C. A., & Rinehart, R. W. (2016). Epistemic cognition and philosophy: Developing
a new framework for epistemic cognition. In Handbook of epistemic cognition (pp.
460–478). Routledge.
Constantin, M. A.. Sample Size Recommendations for Estimating Cross-Sectional
Network Models. https://doi.org/10.17605/OSF.IO/6Z7TN.
Contreras, A., Su´
arez-Falc´
on, J. C., Caprara, M., Pozo, P., G´
omez-Veiga, I., & Cabras, E.
(2023). Psychometric properties of the spanish version of the complex postformal
thought questionnaire: Developmental pattern and signicance and its relationship
with cognitive and personality measures. Collabra: Psychology, 9(1), 67993.
Danielson, R. W., Sinatra, G. M., Trevors, G., Muis, K. R., Pekrun, R., & Heddy, B. C.
(2023). Can multiple texts prompt causal thinking? The role of epistemic emotions.
The Journal of Experimental Education, 91(4), 621–635. Chicago.
Deserno, M. K., Isvoranu, A.-M., Epskamp, S., & Blanken, T. F. (2022). Descriptive
analysis of network structures. In Network psychometrics with r (pp. 45–65).
Routledge.
Dunning, D. (2011). The Dunning–Kruger effect: On being ignorant of one’s own
ignorance. In , Vol. 44. Advances in experimental social psychology (pp. 247–296).
Academic Press.
Dwyer, C. P., Hogan, M. J., & Stewart, I. (2014). An integrated critical thinking
framework for the 21st century. Thinking Skills and Creativity, 12, 43–52.
Dyer, K. D., & Hall, R. E. (2019). Effect of critical thinking education on epistemically
unwarranted beliefs in college students. Research in Higher Education, 60(3),
293–314.
Epskamp, S., Borsboom, D., & Fried, E. I. (2018). Estimating psychological networks and
their accuracy: A tutorial paper. Behavior Research Methods, 50, 195–212.
Epskamp, S., Cramer, A. O., Waldorp, L. J., Schmittmann, V. D., & Borsboom, D. (2012).
Qgraph: Network visualizations of relationships in psychometric data. Journal of
Statistical Software, 48, 1–18.
Erceg, N., Andri´
c, L., Bosilj, L., Britvi´
c, F. D., ˇ
Ceko, A., Dedi´
c, M., et al. (2023). Teaching
actively open-minded thinking online: Encouraging effects of a serious computer game and
an online module. https://doi.org/10.31234/osf.io/hwzk7
Erceg, N., Gali´
c, Z., & Ruˇ
zojˇ
ci´
c, M. (2020). A reection on cognitive reection–testing
convergent/divergent validity of two measures of cognitive reection. Judgment and
Decision making, 15(5), 741–755.
Erwin, T. D. (1983). The scale of intellectual development: Measuring Perry’s scheme.
Journal of College Student Personnel, 24, 6–12.
Evans, J. S. B. (2016). Reasoning, biases and dual processes: The lasting impact of Wason
(1960). Quarterly Journal of Experimental Psychology, 69(10), 2076–2092.
Facione, P. (1990). Critical thinking: A statement of expert consensus for purposes of
educational assessment and instruction (the Delphi report).
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
16
Fagioli, L. P., Baker, R., & Orona, G. A. (2020). The role of non-cognitive variables in
identifying community college students in need of targeted supports. Research in
Higher Education, 61, 725–763.
Fago, G. C. (1995). A scale of cognitive development: Validating Perry’s scheme.
Fernandez-Duque, D., Baird, J. A., & Posner, M. I. (2000). Executive attention and
metacognitive regulation. Consciousness and Cognition, 9(2), 288–307.
Fischer, K. W. (1980). A theory of cognitive development: The control and construction
of hierarchies of skills. Psychological review, 87(6), 477.
Fischer, K. W., & Bidell, T. R. (2006). Dynamic development of action and thought. In ,
vol. 1. Handbook of child psychology (pp. 313–399).
Fischer, K. W., & Pruyne, E. (2003). Reective thinking in adulthood: Emergence,
development, and variation. In Handbook of adult development (pp. 169–198). Boston,
MA: Springer US.
Fischer, K. W., Yan, Z., & Stewart, J. (2003). Adult cognitive development: Dynamics in
the developmental web. In Handbook of developmental psychology (pp. 491–516).
Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The
Nature of Intelligence (pp. 231–235). Erlbaum: Hillsdale.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of
cognitive–developmental inquiry. American Psychologist, 34(10), 906.
Frazier, L. D., Schwartz, B. L., & Metcalfe, J. (2021). The MAPS model of self-regulation:
Integrating metacognition, agency, and possible selves. Metacognition and learning,
16, 297–318.
Frederick, S. (2005). Cognitive reection and decision making. Journal of Economic
Perspectives, 19(4), 25–42.
Friedman, J., Hastie, T., & Tibshirani, R. (2008). Sparse inverse covariance estimation
with the graphical lasso. Biostatistics, 9(3), 432–441.
Ganuthula, V. R. R., & Dyaram, L. (2016). Rationality and the reective mind: A case for
typical performance measure of cognitive ability. Learning and Individual Differences,
49, 216–223.
Greene, J. A. (2022). What can educational psychology learn from, and contribute to,
theory development scholarship? Educational Psychology Review, 34(4), 3011–3035.
Greene, J. A., Cartiff, B. M., & Duke, R. F. (2018). A meta-analytic review of the
relationship between epistemic cognition and academic achievement. Journal of
Educational Psychology, 110(8), 1084.
Greene, J. A., Sandoval, W. A., & Bråten, I. (2016). Handbook of epistemic cognition.
Routledge.
Greene, J. A., & Yu, S. B. (2016). Educating critical thinkers: The role of epistemic
cognition. Policy Insights From the Behavioral and Brain Sciences, 3(1), 45–53.
Greene, J. A., Hollander-Blackmon, C., Kirk, E. A., & Deekens, V. M. (2024). What are
they thinking? Exploring college students’ mental processing and decision making
about COVID-19 (mis) information on social media. Journal of Educational
Psychology, 116(1), 76–2024.
Grimm, J., & Richter, T. (2024). Rational thinking as a general cognitive ability: Factorial
structure, underlying cognitive processes, and relevance for university academic
success. Learning and Individual Differences, 111, Article 102428.
Halpern, D. F., & Dunn, D. S. (2021). Critical thinking: A model of intelligence for solving
real-world problems. Journal of Intelligence, 9(2), 22.
Hao, T., Wang, Z., Dai, S., & Ren, Y. (2023). Ego depletion hampers logical reasoning and
monitoring accuracy: An experimental study. The Journal of Experimental Education,
1–17.
Harrison, G. M., & Vallin, L. M. (2018). Evaluating the metacognitive awareness
inventory using empirical factor-structure evidence. Metacognition and Learning, 13,
15–38.
Healey, M. L., & Grossman, M. (2018). Cognitive and affective perspective-taking:
Evidence for shared and dissociable anatomical substrates. Frontiers in Neurology, 9,
491.
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories:
Beliefs about knowledge and knowing and their relation to learning. Review of
Educational Research, 67(1), 88–140.
Inzlicht, M., Werner, K. M., Briskin, J. L., & Roberts, B. W. (2021). Integrating models of
self-regulation. Annual Review of Psychology, 72, 319–345.
Iordanou, K., Muis, K. R., & Kendeou, P. (2019). Epistemic perspective and online
epistemic processing of evidence: Developmental and domain differences. The
Journal of Experimental Education, 87(4), 531–551.
Ivory, M., Towse, J., Sturdee, M., Levine, M., & Nuseibeh, B. (2023). Recognizing the
known unknowns; the interaction between reective thinking and optimism for
uncertainty among software developer’s security perceptions. Technology, Mind, and
Behavior, 4(3). https://doi.org/10.1037/tmb0000122
Jensen, L. L. (1998). The role of need for cognition in the development of reective judgment
(PhD thesis). ProQuest Information & Learning
Kallio, E. K. (2020). From multiperspective to contextual integrative thinking in
adulthood: Considerations on theorisation of adult thinking and its place as a
component of wisdom 1. In Development of adult thinking (pp. 9–32). Routledge.
Kardash, C. M., & Scholes, R. J. (1996). Effects of preexisting beliefs, epistemological
beliefs, and need for cognition on interpretation of controversial issues. Journal of
Educational Psychology, 88(2), 260.
Kindlinger, M. (2021). Meta-reexivity and epistemic cognition in social science teacher
education. Journal of Social Science Education, 20(3).
King, P. M., & Kitchener, K. S. (1994). Developing reective judgment: Understanding and
promoting intellectual growth and critical thinking in adolescents and adults. San
Francisco: Jossey-Bass.
Kitchner, K. S. (1983). Cognition, metacognition, and epistemic cognition. Human
Development, 26(4), 222–232.
Kozikoglu, I. (2019). Investigating critical thinking in prospective teachers:
Metacognitive skills, problem solving skills and academic self-efcacy. Journal of
Social Studies Education Research, 10(2), 111–130.
Kuhn, D. (2001). How do people know? Psychological Science, 12(1), 1–8.
Kuhn, D. (2022). Metacognition matters in many ways. Educational Psychologist, 57(2),
73–86.
Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological
understanding. Cognitive Development, 15(3), 309–328.
Li, S., Ren, X., Schweizer, K., Brinthaupt, T. M., & Wang, T. (2021). Executive functions
as predictors of critical thinking: Behavioral and neural evidence. Learning and
Instruction, 71, Article 101376.
Lins de Holanda Coelho, G., Hanel, P. H. P., & Wolf, L. J. (2020). The very efcient
assessment of need for cognition: Developing a six-item version. Assessment, 27(8),
1870–1885.
Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking in higher
education: Current state and directions for next-generation assessment. ETS Research
Report Series, 2014(1), 1–23.
Liu, O. L., Mao, L., Frankel, L., & Xu, J. (2016). Assessing critical thinking in higher
education: The HEIghten™ approach and preliminary validity evidence. Assessment
& Evaluation in Higher Education, 41(5), 677–694.
Liu, O. L., Shaw, A., Gu, L., Li, G., Hu, S., Yu, N., et al.others. (2018). Assessing college
critical thinking: Preliminary results from the chinese HEIghten critical thinking
assessment. Higher Education Research & Development, 37(5), 999–1014.
Livingston, J. A. (2003). Metacognition: An overview.
Magno, C. (2010). The role of metacognitive skills in developing critical thinking.
Metacognition and Learning, 5(2), 137–156.
Mason, L. (2016). Psychological perspectives on measuring epistemic cognition. In
J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp.
375–392). New York: Routledge.
Mason, L., & Scirica, F. (2006). Prediction of students’ argumentation skills about
controversial topics by epistemological understanding. Learning and Instruction, 16
(5), 492–509.
McGrew, K. S., Schneider, W. J., Decker, S. L., & Bulut, O. (2023). A psychometric
network analysis of CHC intelligence measures: Implications for research, theory,
and interpretation of broad CHC scores “beyond g”. Journal of Intelligence, 11(1), 19.
McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students
evaluate online sources? Learning from assessments of civic online reasoning. Theory
& Research in Social Education, 46(2), 165–193.
McGrew, S., Ortega, T., Breakstone, J., & Wineburg, S. (2017). The challenge that’s
bigger than fake news: Teaching students to engage in civic online reasoning.
American Educator, 41(3), 4.
Merk, S., Rosman, T., Muis, K. R., Kelava, A., & Bohl, T. (2018). Topic specic epistemic
beliefs: Extending the theory of integrated domains in personal epistemology.
Learning and Instruction, 56, 84–97.
Metts, S., & Bressler, M. S. (2023). From fraudsters to scammers and cyber-villains, tech-
savvy criminals are out to steal your money. Journal of Accounting & Finance (2158-
3625), 23(4).
Mevarech, Z., & Fridkin, S. (2006). The effects of IMPROVE on mathematical knowledge,
mathematical reasoning and meta-cognition. Metacognition and Learning, 1(1),
85–97.
Moeller, J. (2015). A word on standardization in longitudinal studies: Don’t. Frontiers in
Psychology, 6, 1389.
Moshman, D. (2013). Epistemic cognition and development. In The development of
thinking and reasoning (pp. 25–45). Psychology Press.
Moshman, D. (2020). Metacognition and epistemic cognition. In Oxford research
encyclopedia of education.
Muis, K. R., Chevrier, M., Denton, C. A., & Losenno, K. M. (2021). Epistemic emotions
and epistemic cognition predict critical thinking about socio-scientic issues.
Frontiers in Education, 6, 1–18.
Muis, K. R., & Franco, G. M. (2010). Epistemic proles and metacognition: Support for
the consistency hypothesis. Metacognition and Learning, 5(1), 27–45.
Nerantzaki, K., Efklides, A., & Metallidou, P. (2021). Epistemic emotions: Cognitive
underpinnings and relations with metacognitive feelings. New Ideas in Psychology,
63, Article 100904.
Newton, C., Feeney, J., & Pennycook, G. (2024). On the disposition to think analytically:
Four distinct intuitive-analytic thinking styles. Personality and Social Psychology
Bulletin, 50(6), 906–923.
Nuzzo, J. L., & Deaner, R. O. (2023). Men and women differ in their interest and
willingness to participate in exercise and sports science research. Scandinavian
Journal of Medicine & Science in Sports, 33(9), 1850–1865.
O’Leary, M., Reynolds, K., Ling, G., Liu, O. L., Belton, S., O’Reilly, N., & McKenna, J.
(2020). Assessing critical thinking in higher education: Validity evidence for the use
of the HEIghten™ critical thinking test in Ireland. Journal of Higher Education Theory
and Practice, 20(12), 115–130.
Orona, G. A. (2021a). Gotta know why! Preliminary evidence supporting a theory of
virtue learning as applied to intellectual curiosity. Theory and Research in Education,
19(3), 279–295.
Orona, G. A. (2021b). Philosophy’s rematch: A new conceptualization of the study of
higher education. Arts and Humanities in Higher Education, 20(4), 386–402.
Orona, G. A., Eccles, J. S., Zitzmann, S., Fischer, C., & Arum, R. (2023a). Cognitive
development in undergraduate emerging adults: How course-taking breadth
supports skill formation. Contemporary Educational Psychology, 102206.
Orona, G. A., Liu, O. L., & Arum, R. (2023b). The language of 21st century skills: Next
directions for closing the skills gap between employers and postsecondary graduates.
Chinese/English Journal of Educational Measurement and Evaluation| 教育测量与评估双
语期刊, 4(2), 3.
Orona, G. A., & Pritchard, D. (2022). Inculcating curiosity: Pilot results of an online
module to enhance undergraduate intellectual virtue. Assessment & Evaluation in
Higher Education, 47(3), 375–389.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
17
Orona, G. A., Pritchard, D., Arum, R., Eccles, J., Dang, Q.-V., Copp, D., et al. (2024).
Epistemic virtue in higher education: Testing the mechanisms of intellectual
character development. Current Psychology, 1–15.
Orona, G. A., & Trautwein, U. (2024). Thinking disposition education improves cognitive
reection: Experimental results from an intervention study. Thinking Skills and
Creativity, 101569.
Perry, W. G. (1970). Forms of intellectual and ethical development in the college years: A
scheme. New York: Holt, Rinehart; Winston.
Primi, C., Morsanyi, K., Donati, M. A., Galli, S., & Chiesi, F. (2017). Measuring
probabilistic reasoning: The construction of a new scale applying item response
theory. Journal of Behavioral Decision Making, 30(4), 933–950.
Puente-Díaz, R., Cavazos-Arroyo, J., & Vargas-Barrera, F. (2021). Metacognitive feelings
as a source of information in the evaluation and selection of creative ideas. Thinking
Skills and Creativity, 39, Article 100767.
Rivas, S. F., Saiz, C., & Ossa, C. (2022). Metacognitive strategies and development of
critical thinking in higher education. Frontiers in Psychology, 13.
Ross, L. (2020). The virtue of curiosity. Episteme, 17(1), 105–120.
Schraw, G., Dunkle, M. E., & Bendixen, L. D. (1995). Cognitive processes in well-dened
and ill-dened problem solving. Applied Cognitive Psychology, 9(6), 523–538.
Shaw, A., Liu, O. L., Gu, L., Kardonova, E., Chirikov, I., Li, G., et al.others. (2020).
Thinking critically about critical thinking: Validating the russian HEIghten critical
thinking assessment. Studies in Higher Education, 45(9), 1933–1948.
Shin, J., Jian, L., Driscoll, K., & Bar, F. (2018). The diffusion of misinformation on social
media: Temporal pattern, message, and source. Computers in Human Behavior, 83,
278–287.
Shtulman, A., & Young, A. G. (2023). The development of cognitive reection. Child
Development Perspectives, 17(1), 59–66.
Simonovic, B., Vione, K., Stupple, E., & Doherty, A. (2023). It is not what you think it is
how you think: A critical thinking intervention enhances argumentation, analytic
thinking and metacognitive sensitivity. Thinking Skills and Creativity, 101362.
Sinatra, G. M. (2016). Thoughts on knowledge about thinking about knowledge. In
Handbook of epistemic cognition (pp. 479–492). Routledge.
Sinnott, J. D. (1998). The development of logic in adulthood: Postformal thought and its
applications. Springer Science & Business Media.
Sinnott, J. D. (2021). Psychology, politics, and complex thought: A time for postformal
thought in politics. In The psychology of political behavior in a time of change (pp.
147–176).
Sinnott, J. D., Hilton, S., Wood, M., & Douglas, D. (2020). Relating ow, mindfulness,
cognitive exibility, and postformal thought: Two studies. Journal of Adult
Development, 27, 1–11.
Sodian, B., & Kristen, S. (2016). Theory of mind. In Handbook of epistemic cognition (pp.
68–85). Routledge.
Soto, C. J., & John, O. P. (2017). Short and extra-short forms of the big ve inventory—2:
The BFI-2-s and BFI-2-XS. Journal of Research in Personality, 68, 69–81.
Stanovich, K. E. (2016). The comprehensive assessment of rational thinking. Educational
Psychologist, 51(1), 23–34.
Swanson, H. L. (1990). Inuence of metacognitive knowledge and aptitude on problem
solving. Journal of Educational Psychology, 82(2), 306.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive
Science, 12(2), 257–285.
Sweller, J., Van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and
instructional design. Educational Psychology Review, 10, 251–296.
Tamura, A., Ishii, R., Yagi, A., Fukuzumi, N., Hatano, A., Sakaki, M., et al. (2022).
Exploring the within-person contemporaneous network of motivational engagement.
Learning and Instruction, 81, Article 101649.
Tang, K.-S. (2020). The use of epistemic tools to facilitate epistemic cognition &
metacognition in developing scientic explanation. Cognition and Instruction, 38(4),
474–502.
Tarchi, C. (2024). Promoting multiple-text comprehension through thinking dispositions:
A randomized controlled trial. Discourse Processes, 1–16.
Thacker, I., & Sinatra, G. M. (2022). Supporting climate change understanding with
novel data, estimation instruction, and epistemic prompts. Journal of Educational
Psychology, 114(5), 910.
UCI Demographics. (2022–23). https://inclusion.uci.edu/wp-content/uploads/2023/0
4/UCI-Demographic-Charts-2022-23.pdf.
Van Borkulo, C. D., Borsboom, D., Epskamp, S., Blanken, T. F., Boschloo, L.,
Schoevers, R. A., & Waldorp, L. J. (2014). A new method for constructing networks
from binary data. Scientic Reports, 4(1), 1–10.
Vogl, E., Pekrun, R., & Loderer, K. (2021). Epistemic emotions and metacognitive feelings.
Trends and prospects in metacognition research across the life span: A tribute to Anastasia
Efklides (pp. 41–58).
Vukman, K. B. (2005). Developmental differences in metacognition and their
connections with cognitive development in adulthood. Journal of Adult Development,
12(4), 211–221.
Waldorp, L. J., Borsboom, D., & Epskamp, S. (2022). Association and conditional
independence. In Network psychometrics with r (pp. 79–89). Routledge.
Weinstock, M. P., Neuman, Y., & Glassner, A. (2006). Identication of informal reasoning
fallacies as a function of epistemological level, grade level, and cognitive ability.
Journal of Educational Psychology, 98(2), 327.
van der Wel, R. P. (2015). Me and we: Metacognition and performance evaluation of
joint actions. Cognition, 140, 49–59.
Wineburg, S., Breakstone, J., McGrew, S., & Ortega, T. (2016). Evaluating information: The
cornerstone of online civic literacy; executive summary. Stanford, CA: Stanford History
Education Group.
Wrzus, C., & Roberts, B. W. (2017). Processes of personality development in adulthood:
The TESSERA framework. Personality and Social Psychology Review, 21(3), 253–277.
Wyre, S. H. (2007). Critical thinking, metacognition, and epistemological beliefs (PhD thesis).
University of Phoenix
Yan, Z., & Fischer, K. (2002). Always under construction: Dynamic variations in adult
cognitive microdevelopment. Human Development, 45(3), 141–160.
Zagzebski, L. T. (1996). Virtues of the mind: An inquiry into the nature of virtue and the
ethical foundations of knowledge. Cambridge University Press.
˙
Zyluk, N., Karpe, K., & Urba´
nski, M. (2022). The development of epistemological
understanding revisited: Enhancing reliability of the tool by using only abstract
items. SAGE Open, 12(2), 21582440221094602.
G.A. Orona et al.
Learning and Individual Dierences 117 (2025) 102584
18
