ArticlePublisher preview available

Inadequacies in the SES–Achievement model: Evidence from PISA and other studies

Wiley
Review of Education
Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Students’ socioeconomic status (SES) is central to much research and policy deliberation on educational inequalities. However, the SES model is under severe stress for several reasons. SES is an ill‐defined concept, unlike parental education or family income. SES measures are frequently based on proxy reports from students; these are generally unreliable, sometimes endogenous to student achievement, only low to moderately intercorrelated, and exhibit low comparability across countries and over time. There are many explanations for SES inequalities in education, none of which achieves consensus among research and policy communities. SES has only moderate effects on student achievement, and its effects are especially weak when considering prior achievement, an important and relevant predictor. SES effects are substantially reduced when considering parent ability, which is causally prior to family SES. The alternative cognitive ability/genetic transmission model has far greater explanatory power; it provides logical and compelling explanations for a wide range of empirical findings from student achievement studies. The inadequacies of the SES model are hindering knowledge accumulation about student performance and the development of successful policies. Context and implications Rationale for this study This review was written in response to the disconnect between the literature surrounding student achievement studies, and the cognitive psychology and behavioural genetic academic literatures. It is well‐established that student achievement is closely related to cognitive ability and both have sizable genetic components, findings largely ignored in achievement studies. This review’s aim is for more considered responses to socioeconomic inequalities in student achievement by both researchers and policymakers. Why the new findings matter The review provides overwhelming evidence that much of the current thinking about SES and student achievement is mistaken. Implications for researchers and policymakers The current emphasis on SES is misleading and wastes considerable human and financial resources that could much better be utilized. The focus should be on student performance ensuring that low achievers have rewarding educational and occupational careers, and raising the overall skill levels of students, not on the nebulous, difficult to measure, concept of SES, which is only moderately associated with achievement.
Review of Educ ation. 2021;9:e3293.
|
1 of 36
https://doi.org/10.1002/rev3.3293
wileyonlinelibrary.com/journal/roe
Received: 7 August 2020
|
Accepted: 14 May 2021
DOI: 10.1002/rev3.3293
STATE- OF- THE- ART REVIEW
Inadequacies in the SES– Achievement model:
Evidence from PISA and other studies
Gary N. Marks1 | Michael O'Connell2
© 2021 British Educational Research Association
1Social and Political Sciences, University of
Melbourne, Melbourne, VIC, Australia
2School of Psychology, UCD, Dublin,
Ireland
Correspondence
Gary N. Marks, Social and Political
Sciences, University of Melbourne,
Parkville 3052, Victoria, Australia.
Emails: gmarks@unimelb.edu.au;
garymarks2030@gmail.com
Abstract
Students’ socioeconomic status (SES) is central to
much research and policy deliberation on educational
inequalities. However, the SES model is under se-
vere stress for several reasons. SES is an ill- defined
concept, unlike parental education or family income.
SES measures are frequently based on proxy reports
from students; these are generally unreliable, some-
times endogenous to student achievement, only low
to moderately intercorrelated, and exhibit low com-
parability across countries and over time. There are
many explanations for SES inequalities in education,
none of which achieves consensus among research
and policy communities. SES has only moderate ef-
fects on student achievement, and its effects are es-
pecially weak when considering prior achievement,
an important and relevant predictor. SES effects are
substantially reduced when considering parent abil-
ity, which is causally prior to family SES. The alterna-
tive cognitive ability/genetic transmission model has
far greater explanatory power; it provides logical and
compelling explanations for a wide range of empirical
findings from student achievement studies. The inad-
equacies of the SES model are hindering knowledge
accumulation about student performance and the de-
velopment of successful policies.
KEYWORDS
assessment, educational psychology, socioeconomic circumstances
Funding information
None.
... However, the complex relationships of these variables have not been comprehensively investigated in different samples. Moreover, the most frequently repeated determinants of learning outcomes are cognitive capacities, SES, and gender (Boman, 2023;Lee and Borgonovi, 2022;Marks and O'Connell, 2021). Cognitive ability affects academic achievement (Deary et al., 2007;Shi and Qu, 2022;Sternberg et al., 2008). ...
... Students' socioeconomic status (SES) has been used often to explain academic achievement. Based on the SES model, numerous publications have been produced on the scope of socioeconomic inequalities in education, theoretical explanations and measurement of their relationship to education, and policies aimed at reducing inequalities in education (Marks and O'Connell, 2021). SES and academic achievement relationship has been consistently confirmed in national and international contexts with a significant body of research (Bradley and Corwyn, 2002;Coleman et al., 1966;Davis-Kean, 2005;Hattie, 2009;Sirin, 2005). ...
... Although the effects of SES on academic performance have varied across countries and cultures, SES has been directly or indirectly associated with high academic achievement (Bray, 2006;Lee and Borgonovi, 2022;Sackett et al., 2009;Sirin, 2005). However, Marks and O'Connell (2021) have argued that the exact relationship between SES and academic achievement needs to be clarified through further studies. Since cognitive ability cannot fully explain SES and SES cannot fully explain cognitive ability, these two constructs should be studied in multivariate models (Boman, 2022). ...
Article
Full-text available
We utilized PISA-2018 data from 71 countries to investigate the relationship between cognitive constructs employed in reading literacy (locating information, understanding, evaluating and reflecting, single and multiple thinking) and math and science achievement. We found that these cognitive constructs collectively accounted for 56% of the variance in math and 63% in science achievement, even after adjusting for gender, socioeconomic status, and country fixed effects. This means that the majority of cultural differences in math and science achievement (resulting from different education systems) can be explained by cognitive constructs employed in reading. We also noted that, at the country level, coefficients of cognitive constructs employed in reading demonstrated an interesting reconciliatory pattern. Countries with a weaker coefficient on the “locating information” dimension tended to have stronger coefficients on the “understanding” and “evaluation and reflection” dimensions; whereas, countries with a stronger coefficient on “understanding” dimension tended to have a weaker coefficient on “evaluation and reflection” dimension. These findings are particularly significant for STEM interventions aiming to enhance math and science achievement, as they indicate that a substantial portion of the variance in these achievements can be explained by cognitive constructs employed in reading literacy. Furthermore, culture-specific reconciliatory patterns imply that strengths in certain cognitive skills can compensate for weaknesses of others. Therefore, schools should consider modifying their curricula to integrate cognitive constructs employed in reading literacy more into math and science education.
... This journal recently published a commentary by Debouwere (2024) on our article 'Inadequacies in the SES-Achievement model: Evidence from PISA and other studies' (Marks & O'Connell, 2021). Our paper argues that the predominance of the socioeconomic status (SES) model in both research and policy is unwarranted. ...
... After over a half-century of research, and the implementation of policies aiming to reduce SES inequalities in student achievement, the SES-achievement relationship has declined only marginally, if at all. The statistical relationships between SES and achievement cannot be accounted for by early childhood socialisation, parenting, cultural resources, economic resources or schools, or any one of the plethora of theories attempting to explain the relationship (see Marks & O'Connell, 2021). ...
... The most credible explanation for the relationship between SES and student achievement involves parents' and children's cognitive abilities and genetics (see Marks & O'Connell, 2021). Parents' genes influence their cognitive ability, and thus their educational and occupational attainments, and family income. ...
Article
Full-text available
The first section of this paper sets the record straight regarding many of Debouwere's (2024, Review of Education, 12, e3445) specific criticisms. The second section discusses the magnitude of the SES‐achievement relationship, specifically Debouwere's (2024) contention that the correlation is strong around 0.5 or 0.6 compared to observed correlations mostly between 0.2 and 0.3. The third section deals with five issues that Debouwere (2024) raises in his paper: (1) the stability of SES vis‐à‐vis cognitive ability; (2) the accuracy of children's reports of parents' socioeconomic characteristics; (3) whether teachers discriminate by students' SES; (4) the importance of cognitive ability for educational differentiation (i.e., tracking and streaming); and (5) SES effects on student achievement, controlling for prior achievement. The fourth section discusses the role of genetics in student achievement. Meta‐analyses and other studies indicate that about 50%–70% of the variance in student achievement is attributable to genetics (i.e., the heritability). The high heritability of student achievement accounts for its high stability, its strong correlations with cognitive ability and the weak effects of SES, net of prior achievement or cognitive ability.
... On the other hand, other researchers have argued that the impact of family SES on educational outcomes has been overstated (Marks & O'Connell, 2021;O'Connell & Marks, 2022). For example, Dickson et al. (2016), exploiting the exogenous shift in parents' education levels induced by the 1972 minimum school leaving age reform in England, estimated that parental education had a significant but small causal effect on children's academic performance. ...
... For example, Holmlund et al. (2011) indicated that removing genetic transmission reduced the apparent influence of parental education. Moreover, Marks and O'Connell (e.g., Marks & O'Connell, 2023;Marks & O'Connell, 2021;O'Connell & Marks, 2022) have argued that the intergenerational transmission of abilities, rather than sociological processes tied to SES, is more important for students' academic success. ...
Article
Full-text available
A growing body of research has examined the relationship between family socioeconomic status (SES) and educational outcomes. Meta-analyses of raw correlations generally indicate moderate associations, typically between 0.12 and 0.3 for academic achievement and around 0.18 to 0.4 for educational attainment. Causal inference studies, aimed at capturing the true effects of SES, report much weaker associations, usually around 0.1 or less. Despite the importance of these causal estimates, few studies have systematically reviewed evidence from causal research. To address this gap, we conducted a systematic review of studies on the causal effect of SES on educational achievement and attainment. A total of 24 causal inference studies published between 1990 and 2023 were reviewed. The findings contribute to the literature and theory in several ways. First, the meta-analysis revealed a small and non-significant effect of SES on academic achievement (Cohen’s d = 0.03) and a small but statistically significant effect on educational attainment (d = 0.08). Second, moderator analyses indicated that parental education exerted a stronger influence on educational attainment than that of family income. Moreover, the absence of significant differences in SES effects between developed and developing countries, as well as across various causal inference research designs (i.e., sample size, model specification, and methodologies), calls into question the assumed context-dependent nature of SES influence. Overall, the findings challenge SES-centered theories, showing that the causal impact of family SES on educational outcomes is much smaller than generally believed, and suggest that universal mechanisms may underlie the SES-education relationship.
... В совокупности это составляет важнейшие предпосылки для развития у детей интеллектуальных и речевых навыков (Gustafsson et al. 2013). Отмечается, что в основе вклада уровня образования родителей в академические достижения их детей могут лежать когнитивные способности самих родителей, которые, в свою очередь, выступают как значимые детерминанты их образовательного уровня, поскольку показано, что вклад образования родителей существенно снижается при учете их способностей (Marks, O'Connell 2021). На примере лонгитюдного исследования, проведенного американскими учеными на материале анализа более 5000 случаев, было показано, что положительные взаимосвязи между когнитивными возможностями человека (в частности, характеристиками речевых функций, исполнительных функций и, в меньшей степени, памяти) и уровнем образования его родителей сохраняются на протяжении всей жизни, хотя и несколько ослабевают по мере взросления, в отличие от других параметров социально-экономического статуса родительской семьи, вклад которых в когнитивные возможности человека в среднем возрасте становится уже статистически незначимым (Greenfield, Moorman 2019). ...
Article
Full-text available
Введение. Сегодня имеются неопровержимые данные о том, что между уровнем образования родителей и академическими достижениями их детей существуют прямые взаимосвязи. Однако психологические механизмы, лежащие в их основе, описаны довольно фрагментарно, несмотря на то, что их учет важен при планировании и реализации работы, направленной на повышение образовательного уровня родителей в контексте содействия академическим успехам их детей. В связи с этим целью представленного исследования стало обобщение сведений о факторах, которые могут определять вклад уровня образования родителей в академические успехи обучающихся. Материалы и методы. Использовался метод систематического обзора литературы, отбор источников осуществлялся по ключевым словосочетаниям «образование родителей» и «академические достижения» / «академическая успеваемость» (“education of parents ” и “academic achievements” / “academic performance”) в библиографических базах еLibrary, Google Scholar и PubMed по следующим критериям: 1) наличие доступа к полнотекстовой версии статьи или развернутой аннотации; 2) представленность в тексте результатов эмпирических или вторичных (метааналитических) исследований; 3) изложение материалов исследования на русском или английском языке; 4) публикация материалов исследования в рецензируемом научном издании. Первоначально массив данных включал 452 статьи, из которых соответствовали указанным критериям 39 источников. Эти статьи и были включены в обзор. Результаты. Выделены три группы факторов, которые определяют вклад образовательного уровня родительской семьи в академические успехи обучающихся: когнитивные и некогнитивные особенности обучающихся и их родителей (интеллектуальные, мотивационные, регуляторные, рефлексивные и др.), характеристики детско-родительских отношений (академическая вовлеченность родителей, их ожидания в отношении академических успехов детей и др.) и социальный статус семьи, отражающий ее положение в системе социальных отношений (уровень дохода, принадлежность к расовым/ этническим группам большинства/меньшинства и др.). Заключение. Перспективы исследования связаны с эмпирической проверкой гипотезы об опосредующем влиянии указанных факторов на взаимосвязи между уровнем образования родителей и академическими успехами их детей в контексте изучения траекторий образовательной социализации обучающихся на разных образовательных ступенях. Результаты представляются значимыми для повышения эффективности образовательной работы с родителями, направленной на содействие академическим успехам обучающихся.
... Several factors contribute to students' mathematical literacy skills. Socioeconomic status significantly affects their performance on PISA (Perry & Mcconney, 2010), including parental educational attainment (O'Connell, 2019) and changes in family circumstances (Marks & O'Connell, 2021). Access to comprehensive facilities and a supportive learning environment enhance students' learning effectiveness. ...
... Esta desigualdad entre el alumnado nativo y extranjero se ha vinculado a una serie de factores como, por ejemplo, los relacionados con la escuela como pueden ser la segregación escolar, las influencias negativas de los compañeros o la falta de participación de las familias (Brown et al., 2019;Griffin, 2002;Leavitt y Hess, 2019;Whaley, 2020). Y también, a factores asociados a la condición familiar como el desconocimiento del idioma del país acogedor, el nivel educativo de sus familias, o situación económica familiar (Lee et al., 2019;Marks y O'Connell, 2021;Wang et al., 2023). En un ejercicio de síntesis, Schnepf (2007) agrupa todos los factores que podrían estar relacionados con la brecha del rendimiento de los estudiantes extranjeros en dos grandes categorías: las diferencias socioeconómicas y culturales en comparación con la población nativa y la desigual distribución territorial y escolar del alumnado extranjero. ...
Article
Full-text available
España ha experimentado un aumento significativo de la población extranjera en edad escolar y la caída demográfica hace augurar un incremento relativo en las próximas décadas. El objetivo de este estudio es contribuir a una mayor comprensión de la relación entre la concentración de estudiantes extranjeros en las escuelas y su rendimiento académico explorando los factores mediadores. Para ello, se realiza una explotación de datos de 28.708 estudiantes de 887 escuelas de Educación Primaria españolas. Los resultados muestran como el alumnado de origen extranjero tiende a concentrarse en centros públicos lo cual tiene un efecto diferencial sobre el rendimiento en función del grado de concentración. Una presencia inferior al 10% de estudiantes extranjeros se asocia a mayor rendimiento mientras que concentraciones superiores al 20% se vinculan con un rendimiento menor. Las diferencias en el rendimiento tendrían su origen en los recursos dispuestos en el proceso educativo, micro-interacciones desiguales que genera el alumnado, efectos escolares y efectos de composición, lo que lleva a la necesidad de reconfigurar los procesos de asignación de plazas escolares como herramienta para reducir la segregación y aumentar el desarrollo académico en su conjunto.
... Thien & Ong, 2015;Pakpahan, 2016;Kartianom & Ndayizeye, 2018;Sari & Kismiantini, 2023). On the contrary, other study shows that relationship between ESCS and student performance might be weak and less reliable (Marks & O'Connell, 2021;Pokropek et al., 2022). Other studies found that ESCS affect students' performance through mediator variables, such as expected occupational status, enjoyment of reading, mastery-approach orientation of achievement goals, and cultural capital (Michael & Kyriakides, 2023;Xie & Ma, 2019). ...
Article
Full-text available
How Economics-Socio-Cultural Status affect Indonesian students' performance in Mathematics? an Insight from PISA 2012-2022. To date, the average mathematics performance of Indonesian students in PISA tends to be unchanged, and lower than that of OECD countries. However, PISA participants in Indonesia consist of students from five to six different grades. Objectives: This research aims to determine how Economic-Socio-Cultural Status (ESCS) and student grades influence Indonesian students' mathematics performance. Methods: We apply a quantitative approach using multiple linear regression analysis methods using the PISA 2012, 2015, 2018, and 2022 datasets. In this analysis, we also use the weight of each sample and the overall plausible value for mathematics performance. Findings: we found a significant relation between education level and ESCS regarding students' overall mathematics performance in PISA from 2012 to 2022. The interaction coefficient increases as the level increases from grade 8 of junior secondary school to grade 11. Conclusion: ESCS has a contribution to the diversity of students' mathematics performance at the same grade. From 2012 to 2022, the relative position of Indonesian students' ESCS compared to all participant has declined, although its influence on student mathematics performance tends to decline.
... Various measures of students' socioeconomic status are related to mathematics performance 16 . Some research, including the OECD's Programme for International Student Assesssment (PISA, using a composite metric of various SES indicators) has naively attributed a causal nature to these correlations, suggesting that parents possessing higher socioeconomic status can directly improve their children's performance through additional home tutoring or a more nurturing home environment [17][18][19] . In contradiction to this theory, however, more detailed studies have shown that once psychometric intelligence is accounted for, various measures of family socieconomic status are at best a minor contributor to school performance 20,21 . ...
Article
Full-text available
Mathematics is a key school subject for some of the most lucrative and economically important careers. Low mathematics performance in school is associated with low psychometric intelligence, family socioeconomic status (SES), specific mathematical abilities, and high mathematics anxiety. We used a sample of Hungarian schoolchildren (N = 102, mean age = 12.3 years) to directly compare the predictive power of general intelligence, specific mathematical abilities measured by the Pedagogical Examination of Dyscalculia (DPV), mathematics anxiety, and socioeconomic status for mathematics grades. Mathematics grades correlated with IQ, specific mathematic ability, mathematics anxiety, and a composite measure of family SES. The WISC-IV showcased a manifest correlation of 0.62 and a latent correlation of 0.78 with the DPV and high manifest (r=-0.53) and latent (r=-0.59) correlations with mathematics anxiety. IQ alone accounted for 52% of the variance in mathematics grades. IQ, specific mathematical ability, family SES and mathematics anxiety jointly accounted for 56% of the variance in grades, with a non-significant contribution of specific mathematical ability and family SES over IQ and a marginal contribution of mathematics anxiety. Our results show that psychometric intelligence is the most important predictor of mathematics grades, while family SES and specific mathematical abilities are only associated with grades to the extent they reflect psychometric intelligence. The results, however, confirmed a small role of mathematics anxiety over intelligence in predicting grades. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-024-77904-7.
... For example, selfrated abilities (i.e., non-cognitive abilities) tap into cognitive abilities and higher SES are associated with higher cognitive ability. However, these intercorrelations are small (r = 0.30<) with regard to the current data set and in general (Marks, 2022;Marks and O'Connell, 2021). That is also why the mediation models may omit covariates. ...
Article
Full-text available
While well-being does generally constitute a moderate predictor of school achievement, research on the predictive validity of cognitive ability for well-being in school contexts remains scant. The current study analyzed longitudinal relations between cognitive ability measured at age 13 (Grade 6) and well-being measured at age 18 (Grade 12, valid N = 2,705) in a Swedish sample, using several multivariate model techniques. The results indicate that cognitive ability was not a statistically significant predictor when several predictors were entered in a multiple regression model. However, gender was a significant covariate as girls and young women have a substantially lower degree of self-reported well-being. This casts light on the limitations of cognitive ability as a construct for some non-cognitive outcomes, at least in shorter and narrower spatial–temporal contexts.
... However, according to Harwell et al. (2017), this association is not as robust as one might initially assume. Supporting this perspective, Marks and O'Connell (2021) argue that the SES-achievement model in PISA is fraught with inadequacies, necessitating a re-evaluation of how SES is conceptualised and measured when explaining educational outcomes. Selvitopu and Kaya (2021) and Sirin (2005) also contribute to this discourse through meta-analytic methods, examining the relationship between SES and academic performance, thereby highlighting its complex and often nuanced nature. ...
Article
Full-text available
Purpose Given the scientific and practical difficulties inherent in measuring and comparing socioeconomic deprivation (SED), and the further complexity added in cross national measurements, the main aim of this paper was to check the validity of SED measures used in PISA 2018 dataset. The SED measure used in PISA 2018 was the PISA index of economic, social and cultural status abbreviated as the ESCS index. This assessment was important as PISA analysis is based on variables derived from this instrument and the ESCS index and these reports influence and reflect international and comparative education policies and practice. Design/methodology/approach This study critically evaluates the socioeconomic status measures in the PISA 2018 dataset, focusing on their convergent validity and cross-national comparability. Using responses from over 600,000 students in 73 countries, it examines the validity of SES indicators and their comparability across countries. The study employs principal component analysis to construct local SES measures and compares them with the existing Economic, Social, and Cultural Status (ESCS) index. It explores the relationship between these SES measures and academic achievement in reading, science, and mathematics, aiming to understand their predictive validity in diverse educational settings. Statistical analyses were conducted using the IEA’s IDB Analyser and SPSS, ensuring robustness and generalisability across the diverse participant countries. Findings Our research findings challenge the assumed superiority of local measures over broader constructs like the Economic, Social, and Cultural Status (ESCS). It suggests that standardised measures like ESCS may provide more reliable predictions of academic achievement across various educational contexts, underscoring the complex relationship between SES measures and academic performance. Originality/value Our novel analysis shows that local and cross-national SED measures are poorly correlated. Our findings raise questions about the measures' validity while acknowledging the methodological challenges. We provide empirical evidence to support ongoing debates on the topic.
Article
Full-text available
Citare come: Borgna, Camilla. 2021. “Crisi sanitaria e rischi per la povertà educativa”, Volere la Luna, Talpa n.25, “La scuola al tempo dell’incertezza”, 21.01.2021.
Article
Full-text available
International Large-Scale Assessments (LSA) allow comparisons of education systems’ effectiveness in promoting student learning in specific domains, such as reading, mathematics, and science. However, it has been argued that students’ scores in International LSAs mostly reflect general cognitive ability (g). This study examines the extent to which students’ scores in reading, mathematics, science, and a Raven’s Progressive Matrices test reflect general ability g and domain-specific abilities with data from 3,472 Polish students who participated in the OECD’s 2009 Programme for International Student Assessment (PISA) and who were retested with the same PISA instruments, but with a different item set, in 2010. Variance in students’ responses to test items is explained better by with a bifactor Item Response Theory (IRT) model than by the multidimensional IRT model routinely used to scale PISA and other LSAs. The bifactor IRT model assumes that non-g factors (reading, math, science, and Raven’s test) are uncorrelated with g and with each other. The bifactor model generates specific ability factors with more theoretically credible relationships with criterion variables than the multidimensional standard model. Further analyses of the bifactor model indicate that the domain-specific factors are not reliable enough to be interpreted meaningfully. They lie somewhere between unreliable measures of domain-specific abilities and nuisance factors reflecting measurement error. The finding that PISA achievement scores reflect mostly g, which may arise because PISA aims to test broad abilities in a variety of contexts or may be a general characteristic of LSAs and national achievement tests.
Article
The Coleman report, published 12 years after the Brown decision, confirmed that widespread school segregation in the United States created inequality of educational opportunity. This study examines whether racial and socioeconomic segregation, which is on the rise in the United States, is still contributing to the achievement differences among students. The study used data from the National Education Longitudinal Survey of 1988 to estimate multilevel models of achievement growth between Grades 8 and 12 in mathematics, science, reading, and history for a sample of 14,217 students attending a representative sample of 913 U.S. high schools. The study found that the average socioeconomic level of students’ schools had as much impact on their achievement growth as their own socioeconomic status, net of other background factors. Moreover, school socioeconomic status had as much impact on advantaged as on disadvantaged students, and almost as much impact on Whites as on Blacks, raising questions about the likely impact of widespread integration. The impact of socioeconomic composition was explained by four school characteristics: teacher expectations, the amount of homework that students do, the number of rigorous courses that students take, and students’ feelings about safety. The results suggest that schools serving mostly lower-income students tend to be organized and operated differently than those serving more-affluent students, transcending other school-level differences such as public or private, large or small. This article then addresses the question of whether such school characteristics can be changed by policies to reform schools and funding systems versus policies to desegregate schools.
Article
International Large-Scale Assessments (LSA) allow comparisons of education systems’ effectiveness in promoting student learning in specific domains, such as reading, mathematics, and science. However, it has been argued that students’ scores in International LSAs mostly reflect general cognitive ability (g). This study examines the extent to which students’ scores in reading, mathematics, science, and a Raven’s Progressive Matrices test reflect general ability g and domain-specific abilities with data from 3,472 Polish students who participated in the OECD’s 2009 Programme for International Student Assessment (PISA) and who were retested with the same PISA instruments, but with a different item set, in 2010. Variance in students’ responses to test items is explained better by with a bifactor Item Response Theory (IRT) model than by the multidimensional IRT model routinely used to scale PISA and other LSAs. The bifactor IRT model assumes that non-g factors (reading, math, science, and Raven’s test) are uncorrelated with g and with each other. The bifactor model generates specific ability factors with more theoretically credible relationships with criterion variables than the multidimensional standard model. Further analyses of the bifactor model indicate that the domain-specific factors are not reliable enough to be interpreted meaningfully. They lie somewhere between unreliable measures of domain-specific abilities and nuisance factors reflecting measurement error. The finding that PISA achievement scores reflect mostly g, which may arise because PISA aims to test broad abilities in a variety of contexts or may be a general characteristic of LSAs and national achievement tests. Educational Impact and Implications Statement: This study analyzes Programme for International Student Assessment data from Poland to establish how much the achievement of secondary school students in reading, mathematics, science and in a Raven’s Progressive Matrices test reflects general ability and how much it reflects domain-specific abilities. Findings indicate that a scaling model that accounts for general ability, fit the data better than models typically employed in large scale assessments that ignore the influence of general ability on student achievement. The finding that students’ responses to PISA test items reflect general ability rather than domain-specific abilities, if replicated to other countries, could have important implications for the design of large-scale assessments and the interpretation of analyses of large-scale assessment data.
Article
Hierarchical linear modeling (HLM) is often used to estimate the effects of socioeconomic status (SES) on academic achievement at different levels of an educational system. However, if a prior academic achievement measure is missing in a HLM model, biased estimates may occur on the effects of student SES and school SES. Phantom effects describe the phenomenon in which the effects of student SES and school SES disappear once prior academic achievement is added to the model. In the present analysis, partial simulation (i.e., simulated data are used together with real-world data) was employed to examine the phantom effects of student SES and school SES on science achievement, using the national sample of the United States from the 2015 Programme for International Student Assessment. The results showed that the phantom effects of student SES and school SES are rather real. The stronger the correlation between prior science achievement and (present) science achievement, the greater the chance that the phantom effects occur.