Franca Agnoli’s research while affiliated with University of Padua and other places

Solutions to the crisis in confidence in the psychological literature have been proposed in many recent articles, including increased publication of replication studies, a solution that requires engagement by the psychology research community. We surveyed Australian and Italian academic research psychologists about the meaning and role of replication in psychology. When asked what they consider to be a replication study, nearly all participants (98% of Australians and 96% of Italians) selected options that correspond to a direct replication. Only 14% of Australians and 8% of Italians selected any options that included changing the experimental method. Majorities of psychologists from both countries agreed that replications are very important, that more replications should be done, that more resources should be allocated to them, and that they should be published more often. Majorities of psychologists from both countries reported that they or their students sometimes or often replicate studies, yet they also reported having no replication studies published in the prior 5 years. When asked to estimate the percentage of published studies in psychology that are replications, both Australians (with a median estimate of 13%) and Italians (with a median estimate of 20%) substantially overestimated the actual rate. When asked what constitute the main obstacles to replications, difficulty publishing replications was the most frequently cited obstacle, coupled with the high value given to innovative or novel research and the low value given to replication studies.

Many studies have found that males, on average, perform better than females in mathematics, although the size of this gender gap is small and varies considerably across countries. Stereotype threat has been proposed as a principal cause of this gender gap. From this perspective, females’ performance is affected by fear of confirming a negative stereotype about females’ mathematical ability and this stereo-type can be activated by an experimental manipulation that reminds females of the stereotype. Yet, evidence of a stereotype threat effect on mathematics performance in childhood and adolescence has been mixed. The present study replicated a highly cited study of stereotype threat among Italian adolescents with a much larger sample of Italian ninth grade (89 male, 75 female, mean age = 14.2) and eleventh grade (84 male, 80 female, mean age = 16.2) public high school students. Performance in tests administered both before and after the experimental manipulations were analyzed with a series of logistic mixed-effects models. Model comparisons confirmed that males performed better than females, but the probability of a stereotype threat effect was infinitesimal. We conclude that Italian adolescent gender differences in mathematics may not be explained by stereotype threat effects.

Many recent articles have proposed solutions to the crisis in confidence in the psychology literature, including increased publication of replication studies, a solution that requires engagement by the psychology research community. We surveyed Australian and Italian academic research psychologists about the meaning and role of replication in psychology. When asked what they consider to be a replication study, nearly all participants (98% of Australians and 96% of Italians) selected options that correspond to a direct replication. Only 14% of Australians and 8% of Italians selected any options that included changing the experimental method. Majorities of psychologists from both countries agreed that replications are very important, more replications should be done, more resources should be allocated to them, and they should be published more often. Majorities of psychologists from both countries reported that they or their students sometimes or often replicate studies, yet they also reported having no replication studies published the past 5 years. When asked to estimate the percentage of published studies in psychology that are replications, both Australians (with a median estimate of 13%) and Italians (with a median estimate of 20%) substantially overestimated the actual rate. When asked what constitute the main obstacles to replications, difficulty publishing replications was the most frequently cited obstacle, coupled with the high value given to innovative or novel research and the low value given to replication studies.

Le differenze di genere nella performance matematica sono ancora oggi presenti in alcune nazioni, tuttavia non sempre tali differenze sono a vantaggio dei maschi. Dall’ultimo rapporto PISA (2015), in Italia i maschi ottengono punteggi più alti delle femmine (500 vs 480), mentre in Georgia (Caucaso) sono le femmine ad ottenere punteggi migliori: 411 vs 398. In Georgia il ragionamento logico-matematico è valorizzato più di ogni altro tipo di ragionamento e un’ottima conoscenza matematica è necessaria per molte carriere. Abbiamo confrontato gli atteggiamenti verso la matematica di studenti/esse di 14 e 15 anni di Italia e Georgia osservando che gli studenti georgiani considerano la matematica più importante e utile rispetto agli studenti italiani. Anche l’ultimo rapporto TIMSS (2015) evidenzia come in Georgia il 44% degli studenti dia “molto valore alla matematica”, contro il 19% degli studenti italiani. Tutto ciò supporta il ruolo dei fattori socio-culturali coinvolti nelle differenze di genere in matematica.

Le differenze di genere nella performance matematica sono ancora oggi presenti in alcune nazioni, tuttavia non sempre tali differenze sono a vantaggio dei maschi. Dall’ultimo rapporto PISA (2015), in Italia i maschi ottengono punteggi più alti delle femmine (500 vs 480), mentre in Georgia (Caucaso) sono le femmine ad ottenere punteggi migliori: 411 vs 398. In Georgia il ragionamento logico-matematico è valorizzato più di ogni altro tipo di ragionamento e un’ottima conoscenza matematica è necessaria per molte carriere. Abbiamo confrontato gli atteggiamenti verso la matematica di studenti/esse di 14 e 15 anni di Italia e Georgia osservando che gli studenti georgiani considerano la matematica più importante e utile rispetto agli studenti italiani. Anche l’ultimo rapporto TIMSS (2015) evidenzia come in Georgia il 44% degli studenti dia “molto valore alla matematica”, contro il 19% degli studenti italiani. Tutto ciò supporta il ruolo dei fattori socio-culturali coinvolti nelle differenze di genere in matematica.

There have been many contributions to the study of gender differences in mathematics in the last 40 years, and the main findings are captured in several meta-analyses (e.g., Else-Quest, Hyde, & Linn, 2010). These studies point to a gender difference in mathematics favoring males. Two important points, however, emerge from these studies: a) the size of the effect is small; and b) the gender gap has decreased over time and is disappearing in Northern Europe (Guiso, Monte, Sapienza, & Zingales, 2008). The reductions in gender differences in mathematics achievement observed in some age groups and some societies suggest that these gender differences are influenced by social factors. Indeed, stereotype threat (Spencer, Steele, & Quinn, 1999) is one of the explanations offered for gender differences in mathematics; gender differences arise because the performance of females is affected by their unconscious fear of confirming a negative stereotype about their mathematical ability. This explanation for gender differences in mathematics has been challenged in recent research and analyses (Ganley, et al., 2013; Flore & Wicherts, 2015). We suggest that inconsistencies in reported stereotype threat effects are due to flaws in experimental designs (e.g., lack of male participants) and inappropriate statistical analyses (e.g., inappropriate use of covariates). We conducted a study using a method of inducing stereotype threat that had yielded a significant threat effect (Muzzatti & Agnoli, 2007). Participants were 75 females and 89 males (mean age = 14.2) in the ninth grade and 80 females and 84 males (mean age = 16.2) in the eleventh grade. Students in the experimental condition examined photographs of ten famous mathematicians, including nine males and only one female. Students in the control condition examined ten neutral photographs including nine flowers and one fruit. All students completed a pre-test (18 mathematical problems), participated in either the experimental or control condition, and performed the post-test (18 similar mathematical problems). Studies of stereotype threat have generally analyzed data using Analysis of Variance, but this is not an appropriate analysis method because these studies consistently involve two random effects: subject and mathematics problem. Instead, we used Logistic Mixed-Effects Models that concurrently account for both random effects (Jaeger, 2008). Analyzing accuracy in the pre-test, where both gender and grade were fixed effects and both subjects and problems were random effects, we found that males performed better than females in both grades (OR = 1.49, CI = 1.08 - 2.06). To test the decrement in performance for females in the stereotype threat condition, Logistic Mixed-Effects Models were calculated. Two new variables were included in the analysis: condition (no-stereotype threat versus stereotype threat) and time (pre-test versus post-test) as fixed effects. Again, subjects and problems were included as random effects. There was no evidence of a stereotype threat effect on high school girls’ mathematics performance. This failure to replicate the findings that Muzzatti and Agnoli (2007) obtained with 13-year-olds suggests that stereotype threat effects are not robust and probably depend on unidentified aspects of the experimental methodology or analysis methods.

A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these findings generalize to other countries. All the original materials were translated into Italian, and members of the Italian Association of Psychology were invited to participate via an online survey. The percentages of Italian psychologists who admitted to having used ten questionable research practices were similar to the results obtained in the United States although there were small but significant differences in self-admission rates for some QRPs. Nearly all researchers (88%) admitted using at least one of the practices, and researchers generally considered a practice possibly defensible if they admitted using it, but Italian researchers were much less likely than US researchers to consider a practice defensible. Participants’ estimates of the percentage of researchers who have used these practices were greater than the self-admission rates, and participants estimated that researchers would be unlikely to admit it. In written responses, participants argued that some of these practices are not questionable and they have used some practices because reviewers and journals demand it. The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.

Questionnaire in Italian. (PDF)

Lo studio degli effetti della minaccia stereotipica sulla performance matematica in ambito evolutivo vede pochi contributi che si sono talvolta rivelati controversi (Flore & Wicherts, 2014; Ganley, Mingle, Ryan, Ryan, Vasilyeva, & Perry, 2013). Se vi è ormai un sostanziale accordo nel ritenere che lo stereotipo di genere si consolidi durante l’adolescenza, i processi alla base del suo sviluppo sono tutt’altro che chiari. Dal punto di vista metodologico sono inoltre emerse criticità nel disegno sperimentale, nel trattamento statistico dei dati e nella struttura delle prove utilizzate per la valutazione della performance matematica. Nel tentativo di superare tali criticità abbiamo indagato gli effetti della minaccia stereotipica: a) analizzando campioni numerosi di adolescenti (14 e 16 anni); b) progettando un disegno sperimentale altamente controllato; c) utilizzando nuove tecniche di modellizzazione alternative all’ANOVA e all’ANCOVA; d) specificando i criteri utilizzati per la valutazione della performance in matematica. E’ stato condotto uno studio su 328 studenti/esse appartenenti alle classi I e III della scuola secondaria di secondo grado del Veneto ad indirizzo scientifico. L’esperimento ha previsto tre fasi: un pre-test volto a rilevare la performance in matematica; una seconda fase nella quale la manipolazione sperimentale prevedeva l’attivazione dello stereotipo secondo il quale le donne non sono brave quanto gli uomini in matematica; un post-test per valutare il cambiamento nei risultati in matematica a seguito della manipolazione e un potenziale decremento nelle performance femminili in matematica (Muzzatti & Agnoli, 2007). Attraverso un modello di regressione logistica ad effetti misti per l’analisi dell’accuratezza sono state analizzate le risposte al pre-test (genere e classe come effetti fissi, partecipanti e item come effetti random) e al post-test (genere, classe, condizione e pre/post-test come effetti fissi, partecipanti e item come effetti random). Al pre-test i maschi ottengono risultati migliori delle femmine sia nella classe prima che nella classe terza (Z = 2.42, p = .02, OR = 1.49). Al post-test sono state eseguite due analisi separate per classe dalle quali non sono emerse interazioni significative a tre vie tra genere, condizione e pre/post-test. Dalla nostra indagine emerge la presenza di differenze di genere in termini di performance nel pre-test. Nel post-test non si è verificato il decremento nelle performance matematiche delle femmine a seguito della manipolazione. I risultati di questa ricerca, in contrasto con quelli presenti in letteratura, fanno riflettere su un fenomeno presente anche in psicologia evolutiva: il publication bias, cioè la tendenza a privilegiare e pubblicare gli studi che risultano statisticamente significativi.

Secondo la recente letteratura a livello internazionale il divario di genere in termini di performance in matematica si sta riducendo (OCSE, 2015). Eppure lo stereotipo secondo cui le donne sono meno portate per la matematica sussiste tutt’oggi ed è proprio a scuola che tale stereotipo emerge e si consolida. Molti ricercatori ritengono che la presenza dello stereotipo negativo abbia un ruolo centrale nel determinare le minori prestazioni delle studentesse in compiti matematici, costituendo una potenziale barriera per l’accesso delle donne ai settori scientifico-tecnologici. Uno studio effettuato in Veneto su 348 studenti/esse di 14 e 16 anni della scuola secondaria di II grado ad indirizzo scientifico ha cercato di chiarire se tali differenze sussistono in Italia. Esaminando la performance in compiti matematici, i dati evidenziano la presenza di una differenza di genere a vantaggio dei maschi, così come riportato dalle indagini PISA (Programme for International Student Assessment, 2012) e dalle Rilevazioni Nazionali degli apprendimenti (INVALSI, 2014).