Content uploaded by Marta Fraile
Author content
All content in this area was uploaded by Marta Fraile on Jul 10, 2019
Content may be subject to copyright.
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 1
Title: The gender gap in political knowledge: is it all about guessing? An
experimental approach
Monica Ferrín (University of Zürich)
Marta Fraile * (RSCAS-EUI and CSIC–IPP)
Gema García-Albacete (Carlos III University of Madrid)
*Corresponding author (marta.fraile@eui.eu; marta.fraile@csic.es)
Abstract
This article analyzes the extent to which the format and the ‘Do not know’ protocol of
political knowledge questions influence the size of the gender gap. By using a set of
experiments that manipulated the DK protocol and the format (open vs. closed) of
political knowledge questions in a face to face representative survey of the Spanish
population, we show that the format of the questions (open versus closed) is not
relevant in explaining the gender gap. DK protocols, however, influence outspoken
levels of political knowledge differently for men and women. DK discouraging
protocols tend to encourage the emergence of hidden knowledge among women,
whereas they mostly boost guessing among men. This finding suggests that the meaning
and the use of the DK option appear to be gender biased.
Keywords: political knowledge, gender gap, question format, don’t know
protocol, measurement problems, guessing
ACEPTED FOR PUBLICATION AT INTERNATIONAL JOURNAL OF PUBLIC OPINION
RESEARCH (FORTHCOMING IN 2016)
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 2
Research on public opinion suggests that the overall level of information,
knowledge, and understanding of politics of the average citizen is relatively poor.
Previous research also concludes that political knowledge is unequally distributed
among citizens (Converse, 1964, 1970; Delli Carpini & Keeter, 1996). The public’s
apparent political ignorance, and its resulting incompetence in democratic actions such
as voting, or other forms of political participation, has led to widespread pessimism in
the academic literature. A strand of research on public opinion has recently argued,
however, that this pessimism needs to be qualified (Boudreau & Lupia, 2011; Gibson &
Caldeira, 2009; Lupia, 2006; Lupia & McCubbins, 1998; Prior & Lupia, 2008).
According to this approach, the common analytic definition of political knowledge
(citizens’ ability to provide correct answers to a specific set of fact-based survey items)
presents relevant problems of validity (Boudreau & Lupia, 2011), something which
might result in misleading conclusions about the general public’s competence and the
origin and size of knowledge inequalities shown in previous studies. Among these,
some of the problems mentioned by this trend in the literature relate to the format and
the DK protocol of the items used to measure political knowledge (Luskin & Bullock,
2011; Miller & Orr, 2008; Mondak, 2001; Mondak & Davis, 2002; Robison 2015).
This study contributes to the debate in the literature about the effect of the
format and the DK protocol on gender inequalities in knowledge identified in the
literature (Burns, Schlozman & Verba, 2001; Delli Carpini & Keeter, 1996; Fraile,
2014; Frazer & Macdonald, 2003; Kenski & Jamieson, 2000; Verba, Burns &
Schlozman, 1997). The contribution is threefold: First, while previous studies have dealt
with these topics, very few have directly tested the effect of the question format on
gender inequalities in knowledge using an experimental research design that keeps
equal question content, while randomly changing the format (the sole exception is
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 3
Robison, 2015). Moreover, the potential interaction of question format and gender has
attracted no attention at all. Second, we develop previous findings on the interaction
between DK protocols and gender by exploring the extent to which the meaning of DK
differs for men and women. Finally, we conducted experiments with five different
knowledge items in a Spanish representative sample by employing a face to face survey.
This is significant as to date most of the literature on the topic has come from American,
British, and Canadian studies. Therefore we provide evidence from a European country,
contributing to the debate and to the generalizability of the results. Finally, the
experiments conducted here present a high level of external validity since the results can
be inferred to the Spanish electors.
Findings show that the format of the questions (open versus closed) is not
relevant in explaining the gender gap. They also show that women tend to use the DK
option more than men, independently of the format and the DK protocol; a point that
confirms previous studies. The novelty of our findings relates to the effects of
discouraging DK. It barely encourages the emergence of hidden knowledge, but rather
increases the propensity of respondents to guess. Not only this, but under the scenario of
DK discouragement, men appear to guess to a greater extent than women. This finding
suggests that the meaning and the use of the DK option appear to be gender biased.
Therefore DK really means ‘don’t know’ for male respondents, but not for women. We
discuss the implication of these findings in the last section.
The Validity of Political Knowledge Survey Items
From a methodological perspective, the validity of political knowledge measures
has been evaluated in relation to several aspects: the use of the DK option (Miller &
Orr, 2008; Mondak, 2001; Mondak & Anderson, 2004; Mondak & Davis, 2002; Sturgis
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 4
et al. 2008); the closed vs. open format (Luskin & Bullock, 2011; Robison, 2015); the
effect of providing incentives to respondents to provide a correct answer (Prior &
Lupia, 2008); the use of images vs. words (Prior, 2014); etc. Here, we focus on the type
of format – open ended vs. closed ended –, and the DK protocol used to measure
political knowledge. And, more specifically, on the way these different formats affects
estimated levels of knowledge and gender differences in political knowledge.
Regarding the open and closed format debate, there seems to be an agreement in
the literature about the advantages and disadvantages of each of the formats. Two topics
are of particular relevance here. First, the ability to stimulate or dampen correct
responses. While open ended questions tend to produce conservative estimations of
political knowledge since some knowledgeable respondents will not provide a correct
answer because they are not 100% certain (Mondak, 2001; Luskin & Bullock, 2011),
closed ended questions stimulate correct answers, as respondents might try to answer
even if they have only some partial knowledge about the question (they have a list of
possible correct answers from which to choose). Second, the ability to incentivize
guessing. While the closed format increases the propensity of respondents to guess from
a list of possible answers (Luskin & Bullock, 2011), open ended questions minimize the
risk of guessing to a great extent.
Regarding the protocol followed in surveys to present the DK option to the
interviewee, there is still a lively debate in the literature. One line of research
recommends avoiding the use of the DK option or, in the case that the option is
included in the survey item, discouraging respondents to choose this option (Mondak,
2001; Mondak & Anderson, 2004; Miller & Orr, 2008; Mondak & Davis, 2002).
Respondents might differ in their propensity to select DK for reasons unrelated to their
real objective knowledge. For instance, individuals who are risk takers will be more
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 5
willing to guess the answer to a survey question than individuals who are risk averse.
As a consequence, the presence of the DK option as a response category might
underestimate/overestimate knowledge levels for some respondents. However, other
studies urge caution about discouraging DK, given the risk that respondents will be
more motivated to simply provide their best guess, and therefore artificially inflate
general levels of knowledge (Luskin & Bullock 2011; Sturgis et al., 2008).
What are the implications of these debates for explaining the specific case of
gender differences in the levels of political knowledge extensively documented in the
literature (Burns et al., 2001; Kenski & Jamieson, 2000; Verba et al., 1997)? While
previous studies have analyzed the role of the DK protocol of knowledge items in
explaining part of the gender gap (Lizotee & Sidman, 2009; Mondak & Anderson,
2004), to the best of our knowledge no previous studies have explored the extent to
which the format (open vs. closed) has any relevant implication for explaining the
aforementioned gender gap.
Differences Between Men and Women: A Matter of Measurement?
The existence of significant gender differences in levels of political knowledge
is well documented in previous research on public opinion. Numerous studies show that
women tend to provide fewer correct answers than men to standard political knowledge
questions not only in the US but also in Canada and Europe (Burns et. al 2001; Delli
Carpini & Keeter, 1996; Fraile, 2014; Frazer & Macdonald, 2003; Kenski & Jamieson,
200; Mondak &Anderson, 2004; Verba et al., 1997).
Despite many attempts in the literature to explain this gap, the question still
remains unresolved and the debate is very much alive. Common explanations for gender
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 6
inequality in political knowledge point to the traditional social norms (that identify
women as responsible for parenting and other caring activities); as well as to the
socioeconomic disadvantages that women have traditionally suffered (Delli Carpini &
Keeter, 1996; Verba, Burns & Scholzman, 1997). However, the gender gap in
knowledge cannot be totally explained by considering individuals’ varying resources,
opportunities, and motivations (Delli Carpini & Keeter 1996). A growing body of
literature suggests that gender differences might be the product of the characteristic of
the items used to measure political knowledge.
One line of research argues that the gender gap (or at least part of it) is a product
of the content of the items used to measure knowledge. These studies show that the
majority of questions used in conventional surveys are biased in favor of men’s
preferences and interests (Dolan 2011; Stolle and Gidengil 2010). As a result, women
do not systematically know less about politics than men, but their levels of political
knowledge are constantly deflated by the traditional measures of political knowledge.
In fact when knowledge is measured on political areas and issues that differ from
electoral politics, such as local politics, civic rights and social policies, then the
differences between men and women are reduced or even vanish (Dolan 2011; Stolle &
Gidengil 2010; Shaker 2012)
Another line of research suggests that the format and design of the political
knowledge survey questions is also relevant to explain (at least) part of the differences
between women and men (Lizotee & Sidman, 2009; Mondak & Anderson, 2004).
According to these studies, women and men tend to respond differently to political
knowledge questions because women are more risk-averse than men (Frazer &
Macdonald, 2003; Kenski & Jamieson, 2000; Lizotee & Sidman, 2009; Mondak &
Anderson, 2004). In contrast, men take more risks and are more prone to guess what
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 7
they think is the correct answer than women (Lizotee & Sidman, 2009). Consequently,
and given the different propensity of women and men to guess, women’s levels of
political knowledge will be smaller when open -ended items or neutral DK protocols are
used– because the format and DK protocol requires more risk taking.
However what the literature has not explored yet is the possibility that the use of
the DK option (and thereby its meaning) is gender biased. This is what we explore in
this article: the extent to which format and DK protocols interact with gender, resulting
in diverse levels of hidden knowledge for men and women.
We test two competing hypotheses. The first states that the hidden knowledge of
women is more likely to appear when closed ended and DK discouraging items are used
to measure political knowledge. This is because these two formats are easier than open
ended and neutral DK items and entail therefore lower levels of risk when answering.
To put it more succinctly, H1 states that in closed ended/ DK discouraging items,
women should provide a larger number of correct answers (as they decrease the number
of DK), to a greater extent than men (and as compared to open ended items/ DK neutral
items). And vice-versa, men should increase the number of incorrect answers more than
women, as they decrease their number of DK answers (compared to open ended items/
DK neutral items). If this is the case, the implication is that DK has different meanings
for men and women.
An alternative hypothesis suggests that there will be no differences in the
increase of in/correct answers between women and men, since closed ended and DK
discouraging items only encourage guessing. Since the probability that women and men
guess the correct answer is in theory the same (50%/50%), in close ended and DK
discouraging items men and women should provide as many correct as incorrect
answers, rather than DK (H2). If this hypothesis holds, the implication is that the gender
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 8
gap is independent of the type of format or DK protocol used to measure political
knowledge.
Research Design and Data
To test the two aforementioned competing hypotheses we have designed a
survey with the aim of assessing Spanish citizens’ levels of political knowledge. A face
to face survey was carried out in December 2012 on a representative sample of the
Spanish population (n= 2,962), in which we experimented with some of the items
intended to measure political knowledge. The sample was divided into two
representative sub-samples of Spanish electors (age from 18 onwards, n= 1,481 each
sub-sample).
1
As Table A.1 in the Appendix shows, the mean value of the antecedents
of political knowledge (age, education, gender, political interest and sex of the
interviewer) is almost identical for the two samples. This provides the experiments with
a high level of external validity since the results can be generalized across Spanish
electors. The first sample (questionnaire A) was used as a control group, whereas the
second sample (questionnaire B) was exposed to experimental manipulation. We
experimented with five political knowledge items.
2
As for the closed ended vs. open ended format, we experimented with two items.
The first asks for the year the Spanish constitution was signed; and the second relates to
the political organization of Spanish territory. The exact same wording was used in the
two questionnaires, but the open ended format was used in questionnaire A, while
closed ended items were introduced in questionnaire B (see the Appendix for the
specific wording, questions Q12 and Q24 respectively).
Regarding the protocol strategy to introduce the DK option, we experimented
with three close ended items. The first asks about the main function of the Spanish
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 9
Parliament (Q9), the second about the Euribor (the interest rate at which banks lend
money, Q21); and the third about the place where Spaniards can obtain the health card
(Q26). In questionnaire A, respondents were given neither incentive nor disincentive to
answer DK (DK neutral design). More specifically, they were provided with a card
containing the response categories, which did not show the options DK/NA. The
interviewer codified as DK all spontaneous DK responses. In questionnaire B,
respondents were presented with the exact same questions and cards. However, in the
case that they spontaneously answered DK, the interviewer tried to discourage the DK
with a follow-up question (“Even if you are not certain, I would like you to give me
your best guess”). The experimental design is particularly conservative in that we use a
DK neutral protocol instead of a DK encouraging protocol (encouraging respondents to
answer DK if they are not totally sure about the answer) to compare with a DK
discouraging protocol. We contend that under this demanding experimental setting, any
significant differences will allow for reliable conclusions.
The experiments were conducted for a number of items that differed
significantly in content: from basic political facts (the year the constitution was signed)
and economic figures (the question on the Euribor), to more practical matters (where to
get the health card). Such a variation allows us to control for the potential effect of the
content (as previously discussed, see Dolan 2011; Stolle and Gidengil 2010) in our
experiment.
Finally, regarding the level of difficulty of the items, we have used a different
logic for each of the two experiments. First, in the format (open vs. closed) experiment,
we have kept difficulty of the two items used at a similar level. These two items can be
considered to have a relatively high level of difficulty (the question on the year the
Spanish constitution was signed–Q12; and the item on the political organization of
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 10
Spanish territory–Q24).
3
As previously discussed, the main reason to expect an increase
in the percentage of correct answers for the closed format (as compared to the open one)
is that the closed format makes the response (and the possibility of guessing) easier for
the interviewee. For this reason, we choose relatively difficult items and kept constant
this level of difficulty for the two items.
In contrast, for the DK protocol experiment we included one item of
medium/high level of difficulty (the one about the Euribor–Q21),
4
another item of
medium/low level of difficulty (the one about the main function of the Spanish
Parliament–Q9),
5
and the last item with a low level of difficulty (the one about the
place where Spaniards can obtain the health card–Q26).
6
We kept constant the format of
the three items (closed ended) and varied their level of difficulty to test if the expected
difference in propensity for men and women to guess - boosted by the DK discouraging
protocol - is constant across diverse levels of difficulty.
Results
Starting with the first experiment, Table 1 shows percentages of DK, correct,
and incorrect answers of open ended (column 2) vs. closed ended (column 3) items.
Column 4 presents the differences in percentages of DK, correct, and incorrect answers
between the two types of formats (questionnaires).
[Table 1 about here]
Table 1 shows that the percentage of DK answers is higher in open ended
questions. And vice versa, the percentage of correct answers is higher in closed ended
questions. In line with Luskin and Bullock (2011), our evidence shows that open ended
items produce more conservative estimations of political knowledge, while increasing
the propensity of respondents to answer DK if they are not completely sure about the
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 11
answer. Closed ended items, on the other hand, encourage respondents to give an
answer, or to guess.
Table 2 presents similar results for our second experiment: the DK neutral
design (column 2) vs. the DK discouraging design (column 3). Again, column 4 shows
the differences in percentages of DK, correct, and incorrect answers between the two
questionnaires.
[Table 2 about here]
There are significant differences between the DK neutral design and the DK
discouraging design. Clearly, DK is considerably reduced, when respondents are
discouraged to choose this option (questionnaire B). Logically, the magnitude of the
decrease in the percentage of DK is smaller for the case of the health care item (Q26),
which is the easiest one to correctly respond to (and the one that initially presents the
lowest percentage of DK responses).
On the contrary, both correct and incorrect answers are increased if DK is
discouraged, in two items (function of the parliament and Euribor, respectively Q9 and
Q21). The magnitude of the increase in the percentage of correct answers, however, is
smaller than that of the increase in the percentage of incorrect answers, when the DK
discouraging design is used. This might be an indication that discouraging DK does not
in fact favor the emergence of hidden knowledge, but simply increases the propensity of
all respondents to guess.
Now that the descriptive evidence has been provided, we turn to the results of
the test of the two previously mentioned competing hypotheses on the effect of question
format and DK protocol for men and women. Our general expectation is that DK has a
different meaning for men and women – real DK vs. hidden knowledge respectively
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 12
(H1). The alternative hypothesis instead states that DK really means don’t know both
for men and women (H2).
Testing H1 and H2 requires us to estimate the extent to which the effect of each
treatment (the different DK protocol or the open vs. closed ended format followed in the
two versions of the questionnaire) on the propensity to give a DK, correct, and incorrect
answer differs for men and women. In this regard, we have created a categorical
variable for each of our experiments in which DK answers take value 1, correct
answers value 2, and incorrect answers value 3.
We have estimated the probabilities of providing a DK vs. correct vs. incorrect
answer through a multinomial logit equation with robust standard errors for each of the
five items. There are three main independent variables: treatment (closed ended vs.
open ended in Table 3, DK discouraging vs. DK neutral in Table 4); gender; and the
interaction between treatment and gender. The interaction term tests whether the
magnitude of the treatment effect is different for women and men. Previous tests
controlled for the conventional antecedents of political knowledge (abilities, motivation
and resources; see for instance Delli Carpini & Keeter, 1995): education, age, and
political interest, plus the sex of the interviewer, as well as interviewers’ evaluations of
the extent to which respondents answered political knowledge questions at random. The
results regarding the treatment effect (that is, the effect of the type of questionnaire: A
vs. B) show the same pattern, thus we have decided to present the results of the most
simple estimation.
We proceed as follows: we first estimate the probabilities of providing a DK vs.
correct vs. incorrect answer as a function of gender and the treatment for each of the
five dependent variables (that is, the five knowledge items considered here), and we
then specify the interaction between treatment and gender in a second step. The
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 13
empirical expectation from H1 is that in closed ended and DK discouraging items (the
treatment here) women should increase the percentage of correct vs. DK answers more
than men; whereas men should increase the percentage of incorrect vs. DK answers
more than women. Hence we are particularly interested in the contrast between correct
and DK answers; and between incorrect and DK answers. For this reason, we use DK as
the baseline category in the estimations. The results of these estimations are
summarized in Tables 3 and 4.
Does one type of format/ protocol tend to increase the propensity of women and
men to provide a correct or incorrect answer instead of choosing the DK option? A first
glance at Tables 3 and 4 suggests that only the DK protocol treatment have a
differentiated statistically significant effect for men and women and for one single item
– the Euribor question (Q21, see the coefficient of the interaction term in column 9 of
Table 4).
[Tables 3 and 4 about here]
A correct assessment of the impact and significance of these interactions,
however, cannot be completed through the direct interpretation of the multiplicative
terms. This would require not only the inclusion of both constitutive terms in the
estimation equation (as we do), but also the calculus of how the average marginal effect
of the treatment varies for men and women. We follow Brambor, Clark and Golder
(2006) and plot the change in average marginal effects of the treatment (responding to
the closed versus the open item) on the expected probabilities of providing correct,
incorrect and DK answers for men and women (see Figures 1 and 2). The confidence
intervals around the dots indicate whether the change in probabilities of responding to
each format of the questions is significant (at p<0.05) for men and women separately.
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 14
Figures 1 and 2 confirm the descriptive evidence previously commented upon,
that the closed ended format substantially increases the propensity of choosing the
correct option instead of DK; whereas incorrect answers are much less increased for the
two survey items analyzed here (compare the graph in the middle with the graph on the
right for each of the two figures). Since it is unlikely that random guessing would only
increase correct answers, it is an indication that multiple-choice questions in surveys
might provide a more accurate measurement of political knowledge. However, is this
treatment effect of an equal magnitude for women and men? The figures suggest an
affirmative answer since the gender differences appear to be of a small magnitude.
7
The
evidence provided then appears to confirm H2 as regards the format of the question.
[Figures 1 and 2 about here]
Regarding the DK protocol experiments, the treatment increases the propensity
to provide an incorrect answer more than it increases the propensity to provide a correct
answer. This suggests that the DK discouraging protocol might bias the estimation of
levels of political knowledge, since it seems to stimulate guessing.
However, is this effect of an equal magnitude for women and men? Figures 3 to
5 indicate that the effect of the DK discouraging protocol on the expected probability
that respondents provide a correct answer is larger for women than for men. That is,
women provide 4% and 5% more correct answers in questionnaire B than in
questionnaire A; compared to 2% and 2% of men) for the question on the function of
the parliament (Q9) and the Euribor (Q21) respectively (see the middle of the graphs in
Figures 3 and 4). In Q26-Health card, the percentage of correct answers decreases for
both men and women but the change is larger for men (2%) than for women (0.5%, see
the middle of the graph in Figure 5).
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 15
When it comes to incorrect answers, H1 suggests that the DK discouraging
protocol will induce a larger increase in the percentage of incorrect questions for men
than for women. This is confirmed by the findings for the three items analyzed here (see
right hand graphs in Figures 3 to 5). Take for example the item about the Euribor. The
DK discouraging protocol increases the expected probability of providing an incorrect
answer by 10% for men and 5% for women, which implies a 5% gender difference (see
Figure 4). Following the same pattern, the increase in the expected probabilities of
providing an incorrect answer for the items related to parliament and the health card
provision are larger for men (6 and 4%) than for women (5 and 1% respectively).
Although the gender differences are of a relatively small magnitude we trust the results
since the experimental design we are using here is particularly conservative. In fact we
compare a DK neutral protocol (instead of a DK encouraging protocol) with a DK
discouraging one.
[Figures 3, 4 and 5 about here]
These findings suggest that perhaps women and men make different use of the
DK option. This implies not only that women tend to use DK more than men; but also
that “DK really means DK” to a greater extent for men than for women. In other words,
women might use DK even if they have some partial knowledge about a topic; whereas
men would tend to use DK only if they ignore the correct answer. In the following lines
we discuss the implication of these findings for the validity of conventional measures of
political knowledge more frequently used in previous studies.
Discussion and Concluding Remarks
Recent studies argue that a number of methodological issues could potentially
question the validity of political knowledge measures. Among these, some of the
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 16
problems refer to the format and protocol of the items used in surveys. This article
analyzes the extent to which the format and the DK protocol of survey questions
influence part of the gender differences identified in previous studies. What is more, we
test the possibility that the use and meaning of the DK option in survey knowledge
items might be gender biased. Something that to the best of our knowledge has not been
previously analyzed.
We use a set of experiments that manipulated the DK protocol and the format
(open vs. closed) of political knowledge questions in a face to face representative
survey of the Spanish population, a rarely studied case. While previous studies have
examined these topics, very few have used an experimental design by maintaining equal
question content, while randomly changing the format. And the sole exception (Robison,
2015) paid no attention to the potential interaction of question format and gender.
Findings suggest that the way political knowledge items are designed can
potentially affect not only observed levels of knowledge but also the magnitude of the
gender gap, although with some restrictions. First, results of the format experiment
show that multiple choice questions in surveys provide a more accurate measurement of
political knowledge than their open ended counterpart, confirming previous studies also
for the case of Spain (Luskin & Bullock, 2011; Robison, 2015). This evidence appears
to be of the same substantive magnitude for both men and women, thereby suggesting
that the use and meaning of DK in multiple choice survey questions is equal for men
and women, since responding to this type of question requires less effort from the
interviewee.
Second, women say DK more often than men also in Spain. In line with previous
studies (Mondak & Anderson, 2004), we interpret this finding as a product of women’s
higher risk aversion since the greater propensity of females to respond DK compared to
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 17
their male counterparts is clear, independently of the format or the DK protocol.
However, we find an interesting qualification to this evidence. Although the DK
discouraging protocol might bias the estimation of levels of political knowledge, since it
seems to stimulate guessing, we also find that this evidence appears to be of a greater
magnitude for males, compared to their female counterparts. Moreover the DK
discouraging protocol fosters a larger increase in the percentage of incorrect questions
for men than for women and a greater increase in the percentage of correct questions for
women than for men.
Despite the fact that the gender differences found in the use of the DK option
under the DK discouraging protocol are of a relatively small magnitude, it is important
to remember that these differences arise under a very demanding experimental setting,
which make the differences especially reliable. A last derivation from these findings is
that the use (and perhaps the meaning) of the DK option might be different for women
and men. Women appear to be more reluctant to provide an answer, even if they have
some partial knowledge about a specific topic. On the contrary, men seem to answer
DK when they ignore the answer to the question, suggesting that men’s initial DK
responses really mean that: “they do not know”.
The practical implications of the analyses provided here is that when designing
political knowledge questionnaires, multiple choice and DK neutral protocol are
preferable because they provide a more accurate estimation of respondents’ true levels
of political knowledge. However this recommendation goes with some qualifications.
Question content (and their consequential levels of difficulty) appear to also matter. We
wonder whether the format and DK protocol driven effects observed here also translate
to other knowledge items that include different content (and diverse levels of difficulty).
In order to address this question, further experimental manipulation of format and
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 18
(especially) DK protocols with additional knowledge items might be the best way
forward to build upon the findings presented here.
A final remark regards the potential effects of the DK protocol on other
respondents. For instant, to what extent could DK also be biased for other
disadvantaged groups, causing an underestimation of their levels of political
knowledge? Previous empirical tests of our experiments suggest that the format or DK
protocol do not appear to have an effect on levels of knowledge of other disadvantaged
groups (such as the less educated or the unemployed). However, we do find that they
are also more willing to provide an answer when discouraged to answer DK. Further in
depth analysis would be needed in this regard and might be the subject of future
research.
References
Boudreau, Ch. & Lupia, A. (2011). Political knowledge. In J. N. Druckman, D. P.
Green, J. H. Kuklinski, & A. Lupia (eds.), Cambridge handbook of experimental
political science (pp. 171-183). Cambridge: Cambridge University Press.
Brambor, T.; Clark, W. and Golder, M. (2006). Understanding Interaction Models:
Improving Empirical Analyses. Political Analysis, 14(1), 63-82. doi:
10.1093/pan/mpi014.
Burns, N., Schlozman, K. L. & Verba, S. (2001). The private roots of public action.
Cambridge. Cambridge, MA: Harvard University Press
Converse, P. (1964) The Nature of Belief Systems in Mass Publics. In: D. Apter (ed.)
Ideology and Discontent (pp. 206-261) New York: Free Press.
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 19
Converse, P. (1970) Attitudes and Non Attitudes: Continuation of a Dialogue. In: E.
Tufte (ed.) The Quantitative Analysis of Social Problems (pp. 168-189).
Reading, (Mass.): Addison-Wesley.
Delli Carpini, M.X. & Keeter, S. (1996). What Americans know about politics and why
it matters. New Haven: Yale University Press.
Dolan, K. (2011). Do women and men know different things? Measuring gender
differences in political knowledge. The Journal of Politics, 73(1), 97–107.
doi:10.1017/S0022381610000897.
Fraile, M. (2014). Do women know less about politics than men? The gender gap in
political knowledge in Europe. Social Politics, 21(2), 261–289. doi:
10.1093/sp/jxu006
Frazer, E. & Macdonald, K. (2003). Sex differences in political knowledge in Britain.
Political Studies, 51(1), 67–83. doi: 10.1111/1467-9248.00413
Gibson, J. L., & Caldeira, G. A. (2009). Knowing the supreme court? A reconsideration
of public ignorance of the High Court. The Journal of Politics, 71(2), 429–441.
doi:10.1017/S0022381609090379.
Kenski, K., & Jamieson, K. H. (2000). The gap in political knowledge: Are women less
knowledgeable than men about politics? In K. H. Jamieson (ed.), Everything
you think you know about politics . . . And why you’re wrong (pp. 83–89). New
York: Basic Books.
Lizotte, M-K., & Sidman, A. H. (2009). Explaining the gender gap in political
knowledge. Politics & Gender, 5(2), 127–52.
doi:10.1017/S1743923X09000130.
Lupia, A. (2006). How elitism undermines the study of voter competence. Critical
Review, 18 (1), 217 –232 . doi:10 .1080 /08913810608443658 .
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 20
Lupia, A. & McCubbins, M. D. (1998). The democratic dilemma: Can citizens learn
what they need to know? New York: Cambridge University Press.
Luskin, R., & Bullock, J. G. (2011). ‘Don’t know’ means ‘don’t know’: DK responses
and the public’s level of political knowledge. The Journal of Politics, 73(2),
547–557. doi:10.1017/S0022381611000132.
Miller, M. K., & Orr, S. K. (2008). Experimenting with a ‘‘Third Way’’ in political
knowledge estimation. Public Opinion Quarterly, 72 (4), 768 –780 . doi:10
.1093 /poq/nfn057.
Mondak, J. J. (2001). Developing valid knowledge scales. American Journal of
Political Science, 45(1), 224–238. doi: 10.2307/2669369
Mondak, J.J., & Anderson, M. R. (2004). The knowledge gap: A reexamination of
gender-based differences in political knowledge. Journal of Politics, 66(2), 492–
512. doi: 10.1111/j.1468-2508.2004.00161.x
Mondak, J. J., & Davis, B. C. (2002). Asked and answered: knowledge levels when we
will not take “Do not know” for an answer. Political Behavior, 23(3), 199–224.
doi: 10.1023/A:1015015227594.
Prior, M., & Lupia, A. (2008). Money, time, and political knowledge: Distinguishing
quick recall and political learning skills. American Journal of Political Science,
52 (1), 169 –183 . doi:10 .1111 /j.1540 -5907 .2007 .00306 .x
Prior, M. (2014). Visual knowledge: A different road to competence? The Journal of
Politics, 76(1), 41-57. doi:10.1017/S0022381613001096
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 21
Robison, J. (2015). Who knows? Question format and political knowledge.
International Journal of Public Opinion Research, 27(1), 1-21.
doi: 10.1093/ijpor/edu019
Shaker, L. (2012). Local political knowledge and assessments of citizen competence.
Public Opinion Quarterly 76(3), 525-537. doi: 10.1093/poq/nfs018
Stolle, D. & Gidengil E. (2010). What do women really know? A gendered analysis of
varieties of political knowledge. Perspectives on Politics 8(1), 93–109.
doi: http://dx.doi.org/10.1017/S1537592709992684
Sturgis, P., Allum, N., & Smith, P. (2008). An experiment on the measurement of
political knowledge in surveys. Public Opinion Quarterly, 72(1), 90–102. doi:
10.1093/poq/nfm032
Verba, S., Burns, N., & Schlozman, K.L. (1997). Knowing and caring about politics:
Gender and political engagement. Journal of Politics 59(4), 1051–1072.
doi:10.2307/2998592.
Footnotes
1
Random-strata sampling. The sampling error (at 95.5% level) was ± 1.8. More
technical details about the sample design can be found at www.cis.es.
2
We introduced up to 31 different items intended to measure political
knowledge. 9 items reproduce classical factual questions used in previous studies
(related to electoral politics); 18 items aim at measuring knowledge about the
functioning of democratic institutions (11) and specific policies (7); finally, 4 items
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 22
measure knowledge about economic institutions and processes. From all 31 items we
could experiment only with 5 (due to budget restrictions).
3
The percentage of correct answers to these questions varies between 34% and
49% of total respondents (rounded percentages; see Table 1).
4
The percentage of correct answers to this question varies between 47% and
51% of total respondents (rounded percentages; see Table 2).
5
The percentage of correct answers to this question varies between 69% and
72% of total respondents (rounded percentages; see Table 2).
6
The percentage of correct answers to these questions varies between 78% and
80% of total respondents (rounded percentages; see Table 2).
7
For example, the increase in the percentage of correct answers is slightly larger
for men than for women in Q12-Constitution (men provide 13% more correct answers
in questionnaire B than in questionnaire A; compared to 11% of women); whereas
women increase the percentage of correct answers more than men in Q24-AACC (men
provide 6% more correct answers in questionnaire B than in questionnaire A; compared
to 9% of women, see Figure 2). These slight gender differences however are not
statistically different from zero (see the overlapping confidence interval).
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 23
23
List of Tables and Figures
Table 1
Percentage of DK, Correct and Incorrect responses in the two formats: Open ended vs.
closed ended
Open ended
(questionnaire A)
Closed ended
(questionnaire B)
Difference
(questionnaire A-B)
Q12. Year of constitution
DK
40.92
32.21
8.71***
Correct
36.93
49.29
-12.36***
Incorrect
22.15
18.50
3.65**
Q24. Number AACC
DK
38.22
27.01
11.21***
Correct
34.57
42.67
-8.10***
Incorrect
27.22
30.32
-3.10**
Note: Our elaboration of the data CIS2973. *p<.10; **p<.05;***p<.01
Table 2
Percentage of DK, Correct and Incorrect responses in the two formats: DK Neutral vs.
DK discouraging (%)
DK neutral
(questionnaire
A)
DK discouraging
(questionnaire
B)
Difference
(questionnaire A-
B)
Q9. Function of parliament
DK
14.04
6.08
7.96***
Correct
68.87
71.64
-2.77*
Incorrect
17.08
22.28
-5.2***
Q21. Euribor
DK
32.68
21.54
11.14***
Correct
47.13
50.98
-3.85*
Incorrect
20.19
27.48
-7.29***
Q26. Health Card
DK
3.04
1.96
1.08*
Correct
79.74
78.39
1.35
Incorrect
17.22
19.65
-2.43*
Note: Our elaboration of the data CIS2973. *p<.10; **p<.05;***p<.01
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 24
24
Table 3.
Probabilities of Providing a Correct vs. DK answer and an Incorrect vs. DK answer
Q12: Constitution
Q24: AACC
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Treatment a
0.529***
0.0447
0.552***
-0.0297
0.548***
0.456***
0.491***
0.430***
(0.0858)
(0.105)
(0.123)
(0.151)
(0.0911)
(0.0974)
(0.133)
(0.143)
Female
-0.513***
-0.355***
-0.482***
-0.422***
-0.696***
-0.532***
-0.749***
-0.547***
(0.0857)
(0.104)
(0.121)
(0.141)
(0.0912)
(0.0975)
(0.127)
(0.135)
Treatment*Female
-0.0537
0.148
0.110
0.0400
(0.172)
(0.210)
(0.183)
(0.196)
Constant
0.113
-0.428***
0.0986
-0.395***
0.216***
-0.103
0.242***
-0.0927
(0.0743)
(0.0889)
(0.0864)
(0.0998)
(0.0785)
(0.0839)
(0.0903)
(0.0972)
Observations
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
Note: Our elaboration of the data CIS2973. Multinomial estimations with robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1
a Open ended=0; Closed ended=1
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 25
25
Table 4.
Probabilities of Providing a Correct vs. DK answer and an Incorrect vs. DK answer
Q9: Parliament Function
Q21: Euribor
Q26: Health Card
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Correct
Incorrect
Treatment a
0.897***
1.130***
1.014***
1.322***
0.460***
0.707***
0.595***
1.015***
0.439*
0.591**
0.493
0.717**
(0.137)
(0.155)
(0.207)
(0.233)
(0.0920)
(0.108)
(0.140)
(0.165)
(0.248)
(0.259)
(0.309)
(0.326)
Female
-0.338***
-0.176
-0.281*
-0.0409
-0.741***
-0.419***
-0.662***
-0.148
0.761***
0.554**
0.826**
0.697**
(0.128)
(0.147)
(0.156)
(0.192)
(0.0919)
(0.108)
(0.122)
(0.152)
(0.256)
(0.267)
(0.335)
(0.354)
Treatment
-0.198
-0.337
-0.208
-0.540**
-0.169
-0.318
*Female
(0.276)
(0.312)
(0.187)
(0.218)
(0.519)
(0.542)
Constant
1.677***
0.220*
1.644***
0.145
0.701***
-0.307***
0.651***
-0.458***
2.936***
1.476***
2.917***
1.419***
(0.105)
(0.125)
(0.113)
(0.141)
(0.0796)
(0.0950)
(0.0882)
(0.114)
(0.180)
(0.191)
(0.189)
(0.204)
Observations
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
2,962
Note: Our elaboration of the data CIS2973. Multinomial estimations with robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1
a DK neutral=0; DK discouraging=1
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 26
26
FIGURE 1. Contrast of marginal effects of the closed ended question (versus open
ended question) format on the probabilities of giving a DK, correct and incorrect answer
for men and women – year Constitution (Q12)
Note: Calculation on the basis of Table 3 (columns 4 and 5)
FIGURE 2. Contrast of marginal effects of the closed ended question (versus open
ended question) format on the probabilities of giving a DK, correct and incorrect answer
for men and women – number of Autonomous Communities (Q24)
Note: Calculation on the basis of Table 3 (columns 8 and 9)
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 27
27
FIGURE 3. Contrast of marginal effects of the DK discouraging (versus the DKneutral)
protocol on the probabilities of giving a DK, correct and incorrect answer for men and
women – Function of the parliament (Q9)
Note: Calculation on the basis of Table 4 (columns 4 and 5)
FIGURE 4. Contrast of marginal effects of the DK discouraging (versus the DK neutral)
protocol on the probabilities of giving a DK, correct and incorrect answer for men and
women – Euribor (Q21)
Note: Calculation on the basis of Table 4 (columns 8 and 9)
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 28
28
FIGURE 5. Contrast of marginal effects of the DK discouraging (versus the DK neutral)
protocol on the probabilities of giving a DK, correct and incorrect answer for men and
women – Health Card (Q26)
Note: Calculation on the basis of Table 4 (columns 12 and 13)
APPENDIX
Questions wording: Open vs. closed ended experiment
Q.12 Do you remember the year when the Spanish Constitution was passed?
Questionnaire A
Questionnaire B
(WRITE THE YEAR)
- 1973 ………..…1
- 1975 …….….…2
- 1978 ……….….3
- 1979 …….…….4
Q.24 Do you remember how many Autonomous Communities there are in Spain? I’m referring
to Autonomous Communities, not to provinces.
Questionnaire A
Questionnaire B
(WRITE THE YEAR)
- 13 ………. 1
- 16 ………. 2
- 17 ………. 3
- 19 ………. 4
- 21 ………. 5
Questions wording: DK protocol experiment
Q.9 From the following, which is the main function of the Spanish Parliament?
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 29
29
Questionnaire A
Questionnaire B
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
-DK (not shown, used when
the respondents answer ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.9b Even if you are not certain, I would like
you to give me your best guess:
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
Q.21 Could you tell me whether the value of the Euribor has increased, stayed the same or
decreased since January 2012?
Questionnaire A
Questionnaire B
It has increased………………….1
It has stayed the same…………...2
It has decreased………………….3
It has increased………………….1
It has stayed the same…………...2
It has decreased………………….3
-DK (not shown, used when
the respondents answers ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.21b Even if you are not certain, I would
like you to give me your best guess:
It has increased…………………….1
It has stayed the same……………...2
It has decreased……………….……3
Q.26 Do you know where you need to go to obtain the health card?
Questionnaire A
Questionnaire B
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
-DK (not shown, used when
the respondents answers ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.26b Even if you are not certain, I would
like you to give me your best guess:
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
Table A.1. The antecedents of Political Knowledge across the two samples
Antecedents
Sample 1
(Questionnaire A)
Sample 2
(Questionnaire B)
p value of
t- test
Age (18-95)
44,13
44,41
p=0.65
Education (0-15)
5.57
5.52
p=0.62
Gender (1= female)
0.50
0.50
P=0.88
Political Interest (0-3)
1.12
1.14
p=0.45
Interviewers’ sex (1= women)
0.61
0.60
p=0.54
Note: Our elaboration of the data CIS2973
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 30
30
APPENDIX
Questions wording: Open vs. closed ended experiment
Q.12 Do you remember the year when the Spanish Constitution was passed?
Questionnaire A
Questionnaire B
(WRITE THE YEAR)
- 1973 ………..…1
- 1975 …….….…2
- 1978 ……….….3
- 1979 …….…….4
Q.24 Do you remember how many Autonomous Communities there are in Spain? I’m referring
to Autonomous Communities, not to provinces.
Questionnaire A
Questionnaire B
(WRITE THE YEAR)
- 13 ………. 1
- 16 ………. 2
- 17 ………. 3
- 19 ………. 4
- 21 ………. 5
Questions wording: DK protocol experiment
Q.9 From the following, which is the main function of the Spanish Parliament?
Questionnaire A
Questionnaire B
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
-DK (not shown, used when
the respondents answer ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.9b Even if you are not certain, I would like
you to give me your best guess:
- To enforce the law and arrest criminals …1
- To debate and pass laws………………….2
- To control the courts of justice…………...3
- To ensure the stability of laws …………...4
Q.21 Could you tell me whether the value of the Euribor has increased, stayed the same or
decreased since January 2012?
Questionnaire A
Questionnaire B
It has increased………………….1
It has stayed the same…………...2
It has decreased………………….3
It has increased………………….1
It has stayed the same…………...2
It has decreased………………….3
-DK (not shown, used when
the respondents answers ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.21b Even if you are not certain, I would
like you to give me your best guess:
It has increased…………………….1
It has stayed the same……………...2
It has decreased……………….……3
GUESSING AND THE GENDER GAP IN POLITICAL KNOWLEDGE 31
31
Q.26 Do you know where you need to go to obtain the health card?
Questionnaire A
Questionnaire B
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
-DK (not shown, used when
the respondents answers ………………….5
DK spontaneously)
(IF INTERVIEWEE ANSWERS DK)
Q.26b Even if you are not certain, I would
like you to give me your best guess:
To the city council …………………………1
To the health center…………........................2
To the regional Health office ………………3
To the citizens’ information local bureau ….4
Table A.1. The antecedents of Political Knowledge across the two samples
Antecedents
Sample 1
(Questionnaire A)
Sample 2
(Questionnaire B)
p value of
t- test
Age (18-95)
44,13
44,41
p=0.65
Education (0-15)
5.57
5.52
p=0.62
Gender (1= female)
0.50
0.50
P=0.88
Political Interest (0-3)
1.12
1.14
p=0.45
Interviewers’ sex (1= women)
0.61
0.60
p=0.54
Note: Our elaboration of the data CIS2973