Women catch up: gender differences in learning programming concepts.
ABSTRACT This paper describes a multi-institutional study that used categorization exercises (known as constrained card sorts) to investigate gender differences in graduating computer science students' learning and perceptions of programming concepts. Our results show that female subjects had significantly less pre-college programming experience than their male counterparts. However, for both males and females, we found no correlation between previous experience and success in the major, as measured by computer science grade point average at graduation. Data also indicated that, by the time students completed their introductory courses, females reported nearly equal levels of mastery as males of the programming concepts. Furthermore, females generally considered the programming concepts to be no more difficult than did the men.
- SourceAvailable from: José Paulo Leal[Show abstract] [Hide abstract]
ABSTRACT: Several Web-based on-line judges or on-line programming trainers have been developed in order to allow students to train their programming skills. However, their pedagogical functionalities in the learning of programming have not been clearly defined. EduJudge is a project which aims to integrate the “UVA On-line Judge”, an existing on-line programming trainer with an important number of problems and users, into an effective educational environment consisting of the e-learning platform Moodle and the competitive learning tool QUESTOURnament. The result is the EduJudge system which allows teachers to apply different pedagogical approaches using a proven e-learning platform, makes problems easy to search through an effective search engine, and provides an automated evaluation of the solutions submitted to these problems. The final objective is to provide new learning strategies to motivate students and present programming as an easy and attractive challenge. EduJudge has been tried and tested in three algorithms and programming courses in three different Engineering degrees. The students’ motivation and satisfaction levels were analysed alongside the effects of the EduJudge system on students’ academic outcomes. Results indicate that both students and teachers found that among other multiple benefits the EduJudge system facilitates the learning process. Furthermore, the experiment also showed an improvement in students’ academic outcomes. It must be noted that the students’ level of satisfaction did not depend on their computer skills or their gender.Computers & Education. 01/2012; 58:1-10.
Conference Paper: Accessing IT: a curricular approach for girls[Show abstract] [Hide abstract]
ABSTRACT: Learning settings, methods as well as the lessons' contents in schools in Germany have undergone significant changes over the years, developing towards a more gender oriented way of teaching. Yet, the significant gap between girls' and boys' participation in computer science remains. Based on a qualitative study among female teenage students and their teachers in Germany, we have developed and put into practice a curriculum taking the girls' assessment of and access to both computer science and professional activity in the IT domain into account.Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design; 10/2012
Women Catch Up: Gender Differences in Learning
Pacific Lutheran University
University of Puget Sound
This paper describes a multi-institutional study that used
categorization exercises (known as constrained card sorts) to
investigate gender differences in graduating computer science
students’ learning and perceptions of programming concepts. Our
results show that female subjects had significantly less pre-college
programming experience than their male counterparts. However,
for both males and females, we found no correlation between
previous experience and success in the major, as measured by
computer science grade point average at graduation. Data also
indicated that, by the time students completed their introductory
courses, females reported nearly equal levels of mastery as males
of the programming concepts. Furthermore, females generally
considered the programming concepts to be no more difficult than
did the men.
College of Charleston
Briana B. Morrison
Southern Polytechnic State Univ
University of Arizona
SUNY at Potsdam
Categories and Subject Descriptors
K.3.2 [Computers & Education]: Computer & Information
Science Education – Computer Science Education.
card sort, gender differences, programming experience
It is a well-known phenomenon that women students come to
introductory computer science (CS) classes with less pre-college
programming experience than do men (e.g., [5, 7, 11, 13]). There
is also considerable research suggesting that women students have
less confidence in their computing abilities than their male peers
[1, 7, 13]. Data on experience and confidence have typically been
obtained through student self-reports, including survey questions
that ask students to “measure pre-college experience on a scale
from 1 to 7”  or respond to statements such as “I have studied
computer science in school” using a Likert-type scale . Very
few studies have asked students to rate their familiarity with
specific concepts (e.g., selection, loop, procedure, arrays and
pointers in ).
Previous studies investigating gender differences in experience
and confidence levels among students have focused primarily on
students in introductory-level computer science or non-major
courses at a single institution (e.g., [2, 5, 13, 15]). This study was
unique in that it focused on students who have persisted and
succeeded in the major, those graduating with baccalaureate
degrees in CS. It was also multi-institutional, involving 73
students from eight colleges and universities across the United
States. Furthermore, the primary methodology used to elicit data
did not ask students to self-report or subjectively rate their general
knowledge or experience, but rather required that they
characterize their learning and perceptions of 26 specific
programming concepts using a categorization technique adopted
from knowledge acquisition research. Subjects were required to
sort the concepts using prescribed criteria and category names.
This technique is useful in measuring agreement between subjects
. The prescribed criteria focused on when subjects first
learned specific programming concepts, when they mastered those
concepts, and how difficult they had been to learn. This was done
within the context of a study  of students’ general knowledge
of programming concepts and was not specifically designed to
investigate gender differences.
Asking students to identify when they were introduced to or
mastered specific concepts or how difficult they perceived the
concepts to be may have several advantages over more subjective
self-ratings. First, this technique is likely to result in a more
precise assessment of experience. For example, a student who
mastered concepts such as objects and recursion “before college”
comes to an introductory college level computer science class
with a more sophisticated background than one who has not, even
if both students have taken a pre-college programming course.
Comparing which and how many concepts are assigned to the
“before college” category by men and women gives a more
accurate view of how pre-college computing experiences may
differ for these two groups. Secondly, such an approach may be
less susceptible to gender bias caused by male students’ general
tendency to rate their own abilities more highly than equally able
female students, as well as the tendency for both men and women
to rate men’s abilities higher than women’s . The premise is
that even a student with little confidence will find it difficult to
substantially underestimate his or her knowledge of specific
concepts, such as if-then-else or encapsulation. Furthermore,
focusing on students’ perceptions of programming concepts,
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
SIGCSE’06, March 1–5, 2006, Houston, Texas, USA.
Copyright 2006 ACM 1-59593-259-3/06/0003…$5.00.
rather than their abilities compared to those of others, and asking
the questions in a gender-neutral manner, may be less likely to
bias their responses. A study by Spencer et al.  investigating
stereotype threat showed that the mere suggestion that a math test
has gender differences can cause women to perform worse and
men to perform better on the test. This suggests the possibility
that surveys including questions such as “In general, men are
better than women at programming”  might influence the
accuracy of men’s and women’s self-assessments of knowledge or
ability. The gender-neutral protocol used in the study reported
here is likely to result in more accurate results in terms of gender.
We investigated gender differences by examining graduating
students’ experience and knowledge of programming concepts.
This research project is an adaptation of an earlier multi-national
study  that employed the same concept categorization exercise
to investigate the knowledge structures of 243 novice
programmers and 33 educators at 22 universities. The novices’
understanding of programming concepts was elicited through a
knowledge-acquisition technique called a repeated single-
criterion card sort . While the novice study showed little
difference between sorts produced by male and female students or
between students with very little or considerable programming
experience, it was useful in focusing the study of graduating
students reported herein. Specifically, the criteria and categories
chosen for the constrained sorts, discussed in this paper, were
selected based on data observed during analysis of the data from
the study of novices.
2. STUDY METHODOLOGY
Interviews were conducted with graduating CS students. The
interviews consisted of two types of categorization tasks requiring
subjects to sort programming concepts into categories based on a
single criterion. In the first type, subjects were asked to articulate
their own criteria and category names. There is evidence to
suggest that the way subjects categorize concepts reflects their
internal representation of those concepts . We refer to these
sorts as unconstrained sorts.1 The second type, discussed herein,
asked subjects to perform constrained sorts in which the criteria
and category names were provided.
During all card sorts subjects categorized the same set of 26 index
cards, each containing a prompt for a programming concept. (See
Table 1) These concepts ranged from specific programming
entities, such as if-then-else and variable, to more abstract
concepts, such as decomposition and encapsulation. However, all
were general in nature and not limited to a particular language
syntax or programming task.
Table 1: Stimuli used in card sorts
1 See  and  for analysis of the unconstrained sorts
2.1 Constrained Sorts
Subjects were asked to perform four sorts using specific
categories and criteria provided by the interviewer. (See Table 2.)
Table 2: Constrained sort criteria and categories
For each criterion, subjects were asked to group the cards into
categories for the given criterion. For example, for the “When I
was first introduced to it” criterion, one subject provided the
following groupings of cards into categories:
Before college: <no cards>
Lower-level CS courses: dependency, object, abstraction,
scope, list, recursion, state, encapsulation
Upper-level CS courses: tree, thread, event
On the job: <no cards>
On my own: function, method, procedure, if-then-else,
boolean, choice, parameter, variable, constant, type,
loop, expression, iteration, array
Don’t know the term: decomposition
This paper focuses on the analysis by gender of data from three of
the four constrained sorts: “When I was first introduced to it”,
“When I mastered it”, and “Difficulty level”.
The subjects were 73 undergraduate CS students at eight colleges
and universities in the USA, including both private and public
institutions, ranging from small liberal arts colleges to large
research universities. All subjects were eligible to complete
baccalaureate degrees in CS at some time during the calendar year
in which they were interviewed. At each school, the study was
advertised and students volunteered to participate. An effort was
made to recruit female students and students with a range of
academic abilities. The recruitment techniques, research protocol
and interview procedures were approved by the Institutional
Review Boards at all institutions.
2.3 Data Collection
Data were collected during Spring 2004 and Spring 2005. Data
collection followed the standard protocol established for the
earlier study of novice programmers . All investigators had
participated in the novice study and were familiar with the sorting
and data collection procedures. The following data were collected
for each of the 73 subjects:
When I was first introduced to it
before college, lower-level CS courses, upper-level CS
courses, on the job, on my own, don’t know the term
When I mastered it
before college, lower-level CS courses, upper-level CS
courses, on the job, on my own, haven’t mastered it
yet, don’t know the term
procedural, functional, object-oriented, logic, not sure,
don’t know the term
easy, intermediate, advanced, don’t know the term
Demographic and background data included expected
graduation date, age, gender, first spoken language, first and
second programming languages, age when they began
programming, level of experience with a variety of
programming languages, overall grade point average (GPA2),
CS GPA, and grades in CS courses.
Constrained card-sort data included a record of the cards sorted
into each given category for each of the four constrained sort
criteria. (See Table 2.)
3. RESULTS AND DISCUSSION
3.1 Demographic variables
The subjects were 22 (30%) females and 51 (70%) males. While
the mean age of male and female subjects was nearly the same
(24.5 years for females and 24.0 years for males), on average the
females began programming at a later age (18.4 years of age) than
the males (16.5 years of age). This disparity in ages at which
males and females begin to program is similar to those reported in
other studies . For this group, the mean ages imply that on
average these females began programming in college, while males
began while still in high school.
Academic performance was nearly the same for both male (GPA =
3.28, CS GPA = 3.29) and female (GPA = 3.34, CS GPA = 3.32)
subjects. It should be noted here, that an effort was made to
recruit additional female subjects with high CS GPAs in Spring
2005 since they were not well represented in Spring 2004.
3.2 Difficulty Level Sorts
For the “Difficulty Level” constrained sorts, subjects assigned
each of the 26 programming concepts to one of the categories
labeled “easy,” “intermediate,” “advanced,” and “don’t know the
term.” The objective of the analysis of the difficulty level sorts
was to determine if there were differences in perceptions of
concept difficulty between male and female subjects. Table 3
2 GPA is a measure of academic achievement that represents a
weighted average of grades from courses taken. All institutions
in this study used a four-point grading scale where 4 is the
shows females designated slightly more concepts as “easy”
(females 54%, males 49%) and slightly fewer as “intermediate”
(females 25%, males 29%) than did the males. On average, male
subjects considered slightly more (18%) of the terms to be
“advanced” than female subjects did (15%). Both groups reported
knowing all but a very few of the terms.
Table 3: Mean concepts per difficulty category
Easy Intermediate Advanced Don’t Know
Females 14.0 6.5 4.0 1.5
Males 12.8 7.5 4.8 0.9
To quantify difficulty ratings for individual concepts each
category was assigned a numeric value (1=easy, 2=intermediate,
3=advanced, 4=don’t know) and the mean difficulty rating of each
concept was calculated for male and female subjects. Little
difference was observed in male and female subjects’ difficulty
ratings of the individual concepts, with both rating the concepts
decomposition, abstraction, dependency and thread as most
difficult and constant, boolean and variable as easiest. The mean
difficulty ratings for all 26 concepts by individual subjects was
nearly identical for males (N=51, M=1.76, SD=.249) and females
(N=22, M=1.73, SD=.341).
3.3 Introduction and Mastery Sorts
The categories for both the “When I was first introduced to it” and
“When I mastered it” sorts were “before college,” “introductory
CS classes,” “upper-level CS classes,” “on the job,” “on my own,”
and “don’t know.” The category “haven’t mastered yet” was also
used in the mastery sort. Figure 1 reveals the substantial
differences between males and females, with males being both
introduced to and mastering many more programming concepts
than females before college. Results of chi-square tests on the
number of concepts assigned to the “before college” category,
versus those assigned to one of the categories representing college
or after, indicated these differences were significant (p <.005) for
both the “When I was first introduced to it” and “When I mastered
Figure 1: Mean concepts per category for introduction and mastery sorts
Figure 2: Early mastery of concepts by gender
(differences in percentages of males and females mastering each
concept early are given under each concept name)
Interestingly, subjects of both genders reported learning very few
concepts on their own. This was surprising in light of previous
research suggesting that men are much more likely than women to
engage in self-initiated computer use [4, 7]. This result suggests
that self-initiated computer experiences may have little or no
impact on learning fundamental programming concepts.
Analysis of the individual concepts reveals that, while women on
average are introduced to and master far fewer concepts than men
before college, by the time they have taken introductory CS
classes they appear to “catch up” with the men. This is depicted in
Figure 2, which shows the cumulative percentage of subjects, by
gender, mastering each concept “before college” and in an
“introductory CS class”. The percentage of females mastering
concepts by the time they have completed their introductory
classes is at least as high as for males for 12 of the 26 concepts.
Additionally, the highest levels of early mastery reported by either
gender were reported by 95% of females for the concepts type,
variable, boolean and if-then-else.
To assess the impact of pre-college experience on success in the
major, students were placed into one of four groups based on their
level of pre-college programming experience as determined by the
number of concepts they were introduced to before college. Table
4 shows the percentage of students in each of the four groups and
compares CS GPAs for male and female subjects in each group.
Table 4: CS GPAs relative to pre-college experience
# of concepts
M F All M F All
33% 3.11 3.20 3.16
1 to 7
22% 18% 21% 3.30 3.55 3.37
8 to 14
33% 14% 27% 3.33 3.42 3.35
15 to 26
25% 5% 19% 3.34 3.82 3.38
The distribution of male and female subjects, in Table 4, was not
surprising, and revealed that women CS majors came to college
without prior introduction to programming concepts at more than
three times the rate of men. Furthermore, while only one female
subject entered college with an extensive introduction to the
concepts, one quarter of the male subjects had. However, given
previous research suggesting the importance of pre-college
experience for success in introductory programming courses ,
the mean CS GPAs are somewhat unexpected. While subjects
with no previous experience have slightly lower CS GPAs than
subjects in the other three groups, a Kruskal-Wallis independent
samples test revealed these differences were not significant.
The most interesting gender-related result of this research so far
appears to be that, while female subjects enter college having
been introduced to and having mastered far fewer concepts before
college than males, they “catch up” with the males in their
introductory classes, reporting (at least retrospectively) having
achieved similar mastery of the concepts. The retrospective nature
of these assessments should not be discounted, as research has
indicated that students’ perceptions of factors such as
preparedness  and gender discrimination  can improve over
time, particularly for female students. It is possible that the
females in this study may have had different perceptions of
concept difficulty or mastery had they made these evaluations
earlier in their academic careers.
Similar to women subjects in other studies [7, 11, 13], female
subjects in this study had substantially less pre-college
programming experience than did their male peers. Our results
also support those in  and , and clearly indicate that both
male and female students were able to succeed in CS with little or
no pre-college programming experience. That 64% of women in
this study were able to overcome this deficit is encouraging.
However, since this study only examined graduating students, we
do not know how many women were retained from their
While it is not known what factors enabled the female subjects to
achieve equal standing, one plausible explanation is that pre-
college experience in a non-object oriented approach may not
provide an advantage when followed by introductory college
programming courses taught using an object-oriented approach
. Possible evidence of this phenomenon can be seen in Figure
2, which shows that less than 10% of male subjects mastered
object before college, while over 50% mastered variable,
suggesting their pre-college experience was not object-oriented.
Furthermore, none of the women mastered object before college,
yet nearly 60% mastered it in their introductory classes.
The trend of women coming to college with less programming
experience than men is unlikely to change in the foreseeable
future. Therefore, if we hope to increase female participation in
CS, it seems prudent to closely examine factors that enable less
experienced women to succeed. As suggested in , the playing
field may have been leveled somewhat for women in this study
because their male peers’ pre-college experience appears not to
have been object-oriented. This equalizer may disappear as
object-oriented teaching filters down into high schools. Perhaps
changes departments have made to facilitate the success of
students with less experience (e.g. multiple entry points for
incoming students, collaborative learning environments, pair-
programming opportunities) are working. This is not obvious,
since the number of women CS majors in recent years has not
increased, and fluctuations in enrollments due to economic and
societal factors make the impact of departmental changes difficult
to assess. It is worth noting however that 32% of women in this
study attended a large research university offering multiple
introductory tracks. To begin to answer these questions would
require a longitudinal study that follows women, starting with
their introductory classes on through their graduation in CS or
their choice of alternate paths.
We are grateful to Susan Haller for data collection, statistical
analysis and editorial assistance; to Kate Sanders, Rhode Island
College, and Ruth Anderson, University of Virginia for data
collection; and to Carol Zander for her participation in previous
phases of this research. We thank Josh Tenenberg for his
suggestions. We are grateful to Sally Fincher, Marian Petre, and
the participants of the Bootstrapping Research in Computer
Science Education project for feedback and encouragement on
this study. This study was partially funded by a University of
Arizona Internet Technology, Commerce and Design Institute
grant. This material is based upon work supported by the National
Science Foundation under Grant No. DUE-0243242. Any
opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not
necessarily reflect the views of the National Science Foundation.
 S. Beyer, M. DeKeuster, K. Walter, M. Colar, and C. Holcomb.
Changes in CS students’ attitudes towards CS over time: an
examination of gender differences. In Proceedings of the 36th
SIGCSE technical symposium on Computer science education,
392-396. ACM Press, 2005.
 S. Beyer, K. Rynes, J. Perrault, K. Hay, and S. Haller. Gender
differences in computer science students. In Proceedings of the
34th SIGCSE technical symposium on Computer science
education, 49-53. ACM Press, 2003.
 J. Carter and T. Jenkins. Gender and programming: what’s going
on? In Proceedings of the 4th annual SIGCSE/ SIGCUE ITiCSE
conference on Innovation and technology in computer science
education, 1-4. ACM Press, 1999.
 Z. A. Kersteen, M. C. Linn, M. Clancy, and C. Hardyck.
Previous experience and the learning of computer programming:
The computer helps those who help themselves. Journal of
Educational Computing Research, 4(3):321-334, 1988.
 H. M. Lips and L. Temple. Majoring in computer science:
Causal models for women and men. Research in Higher
Education, 31(1), 1990.
 N. Maiden and M. Hare. Problem domain categories in
requirements engineering. International Journal of Human-
Computer Studies, 49:281-304, 1998.
 J. Margolis and A. Fisher. Unlocking the clubhouse: women in
computing. MIT Press, 2002.
 R. McCauley, et al. What do successful computer science
students know? an integrative analysis using card-sort measures
and content analysis to evaluate graduating students’ knowledge
of programming concepts. Expert Systems, 22(3):147-159, 2005.
 L. Murphy, et al. A multi-institutional investigation of computer
science seniors’ knowledge of programming concepts. In
Proceedings of the 36th SIGCSE technical symposium on
Computer science education, 510-514. ACM Press, 2005.
 G. Rugg and P. McGeorge. The sorting techniques: a tutorial
paper on card sorts, picture sorts, and item sorts. Expert
Systems, 22(3), 2005.
 M. G. Sackrowitz and A. P. Parelius. An unlevel playing field:
women in the introductory computer science courses. In
Proceedings of the 27th SIGCSE technical symposium on
Computer science education, 37-41. ACM Press, 1996.
 K. Sanders, et al. A multi-institutional, multinational study of
programming concepts using card sort data. Expert Systems,
 G. Scragg and J. Smith. A study of barriers to women in
undergraduate computer science. In Proceedings of the 29th
SIGCSE technical symposium on Computer science education,
82-86. ACM Press, 1998.
 S. J. Spencer, C. M. Steele, and D. M. Quinn. Stereotype threat
and women’s math performance. Journal of Experimental Social
Psychology, 35:4-28, 1999.
 P. Ventura and B. Ramamurthy. Wanted: CS1 students, no
experience required. In Proceedings of the 35th SIGCSE
technical symposium on Computer science education, 240-244.
ACM Press, 2004.