ArticlePDF Available


This article presents a technical opinion in the case of collaborative programming for a large, complex system, in which two programmers are working jointly on the same algorithm and code. A field experiment was conducted using experienced programmers who worked on a challenging problem important to their organization, in their own environments, and with their own equipment. Findings revealed that all the teams outperformed the individual programmers, enjoyed the problem-solving process more, and had greater confidence in their solutions. Several aspects of this experiment make the results significant. First, there may be a tendency to dismiss results using small groups. Questions may arise at the value of these results if the collaborators do not perform twice as well as individuals, at least in the amount of time spent. The article concludes that the crunch-time information systems development will demand innovative ways to produce high-quality systems in a short time, with companies increasingly introducing new products. This rare experimental research is phenomenon considering the luxury of using collaborative programming while many companies are experiencing shortages of experienced programmer/analysts.
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 105
Team programming usually
means coordinating efforts
of individual programmers
who divide up the programming
tasks for a large, complex system.
Collaborative programming is
used here to mean two program-
mers working jointly on the
same algorithm and code. Previ-
ous research indicates that stu-
dent programmers working
collaboratively outperformed
individual programmers. A fol-
low-up field experiment was con-
ducted using experienced
programmers who worked on a
challenging problem important
to their organization, in their
own environments, and with
their own equipment. To the sur-
prise of the managers and partici-
pants, all the teams outperformed
the individual programmers,
enjoyed the problem-solving
process more, and had greater
confidence in their solutions.
Description of the
“Effective problem-solving per-
formance” is defined in terms of
(a) the readability of the pro-
posed solution, that is, the
degree to which the problem-
solving strategy could be deter-
mined from the subject’s work;
and (b) the functionality of the
proposed solution, that is, the
degree to which the strategy
accomplishes the objectives
stated in the problem descrip-
tion. The variables READABIL-
were defined accordingly. Read-
ability is a component of overall
score since it is possible for a
subject to use a reasonable strat-
egy and to use programming lan-
guage structures appropriately
and yet fail to solve the problem
(in the sense of generating the
correct output). In such cases, the
programmer may have misinter-
preted the problem description
or overlooked a critical compo-
nent. Overall success on the
problem-solving effort is mea-
sured by a variable SCORE,
which is a simple sum of compo-
nent variables READABILITY
on previous results, four predic-
tions were made:
1. Programmers working in pairs
will produce more readable
and functional solutions to a
programming problem than
will programmers working
2. Groups will take less time on
average to solve the problem
than individuals working
3. Programmers working in pairs
will express higher levels of
confidence about their work
(CONFID) and enjoyment of
the process (ENJOY) immedi-
ately following the problem-
solving session than will
programmers working alone.
(Positive feelings about the
problem-solving process can
affect performance. These feel-
ings were assessed immedi-
ately following the
problem-solving session by a
two-item questionnaire.)
4. Programmers with more years
of experience will perform bet-
ter than programmers with
fewer years of experience.
Aside from the pairing, condi-
tions and materials used in the
experiment were the same for
both experimental (teams) and
control (individuals) groups.
These subjects were 15 full-time
system programmers from a pro-
gram trading firm working on
system maintenance of three
Unix networks and a large data-
base running Sybase. They used
The Case for
Collaborative Programming
John T. Nosek
106 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
the X-window system with the C
language. The firm uses a very
large database to get information
on program trading. The subjects
were asked to write a script that
performs a database consistency
check (DBCC) with the output
for errors to be written to a file.
If no errors are found then the
script should
write to the out-
put file saying
that no errors
were encoun-
tered. Also, the
script should dis-
play on the
screen whether
the results of this
output are to be
mailed to all the
system adminis-
trators. To evaluate
the READ-
able, the subjects
were asked to
properly com-
ment on each of the processes
within the script they were pro-
gramming. None of the subjects
had worked on this kind of prob-
lem before. DBCC checks are
considered so critical to the orga-
nization’s success and generally
beyond the skill of in-house pro-
grammers that outside consul-
tants are usually hired to perform
them. The subjects in the experi-
mental paired groups and the
control group of individuals were
randomly assigned using a set of
randomly generated numbers. All
subjects were given 45 minutes
to solve the problem.
During the problem-solving
phase, the pairs were allowed to
communicate freely with their
randomly assigned partner; sub-
jects working alone were asked to
work without any communica-
tion. To solve the problem, all
subjects used Sun Microsystems
SPARCstations, equipment with
which they were very familiar.
The post-treatment questionnaire
was given to each subject after he
or she had handed in all the
problem-solving materials. A
stopwatch was used to time each
group and individual. Each set of
materials was evaluated on the
degree to which it solved the
and on the readability of the
solution (READABILITY).
The functionality score was
obtained as a sum of three parts
corresponding to the three out-
put requirements stipulated in
the problem description. The
READABILITY scores could
range between 0 to 2. READ-
ABILITY was assigned 0 for an
entirely unreadable solution, 2
for an entirely readable solution,
and 1 otherwise. FUNCTION-
ALITY scores ranged from 0 for
a solution which did not achieve
the goal at all, to 6 if the solu-
tion achieved the goal entirely.
With the exception of time, per-
formance measurements were the
average scores of two separate
graders. Overall SCORE was
obtained by summing READ-
a perfect overall score being 8.
The t-test, a statisti-
cal technique
designed to compare
the means of two
small samples, was
used. The two-sided
t-test, which looks at
the ends of both
sides of a flattened
normal distribution
curve, was used
because this proce-
dure tests for results
that are in the pre-
dicted as well as in
the opposite direc-
tion expected, the
standard procedure in this kind
of experiment. The significance
level for all tests was set so that
the probability was less than 1 in
20 that results were due to
chance. Two graders separately
evaluated the problem solutions
with inter-grader reliability of
over 90%.
The table presents the means
and results of the two-sided t test
for performance and satisfaction
measurements, the test created
for small sample sizes. Predic-
tions 1 and 3 were supported.
For example, groups outper-
formed individuals, enjoyed their
collaborative problem-solving
process more, and were more
confident in their solutions. In
fact, all groups outperformed
TIME (minutes)
n = 5
1.40 (0.894)
4.20 (1.788)
5.60 (2.607)
42.60 (3.361)
n = 5
3.80 (2.049)
4.00 (1.870)
n = 5
2.00 (0.000)
5.60 (0.547)*
7.60 (0.547)*
30.20 (1.923)
n = 10
6.50 (0.500)*
6.60 (0.418)*
Comparison of Individual and Team Measurements
Control Group
mean (st. dev.)
Experimental Group
mean (st. dev.)
*less than 1 in 20 that results are due to chance
Technical Opinion
individuals. Although the aver-
age time for completion was
more than 12 minutes (41%)
longer for individuals, prediction
2 was not statistically supported
because there is more than a 1 in
20 chance that this is due to
chance. Prediction 4 was sup-
ported. Experience and problem
scores were highly correlated for
both the control (73.5%) and the
experimental (87.2%) groups.
The results provide additional
evidence that collaboration
improves the problem-solving
process. Several aspects of this
experiment make the results sig-
nificant. First, there may be a
tendency to dismiss results using
small groups. However, it is rare,
if not impossible, to obtain the
time and cooperation of experi-
enced programmers to perform
the identical task. Much experi-
mental research is automatically
dismissed because it is done with
students, who are more plentiful
but who do not possess the skills
and knowledge of experienced
programmers. The subjects of
this experiment were experienced
programmers, working on an
important, challenging problem,
in a natural setting. Second,
because of the inherent bias
against small samples in statisti-
cal models, Miller [2]emphasizes
that a small sample, with a prob-
ability of only 1 in 20 that
results were due to chance, may
be far more striking than results
from larger sample sizes that have
the same or lower probability.
Third, this experiment was an
extension of previous ones, not
just a one-time event.
The results are consistent with
previous experiments that used
simpler, canned problems. The
qualitative data also provides
some interesting insights. The
majority of programmers were
somewhat skeptical of the value
of collaboration in working on
the same algorithm/program
module and thought that the
process would not be enjoyable.
However, as the results indicate,
and supported by their com-
ments, collaboration did improve
their performance and they
enjoyed their efforts. One pro-
gramming pair experienced a
“qualitative jump” as compared
to the other groups and individu-
als in the experiment. To the
amazement of the manager, their
programming solution was better
than previous scripts written for
the company. It is costly to run
these scripts and efficiently writ-
ten scripts are considered so diffi-
cult to create that the company
hires expert outside consultants
to write them. However, the
script written by this program-
ming team was twice as efficient
as previously purchased scripts.
Some may question the
value of these results if the
collaborators do not per-
form “twice” as well as individu-
als, at least in the amount of time
spent. For instance, if the collab-
orators did not perform the task
in half the time it takes an indi-
vidual, it would still be more
expensive to employ two pro-
grammers. However, there are at
least two scenarios where some
improved performance over what
is expected or possible by a single
programmer may be the goal: (1)
speeding up development and (2)
improving software quality.
In the first case, bringing a
product to market a month ear-
lier can mean a competitive edge,
or gaining, in some cases, even
survival. As organizations find it
increasingly difficult to produce
current products more efficiently,
they are escalating the number of
new products to market, where
the profit margins are greater.
According to the Product Devel-
opment and Management Associ-
ation, new products are expected
to account for 37% of total sales
by 2000, up from 28% for the
five years that ended in 1995 [3].
A reduced product life cycle,
combined with an increased
informational component of the
product, is a growing, critical
problem for information systems
“Crunch mode is not a matter
of opportunity—it’s a matter of
survival.… The ability to get
working software quickly into
the hands of users will be charac-
teristic of successful data-process-
ing organizations for the
foreseeable future. Groups that
can produce and install software
systems within tight time frames
will prosper. Those [that]
can’t will fail and, in some cases,
they will bring the enterprises of
which they are a part down with
them. Fast response to changing
information-processing require-
ments is a necessity in today’s
world.” [1]
Additionally, while there was
more than a 1 in 20 probability
that the time saved by groups
was due to chance, the groups
completed the task 40% more
quickly and more effectively. The
collaborative programmers pro-
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 107
duced better algorithms and code
in less time. As we all know,
poorly written code completed
quickly may in fact cause greater
delays in the overall development
and implementation time. Prior
to this experiment, standard
practice would limit the
resources that could be brought
to bear to speed up the process.
For example, additional program-
mers could be assigned to addi-
tional program modules.
However, programmers would
generally not be assigned to the
same program modules.
Group support technologies
may be employed to marshal pro-
grammer resources wherever they
exist to work on critical, time-
pressing systems.
The second scenario has to do
with improved quality in soft-
ware writing and testing. We
group together pseudocoding,
programming, and human-test-
ing procedures, such as struc-
tured walk-throughs, because
they involve closely related cog-
nitive skills, where programming
is the syntactical translation of
pseudocode into specific com-
puter languages and human-test-
ing procedures validate the
process. As program generators
and packages become more
prevalent, the greatest value
added by the programmer/analyst
may be as a contributor to team
development of innovative algo-
rithms and novel implementa-
tions of process and data models.
With companies increasingly
introducing new products,
crunch-time information systems
development will demand innov-
ative ways to produce high-qual-
ity systems in a short time. Pro-
fessional collaborative program-
mers who jointly developed
algorithms and code outper-
formed individual programmers
in this real-world field experi-
ment, a rarity in experimental
research. While the number of
subjects was small, this experi-
ment is not a single experiment,
but builds upon previous experi-
mental studies done with student
collaborative programmers. Some
may wonder at the luxury of
using collaborative programming
while many companies are expe-
riencing shortages of experienced
programmer/analysts. However,
the results raise intriguing ques-
tions. Can two average or less-
experienced workers collaborate
to perform tasks that may be too
challenging for each one alone?
Can a company bring a product
to market substantially earlier by
using collaborative program-
ming? Can collaborative pro-
gramming using worldwide
commodity programming
resources offer a competitive
1. Abdel-Hamid, T.K. Investigating the
cost/schedule trade-off in software develop-
ment. IEEE Software (Jan. 1990), 97–105.
2. Miller, R.G. Beyond Anova, Basics of Applied
Statistics. John Wiley and Sons, New York,
3. Warner, S. A dreamer finds the Expertise to
Develop an Idea. The Philadelphia Inquirer,
Philadelphia, Penn., Dec. 1, 1996, D1.
John T. Nosek (
is a professor in the Computer and
Information Sciences Department at
Temple University, Philadelphia.
© ACM 0002-0782/98/0300 $3.50
108 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
Technical Opinion
The designation “ACM Fellow” may be
conferred upon those ACM Members
who have distinguished themselves by
outstanding technical and professional
achievements in information technology,
who are current voting members of ACM
and have been voting members for the
preceding five years. Any voting member
of ACM may nominate another member
for this distinction. Nominations must
be received by the ACM Fellows Com-
mittee no later than August 1 of each
year and must be delivered to the Com-
mittee on forms provided for this purpose
(see below).
Nomination information organized by a
principal nominator includes:
1) excerpts from the candidate’s current
curriculum vitae, listing selected pub-
lications, patents, technical achieve-
ments, honors, and other awards.
2) a description of the work of the nomi-
nee, drawing attention to the contribu-
tions which merit designation as Fellow.
3) supporting endorsements from five
ACM Members.
ACM Fellows nomination forms and
endorsement forms may be obtained
from ACM by writing to:
ACM Fellows Nomination Committee
ACM Headquarters 1515 Broadway
New York, New York 10036-5701
The forms can also be accessed on the
Completed forms should be sent by
August 1, 1998
to one of the following:
ACM Fellows Committee
ACM Headquarters
1515 Broadway
New York, New York 10036-5701, USA
+1-212-869-0824 - fax
ACM Fellows
... Working in pairs produces more rapid and effective solutions than working alone. In addition, students with high programming skills will help a partner who has lower programming skills to complete their task [67]. Getting students to write code together in pairs or small groups can also enhance students' programming performance and confidence [52][53][54]. ...
... Working in pairs produces more rapid and effective solutions than working alone. In addition, students with high programming abilities will assist their partners with low programming abilities in completing their work [67]. ...
... Pair programming produces more rapid and effective solutions than solo programming [75]. Furthermore, students with high programming skills will help a partner who has lower programming skills to complete their task [67]. Curriculum-makers and instructors should pay attention to the student attitudes toward programming because negative student attitudes toward programming can increase the dropout rate and grade failures. ...
Full-text available
The purpose of this study was to investigate the relationship between students’ attitudes toward programming, gender, and learning performances. The survey used for measuring students’ attitudes toward programming consisted of 20 questions on a five-point Likert scale in five dimensions (meaningfulness, interest in programming, self-efficacy, creativity, and collaboration). Ninety freshmen who had basic programming experience by using block-based programming in the Innovation in Educational Technology course were asked to take the survey. The overall reliability of the survey was found to be 0.93. The results showed that there was no significant difference between male and female freshmen in attitude toward programming, but there was a significant difference among different learning performances in dimensions of interest in programming, self-efficacy, and creativity. We performed pairwise comparisons at the same level of significance by using Fisher’s least significant difference (LSD) method to test which group differs from the other groups. The results found that low-performing students’ attitudes toward programming in dimensions of interest in programming, self-efficacy, and creativity were the lowest of all types of students. This is a challenge for instructors in planning learning activities to encourage low-performing students to have a more positive attitude toward programming.
... A formal study [6] was conducted in a classroom setting between inexperienced middle school and high school programmers using the Intelligent Programming (IPRO) framework. A collaborative programming environment [53] was introduced where two programmers working jointly on the same algorithm and code. Nosek et al. performed a comparative study, which found that student programmers working collaboratively outperformed individual programmers for aspects of performance, which included readability, functionality, and time and satisfaction (as confidence and enjoyment). ...
... Similar to Nosek et al. [53], we logged how long it took the participants to complete the tasks. Both synchronous and asynchronous modes were faster than the control and the order effect was not significant. ...
... The unsurprising part of the findings was that both collaborative modes were faster than the control, which is typical to expect when the work is shared. This has been also known to be the case for code-based programming for decades [53]. It is worth acknowledging that a limitation in the form of a possibility may exist that the task itself could be a confounding factor since it varied across the levels of the independent variable. ...
Full-text available
Game development is a collective process in which a variety of different professionals from different backgrounds collaborate together not only by means of conversational interaction but also collaborative participation, one of which is programming. While collaborative and pair programming solutions exist for text-based programming languages, visual programming has not enjoyed as much attention. These solutions would not only address advanced forms of business communication among team members but could find their use in distance learning, which would have been useful during the pandemic. In our work, we propose a solution for collaborative behavioral animation of NPCs using behavior trees through synchronous and asynchronous modes of collaboration. We conducted a user study with 12 moderately skilled game development university students who were placed in groups of two and engaged in joint fixed behavior tree development tasks using the synchronous and asynchronous modes and auxiliary features of live preview, access and restoration of previous states from behavior tree history, conflict resolution, and instant messaging. Participants also completed a control task where no collaboration was involved and auxiliary features were not available. Feedback form Creativity Support Index, a self-developed questionnaire, and a semi-structured interview were collected. Additionally, task completion times were logged. The results indicate that the two collaborative modes provide expected improvement over the control condition. No significant differences were found between the two collaborative modes. However, the semi-structed interview revealed that the synchronous mode could be useful for quick prototyping, while the asynchronous mode - for most other situations. Supplementary information: The online version contains supplementary material available at 10.1007/s11042-022-12307-2.
... To improve programming skills, collaborative programming has been widely adopted in many schools. Collaborative programming, in which a group of learners work on the same code and complete programming tasks together, is considered an effective pedagogical approach (Nosek, 1998). Collaborative programming aims to improve learners' programming skills through writing code and refining programs with peers (Lu et al., 2017). ...
Full-text available
Programming skills have gained increasing attention in recent years because digital technologies have become an indispensable part of life. However, little is known about the roles of fade-in and fade-out scaffolding in online collaborative programming settings. To close this research gap, the present study aims to examine the roles of fade-in and fade-out scaffolding for novice programmers in online collaborative programming. A total of 90 undergraduate students participated in the exploratory study and were assigned to 15 fade-in groups and 15 fade-out groups. All of the participants completed the same programming task. The findings reveal that fade-in scaffolding can significantly improve collaborative knowledge building, programming skills, metacognitive behaviors, emotions, and collective efficacy. Goal setting, planning, monitoring and control, enacting strategies, and evaluation and reflection are identified as the crucial metacognitive behaviors. The main contribution of this exploratory study is to shed light on how to design and implement scaffolding for novice programmers.
... Pair programming, for example, is a practice that involves two developers collaborating as a single individual on the design, coding, and testing of the same programming task. This practice has been shown to be productive and to render higher-quality code than either developer may produce alone (Lui & Chan, 2006;Nosek, 1998;Williams, 2001;Williams, Kessler, Cunningham, & Jeffries, 2000;Williams & Upchurch, 2001), especially with novice programmers or challenging programming problems (Lui & Chan, 2006), and positive effects have been found in educational contexts as well (Braught, Wahls, & Eby, 2011;Tunga & Tokel, 2018;Umapathy & Ritzhaupt, 2017). Another popular practice involves explaining line by line what a nonworking piece of code is supposed to do. ...
Full-text available
There is growing interest in teaching computer science and programming skills in schools. Here we investigated the efficacy of peer tutoring, which is known to be a useful educational resource in other domains but never before has been examined in such a core aspect of applied logical thinking in children. We compared (a) how children (N = 42, age range = 7 years 1 month to 8 years 4 months) learn computer programming from an adult versus learning from a peer and (b) the effect of teaching a peer versus simply revising what has been learned. Our results indicate that children taught by a peer showed comparable overall performance—a combination of accuracy and response times—to their classmates taught by an adult. However, there was a speed–accuracy trade-off, and peer-taught children showed more exploratory behavior, with shorter response times at the expense of lower accuracy. In contrast, no tutor effects (i.e., resulting from teaching a peer) were found. Thus, our results provide empirical evidence in support of peer tutoring as a way to help teach computer programming to children. This could contribute to the promotion of a widespread understanding of how computers operate and how to shape them, which is essential to our values of democracy, plurality, and freedom.
... Although the concept of pair programming existed informally much earlier [8], it was popularized and formally defined in Extreme Programming in the late 1990s [17]. In pair programming [18], two programmers work jointly to produce software. One programmer, known as the driver, writes the software (i.e., works on the problem). ...
... These studies have taken place in both academic and commercial environments. In the commercial area, two studies are particularly noteworthy: Nosek [12] showed that pair teams significantly outperformed individuals on program quality. Jensen [13] found that the error rate for a project with pair-programming was three orders of magnitude smaller than for other similar projects. ...
Conference Paper
This Research to Practice Full Paper depicts and evaluates a secondary school project on using pair-and solo-programming of mini-games in introductory programming classes. In addition to investigating various factors influencing students' problem-solving skills (K9; age 14-15), we introduce the software metric Lines Of Code (LOC) to compare outcomes on that specific measure in the pair-and solo-programming setting. The mini-games were developed with the free personal edition of the game development engine Unity™ and C#. In the current study, four different classes at the secondary level were instructed and researched. All classes had approximately the same number of students, the same tasks, the same tutorials, but were using a different social setting for programming. In response to the worldwide pandemic in the years 2020 and 2021, instruction and research proceeded either in virtual or in hybrid-learning mode. We chose participatory action research to accommodate for the complexity of factors inherent in the field as well as for its iterative, cyclic nature. The current cycle is the third of a series on studies that have investigated various aspects of introductory pair-programming. The evaluation phase employs a digital questionnaire with open and closed questions aimed to capture student's perceptions regarding problem solving. In addition, the software metric Lines Of Code" (LOC), traditionally used to measure the size of a computer program by counting the number of lines of the program's source code, was adapted to measure students' achievement in pair-and solo-programming. With our research we aim to contribute to make learning to program more effective, engaging, and inclusive, and we would like to promote 21 st century competences besides programming skills. In addition, we are eager to share our practice of pair-programming with educators in order to inspire them to experiment with pair-programming as a social setting with high potential, even in times of required social distancing.
his paper describes a qualitative study of how undergraduate students majoring in Information Technology perceive the effectiveness and evaluate the learning experience of pair-programming. The phenomenographic research approach was used to analyze student interviews and revealed 4 categories of descriptions: Effective Problem Solving, Participation, Enjoyment and Coding. Pair-programming as a teaching methodology was commonly perceived as a positive experience. The resulting outcome space maps a logical hierarchy of students’ conceptions of reality (categories of description). Findings of this research identify the factors that affect student engagement in a problem-solving process and can be used as a guiding principle on how to improve students’ learning experience of computer programming.
The author has studied the effects of schedule compression or stretch-out on total project cost within a much broader effort to study and predict the dynamics of the entire development process. The resulting cost/schedule tradeoffs were examined. Much of this project involved developing a comprehensive system-dynamics model. He used the model to conduct three simulation experiments. (1) He investigated the effects of different levels of schedule compression and stretch-out on total project cost in man-days and compared the results to those reported in the literature. (2) He addressed the stealthy role undersizing plays in schedule compression. (3) He investigated how different levels of managerial commitment affect the project's final cost and completion time. The results of all three experiments are presented and discussed.< >
  • R G Miller
  • Anova
Miller, R.G. Beyond Anova, Basics of Applied Statistics. John Wiley and Sons, New York, 1986.
A dreamer finds the Expertise to Develop an Idea. The Philadelphia Inquirer
  • S Warner
Warner, S. A dreamer finds the Expertise to Develop an Idea. The Philadelphia Inquirer, Philadelphia, Penn., Dec. 1, 1996, D1.
  • R G Miller
  • Beyond Anova
Miller, R.G. Beyond Anova, Basics of Applied Statistics. John Wiley and Sons, New York, 1986.