ArticlePDF Available


This article presents a technical opinion in the case of collaborative programming for a large, complex system, in which two programmers are working jointly on the same algorithm and code. A field experiment was conducted using experienced programmers who worked on a challenging problem important to their organization, in their own environments, and with their own equipment. Findings revealed that all the teams outperformed the individual programmers, enjoyed the problem-solving process more, and had greater confidence in their solutions. Several aspects of this experiment make the results significant. First, there may be a tendency to dismiss results using small groups. Questions may arise at the value of these results if the collaborators do not perform twice as well as individuals, at least in the amount of time spent. The article concludes that the crunch-time information systems development will demand innovative ways to produce high-quality systems in a short time, with companies increasingly introducing new products. This rare experimental research is phenomenon considering the luxury of using collaborative programming while many companies are experiencing shortages of experienced programmer/analysts.
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 105
Team programming usually
means coordinating efforts
of individual programmers
who divide up the programming
tasks for a large, complex system.
Collaborative programming is
used here to mean two program-
mers working jointly on the
same algorithm and code. Previ-
ous research indicates that stu-
dent programmers working
collaboratively outperformed
individual programmers. A fol-
low-up field experiment was con-
ducted using experienced
programmers who worked on a
challenging problem important
to their organization, in their
own environments, and with
their own equipment. To the sur-
prise of the managers and partici-
pants, all the teams outperformed
the individual programmers,
enjoyed the problem-solving
process more, and had greater
confidence in their solutions.
Description of the
“Effective problem-solving per-
formance” is defined in terms of
(a) the readability of the pro-
posed solution, that is, the
degree to which the problem-
solving strategy could be deter-
mined from the subject’s work;
and (b) the functionality of the
proposed solution, that is, the
degree to which the strategy
accomplishes the objectives
stated in the problem descrip-
tion. The variables READABIL-
were defined accordingly. Read-
ability is a component of overall
score since it is possible for a
subject to use a reasonable strat-
egy and to use programming lan-
guage structures appropriately
and yet fail to solve the problem
(in the sense of generating the
correct output). In such cases, the
programmer may have misinter-
preted the problem description
or overlooked a critical compo-
nent. Overall success on the
problem-solving effort is mea-
sured by a variable SCORE,
which is a simple sum of compo-
nent variables READABILITY
on previous results, four predic-
tions were made:
1. Programmers working in pairs
will produce more readable
and functional solutions to a
programming problem than
will programmers working
2. Groups will take less time on
average to solve the problem
than individuals working
3. Programmers working in pairs
will express higher levels of
confidence about their work
(CONFID) and enjoyment of
the process (ENJOY) immedi-
ately following the problem-
solving session than will
programmers working alone.
(Positive feelings about the
problem-solving process can
affect performance. These feel-
ings were assessed immedi-
ately following the
problem-solving session by a
two-item questionnaire.)
4. Programmers with more years
of experience will perform bet-
ter than programmers with
fewer years of experience.
Aside from the pairing, condi-
tions and materials used in the
experiment were the same for
both experimental (teams) and
control (individuals) groups.
These subjects were 15 full-time
system programmers from a pro-
gram trading firm working on
system maintenance of three
Unix networks and a large data-
base running Sybase. They used
The Case for
Collaborative Programming
John T. Nosek
106 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
the X-window system with the C
language. The firm uses a very
large database to get information
on program trading. The subjects
were asked to write a script that
performs a database consistency
check (DBCC) with the output
for errors to be written to a file.
If no errors are found then the
script should
write to the out-
put file saying
that no errors
were encoun-
tered. Also, the
script should dis-
play on the
screen whether
the results of this
output are to be
mailed to all the
system adminis-
trators. To evaluate
the READ-
able, the subjects
were asked to
properly com-
ment on each of the processes
within the script they were pro-
gramming. None of the subjects
had worked on this kind of prob-
lem before. DBCC checks are
considered so critical to the orga-
nization’s success and generally
beyond the skill of in-house pro-
grammers that outside consul-
tants are usually hired to perform
them. The subjects in the experi-
mental paired groups and the
control group of individuals were
randomly assigned using a set of
randomly generated numbers. All
subjects were given 45 minutes
to solve the problem.
During the problem-solving
phase, the pairs were allowed to
communicate freely with their
randomly assigned partner; sub-
jects working alone were asked to
work without any communica-
tion. To solve the problem, all
subjects used Sun Microsystems
SPARCstations, equipment with
which they were very familiar.
The post-treatment questionnaire
was given to each subject after he
or she had handed in all the
problem-solving materials. A
stopwatch was used to time each
group and individual. Each set of
materials was evaluated on the
degree to which it solved the
and on the readability of the
solution (READABILITY).
The functionality score was
obtained as a sum of three parts
corresponding to the three out-
put requirements stipulated in
the problem description. The
READABILITY scores could
range between 0 to 2. READ-
ABILITY was assigned 0 for an
entirely unreadable solution, 2
for an entirely readable solution,
and 1 otherwise. FUNCTION-
ALITY scores ranged from 0 for
a solution which did not achieve
the goal at all, to 6 if the solu-
tion achieved the goal entirely.
With the exception of time, per-
formance measurements were the
average scores of two separate
graders. Overall SCORE was
obtained by summing READ-
a perfect overall score being 8.
The t-test, a statisti-
cal technique
designed to compare
the means of two
small samples, was
used. The two-sided
t-test, which looks at
the ends of both
sides of a flattened
normal distribution
curve, was used
because this proce-
dure tests for results
that are in the pre-
dicted as well as in
the opposite direc-
tion expected, the
standard procedure in this kind
of experiment. The significance
level for all tests was set so that
the probability was less than 1 in
20 that results were due to
chance. Two graders separately
evaluated the problem solutions
with inter-grader reliability of
over 90%.
The table presents the means
and results of the two-sided t test
for performance and satisfaction
measurements, the test created
for small sample sizes. Predic-
tions 1 and 3 were supported.
For example, groups outper-
formed individuals, enjoyed their
collaborative problem-solving
process more, and were more
confident in their solutions. In
fact, all groups outperformed
TIME (minutes)
n = 5
1.40 (0.894)
4.20 (1.788)
5.60 (2.607)
42.60 (3.361)
n = 5
3.80 (2.049)
4.00 (1.870)
n = 5
2.00 (0.000)
5.60 (0.547)*
7.60 (0.547)*
30.20 (1.923)
n = 10
6.50 (0.500)*
6.60 (0.418)*
Comparison of Individual and Team Measurements
Control Group
mean (st. dev.)
Experimental Group
mean (st. dev.)
*less than 1 in 20 that results are due to chance
Technical Opinion
individuals. Although the aver-
age time for completion was
more than 12 minutes (41%)
longer for individuals, prediction
2 was not statistically supported
because there is more than a 1 in
20 chance that this is due to
chance. Prediction 4 was sup-
ported. Experience and problem
scores were highly correlated for
both the control (73.5%) and the
experimental (87.2%) groups.
The results provide additional
evidence that collaboration
improves the problem-solving
process. Several aspects of this
experiment make the results sig-
nificant. First, there may be a
tendency to dismiss results using
small groups. However, it is rare,
if not impossible, to obtain the
time and cooperation of experi-
enced programmers to perform
the identical task. Much experi-
mental research is automatically
dismissed because it is done with
students, who are more plentiful
but who do not possess the skills
and knowledge of experienced
programmers. The subjects of
this experiment were experienced
programmers, working on an
important, challenging problem,
in a natural setting. Second,
because of the inherent bias
against small samples in statisti-
cal models, Miller [2]emphasizes
that a small sample, with a prob-
ability of only 1 in 20 that
results were due to chance, may
be far more striking than results
from larger sample sizes that have
the same or lower probability.
Third, this experiment was an
extension of previous ones, not
just a one-time event.
The results are consistent with
previous experiments that used
simpler, canned problems. The
qualitative data also provides
some interesting insights. The
majority of programmers were
somewhat skeptical of the value
of collaboration in working on
the same algorithm/program
module and thought that the
process would not be enjoyable.
However, as the results indicate,
and supported by their com-
ments, collaboration did improve
their performance and they
enjoyed their efforts. One pro-
gramming pair experienced a
“qualitative jump” as compared
to the other groups and individu-
als in the experiment. To the
amazement of the manager, their
programming solution was better
than previous scripts written for
the company. It is costly to run
these scripts and efficiently writ-
ten scripts are considered so diffi-
cult to create that the company
hires expert outside consultants
to write them. However, the
script written by this program-
ming team was twice as efficient
as previously purchased scripts.
Some may question the
value of these results if the
collaborators do not per-
form “twice” as well as individu-
als, at least in the amount of time
spent. For instance, if the collab-
orators did not perform the task
in half the time it takes an indi-
vidual, it would still be more
expensive to employ two pro-
grammers. However, there are at
least two scenarios where some
improved performance over what
is expected or possible by a single
programmer may be the goal: (1)
speeding up development and (2)
improving software quality.
In the first case, bringing a
product to market a month ear-
lier can mean a competitive edge,
or gaining, in some cases, even
survival. As organizations find it
increasingly difficult to produce
current products more efficiently,
they are escalating the number of
new products to market, where
the profit margins are greater.
According to the Product Devel-
opment and Management Associ-
ation, new products are expected
to account for 37% of total sales
by 2000, up from 28% for the
five years that ended in 1995 [3].
A reduced product life cycle,
combined with an increased
informational component of the
product, is a growing, critical
problem for information systems
“Crunch mode is not a matter
of opportunity—it’s a matter of
survival.… The ability to get
working software quickly into
the hands of users will be charac-
teristic of successful data-process-
ing organizations for the
foreseeable future. Groups that
can produce and install software
systems within tight time frames
will prosper. Those [that]
can’t will fail and, in some cases,
they will bring the enterprises of
which they are a part down with
them. Fast response to changing
information-processing require-
ments is a necessity in today’s
world.” [1]
Additionally, while there was
more than a 1 in 20 probability
that the time saved by groups
was due to chance, the groups
completed the task 40% more
quickly and more effectively. The
collaborative programmers pro-
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 107
duced better algorithms and code
in less time. As we all know,
poorly written code completed
quickly may in fact cause greater
delays in the overall development
and implementation time. Prior
to this experiment, standard
practice would limit the
resources that could be brought
to bear to speed up the process.
For example, additional program-
mers could be assigned to addi-
tional program modules.
However, programmers would
generally not be assigned to the
same program modules.
Group support technologies
may be employed to marshal pro-
grammer resources wherever they
exist to work on critical, time-
pressing systems.
The second scenario has to do
with improved quality in soft-
ware writing and testing. We
group together pseudocoding,
programming, and human-test-
ing procedures, such as struc-
tured walk-throughs, because
they involve closely related cog-
nitive skills, where programming
is the syntactical translation of
pseudocode into specific com-
puter languages and human-test-
ing procedures validate the
process. As program generators
and packages become more
prevalent, the greatest value
added by the programmer/analyst
may be as a contributor to team
development of innovative algo-
rithms and novel implementa-
tions of process and data models.
With companies increasingly
introducing new products,
crunch-time information systems
development will demand innov-
ative ways to produce high-qual-
ity systems in a short time. Pro-
fessional collaborative program-
mers who jointly developed
algorithms and code outper-
formed individual programmers
in this real-world field experi-
ment, a rarity in experimental
research. While the number of
subjects was small, this experi-
ment is not a single experiment,
but builds upon previous experi-
mental studies done with student
collaborative programmers. Some
may wonder at the luxury of
using collaborative programming
while many companies are expe-
riencing shortages of experienced
programmer/analysts. However,
the results raise intriguing ques-
tions. Can two average or less-
experienced workers collaborate
to perform tasks that may be too
challenging for each one alone?
Can a company bring a product
to market substantially earlier by
using collaborative program-
ming? Can collaborative pro-
gramming using worldwide
commodity programming
resources offer a competitive
1. Abdel-Hamid, T.K. Investigating the
cost/schedule trade-off in software develop-
ment. IEEE Software (Jan. 1990), 97–105.
2. Miller, R.G. Beyond Anova, Basics of Applied
Statistics. John Wiley and Sons, New York,
3. Warner, S. A dreamer finds the Expertise to
Develop an Idea. The Philadelphia Inquirer,
Philadelphia, Penn., Dec. 1, 1996, D1.
John T. Nosek (
is a professor in the Computer and
Information Sciences Department at
Temple University, Philadelphia.
© ACM 0002-0782/98/0300 $3.50
108 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
Technical Opinion
The designation “ACM Fellow” may be
conferred upon those ACM Members
who have distinguished themselves by
outstanding technical and professional
achievements in information technology,
who are current voting members of ACM
and have been voting members for the
preceding five years. Any voting member
of ACM may nominate another member
for this distinction. Nominations must
be received by the ACM Fellows Com-
mittee no later than August 1 of each
year and must be delivered to the Com-
mittee on forms provided for this purpose
(see below).
Nomination information organized by a
principal nominator includes:
1) excerpts from the candidate’s current
curriculum vitae, listing selected pub-
lications, patents, technical achieve-
ments, honors, and other awards.
2) a description of the work of the nomi-
nee, drawing attention to the contribu-
tions which merit designation as Fellow.
3) supporting endorsements from five
ACM Members.
ACM Fellows nomination forms and
endorsement forms may be obtained
from ACM by writing to:
ACM Fellows Nomination Committee
ACM Headquarters 1515 Broadway
New York, New York 10036-5701
The forms can also be accessed on the
Completed forms should be sent by
August 1, 1998
to one of the following:
ACM Fellows Committee
ACM Headquarters
1515 Broadway
New York, New York 10036-5701, USA
+1-212-869-0824 - fax
ACM Fellows
... A formal study [6] was conducted in a classroom setting between inexperienced middle school and high school programmers using the Intelligent Programming (IPRO) framework. A collaborative programming environment [53] was introduced where two programmers working jointly on the same algorithm and code. Nosek et al. performed a comparative study, which found that student programmers working collaboratively outperformed individual programmers for aspects of performance, which included readability, functionality, and time and satisfaction (as confidence and enjoyment). ...
... Similar to Nosek et al. [53], we logged how long it took the participants to complete the tasks. Both synchronous and asynchronous modes were faster than the control and the order effect was not significant. ...
... The unsurprising part of the findings was that both collaborative modes were faster than the control, which is typical to expect when the work is shared. This has been also known to be the case for code-based programming for decades [53]. It is worth acknowledging that a limitation in the form of a possibility may exist that the task itself could be a confounding factor since it varied across the levels of the independent variable. ...
Full-text available
Game development is a collective process in which a variety of different professionals from different backgrounds collaborate together not only by means of conversational interaction but also collaborative participation, one of which is programming. While collaborative and pair programming solutions exist for text-based programming languages, visual programming has not enjoyed as much attention. These solutions would not only address advanced forms of business communication among team members but could find their use in distance learning, which would have been useful during the pandemic. In our work, we propose a solution for collaborative behavioral animation of NPCs using behavior trees through synchronous and asynchronous modes of collaboration. We conducted a user study with 12 moderately skilled game development university students who were placed in groups of two and engaged in joint fixed behavior tree development tasks using the synchronous and asynchronous modes and auxiliary features of live preview, access and restoration of previous states from behavior tree history, conflict resolution, and instant messaging. Participants also completed a control task where no collaboration was involved and auxiliary features were not available. Feedback form Creativity Support Index, a self-developed questionnaire, and a semi-structured interview were collected. Additionally, task completion times were logged. The results indicate that the two collaborative modes provide expected improvement over the control condition. No significant differences were found between the two collaborative modes. However, the semi-structed interview revealed that the synchronous mode could be useful for quick prototyping, while the asynchronous mode - for most other situations. Supplementary information: The online version contains supplementary material available at 10.1007/s11042-022-12307-2.
... Such additions will change the state of the original notebook as presented to them and, if not documented, may lead to loss of information like the author who created the analysis. As a result, tools that support comprehension should also consider and facilitate different modes of collaboration (Nosek 1998). Moreover, the participants in our experiment also exhibited an interest in an assistant that would automatically present them with an initial set of analyses. ...
Full-text available
Data science is an exploratory and iterative process that often leads to complex and unstructured code. This code is usually poorly documented and, consequently, hard to understand by a third party. In this paper, we first collect empirical evidence for the non-linearity of data science code from real-world Jupyter notebooks, confirming the need for new approaches that aid in data science code interaction and comprehension. Second, we propose a visualisation method that elucidates implicit workflow information in data science code and assists data scientists in navigating the so-called garden of forking paths in non-linear code. The visualisation also provides information such as the rationale and the identification of the data science pipeline step based on cell annotations. We conducted a user experiment with data scientists to evaluate the proposed method, assessing the influence of (i) different workflow visualisations and (ii) cell annotations on code comprehension. Our results show that visualising the exploration helps the users obtain an overview of the notebook, significantly improving code comprehension. Furthermore, our qualitative analysis provides more insights into the difficulties faced during data science code comprehension.
... Collaborative programming is used to mean two programmers working jointly on the algorithm and code (Nosek, 1998). The previous research indicated that programmers (adult) programming collaboratively outperformed individual programmers, and they had greater confidence and enjoyed the process of problem-solving. ...
Full-text available
In order to cultivate children’s computational thinking, researchers have developed many excellent programming systems. Among them, the tangible programming systems combined with graphic output have been widely accepted because of the intuitive input method and diversified visual feedback. However, few of them support collaborative learning or programming. Little is known about children’s experiences, emotional states and behaviours on collaborative programming. Motivated by this gap, we present a novel tangible and collaborative enabled programming system named Lighters, which designed for children aged 7–10 and had two types of collaboration modes (block-based and role-based). We conducted an user experiment with 24 children, and collected physiological (EDA), questionnaire, video recording and interview data for analysis. Based on our experiment results, Lighters are effective in helping children learn to program collaboratively. In addition, Lighters can mobilize children’s positive emotions and enthusiasm to learn programming. Compared with block-based collaboration, role-based collaboration is more likely to stimulate children’s emotional states and has a better effect on learning programming.
... To improve programming skills, collaborative programming has been widely adopted in many schools. Collaborative programming, in which a group of learners work on the same code and complete programming tasks together, is considered an effective pedagogical approach (Nosek, 1998). Collaborative programming aims to improve learners' programming skills through writing code and refining programs with peers (Lu et al., 2017). ...
Full-text available
Programming skills have gained increasing attention in recent years because digital technologies have become an indispensable part of life. However, little is known about the roles of fade-in and fade-out scaffolding in online collaborative programming settings. To close this research gap, the present study aims to examine the roles of fade-in and fade-out scaffolding for novice programmers in online collaborative programming. A total of 90 undergraduate students participated in the exploratory study and were assigned to 15 fade-in groups and 15 fade-out groups. All of the participants completed the same programming task. The findings reveal that fade-in scaffolding can significantly improve collaborative knowledge building, programming skills, metacognitive behaviors, emotions, and collective efficacy. Goal setting, planning, monitoring and control, enacting strategies, and evaluation and reflection are identified as the crucial metacognitive behaviors. The main contribution of this exploratory study is to shed light on how to design and implement scaffolding for novice programmers.
Medical Device Software (MDS) defects have caused death of patients and continue to be the major cause of recalls of medical devices in the US and Europe. Despite various approaches proposed to address defects, dealing with defects in MDS is an increasingly difficult task as MDS has become more complex to support a growing number of functions. To increase quality in any software development project, it is essential that defects are identified and addressed quickly in the early stages of the software development life cycle. Agile methods have been advocated to increase software quality by minimising defects through their agile practices. However, agile methods on their own are deficient in satisfying the regulatory requirements for the MDS domain. Instead, the common approach is to integrate agile practices into the plan driven methods. Consequently, frameworks have been developed to help developers in the MDS domain to accrue the benefits of agile development while fulfilling regulatory requirements. Despite the adoption of agile practices in MDS development, it is still unclear as to which agile practice(s) is effective and how it is applied to address MDS defects. The purpose of this research is to identify agile practices that can assist in addressing defects in MDS development. This will help MDS developers to select the appropriate agile practice(s) to address defects.KeywordsMedical Device SoftwareAgile PracticesMedical Device Software DefectsSoftware FaultsMedical Device Recalls
Background and Context Computational thinking (CT) is a critical part of computing education in middle school. The existing practices of collaboration and collaborative design activities at this education level pairs well with CT practices, but this interaction has previously been under-explored in the existing literature. Objective In this study, we investigate whether students can learn CT through collaborative design activities and what patterns emerge over time and across instructional phases. Method We coded 6.3 hours of video observations for CT practices to create visualizations of 10-second segments of middle school students programming together. We use these visualizations to unpack nuances regarding how students demonstrate CT practices. Additionally, we provide three vignettes to highlight the interactions between students within and across the three instructional phases. Findings The findings suggest that middle school students can learn CT concepts and practices through collaborative design activities. The results demonstrate the patterns of CT practices and detail the transitions between each CT practice over time within a small group. Implications This study applies novel analysis techniques on student interaction data to examine CT through collaborative design. By bridging a CT framework with collaborative design activities, this study enhances the understanding of CT in collaborating, learning, and creating computing project-based designs.
his paper describes a qualitative study of how undergraduate students majoring in Information Technology perceive the effectiveness and evaluate the learning experience of pair-programming. The phenomenographic research approach was used to analyze student interviews and revealed 4 categories of descriptions: Effective Problem Solving, Participation, Enjoyment and Coding. Pair-programming as a teaching methodology was commonly perceived as a positive experience. The resulting outcome space maps a logical hierarchy of students’ conceptions of reality (categories of description). Findings of this research identify the factors that affect student engagement in a problem-solving process and can be used as a guiding principle on how to improve students’ learning experience of computer programming.
The COVID-19 pandemic raised the need to examine online learning methods also in young children. This study examined elementary school children’s performance and attitudes during and toward an online programming learning activity utilizing the pair-programming Agile method that may foster 21st-century skills, including collaboration and computational thinking. Forty 4th–6th grade children with basic programming knowledge of Scratch were randomly assigned to either a pair-programming or solo-programming condition. Overall, children in both conditions enjoyed the online learning activity and completed it successfully. In particular, pair-programming seemed to entail an extra benefit to girls who generally preferred working in pairs. Nevertheless, children in the pair condition took longer to complete all tasks, perceived the third task, which was completed individually, as more difficult, and were less active when their partner was more competent. Implications for post-COVID-19 learning are discussed.
The author has studied the effects of schedule compression or stretch-out on total project cost within a much broader effort to study and predict the dynamics of the entire development process. The resulting cost/schedule tradeoffs were examined. Much of this project involved developing a comprehensive system-dynamics model. He used the model to conduct three simulation experiments. (1) He investigated the effects of different levels of schedule compression and stretch-out on total project cost in man-days and compared the results to those reported in the literature. (2) He addressed the stealthy role undersizing plays in schedule compression. (3) He investigated how different levels of managerial commitment affect the project's final cost and completion time. The results of all three experiments are presented and discussed.< >
  • R G Miller
  • Anova
Miller, R.G. Beyond Anova, Basics of Applied Statistics. John Wiley and Sons, New York, 1986.
A dreamer finds the Expertise to Develop an Idea. The Philadelphia Inquirer
  • S Warner
Warner, S. A dreamer finds the Expertise to Develop an Idea. The Philadelphia Inquirer, Philadelphia, Penn., Dec. 1, 1996, D1.
  • R G Miller
  • Beyond Anova
Miller, R.G. Beyond Anova, Basics of Applied Statistics. John Wiley and Sons, New York, 1986.