Content uploaded by John T. Nosek
Author content
All content in this area was uploaded by John T. Nosek on Jan 03, 2015
Content may be subject to copyright.
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 105
Team programming usually
means coordinating efforts
of individual programmers
who divide up the programming
tasks for a large, complex system.
Collaborative programming is
used here to mean two program-
mers working jointly on the
same algorithm and code. Previ-
ous research indicates that stu-
dent programmers working
collaboratively outperformed
individual programmers. A fol-
low-up field experiment was con-
ducted using experienced
programmers who worked on a
challenging problem important
to their organization, in their
own environments, and with
their own equipment. To the sur-
prise of the managers and partici-
pants, all the teams outperformed
the individual programmers,
enjoyed the problem-solving
process more, and had greater
confidence in their solutions.
Description of the
Experiment
“Effective problem-solving per-
formance” is defined in terms of
(a) the readability of the pro-
posed solution, that is, the
degree to which the problem-
solving strategy could be deter-
mined from the subject’s work;
and (b) the functionality of the
proposed solution, that is, the
degree to which the strategy
accomplishes the objectives
stated in the problem descrip-
tion. The variables READABIL-
ITY and FUNCTIONALITY
were defined accordingly. Read-
ability is a component of overall
score since it is possible for a
subject to use a reasonable strat-
egy and to use programming lan-
guage structures appropriately
and yet fail to solve the problem
(in the sense of generating the
correct output). In such cases, the
programmer may have misinter-
preted the problem description
or overlooked a critical compo-
nent. Overall success on the
problem-solving effort is mea-
sured by a variable SCORE,
which is a simple sum of compo-
nent variables READABILITY
and FUNCTIONALITY. Based
on previous results, four predic-
tions were made:
1. Programmers working in pairs
will produce more readable
and functional solutions to a
programming problem than
will programmers working
alone.
2. Groups will take less time on
average to solve the problem
than individuals working
alone.
3. Programmers working in pairs
will express higher levels of
confidence about their work
(CONFID) and enjoyment of
the process (ENJOY) immedi-
ately following the problem-
solving session than will
programmers working alone.
(Positive feelings about the
problem-solving process can
affect performance. These feel-
ings were assessed immedi-
ately following the
problem-solving session by a
two-item questionnaire.)
4. Programmers with more years
of experience will perform bet-
ter than programmers with
fewer years of experience.
Aside from the pairing, condi-
tions and materials used in the
experiment were the same for
both experimental (teams) and
control (individuals) groups.
These subjects were 15 full-time
system programmers from a pro-
gram trading firm working on
system maintenance of three
Unix networks and a large data-
base running Sybase. They used
The Case for
Collaborative Programming
John T. Nosek
106 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
the X-window system with the C
language. The firm uses a very
large database to get information
on program trading. The subjects
were asked to write a script that
performs a database consistency
check (DBCC) with the output
for errors to be written to a file.
If no errors are found then the
script should
write to the out-
put file saying
that no errors
were encoun-
tered. Also, the
script should dis-
play on the
screen whether
the results of this
output are to be
mailed to all the
system adminis-
trators. To evaluate
the READ-
ABILITY vari-
able, the subjects
were asked to
properly com-
ment on each of the processes
within the script they were pro-
gramming. None of the subjects
had worked on this kind of prob-
lem before. DBCC checks are
considered so critical to the orga-
nization’s success and generally
beyond the skill of in-house pro-
grammers that outside consul-
tants are usually hired to perform
them. The subjects in the experi-
mental paired groups and the
control group of individuals were
randomly assigned using a set of
randomly generated numbers. All
subjects were given 45 minutes
to solve the problem.
During the problem-solving
phase, the pairs were allowed to
communicate freely with their
randomly assigned partner; sub-
jects working alone were asked to
work without any communica-
tion. To solve the problem, all
subjects used Sun Microsystems
SPARCstations, equipment with
which they were very familiar.
The post-treatment questionnaire
was given to each subject after he
or she had handed in all the
problem-solving materials. A
stopwatch was used to time each
group and individual. Each set of
materials was evaluated on the
degree to which it solved the
problem (FUNCTIONALITY)
and on the readability of the
solution (READABILITY).
The functionality score was
obtained as a sum of three parts
corresponding to the three out-
put requirements stipulated in
the problem description. The
READABILITY scores could
range between 0 to 2. READ-
ABILITY was assigned 0 for an
entirely unreadable solution, 2
for an entirely readable solution,
and 1 otherwise. FUNCTION-
ALITY scores ranged from 0 for
a solution which did not achieve
the goal at all, to 6 if the solu-
tion achieved the goal entirely.
With the exception of time, per-
formance measurements were the
average scores of two separate
graders. Overall SCORE was
obtained by summing READ-
ABILITY and FUNCTIONALITY,
a perfect overall score being 8.
Results
The t-test, a statisti-
cal technique
designed to compare
the means of two
small samples, was
used. The two-sided
t-test, which looks at
the ends of both
sides of a flattened
normal distribution
curve, was used
because this proce-
dure tests for results
that are in the pre-
dicted as well as in
the opposite direc-
tion expected, the
standard procedure in this kind
of experiment. The significance
level for all tests was set so that
the probability was less than 1 in
20 that results were due to
chance. Two graders separately
evaluated the problem solutions
with inter-grader reliability of
over 90%.
The table presents the means
and results of the two-sided t test
for performance and satisfaction
measurements, the test created
for small sample sizes. Predic-
tions 1 and 3 were supported.
For example, groups outper-
formed individuals, enjoyed their
collaborative problem-solving
process more, and were more
confident in their solutions. In
fact, all groups outperformed
Performance
Scores:
READABILITY
FUNCTIONALITY
SCORE
TIME (minutes)
Satisfaction
Measures:
CONFID
ENJOY
n = 5
1.40 (0.894)
4.20 (1.788)
5.60 (2.607)
42.60 (3.361)
n = 5
3.80 (2.049)
4.00 (1.870)
n = 5
2.00 (0.000)
5.60 (0.547)*
7.60 (0.547)*
30.20 (1.923)
n = 10
6.50 (0.500)*
6.60 (0.418)*
Comparison of Individual and Team Measurements
Variable
Control Group
(Individuals)
mean (st. dev.)
Experimental Group
(Teams)
mean (st. dev.)
*less than 1 in 20 that results are due to chance
Technical Opinion
individuals. Although the aver-
age time for completion was
more than 12 minutes (41%)
longer for individuals, prediction
2 was not statistically supported
because there is more than a 1 in
20 chance that this is due to
chance. Prediction 4 was sup-
ported. Experience and problem
scores were highly correlated for
both the control (73.5%) and the
experimental (87.2%) groups.
Discussion
The results provide additional
evidence that collaboration
improves the problem-solving
process. Several aspects of this
experiment make the results sig-
nificant. First, there may be a
tendency to dismiss results using
small groups. However, it is rare,
if not impossible, to obtain the
time and cooperation of experi-
enced programmers to perform
the identical task. Much experi-
mental research is automatically
dismissed because it is done with
students, who are more plentiful
but who do not possess the skills
and knowledge of experienced
programmers. The subjects of
this experiment were experienced
programmers, working on an
important, challenging problem,
in a natural setting. Second,
because of the inherent bias
against small samples in statisti-
cal models, Miller [2]emphasizes
that a small sample, with a prob-
ability of only 1 in 20 that
results were due to chance, may
be far more striking than results
from larger sample sizes that have
the same or lower probability.
Third, this experiment was an
extension of previous ones, not
just a one-time event.
The results are consistent with
previous experiments that used
simpler, canned problems. The
qualitative data also provides
some interesting insights. The
majority of programmers were
somewhat skeptical of the value
of collaboration in working on
the same algorithm/program
module and thought that the
process would not be enjoyable.
However, as the results indicate,
and supported by their com-
ments, collaboration did improve
their performance and they
enjoyed their efforts. One pro-
gramming pair experienced a
“qualitative jump” as compared
to the other groups and individu-
als in the experiment. To the
amazement of the manager, their
programming solution was better
than previous scripts written for
the company. It is costly to run
these scripts and efficiently writ-
ten scripts are considered so diffi-
cult to create that the company
hires expert outside consultants
to write them. However, the
script written by this program-
ming team was twice as efficient
as previously purchased scripts.
Some may question the
value of these results if the
collaborators do not per-
form “twice” as well as individu-
als, at least in the amount of time
spent. For instance, if the collab-
orators did not perform the task
in half the time it takes an indi-
vidual, it would still be more
expensive to employ two pro-
grammers. However, there are at
least two scenarios where some
improved performance over what
is expected or possible by a single
programmer may be the goal: (1)
speeding up development and (2)
improving software quality.
In the first case, bringing a
product to market a month ear-
lier can mean a competitive edge,
or gaining, in some cases, even
survival. As organizations find it
increasingly difficult to produce
current products more efficiently,
they are escalating the number of
new products to market, where
the profit margins are greater.
According to the Product Devel-
opment and Management Associ-
ation, new products are expected
to account for 37% of total sales
by 2000, up from 28% for the
five years that ended in 1995 [3].
A reduced product life cycle,
combined with an increased
informational component of the
product, is a growing, critical
problem for information systems
departments:
“Crunch mode is not a matter
of opportunity—it’s a matter of
survival.… The ability to get
working software quickly into
the hands of users will be charac-
teristic of successful data-process-
ing organizations for the
foreseeable future. Groups that
can produce and install software
systems within tight time frames
will prosper. Those [that]
can’t will fail and, in some cases,
they will bring the enterprises of
which they are a part down with
them. Fast response to changing
information-processing require-
ments is a necessity in today’s
world.” [1]
Additionally, while there was
more than a 1 in 20 probability
that the time saved by groups
was due to chance, the groups
completed the task 40% more
quickly and more effectively. The
collaborative programmers pro-
COMMUNICATIONS OF THE ACM March 1998/Vol. 41, No. 3 107
duced better algorithms and code
in less time. As we all know,
poorly written code completed
quickly may in fact cause greater
delays in the overall development
and implementation time. Prior
to this experiment, standard
practice would limit the
resources that could be brought
to bear to speed up the process.
For example, additional program-
mers could be assigned to addi-
tional program modules.
However, programmers would
generally not be assigned to the
same program modules.
Group support technologies
may be employed to marshal pro-
grammer resources wherever they
exist to work on critical, time-
pressing systems.
The second scenario has to do
with improved quality in soft-
ware writing and testing. We
group together pseudocoding,
programming, and human-test-
ing procedures, such as struc-
tured walk-throughs, because
they involve closely related cog-
nitive skills, where programming
is the syntactical translation of
pseudocode into specific com-
puter languages and human-test-
ing procedures validate the
process. As program generators
and packages become more
prevalent, the greatest value
added by the programmer/analyst
may be as a contributor to team
development of innovative algo-
rithms and novel implementa-
tions of process and data models.
Conclusion
With companies increasingly
introducing new products,
crunch-time information systems
development will demand innov-
ative ways to produce high-qual-
ity systems in a short time. Pro-
fessional collaborative program-
mers who jointly developed
algorithms and code outper-
formed individual programmers
in this real-world field experi-
ment, a rarity in experimental
research. While the number of
subjects was small, this experi-
ment is not a single experiment,
but builds upon previous experi-
mental studies done with student
collaborative programmers. Some
may wonder at the luxury of
using collaborative programming
while many companies are expe-
riencing shortages of experienced
programmer/analysts. However,
the results raise intriguing ques-
tions. Can two average or less-
experienced workers collaborate
to perform tasks that may be too
challenging for each one alone?
Can a company bring a product
to market substantially earlier by
using collaborative program-
ming? Can collaborative pro-
gramming using worldwide
commodity programming
resources offer a competitive
edge?
References
1. Abdel-Hamid, T.K. Investigating the
cost/schedule trade-off in software develop-
ment. IEEE Software (Jan. 1990), 97–105.
2. Miller, R.G. Beyond Anova, Basics of Applied
Statistics. John Wiley and Sons, New York,
1986.
3. Warner, S. A dreamer finds the Expertise to
Develop an Idea. The Philadelphia Inquirer,
Philadelphia, Penn., Dec. 1, 1996, D1.
John T. Nosek (nosek@cis.temple.edu)
is a professor in the Computer and
Information Sciences Department at
Temple University, Philadelphia.
© ACM 0002-0782/98/0300 $3.50
c
108 March 1998/Vol. 41, No. 3 COMMUNICATIONS OF THE ACM
Technical Opinion
CALL FOR 1999
ACM FELLOWS NOMINATIONS
The designation “ACM Fellow” may be
conferred upon those ACM Members
who have distinguished themselves by
outstanding technical and professional
achievements in information technology,
who are current voting members of ACM
and have been voting members for the
preceding five years. Any voting member
of ACM may nominate another member
for this distinction. Nominations must
be received by the ACM Fellows Com-
mittee no later than August 1 of each
year and must be delivered to the Com-
mittee on forms provided for this purpose
(see below).
Nomination information organized by a
principal nominator includes:
1) excerpts from the candidate’s current
curriculum vitae, listing selected pub-
lications, patents, technical achieve-
ments, honors, and other awards.
2) a description of the work of the nomi-
nee, drawing attention to the contribu-
tions which merit designation as Fellow.
3) supporting endorsements from five
ACM Members.
ACM Fellows nomination forms and
endorsement forms may be obtained
from ACM by writing to:
ACM Fellows Nomination Committee
ACM Headquarters 1515 Broadway
New York, New York 10036-5701
nominate-fellows@acm.org
The forms can also be accessed on the
following:
http://www.acm.org/awards/fellows/
nomination_packet.html
Completed forms should be sent by
August 1, 1998
to one of the following:
ACM Fellows Committee
ACM Headquarters
1515 Broadway
New York, New York 10036-5701, USA
or
nominate-fellows@acm.org
or
+1-212-869-0824 - fax
ACM Fellows
