Developing and Evaluating Software Engineering
Department of Computer Science
University of Auckland
Auckland, New Zealand
Abstract - A process theory is an explanation of how an entity
changes and develops. While software engineering is fundamen-
tally concerned with how software artifacts change and develop,
little research explicitly builds and empirically evaluates software
engineering process theories. This lack of theory obstructs scien-
tific consensus by focusing the academic community on methods.
Methods inevitably oversimplify and over-rationalize reality,
obfuscating crucial phenomena including uncertainty, problem
framing and illusory requirements. Better process theories are
therefore needed to ground software engineering in empirical
reality. However, poor understanding of process theory issues
impedes research and publication. This paper therefore attempts
to clarify the nature and types of process theories, explore their
development and provide specific guidance for their empirically
Index Terms—Methodology, process theory, reviewing, check-
I. WHAT IS A PROCESS THEORY?
The purpose of this paper is to adapt existing guidance on
developing and evaluating process theory research to the soft-
ware engineering (SE) context. A process theory is a system of
ideas intended to explain (and possibly to describe, to predict
or to analyze) how an entity changes and develops (cf. ,
). For example, the theory that organisms evolve through
mutation and natural selection (Fig. I) is a process theory be-
cause it explains how species change and develop over time. It
explains a higher level of process (evolution) by dividing it
into several lower-level component processes (mutation, selec-
tion and reproduction).
Process theories may be contrasted with variance theories,
e.g., The Unified Theory of the Adoption and Use of Technol-
ogy  (Fig. II). A variance theory explains and predicts the
variance in a dependent variable (e.g. Use Behavior) using
independent variables (e.g. Performance Expectancy), mediat-
ing variables (e.g. Behavioral Intention) and moderating vari-
ables (e.g. Gender). While a variance theory comprises varia-
bles and causal relationships, a process theory may include
activities, actors, phases, steps, subprocesses and diverse non-
Furthermore, process theories are not methods. A meth-
od(ology) prescribes while a process theory explains. Scrum
, Lean  and the V-model  are methods because they
prescribe specific practices, techniques, tools or sequences,
which are ostensibly better than their alternatives. A process
theory, contrastingly explains how something actually occurs,
regardless of effectiveness. While a method claims this way
will work; a process theory claims it happens this way. The
theory of natural selection posits that evolution follows a spe-
cific pattern, not that this pattern is particularly efficient.
Scrum similarly claims that organizing development into time-
boxed sprints is wise not that everyone does so .
Moreover, process theory is not a synonym for process
model. While all theories are models in a sense, process theory
research differs significantly from process model research. “A
process model is an abstract description of an actual or pro-
posed process that represents selected process elements that
are considered important to the purpose of the model”[8, p.
76]. A process model describes a specific process while a pro-
cess theory explains a process in general.
FIGURE I. EVOLUTION THROUGH MUTATION AND SELECTION
Note: by Elembis - Licensed under Creative Commons Attribution-
Share Alike 3.0 via
For example, Fig. III shows how a manufacturer might
build custom components. It does not claim that all manufac-
turers act this way or that a manufacturer acts this way all of
the time. The theory of natural selection, in contrast, posits
that all complex life arises through mutation, selection and
reproduction. Process models (like methods) often describe
one or a few sequences while process theories seek to encom-
pass all of the ways an entity can change.
Finally, a process theory is not a set of steps. While soft-
ware engineers often think of a process as a set of steps in a
FIGURE II. THE UNIFIED THEORY OF THE ADOPTION AND USE OF TECHNOLOGY (FROM )
FIGURE III. MADE-TO-ORDER PROCESS MODEL (FROM )
specific order (an algorithm)process theories do not necessari-
ly posit sequential orders or divide processes into steps. Think-
ing of selection, retention and reproduction as evolutionary
steps is deeply misleading. These activities occur chaotically
and in parallel across organism populations rather than in a
disciplined cycle (see Section V).
The paper proceeds by clarifying why process theories are
needed specifically in SE, defining the four types of process
theories and providing specific guidance for developing and
testing process theories and, by extension, for writing and re-
viewing process theory papers. The paper also specifically
addresses problems with lifecycle process theories in SE.
II. WHY DO WE NEED PROCESS THEORIES?
Despite numerous calls for more theory in Software Engi-
neering (e.g. -), theories of SE processes remain elu-
sive. SE needs process theories for three main reasons.
1) Without process theories, SE academics conceptualize
their research using concepts from methods, which inevitably
oversimplify and over-rationalize reality , . Research-
ers indoctrinated in methods may therefore struggle to recog-
nize phenomena that are observable in real projects but down-
played in the methods literature, e.g. problem framing ,
illusory requirements  and coevolution , . Fur-
thermore, without process theories, researchers base surveys
and qualitative coding schemes on the concepts and assump-
tions of methods (e.g. ). This may introduce insidious,
poorly-understood biases; for instance, a survey question
might ask, during which phase of the project were the re-
quirements agreed –
analysis, design, coding, testing, imple-
mentation or maintenance? This presumes: 1) all projects have
the same phases; 2) project phases map to those listed; 3) all
projects have requirements; 4) requirements are always
agreed. In summary, process theories, which encapsulate em-
pirical, explanatory knowledge, counterbalance methods,
which provide prescriptions.
2) Absence of communicable theories of SE processes
hinders SE education. Educators need some basis for describ-
ing SE to students. Without process theories, they continue to
base curricula on methods , , . This gives students
an idealistic, oversimplified, incorrect view of the great diver-
sity of SE approaches and projects.
3) Lack of testable process theories permit pseudoscien-
tific nonsense to infest both the academic and popular SE lit-
erature. For example, the Gartner Hype Cycle posits that in-
formation technologies exhibit a common pattern of visibility
as they mature. Simply comparing the hype cycle in subse-
quent years reveals that many technologies do not follow the
posited path. However, without a rigorously evaluated theory
of technology visibility to displace Gartner’s nonsense, the
Hype Cycle continues to enjoy substantial media coverage and
SE-related processes that may benefit from a process theo-
ry approach include the following.
• The process of forming SE teams
• The process through which agility arises
• The process of evaluating the coverage of a test suite
• The process of integrating a new developer into an
• The process of communicating a software design to a
• The process of selecting an architectural pattern (e.g.
tiered, service-oriented, model-view controller)
• The process by which open source projects gain
momentum, contributors and market share
• The process of finding a bug
• The process of finding and eliminating security
• The process of problem/solution coevolution
• The software development process in general
III. TYPES OF PROCESS THEORIES
At least four types of process theories are evident in the lit-
erature –teleological, dialectic, evolutionary and lifecycle 
In a teleological process theory, an agent “constructs an
envisioned end state, takes action to reach it and monitors the
progress”[2, p. 518]. In other words, teleological theories ex-
plain the behavior of agents taking actions to reach goals, but
the agent chooses its own sequence of actions. Teleological
process theories are very common in the social sciences .
Sensemaking-Coevolution-Implementation Theory (SCI) is
a teleological theory that explains how a cohesive develop-
ment team builds complex software systems (Fig. IV). It posits
that development teams engage in three basic activities: 1)
making sense of an ambiguous, problematic context (Sense-
making); 2) rapidly oscillating between ideas about the context
and ideas about the space of possible design artifacts (Coevo-
lution) and building the system (Implementation) , .
These activities may occur serially (in any order) or in paral-
In a dialectic process theory, “stability and change are ex-
plained by reference to the balance of power between oppos-
ing entities”[2, p. 517]. Possible entities include people,
teams, organizations , activities and social forces. Dialectic
process theories therefore posit two or more conflicting enti-
ties and model change in inter-entity power. These theories are
rooted in the argumentative methods of classical philosophy.
For example, Allison and Merali  proposed a dialectic
theory of software process improvement (Fig. V). It posits a
dialectic interplay between software development and soft-
ware process improvement, where each informs the other.
Both process and product metamorphose over time, changing
and being changed by their surrounding context.
In an evolutionary process theory, a population undergoes
structural changes through variation (producing new entities
through chance occurrences), selection (the preservation of
entities with higher fitness and elimination of those with lower
fitness) and retention (forces (including inertia and persis-
tence) perpetuate and maintain certain organizational
forms” (p. 518)).
For example, the Problem-Design Exploration Model 
is an evolutionary process theory that explains how genetic
algorithms may be used to design systems (Fig. VI). The prob-
lem space (constraints) and solution space (designs) are mod-
eled as coevolving populations each affecting the other.
A lifecycle theory “is a unitary sequence (it follows a sin-
gle sequence of stages or phases), which is cumulative (char-
acteristics acquired in earlier stages are retained in later stag-
es) and conjunctive (the stages are related such that they de-
rive from a common underlying process)”[2, p. 515]. This
progression occurs because “the trajectory to the final end
state is prefigured and requires a particular historical sequence
of events”[2, p. 515]. Lifecycle theories have their roots in
biological life cycles. Despite their name, lifecycle theories
need not contain loops.
If it were a theory, the Waterfall Model  either the
linear version Royce condemned or the more iterative version
he proposed would be a lifecycle theory. Outside of SE, the
nitrogen cycle and continental drift are lifecycle theories.
While process theories do not posit causal relationships be-
tween constructs, they may adopt a particular approach to cau-
sality . Evolutionary process theories, for instance, adopt a
probabilistic approach to causality in that entities with higher
fitness have a greater probability of surviving and reproduc-
ing. Teleological process theories adopt teleological causality,
i.e., actions are caused by the choices of agents with free will.
IV. DEVELOPING PROCESS THEORIES
When developing a process theory, the following four
questions may be useful.
1. What entities are changing? Process theories explain how
an entity changes, so what is the entity whose change pro-
cess we want to explain? Example entities include soft-
ware artifacts, tests, documentation, projects, teams, indi-
viduals, and organizations.
2. Is there a pattern at all? Some processes are so chaotic,
lawless or context-dependent that they lack sufficient
common structure on which to base a process theory. In-
formation systems development may even be such a pro-
cess . It is crucial to recognize our propensity for
apophenia seeing patterns in randomness.
3. If there is a pattern, what type is it? If we find a person or
cohesive team pursuing one or a few goals through nu-
merous activities, we may adopt a teleological approach.
If we find a small number of things engaged in a power
struggle, or several conflicting forces, a dialectic approach
may help. If we are trying to understand how the structure
of a large set of entities changes over time, an evolution-
ary approach my be preferred. If we discover a common
sequence of phases, we may take a lifecycle approach. Fi-
nally, if we find two or more of the above, we may adopt
a hybrid approach.
4. What are the specific concepts and relationships of the
theory? For a teleological theory, what are the agent, its
goals and activities? For a dialectic theory, what are the
entities, what is the nature of their power structure and
how do they conflict? For an evolutionary theory, what is
the population, how do entities exhibit variation, selection
and retention, and what is the fitness function? For a
lifecycle theory, what are the stages or phases and what is
the sequence. For a hybrid theory, which types are we hy-
bridizing and how do the elements from the different
types relate to each other?
FIGURE IV. SENSEMAKING-COEVOLUTION-IMPLEMENTATION THEORY
(ADAPTED FROM )
FIGURE V. AN EMERGENT VIEW OF SOFTWARE PROCESS IMPROVEMENT
FIGURE VI. PROBLEM-DESIGN EXPLORATION MODEL (FROM 
To investigate these questions, we can apply at least three
theory development strategies, as follows.
1. Synthesize a process theory using existing theories or
models from SE or reference disciplines. Some SE pro-
cesses are special cases of more general processes. For
example, making software design decisions is a special
case of decision making. Therefore, it may be possible to
synthesize a process theory of software design decision
making by extending or adapting a more general theory of
decision making (e.g. ). General theories of negotia-
tion, group mind, contracts, learning, communication,
creativity, problem structuring and crime may be similarly
applied to SE special cases. SCI, for example, was devel-
oped by refining an older model of the self-conscious
design process , . Benefits of the synthesizing ap-
proach include speed (compared to months or years of
field work), natural linkages to existing theories and pos-
sibly greater scientific rigor (in that, working with rigor-
ous theories may encourage us to produce more rigorous
theories). Source theories however may be incorrect in
general or not translate well to SE.
2. Develop theory based on field studies. Another approach
is to develop a theory from extensive field work, e.g., eth-
nography or interpretivist case studies. As many SE pro-
cesses are observable, the researcher can observe the pro-
cess, consult with its participants and develop a theory in
much the same way we develop conceptual models. Pro-
cess coding  may be especially helpful here. The pri-
mary advantage of this approach is that it encourages deep
immersion in the data, which may protect researchers
from oversimplifying or overrationalizing the process.
However, this approach is time-consuming and interpre-
tivist methods focus on participants’perceptions of reali-
ty, which may be systematically biased - or simp-
ly wrong. Moreover, over-reliance on interviews and fo-
cus groups can be detrimental (see below).
3. Use Grounded Theory. Grounded theory refers to a col-
lection of similar methodologies originally developed by
Glaser, Strauss and Corbin  for generating theory
from mostly qualitative data. Grounded theory is not a
synonym for qualitative research and differs substantially
from ethnography and case studies. Researchers who
adopt grounded theory should read at least one seminal
book on grounded theory and clearly indicate whether
they are adopting Glaser’s approach, Strauss and Corbin’s
approach or constructivist grounded theory . Ground-
ed theory benefits from a rigorous and systematic analyti-
cal approach and high potential for immersion; however,
like ethnography, it is very time consuming.
Furthermore, some approaches are highly unsuited to de-
veloping process theories, including the following.
1. Do not base process theories on methods. A method is a
system of prescriptions for doing something effectively. A
process theory is a system of ideas for explaining how
something happens. Methods are inappropriate founda-
tions for process theories since process theories explain
both effective and ineffective behavior , . Adapt-
ing a method into a theory is therefore likely to overra-
tionalize reality. To overrationalize reality means to pre-
sent it as we think it should be rather than as it is. For ex-
ample, presenting something messy as clean, presenting
something chaotic as lawful, presenting something unrea-
sonable as reasonable. When economists model humans
as rational agents, they are over-rationalizing reality.
2. Do not base process theories predominately on expert
opinion. An expert is one who is skilled in a specific do-
main. However, expertise does not inoculate people
against cognitive and perceptual biases , . For ex-
ample, experts are susceptible to system justification –the
tendency to irrationally defend the status quo , .
Furthermore, skill in software development or managing
software projects, for example, does not imply expertise
in theorizing software processes. An expert programmer
probably spends much of his or her time programming,
not reflecting on the nature of programming processes or
systematically recording process variations between dif-
ferent kinds of programmers in different kinds of projects.
Unless the experts in question derive their expertise from
lifetimes of systematic, rigorous research and meta-
analysis, expert opinion is intrinsically anecdotal and un-
reliable. That said, experts may be an excellent source of
data; they are simply not substitutes for observation.
A related pitfall in process theory development is the ten-
dency to blend explanation and description, or combine a theo-
ry with a method. Simultaneously modeling how something
should happen and how it actually happens is difficult and
confusing. Therefore, avoid adding any prescriptive elements
to a process theory.
In summary, three ways of developing a process theory are
to synthesize it from existing theories, to base it on field stud-
ies or to use grounded theory. Process theories should not be
based on methods or include prescriptions. While expert opin-
ion may be helpful, it is no substitute for empirical data. Dur-
ing theory development, researchers should ask what is chang-
ing, is there really a pattern, and if so, what kind of pattern?
before determining the theory’s concepts and relationships.
V. THE LIFECYCLE PROBLEM
At this juncture, a slight tangent is needed to address SE’s
lifecycle problem. Many SE researchers, consultants and prac-
titioners emphasize lifecycle models including the Waterfall
Model , Systems Development Lifecycle , V-Model
, and Spiral Model . The proliferation of lifecycle mod-
els superficially suggests adopting a lifecycle approach for
theorizing SE processes. However, lifecycle models are intrin-
sically inappropriate for theorizing most human behavior.
Lifecycle theories posit that the configuration of a process
is immutable. SE processes, meanwhile, involve human agents
who can choose to change their goals, change their activity
sequence, invent new activities, give up part way through a
process and generally take unexpected actions. Modeling the
process of fixing nitrogen as a lifecycle is useful partially be-
cause nitrogen will not spontaneously refuse to interact with
plants or invent a new method of bonding directly to the soil.
Human actors, in contrast, can spontaneously change their
processes. Theorizing software development, for example, as a
defined sequence of phases is problematic because software is
built by people who can decide to take different actions in a
different order. This possibility of spontaneous variation is
incommensurate with a lifecycle approach.
Put another way, mathematician George Polya modeled
problem solving as a four-stage process: 1) understand the
problem, 2) devise a plan, 3) carry out the plan, 4) check the
result . His work is not only a parsimonious lifecycle for
mathematical problem-solving but also one of the clearest ex-
positions of the planning model of human action . Howev-
er, Polya’s model would make a terrible foundation for an SE
process theory because, in many projects, people act without
ever agreeing on a clear problem , expert designers do not
separate thinking from doing , and most human action is
better understood as improvisation rather than enacting a plan
. In short, lifecycle theories and improvisation are incom-
patible. Nitrogen does not improvise.
Moreover, the rise of Agile methods – specifically the prin-
ciple of “responding to change over following a plan”  –
reflects a growing understanding of the limitations of lifecy-
cles. Unlike previous frameworks including the Rational Uni-
fied Process , Agile methods including Scrum  and
Lean  present sets of principles, values and practices rather
than steps, stages and phases.
Contrastingly, teleological, evolutionary and dialectic ap-
proaches do not make assumptions so misaligned with SE pro-
cesses. Teleological theories posit agents (e.g. SE teams) who
form goals (e.g. eliminate bugs). Evolutionary theories posit
populations of competing entities (e.g. mobile applications,
open source projects, design concepts). Dialectic theories posit
conflict between groups with differing levels of power (e.g.
developers vs. managers, producers vs. consumers). Conse-
quently, of the four process theory approaches, lifecycle ap-
pears least suited to theorizing SE processes.
In summary, while lifecycle theories may explain deter-
ministic processes (e.g. how compilers work), lifecycle theo-
ries of social processes are fundamentally flawed. No matter
what phases and sequence we hypothesize, particular develop-
ers can choose to take different actions in a different order.
VI. HOW TO EVALUATE PROCESS THEORIES
A. Compare Rival Theories
This section extends and clarifies previous guidance for
empirically evaluating process theories, which emphasizes
questionnaires and field studies , . More recent work
has explored process theory evaluation in communication 
and decision-making .
Whether an evaluation method is appropriate for a theory
depends on the kind of pattern the theory posits. For example,
since the hypothesis “all swans are white”is existential rather
than causal, experimental methods would be inappropriate.
Process and variance theories claim different kinds of patterns.
Therefore, they need different kinds of evaluation –both em-
pirically and conceptually.
Randomized controlled trials are not appropriate for testing
process theories. As process theories do not have independent
or dependent variables, an experiment where the investigator
manipulates the independent variable(s) and observes the de-
pendent variable(s) while controlling everything else to estab-
lish a causal relationship does not make sense.
More generally, null hypothesis testing, where a hypothesis
(e.g. X causes Y) is tested against a null hypothesis (e.g. X and
Y are unrelated), is inappropriate for process theory testing.
Null hypothesis testing is problematic in at least three ways.
1. Null hypothesis testing is susceptible to confirmation bias.
“Confirmation bias means that information is searched
for, interpreted, and remembered in such a way that it sys-
tematically impedes the possibility that the hypothesis
could be rejected”[55, p. 79]. Confirmation bias is related
to the tendency in questionnaire-based research, to uncon-
sciously exploit response bias for positive results.
2. The null hypothesis is often a straw man. In this context, a
straw man is a weak proposition presented only to be re-
futed. For example, the Technology Adoption Model hy-
pothesizes that the perceived usefulness of a technology
leads to intention to use it . The null hypothesis that
perceived usefulness and intention to use are unrelated
is a straw man.
3. Null hypothesis testing answers the wrong question. In
field studies, one will usually make some observations
consistent with the theory (unless it is a straw man) and
some observations incongruous with the theory. The key
question is not “is the theory correct?”but “are we on the
right track?”Null hypothesis testing does not tell us
whether we are on the right track.
Consequently, Platt proposes the strong inference model,
which simply emphasizes comparing several plausible alterna-
tive hypotheses, and argues that it leads to faster scientific
progress . Comparing rival theories is now commonly
recommended for case study research  and some have
even argued that theory testing is essentially comparative .
SCI, for example, posits that designers engage in sense-
making, i.e., giving meaning to an ambiguous situation. The
null hypothesis, “situations are never ambiguous or designers
do not make sense of them”, is a straw man. Cherry-picking
observations to support sensemaking would be straightfor-
ward. Moreover, a field study is likely to produce observations
of both clear situations participants take for granted and am-
biguous situations participants have to make sense of. The
relevant question is not whether we can view some design
activities as sensemaking, but whether sensemaking has more
explanatory power than an alternative concept, e.g., analyzing.
SCI was therefore tested against a rival theory, the Function-
Behavior-Structure Framework , .
Selecting an appropriate rival may be challenging. If sev-
eral potential rivals are evident, the choice is inherently sub-
jective and reviewers may complain that the rival is arbitrary.
While authors should attempt to justify their choice of rival
theory; reviewers should not criticize this choice unless they
can name and cite a different rival and give definitive reasons
for its superior appropriateness. Imperfect rivals are often bet-
ter than no rivals.
Comparative evaluation may also be augmented by team-
based research. Having one team member seek evidence for
Theory A while another seeks evidence for Theory B may
ameliorate confirmation bias. The process of reconciling con-
flicting interpretations through discussion may furthermore
spur more nuanced analysis and reflection.
In summary, comparing rival process theories is simply
more tractable and defensible than null hypothesis testing.
Both field students and questionnaire studies support compar-
ing rival theories.
B. Use Observational Studies
Observational studies may provide extensive evidence with
which to discriminate between rival theories. To be clear,
while interpretivist field studies were recommended for theory
building, positivist observational studies are recommended for
Data collection depends on the specific approach. In a field
study, data may include field notes, interview transcripts, cop-
ies of relevant documents, diagrams and email, segments of
source code, code commit logs, screen shots, photographs of
the physical environment and video of meetings or activities
cf. . In a lab-based simulation, data may include video of
the simulation, copies of artifacts produced and the written or
oral reflections of participants. In a think-aloud protocol study
(where participants are asked to think aloud during a task) data
would include the protocol transcripts (cf. , ). In a
retrospective analysis, data would consist of previous case
data (e.g. transcripts, documents) or case narratives (i.e. de-
scriptions of cases written by researchers). In a discourse anal-
ysis, the data would consist of a corpus of relevant texts (e.g.
journal articles, project plans). In all cases, observational stud-
ies may produce both quantitative data (e.g. bug counts, num-
ber of unit tests, number of code commits, time spent in meet-
ings, word frequencies) and qualitative data (e.g. field notes,
interview transcripts, software documentation).
For data analysis, researchers may choose between two
basic strategies, or employ both serially or in parallel.
In one strategy, the researcher develops separate coding
schemes for each theory (Table I). Each coding scheme lists
all of the components (e.g. phases, activities, actors) and rela-
tionships (e.g. sequence, dependency) of the theory. The cod-
ing scheme provides space for evidence both for and against
each component and relationship. It may be useful to brain-
storm examples of each evidence category before data collec-
tion begins. Again, evidence may be qualitative or quantita-
tive. The researcher organizes the data into this coding scheme
and then weighs the evidence to reach a conclusion about
which theory is better supported. While the evidence may bet-
ter support the proposed theory or the rival theory, other con-
clusions are possible, e.g., that both theories are deficient or
both are partly correct and can be merged. This strategy pro-
motes thorough consideration of the data but is messy and
In the other strategy, the researcher first lists as many
propositions as possible for both theories. The researcher then
identifies the conflicting propositions, i.e., those where the two
theories make contrasting predictions. Next, the researcher
creates a coding scheme around the conflicting propositions
(Table II). Again, brainstorming examples for each category
before data collection begins may help. The researcher organ-
izes the data into this coding scheme. For each proposition, the
researcher decides which theory is favored by the evidence.
Ideally, one theory is favored on most or all propositions and
is therefore supported. This strategy relinquishes some degree
of interpretive depth in favor of a cleaner, simpler analysis.
It is also possible to apply both strategies simultaneously.
This may force the researcher to consider observations more
deeply and from different angles. However, I personally found
it challenging to conceptualize observations in terms of two
different coding schemes, especially when events unfolded
rapidly in situ.
In any case, it is both normal and desirable to adjust coding
schemes as data collection progresses. As the researcher is
TABLE I. Coding Scheme Template (First Strategy)
TABLE II. Coding Scheme Template (Second Strategy)
Evidence for Theory A
Evidence for Theory B
immersed in the observational context and begins classifying
data, new insights may emerge, motivating refinements to the
C. Use Questionnaire Studies
Questionnaires may also be used to discriminate between
Concerning data collection, sampling is hindered by the
fact that SE researchers have no comprehensive lists of soft-
ware developers, projects or companies. Two strategies are
evident: 1) randomly sample from a specific community that
does have a population list (e.g. Source Forge); 2) try to max-
imize the diversity of a convenience sample. Neither approach
supports statistical generalization to the population of interest.
Both approaches have benefits and drawbacks; reviewers
should accept both as no superior option is available without a
defensible population list. A more ambitious approach might
utilize respondent-driven sampling  –a method of ad-
dressing bias in referral-chain (snowball) samples.
Two instrument development strategies (analogous to the
two data analysis strategies above) are evident.
In one strategy, the questionnaire comprises both open-
ended questions about participant’s perceptions of the process
and closed-ended questions about specific propositions of the
theories. Examples of open-ended questions include “How are
programming tasks distributed among team members in your
current project?”and “please list all of the work activities (e.g.
writing code, reading documentation) that you remember do-
ing on your last day of work.”Examples of closed-ended
questions include “On a scale of 1 (strongly disagree) to 9
(strongly agree), to what extent do you agree with the follow-
ing: I) our project goals were clear from the beginning; II)
programming tasks sometimes require subject judgments; III)
we often face impractical deadlines.”Open-ended questions
are designed to elicit predominately qualitative data, which
can be classified using a coding scheme as in the first observa-
tional strategy / Table I. Closed-ended are designed to directly
test specific theory propositions. For example, a median re-
sponse of 7 (agree) to “Our project goals were clear from the
beginning”would be evidence against the sensemaking propo-
sition of SCI. Adopting this strategy may produce compara-
tively more data per respondent; however, the length of the
instrument may reduce response rates.
In the other strategy, the researcher again identifies con-
flicting propositions. The researcher then generates numerous
questions that reflect these differences. Specifically, one can
generate bipolar scales (including Likert and semantic differ-
ential scales) where one pole is consistent with Theory A and
the other pole is consistent with Theory B, e.g., The goals of
my current project were...
a) ... completely provided by the client (Rival Theory)
mostly provided by the client
c) ... determined equally by the client and development team
d) ... mostly determined by the development team
e) ... completely determined by the development team (SCI)
Validating these kinds of questionnaires fundamentally dif-
fers from validating a variance theory testing questionnaire. In
social science, variance theories usually posit causal relation-
ships between constructs. A construct is a postulated (often
psychometric) variable that cannot be measured directly, e.g.,
trust, extroversion, team cohesion, software quality. A variable
is a quantity that may have different values. Therefore, the
researcher generates several questionnaire items (questions)
intended to reflect each construct. To analyze validity, the
researcher pilots the questionnaire and analyzes the correla-
tions between the items. Items that reflect the same construct
should be highly correlated with each other (convergent validi-
ty) and less correlated to items reflecting other constructs (dis-
criminate validity) .
Running the survey will produce a response distribution for
each question; e.g., a) 5; b) 10; c) 20; d) 30; e) 15. In this ex-
ample, visual inspection suggest the distribution favors SCI.
To determine statistical significance we can use non-
parametric tests including Chi-Square Test and Kolmogorov-
Smirnov. These tests require an expected distribution, three
options for which are:
1) uniform distribution, e.g., 20, 20, 20, 20, 20
2) pseudo-normal distribution, e.g., 6, 19, 30, 9, 6
3) reflected distribution, e.g., 15, 30, 20, 10, 5
Using a uniform distribution makes little sense as it does
not answer the question at hand. Using the pseudo-normal
distribution addresses the question, is the extent of skew in the
observed distribution significant? However, treating discon-
tinuous, ordinal data as normally distributed is statistically
problematic. The reflected distribution is the most defensible
as it addresses the question, is the observed distribution signif-
icantly different from an equally compelling distribution sup-
porting the rival theory? and reflecting a distribution does not
obviously violate statistical norms (cf.  for a detailed ex-
ample of this approach).
Process theories do not necessarily have constructs.
Sensemaking and coevolution  are sub-processes. The
problem space and the solution space  are populations of
ideas. “Action to improve software process”and “action to
develop software products” are categories of activities.
None of these process theory elements are constructs because
none of them can take on quantities. Similarly, rival proposi-
tions, e.g., goals are given by stakeholders / constructed by
developers, are not constructs. When researchers devise sever-
al questions about a process theory proposition, the answers
are variables but these variables are not reflective indicators of
the same underlying construct. Without a priori reasons to
believe that items associated with one theory difference will
be more closely related to each other than to items associated
with other differences, convergent/discriminate validity analy-
sis does not apply. Instead, process theory testing question-
naires may be validated using pilot studies with both experts
and members of the target population. Researchers should
request question-by-question feedback from pilot participants
–some familiar with the purpose of the study, others not (see
,  for a detailed example).
D. Multi-Methodological and Team Approaches
Process theories may also be evaluated using a multi-
methodological approach. Researchers can combine question-
naire studies with case studies or other observational ap-
proaches to achieve good breadth and depth simultaneously.
One way to approach the data analysis is to analyze the two
theories separately in the observational study (which promotes
deeper analysis) and focus on conflicting propositions in the
questionnaire-based study (to keep the instrument short and
simple). While other configurations may be preferable in some
contexts, this one appears to make the most of each study type
see  for a detailed example.
Combining case studies and questionnaire studies in this
way overcomes many of the limitations of each individually.
Observational studies benefit from extensive data collection
and triangulation; however, the number of organizations stud-
ied is limited. Questionnaire studies, in contrast, produce more
shallow data but can reach a large number or organizations.
Observational studies also produce real-time observations,
which are not subject to the participant memory effects that
threaten questionnaire studies. While both questionnaire and
observational approaches are valid on their own, combining
one or more observational studies with a questionnaire study
facilitates greater data triangulation, mitigates mono-method
bias and generally encourages more nuanced reflection.
Improved rigor notwithstanding, multi-methodological re-
search is much more difficult to present. Presenting each indi-
vidual case, the cross case analysis, the questionnaire results
and the case study / questionnaire triangulation will consume
many pages. It may be better to omit individual case analyses
and provide only the cross case analysis in a question-and-
answer format . Reviewers should accept this as a neces-
sary abbreviation. Reviewers should more generally give cred-
it for the added robustness of multimethodological research.
E. Conceptual Evaluation
Process theories may also be evaluated conceptually on
numerous dimensions including testability, usefulness, sim-
plicity and communicability. Regarding testability, researchers
do not categorically prove or disprove theories . However,
if we cannot imagine any specific observation that would con-
stitute evidence against a theory, it may be tautological. While
a process theory may not be directly useful to software devel-
opers in general, it may be useful to those who develop SE
tools and practices or to researchers studying SE phenomena.
Similarly, while a process theory should not oversimplify it
should not be so complicated that few people can understand it
–a charge sometimes leveled against the Situated Function-
Behavior-Structure Framework  for example. Finally, pro-
cess theories obviously should be communicable. Authors can
demonstrate communicability by providing clear definitions of
their theorys elements, possibly in tabular form. Giving ex-
amples of the theorys applications and discussing its implica-
tions also helps to demonstrate communicability.
Moreover, some process theories may be evaluated by
analogy to similar kinds of artifacts with well-defined evalua-
tion criteria. For example, a teleological theory posits an
agent, pursuing goals through activities. However, the “activi-
ties”posited by a teleological theory may actually be catego-
ries of activities, making the theory in part a categorization
system. A good category is one that supports specific infer-
ences about its members . Consequently, it may be in-
structive to apply the supports-inferences criteria to evaluate a
VII. DISCUSSION AND CONCLUSION
Section III listed some topics that might benefit from a
process theory approach. The following discussion suggests
how some of these process may be theorized.
For example, we know that designers simultaneously re-
frame problems and design artifacts to address them a pro-
cess called coevolution . However, we do not know exact-
ly how the coevolution process unfolds in SE. Because coevo-
lution involves the interaction between entities (e.g. the devel-
opers, solution conjectures, problem frames), it a dialectical
process theory may be appropriate.
Similarly, while many believe that agility is important, we
lack a clear understanding what agility is and how the agility,
the team property, differs from ostensibly Agile methods (cf.
). Since agility involves the interaction between the need
for change and responses for change, a dialectical process the-
ory of how developers experience change might help explain
the role of agility in SE projects.
Open source projects, meanwhile depend on networks of
contributors to implement features, improve quality and gain
market share . A process theory may help explain how
some projects succeed while others stagnate. Because so many
open source projects compete for attention among developers
and users, an evolutionary approach may be appropriate.
Meanwhile, my own research predominately concerns pro-
cess theories of software engineering in general , ,
. As most software is developed by human agents, a teleo-
logical approach has been helpful.
From my own limited experience, the greatest barriers to
process theory research in SE are 1) many SE academics are
unfamiliar with process theory; 2) little SE-specific methodo-
logical guidance is available; 3) reviewers incorrectly evaluate
process theory research using the criteria and norms of vari-
ance theory research; 4) reviewers evaluate process theory
papers based on tone and argumentum ad hominem rather than
This paper therefore describes process theories and synthe-
sizes the methodological concepts I have acquired and devel-
oped through working with process theories. Its core contribu-
tion is its exploration of observational, questionnaire and mul-
timethodological empirical studies as they pertain to process
theories. I am not aware of any previous research, in SE or
elsewhere, which describe the coding-scheme / instrument
development approaches for comparing rival process theories
Despite their many benefits, however, process theories are
intrinsically limited in several ways. Lifecycle theories en-
courage us to oversimplify and over-rationalize complex hu-
man behaviors. Teleological and dialectical theories help us
understand but rarely produce useful predictions. Evolutionary
theories may predict (probabilistically) success and failure;
however, quantifying the fitness function may be difficult.
This paper also has several limitations. It does not present
a taxonomy of quality dimensions for process theory studies,
such as internal validity, external validity, reliability and ob-
jectivity  or credibility, transferability, dependability and
confirmability . This is an intentional choice following
Rolfe’s  convincing argument that research quality cannot
be reliably ascertained from a pre-ordained set of criteria. Fur-
thermore, this paper is intended to share a conceptualization of
process theory and ideas that seemed useful to the author in
his own research. Evaluating such guidance requires other
experienced researchers to apply it in diverse contexts. Future
work may therefore include reflections, refinements or refuta-
tions from other researchers who use or disregard the above
guidance in process theory studies. Moreover, my own re-
search primarily concerns teleological and lifecycle theories,
and the tension between the two. The discussion of evolution-
ary and dialectic theories is therefore less developed and fu-
ture work may include better treatment of these theory types.
Future work should moreover address SEs sampling problem,
i.e., the lack of population lists from which to randomly sam-
ple and the tendency for reviewers to capriciously reject some
papers for convenience sampling despite the absence of any
strongly defensible representative samples in our literature.
These limitations notwithstanding, the Software Engineer-
ing academic discipline is fundamentally concerned (among
other things) with the process of developing software. A pro-
cess theory explains how a process unfolds. Consequently,
process theories may be useful for capturing and reasoning
about scientific accounts of software processes. SE needs ex-
planatory process theories to counterbalance more prescriptive
methods. The lack of SE process theory research obstructs
scientific consensus by focusing the academic community on
methods, which inevitably oversimplify and over-rationalize
complex and unpredictable human phenomena , . Spe-
cifically, SE needs more evolutionary, dialectic and teleologi-
cal process theories to overcome the tendency to oversimplify
and over-rationalize human processes into lifecycles.
Finally, this work may be particularly useful for develop-
ing general theories of software engineering. As such theories
should include process theory elements , , , the
lack of guidance for and acceptance of process theory research
has perhaps hindered their development.
Thanks are due to Marshall Scott Pool and the participants
of the 2014 SEMAT Workshop on General Theory of Soft-
ware Engineering for their constructive feedback on an early
version of this paper.
 A. H. Van de Ven, Engaged scholarship: a guide for
organizational and social research. Oxford, UK: Oxford
University Press, 2007.
 A. H. Van de Ven and M. S. Poole, Explaining development
and change in organizations, The Academy of Management
Review, vol. 20, no. 3, pp. 510540, 1995.
 V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, User
acceptance of information technology: Toward a unified view,
MIS Quarterly, vol. 27, no. 3, pp. 425478, 2003.
 K. Schwaber and J. Sutherland, The Scrum Guide, 2010.
[Online]. Available: http://www.scrum.org/scrumguides/.
 M. Poppendieck and T. Poppendieck, Lean Software
Development: An Agile Toolkit. Addison-Wesley Professional,
 K. Forsberg and H. Mooz, The Relationship of System
Engineering to the Project Cycle, Engineering Management
Journal,, vol. 4, no. 3, pp. 3643, 1992.
 T. Allweyer, BPMN 2.0- Business Process Model and Notation.
OMG Standard formal/2011-01-03, 2011.
 B. Curtis, M. I. Kellner, and J. Over, Process Modeling,
Communications of the ACM, vol. 35, no. 9, pp. 7590, 1992.
 P. Johnson, M. Ekstedt, and I. Jacobson, Where's the Theory for
Software Engineering?, IEEE Software, vol. 29, no. 5, pp. 94
 P. Ralph, P. Johnson, and H. Jordan, Report on the First
SEMAT Workshop on a General Theory of Software
Engineering (GTSE 2012), ACM SIGSOFT Software
Engineering Notes, vol. 38, no. 2, pp. 2628, 2013.
 P. Johnson, P. Ralph, M. Goedicke, P.-W. Ng, K.-J. Stol, K.
Smolander, I. Exman, and D. E. Perry, Report on the Second
SEMAT Workshop on General Theory of Software Engineering
(GTSE 2013), ACM SIGSOFT Software Engineering Notes,
vol. 38, no. 5, pp. 4750, 2013.
 P. Ralph, Possible core theories for software engineering, in
Proceedings of the 2nd SEMAT Workshop on a General Theory
of Software Engineering, San Francisco, CA, USA, IEEE, 2013,
 P. Ralph, I. Exman, P.-W. Ng, P. Johnson, M. Goedicke, A. T.
Kocatas, and K. L. Yan, How to Develop a General Theory of
Software Engineering: Report on the GTSE 2014 Workshop,
ACM SIGSOFT Software Engineering Notes, vol. 39, no. 6, pp.
 J. Hannay, D. I. K. I. K. Sjberg, and T. Dyba, A Systematic
Review of Theory Use in Software Engineering Experiments,
IEEE Transaction on Software Engineering, vol. 33, no. 2, pp.
87107, Feb. 2007.
 F. P. Brooks, The Design of Design: Essays from a Computer
Scientist. Addison-Wesley Professional, 2010.
 M. W. Lewis, Exploring Paradox: Toward a More
Comprehensive Guide, The Academy of Management Review,
vol. 25, no. 4, pp. 760776, Jan. 2000.
 D. A. Schn, The reflective practitioner: how professionals think
in action. USA: Basic Books, 1983.
 P. Ralph, The Illusion of Requirements in Software
Development, Requirements Engineering, vol. 18, no. 3, pp.
 P. Ralph, The Sensemaking-Coevolution-Implementation
Theory of Software Design, Science of Computer
Programming, vol. 101, pp. 21-41, 2015.
 M. Maher, J. Poon, and S. Boulanger, Formalising design
exploration as co-evolution: A combined gene approach,
Preprints of the Second IFIP WG5.2 Workshop on Advances in
Formal Design Methods for CAD, pp. 128, 1995.
 P. Palvia and J. Nosek, An empirical evaluation of system
development methodologies, Information Resource
Management Journal, vol. 3, no. 3, pp. 2332, 1990.
 P. Ralph, Introducing an Empirical Model of Design, in
Proceedings of The 6th Mediterranean Conference on
Information Systems, Limassol, Cyprus: AIS, 2011.
 P. Ralph, Improving coverage of design in information systems
education, in Proceedings of the 2012 International Conference
on Information Systems, Orlando, FL, USA: AIS, 2012.
 P. Ralph, Comparing Two Software Design Process Theories,
in Global Perspectives on Design Science Research:
Proceedings of the 5th International Conference, DESRIST
2010, LNCS 6105, Springer, 2010, pp. 139153.
 I. Allison and Y. Merali, Software process improvement as
emergent change: A structurational analysis, Information and
Software Technology, vol. 49, no. 6, pp. 668681, Jun. 2007.
 W. Royce, Managing the development of large software
systems, in Proceedings of WESCON, Los Angeles, USA:
IEEE, 1970, pp. 19.
 J. Kim, Causation, in The Cambridge Dictionary of
Philosophy, 2nd ed., R. Audi, Ed. Cambridge, UK: Cambridge
University Press, 1999, pp. 125127.
 D. P. Truex, R. Baskerville, and J. Travis, Amethodical systems
development: the deferred meaning of systems development
methods, Accounting, Management and Information
Technologies, vol. 10, no. 1, pp. 5379, 2000.
 J. S. Dyer, P. C. Fishburn, R. E. Steuer, J. Wallenius, and S.
Zionts, Multiple Criteria Decision Making, Multiattribute
Utility Theory: The Next Ten Years, Management Science, vol.
38, no. 5, pp. p645654, May 1992.
 C. W. Alexander, Notes on the synthesis of form. Harvard
University Press, 1964.
 J. Saldana, The Coding Manual for Qualitative Researchers.
SAGE Publications, 2012.
 W. Stacy and J. MacMillan, Cognitive bias in software
engineering, Communications of the ACM, vol. 38, no. 6, pp.
 P. Ralph, Toward a Theory of Debiasing Software
Development, in Research in Systems Analysis and Design:
Models and Methods: 4th SIGSAND/PLAIS EuroSymposium
2011, no. 93, S. Wrycza, Ed. Gdansk, Poland: Springer, 2011,
 D. Arnott, Cognitive biases and decision support systems
development: a design science approach, Information Systems
Journal, vol. 16, no. 1, pp. 5578, 2006.
 J. Parsons and C. Saunders, Cognitive Heuristics in Software
Engineering: Applying and Extending Anchoring and
Adjustment to Artifact Reuse, IEEE Transaction on Software
Engineering, vol. 30, pp. 873888, 2004.
 B. G. Glaser and A. L. Strauss, The Discovery of grounded
theory: Strategies for qualitative research. Chicago: Aldine Pub.
 J. Mills, A. Bonner, and K. France, The Development of
Constructivist Grounded Theory, International Journal of
Qualitative Methods, vol. 5, no. 1, 2006.
 P. Ralph, Fundamentals of Software Design Science, PhD
Dissertation, University of British Columbia, Vancouver,
 P. E. Vermaas and K. Dorst, On the Conceptual Framework of
John Gero's FBS-Model and the Prescriptive Aims of Design
Methodology, Design Studies, vol. 28, no. 2, pp. 133157,
 B. Fischoff, Debiasing, in Judgment Under Uncertainty:
Heuristics and Biases, no. 31, D. Kahneman, P. Slovic, and A.
Tversky, Eds. Cambridge, USA: Cambridge University Press,
 R. F. Pohl, Ed., Cognitive Illusions. East Sussex, UK:
Psychology Press, 2004.
 J. T. Jost and R. Andrews, System Justification Theory, in The
Encyclopedia of Peace Psychology, Blackwell Publishing Ltd,
 J. T. Jost, M. R. Banaji, and B. A. Nosek, A Decade of System
Justification Theory: Accumulated Evidence of Conscious and
Unconscious Bolstering of the Status Quo, Political
Psychology, vol. 25, no. 6, pp. 881919, 2004.
 M. Kennaley, SDLC 3.0: Beyond a Tacit Understanding of
Agile, vol. 1. Fourth Medium Press, 2010.
 B. Boehm, A spiral model of software development and
enhancement, IEEE Computer, vol. 21, no. 5, pp. 6172, May
 G. Polya, How to Solve It: A New Aspect of Mathematical
Method, 2nd ed. Princeton, New Jersey, USA: Princeton
University Press, 1957.
 L. Suchman, Human-machine reconfigurations: Plans and
situated actions. Cambridge University Press, 2007.
 P. Checkland, Systems Thinking, Systems Practice. Chichester:
 K. Beck, M. Beedle, A. van Bennekum, A. Cockburn, W.
Cunningham, M. Fowler, J. Grenning, J. Highsmith, A. Hunt, R.
Jeffries, J. Kern, B. Marick, R. C. Martin, S. Mellor, K.
Schwaber, J. Sutherland, and D. Thomas, Manifesto for agile
software development, 2001. [Online]. Available:
http://www.agilemanifesto.org/. [Accessed: 02-Jul-2012].
 P. Kruchten, The Rational Unified Process: An Introduction, 3rd
ed. Addison-Wesley Professional, 2003.
 R. A. Wolfe, Organizational innovation: review, critique and
suggested research directions, Journal of Management Studies,
vol. 31, no. 3, pp. 405431, 1994.
 M. Poole, A. H. Van de Ven, K. Dooley, and M. E. Holmes,
Organizational change and innovation processes theory and
methods for research. New York, NY, USA: Oxford University
 M. S. Poole, On the Study of Process in Communication
Research, in Communication Yearbook 36, C. T. Salmon, Ed.
New York, USA: Routledge, 2012, pp. 371409.
 M. S. Poole and A. H. Van de Ven, Empirical Methods for
Research on Organizational Decision-Making Processes, in The
Blackwell Handbook of Decision Making, Nutt, P. C. and D.
Wilson, Eds. Oxford: Blackwell, 2010, pp. 543580.
 M. E. Oswald and S. Grosjean, Confirmation Bias, in
Cognitive Illusions: A Handbook on Fallacies and Biases in
Thinking, Judgement and Memory, no. 4, R. F. Pohl, Ed. Hove,
UK: Psychology Press, 2004, pp. 7996.
 F. D. Davis, Perceived Usefulness, Perceived Ease of Use, and
User Acceptance of Information Technology, MIS Quarterly,
vol. 13, no. 3, pp. 319340, Sep. 1989.
 J. R. Platt, Strong inference, Science, vol. 146, no. 3642, 1964.
 L. Dube and G. Pare, Rigor in information systems positivist
case research: Current practices, trends and recommendations,
MIS Quarterly, vol. 27, no. 4, pp. 597635, Dec. 2003.
 E. Sober, Testability, Proceedings and Addresses of the
American Philosophical Association, vol. 73, no. 2, pp. 4776,
 J. S. Gero, Design prototypes: A knowledge representation
schema for design, AI Magazine, vol. 11, no. 4, pp. 2636,
 R. K. Yin, Case study research: Design and methods, 4 ed.
California, USA: Sage, 2008.
 M. E. Fonteyn, B. Kuipers, and S. J. Grobe, A Description of
Think Aloud Method and Protocol Analysis, Qualitative Health
Research, vol. 3, no. 4, pp. 430441, Nov. 1993.
 K. Dorst and J. Dijkhuis, Comparing Paradigms for Describing
Design Activity, Design Studies, vol. 16, no. 2, pp. 261274,
 D. D. Heckathorn, Respondent-Driven Sampling: A New
Approach to the Study of Hidden Populations, Social Problems,
vol. 44, no. 2, pp. 174199, 1997.
 D. T. Campbell and D. W. Fiske, Convergent and discriminant
validation by the multitrait-multimethod matrix, Psychological
Bulletin, vol. 56, no. 2, pp. 81105, 1959.
 P. Ralph, Software Engineering Process Theory: A Multi-
Method Comparison of Sensemaking-Coevolution-
Implementation Theory and Function-Behavior-Structure
Theory, arXiv [cs.SE]: 1307.1019.
 J. S. Gero and U. Kannengiesser, The Situated Function-
Behaviour-Structure Framework, Design Studies, vol. 25, no. 4,
pp. 373391, 2004.
 J. Parsons and Y. Wand, A question of class, Nature, vol. 455,
no. 7216, pp. 10401041, Oct. 2008.
 K. Dorst and N. Cross, Creativity in the design process: co-
evolution of problemsolution, Design Studies, vol. 22, no. 5,
pp. 425437, Sep. 2001.
 K. Conboy, Agility from First Principles: Reconstructing the
Concept of Agility in Information Systems Development,
Information Systems Research, vol. 20, no. 3, pp. 329354,
 W. Scacchi, J. Feller, B. Fitzgerald, S. Hissam, and K. Lakhani,
Understanding Free/Open Source Software Development
Processes, Software Process: Improvement and Practice, vol.
11, no. 2, pp. 95105, 2006.
 W. Trochim, Research Methods Knowledge Base. Cincinnati,
OH, USA: Atomic Dog Publishing, 2001.
 Y. S. Lincoln and E. G. Guba, Naturalistic Inquiry. SAGE
 G. Rolfe, Validity, trustworthiness and rigour: quality and the
idea of qualitative research, J Adv Nurs, vol. 53, no. 3, pp. 304
310, Feb. 2006.