Conference PaperPDF Available

Examining the Effects of Agile Methods and Process Maturity on Software Product Development Performance


Abstract and Figures

This paper examines the effects of agile methods and software process maturity on software product development performance. Through a mail survey, we obtained data from 72 small and medium-sized software firms that predominantly were not CMMI-certified. Findings from our partial least squares analysis suggest that the use of agile methods has a positive impact on product development efficiency and effectiveness, but CMMI practices do not have this effect. Our results suggest that software process improvement initiatives in software product firms create the highest benefits through first adopting agile methods and only then moving on to implementing CMMI-like process improvement initiatives.
Content may be subject to copyright.
B. Regnell, I. van de Weerd, and O. De Troyer (Eds.): ICSOB 2011, LNBIP 80, pp. 85–97, 2011.
© Springer-Verlag Berlin Heidelberg 2011
Examining the Effects of Agile Methods and Process
Maturity on Software Product Development Performance
Mikko Rönkkö, Juhana Peltonen, and Christian Frühwirth
Software Business Lab, Aalto University
FI-00076 Aalto, Finland
Abstract. This paper examines the effects of agile methods and software
process maturity on software product development performance. Through a
mail survey, we obtained data from 72 small and medium-sized software firms
that predominantly were not CMMI-certified. Findings from our partial least
squares analysis suggest that the use of agile methods has a positive impact on
product development efficiency and effectiveness, but CMMI practices do not
have this effect. Our results suggest that software process improvement
initiatives in software product firms create the highest benefits through first
adopting agile methods and only then moving on to implementing CMMI-like
process improvement initiatives.
Keywords: Product development, agile methods, process improvement,
1 Introduction
The performance of agile information systems development methods has received
considerable interest among information systems scholars [e.g., 1-3]. However, the
current level of empirical support for the arguments about performance of agile
methods can be considered to be inadequate [4]. In this paper, we compare the effects
of agile software development practices and process maturity on the firm software
product development performance.
Research has provided us with strong evidence of the positive effect of process
maturity on software development performance [5, 6]. These studies have pre-
dominantly focused on process maturity from a more traditional CMMI perspective.
However, recently the use of agile methods has gained popularity in the software
industry. These methods have been advocated as a solution to problems in software
development [7-10] in a dynamic environment such as the software product industry
[11, 12]. Regardless of the relative popularity of this alternative development
approach, only a limited amount of research compares agile methods with other
development approaches [13]. Clearly, more studies comparing the effectiveness of
different development approaches are needed. Moreover, a large share of the studies
testing the performance of CMMI uses data from CMMI appraisals that are used
predominantly by larger firms [14]. While the CMMI based approach on software
86 M. Rönkkö, J. Peltonen, and C. Frühwirth
process improvement is well suited for large project-based development organizations,
the effectiveness of different development approaches in small firm context is currently
In this paper we investigate the following research question: “What is the impact of
process maturity and use of agile practices on software product development
performance?” This paper addresses two important gaps in the present body of
knowledge: First, we conduct a statistical comparative test between the effects of
process maturity and agility on software product development performance. To our
knowledge, only a few comparative studies like this exist [e.g., 13]. Second, in
contrast to previous studies using projects in larger firms, our sample consist of small
and medium sized firms that develop software products hence extending the
generalizability of previous studies.
The rest of the paper is organized as follows. In the second section we briefly
review the key literature and formulate hypotheses for testing. The next sections
introduce our sample, constructs and measures that are followed by data collection
and analysis. Thereafter, the sixth section introduces the results of hypothesis testing
using structural equation modeling. The final section presents a discussion on our
results and on their broader implications.
2 Literature Review and Hypothesis Development
Compatibility of agile development and process maturity has been of interest for the
information systems community. Agile methods were initially seen as alternative,
contradicting ways to develop software that are based on profoundly different values
and principles [15]. While agile development and process maturity were initially often
considered to be incompatible targets, information systems and software engineering
communities have recently started to consider these as more compatible than
conflicting targets [15-18].
Process maturity, often measured through frameworks such as CMMI, refers to the
degree of use of well-established software engineering methods [19]. Development
organizations often strive for maturity to improve the quality of the systems developed,
to improve the manageability of the process, and to make the development projects
more predictable. During the recent more than a decade, the CMM family – including
the most recent CMMI model – has established itself as the leading software process
improvement framework, through which increased maturity is pursued.
When combined with traditional plan-driven development methods, increasing
process maturity generally emphasizes and accommodates issues such as planning of
the system and development organization, repeatability predictability, and relatively
rigid control procedures [20, 21]. Combining software engineering best practices into
development frameworks has been a big success in both terms of popularity and
results. Indeed, there is no lack of research supporting the existence of a link between
maturity and development process performance [5, 6, 22, 20]. However, software
process improvement with CMMI or any other leading framework does not come
without a cost. First, CMMI adoption and appraisals required for official certification
are costly and require significant effort thus making it more accessible for larger than
smaller firms [14]. Second, opponents of these development frameworks argue that
Examining the Effects of Agile Methods and Process Maturity 87
they can lead to too much overhead making the process less efficient and responsive
to changes [23] While these potential weaknesses can be justified especially in large
software development projects, the relative cost in terms of lost opportunities can be
high in more market driven software product development [24, 25].
Agile methods were initially presented as an alternative to the increased overhead
in the development process as well as to decrease the effort required for adopting a
new development framework. In contrast to several of the major SPI frameworks that
are commonly documented as books [26], agile methods commonly define only
minimum amount of processes required for a development team to function efficiently
and can be codified as a relative small number of principles [27]. Another distinctive
feature of agile methods is that they emphasize simplicity, speed, and flexibility to
accommodate changes in requirements. For achieving the pursued characteristics,
developers of the agile methods feel the planning – including documentation – and
control should be, to certain degree, discarded [28]. Agile methods emphasize self-
organization of the development team, quick and frequent delivery of working
software in an incremental manner, and intensive communication both within the
development team and with the customer. Not surprisingly, agile methods have been
recently advocated as a solution to problems in software product development [7-10]
as a response to dynamic environment where technologies change fast and often in an
unpredictable manner [11, 12].
The topic of the compatibility of agile methods and process maturity has been
recently under investigation in the information systems community [2]. Within the
field of software engineering, insights have been gained through case studies of
companies that have adopted agile methods into traditional development organization
[16], and of companies using agile development processes that have implemented
CMMI, even up to maturity level 5 [29]. In general, these efforts are usually not
without challenges, but most studies report positive outcomes. In general, recently
more evidence for than against the compatibility of agile development and process
discipline has been presented [15-18]. The current understanding is that agile process
can be disciplined and mature, and that the development process should be selected
base on situation characteristics [15, 30].
Since most studies focusing on the performance of different software development
approaches focus on software projects, this poses a limitation on how well these
approaches generalize to non-project oriented settings. Particularly, cases where
software is developed as products has received limited attention in the studies
investigating the performance of different development approaches. We start the
discussion on the product development effects of these frameworks by looking at how
product development performance is defined in the literature. While information
systems and software engineering research communities view the development
performance as a two dimensional construct composed of product quality and
development project performance [31], product development research has developed
several alternative conceptualizations, most notably one consisting of efficiency,
effectiveness and innovativeness [32], and an alternative separating product
development and product management as distinct dimensions [33]. The differences on
these conceptualizations can be best understood by examining them in the context:
Incremental product development - the improvement of existing products or product
lines - and radical product development - the development of completely new
88 M. Rönkkö, J. Peltonen, and C. Frühwirth
products or product lines – require different capabilities and hence it is natural that
performance of these processes are considered as separate constructs [34, 35]. In all,
the operationalization of incremental product development include the dimensions of
efficiency and effectiveness, while the goodness of radical product development
performance is often measured through an additional dimension of innovativeness.
Efficiency refers to the goodness of rate that the resources are transformed into
outputs [32], and is composed of two main factors, lead-time and cost-efficiency. This
dimension has a close fit to the process performance dimension of software
development performance [20, 36]. Effectiveness refers to the goodness of the
product not only in terms of product quality – a common measurement of software
development performance [20, 36] – but also how well it fits the needs of the markets.
Finally, innovativeness refers to the ability to conceive new ideas and develop these
into commercially successful products or product features.
Traditional plan-driven methods have been designed to for developing well-
defined software solutions in time and on budget. These objectives are normally
achieved through solid planning and by avoiding wasted programming work [19].
Considering the strong evidence of process maturity on development project
performance [5, 6, 20] and the close fit between the performance dimensions of
incremental product development and software project performance presented earlier,
we hypothesize:
Hypothesis 1: Process maturity increases product development efficiency.
Hypothesis 2: Process maturity increases product development effectiveness.
While agile methods are not about efficiency in the same sense as plan-driven
methods [7], they aim to reduce the amount of administrative work and control for
wasted programming work through frequent integration, testing and delivery of
program versions. In this way, agile methods provide an improvement in efficiency
compared to having less defined processes. Considering the suggested fit between
agile methods to software product development [7-10] we hypothesize:
Hypothesis 3: Process agility increases product development efficiency.
Product development effectiveness refers to how well the developed product matches
the markets. While software product industry is obviously one of the more turbulent
industries, it is likely that the market requirements change during the development
cycle of a new product. In traditional project and contracting oriented software
engineering requirements changes cause schedule and budget overruns. However, in
product oriented development these can result in product with bad market fit or delay,
significantly affecting how the product performs in the markets [37-39]. Hence it is
imperative for firms to maintain flexibility in product development. We hypothesize:
Hypothesis 4: Process agility increases product development effectiveness.
Regarding the last dimension of product development performance, innovativeness is
has received relatively little attention in the software process research. In management
research, one of the most enduring findings about innovativeness is that innovation
Examining the Effects of Agile Methods and Process Maturity 89
occurs at organizational interfaces [40]. Considering that several of the SPI processes
are organizational level tools that define interfaces also between engineering and other
units within the firm, process maturity should improve innovativeness. In addition,
several of the frameworks specify that organization should have processes for managing
innovations [7].
Hypothesis 5: Process maturity increases product development innovativeness.
It has been suggested that agile methods are associated with innovation [41-43].
Moreover, agile methods give significant responsibilities to individuals, and this
empowerment of individuals is a prerequisite for innovative behavior [28]. Moreover,
agile methods emphasize the importance of communications with the customer
interface and emphasize building the work motivation at an individual level. Hence
we hypothesize:
Hypothesis 6: Process agility increases product development innovativeness.
3 Empirical Study Design
We use a subset of a larger data collection that was collected as a part of a research
project surveying software companies in Finland [44]. We use the term “software
product firm” to refer to any firm that owns and markets a software product –
regardless of the official company classification. The sampling frame for the primary
survey, which is described in details elsewhere [44], contained 2616 firms mainly
under the NACE (rev 1.2) codes 72.21 and 72.22, but also a small number of other
firms that conduct software product business as a secondary area. For this sample, the
CEO or other high ranking employee of a firm was used as an informant. After filling
in the general information including questions about product development
performance the informant nominated another person that was intimately familiar
with the software development process of the firm, most commonly the manager of
the software development function. These nominates acted as the population for the
secondary survey where the software development related questions were asked.
Asking the independent and dependent variables from a different person helped us to
avoid common method bias, which is commonly present in survey research.
The dependent variable, product development performance, was measured with a
scale developed by Kusunoki et al. [32]. It contains three dimensions; product
development efficiency, product development effectiveness and innovativeness.
These were measured with 10 items using 7-point Likert scales. Principal factor
analysis revealed only one dominant factor and a weaker three-factor solution.
However due to the theoretical foundations of the scale, we decided to use the three
dimensional structure of efficiency, effectiveness, and innovativeness for which we
standardized and calculated the Cronbach’s alphas of .76, .75, and .85, respectively.
The constructs for development process maturity and agility were adapted directly
from a paper by Rönkkö, Järvi, and Mäkelä [45]. Although their scale was originally
developed as a Rasch scale [cf., 46], it consists of simple agree-disagree questions and
can hence be adapted to a study using more traditional measurement approaches. The
90 M. Rönkkö, J. Peltonen, and C. Frühwirth
scale for measuring the process maturity was based on a well-known Capability
Maturity Model Integration (CMMI) software process maturity framework: CMMI
for Development, Version 1.2 [26]. To improve the discriminant validity of the
construct, i.e. its ability to measure software process maturity as a distinct construct
from more general product development capability, we decided to exclude the so-
called integrated product and process development (IPPD) additions to the CMMI. In
total this scale consisted of 21 items. The adoption of agile methods was measured
using a scale derived from on the basis of the two most well known agile methods and
consisted of 16 items.
All scales that were adapted from previously published research were translated to
Finnish using the double blind translate and back-translate procedure [47]. The
translations were performed by the first authors of this paper, one researcher not
related to this paper, and two research assistants. The translating protocol included
translating also the context of the items, if the content validity of the original item was
considered poor by the translator. The clarity and general validity of the items were
checked by pre-testing all survey instruments with several experts and practitioners
that had roles closely matching the ones of the informants. Based on these reviews, a
number of items were reformulated.
The primary survey was implemented in two stages as follows. First, a pilot survey
with about 11 percent of the sampling frame of 2616 firms was launched in April
2007. The pilot survey, mailed to 291 firms, was used to test the primary survey
instrument. Minor modifications were made to the questionnaire. The main primary
survey was thereafter sent to 2550 firms (2616 firms, from which we have excluded
66 firms that had responded to the pilot survey).
Both stages were implemented following a modified version of the tailored survey
design method [48]. Mailings began with a pre-notice letter followed by the main
survey package using postal mail. Two email reminders and one round of telephone
calls were used to improve the response rate. A printed questionnaire and an online
form were offered as alternative options to the informants. This phase of our data
collection lasted from April through July 2007. Altogether, this phase produced 287
usable responses. After excluding firms with less than five employees as too small,
the secondary survey was sent to 123 managers of software development or person in
similar knowledgeable position with regards to software development in the firm.
After several reminders by email and telephone, 83 firms provided responses to both
surveys and were used in the main analyses. 9 cases were dropped due to the amount
of missing data resulting in final sample of 72 companies.
We screened and prepared the data with Stata version 10. The effect of non-
response was tested using two different methods. First, we compared the means of the
key study variables between early and late respondents, as suggested by Oppenheim
[49]. For the primary survey, we found that late respondents were larger (p < .001)
and older (p < .05) than the early respondents. No significant differences were found
for the secondary survey. Second, we compared the sampling frame with the
respondents. For the primary survey, we found that industry codes 72.21 and 72.22
were overrepresented, which was expected considering that the survey targeted the
software product industry and oversampling was used. We did not find any significant
differences between geographical distribution and age of the sampling frame and the
respondents. When comparing the respondents of the secondary survey to the
Examining the Effects of Agile Methods and Process Maturity 91
sampling frame, no differences were found. In all, the results of our non-response
analysis indicate that the results of this study are likely to be more valid for smaller
than larger software product firms.
We used Partial Least Squares Path Modeling (PLS) as our main data analysis
method. The reason for using this approach is the convenience it provides by testing
all hypotheses and the validity of the measures in a single set of analyses. Although
structural equation modeling would perform similar tests, we chose PLS because it
has less stringent sample size and indicator distribution requirements than traditional
SEM approaches [50]. Moreover, the method has increased popularity in IS recently
[51] and hence we considered it more appropriate.
4 Results
We start examining the results of PLS by examining the measurement model. Table 1
shows the summary statistics for the constructs and Table 3 shows the construct-
indicator cross loading matrix. Indicator loadings for the product development
performance constructs are all above the recommended .707 threshold except the first
two items. The first item is substantially below the limit for acceptable, but we
decided to keep it nevertheless. The reason for this is that PLS model works better
when the number of indicators is larger, all reliability indicators in Table 2 were good
for this construct, and this poor item is weighted less than others so it should not
cause bias in the results. All indicators in these scales load higher on their respective
constructs than other constructs indicating discriminant validity of the three-
dimensional product development performance scale.
The process agility and process maturity scales are more problematic since they have
so many items. We considered parceling of the items, but this was not done due to the
fact that to get unbiased results in PLS, the number of indicators needs to be large [52].
However, when the ratio of number of indicators to the number of cases is large, the
indicator loadings tend to be unstable. Nevertheless, the reliability indices for these two
constructs were high (Process Maturity) and acceptable (Process Agility) and hence we
concluded that the overall degree of quality of measurement is sufficient.
Figure 1 shows the results of PLS estimation in the form of a path diagram and
Table 2 shows the results of bootstrapping. All path coefficients are positive except
for the path from Process Maturity to Efficiency. None of the paths between Process
Table 1. Construct reliability and validity
AVE Composite
Reliability R
Square Cronbachs
Alpha Commun-
ality Redund-
Agility 0,1932 0,7772 0,0000 0,7531 0,1932 0,0000
Effectiveness 0,4225 0,7356 0,1649 0,5329 0,4225 0,0665
Efficiency 0,6093 0,8236 0,1929 0,6850 0,6093 0,0997
0,7640 0,9066 0,0856 0,8454 0,7640 0,0521
Maturity 0,3742 0,9237 0,0000 0,9147 0,3742 0,0000
92 M. Rönkkö, J. Peltonen, and C. Frühwirth
Fig. 1. Results of PLS estimation
Table 2. Bootstrapping results
Sample Sample
Mean Standard
Error T
Statistics p
Agility -> Effectiveness 0,3614 0,4046 0,1401 2,5787 0,05981
Agility -> Efficiency 0,5037 0,4787 0,1527 3,2974 0, 0007589
Agility -> Innovativeness 0,1981 0,2532 0,1535 1,2908 0,1005
Maturity -> Effectiveness 0,0726 0,1223 0,1272 0,5706 0,2850
Maturity -> Efficiency -0,1485 -0,0693 0,1773 0,8378 0,2025
Maturity ->
Innovativeness 0,1320 0,1590 0,1295 1,0189 0,1556
Examining the Effects of Agile Methods and Process Maturity 93
Table 3. Construct-indicator cross-loading matrix
Agility Effectiveness Efficiency Inno-
Agile1 0,5515 0,3127 0,1299 0,2173 0,5430
Agile10 0,3789 0,1255 0,2487 -0,0273 0,0404
Agile11 0,5548 0,1869 0,2705 0,1668 0,4009
Agile12 0,3367 0,0768 0,2214 -0,0083 0,0376
Agile13 0,3031 -0,0318 -0,0773 -0,0028 0,5196
Agile14 0,2710 -0,0128 0,0527 -0,0955 0,3336
Agile15 0,2448 -0,0225 -0,0154 0,0362 0,3367
Agile16 0,5672 0,1005 0,3635 0,0891 0,3932
Agile2 0,4581 0,1095 0,0276 0,0608 0,3085
Agile3 0,4256 0,0781 0,0076 0,0688 0,3247
Agile4 0,2935 -0,0001 0,0229 0,1066 0,5750
Agile5 0,5457 0,2462 0,1487 0,1581 0,3604
Agile6 0,7113 0,3264 0,3295 0,2493 0,2257
Agile7 0,4541 0,1525 0,0068 -0,0359 0,4008
Agile8 0,3141 0,2139 0,0911 0,1171 -0,0456
Agile9 0,2999 0,1206 0,1487 0,1379 0,2503
CMMI1 0,2075 0,0245 0,0326 0,0685 0,3686
CMMI10 0,5046 0,2144 0,2208 0,0453 0,6608
CMMI11 0,4052 -0,0227 -0,1179 -0,0375 0,6358
CMMI12 0,4344 0,1157 0,1575 0,0425 0,6293
CMMI13 0,3204 0,1073 0,0087 0,1387 0,7229
CMMI14 0,3130 0,1407 0,0694 0,2226 0,7038
CMMI15 0,1868 0,1349 -0,1231 0,0695 0,6136
CMMI16 0,2867 0,0802 -0,1057 0,0536 0,6906
CMMI17 0,1609 0,2231 -0,1458 0,1694 0,6626
CMMI18 0,1469 0,1367 0,0535 0,0439 0,2506
CMMI19 0,6020 0,2361 0,2840 0,0237 0,5287
CMMI2 0,1281 0,1380 -0,0326 0,2104 0,4804
CMMI20 0,4241 0,1507 0,1119 0,0457 0,5156
CMMI21 0,2926 0,1742 -0,0233 0,0957 0,6183
CMMI3 0,3116 0,0983 0,0007 0,2018 0,6391
CMMI4 0,2585 0,2190 0,0707 0,2861 0,7086
CMMI5 0,5284 0,1554 0,2045 0,2393 0,7431
CMMI6 0,4737 0,2091 0,2277 0,1620 0,6023
CMMI7 0,3866 0,0686 -0,0611 0,0388 0,6062
CMMI8 0,1414 0,0767 -0,0478 0,1276 0,6392
CMMI9 0,2022 0,1720 -0,0713 0,1647 0,5934
PDEffectiveness1 0,0764 0,3985 0,1221 0,1919 0,1971
PDEffectiveness2 0,2938 0,6317 0,2349 0,2655 0,1857
PDEffectiveness3 0,3106 0,7509 0,4392 0,5817 0,0940
PDEffectiveness4 0,2969 0,7539 0,4742 0,6027 0,2428
PDEfficiency1 0,3087 0,2378 0,7466 0,2607 0,0174
PDEfficiency2 0,3045 0,4715 0,8316 0,4067 0,0402
PDEfficiency3 0,3594 0,4760 0,7609 0,4216 0,2093
PDInnovativeness1 0,2505 0,6057 0,4483 0,8841 0,1661
PDInnovativeness2 0,2519 0,6588 0,5216 0,9004 0,2425
PDInnovativeness3 0,2073 0,4617 0,2596 0,8365 0,2203
94 M. Rönkkö, J. Peltonen, and C. Frühwirth
Maturity to the product development constructs are significant and hence we can
conclude that hypotheses 1, 2, and 5 are not supported. When we look at the
significance tests for the paths from Process Agility to the product development
constructs, we can see that the path to Efficiency is significant, path to Effectiveness
is marginally significant and path to Innovativeness is close to marginally significant.
Hence we conclude that Hypothesis 3 is supported, Hypothesis 4 is weakly supported
and Hypothesis 6 does not receive support from our data. The lack of support for the
last hypothesis is probably due to lack of statistical power due to a small sample size.
5 Discussion and Conclusions
In this paper we studied the product development performance effects of agile
software development practices and mature software process. To our understanding,
this undertaking is one of the first studies investigating the relative merits of these
models using survey research. Our key findings are that agile methods have a positive
effect on product development performance, but process maturity does not seem to
have an effect. This finding is a bit surprising, but can be explained by the fact that
the companies in the sample are small firms and the process development frameworks
are designed for predominantly large organizations that do larger software projects.
The fact that software process and innovativeness seem unrelated can be
interpreted as meaning that there are other organizational factors that affect
innovativeness much more than what software development methods are in use.
Regarding the limitations of the study, non-response analysis and descriptive
statistics indicate that the results of the study are probably rather well generalizable
among small and the smaller medium-sized software product firms. However, it needs
to be pointed out that we have studied only Finnish firms. A number of issues of
national culture or the peculiarities of Finland as a small economy that may influence
business operations can quite possibly affect the results..
Also, as with any paper relating to self-administered surveys and related to
composite variables is that our paper relies on self-reports. Although we spent
considerable efforts to make the survey items as clear as possible [45] and tried to
make the items such that also persons not familiar with CMMI or the particular agile
methods used could reliably answer the questionnaire, it is possible that some of the
items were too difficult for the respondents. However, our more detailed analysis [45]
indicated that these problems should not be more serious than what is typical for a
paper based on survey data. Finally, there have recently been some concerns related to
the validity of the PLS approach [53-54], and should these concerns be validated in
follow-up studies, they can have implications for this paper.
This paper has two managerial implications: First, at least in small firm context,
using agile methods can substitute for process maturity. Although our sample includes
only firms from CMMI levels four and below, this is an important finding that can, if
supported by further research, help guide initial software process improvement efforts
in smaller firms. Second, our data indicated that agile methods are probably more
appropriate for product development than non-agile.
Examining the Effects of Agile Methods and Process Maturity 95
1. Kautz, K., Madsen, S., Norbjerg, J.: Persistent problems and practices in information
systems development. Information Systems Journal 17, 217–239 (2007)
2. Vinekar, V., Slinkman, C.W., Nerur, S.: Can Agile and Traditional Systems Development
Approaches Coexist? an Ambidextrous View. Information Systems Management 23, 31–
42 (2006)
3. Balijepally, V., Mahapatra, R., Nerur, S., Price, K.H.: Are Two Heads Better Than One for
Software Development? The Productivity Paradox of Pair Programming. MIS
Quarterly 33, 91–118 (2009)
4. Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: A systematic
review. Information and Software Technology (2008)
5. Agrawal, M., Chari, K.: Software effort, quality, and cycle time: A study of CMM level 5
projects. IEEE Transactions on Software Engineering 33, 145–156 (2007)
6. Galin, D., Avrahami, M.: Are CMM program investments beneficial? Analyzing past
studies. IEEE Software 23, 81–87 (2006)
7. Highsmith, J., Cockburn, A.: Agile Software Development: The Business of Innovation.
IEEE Computer 34, 120–122 (2001)
8. Meso, P., Jain, R.: Agile Software Development: Adaptive Systems Principles and Best
Practices. Information Systems Management 23, 19–30 (2006)
9. Poppendieck, M.: Lean Software Development. In: Companion to the Proceedings of the
29th International Conference on Software Engineering, pp. 165–166. IEEE Computer
Society, Los Alamitos (2007)
10. Taylor, P.S., Greer, D., Sage, P., Coleman, G., McDaid, K., Lawthers, I., Corr, R.:
Applying an agility/discipline assessment for a small software organisation. In:
Proceedings of Product-Focused Software Process Improvement, pp. 290–304. Springer,
Berlin (2006)
11. Baskerville, R., Pries-Heje, J.: Short cycle time systems development. Information
Systems Journal 14, 237–264 (2004)
12. MacCormack, A., Verganti, R., Iansiti, M.: Developing products on “Internet time": The
anatomy of a flexible development process. Management Science 47, 133–150 (2001)
13. Germain, É., Robillard, P.N.: Engineering-based processes and agile methodologies for
software development: a comparative case study. The Journal of Systems & Software 75,
17–27 (2005)
14. Staples, M., Niazi, M., Jeffery, R., Abrahams, A., Byatt, P., Murphy, R.: An exploratory
study of why organizations do not adopt CMMI. Journal of Systems and Software 80,
883–895 (2007)
15. Boehm, B., Turner, R.: Using risk to balance agile and plan-driven methods. Computer 36,
57–66 (2003)
16. Baker, S.W.: Formalizing agility: an agile organization’s journey toward CMMI
accreditation. In: Proceedings of Agile Conference, pp. 185–192 (2005)
17. Merisalo-Rantanen, H., Tuunainen, T., Rossi, M.: Is extreme programming just old wine in
new bottles: A comparison of two cases. Journal of Database Management 16, 41–61
18. Paulk, M.C.: Extreme programming from a CMM perspective. IEEE Software 18, 19–26
19. Pressman, R.: Software Engineering: A Practitioner’s Approach. McGraw-Hill
Science/Engineering/Math. (2009)
96 M. Rönkkö, J. Peltonen, and C. Frühwirth
20. Jiang, J.J., Klein, G., Hwang, H.G., Huang, J., Hung, S.Y.: An exploration of the
relationship between software development process maturity and project performance.
Information & Management 41, 279–288 (2004)
21. Schach, S.R.: Object-oriented and classical software engineering. McGraw-Hill, Boston
22. Herbsleb, J., Zubrow, D., Goldenson, D., Hayes, W., Paulk, M.: Software quality and the
Capability Maturity Model. Communications of the ACM 40, 30–40 (1997)
23. Turner, R., Jain, A.: Agile meets CMMI: Culture clash or common cause? Extreme
Programming and Agile Methods—XP/Agile Universe 2002, 153–165 (2002)
24. Carlshamre, P.: Release Planning in Market-Driven Software Product Development:
Provoking an Understanding. Requirements Engineering 7, 139–151 (2002)
25. Jantunen, S., Smolander, K.: Towards global market-driven software development
processes: an industrial case study. In: Proceedings of the 2006 International Workshop on
Global Software Development for the Practitioner, pp. 94–100. ACM, Shanghai (2006)
26. SEI: CMMI for Development, version 1.2 (2006)
27. Beck, K., Andres, C.: Extreme programming explained: embrace change. Addison-Wesley
Professional, Reading (2004)
28. Turk, D., France, R., Rumpe, B.: Assumptions underlying agile software-development
processes. Journal of Database Management 16, 62–87 (2005)
29. Sutherland, J., Jakobsen, R., Johnson, K.: Scrum and cmmi level 5: The magic potion for
code warriors. In: Proceedings of the 41st Annual. Hawaii International Conference on
System Sciences, p. 466 (2008)
30. Cockburn, A.: Selecting a project’s methodology. IEEE Software 17, 64–71 (2000)
31. Subramanian, G.H., Jiang, J.J., Klein, G.: Software quality and IS project performance
improvements from software development process maturity and IS implementation
strategies. Journal of Systems and Software 80, 616–627 (2007)
32. Kusunoki, K., Nonaka, I., Nagata, A.: Organizational capabilities in product development
of Japanese firms: a conceptual framework and empirical findings. Organization Science 9,
699–718 (1998)
33. Kahn, K.B.: Market orientation, interdepartmental integration, and product development
performance. The Journal of Product Innovation Management 18, 314–323 (2001)
34. McDermott, C.M., O’Connor, G.C.: Managing radical innovation: an overview of
emergent strategy issues. Journal of Product Innovation Management 19, 424–438 (2002)
35. Veryzer, R.W.: Discontinuous Innovation and the New Product Development Process.
Journal of Product Innovation Management 15, 304–321 (1998)
36. Nidumolu, S.R.: Standardization, requirements uncertainty and software project
performance. Information and Management 31, 135–150 (1996)
37. Citrin, A.V., Lee, R.P., McCullough, J.: Information use and new product outcomes: The
contingent role of strategy type. Journal of Product Innovation Management 24, 259–273
38. Karlsson, L., Dahlstedt, A.G., Regnell, B., Nattoch Dag, J., Persson, A.: Requirements
engineering challenges in market-driven software development - An interview study with
practitioners. Information and Software Technology 49, 588–604 (2007)
39. Slaughter, S.A., Levine, L., Ramesh, B., Pries-Heje, J., Baskerville, R.: Aligning software
processes with strategy. Mis Quarterly 30, 891–918 (2006)
40. Tushman, M.L.: Special boundary roles in the innovation process. Administrative Science
Quarterly 22, 587–605 (1977)
41. McDonough, E.F.: Investigation of factors contributing to the success of cross-functional
teams. Journal of Product Innovation Management 17, 221–235 (2000)
Examining the Effects of Agile Methods and Process Maturity 97
42. Spreitzer, G.M.: Psychological empowerment in the workplace: Dimensions,
measurement, and validation. Academy of Management Journal 38, 1442–1465 (1995)
43. Tierney, P., Farmer, S.M.: Creative self-efficacy: Its potential antecedents and relationship
to creative performance. Academy of Management Journal 45, 1137–1148 (2002)
44. Rönkkö, M., Eloranta, E., Mustaniemi, H., Mutanen, O., Kontio, J.: Mustaniemi, H.,
Mutanen, O., Kontio, J.: Finnish Software Product Business: Results of National Software
Industry Survey 2007. Helsinki University of Technology (2007)
45. Rönkkö, M., Järvi, A., Mäkelä, M.M.: Measuring and comparing the adoption of software
process practices in the software product industry. In: Proceedings of Internationl
Conference on Software Process, Leipzig, Germany, pp. 407–419 (2008)
46. Dekleva, S., Drehmer, D.: Measuring software engineering evolution: A rasch calibration.
Information Systems Research 8, 95–104 (1997)
47. Brislin, R.W.: Back-Translation for Cross-Cultural Research. Journal of Cross-Cultural
Psychology 1, 185–216 (1970)
48. Dillman, D.A.: Mail and Internet surveys: the tailored design method. Wiley, New York
49. Oppenheim, A.N.: Questionnaire Design and Attitude Measurement Heinemann, London
50. Chin, W.W.: The partial least squares approach to structural equation modeling. In:
Marcoulides, G.A. (ed.) Modern Methods for Business Research, pp. 295–336. Lawrence
Erlbaum Associates Publishers, Mahwah (1998)
51. Ahlemann, F., Urbach, N.: Structural Equation Modeling in Information Systems Research
Using Partial Least Squares. Journal of Information Technology Theory and Application
(JITTA) 11 (2010)
52. Dijkstra, T.: Some comments on maximum likelihood and partial least squares methods.
Journal of Econometrics 22, 67–90 (1983)
53. Evermann, J., Tate, M.: Testing Models or Fitting Models? Identifying Model
Misspecification in PLS. In: Proceedings of the ICIS 2010 (2010)
54. Rönkkö, M., Ylitalo, J.: Construct Validity in Partial Least Squares Path Modeling. In:
Proceedings of the ICIS 2010 (2010)
... According to [15] maturity is defined as the degree to which processes are institutionalized and efficient. Also, the maturity of the process relates to the degree of utilization of the established software engineering method [23]. Instruments used to assess the maturity of elements (persons, objects or systems) and selecting appropriate actions for raising this element to a higher level of maturity are called models of maturity. ...
... This is confirmed when looking at articles where agile is introduced into an organisation and improves the quality of software delivery, without regard of the achieving any CMMI maturity level Jakobsen & Sutherland, 2009;Koutsoumpos & Marinelarena, 2013;Leusink, 2012;Morris, 2012;Rönkkö, Peltonen, & Frühwirth, 2011). Many of this research was con-ducted by introducing agile practices into either an already (CMMI) mature environment or where the primary goal was not necessarily maturity but instead successful software delivery. ...
Full-text available
Background/Aim/Purpose: A commonly implemented software process improvement framework is the capability maturity model integrated (CMMI). Existing literature indicates higher levels of CMMI maturity could result in a loss of agility due to its organizational focus. To maintain agility, research has focussed attention on agile maturity models. The objective of this paper is to find the common research themes and conclusions in agile maturity model research. Methodology: This research adopts a systematic approach to agile maturity model research, using Google Scholar, Science Direct, and IEEE Xplore as sources. In total 531 articles were initially found matching the search criteria, which was filtered to 39 articles by applying specific exclusion criteria. Contribution:: The article highlights the trends in agile maturity model research, specifically bringing to light the lack of research providing validation of such models. Findings: Two major themes emerge, being the coexistence of agile and CMMI and the development of agile principle based maturity models. The research trend indicates an increase in agile maturity model articles, particularly in the latter half of the last decade, with concentrations of research coinciding with version updates of CMMI. While there is general consensus around higher CMMI maturity levels being incompatible with true agility, there is evidence of the two coexisting when agile is introduced into already highly matured environments. Future Research: Future research direction for this topic should include how to attain higher levels of CMMI maturity using only agile methods, how governance is addressed in agile environments, and whether existing agile maturity models relate to improved project success.
The objective of the present work is to build and test a framework which makes use of process-based software metrics to determine the defects in software projects in an agile software development environment. A methodological framework based on fuzzy linguistic modelling has been proposed to predict the defect density using various process metrics derived from literature studies to measure the attributes stated in the agile manifesto. Further, the model is investigated by using the data set from the PROMISE software engineering repository, and its performance has been compared with existing models from the literature. The proposed model shows better accuracy (for projects with size ≥ 50 KLOC) as observed from statistical results, i.e. RMSE (18.69), NRMSE (0.0110), MMRE (0.0539) and BMMRE (0.0585). The value of R2 for all projects size up to 10 KLOC is 0.993, projects with size 10–50 KLOC is 0.998, and projects with size ≥ 50 KLOC is 0.997. The main contribution of the framework lies in the use of the process metrics and their linguistic assessment. Results obtained from the linguistic model emphasise the value of concepts related to customer involvement and interactions, the collaboration between stakeholders, responding to change, i.e. flexibility, team experience, skills, communication and coordination, as per agile manifesto.
Context Practitioners and researchers often claim that agile methods have moved into the mainstream for the last few years. To support this claim they refer to recent industrial surveys which tend to report high rates of agile method usage. However many of these industrial surveys are conducted by agile consultants, tool vendors, professional societies and independent technology and market research organizations. This raises some important concerns about the possible conflict of interest and the overall trustworthiness of these studies. Objective In response to the above concerns, a secondary study was carried out. Its objective was to examine industrial surveys published in 2011 and 2012, determine the extent to which we could trust their reported high rates of agile method usage and provide recommendations on how quality of research could be imporved in the future. Method Following a rigorous search procedure, nine industrial surveys on agile method usage published in 2011 and 2012 were extracted from both academia and industry. Their thoroughness in reporting and trustworthiness were evaluated using a newly proposed assessment framework based on Guba's four attributes of trustworthiness (truth value, applicability, consistency and neutrality) and a number of methods for assessing survey research in related fields as information, communication and management studies. Results The careful examination of the reviewed surveys shows that most of the studies have insufficient thoroughness in reporting and (subsequently) low trustworthiness. Only one (out of nine) study is considered as a scientific contribution in determining the current 2011/2012 rate of agile method usage. Conclusions The obtained results support our initial considerations about the trustworthiness of recent industrial surveys on agile method usage and suggest a number of recommendations for increasing the quality and value of future survey research in this regard.
Full-text available
Provides a nontechnical introduction to the partial least squares (PLS) approach. As a logical base for comparison, the PLS approach for structural path estimation is contrasted to the covariance-based approach. In so doing, a set of considerations are then provided with the goal of helping the reader understand the conditions under which it might be reasonable or even more appropriate to employ this technique. This chapter builds up from various simple 2 latent variable models to a more complex one. The formal PLS model is provided along with a discussion of the properties of its estimates. An empirical example is provided as a basis for highlighting the various analytic considerations when using PLS and the set of tests that one can employ is assessing the validity of a PLS-based model. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
Despite differences in definitions, researchers understand that radical innovation within an organization is very different from incremental innovation and that it is critical to the long-term success of firms. Unfortunately, research has also shown that it is often difficult to get support for radical projects in large firms, where internal cultures and pressures often push efforts toward more low risk, immediate reward, incremental projects. Interestingly, we know considerably less about the effective management of the product development process in the radical than in an incremental context. The purpose of this study is to explore the process of radical new product development from a strategic perspective, and to outline key observations and challenges that managers face as they move these projects to market. The findings presented here represent the results of a longitudinal (since 1995), multidisciplinary study of radical innovation projects. A multiple case study design was used to explore the similarities and differences in management practices applied to twelve radical innovation projects in ten large, established North American firms. The findings are grouped into three high-level strategic themes. The first theme, market scope, discusses the challenges associated with the pursuit of familiar versus unfamiliar markets for radical innovation. The second theme of competency management identifies and discusses strategic challenges that emerge as firms stretch themselves into new and unfamiliar territory. The final theme relates to the people issues that emerge as both individuals and the project teams themselves try to move radical projects forward in organizations that are not necessarily designed to support such uncertainty. A breadth of subtopics emerge within and across this framework relating to such ideas as risk management, product cannibalization, team composition, and the search for a divisional home. Taken together, our observations reinforce the emerging literature that shows that project teams engaging in radical innovation encounter a much different set of challenges than those typically faced by NPD teams engaged in incremental innovation.
Full-text available
This article explores extreme programming (XP) as an information systems development approach and argues that it is mainly old wine in new bottles. We take an interpretive and critical view of the phenomenon. We made an empirical study of two companies that apply an XP-style development approach throughout the information systems development life cycle. The results of our research suggest that XP is a combination of best practices of traditional information systems development methods. It is hindered by its reliance on talented individuals, which makes its large-scale deployment as a general-purpose method difficult. We claim that XP can be useful for small teams of domain experts who are physically close together and able to communicate well with the end users, and who are good designers and implementers. However these skilled and motivated individuals with high working moral can exhibit high productivity regardless of the methods used if they are not overly constrained by bureaucracy.
In its eighth edition, the book has again been revised and redesigned, undergoing a substantial content update that addresses new topics in what many have called “the engineering discipline of the 21st-century.” Entertaining and informative sidebars and marginal content have been expanded and make the book still easier-to-use in the classroom and as a self-study guide. Four new chapters, emphasizing software security and the unique challenges of developing software for mobile applications, have been added to this edition. In addition, new content has been added to many other chapters. The eighth edition is organized into 5 parts: • Part 1, The Software Process, presents both prescriptive and agile process models. • Part 2, Modeling, presents modern analysis and design methods with an emphasis on you UML-based modeling. • Part 3, Quality Management, addresses all aspects of software testing and quality assurance, formal verification techniques, and change management. • Part 4, Managing Software Projects, presents software topics that are relevant to those who plan, manage, and control a software project. • Part 5, Advanced Topics, presents dedicated chapters that address software process improvement and future software engineering trends.
Although many new-products professionals may harbor hopes of developing "the next big thing" in their respective industries, most product development efforts focus on incremental innovations. Accordingly, most research on the new-product development (NPD) process focuses on the development of evolutionary products. For new-products professionals seeking insights into the means for achieving breakthrough innovations, a fundamental question remains unanswered: Does the NPD process for discontinuous products differ from the process for incremental, or continuous, products? To provide a better understanding of managerial practices associated with discontinuous innovation, Robert Veryzer presents findings from an in-depth study of eight discontinuous product development projects. The study explores the key factors that affect the discontinuous NPD process, as well as the methods that the firms in this study use for assessing the radically new products they have in development. From the findings in this study, he develops a descriptive model of the discontinuous product development process, and he offers insights into the requirements for effective management of discontinuous innovation projects. Although half the firms in the study use a formal process for evaluating radically innovative products, the participants in the study generally do not employ a formal, highly structured process for managing discontinuous NPD efforts. However; these firms do follow a consistent, logical process in the development of radical innovations, and their process differs significantly from incremental NPD processes. The processes used by the firms in this study are more exploratory and less customer driven than the typical, incremental NPD process. The impetus for all the projects in this study comes from the convergence of developing technologies, various contextual or environmental factors (for example, government regulations), and a product champion or visionary. Starting from these drivers, the discontinuous NPD process focuses on formulating a product application for the emerging technologies. In all cases, these firms developed prototypes at nit earlier stage than the typical, incremental NPD process. To aid in the formulation of a new product application from emerging technologies, prototype construction in these discontinuous NPD projects precedes opportunity analysis, assessment of market attractiveness, market research, and financial analysis. (C) 1998 Elsevier Science Inc.
Various research studies have shown that a market orientation and interdepartmental integration can positively influence product development performance. Addressed in this article is whether market orientation and interdepartmental integration both equally influence product development performance, whether one of these constructs is more influential than the other, and whether such influence is dependent on the type of department being examined? Analyzing survey data from 156 marketing, manufacturing, and R&D managers, the tentative results suggest that a market orientation and interdepartmental integration correlate to improved product development and product management performance in varying degrees across these three manager sets. It appears that a positive relationship between market orientation and product development performance is likely to be reflected by the marketing department, while marketing and manufacturing departments are likely to reflect a positive relationship between the general c onstruct of market orientation and product management performance. Manufacturing managers also reflect a positive relationship between interdepartmental integration and product development and product management performance. Further analyses involving the elements of a market orientation and interdepartmental integration find that a customer orientation appears important to performance in the case of marketing managers, and that collaboration is important to performance in the case of manufacturing managers. R&D managers did not reflect any statistically significant relationships between market orientation, interdepartmental integration, their constructs, and performance. These results should not be taken as refuting the claim of an important relationship between market orientation and product development performance, however. The present results refine our understanding of market orientation to consider department-specific effects, as well as temper the claims that implementing a market orientation will readily lead to improved product development performance across all departments in an organization. This may or may not be the case, depending on the focal department.
Using data from two different firms, this study tested a new construct, creative selfefficacy, tapping employees' beliefs that they can be creative in their work roles. Results support the discriminant validity of the construct and indicate that job tenure, job self-efficacy, supervisor behavior, and job complexity contribute to creative efficacy beliefs. Creative self-efficacy also predicted creative performance beyond the predictive effects of job self-efficacy. Differences in results between white-collar and blue-collar samples suggest considerations for both theory and practice.