Conference PaperPDF Available

Validity of research on large-scale agile projects


Abstract and Figures In this extended abstract we introduce and briefly discuss key challenges that possess threats to validity of empirical studies conducted within the context of distributed agile software development in the large scale. These challenges are based on our own ...
Content may be subject to copyright.
Validity of Research on Large-Scale Agile Projects
Darja Šmite
, Burak Turhan
School of Computing, Blekinge Institute of Technology,
SE-371 79 Karlskrona, Sweden
Department of Information Processing Science, University of Oulu
90014 Oulu, Finland
1 ,
In this extended abstract we introduce and briefly discuss key challenges that possess
threats to validity of empirical studies conducted within the context of distributed
agile software development in the large scale. These challenges are based on our own
research experience with agile and global software development.
To lead our discussion we use a hypothetical research project (see Fig.1), in which
a team of researchers conducts empirical studies on a large-scale agile project
consisting of three agile teams in two geographically distant locations. Common to
many studies in such environments, researchers in our example have access only to
one of the sites, and therefore data collection is performed primarily onshore. To get
the offshore perspective researchers in our example distribute questionnaires to the
remote sites with the help of the onshore team, as well as elicit additional information
regarding the remote teams from the onshore team members.
Agile !
Data !
Fig. 1. Illustration of data collection in a hypothetical research project
Data Collection Surveys: When data collection from the offshore teams is
mediated by the onshore team, the researchers risk in receiving politically-correct
responses due to fears and self-protective behavior. Besides, due to cultural
peculiarities in collectivistic societies the respondents might even answer the surveys
all together [3]. Similarly, in high power distance societies respondents might seek the
“correct” answer to please their management and not share the actual course of events
[3]. This means that researchers shall be especially careful in designing survey studies
and perhaps consider electronically mediated interviews instead to address the threats
to internal and construct validity.
Data Collection Reliability of Perception-Based Studies: When eliciting
information about the remote teams and environment from their onshore counterparts,
the researchers risk in receiving that site’s (potentially biased) view on the
collaboration, which may be influenced by perceptions and prejudices, and is often
found to be more pessimistic as opposed to direct measurements [1]. Methodological
and data source triangulation in such studies is therefore essential to address the
threats to internal and construct validity, respectively.
Data Analysis Process Conformance: Due to the lack of collocation, agile
practices are often implemented in the spirit of “lightweight” methods, but with
adaptations to meet the demands of distribution [2, 4]. First of all, many cross-site
practices might be reduced to the minimum due to geographic dispersion or even
skipped due to a limited collaboration window caused by temporal distance.
Furthermore, project members in different locations may have different interpretation
of agile methods and concepts as a result of cultural differences and disparate
guidance that shaped their perceptions. Therefore, each team might function
differently. This means that teams at different locations might claim that they practice
agile development, but a number of culture-related concerns suggest several important
limitations. For example, autonomy and shared leadership are hindered in cultures
having higher power distance [3]. Although team members possessing collectivistic
ideas might have a strong inclination towards the group, promotion-seeking behavior
might cause willingness to be evaluated upon individual performance. Such cultures
are also associated with high attrition rates that endanger the stability of agile teams.
Concerns about conformance with cross-team and within-team practices pose serious
threats to internal validity of the study. Therefore, researchers shall capture the
heterogeneity of the project environment and avoid synthesizing the use of agile
methods while ignoring the differences in their implementation and interpretation.
Conclusion: To address the validity threats in research on large scale agile
projects, we recommend researchers: (i) to employ direct data collection methods and
observation based research strategies to the best extent possible, (ii) and to triangulate
data sources and methodologies whenever the former is not feasible.
Acknowledgments. This work is supported in part by the Knowledge Foundation
under KK-Hög grant 2012/0200 in Sweden and by the Finnish Funding Agency for
Technology and Innovation (TEKES) under Cloud Software Program in Finland.
1. Nguyen Duc, A., Cruzes, D.S., Conradi, R.: Dispersion, coordination and performance in
global software teams: A systematic review. In: ACM-IEEE International Symposium on
Empirical Software Engineering and Measurement, pp.129-138. (2012)
2. Ramesh, B., Cao, L., Mohan, K., Xu, P.: Can distributed software development be agile?
Commun. ACM 49(10): 41-46 (2006)
3. Hofstede, G., Hofstede, G. J., and Minkov, M.: Cultures and Organizations: Intercultural
Cooperation and Its Importance for Survival. 3
edition, McGraw-Hill (2010)
4. Reifer, D.J.; Maurer, F.; Erdogmus, H.: Scaling agile methods. IEEE Software 20(4): 12,14
... Coordination of software development activities and resource management are significant challenges, especially within multi-site and large-scale project contexts [2,6]. One important issue is the recognition and management of dependencies between the parts of software that is to be developed by different teams, and correspondingly tools, approaches and principles have been developed for the purpose, like configuration management and issue-tracking systems. ...
Conference Paper
Full-text available
It is important to identify and act upon coordination problems within and across distributed software development teams in the context of global software development (GSD). Such problems jeopardize the quality of the system under development, and their identification is paramount to mitigate quality related risks. In this paper, we introduce a visualization tool, namely error-handling graph, that shows the interactions of development teams in terms of the reported and fixed errors during software development, and guides project managers to focus their efforts towards locating root problem causes. We present an example based on the data collected from the issue repository of an industrial project and discuss error-handling graph's potential implications. The initial feedback from the piloting industrial team is positive, though a thorough evaluation is still needed. In practice, error graphs could be helpful in identifying the sources of problems and the opportunities for improvement, by creating awareness through data-driven insights for enhancing the way of working within and across software development teams.
... Even agile software development can be a complicated process, especially when dealing with multi-site large scale projects [1]. The variety of tools used in software development , while directly guiding the process itself, can also offer meaningful statistics based on the process data logged in such tools. ...
Conference Paper
Full-text available
Effective team coordination is crucial for successful global software projects. Although considerable research effort has been made in this area, no agreement has been reached on the influence of dispersion on team coordination and performance. The objective of this paper is to summarize the evidence on the relationship among context dispersion, team coordination and performance in global software projects. We have performed a Systematic literature review (SLR) to collect relevant studies and a thematic analysis to synthesize the extracted data. We found 28 primary studies reporting the impact of five dispersion dimensions on team performance. Previously, only two primary studies considered and distinguished all of these dispersion dimensions in studying dispersed team performance. The dispersion dimensions affect team outcomes indirectly through influencing organic and mechanistic coordination processes. Empirical evidence show that geographical dispersion impacts negatively and temporal dispersion has a mixed effect on team performance. While studies with teams working across different time zones shows a tendency that the team performance is pessimistically perceived, studies that use direct measure on task performance shows a positive association to temporal dispersion. The paper provides implications for future research and practitioners in establishing effective distributed team coordination.
Full-text available
Software development organizations are striving to blend agile software development methods such as extreme programming and distributed development to develop and evolve software systems at Internet speed. Agile and distributed development approaches differ significantly in their tenets, while agile methods mainly rely on informal process, distributed software development typically relies on formal mechanisms. Some organizations adopted practices of blending the unique characteristics of agile and distributed development. These practices are in the spirit of lightweight methods, but have been adopted to meet the competing demands of distributed development as well. These practices have classified into five groups, continuously adjust the process, facilitate knowledge sharing, improve communication, build trust, trust but verify. Careful incorporation of agility in distributed software development environments is essential in addressing several challenges.
Full-text available
Using agile methods to develop large systems presents a thorny set of issues. If large teams are to produce lots of software functionality quickly, the agile methods involved must scale to meet the task. After all, a small team could create the software if the functionality to be delivered was small and, conversely, could be delivered given we had the time. Scaling agile teams thus becomes an issue if the only option for meeting a system delivery deadline is to have many developers working concurrently.