Conference PaperPDF Available

Decoding Grounded Theory for Software Engineering

Abstract

Grounded Theory, while becoming increasingly popular in software engineering, is also one of the most misunderstood, misused, and poorly presented and evaluated method in software engineering. When applied well, GT results in dense and valuable explanations of how and why phenomena occur in practice. GT can be applied as a full research method leading to mature theories and also in limited capacity for data analysis within other methods, using its robust open coding and constant comparison procedures. This technical briefing will go through the social origins of GT, present examples of grounded theories developed in SE, discuss the key challenges SE researchers face, and provide a gentle introduction to socio-technical grounded theory, a variant of GT for software engineering research.
Decoding Grounded Theory for Software
Engineering
Rashina Hoda
Faculty of Information Technology
Monash University
Melbourne, Australia
rashina.hoda@monash.edu
Abstract—Grounded Theory, while becoming increasingly pop-
ular in software engineering, is also one of the most misunder-
stood, misused, and poorly presented and evaluated method in
software engineering. When applied well, GT results in dense
and valuable explanations of how and why phenomena occur in
practice. GT can be applied as a full research method leading
to mature theories and also in limited capacity for data analysis
within other methods, using its robust open coding and constant
comparison procedures. This technical briefing will go through
the social origins of GT, present examples of grounded theories
developed in SE, discuss the key challenges SE researchers face,
and provide a gentle introduction to socio-technical grounded
theory, a variant of GT for software engineering research.
Index Terms—socio-technical grounded theory, grounded the-
ory, research method, software engineering
I. MOTI VATI ON
Grounded Theory (GT) is one of the only research methods
that enables rigorous and systematic theory development. At
its core, GT enables the identification of patterns in data
collected from a variety of sources to explain how and why
phenomena occur.
Introduced in the mid 1960’s in Sociology, GT is a complete
research method including basic steps of literature review, data
collection, analysis, and advanced steps of theory development
[3]. GT comes in three traditional versions: Classic or Glase-
rian GT [3], Strauss-Corbinian GT [14], and Constructivist
GT [1]. All three have wide ranging applications.
GT has gained steady popularity in software engineering
(SE) [12]. This growth can be attributed to GT enabling the
study of human and social aspects of software engineering,
an increasingly growing area of interest, and the need for
theory development in a young discipline such as software
engineering [2]. Because GT enables the study of real world
phenomenon, it is well suited to investigating industrial prac-
tice of software engineering. Arguably, GT addresses what has
been dubbed as a “grand challenge” of research – combining
research rigour with industrial relevance.
And yet, GT is also one of the most misunderstood, mis-
used, and poorly presented and evaluated method in software
engineering. A critical review of GT in software engineering
identified around 1,763 papers under “grounded theory” and
“software engineering” [12]. The same review highlighted how
GT practice in SE suffers from lack of application details,
weak applications, and even method misuse and abuse.
Given the growing interest in GT as well as widespread mis-
conceptions, this technical briefing aims to help SE researchers
and practitioners better understand GT and the challenges
faced by SE researchers in applying and evaluating GT studies.
It provides a gentle introduction and an overview of Socio-
Technical Grounded Theory (STGT) – a GT method variant
developed for socio-technical research contexts.
II. BR IE FIN G DETAI LS
A. Intended Audience
This technical briefing aims to cater to both researchers
and industry practitioners, especially those involved in user
experience, user centred design, end user development, and
engaging with particular user communities. As research and
development (R&D) becomes a staple and prominent part of
most global software companies such as Google, Microsoft,
and Facebook, in addition to traditional research and academic
organisations, the importance of robust methods for studying
human behaviour in software development and use has become
more important than ever.
B. Application Contexts
Applying GT to study the users of software systems will
enable deep and rich understanding of their human characteris-
tics, motivations, needs, preferences, strengths, and limitations
so that more ethical and human-centred software can be
produced. Similarly, GT studies of software developers, teams,
managers, customers, and stakeholders can fuel human bias
management and process improvement, leading to improved
outcomes. In the age of artificial intelligence competing
to mimic and improve human capabilities, GT studies are
particularly important to help explain the human-technology
relationship that pervades all aspects of life and instill core
human values (e.g. ethics, privacy, social justice) into the AI-
powered software systems we develop.
C. Learning Outcomes
The expected learning outcomes from this briefing include:
Understanding GT’s sociological traditions.
Learning about the outcomes of GT studies in SE re-
search.
Understanding the challenges faced by SE researchers in
applying and evaluating GT studies.
Understanding the misalignment between traditional GT
and the SE research contexts.
Learning about socio-technical grounded theory (STGT)
for SE research.
Understanding STGT’s potential and context of use.
D. Planned Structure
While it is difficult to provide a complete overview of GT
with all its techniques and procedures in just over an hour, this
technical briefing will aim to cover the following key areas:
Origin and social traditions of GT.
A brief summary of GT in software engineering, includ-
ing state of practice and key challenges.
Examples of theories resulting from GT studies in SE
research.
A gentle introduction to socio-technical grounded theory,
including guidelines for practice and evaluation.
E. Relevant Study Materials
Participants will benefit from reading the article “developing
a grounded theory to explain the practices of self-organizing
agile teams” [6]. A video of “a talk and QA with Rashina
Hoda on Grounded Theory in Software Engineering”, hosted
by Prof Margaret-Anne Storey at the Empirical Software En-
gineering course, University of Victoria, Canada, also provides
an overview of GT in SE research along with discussions on
its key concepts and practices [13].
III. PRE SE NT ER BI O & NE XT ST EPS
Associate Professor Rashina Hoda is a leading interna-
tional expert in the use of Grounded Theory in Software
Engineering. With her research teams, over the years, she
has developed several grounded theories to explain important
software engineering phenomena such as agile transformations
[4], impact of inadequate customer involvement [5], self-
organising agile teams [7], variations in scrum practice [9],
how self-assignment works [8], the role of the scrum master
[11], and the role of the project manager on agile projects[10].
Based on over 15 years of experience in actively conducting,
supervising, reviewing and editing numerous GT studies, Hoda
has developed the Socio-Technical Grounded Theory method,
a variant of GT for socio-technical research. Full details
of the method can be found in her upcoming book on the
subject, including clear and practical guidance on practicing
STGT in socio-technical fields such as software engineering,
computer science, information system, artificial intelligence,
user experience, and more.
REFERENCES
[1] Charmaz, K. [2006]. Constructing grounded theory: A
practical guide through qualitative analysis, sage.
[2] Ebert, C. [2018]. 50 years of software engineering:
Progress and perils, IEEE Software 35(5): 94–101.
[3] Glaser, B. G. and Strauss, A. L. [1967]. Discovery
of grounded theory: Strategies for qualitative research,
Adline.
[4] Hoda, R. and Noble, J. [2017]. Becoming agile: a
grounded theory of agile transitions in practice, 2017
IEEE/ACM 39th International Conference on Software
Engineering (ICSE), IEEE, pp. 141–151.
[5] Hoda, R., Noble, J. and Marshall, S. [2011]. The
impact of inadequate customer collaboration on self-
organizing agile teams, Information and Software Tech-
nology 53(5): 521–534.
[6] Hoda, R., Noble, J. and Marshall, S. [2012a]. Develop-
ing a grounded theory to explain the practices of self-
organizing agile teams, Empirical Software Engineering
17(6): 609–639.
[7] Hoda, R., Noble, J. and Marshall, S. [2012b]. Self-
organizing roles on agile software development teams,
IEEE Transactions on Software Engineering 39(3): 422–
444.
[8] Masood, Z., Hoda, R. and Blincoe, K. [2020a]. How agile
teams make self-assignment work: a grounded theory
study, Empirical Software Engineering .
URL: https://doi.org/10.1007/s10664-020-09876-x
[9] Masood, Z., Hoda, R. and Blincoe, K. [2020b]. Real
world scrum a grounded theory of variations in practice,
IEEE Transactions on Software Engineering .
[10] Shastri, Y., Hoda, R. and Amor, R. [2021a]. The role
of the project manager in agile software development
projects, Journal of Systems and Software 173: 110871.
[11] Shastri, Y., Hoda, R. and Amor, R. [2021b]. Spearhead-
ing agile: The role of the scrum master in agile projects,
Empirical Software Engineering 26(3).
[12] Stol, K.-J., Ralph, P. and Fitzgerald, B. [2016]. Grounded
theory in software engineering research: a critical review
and guidelines, Proceedings of the 38th International
Conference on Software Engineering, pp. 120–131.
[13] Storey, M.-A. and Hoda, R. [2020]. A talk and QA
with Rashina Hoda on grounded theory in software
engineering.
URL: https://youtu.be/z-Yr27qtePM
[14] Strauss, A. and Corbin, J. [1990]. Basics of qualitative
research, Sage publications.
... Since the focus of this study is to understand processes, examine strategies and explore underlying behaviors, grounded theory was particularly suitable (Masood et al., 2020a). In addition, the grounded theory method is also applicable to explaining how and why phenomena occur (Hoda, 2021). Hence, in this study, the grounded theory approach (Glaser and Strauss, 1967) was used as a qualitative research method. ...
Article
Full-text available
Purpose This study aims to create a better understanding of how practitioners implement and work Agile while balancing the tensions arising between stability and change. Design/methodology/approach A grounded theory approach was used to explore what happens in practice when software development teams implement and work Agile. The empirical data consists of twenty semi-structured interviews with practitioners working in fourteen different organizations and in six different Agile roles. Findings As a result, a substantive theory was presented of continuously balancing between stability and change in Agile teams. In addition, the study also proposes three guidelines that can help organizations about to change their way of working to Agile. Research limitations/implications The inherent limitation of a grounded theory study is that a substantial theory can only explain the specific contexts explored in that study. Thus, this study's contribution is a substantial theory that needs to be further developed and improved. Practical implications The proposed guidelines can help organizations about to change their way of working to Agile. They can also assist organizations in switching from “doing Agile” to “being Agile”, thus becoming more successful. Originality/value The new perspective that this study contributes is the fact that our discovered categories show that several inherent processes are ongoing at the same time in order to balance the need to have both stability and change.
Article
Full-text available
Scrum innovated the role of the scrum master in software engineering. The scrum master is envisioned in agile literature as the “servant leader” who serves the team in a multitude of different ways, which include promoting scrum, facilitating the team’s functioning, and removing obstacles. However, empirical studies focusing on the role of the scrum master in practice are scarce. To address this gap, a Grounded Theory study with a mixed methods approach was carried out which included semi-structured interviews with 39 software practitioners and a questionnaire with 47 respondents. In this study, we present and describe the scrum master’s role in agile projects in terms of (a) the grounded theory of the role of the scrum master which involves everyday activities of facilitating, mentoring, negotiating, process adapting, coordinating, and protecting; (b) the varying involvement of the scrum master in selected agile practices carried out by the team; and (c) a positive association between the presence of the scrum master and the frequency with which agile practices are carried out by the team. This study presents for the first time a multifaceted study of the multiple dimensions of the scrum master role and will enable practitioners to better manage expectations of this role in practice.
Article
Full-text available
Agile teams are not meant to have project managers. Instead, agile methods such as Scrum and XP define roles such as product owner, scrum master, and coach. Studies have uncovered the existence of the project manager in agile projects, pointing to disconnect between theory and practice. To address this gap, a Grounded Theory study with a mixed methods approach was conducted using multiple sources of data including over 45 h of interviews with 39 software practitioners and quantitative data from 57 questionnaire respondents. We present and describe the project manager’s role in agile projects in terms of (a) everyday activities: facilitating, mentoring, negotiating, coordinating, and protecting, performed by the project manager using; (b) three management approaches: hard, moderate, and soft; (c) four traditional project management activities continued to be performed by them, including: tracking project progress, reporting on project status, budgeting and forecasting, and managing personnel; and (d) the influence of the presence of the project manager on the frequency with which agile activities are carried out by the teams. Our study highlights the continued presence of the role of the project manager in agile software projects as a part of the transition from traditional to agile ways of working.
Article
Full-text available
Scrum, the most popular agile method and project management framework, is widely reported to be used, adapted, misused, and abused in practice. However, not much is known about how Scrum actually works in practice, and critically, where, when, how and why it diverges from Scrum by the book. Through a Grounded Theory study involving semi-structured interviews of 45 participants from 30 companies and observations of five teams, we present our findings on how Scrum works in practice as compared to how it is presented in its formative books. We identify significant variations in these practices such as work breakdown, estimation, prioritization, assignment, the associated roles and artefacts, and discuss the underlying rationales driving the variations. Critically, we claim that not all variations are process misuse/abuse and propose a nuanced classification approach to understanding variations as standard, necessary, contextual, and clear deviations for successful Scrum use and adaptation.
Article
Full-text available
Self-assignment, a self-directed method of task allocation in which teams and individuals assign and choose work for themselves, is considered one of the hallmark practices of empowered, self-organizing agile teams. Despite all the benefits it promises, agile software teams do not practice it as regularly as other agile practices such as iteration planning and daily stand-ups, indicating that it is likely not an easy and straighforward practice. There has been very little empirical research on self-assignment. This Grounded Theory study explores how self-assignment works in agile projects. We collected data through interviews with 42 participants representing 28 agile teams from 23 software companies and supplemented these interviews with observations. Based on rigorous application of Grounded Theory analysis procedures such as open, axial, and selective coding, we present a comprehensive grounded theory of making self-assignment work that explains the (a) context and (b) causal conditions that give rise to the need for self-assignment, (c) a set of facilitating conditions that mediate how self-assignment may be enabled, (d) a set of constraining conditions that mediate how self-assignment may be constrained and which are overcome by a set of (e) strategies applied by agile teams, which in turn result in (f) a set of consequences, all in an attempt to make the central phenomenon, self-assignment, work. The findings of this study will help agile practitioners and companies understand different aspects of self-assignment and practice it with confidence regularly as a valuable practice. Additionally, it will help teams already practicing self-assignment to apply strategies to overcome the challenges they face on an everyday basis.
Conference Paper
Full-text available
The project manager has been a ubiquitous feature of traditional software development projects. However, agile software development (ASD) methods which emphasize self-organizing teams and rapid response to change have done away with the project manager’s title. New job titles such as the scrum master and product owner have been introduced instead. It is unclear as to what extent the “project manager” is still encountered in the agile software industry. An online survey was posted out to agile special interest groups on popular social media platforms to discover the frequency of the job title “project manager” in agile projects. Analysis of the 97 responses from 31 countries around the world revealed that: a) the title of project manager is still widely used (67%); b) there is a correlation between the team size and presence of project manager such that there is a higher probability the project manager will be present in teams of 5-10 members and those over 25 members; and c) there is an inverse correlation between the co-location of a team and presence of project manager. Further research is needed to better understand why the project manager continues to be present on ASD projects and how their role may have changed.
Article
Full-text available
Self-organizing teams have been recognized and studied in various forms-as autonomous groups in socio-technical systems, enablers of organizational theories, agents of knowledge management, and as examples of complex-adaptive systems. Over the last decade, self-organizing teams have taken center stage in software engineering when they were incorporated as a hallmark of Agile methods. Despite the long and rich history of self-organizing teams and their recent popularity with Agile methods, there has been little research on the topic within software wngineering. Particularly, there is a dearth of research on how Agile teams organize themselves in practice. Through a Grounded Theory research involving 58 Agile practitioners from 23 software organizations in New Zealand and India over a period of four years, we identified informal, implicit, transient, and spontaneous roles that make Agile teams self-organizing. These roles-Mentor, Coordinator, Translator, Champion, Promoter, and Terminator-are focused toward providing initial guidance and encouraging continued adherence to Agile methods, effectively managing customer expectations and coordinating customer collaboration, securing and sustaining senior management support, and identifying and removing team members threatening the self-organizing ability of the team. Understanding these roles will help software development teams and their managers better comprehend and execute their roles and responsibilities as a self-organizing team.
Article
Full-text available
Software Engineering researchers are constantly looking to improve the quantity and quality of their research findings through the use of an appropriate research methodology. Over the last decade, there has been a sustained increase in the number of researchers exploring the human and social aspects of Software Engineering, many of whom have used Grounded Theory. We have used Grounded Theory as a qualitative research method to study 40 Agile practitioners across 16 software organizations in New Zealand and India and explore how these Agile teams self-organize. We use our study to demonstrate the application of Grounded Theory to Software Engineering. In doing so, we present (a) a detailed description of the Grounded Theory methodology in general and its application in our research in particular; (b) discuss the major challenges we encountered while performing Grounded Theory’s various activities and our strategies for overcoming these challenges; and (c) we present a sample of our data and results to illustrate the artifacts and outcomes of Grounded Theory research. KeywordsEmpirical research–Software engineering–Grounded theory–Agile software development–Self-organizing
Article
A survey of software professionals worldwide suggests the past, present, and future challenges of software engineering. This article is part of a theme issue on software engineering’s 50th anniversary.