PreprintPDF Available

Towards Just-Enough Documentation for Agile Effort Estimation: What Information Should Be Documented?

Authors:

Abstract and Figures

Effort estimation is an integral part of activities planning in Agile iterative development. An Agile team estimates the effort of a task based on the available information which is usually conveyed through documentation. However, as documentation has a lower priority in Agile, little is known about how documentation effort can be optimized while achieving accurate estimation. Hence, to help practitioners achieve just-enough documentation for effort estimation, we investigated the different types of documented information that practitioners considered useful for effort estimation. We conducted a survey study with 121 Agile practitioners across 25 countries. Our survey results showed that (1) despite the lower priority of documentation in Agile practices, 98% of the respondents considered documented information moderately to extremely important when estimating effort, (2) 73% of them reported that they would re-estimate a task when the documented information was changed, and (3) functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the most useful types of documented information for effort estimation. Nevertheless, many respondents reported that these useful types of documented information were occasionally changing or missing. Based on our study results, we provide recommendations for agile practitioners on how effort estimation can be improved by focusing on just-enough documentation.
Content may be subject to copyright.
Towards Just-Enough Documentation for Agile Effort
Estimation: What Information Should Be Documented?
Jirat Pasuksmit, Patanamon Thongtanunam, Shanika Karunasekera
The University of Melbourne, Australia
jpasuksmit@student.unimelb.edu.au, patanamon.t@unimelb.edu.au, karus@unimelb.edu.au
Abstract—Effort estimation is an integral part of activities
planning in Agile iterative development. An Agile team estimates
the effort of a task based on the available information which is
usually conveyed through documentation. However, as documen-
tation has a lower priority in Agile, little is known about how
documentation effort can be optimized while achieving accurate
estimation. Hence, to help practitioners achieve just-enough
documentation for effort estimation, we investigated the different
types of documented information that practitioners considered
useful for effort estimation. We conducted a survey study with
121 Agile practitioners across 25 countries. Our survey results
showed that (1) despite the lower priority of documentation in
Agile practices, 98% of the respondents considered documented
information moderately to extremely important when estimating
effort, (2) 73% of them reported that they would re-estimate
a task when the documented information was changed, and
(3) functional requirements, user stories, definition of done, UI
wireframes, acceptance criteria, and task dependencies were
ranked as the most useful types of documented information for
effort estimation. Nevertheless, many respondents reported that
these useful types of documented information were occasionally
changing or missing. Based on our study results, we provide
recommendations for agile practitioners on how effort estimation
can be improved by focusing on just-enough documentation.
I. INTRODUCTION
Effort estimation is an important practice for planning
activities in Agile iterative development. It helps a software
development team to approximates the size of a task, enabling
an optimal workload allocation for a short iteration of Agile [8,
17]. The estimated effort of the tasks in previous iterations
also can be used to calculate the delivery capability, which
helps the team to better plan the next iteration [7, 40]. Hence,
accurate effort estimation benefits the team in creating a
reliable iteration plan to deliver the product increment [8, 40].
The effort of a task is typically estimated based on the
available information [40, 48]. One of the common approaches
to manage and convey information is to use documenta-
tion [3, 19, 33]. For example, backlog items are used to
document the information related to a feature (e.g., user
stories, acceptance criteria) [29, 42]. A lower-level backlog
item (like development tasks) may also include the detail
information related to the implementation of the feature (e.g.,
a description of a function to be developed) [34].
On the other hand, in Agile practices, documentation has
a lower priority than working software [16]. Based on this
concept, it is possible that the information that is recorded
in documentation (documented information, henceforth) may
not be well maintained, resulting in inadequate information for
the team to understand development tasks [29]. Several studies
also discovered that documented information in Agile projects
can be incorrect, incomplete, outdated, and changing [2, 29,
52]. Yet, little is known about how documentation effort can
be minimized while achieving accurate estimation.
In this paper, we aim to understand how just-enough doc-
umentation can be provided (i.e., what information should
be documented). Hence, practitioners can save documentation
effort while achieving accurate effort estimation. Therefore,
we conducted a survey study with Agile practitioners to
investigate the importance and the types of documented in-
formation that are useful for effort estimation in the current
practices. Our findings will be beneficial to Agile practitioners,
tools developers, and researchers. Agile practitioners can pay
attention to the quality of useful documented information
to achieve more accurate estimation. Tool developers and
researchers can focus their attention on developing tools
and techniques to help practitioners maintain documentation
for effort estimation. Based on the responses of 121 Agile
practitioners, we answered the following research questions:
(RQ1) To what degree is documented information impor-
tant for effort estimation?
Motivation: Accurate effort estimation requires an ad-
equate understanding of a task [28, 44, 47, 48]. While
documentation can convey the information to understand
the task [3, 19, 24], it has a lower priority than deliv-
ering working software in the Agile concept [16]. Hence,
we investigated how Agile practitioners perceive the impor-
tance of the documented information for effort estimation.
Results: 97% of the respondents used documentation to
assist effort estimation, and 98% of them reported that doc-
umented information was moderately to extremely important
when estimating effort. The majority of them attempted to
keep documented information and documented effort up to
date most of the time. Although documentation has a low
priority in Agile, our result shows that the documented in-
formation is considered important for effort estimation.
(RQ2) Is a change of documented information more likely
to influence re-estimation than the information from other
sources?
Motivation: Prior studies reported that effort may be re-
estimated when the information is changed [5, 23]. How-
ever, in Agile, the information can be conveyed through
various channels, e.g., documentation, in-person, discussion
threads [3, 19, 46]. Hence, we set out to investigate how often
arXiv:2107.02420v1 [cs.SE] 6 Jul 2021
a task was re-estimated when the documented information was
changed in comparison to the information from other sources.
Results: The reported frequency of re-estimation when
documented information was changed is statistically higher
than the reported frequency of re-estimation when information
from other sources was changed. In general, the respondents
noted that re-estimation will be performed if the information
change is significant or is from a reliable source like documen-
tation. This result shows that when documented information
is changed, a task is often re-estimated.
(RQ3) What types of documented information are useful
for effort estimation?
Motivation: Comprehensive documentation can be consid-
ered as an overhead in Agile [16, 24]. On the other hand,
inadequate documentation may lead to a misunderstanding
of a task [29, 44]. Hence, just-enough and just-in-time doc-
umentation would be suitable for Agile practices [21]. Yet,
little is known what types of information are considered
useful for effort estimation and should be documented.
Results: Functional requirements (i.e., the task information
about the functionality of the system), user stories, defini-
tion of done, UI wireframes, acceptance criteria, and task
dependencies were ranked as the six most useful documented
information for effort estimation. Yet, many respondents re-
ported that these useful types of documented information were
occasionally (or more) changing or missing during (or after)
effort estimation. Our results suggest that the information that
is useful for effort estimation (especially for the six useful
types of documented information) should be documented and
reviewed prior to finalizing effort estimates.
Our findings highlight that documented information is im-
portant for effort estimation and a change of documented
information can lead to re-estimation. Aligning with the Agile
practices, our recommendations can be done in a just-in-time
manner when the information is needed for effort estimation.
In other words, attention should be paid to ensure that the
useful documented information for effort estimation has suffi-
cient detail prior to finalizing effort estimates to avoid making
unnecessary assumptions. In addition, to help practitioners
achieve accurate effort estimation, our work highlights the
need for tools and techniques to timely assess the quality of
documented information.
Paper organization. Section II defines the terms used in
this paper. Section III presents our research methodology.
Section IV presents the study results. Section V provides a
broader implication of the results. Section VI discusses related
work. Section VII discloses the threats to validity. Finally,
Section VIII draws the conclusion.
II. TE RM S & DE FIN IT IO NS
This section defines the terms that are used in this paper.
Information refers to a description that provides an under-
standing of a software product under development, e.g.,
software features [1, 11].
Documented information refers to the information related
to a software product that is captured and recorded in
documentation (i.e., a collection of documents) [1, 15].
The documented information can be in various forms.
For example, the information related to a feature (e.g.,
user stories, acceptance criteria) can be documented in
a backlog item [29, 42]. A description of a function to
be developed can be documented in a lower-level backlog
item like a development task [34]. The documentation we
consider can be recorded on a physical paper or in a task
management system (e.g., JIRA) [19, 23, 24, 31, 41].
We do not refer the documentation to textual descriptions
related to source code (e.g., code comments) [12, 52], soft-
ware development process [11], user manuals, and informal
communication (e.g., discussion thread, chat) [19, 46].
III. RESEARCH MET HO D
In this section, we describe our survey design, survey
questions, recruitment process, and data analysis.
A. Survey Design
The survey was designed based on the recommended sur-
vey design practice in software engineering to avoid known
problems [18, 32]. Our survey consisted of seven sections
which included 11 questions to address our research questions
(see Table I) and 11 questions to collect demographics about
the respondents. Participation was voluntary and the estimated
time to complete a survey was 10-15 minutes. The survey
responses were recorded anonymously. The respondent could
drop out anytime, and the incomplete response was automati-
cally deleted. The survey was created using an online survey
platform named Qualtrics. To ensure the overall quality of
responses, we used the features of Qualtric to prevent bots
and multiple/duplicate responses from a single respondent.1
Pilot Tests: Comprehensibility of the survey questions was
our main construct concern when designing the survey [30].
As suggested by Ghazi et al. [18], we conducted two pilot
tests with five software engineers from our connections. These
five participants of the pilot tests were software engineers
who used an Agile method and used effort estimation (i.e.,
the target population of our survey). They had 5 to 12 years
(7.6 years on average) of experience. After the pilot tests,
the comprehensibility of the survey was improved by adding
details, rewording, adding and merging options, and providing
definitions. Finally, our survey and related documents were
reviewed and approved by the Human Ethics Advisory Group
of the authors’ institution.
B. Survey Questions
Table I provides an overview of our survey questions
that address our three RQs.2The questions with an asterisk
(*) included an optional open-ended question to collect an
elaboration for an answer of the corresponding question. We
now briefly describe each section of our survey.
1http://qualtrics.com/support/survey-platform/survey-module/
survey-checker/response-quality
2The full survey and the replication package is available at http://figshare.
com/s/74c2509bdb23de953da3
TABLE I: Survey questions (excluding demographics questions2).
Item Question
S2: Preliminary questions
PQ1* Does your team perform effort estimation in any stage of the software development lifecycle? (yes/no)
PQ2 Have you ever participated in effort estimation of your team? (yes/no)
S4: Importance of documented information (RQ1)
Q1 Does your team use any documentation when performing effort estimation? (yes/no)
Q1.1 What is the source of information (other than the information in the documentation) that your team uses in effort estimation? (Open-ended)
Q2In general, how important is the task-related documented information when you estimate the task?
Q3How often do you update the documentation when new information is available?
Q4How often do you update the estimated effort in the documentation when the task is re-estimated?
S5: Re-estimation (RQ2)
Q5*Before you work on a task, do you re-estimate the task if the documented information that is relevant to the task is changed or updated?
Q6*Before you work on a task, do you re-estimate the task if the information from other sources than documentation is changed or updated?
Q7*While you are working on a task, do you re-estimate the task if the documented information that is relevant to the task is changed or updated?
Q8*While you are working on a task, do you re-estimate the task if the information from other sources than documentation is changed or updated?
S6: Useful information and its quality (RQ3)
Q9 On average, which of the following documented information do you find most useful in effort estimation? (Ranking)
Q10 For each of the selected types of documented information, please select the following quality issues that you occasionally (or more) encounter
during (or after) effort estimation for an average task that you estimate.
Multiple selections of quality issues: @A missing, @A incorrect, @A outdated, @A changing, or generally no issue
* Questions with asterisk included an optional open-ended answer for further elaboration.
A Likert scale of importance: Extremely important, Very important, Moderately important, Slightly important, Not at all important
A Likert scale of frequency: Always (˜100%), Most of the time (˜75%), About half the time (˜50%), Sometimes (˜25%), Never (˜0%)
S1: Informed consent. This section informed the respon-
dent about the researchers’ contact, the purpose of this study,
how a response will be recorded, and how long will it take to
complete the survey. Before proceeding to the next section, the
respondent was asked to consent to participate in the study.
S2: Preliminary questions. We asked two preliminary
questions to check whether the respondent had been involved
in effort estimation. To be specific, we asked whether the team
of the respondent performed effort estimation at any stage
of the software development process (PQ1) and whether the
respondent had ever participated in effort estimation of the
team (PQ2). Given that a respondent who answered ‘no’ to
one of these two questions is unlikely to have background
knowledge about effort estimation of the team, the rest of the
survey questions were skipped and the respondent was directed
to the Demographics section.
S3: Terms and Definitions. This section provided the terms
and definitions that we used in this survey (see section II).
This is to ensure that the respondent will have the correct
understanding of the survey questions. We also informed
the respondent that the survey questions focused on effort
estimation at the lowest granularity of a work item that
s/he estimated. Since the respondent may be involved in
multiple projects, we asked the respondent to answer the
survey questions based on the project in which s/he was most
involved in. Finally, we asked the respondent to confirm that
s/he understood the terms and definitions.
S4: Importance of documented information. The survey
questions in this section aimed to address our RQ1. We
asked whether the respondent used any documentation when
performing effort estimation (Q1). If the respondent answered
no’, we asked what is the other sources of information that
the respondent used (Q1.1). Then, the respondent was directed
to the Demographics section (S7). If the respondent answered
Time
Task
3 points
Time T3
Information
change*#2
Time T2
Information
change*#1
work had
started
/01-*23"%&'%- )&5*,"(')*& /91-*23"%&'%- )&5*,"(')*&
/71)&5*,"(')*& 5,*" *'8%,
+*3,2%+ '8(& -*23"%&'(')*&
/:1)&5*,"(')*& 5,*" *'8%,
+*3,2%+ '8(& -*23"%&'(')*&
!"#$ !'
2'34 .3()&'(.5
Fig. 1: Illustrative Scenario for Re-estimation section (S5).
yes’ to Q1, we asked the respondent to rate the importance
of documented information for effort estimation (Q2) using a
5-points Likert scale of importance (see Table I). Since the
respondent who answered ‘yes’ to Q1 used documentation
for effort estimation, we further asked the respondent about
documentation maintenance. Specifically, we asked how often
the respondent updated the document when new information
is available (Q3) and how often s/he updated the documented
effort when a task is re-estimated (Q4). To answer Q3 and Q4,
we provided a 5-points Likert scale of frequency (see Table I).
S5: Re-estimation. The survey questions in this section
aimed to address our RQ2. We asked how often the respon-
dent re-estimates a task when the documented information
was changed in comparison to the information from other
sources [3, 19, 46]. We also considered whether the infor-
mation was changed before or after the work had started
as this may be a factor that influences a decision of re-
estimation [23, 37, 50].
Hence, we formulated four survey questions (Q5-Q8), where
Q5 and Q7 focused on the changes of documented informa-
tion, while Q6 and Q8 focused on the changes of information
from other sources. Furthermore, Q5-6 focused on the re-
estimation when the information was changed before the work
had started, while Q7-8 focused on the re-estimation when the
information was changed after the work had started. To ease
the respondent’s understanding, we provided an illustrative
scenario (Figure 1) in our survey. Additionally, we included
open-ended questions for the respondents to elaborate their
answers to Q5-8.
S6: Useful information and its quality. The survey ques-
tions in this section aimed to address our RQ3. We asked the
respondent to select five types of documented information and
rank them based on the usefulness for effort estimation at the
lowest granularity that s/he estimated (Q9). We provided nine
common types of documented information as the options. We
also provided open-ended options where the respondent can
fill out other types of documented information that were not
on the list. We derived the nine common types of documented
information from the literature which are as follows:
User Stories [4, 26, 40, 42],
Definition of Done [4, 26, 40],
– Acceptance criteria (including acceptance tests and test
cases) [9, 19, 26, 40, 42],
Functional requirements [34],
Non-functional requirements [4, 34],
Task dependencies [19, 42, 45],
Code examples [2, 9, 53],
UI wireframes (or pre-designed UI) [21, 42], and
API reference guides [2].
In our survey, we provided a definition for each of these types
to ensure that the respondent had the same understanding.2
Note that, since we focused on Agile practices, functional
requirements are referred to the task information about the
functionality of the system [34] as opposed to software
requirement specification (SRS) that is prepared upfront in
non-Agile software development practices. Both user stories
and functional requirements capture the information about
the system behaviors. The key difference between the two
types is that a user stories is a lightweight expression that
focuses on the desired business value. In contrast, functional
requirements focus on the technical aspect of a functionality
to be developed.
For each selected type of documented information, we
further asked the respondent to select the quality issues that
s/he occasionally (or more) encountered during (or after) effort
estimation (Q10). In this work, we study four common quality
issues of documented information: (1) changing, (2) missing,
(3) outdated, and (4) wrong [2, 29, 52]. In our survey, we also
provided an option of ‘generally no issue’ if the respondent
did not encounter any of these four issues.
S7: Demographics. The survey questions in this section
aimed to collect the general background and effort estimation
practice of the respondent.2We asked the respondent about
the software development methodology, experience in the IT
industry (years), experience in effort estimation (years), the
duration of the most involved project (years), office location
(country), working team size, and team location (co-located,
distributed-inshore, or distributed-offshore).
We asked what the effort estimation technique(s) and unit(s)
that the team of the respondent used, and when the team
performed effort estimation (e.g., iteration planning, backlog
grooming, project planning). As the respondent’s team might
use different effort estimation techniques, the questions in this
section were multiple selections. We also asked an optional
open-ended question about the benefit of effort estimation.
C. Recruitment Process
To recruit software engineers to participate in our survey,
we used purposive sampling and snowballing methods.
Purposive Sampling. We used purposive sampling as our
main recruitment approach. This method allowed us to identify
software engineers who are likely to use an Agile method
and effort estimation [14]. In particular, we targeted software
engineers who (1) work (or have worked) in an international
software company or a top local software company; and (2)
have at least one year of experience in software development.
We mainly considered the international and top local compa-
nies as they are established companies which are more likely
to use effort estimates in feature prioritization, while start-up
companies tend to use business values as the prioritization
target [33]. To find the top local companies, we searched for
a list of top companies for each country.
We recruited software engineers via LinkedIn (i.e., a profes-
sional networking platform) as it allows us to access the target
group of interest [18, 38]. We identified the target group of
software engineers using publicly available LinkedIn profiles.
Then, we sent a survey invitation to the targeted software
engineers via the LinkedIn direct message.
To ensure the diversity of respondents, we controlled the
number of invitations per software company. More specifically,
for a small company (<200 employees), we randomly selected
10-20 targeted software engineers. For a large company (>200
employees), we randomly selected 20-50 targeted software
engineers. Since LinkedIn limits the number of invitations,3
we sent the survey invitation to about 20-30 software engineers
per day for 60 consecutive days. Based on this process,
we sent a survey invitation to 1,500 software engineers via
LinkedIn. Broadly speaking, 36%, 20%, 33%, and 10% of
direct invitations via LinkedIn were sent to software engineers
in the Americas, Europe, Asia, and Oceania, respectively.
Snowballing. To further reach out to the target population,
we performed a snowballing approach [10]. Specifically, if
the respondent informed us that s/he completed the survey,
we asked them to spread the survey link to other software
engineers who use Agile methods and effort estimation. In
total, we asked 16 respondents to spread the link.
D. Data Preprocessing
In this study, we focused on effort estimation in Agile
projects. However, when we sent the survey invitation, we
could not determine whether a software engineer has worked
in an Agile project or his/her team used effort estimation.
3https://www.linkedin.com/help/linkedin/topics/6096/6097/4800
Therefore, before analyzing the survey results, we selected
the responses of software engineers of interest based on their
answers to the preliminary and demographic questions. To be
specific, we selected the responses using the following criteria.
Criterion 1 - Use an Agile method: Since we focused on
Agile practices, we excluded the respondents who used
non-Agile methods, i.e., Waterfall model, Incremental
model, and not sure or N/A.
Criterion 2 - Have Effort Estimation Experience: To ensure
that the respondents had experience in effort estimation,
we excluded the respondents who answered ‘no’ to the
preliminary questions about the involvement in effort
estimation (i.e., PQ1 and/or PQ2).
E. Card Sorting
Similar to prior work [12, 49], we applied card sorting
to analyze the open-ended answers of Q5-8. As suggested
by Spencer [43], we first conducted an open card sorting
procedure to extract themes of the answers (cards). For each
open-ended question, the first two authors of this paper
independently grouped the answers based on the thematic
similarities and defined a theme for each group. Then, the
first two authors discussed to develop the agreed-upon themes
of the answers. To validate the agreed-upon themes, the
third author of this paper performed closed-card sorting, i.e.,
sorting the answers into the themes that were developed by
the first two authors. Finally, we measured the inter-rater
reliability using Cohen’s Kappa coefficient [35]. The overall
Kappa values ranged from 0.65-0.80, indicating substantial
agreements between the sorters.
IV. RES ULT S
In this section, we present the results of our survey study. In
particular, we first provide an overview of the survey results.
We then analyze the survey results to answer each of our RQs.
A. Survey Results Overview
We received 148 responses in total from the 1,500 direct
LinkedIn invitations and snowballing (a response rate of
10%).4The median time taken by the respondents to complete
the survey was 14 minutes. After the data pre-processing (see
Section III-D), we retained 121 respondents of interest, i.e.,
software engineers who used an Agile method and had been
involved in effort estimation.
Respondent Demographics. Figure 2 presents the distri-
butions of respondents across different demographic groups.
Most of the respondents used Scrum (78%) or Kanban (18%)
as their development methodology. A majority of them (72%)
had more than five years of experience in the IT industry. The
software project on which a respondent reported typically had
a duration of three years or under (83%). The typical team size
was 6-10 people (50%) and 11-30 people (27%). About 52%
of the respondents worked in a co-located team, while others
4Since we used a snowballing approach and the responses are anonymized,
we do not know the number of snowballed responses. Hence, the reported
response rate may be varied.
Development
methodology
Experience
in IT
Project
duration
Team size
Team
location
Scrum
Kanban
Others
<1 yr
1−5 yrs
5−10 yrs
>10 yrs
<6 mo
6 mo − 1 yr
1−3 yrs
3−5 yrs
>5 yrs
1−5 ppl
6−10 ppl
11−30 ppl
>30 ppl
Co−located
Distributed−inshore
Distributed−offshore
0%
10%
20%
30%
40%
50%
0%
10%
20%
30%
40%
50%
0%
10%
20%
30%
0%
10%
20%
30%
40%
0%
20%
40%
60%
80%
Number of respondents (%)
Experience in
estimation
Estimation
methodology
Estimation
unit
When to
estimate
<1 yr
1−5 yrs
5−10 yrs
>10 yrs
Expert's judgement
Planning poker
Use Case Points
Function Points
Others
Story points
Person−days
Person−hours
T−shirt sizes
Others
Iteration planning
Backlog Grooming
Release planning
Project bidding
0%
20%
40%
60%
80%
0%
20%
40%
60%
0%
10%
20%
30%
40%
50%
0%
20%
40%
Number of respondents (%)
Fig. 2: The distributions of respondents across different groups
of demographics and effort estimation practices. A respondent
can select multiple estimation methodologies, estimation units,
and time points when estimating effort.
worked in a distributed team. Our respondents were from 25
countries, e.g., Singapore (18), the United States of America
(14), Thailand (13), Australia (12), India (11), others (53).
Effort Estimation Practices. Figure 2 also shows that
our respondents had experience in diverse effort estimation
practices. Figure 2 shows the effort estimation practices of
our respondents. The majority of the respondents had 1-5
years (53%) or 5-10 years (27%) of experience in estimat-
ing effort. They performed effort estimation during iteration
planning (83%) and/or backlog grooming (37%). The common
estimation methods were Expert Judgement (50%) and/or
Planning Poker (48%). The common effort unit(s) were story
points (61%) and/or person-days (43%). Based on the open-
ended questions, the respondents reported that they used effort
estimation to 1) know the size and complexity of development
tasks, 2) achieve better resource allocation, 3) accurately
estimate the delivery capability and the time to deliver, and
4) build trust with stakeholders based on an accurate and
transparent workflow.
B. (RQ1) To what degree is documented information important
for effort estimation?
A vast majority of the respondents (117
121 ; 97%) reported that
they used documentation when performing effort estimation
(Q1). Figure 3 shows that most of the respondents (115
117 ;
98%) who used documentation when estimating effort rated
the importance of documented information as moderately to
extremely important for effort estimation (Q2). On the other
hand, only two respondents rated it as slightly importance and
229
(Q2) Importance of
documented information
for effort estimation 0 30 60 90 117
# Participants
Extremely
important Very
important Moderately
important Slightly
important
35 51
Fig. 3: Importance of documented information when perform-
ing effort estimation.
31415
1314
(Q4) Documented effort
updating frequency
(Q3) Documentation
updating frequency
0 30 60 90 117
# Participants
Always
(~100%) Most of the
time (~75%) About half the
time (~50%) Sometimes
(~25%) Never
(~0%)
2362
5040
Fig. 4: The frequencies that the respondents updated the doc-
umentation when new information was available and updated
the documented effort after re-estimation.
none rated it as not at all important. This result indicates
that practitioners considered documented information to be
moderately to extremely important for effort estimation.
Based on Q1.1, the respondents who did not use documenta-
tion when estimating effort (4 out of 121 respondents) reported
that, instead of using documentation, they relied on 1) team
velocity in the past iterations, 2) verbal communication, or 3)
code-based statistics from code indexer (e.g., Google Kythe).
The survey results of Q3 and Q4 also show that the respon-
dents who used documentation when estimating effort tend to
keep both documented information and documented efforts up
to date. Figure 4 shows that 34% ( 40
117 ) of the respondents
always updated documentation when new information was
available (Q3); and 42% ( 50
117 ) of them updated the document
most of the time. Similarly, 53% ( 62
117 ) of the respondents
always updated the documented effort after re-estimation (Q4).
These results suggest that the respondents attempted to keep
the documentation up to date.
Prior studies pointed out that the importance of docu-
mented information may be varied based on the team set-
tings [19, 22, 25, 29]. Hence, we further checked whether the
documented information has a different level of importance
given a different team setting. We considered three aspects
of team settings, i.e., the team location, size, and project
duration. For each aspect, we used the Kruskal-Wallis rank
sum test to compare the distributions of the importance level
of documented information across the groups of respondents.
For example, we examined whether the distributions of the
importance level of documented information are statistically
different among the three groups of team location (i.e.,
co-located, distributed-inshore, and distributed-offshore). The
Kruskal-Wallis rank sum test is a non-parametric test that is
robust to non-homogenous and a small sample. Similar to
prior work [20, 27, 51], we assigned the numerical values
to the Likert scale, i.e., 5 (for Extremely important) to 1
(for Not at all important) when performing a statistical test.
2142
1628
1048
428
0 30 60 90 117
# Participants
Do you re−estimate the task
Always
(~100%) Most of the
time (~75%) About half the
time (~50%) Sometimes
(~25%) Never
(~0%)
2217
2526
2815
3824
15
22
16
23
before the work had started
(Q5) when the documented
information was changed
(Q6) when the information from
other sources was changed
before the work had started
(Q7) when the documented
information was changed
after the work had started
(Q8) when the information from
other sources was changed
after the work had started
Fig. 5: The frequencies that the respondents would re-estimate
if the documented information (or the information from other
sources) was changed before (or after) the work had started.
For the three aspects of team settings, all the tests yielded
not statistically significant (p0.05), indicating that the
distributions of importance levels are not statistically different
for the respondents from different team settings.
Findings: 97% of the respondents used documentation to as-
sist effort estimation, and 98% of these respondents reported
that documented information is moderately to extremely
important when estimating effort. Furthermore, a majority of
the respondents attempted to keep documented information
and documented effort up to date most of the time.
C. (RQ2) Is a change of documented information more likely
to influence re-estimation than the information from other
sources?
We found that the respondents tend to re-estimate a task
when the documented information was changed more often
than when the information from other sources was changed.
Figure 5 shows that 73% ( 85
117 ) of the respondents reported
that they would re-estimate at least 50% of the time when the
documented information was changed before the work had
started (Q5).5On the other hand, 50% of the respondents
reported that they would re-estimate at least 50% of the
time when the information from other sources was changed
before the work had started (Q6). Similarly, for the after-start-
working case, 62% of the respondents reported that they would
re-estimate at least 50% of the time when the documented
information was changed (Q7); while 46% of the respondents
reported that they would re-estimate at least 50% of the time
when the information from other sources was changed (Q8).
To statistically confirm our finding, we used the one-sided
Wilcoxon signed-rank test to examine whether a respondent
reported the frequency in Q5 (i.e., when the documented
information is changed) higher than the frequency in Q6 (i.e.,
when the information from other sources was changed) for
the before-start-working case. Similarly, we performed the
test to compare the difference of frequencies in Q7 and Q8
5Note that we excluded the four respondents who did not use any documents
in effort estimation since we focus on the information in documents.
TABLE II: Themes derived from the answers of the open-
ended questions of Q5-8.
Theme #
Reasons to re-estimate
T1) Only re-estimate if the change is significant or from a
reliable source like documentation.
52
T2) Always re-estimate to correct the accuracy, velocity, or
development plan.
26
Reasons to not re-estimate
T3) The team has a policy (or norm) to not re-estimate after
the work had started.
12
T4) Absorb the change (attempt to complete within the original
time frame) and/or roll the work over to the next iteration.
9
T5) Over-estimate to cover future change. 2
Total responses 101
for the after-start-working case. We opted to use the one-
sided Wilcoxon signed-rank test as it is a non-parametric
test which is robust to non-homogenous and a small sample.
Before performing the tests, we converted the Likert scale
of frequency into numerical values (e.g., ‘Always (˜100%)’
to 100). The tests showed that the reported frequency in Q5
is statistically higher than the reported frequency in Q6; and
the reported frequency in Q7 is statistically higher than the
reported frequency in Q8 (p < 0.05). These results confirm
that the change of the documented information is more likely
to influence re-estimation than the change of the information
from other sources.
For the optional open-ended questions of Q5-8, we received
101 responses. We sorted the responses into five themes of rea-
sons based on the card sorting procedure (see Section III-E).
Table II lists the themes which are grouped into (1) reasons
to re-estimate and (2) reasons to not re-estimate.
In general, 52 responses indicated that the decision of re-
estimation was based on the significance of the information
change (T1): "If the change is not significant, and does not
change the scope, the same points may remain. However, if due
to the change, the original estimate is not accurate, new point
will be estimated. [...]". In addition to the significance of the
change, some responses also emphasized that the re-estimation
decision also depended on the source of information: "If it’s
not documented, it likely won’t be important. This is the
motivation for everything to be documented." Another 26
responses pointed out that re-estimation was required in order
to keep the delivery capability and development plan realistic
(T2): "Any re-estimation is always needed to have a correct
velocity sprint productivity [...]."
The common reason for not re-estimating was that they
had a policy (or norm) to not re-estimate after the work
had started (T3): "At my team, estimation is frozen since
task is scheduled for current sprint and sprint is ongoing."
Another nine responses indicated that they tried to absorb
the change and/or rolled the remaining work over to the
next iteration (T4): "Mostly, our team just try to done the
task even it’s exceed the estimation. However, if the sprint
ended and this task still exist and need to carry over to next
TABLE III: The number of respondents who ranked the type
of documented information as one of the five most useful
information; and the number of respondents who indicated
that the type of documented information occasionally (or more
frequent) had a quality issue during (or after) effort estimation.
Type of documented information Ranked as Have Quality
Useful Issue(s)
Functional requirements 91 67 (74%)
User stories 74 41 (55%)
Definition of Done 76 41 (55%)
UI wireframes 61 46 (75%)
Acceptance criteria 79 65 (82%)
Task dependencies 61 51 (84%)
Non-functional requirements 40 36 (90%)
API reference guides 33 24 (73%)
Code examples 6 3 (50%)
1212
5810
13108
111818
211625
61515
141618
8710
795
0 25 50 75 91
# participants
Importance rank 12345
2347
1732
134
178
1117
122
2
161
2
Functional requirements
Nonfunctional requirements
Definition of done
User stories
UI wireframes
Acceptance criteria
Task dependencies
API references guides
Code examples
Fig. 6: The distributions of importance rank of documented
information, where the rank of 1 indicates that that type of
documented information is the most useful.
sprint, we will re-estimate the remaining task [...]." Other two
responses noted that they intentionally over-estimated to cover
the future change (T5): "Because I usually expect delays based
on missing or incorrect documentation, I overestimate my tasks
to take into account these issues. This allows me not to have
to re-estimate."
Findings: The reported frequency of re-estimation when
the documented information was changed is statistically
higher than the reported frequency of re-estimation when
the information from other sources was changed. In general,
the respondents noted that re-estimation will be performed
if the information change is significant or from a reliable
source like documentation.
D. (RQ3) What types of documented information are useful
for effort estimation?
Table III shows that functional requirements, user stories,
definition of done, UI wireframes, acceptance criteria, and
task dependencies were frequently rated as useful documented
information for effort estimation (Q9).5Moreover, Figure 6
shows that functional requirements (i.e., the description of a
function to be developed [34]) were ranked as the most useful
type of documented information for effort estimation where
18
17
54
16
14
33
13
15
24 10
26
20 13
41
35
12
26
25
0
20
40
Functional
requirements Pre−
designed UI
User
stories Definition
of done Acceptance
criteria Task
dependencies
# respondents
Quality issues Changing Missing Outdated Wrong
15 13
15
5
13
9
Fig. 7: The number of respondents who reported that the type
of documented information occasionally (or more) had quality
issues during (or after) effort estimation. Note that multiple
quality issues could be selected.
52% ( 47
91 ) of the respondents ranked this type as the top one.
The user stories was also ranked as the second most useful
type where 43% ( 32
74 ) of the respondents ranked this type
as the top one. These results suggest that the documented
information that provides an understanding of the software
system’s behavior is important when estimating effort.
Nine respondents also provided additional types of infor-
mation that are useful for effort estimation in the open-ended
options. Most of the reported information is related to the
team context, e.g., the knowledge, expertise, or workload of
developers. In addition, one respondent pointed out that steps-
to-reproduce and observed behaviors are useful documented
information when estimating the effort of a bug-fixing task.
Table III shows that many respondents reported that these
useful types of documented information occasionally (or more)
had at least one kind of quality issue during (or after) effort
estimation. For example, 74% (67
91 ) of the respondents who
ranked functional requirements as useful indicated that this
type of documented information had quality issues. Figure 7
shows that 81% (54
67 ) of the respondents indicated that the func-
tional requirements in the documentation were occasionally
changing. Furthermore, Table III shows that other useful types
documented information, i.e., user stories, definition of done,
UI wireframes, acceptance criteria, and task dependencies
were reported as having quality issues by many respondents
(55%-84%; see Table III). Figure 7 also shows that user
stories and UI wireframes were occasionally changing, while
definition of done, acceptance criteria, and task dependencies
were occasionally missing. These results suggest that despite
the usefulness of these types of documented information for
effort estimation, they were occasionally changing or missing.
Findings: Functional requirements, user stories, definition
of done, UI wireframes, acceptance criteria, and task de-
pendencies were ranked as the six most useful documented
information for effort estimation. Yet, many respondents
reported that these useful types of documented information
were occasionally (or more) changing or missing during (or
after) effort estimation.
V. DISCUSSION
We now discuss a broader implication and provide a rec-
ommendation based on our results.
A. Implications
Despite the lower priority of documentation in Agile, the
documented information is used and considered important
for effort estimation. More specifically, our RQ1 shows
that 117 out of the 121 respondents used documentation for
effort estimation. Moreover, regardless of the team settings
(i.e., the team location, size, or project duration), 98% of the
respondents (115
117 ) considered documented information as mod-
erately to extremely important for effort estimation (see Figure
3). From the open-ended question of Q6, some respondents
emphasized the importance of documented information, e.g.,
"I prioritize the documentation over information from other
sources. The information outside documentation is needed to
be filtered if it is important or it is just good to have/know.
[...]." Furthermore, the responses of Q3-4 show that a majority
of the respondents tend to maintain documentation as they
reported that they updated documentation of both information
and estimated effort most of the time (see Figure 4). These
results highlight the importance of documented information
for effort estimation for Agile practitioners.
The changing documented information can have an
impact on estimation accuracy. Figure 5 shows that 73%
(85
117 ) of the respondents re-estimated the task at least 50%
of the time when the documented information was changed,
which is more often than the change of the information from
other sources. The respondents also noted in the open-ended
questions of Q5-8 that they would re-estimate if the change
was significant or was from a reliable source like docu-
mentation. Our statistical tests also confirm that the reported
frequency of re-estimation when documented information was
changed is statistically higher than that of the information from
other sources. These results suggest that the practitioners are
more likely to re-estimate the task if documented information
was changed in comparison to a change of the information
from other sources.
B. Recommendations
Our survey results highlight the importance of documented
information for effort estimation. The results also show that
the quality of documented information can have an impact
on estimation. Hence, we provide two recommendations on
how documentation should be maintained while achieving
accurate estimation. Aligning with the Agile practices, our
recommendations can be done in the just-enough (i.e., focusing
on the useful types of documented information) and just-in-
time (e.g., before iteration planning) manner.
Prior to finalizing effort estimates (e.g., at iteration
planning), attention should be paid to ensure that the
information that is useful for effort estimation is docu-
mented to avoid making unnecessary assumptions. Our
RQ3 shows the six types of documented information that were
ranked as the most useful for effort estimation (see Figure 6).
However, several respondents reported that these useful types
of documented information had quality issues, especially for
the changing issue (Table III). The responses of the open-
ended question of Q5 also highlight that a change of such
useful information often caused re-estimation: "Depends on
changes. [...] the change of user stories/functional require-
ments changes the original estimation often." Since the effort
estimates are used in iteration planning, such uncertainty may
cause the plan to become unreliable [8, 44]. Hence, to facilitate
effort estimation and improve its accuracy, these useful types
of information should be documented and reviewed. One
possible actionable approach is to confirm the information
with stakeholders. This is also aligned with one respondent
who noted that "[...] It’s actually good to communicate with
all stakeholders earlier rather than later about the assumption
[...]." While it is not required to document all the information
at the very early stages of development, our recommendation
can be done in the timely manner. In other words, before
iteration planning, practitioners can check whether these useful
types of information are documented, correct, and up to date
to avoid making unnecessary assumptions.
Future research should focus on developing tools and
techniques to assess the quality of documented information
for effort estimation. While it is recommended to maintain
the documentation for effort estimation, it may still be tedious
for practitioners to assess the quality of information of all
tasks. Techniques to detect missing or changing information
should help practitioners to timely manage the useful docu-
mented information, which in turn will lead to more accurate
effort estimation. Yet, less focus has been given to developing
techniques that detect missing or changing information that is
useful for effort estimation. To the best of our knowledge,
existing work focused on developing techniques to detect
missing information for software development, e.g., the in-
formation for bug-fixing tasks [6], feature requests in issue
tracking systems [36], and user stories from developer-client
conversation transcripts [39]. Therefore, future research should
develop techniques that help practitioners be aware of the
quality issues of documented information that could impact
the accuracy of effort estimation.
VI. RE LATE D WORK
In this section, we discuss related work with respect to effort
estimation and documentation.
Information and Estimation. Several studies investigated
the impact of inadequate information on effort estimation.
Jørgensen et al. [28] found that unclear task information
was one of the reasons for inaccurate estimation. Svensson
et al. [44] pointed out that inadequate information could
lead the team to misunderstood the task, which may lead
to inaccurate estimation. Usman et al. [47, 48] reported
that Agile practitioners perceived that a lack of details in
requirements could lead to inaccurate estimation. Hoda and
Murugesan [23] conducted a grounded theory study with Agile
practitioners and reported that some information could be
missing or delayed, which could lead to a wrong assumption
and inaccurate estimation. Bick et al. [5] conducted a case
study with Agile teams and reported that a task is re-estimated
when the information change caused the estimated effort to
became unrealistic. Complimenting the prior studies, this work
shows that a change of documented information is more likely
to influence re-estimation than a change of the information
from other sources.
Documentation in Agile. Documentation is one of the
approaches to record and convey information related to a
software product, which a team can use as a reference [3].
Sedano et al. [42] reported that a product backlog could be
used to remind the work to be done and assist the face-to-face
communication. Kasauli et al. [29] reported that practitioners
relied on information recorded in documentation (i.e., require-
ments specifications, backlog items) to build and maintain
understanding for several purposes, e.g., task implementation,
system maintenance, defining test cases. Aligning with the
prior work, our results highlight that documentation is com-
monly used for effort estimation and documented information
is considered moderately to extremely important when esti-
mating effort.
Minimal Documentation. Comprehensive documentation
is considered overhead based on the Agile manifesto [16]. On
the other hand, Kasauli et al. [29] discussed that information
might be inadequate for developers if the team aimed to
minimize documentation effort by only documenting user
stories and test cases. Ernst and Murphy [13] reported that
information could be documented in a just-in-time fashion,
i.e., requirements were first sketched out with simple natural
language statements and then fully elaborated just before or
during development. Heck and Zaidman [21] found that some
types of information (e.g., use cases, UI mock-ups) might not
be mandatory at the early stage. Motivated by the prior studies,
we identified the useful types of information that should be
documented for effort estimation, i.e., functional requirements,
user stories, definition of done, UI wireframes, acceptance
criteria, and task dependencies.
Quality Issues. Several studies analyzed the quality of
documentation. Aghajani et al. [2] manually analyzed 878
documentation-related artifacts and found that documented in-
formation was often found incorrect, incomplete, and outdated.
Kasauli et al. [29] reported that the information documented
in backlog items was often changing while the information
documented in requirement specifications was often outdated.
Prior studies also reported that documented information could
be missing or did not match the developer needs for bug-
fixing tasks [6, 53]. Similarly, in the context of Agile effort
estimation, we found that the useful types of documented
information for effort estimation were occasionally (or more)
changing or missing during or after effort estimation.
VII. THR EATS T O VALIDITY
Construct validity is related to the concerns on the survey
design. The survey respondents might misinterpret the survey
questions or definitions of the terms. To mitigate this threat, we
conducted two pilot tests with five experienced software engi-
neers to check the comprehensibility of our survey questions.
We also provided the definitions of terms and the nine types
of documented information we used in the survey. In addition,
we informed the invitees that they could directly contact the
authors for clarification or requesting support via video calls.
It might also be possible that the term "functional require-
ments" might be misinterpreted as the software requirement
specification (SRS) of the non-agile processes. However, our
study focused only on the software engineers who used Agile
methods and we asked them to answer the survey questions at
the lowest granularity that they estimate. We also confirmed
the interpretation with the pilot test participants and they
explained that the functional requirements were understood
as the task information about the functionality of the system.
Hence, it is less likely that the functional requirements were
interpreted as the SRS.
Internal validity concerns the confounding factors that
might impact our findings. Our survey results might be varied
based on other factors, e.g., company practices or product
domain. However, we did not ask the participants about their
company or product because such sensitive information might
cause them to be reluctant to participate, increased the risk
of evaluation apprehension [18] and self-selection bias [30].
Some invitees also responded that they would not participate in
the survey if it collects such information. Other study methods
(e.g., interviews, Delphi sessions) might provide additional
insights about the variation based on other factors.
In RQ2, the responses of Q5-8 might be biased toward
re-estimation if documented information was changed since
most of our respondents consider documentation important
(RQ1). To check this, we used the Spearman rank correlation
test to measure whether the responses of Q2 and Q5-8 are
highly correlated (i.e., the higher the perceived importance of
documented information, the more likely the re-estimation is
performed when documented information was changed). We
found that the correlations are small to moderate (i.e., |ρ|is
0.2-0.39). Hence, the responses in RQ2 are less likely biased.
The first two authors conducted the analysis of open-ended
answers (i.e., card sorting). Personal experiences of the authors
might influence the analysis. Therefore, we validated the
analysis by the closed-card sorting performed by the third
author. The Cohen’s Kappa coefficient [35] ranged from 0.65-
0.80, indicating substantial agreements between the sorters.
When we performed statistical tests, we converted the Likert
scale into numerical values. The statistical test results could
be impacted because the distance between the scale values
were not ordinal. However, as suggested by Harpe [27] and
Jamieson [20], it is a common practice to treat rating items
with at least five numerical categories as continuous data.
External validity is related to the concern on the generaliz-
ability of our findings. Despite the diverse background of our
respondents (c.f. Section IV-A), the survey results might not
generalize to all kinds of Agile projects or effort estimation
practices. Nevertheless, to facilitate a replicate study of this
work, we provided a full set of survey questions and relevant
documents in the replication package.2
VIII. CONCLUSIONS
Effort estimation is an important practice of Agile plan-
ning activities. To estimate the effort, practitioners use the
information to understand the task. One of the common
approaches to convey the information is to use documentation.
However, documentation has a lower priority than delivering
working software in the Agile practices. It is possible that
documentation may not be well maintained and may impact
the accuracy of effort estimation. Yet, little is known about
how documented effort can be minimized while achieving
accurate estimation. Hence, in this paper, we investigated the
importance of documentation and the types of documented
information that are useful for effort estimation in the current
Agile practices. Through a survey study with 121 Agile
practitioners, we found that:
Despite the low priority of documentation in Agile,
a majority of our respondents used documentation to
assist effort estimation and considered the documented
information as moderately to extremely important when
estimating effort.
– The reported frequency of re-estimation when the doc-
umented information was changed is statistically higher
than the reported frequency of re-estimation when the in-
formation from other sources was changed. Furthermore,
their re-estimation decision was based on the significance
or the source of the information change.
Functional requirements, user stories, definition of done,
UI wireframes, acceptance criteria, and task dependencies
were ranked as the six most useful documented infor-
mation for effort estimation, however many respondents
reported that these useful types of information were
occasionally (or more) changing or missing during (or
after) effort estimation.
The key contribution of our work is to shed light on
the importance of documented information and its impact
on the accuracy of effort estimation in Agile practices. Our
findings suggest that the quality of documented information
can have an impact on effort estimation. Nevertheless, we
do not expect Agile practitioners to prepare comprehensive
documentation. Indeed, to achieve a practice of minimal
documentation of Agile, our work highlights the six types of
documented information that practitioners should pay attention
before performing effort estimation. Our recommendations can
be done in a timely manner, which helps practitioners achieve
more accurate effort estimation with an optimal maintenance
effort of documentation.
Acknowledgement. P. Thongtanunam was partially sup-
ported by the Australian Research Council’s Discovery
Early Career Researcher Award (DECRA) funding scheme
(DE210101091).
REFERENCES
[1] “ISO/IEC/IEEE International Standard - Systems and software engineer-
ing – Vocabulary,” ISO/IEC/IEEE 24765:2010(E), pp. 1–418, 2010.
[2] E. Aghajani, C. Nagy, O. L. Vega-Márquez, M. Linares-Vásquez,
L. Moreno, G. Bavota, and M. Lanza, “Software Documentation Issues
Unveiled,” in Proc. of the ICSE, 2019, pp. 1199–1210.
[3] Y. Andriyani, R. Hoda, and R. Amor, “Understanding Knowledge
Management in Agile Software Development Practice,” in Proc. of the
KSEM, 2017, pp. 195–207.
[4] W. Behutiye, P. Karhapää, D. Costal, M. Oivo, and X. Franch, “Non-
functional Requirements Documentation in Agile Software Develop-
ment: Challenges and Solution Proposal,” in Proc. of the PROFES, 2017,
pp. 515–522.
[5] S. Bick, K. Spohrer, R. Hoda, A. Scheerer, and A. Heinzl, “Coordination
Challenges in Large-Scale Software Development: A Case Study of
Planning Misalignment in Hybrid Settings,” TSE, vol. 44, no. 10, pp.
932–950, 2018.
[6] O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus,
G. Bavota, and V. Ng, “Detecting Missing Information in Bug Descrip-
tions,” in Proc. of the ESEC/FSE, 2017, pp. 396–407.
[7] E. Coelho and A. Basu, “Effort Estimation in Agile Software Develop-
ment using Story Points,” IJAIS, vol. 3, no. 7, pp. 7–10, 2012.
[8] M. Cohn, Agile estimating and planning. Pearson Education, 2006.
[9] S. Davies and M. Roper, “What’s in a Bug Report?” in Proc. of the
ESEM, no. 26, 2014, pp. 1–10.
[10] M. Denscombe, The Good Research Guide: for small-scale social
research projects. McGraw-Hill Education (UK), 2014.
[11] C. Ebert and J. De Man, “Effectively utilizing project, product and
process knowledge,IST, vol. 50, no. 6, pp. 579–594, 2008.
[12] F. Ebert, F. Castor, N. Novielli, and A. Serebrenik, “Confusion in
code reviews: Reasons, impacts, and coping strategies,” in Proc. of the
SANER, 2019, pp. 49–60.
[13] N. A. Ernst and G. C. Murphy, “Case Studies in Just-In-Time Require-
ments Analysis,” in Proc. of the EmpiRE, 2012, pp. 25–32.
[14] I. Etikan, S. A. Musa, and R. S. Alkassim, “Comparison of Convenience
Sampling and Purposive Sampling,AJTAS, vol. 5, no. 1, pp. 1–4, 2016.
[15] A. Forward and T. C. Lethbridge, “The Relevance of Software Doc-
umentation, Tools and Technologies: A Survey,” in Proc. of the ACM
Symposium on Document Engineering, 2002, pp. 26–33.
[16] M. Fowler and J. Highsmith, “The agile manifesto,” Software Develop-
ment, vol. 9, no. 8, pp. 28–32, 2001.
[17] D. Fucci, C. Palomares, X. Franch, D. Costal, M. Raatikainen, M. Stet-
tinger, Z. Kurtanovic, T. Kojo, L. Koenig, A. Falkner et al., “Needs
and Challenges for a Platform to Support Large-scale Requirements
Engineering: a Multiple-case Study,” in Proc. of the ESEM, no. 19,
2018, pp. 1–10.
[18] A. N. Ghazi, K. Petersen, S. S. V. R. Reddy, and H. Nekkanti, “Survey
Research in Software Engineering: Problems and Mitigation Strategies,
IEEE Access, vol. 7, pp. 1–24, 2018.
[19] C. Gralha, D. Damian, A. Wasserman, M. Goulão, and J. Araújo, “The
Evolution of Requirements Practices in Software Startups,” in Proc. of
the ICSE, 2018, pp. 823–833.
[20] S. E. Harpe, “How to analyze Likert and other rating scale data,” CPTL,
vol. 7, no. 6, pp. 836–850, 2015.
[21] P. Heck and A. Zaidman, “A framework for quality assessment of just-
in-time requirements: the case of open source feature requests,” RE,
vol. 22, no. 4, pp. 453–473, 2017.
[22] ——, “A Systematic Literature Review on Quality Criteria for Agile
Requirements Specifications,” SQJ, vol. 26, no. 1, pp. 127–160, 2018.
[23] R. Hoda and L. K. Murugesan, “Multi-Level Agile Project Management
Challenges: A Self-Organizing Team Perspective,” JSS, vol. 117, pp.
245–257, 2016.
[24] R. Hoda, J. Noble, and S. Marshall, “Documentation Strategies on Agile
Software Development Projects,IJAESD, vol. 1, no. 1, pp. 23–37, 2012.
[25] I. Inayat, S. S. Salim, S. Marczak, M. Daneva, and S. Shamshirband, “A
systematic literature review on agile requirements engineering practices
and challenges,” Computers in human behavior, vol. 51, pp. 915–929,
2015.
[26] M. James and L. Walter, “Scrum Reference Card,CollabNet, 2010.
[27] S. Jamieson, “Likert scales: How to (ab) use them?” Medical education,
vol. 38, no. 12, pp. 1217–1218, 2004.
[28] M. Jørgensen and T. M. Gruschke, “The Impact of Lessons-Learned
Sessions on Effort Estimation and Uncertainty Assessments,” TSE,
vol. 35, no. 3, pp. 368–383, 2009.
[29] R. Kasauli, G. Liebel, E. Knauss, S. Gopakumar, and B. Kanagwa,
“Requirements Engineering Challenges in Large-Scale Agile System
Development,” in RE, 2017, pp. 352–361.
[30] M. Kasunic, “Designing an effective survey,” Carnegie-Mellon Univ
Pittsburgh PA Software Engineering Inst, Tech. Rep., 2005.
[31] C. Khalil and S. Khalil, “Exploring knowledge management in agile
software development organizations,IEMJ, vol. 16, no. 2, pp. 555–
569, 2020.
[32] B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C.
Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary Guidelines for
Empirical Research in Software Engineering,” TSE, vol. 28, no. 8, pp.
721–734, 2002.
[33] E. Klotins, M. Unterkalmsteiner, P. Chatzipetrou, T. Gorschek, R. Prik-
ladnicki, N. Tripathi, and L. Pompermaier, “A progression model of
software engineering goals, challenges, and practices in start-ups,” TSE,
p. To appear, 2019.
[34] S. Lauesen, Software Requirements: Styles and Techniques. Pearson
Education, 2002.
[35] M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia
medica: Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012.
[36] T. Merten, M. Falis, P. Hübner, T. Quirchmayr, S. Bürsner, and B. Paech,
“Software Feature Request Detection in Issue Tracking Systems,” in
Proc. of the RE, 2016, pp. 166–175.
[37] F. Paetsch, A. Eberlein, and F. Maurer, “Requirements Engineering and
Agile Software Development,” in Proc. of the WETICE, 2003, pp. 308–
313.
[38] C. Palomares, C. Quer, and X. Franch, “Requirements Reuse and
Requirement Patterns: A State of the Practice Survey,” ESE, vol. 22,
no. 6, pp. 2719–2762, 2017.
[39] P. Rodeghero, S. Jiang, A. Armaly, and C. McMillan, “Detecting
User Story Information in Developer-Client Conversations to Generate
Extractive Summaries,” in Proc. of the ICSE, 2017, pp. 49–59.
[40] K. S. Rubin, Essential Scrum: A Practical Guide to the Most Popular
Agile Process. Addison-Wesley, 2012.
[41] E.-M. Schön, J. Thomaschewski, and M. J. Escalona, “Agile Require-
ments Engineering: A systematic literature review,” Computer Standards
& Interfaces, vol. 49, pp. 79–91, 2017.
[42] T. Sedano, P. Ralph, and C. Péraire, “The Product Backlog,” in Proc. of
the ICSE, 2019, pp. 200–211.
[43] D. Spencer, Card sorting: Designing usable categories. Rosenfeld
Media, 2009.
[44] R. B. Svensson, T. Gorschek, P.-O. Bengtsson, and J. Widerberg, “BAM-
Backlog Assessment Method,” in Proc. of the XP, 2019, pp. 53–68.
[45] B. Tanveer, L. Guzmán, and U. M. Engel, “Understanding and improving
effort estimation in agile software development: an industrial case
study,” in Proc. of the ICSSP, 2016, pp. 41–50.
[46] J. Tizard, H. Wang, L. Yohannes, and K. Blincoe, “Can a Conversation
Paint a Picture? Mining Requirements in Software Forums,” in Proc. of
the RE, 2019, pp. 17–27.
[47] M. Usman and R. Britto, “Effort Estimation in Co-located and Globally
Distributed Agile Software Development: A Comparative Study,” in
Proc. of the IWSM-MENSURA, 2016, pp. 219–224.
[48] M. Usman, R. Britto, L.-O. Damm, and J. Börstler, “Effort estimation in
large-scale software development: An industrial case study,” IST, vol. 99,
pp. 21–40, 2018.
[49] C. Vassallo, S. Panichella, F. Palomba, S. Proksch, H. C. Gall, and
A. Zaidman, “How developers engage with static analysis tools in
different contexts,EMSE, vol. 25, no. 2, pp. 1419–1457, 2020.
[50] X. Wang, L. Zhao, Y. Wang, and J. Sun, “The Role of Requirements
Engineering Practices in Agile Development: An Empirical Study,RE,
vol. 432, pp. 195–209, 2014.
[51] X. Xia, Z. Wan, P. S. Kochhar, and D. Lo, “How Practitioners Perceive
Coding Proficiency,” in Proc. of the ICSE, 2019, pp. 924–935.
[52] J. Zhi, V. Garousi-Yusifo ˘
glu, B. Sun, G. Garousi, S. Shahnewaz, and
G. Ruhe, “Cost, benefits and quality of software development documen-
tation: A systematic mapping,” JSS, vol. 99, pp. 175–198, 2015.
[53] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and
C. Weiss, “What Makes a Good Bug Report?” TSE, vol. 36, no. 5, pp.
618–643, 2010.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Automatic static analysis tools (ASATs) are instruments that support code quality assessment by automatically detecting defects and design issues. Despite their popularity, they are characterized by (i) a high false positive rate and (ii) the low comprehensibility of the generated warnings. However, no prior studies have investigated the usage of ASATs in different development contexts (e.g., code reviews, regular development), nor how open source projects integrate ASATs into their workflows. These perspectives are paramount to improve the prioritization of the identified warnings. To shed light on the actual ASATs usage practices, in this paper we first survey 56 developers (66% from industry and 34% from open source projects) and interview 11 industrial experts leveraging ASATs in their workflow with the aim of understanding how they use ASATs in different contexts. Furthermore, to investigate how ASATs are being used in the workflows of open source projects, we manually inspect the contribution guidelines of 176 open-source systems and extract the ASATs’ configuration and build files from their corresponding GitHub repositories. Our study highlights that (i) 71% of developers do pay attention to different warning categories depending on the development context; (ii) 63% of our respondents rely on specific factors (e.g., team policies and composition) when prioritizing warnings to fix during their programming; and (iii) 66% of the projects define how to use specific ASATs, but only 37% enforce their usage for new contributions. The perceived relevance of ASATs varies between different projects and domains, which is a sign that ASATs use is still not a common practice. In conclusion, this study confirms previous findings on the unwillingness of developers to configure ASATs and it emphasizes the necessity to improve existing strategies for the selection and prioritization of ASATs warnings that are shown to developers.
Chapter
Full-text available
The necessity of software as stand-alone products, and as central parts of non-traditional software products have changed how software products are developed. It started with the introduction of the agile manifesto and has resulted in a change of how software process improvements (SPI) are conducted. Although there are agile SPI methods and several agile practices for evaluating and improving current processes and ways-of-working, no method or practices for evaluating the backlog exists. To address this gap, the Backlog Assessment Method (BAM) was developed and applied in collaboration with Telenor Sweden. BAM enables agile organizations to assess backlogs, and assure that the backlog items are good-enough for their needs and well aligned with the decision process. The results from the validation show that BAM is feasible and relevant in an industrial environment, and it indicates that BAM is useful as a tool to perform analysis of items in a specific backlog.
Article
Full-text available
Knowledge represents a sustainable asset for innovative firms and specifically software industries working in unpredictable environments. Until the late 90’s, the most widely used software development approaches advocated extensive documentation and traceability. The emphasized knowledge is mainly explicit. However, with the dynamics of progress of technologies and the market changing demands, extensive planning and documentation are quickly becoming obsolete. As such, more flexible methods, called agile methods, have gained popularity. These methods value collective learning and close collaboration between team members. To this day, little research has shown how agile software development supports knowledge management initiatives. This research helps fill this gap by reviewing the related work on agile software development by focusing on and examining knowledge management initiatives in agile organizations. Our findings highlight the way knowledge management is embedded in agile practices, including continuous communication, iterative development, knowledge repositories and engineering practices. It also emphasizes the importance of knowledge management in Information Technology development organizations.
Article
Full-text available
Background: The need for empirical investigations in software engineering is growing. Many researchers nowadays, conduct and validate their solutions using empirical research. The Survey is an empirical method which enables researchers to collect data from a large population. The main aim of the survey is to generalize the findings. Aims: In this study, we aim to identify the problems researchers face during survey design and mitigation strategies. Method: A literature review, as well as semi-structured interviews with nine software engineering researchers, were conducted to elicit their views on problems and mitigation strategies. The researchers are all focused on empirical software engineering. Results: We identified 24 problems and 65 strategies, structured according to the survey research process. The most commonly discussed problem was sampling, in particular, the ability to obtain a sufficiently large sample. To improve survey instrument design, evaluation and execution recommendations for question formulation and survey pre-testing were given. The importance of involving multiple researchers in the analysis of survey results was stressed. Conclusions: The elicited problems and strategies may serve researchers during the design of their studies. However, it was observed that some strategies were conflicting. This shows that it is important to conduct a trade-off analysis between strategies.
Conference Paper
Background: Requirement engineering is often considered a critical activity in system development projects. The increasing complexity of software as well as number and heterogeneity of stakeholders motivate the development of methods and tools for improving large-scale requirement engineering. Aims: The empirical study presented in this paper aim to identify and understand the characteristics and challenges of a platform, as desired by experts, to support requirement engineering for individual stakeholders, based on the current pain-points of their organizations when dealing with a large number requirements. Method: We conducted a multiple case study with three companies in different domains. We collected data through ten semi-structured interviews with experts from these companies. Results: The main pain-point for stakeholders is handling the vast amount of data from different sources. The foreseen platform should leverage such data to manage changes in requirements according to customers' and users' preferences. It should also offer stakeholders an estimation of how long a requirements engineering task will take to complete, along with an easier requirements dependency identification and requirements reuse strategy. Conclusions: The findings provide empirical evidence about how practitioners wish to improve their requirement engineering processes and tools. The insights are a starting point for in-depth investigations into the problems and solutions presented. Practitioners can use the results to improve existing or design new practices and tools.
Conference Paper
We use Grounded Theory to study the evolution of requirements practices of 16 software startups as they grow and introduce new products and services. These startups operate in a dynamic environment, with significant time and market pressure, and rarely have time for systematic requirements analysis. Our theory describes the evolution of practice along six dimensions that emerged as relevant to their requirements activities: requirements artefacts, knowledge management, requirements-related roles, planning, technical debt and product quality. Beyond the relationships among the dimensions, our theory also explains the turning points that drove the evolution along these dimensions. These changes are reactive, rather than planned, suggesting an overall pragmatic lightness, i.e., flexibility, in the startups' evolution towards engineering practices for requirements. Our theory organises knowledge about evolving requirements practice in maturing startups, and provides practical insights for startups' assessing their own evolution as they face challenges to their growth. Our research also suggests that a startup's evolution along the six dimensions is not fundamental to its success, but has significant effects on their product, their employees and the company.