Content uploaded by Pariya Kashfi
Author content
All content in this area was uploaded by Pariya Kashfi on Jul 25, 2020
Content may be subject to copyright.
Evidence-based Timelines for User eXperience
Software Process Improvement Retrospectives
Pariya Kashfi∗, Robert Feldt∗†, Agneta Nilsson∗, Richard Berntsson Svensson†
∗Software Engineering Division
Department of Computer Science and Engineering
Chalmers University of Technology and Gothenburg University
{pariya.kashfi,robert.feldt,agneta.nilsson}@chalmers.se
†Software Engineering Research Lab
School of Computing
Blekinge Institute of Technology
{robert.feldt, richard.berntsson.svensson}@bth.se
Abstract—We performed a retrospective meeting at a case
company to reflect on its decade of Software Process Improve-
ment (SPI) activities for enhancing UX integration. We supported
the meeting by a pre-generated timeline of the main activities.
This approach is a refinement of a similar approach that is
used in Agile projects to improve effectiveness and decrease
memory bias of retrospective meetings. The method is evaluated
through gathering practitioners’ view using a questionnaire. We
conclude that UX research and practice can benefit from the SPI
body of knowledge. We also argue that a cross-section evidence-
based timeline retrospective meeting is useful for enhancing UX
work in companies, especially for identifying and reflecting on
‘organizational issues’. This approach also provides a cross-
section longitudinal overview of the SPI activities that cannot
easily be gained in other common SPI learning approaches.
Keywords: user experience, software process improvement, orga-
nizational change, organizational issues, timeline, retrospective,
lessons learned, postmortem
I. INTRODUCTION
To deliver a piece of software that is consistent and of high
quality, practitioners need to consider a large number of soft-
ware quality characteristics, including User eXperience (UX).
UX is a user’s holistic perception of functionalities and quality
characteristics of a piece of software [1]. Research shows that
certain practices can increase the likelihood of delivering a
better UX [1] (hereafter, UX practices). But simply applying
UX practices in isolation is not enough [2]. Like methods
and practices to support other software quality characteristics,
UX practices need to be integrated into development processes
and considered throughout projects (hereafter, UX integration).
In other words, software development processes need to be
improved in order to better support UX.
Research shows that different Software Process Improve-
ment (SPI) methodologies have different impacts on various
software quality characteristics [3]. For instance, Asharfi [3]
shows that ISO 9000-3 [4] has a better impact on usability
while Capability Maturity Model (CMM) [5] on integrity,
reliability, and testability. The impact of various SPI method-
ologies on UX of the software, however, remains an open
question. Considering the general lack of theories on the
relation between SPI and UX of the product, it is expected to
observe that often companies approach UX integration in an
ad-hoc manner. They, therefore, often fail to benefit from SPI
body of knowledge or align UX integration activities with their
other ongoing SPI activities. To the best of our knowledge,
none of the current studies on UX work has investigated
the role of SPI principles in success of UX integration.
The use of SPI principles or practices, such as retrospective
meetings [6] remains unexplored in this field. Therefore,
we designed a study to investigate the potential usefulness
and benefit of retrospective meetings for reflecting on UX
integration activities. A decade of UX integration activities in
the case company spanned across various organizational units
and projects. So we found it important for practitioners to get
an overview of such activities and their interrelations since
UX integration, similar to other SPI activities, is emergent,
and shaped by intertwined activities in their context [7]. In
addition, considering the nature of UX, being originated in
non-technical fields, it is even more important that technical
practitioners get a better understanding of what UX integration
entails, and how that affects their every day work. This implied
that project retrospectives were a less suitable choice for this
case, and therefore motivated running a retrospective meeting
across sections, covering the main projects running over the
last decade, across various units in the case company.
When a project has ended and project members get involved
in other projects, they might quickly forget the details of
the activities and their interrelations [8]. This may result in
biased discussions in retrospective meetings. In our case, this
problem was expected to be compounded considering the
larger scope of the retrospective. To address this concern,
researchers [8] suggest Evidence-based timeline Retrospective
(EBTR) meetings. The idea behind EBTR is to use pre-
constructed timelines instead of creating them during the
meeting. These timelines include various view points, that are
time stamped and reflect different activities performed over
the course of a project (e.g. decisions made). Bjarnason et
al. [8] argue that EBTs can prompt memory, save meeting
time, provide objective information for the discussion, and
more importantly minimize the risk of memory bias. They
argue that EBTRs are generic enough to be customized for
different retrospective goals.
This paper1reports on our experience in performing a
EBTR meeting. The meeting was run across organizational
units and projects (i.e. sections) to facilitate reflecting on, and
coordinating UX integration activities.
II. ME THO DS
The case company is an international software development
company in Gothenburg, Sweden. It currently has around 2500
employees, and develops software for various domains, mainly
aviation, using an agile development process. The company is
organized in three different units: (i) the first unit is responsible
for developing the core features of the products, (ii) the
second unit is responsible for customization of the features
according to the needs of each customer, (iii) the third unit
involves product owners and product managers and in large
is responsible for strategic decisions and long term business
planning. In 2016, the company hired a UX integration expert
to take the responsibility of UX integration, and coordinate
related activities, taking the role of a change agent.
To prepare for and run the meeting, we adapted the steps
suggested by Bjarnason et al. [8]:
Preparation The first author was assigned as the meeting
coordinator. To create the timelines, she ran 13 interviews
in the company with practitioners from all three units. The
interviewees were selected together with our main collabora-
tors in the company. The interviews were face-to-face, audio
recorded, and lasted between 30 to 60 minutes. Analyzing the
interview data resulted in four main categories of activities:
(i) People: People who have contributed to UX integration,
including those who have been hired based on their UX-
related competences, assigned UX-related roles, or advocated
UX integration, (ii) Direct-events: Activities that have been
performed with an explicit aim to improve UX integration,
e.g., ‘starting study groups to learn about the concept of UX’,
(iii) Indirect-events: Activities that have been performed for
other purposes but indirectly influenced UX integration, e.g.,
‘process change to agile’, (iv) UX-artifacts: Tangible outputs
of UX practices in projects, e.g., a ‘UX guideline’.
Timeline construction To increase readability of the time-
lines, for each of the above group of activities, one timeline
was generated. When required, document analysis was per-
formed to discover more details about the activities. Based
on the distribution of the activities we chose ‘year’as the time
unit on the timelines. Because of the longitudinal nature of the
timelines, we also applied theories on longitudinal research [9]
when creating them.
Retrospective meeting Nine people attended the meeting,
representing the three units, at least two people from each
unit. The meeting lasted 90 minutes and was started by
going through the aims of the meeting, and definitions of the
terms used on the timelines. Then the participants received
1A more complete version of the paper is accessible online: https://arxiv.
org/abs/1605.03883
15 minutes to add their post-it notes to the timelines. The first
and the last authors moderated the meeting, and took extensive
notes to assure all the main discussion points are covered. The
participants had open discussions about the timelines, and were
guided by some focus questions if required.
Reporting the findings of the retrospective The sum-
mary of the meeting, and updated timelines were sent to the
participants for validation, and getting approval. As a result,
the timelines were finalized, and a retrospective report was
prepared and distributed among the participants, management,
and the UX integration expert.
Validating the cross-section EBT retrospective At the
end of the meeting, a questionnaire was distributed among
the participants to gather their thoughts about the EBTR. The
questionnaire was inspired by the questionnaire by Bjarnason
et. al [8]. After initial analysis of the responses, a number of
follow up open-ended questions were sent to the participants to
gather some additional data. In order to investigate usefulness
of the findings of the meeting for the UX integration expert, the
results of the meeting (the retrospective report and the updated
timelines) were shared with her. She also received a series of
open-ended questions to reflect on the findings. Data analysis
is descriptive, and qualitative and emphasizes indication of
the responses rather than their statistical significance, since
the sample number was too small.
To increase construct validity [10], the subjects were se-
lected together with our contacts in the company, following
a list of criteria prepared based on the goals of the meeting.
Also, to minimize the impact of presence of a researcher on the
subjects’ behavior and responses, confidentiality of the data
was guaranteed. To increase internal validity, the interviews
and the meeting were recorded either in audio or in form of
extensive notes. The method is validated in other cases [8]
that can help in comparing how EBTs can be used in different
settings to increase external validity.
III. RES ULTS AN D ANA LYSIS
During the meeting, the participants repeatedly pointed to
the timelines to refer to different activities (i.e. events, people
or artifacts). They also discussed the interrelation among a
number of them, in particular the UX artifact timeline and
the direct-event timeline. We also observed that the printed
timelines facilitated discussions about details of the activities
and how the participants’ perceived positive or negative impact
of these activities on UX integration in the company. Here, we
report on the experiences from using the method, based on our
observations in the meeting, the participants’ responses to the
questionnaire, and reflection of the company’s UX integration
expert and management.
According to the questionnaire responses, the participants
generally found this method useful for learning about, and
reflecting on UX integration in the company. They emphasized
that the method provided a ‘good overview’ and a ‘big picture’
of the UX integration activities which made it easier to reflect
on the activities. They also highlighted that the timeline helped
them get informed about the activities performed in different
units. This was not clear to all participants before the meeting
in particular since these units lack communication (at least)
concerning UX integration.
In addition, according to the participants, the meeting
facilitated communication among them. Regarding this one
participant stated: “I think this is a very good technique. It
keeps everyone focused, one can come prepared, and you
get the chance to steer the meeting to what you believe is
important. Of course there is always the risk to steer too
much and miss out on some of the ‘out-of-the-box’ thinking.”
Another participant stressed the timeline is useful as a ‘base
for the discussions’: “The timelines helped us to get a common
understanding about what had happened during the years, and
we could point on the timelines and have them as a base for the
discussion.” According to the participants, the timelines pro-
vided ‘a neutral’ and ‘factual’ basis to reflect on the activities.
Also, one participant highlighted that the retrospective meeting
was much ‘calmer’ than other informal discussion meetings
about UX integration because of the neutral and factual basis,
and presence of ‘a neutral moderator’. Especially since current
conflicts among various units, or even individuals often prevent
’constructive discussions’ about UX integration. Emphasizing
this, one participant stated: “I liked that people from all three
departments sat together to share some of their views. This
helps us align for future actions”
In the meeting, we observed that the ‘direct-event’, and
‘UX artifact’ timelines opened up a series of discussions
about different UX integration activities that have mostly
been bottom-up in some teams and based on personal interest
of individual developers. Through the people timeline, it
also became obvious that some of these activities have been
performed by people with no UX-related role or responsibility.
Some participants believed that it was not easy to identify the
connections among activities while the others expressed they
could identify some. For instance a practitioner highlighted: “I
liked the overview and the visibility of when things happened
in relation to each other.”
The participants also highlighted that the meeting facili-
tated identifying, and making the main points of agreement
and disagreement concerning UX integration explicit. For
instance, it became clear that most activities have been ’ad-
hoc’, with limited leadership, support from management, or
even mandate. As one of the participants pointed out: “the
difference between people and the lack of formal work became
obvious.”. It also became clear that these activities have not
been coordinated (e.g. similar activities duplicated in different
units without alignment, or artifacts being introduced without
informing roles that would receive these artifacts in later
stages of the project). Another participant responded that the
meeting helped them to agree on what main activities have
been performed in the past that influenced UX integration.
Similarly, another participant stated: “It is clear that we don’t
agree on the UX methods and processes and it is good that
is out in the open now.” In addition, the participants generally
agreed that the meeting was helpful in emphasizing the limited
communication concerning UX integration. For instance, one
participant highlighted that the meeting facilitated ‘communi-
cation’ between different roles with different perspectives, and
further emphasized: “I think it was clear why we don’t have
a common understanding about how and what to do with UX:
[limited] communication.”
We observed that the meeting discussions, and in particular
learning about different viewpoints helped generating ideas for
future improvements. For instance, one participant stated: “We
need one person that is in charge [of UX integration] . .. and to
look at which kind of profile we need for these tasks.” Another
participant stated: “I think I got a good understanding of what
needs to be changed.”
The UX integration expert was very positive towards the
findings of the meeting, and using them to plan future UX
integration improvement activities for the company: “Based
on the information from the meeting I think I have a better
understanding of where they are coming from . . . I have to
show them what use they can get from a UX-person.” Accord-
ing to her, the findings provide useful and valuable insight
to understand the challenges with UX integration over the
years and how they have influenced the attitude of practitioners
towards UX. She stressed: “I think this kind of information is
very useful for a new UX person coming in to the company.”
But she emphasized such information is often difficult to
gather in particular for practitioners like her that are new to
an organization. One example of such information is how over
the years various people have influenced UX integration (i.e.
the people timeline) and how this has been perceived by other
practitioners (i.e. the discussions around the people timeline
in the meeting). Another use of the findings according to her
was learning about the ‘expectations’ for her role as ‘UX
integration expert’, and the people’s attitude towards UX in
general. Regarding this, she stated: “They expected a totally
different approach than the one I am offering to them right
now.” The expert argued for another potential use of cross-
section EBT retrospective meetings by stating: “this kind of
meeting could be fruitful to do as a UX-consulting company
. . . to visualize how the clients company have treated UX over
the years and come with possible solutions on how to work
better with it.” As we saw, this method can also facilitate a
longitudinal analysis of UX integration (or other SPI activities)
although this a bonus rather the main focus.
IV. DISCUSSION
Our hypothesis that a cross-section EBT retrospective meet-
ing could be useful for reflecting on, learning from, and
coordinating UX integration activities is plausible based on
our findings. According to the questionnaire responses, the
participants generally agreed that the meeting helped them to
(i) have a factual discussion in the meeting, (ii) remember
and agree on UX integration activities, and activities that
indirectly influenced UX integration (iii) remember details
of those activities to a good extent, (iv) get an overview of
the activities. These findings are in line with previous studies
on EBTRs in context of agile projects [8]. The participants,
however, had a divided view on whether the meeting (and
in particular the EBT) provided sufficient data to identify
connections between the activities. Still in previous studies,
this is reported as one of the benefits of EBTs. Below we
discuss the relevance of our findings in the context of UX
integration and SPI in general.
Providing an overview of the activities: Practitioners’ stated
that this approach gave them a ‘good overview of the activities’
in particular ‘activities performed in different units’. This is
evidence that the method is useful in providing a cross-section
overview of the activities across projects and organizational
units. Such an overview is not likely to be gained through
separate project retrospectives that is a more common learning
practice in SPI. Because project retrospectives focus on a
certain project and cannot directly provide a cross-section
overview of the improvement activities. As emphasized by
the UX integration expert, one use of such an overview is
to find out ‘how the company has so far worked with UX
integration’. This provides more insight about the company’s
history, a useful input for planning future improvements.
Visualizing connection between the activities: An overview
of the activities can facilitate gaining insight about how they
have affected each other. The meeting participants were, to
some extent, able to identify the connection between various
activities on a timeline or across timelines. Insight about the
intertwined activities is an important input for analyzing SPI
activities and increasing its success [7].
Facilitating communication: Our experience shows that the
cross-section EBT retrospective meeting can facilitate practi-
tioners’ communication about UX integration through helping
in identifying the main challenges to UX integration, and
providing an overview of the improvement activities. One
benefit of the cross-section EBT retrospective was to make
points of agreement and disagreement explicit. This can be in
particular useful in case of conflicts in the organizations. For
instance, the participants appreciated that some of the main
points of disagreement concerning UX integration activities
‘became obvious’. Because by using factual and unbiased
timelines, this approach prevents biased discussions [8] as
also emphasized by the participants. They for instance stressed
that the meeting was ‘calmer’ than expected considering the
existing conflicts among various units. The UX integration
expert found the results of the meeting informative, e.g.,
concerning ‘expectations for her role’ that as he stated is
valuable information for a ‘new person coming in’ to any
organization as a change agent to improve UX integration.
Identifying improvements for future: Our experience also
shows that the meeting facilitates generating ideas for future
improvements and helps practitioners to collectively identify
‘what needs to be changed’ as expressed by one of the par-
ticipants. As another participant emphasized, the discussions
in the meeting can help them better ‘align for future’. The
UX integration expert also stated that the information gained
from the meeting provided ideas for ‘possible solutions’ and
inspired ideas for ‘how to work better with UX integration’
considering the context, i.e., the history of the UX integration
activities in the company.
Facilitating active staff involvement: We observed that the
cross-section EBT retrospective meeting provided a venue
for practitioners with different backgrounds to actively get
involved in this reflection and learning activity. Through this
meeting, both technical and business staff got a chance to raise
their concerns about UX integration and how that could impact
their everyday work or long term plans. There is however
no guarantee that the involvement will be pursued throughout
future UX integration improvement activities, but the meeting
is at least a starting point for that. We hypothesize that frequent
cross-section EBT retrospectives can be a ‘milestone’ to assure
active staff involvement, an open research questions.
The topics we discussed above belong to the organizational
aspects of UX integration (or SPI in general). Therefore, we
argue that this method can support practitioners in dealing
with organizational issues. This is in particular important since
organizational issues related to UX integration still remain less
explored [11]. The findings of this study however should be
used with caution since they are based on a single case study.
This study is an evidence that UX integration practice
can benefit from the SPI body of knowledge considering the
similarities between the two topics. We therefore encourage
the UX community to take a SPI perspective when focusing on
UX integration. It is also reasonable to assume that the benefits
of cross-section EBT retrospective meetings experienced in
this case study can be observed for other SPI activities as
well, an interesting subject to investigate in future research.
REFERENCES
[1] M. Hassenzahl, Experience Design: Technology for All the Right Rea-
sons. San Francisco, CA: Morgan & Claypool, 2010.
[2] J. Winter and K. R¨
onkk¨
o, “SPI success factors within product usability
evaluation,” Journal of Systems and Software, vol. 83, no. 11, pp. 2059–
2072, 2010.
[3] N. Ashrafi, “The impact of software process improvement on quality:
in theory and practice,” Information & Management, vol. 40, no. 7, pp.
677–690, 2003.
[4] Quality management and quality assurance standards – Part 3: Guide-
lines for the application of ISO 9001:1994 to the development, supply,
installation and maintenance of computer software. ISO 9000-3:1997,
1997.
[5] M. Paulk, B. Curtis, M. Chrissis, and C. Weber, “Capability maturity
model, version 1.1,” IEEE Software, vol. 10, no. 4, pp. 18–27, 1993.
[6] T. Dingsøyr, N. Moe, J. Schalken, and T. St˚
alhane, “Organizational
Learning Through Project Postmortem Reviews - An Explorative Case
Study,” in Proceedings of the 14th European software process improve-
ment conference (EuroSPI 2007), 2007, pp. 136–147.
[7] I. Allison and Y. Merali, “Software process improvement as emergent
change: A structurational analysis,” Information and Software Technol-
ogy, vol. 49, no. 6, pp. 668–681, 2007.
[8] E. Bjarnason, A. Hess, R. Berntsson Svensson, B. Regnell, and J. Doerr,
“Reflecting on Evidence- Based Timelines,” IEEE Software, vol. 31,
no. 4, pp. 37–43, 2014.
[9] C. T. Street and K. W. Ward, “Improving validity and reliability in
longitudinal case study timelines,” European Journal of Information
Systems, vol. 21, no. 2, pp. 160–175, 2012.
[10] P. Runeson and M. H¨
ost, “Guidelines for conducting and reporting case
study research in software engineering,” Empirical Software Engineer-
ing, vol. 14, no. 2, pp. 131–164, 2008.
[11] J. Winter, K. R¨
onkk¨
o, and M. Rissanen, “Identifying organizational
barriers - A case study of usability work when developing software
in the automation industry,” Journal of Systems and Software, vol. 88,
no. 1, pp. 54–73, 2014.
