Conference PaperPDF Available

Evidence-Based Timelines for User eXperience Software Process Improvement Retrospectives

Evidence-based Timelines for User eXperience
Software Process Improvement Retrospectives
Pariya Kashfi, Robert Feldt∗†, Agneta Nilsson, Richard Berntsson Svensson
Software Engineering Division
Department of Computer Science and Engineering
Chalmers University of Technology and Gothenburg University
Software Engineering Research Lab
School of Computing
Blekinge Institute of Technology
{robert.feldt, richard.berntsson.svensson}
Abstract—We performed a retrospective meeting at a case
company to reflect on its decade of Software Process Improve-
ment (SPI) activities for enhancing UX integration. We supported
the meeting by a pre-generated timeline of the main activities.
This approach is a refinement of a similar approach that is
used in Agile projects to improve effectiveness and decrease
memory bias of retrospective meetings. The method is evaluated
through gathering practitioners’ view using a questionnaire. We
conclude that UX research and practice can benefit from the SPI
body of knowledge. We also argue that a cross-section evidence-
based timeline retrospective meeting is useful for enhancing UX
work in companies, especially for identifying and reflecting on
‘organizational issues’. This approach also provides a cross-
section longitudinal overview of the SPI activities that cannot
easily be gained in other common SPI learning approaches.
Keywords: user experience, software process improvement, orga-
nizational change, organizational issues, timeline, retrospective,
lessons learned, postmortem
To deliver a piece of software that is consistent and of high
quality, practitioners need to consider a large number of soft-
ware quality characteristics, including User eXperience (UX).
UX is a user’s holistic perception of functionalities and quality
characteristics of a piece of software [1]. Research shows that
certain practices can increase the likelihood of delivering a
better UX [1] (hereafter, UX practices). But simply applying
UX practices in isolation is not enough [2]. Like methods
and practices to support other software quality characteristics,
UX practices need to be integrated into development processes
and considered throughout projects (hereafter, UX integration).
In other words, software development processes need to be
improved in order to better support UX.
Research shows that different Software Process Improve-
ment (SPI) methodologies have different impacts on various
software quality characteristics [3]. For instance, Asharfi [3]
shows that ISO 9000-3 [4] has a better impact on usability
while Capability Maturity Model (CMM) [5] on integrity,
reliability, and testability. The impact of various SPI method-
ologies on UX of the software, however, remains an open
question. Considering the general lack of theories on the
relation between SPI and UX of the product, it is expected to
observe that often companies approach UX integration in an
ad-hoc manner. They, therefore, often fail to benefit from SPI
body of knowledge or align UX integration activities with their
other ongoing SPI activities. To the best of our knowledge,
none of the current studies on UX work has investigated
the role of SPI principles in success of UX integration.
The use of SPI principles or practices, such as retrospective
meetings [6] remains unexplored in this field. Therefore,
we designed a study to investigate the potential usefulness
and benefit of retrospective meetings for reflecting on UX
integration activities. A decade of UX integration activities in
the case company spanned across various organizational units
and projects. So we found it important for practitioners to get
an overview of such activities and their interrelations since
UX integration, similar to other SPI activities, is emergent,
and shaped by intertwined activities in their context [7]. In
addition, considering the nature of UX, being originated in
non-technical fields, it is even more important that technical
practitioners get a better understanding of what UX integration
entails, and how that affects their every day work. This implied
that project retrospectives were a less suitable choice for this
case, and therefore motivated running a retrospective meeting
across sections, covering the main projects running over the
last decade, across various units in the case company.
When a project has ended and project members get involved
in other projects, they might quickly forget the details of
the activities and their interrelations [8]. This may result in
biased discussions in retrospective meetings. In our case, this
problem was expected to be compounded considering the
larger scope of the retrospective. To address this concern,
researchers [8] suggest Evidence-based timeline Retrospective
(EBTR) meetings. The idea behind EBTR is to use pre-
constructed timelines instead of creating them during the
meeting. These timelines include various view points, that are
time stamped and reflect different activities performed over
the course of a project (e.g. decisions made). Bjarnason et
al. [8] argue that EBTs can prompt memory, save meeting
time, provide objective information for the discussion, and
more importantly minimize the risk of memory bias. They
argue that EBTRs are generic enough to be customized for
different retrospective goals.
This paper1reports on our experience in performing a
EBTR meeting. The meeting was run across organizational
units and projects (i.e. sections) to facilitate reflecting on, and
coordinating UX integration activities.
The case company is an international software development
company in Gothenburg, Sweden. It currently has around 2500
employees, and develops software for various domains, mainly
aviation, using an agile development process. The company is
organized in three different units: (i) the first unit is responsible
for developing the core features of the products, (ii) the
second unit is responsible for customization of the features
according to the needs of each customer, (iii) the third unit
involves product owners and product managers and in large
is responsible for strategic decisions and long term business
planning. In 2016, the company hired a UX integration expert
to take the responsibility of UX integration, and coordinate
related activities, taking the role of a change agent.
To prepare for and run the meeting, we adapted the steps
suggested by Bjarnason et al. [8]:
Preparation The first author was assigned as the meeting
coordinator. To create the timelines, she ran 13 interviews
in the company with practitioners from all three units. The
interviewees were selected together with our main collabora-
tors in the company. The interviews were face-to-face, audio
recorded, and lasted between 30 to 60 minutes. Analyzing the
interview data resulted in four main categories of activities:
(i) People: People who have contributed to UX integration,
including those who have been hired based on their UX-
related competences, assigned UX-related roles, or advocated
UX integration, (ii) Direct-events: Activities that have been
performed with an explicit aim to improve UX integration,
e.g., ‘starting study groups to learn about the concept of UX’,
(iii) Indirect-events: Activities that have been performed for
other purposes but indirectly influenced UX integration, e.g.,
‘process change to agile’, (iv) UX-artifacts: Tangible outputs
of UX practices in projects, e.g., a ‘UX guideline’.
Timeline construction To increase readability of the time-
lines, for each of the above group of activities, one timeline
was generated. When required, document analysis was per-
formed to discover more details about the activities. Based
on the distribution of the activities we chose ‘year’as the time
unit on the timelines. Because of the longitudinal nature of the
timelines, we also applied theories on longitudinal research [9]
when creating them.
Retrospective meeting Nine people attended the meeting,
representing the three units, at least two people from each
unit. The meeting lasted 90 minutes and was started by
going through the aims of the meeting, and definitions of the
terms used on the timelines. Then the participants received
1A more complete version of the paper is accessible online: https://arxiv.
15 minutes to add their post-it notes to the timelines. The first
and the last authors moderated the meeting, and took extensive
notes to assure all the main discussion points are covered. The
participants had open discussions about the timelines, and were
guided by some focus questions if required.
Reporting the findings of the retrospective The sum-
mary of the meeting, and updated timelines were sent to the
participants for validation, and getting approval. As a result,
the timelines were finalized, and a retrospective report was
prepared and distributed among the participants, management,
and the UX integration expert.
Validating the cross-section EBT retrospective At the
end of the meeting, a questionnaire was distributed among
the participants to gather their thoughts about the EBTR. The
questionnaire was inspired by the questionnaire by Bjarnason
et. al [8]. After initial analysis of the responses, a number of
follow up open-ended questions were sent to the participants to
gather some additional data. In order to investigate usefulness
of the findings of the meeting for the UX integration expert, the
results of the meeting (the retrospective report and the updated
timelines) were shared with her. She also received a series of
open-ended questions to reflect on the findings. Data analysis
is descriptive, and qualitative and emphasizes indication of
the responses rather than their statistical significance, since
the sample number was too small.
To increase construct validity [10], the subjects were se-
lected together with our contacts in the company, following
a list of criteria prepared based on the goals of the meeting.
Also, to minimize the impact of presence of a researcher on the
subjects’ behavior and responses, confidentiality of the data
was guaranteed. To increase internal validity, the interviews
and the meeting were recorded either in audio or in form of
extensive notes. The method is validated in other cases [8]
that can help in comparing how EBTs can be used in different
settings to increase external validity.
During the meeting, the participants repeatedly pointed to
the timelines to refer to different activities (i.e. events, people
or artifacts). They also discussed the interrelation among a
number of them, in particular the UX artifact timeline and
the direct-event timeline. We also observed that the printed
timelines facilitated discussions about details of the activities
and how the participants’ perceived positive or negative impact
of these activities on UX integration in the company. Here, we
report on the experiences from using the method, based on our
observations in the meeting, the participants’ responses to the
questionnaire, and reflection of the company’s UX integration
expert and management.
According to the questionnaire responses, the participants
generally found this method useful for learning about, and
reflecting on UX integration in the company. They emphasized
that the method provided a ‘good overview’ and a ‘big picture’
of the UX integration activities which made it easier to reflect
on the activities. They also highlighted that the timeline helped
them get informed about the activities performed in different
units. This was not clear to all participants before the meeting
in particular since these units lack communication (at least)
concerning UX integration.
In addition, according to the participants, the meeting
facilitated communication among them. Regarding this one
participant stated: “I think this is a very good technique. It
keeps everyone focused, one can come prepared, and you
get the chance to steer the meeting to what you believe is
important. Of course there is always the risk to steer too
much and miss out on some of the ‘out-of-the-box’ thinking.
Another participant stressed the timeline is useful as a ‘base
for the discussions’: “The timelines helped us to get a common
understanding about what had happened during the years, and
we could point on the timelines and have them as a base for the
discussion. According to the participants, the timelines pro-
vided ‘a neutral’ and ‘factual’ basis to reflect on the activities.
Also, one participant highlighted that the retrospective meeting
was much ‘calmer’ than other informal discussion meetings
about UX integration because of the neutral and factual basis,
and presence of ‘a neutral moderator’. Especially since current
conflicts among various units, or even individuals often prevent
’constructive discussions’ about UX integration. Emphasizing
this, one participant stated: “I liked that people from all three
departments sat together to share some of their views. This
helps us align for future actions”
In the meeting, we observed that the ‘direct-event’, and
‘UX artifact’ timelines opened up a series of discussions
about different UX integration activities that have mostly
been bottom-up in some teams and based on personal interest
of individual developers. Through the people timeline, it
also became obvious that some of these activities have been
performed by people with no UX-related role or responsibility.
Some participants believed that it was not easy to identify the
connections among activities while the others expressed they
could identify some. For instance a practitioner highlighted: “I
liked the overview and the visibility of when things happened
in relation to each other.
The participants also highlighted that the meeting facili-
tated identifying, and making the main points of agreement
and disagreement concerning UX integration explicit. For
instance, it became clear that most activities have been ’ad-
hoc’, with limited leadership, support from management, or
even mandate. As one of the participants pointed out: “the
difference between people and the lack of formal work became
obvious.. It also became clear that these activities have not
been coordinated (e.g. similar activities duplicated in different
units without alignment, or artifacts being introduced without
informing roles that would receive these artifacts in later
stages of the project). Another participant responded that the
meeting helped them to agree on what main activities have
been performed in the past that influenced UX integration.
Similarly, another participant stated: “It is clear that we don’t
agree on the UX methods and processes and it is good that
is out in the open now. In addition, the participants generally
agreed that the meeting was helpful in emphasizing the limited
communication concerning UX integration. For instance, one
participant highlighted that the meeting facilitated ‘communi-
cation’ between different roles with different perspectives, and
further emphasized: “I think it was clear why we don’t have
a common understanding about how and what to do with UX:
[limited] communication.
We observed that the meeting discussions, and in particular
learning about different viewpoints helped generating ideas for
future improvements. For instance, one participant stated: “We
need one person that is in charge [of UX integration] . .. and to
look at which kind of profile we need for these tasks. Another
participant stated: “I think I got a good understanding of what
needs to be changed.
The UX integration expert was very positive towards the
findings of the meeting, and using them to plan future UX
integration improvement activities for the company: “Based
on the information from the meeting I think I have a better
understanding of where they are coming from . . . I have to
show them what use they can get from a UX-person. Accord-
ing to her, the findings provide useful and valuable insight
to understand the challenges with UX integration over the
years and how they have influenced the attitude of practitioners
towards UX. She stressed: “I think this kind of information is
very useful for a new UX person coming in to the company.
But she emphasized such information is often difficult to
gather in particular for practitioners like her that are new to
an organization. One example of such information is how over
the years various people have influenced UX integration (i.e.
the people timeline) and how this has been perceived by other
practitioners (i.e. the discussions around the people timeline
in the meeting). Another use of the findings according to her
was learning about the ‘expectations’ for her role as ‘UX
integration expert’, and the people’s attitude towards UX in
general. Regarding this, she stated: “They expected a totally
different approach than the one I am offering to them right
now. The expert argued for another potential use of cross-
section EBT retrospective meetings by stating: “this kind of
meeting could be fruitful to do as a UX-consulting company
. . . to visualize how the clients company have treated UX over
the years and come with possible solutions on how to work
better with it. As we saw, this method can also facilitate a
longitudinal analysis of UX integration (or other SPI activities)
although this a bonus rather the main focus.
Our hypothesis that a cross-section EBT retrospective meet-
ing could be useful for reflecting on, learning from, and
coordinating UX integration activities is plausible based on
our findings. According to the questionnaire responses, the
participants generally agreed that the meeting helped them to
(i) have a factual discussion in the meeting, (ii) remember
and agree on UX integration activities, and activities that
indirectly influenced UX integration (iii) remember details
of those activities to a good extent, (iv) get an overview of
the activities. These findings are in line with previous studies
on EBTRs in context of agile projects [8]. The participants,
however, had a divided view on whether the meeting (and
in particular the EBT) provided sufficient data to identify
connections between the activities. Still in previous studies,
this is reported as one of the benefits of EBTs. Below we
discuss the relevance of our findings in the context of UX
integration and SPI in general.
Providing an overview of the activities: Practitioners’ stated
that this approach gave them a ‘good overview of the activities’
in particular ‘activities performed in different units’. This is
evidence that the method is useful in providing a cross-section
overview of the activities across projects and organizational
units. Such an overview is not likely to be gained through
separate project retrospectives that is a more common learning
practice in SPI. Because project retrospectives focus on a
certain project and cannot directly provide a cross-section
overview of the improvement activities. As emphasized by
the UX integration expert, one use of such an overview is
to find out ‘how the company has so far worked with UX
integration’. This provides more insight about the company’s
history, a useful input for planning future improvements.
Visualizing connection between the activities: An overview
of the activities can facilitate gaining insight about how they
have affected each other. The meeting participants were, to
some extent, able to identify the connection between various
activities on a timeline or across timelines. Insight about the
intertwined activities is an important input for analyzing SPI
activities and increasing its success [7].
Facilitating communication: Our experience shows that the
cross-section EBT retrospective meeting can facilitate practi-
tioners’ communication about UX integration through helping
in identifying the main challenges to UX integration, and
providing an overview of the improvement activities. One
benefit of the cross-section EBT retrospective was to make
points of agreement and disagreement explicit. This can be in
particular useful in case of conflicts in the organizations. For
instance, the participants appreciated that some of the main
points of disagreement concerning UX integration activities
‘became obvious’. Because by using factual and unbiased
timelines, this approach prevents biased discussions [8] as
also emphasized by the participants. They for instance stressed
that the meeting was ‘calmer’ than expected considering the
existing conflicts among various units. The UX integration
expert found the results of the meeting informative, e.g.,
concerning ‘expectations for her role’ that as he stated is
valuable information for a ‘new person coming in’ to any
organization as a change agent to improve UX integration.
Identifying improvements for future: Our experience also
shows that the meeting facilitates generating ideas for future
improvements and helps practitioners to collectively identify
‘what needs to be changed’ as expressed by one of the par-
ticipants. As another participant emphasized, the discussions
in the meeting can help them better ‘align for future’. The
UX integration expert also stated that the information gained
from the meeting provided ideas for ‘possible solutions’ and
inspired ideas for ‘how to work better with UX integration’
considering the context, i.e., the history of the UX integration
activities in the company.
Facilitating active staff involvement: We observed that the
cross-section EBT retrospective meeting provided a venue
for practitioners with different backgrounds to actively get
involved in this reflection and learning activity. Through this
meeting, both technical and business staff got a chance to raise
their concerns about UX integration and how that could impact
their everyday work or long term plans. There is however
no guarantee that the involvement will be pursued throughout
future UX integration improvement activities, but the meeting
is at least a starting point for that. We hypothesize that frequent
cross-section EBT retrospectives can be a ‘milestone’ to assure
active staff involvement, an open research questions.
The topics we discussed above belong to the organizational
aspects of UX integration (or SPI in general). Therefore, we
argue that this method can support practitioners in dealing
with organizational issues. This is in particular important since
organizational issues related to UX integration still remain less
explored [11]. The findings of this study however should be
used with caution since they are based on a single case study.
This study is an evidence that UX integration practice
can benefit from the SPI body of knowledge considering the
similarities between the two topics. We therefore encourage
the UX community to take a SPI perspective when focusing on
UX integration. It is also reasonable to assume that the benefits
of cross-section EBT retrospective meetings experienced in
this case study can be observed for other SPI activities as
well, an interesting subject to investigate in future research.
[1] M. Hassenzahl, Experience Design: Technology for All the Right Rea-
sons. San Francisco, CA: Morgan & Claypool, 2010.
[2] J. Winter and K. R¨
o, “SPI success factors within product usability
evaluation, Journal of Systems and Software, vol. 83, no. 11, pp. 2059–
2072, 2010.
[3] N. Ashrafi, “The impact of software process improvement on quality:
in theory and practice,” Information & Management, vol. 40, no. 7, pp.
677–690, 2003.
[4] Quality management and quality assurance standards Part 3: Guide-
lines for the application of ISO 9001:1994 to the development, supply,
installation and maintenance of computer software. ISO 9000-3:1997,
[5] M. Paulk, B. Curtis, M. Chrissis, and C. Weber, “Capability maturity
model, version 1.1,” IEEE Software, vol. 10, no. 4, pp. 18–27, 1993.
[6] T. Dingsøyr, N. Moe, J. Schalken, and T. St˚
alhane, “Organizational
Learning Through Project Postmortem Reviews - An Explorative Case
Study,” in Proceedings of the 14th European software process improve-
ment conference (EuroSPI 2007), 2007, pp. 136–147.
[7] I. Allison and Y. Merali, “Software process improvement as emergent
change: A structurational analysis,” Information and Software Technol-
ogy, vol. 49, no. 6, pp. 668–681, 2007.
[8] E. Bjarnason, A. Hess, R. Berntsson Svensson, B. Regnell, and J. Doerr,
“Reflecting on Evidence- Based Timelines, IEEE Software, vol. 31,
no. 4, pp. 37–43, 2014.
[9] C. T. Street and K. W. Ward, “Improving validity and reliability in
longitudinal case study timelines,” European Journal of Information
Systems, vol. 21, no. 2, pp. 160–175, 2012.
[10] P. Runeson and M. H¨
ost, “Guidelines for conducting and reporting case
study research in software engineering,” Empirical Software Engineer-
ing, vol. 14, no. 2, pp. 131–164, 2008.
[11] J. Winter, K. R¨
o, and M. Rissanen, “Identifying organizational
barriers - A case study of usability work when developing software
in the automation industry, Journal of Systems and Software, vol. 88,
no. 1, pp. 54–73, 2014.
... Second, companies should concentrate on the interplay between UX experts, developers and other stakeholders during the actual design and development. We present our early results on stakeholder involvement that are found through empirical studies in a number of software development companies (for more details, see [12,13]). ...
... UX integration can be considered a type of Software Process Improvement (SPI) [12]. The idea behind SPI is that development processes often need to be managed and improved in order for the outcome software to be of better quality [19]. ...
... In particular, since the software development community already has access to ample SPI guidelines and best practices [19]. Since SPI guidelines are generic and they do not directly address UX or other specific software quality characteristics [12], they need to be customized to better suit UX integration. ...
Full-text available
Stakeholder involvement is one of the major success factors in integrating user experience (UX) practices into software development processes and organizations. It is also a necessity for agile software development. However, practitioners still have limited access to guidelines on successful involvement of UX stakeholders in agile settings. Moreover, agile UX literature does not well address the specific characteristics of UX and it does not clearly differentiate between UX and usability work. This paper presents two guidelines for supporting stakeholder involvement in both UX integration and the daily UX work. In particular, we focus on the special characteristics of UX: being dynamic, subjective, holistic, and context-dependent. The guidelines clarify practical implications of these characteristics for practitioners. In addition, they can help researchers in addressing these characteristics better in agile UX research.
... Kashfi et al. (2016) identified four aspects (they called them activities): people (contributing to UX), direct events (aiming to improve UX), indirect events (indirectly improving UX), and UX artefacts (outputs from UX practices). We used the same aspects; however, we did not separate events into direct and indirect as this separation was not relevant to our investigation, all events with direct or indirect contribution to UX were regarded as important. ...
Full-text available
Empirical research on software development practices in startups is growing. However, little has been investigated about how User eXperience (UX) work has been carried out in software startups. The primary objective of this paper is to investigate what software startups need from UX work. To achieve this goal, we conducted open-ended interviews and retrospective meetings with 16 software professionals from two software startups in Brazil. We analysed the data qualitatively using different coding approaches: initial coding, focused coding, and theoretical coding. We found 14 UX work-related needs which emerged from the daily practices used for software development in the two startups studied. Based on our findings, we propose an initial theoretical framework that highlights two theoretical themes and four groups underlying the needs identified. Our study reveals several relationships between UX work-related needs which are helpful to understand in order to identify what startups need from UX work in practice and to focus startup teams' efforts on the most urgent needs. As future work, we plan to explore ways in which these needs may be addressed so that UX work may be put into practice in software startups.
... The objective of the UX Retrospective is to better estimate the potential UX, using UX Poker. Conducting a retrospective can generally identify areas for improvement (Schwaber and Sutherland, 2020), especially for software development (Kashfi et al., 2016). ...
Context. Agile methods are increasingly being used by companies, to develop digital products and services faster and more effectively. Today's users not only demand products that are easy to use, but also products with a high User Experience (UX). Agile methods themselves do not directly support the development of products with a good user experience. In combination with UX activities, it is potentially possible to develop a good UX. Objective. The objective of this PhD thesis is to develop a UX Lifecycle, to manage the user experience in the context of Agile methods. With this UX Lifecycle, Agile teams can manage the UX of their product, in a targeted way. Method. We developed the UX Lifecycle step by step, according to the Design Science Research Methodology. First, we conducted a Structured Literature Review (SLR) to determine the state of the art of UX management. The result of the SLR concludes in a GAP analysis. On this basis, we derived requirements for UX management. These requirements were then implemented in the UX Lifecycle. In developing the UX Lifecycle, we developed additional methods (UX Poker, UEQ KPI, and IPA), to be used when deploying the UX Lifecycle. Each of these methods has been validated in studies, with a total of 497 respondents from three countries (Germany, England, and Spain). Finally, we validated the UX Lifecycle, as a whole, with a Delphi study, with a total of 24 international experts from four countries (Germany, Argentina, Spain, and Poland). Results. The iterative UX Lifecycle (Figure 1) consists of five steps: Initial Step 0 ‘Preparation’, Step 1 ‘UX Poker’ (before development/Estimated UX), Step 2 ‘Evaluate Prototype’ (during development/Probable UX), Step 3 ‘Evaluate Product Increment’ (after development/Implemented UX), and a subsequent Step 4 ‘UX Retrospective’. With its five steps, the UX Lifecycle provides the structure for continuously measuring and evaluating the UX, in the various phases. This makes it possible to develop the UX in a targeted manner, and to check it permanently. In addition, we have developed the UX Poker method. With this method, the User Experience can be determined by the Agile team, in the early phases of development. The evaluation study of UX Poker has indicated that UX Poker can be used to estimate the UX for user stories. In addition, UX Poker inspires a discussion about UX, that results in a common understanding of the UX of the product. To interpret the results from the evaluation of a prototype and product increment, we developed or derived the User Experience Questionnaire KPI and Importance-Performance Analysis. In a first study, we were able to successfully apply the two methods and, in combination with established UEQ methods, derive recommendations for action, regarding the improvement of the UX. This would not have been possible without their use. The results of the Delphi study, to validate the UX Lifecycle, reached consensus after two rounds. The results of the evaluation and the comments lead to the conclusion, that the UX Lifecycle has a sufficiently positive effect on UX management. Conclusion. The goal-oriented focus on UX factors and their improvement, as propagated in the UX Lifecycle, are a good way of implementing UX management in a goal-oriented manner. By comparing the results from UX Poker, the evaluation of the prototype, and product increment, the Agile team can learn more about developing a better UX, within a UX retrospective. The UX Lifecycle will have a positive effect on UX management. The use of individual components of the UX Lifecycle, such as UX Poker or Importance-Performance Analysis, already helps an Agile team to improve the user experience. But only in combination with the UX Lifecycle and the individual methods and approaches presented in this PhD thesis, is a management of the user experience in a targeted manner possible, in our view. This was the initial idea of this PhD thesis, which we are convinced we could implement.
... As our previous studies showed, these characteristics have various impli- cations for day-to-day work of practitioners and also integration. In our view, simply applying SPI guidelines does not satisfy the particular needs of UX integration as these guidelines are generic and do not directly address UX or any other specic software quality characteristics for that matter [89]. Hence, this paper aimed to create and propose customization of a number of SPI recommended practices to support UX integration eorts. ...
Full-text available
Background: To be effective, User eXperience (UX) principles and practices need to be integrated into development processes and organizations, what we refer to as UX integration. However, software companies often face various challenges that hinder a successful UX integration. Objective: The aim of this thesis is to facilitate and improve the current state of UX integration in the software industry. To that end, we present an empirical investigation of current UX integration challenges and success factors and analyze them in relation to other software quality characteristics, in particular, usability. Method: We performed a series of studies, mainly in the Swedish software industry and applied a variety of methods including interviews, observations, and workshops. We used Grounded Theory (GT) and thematic analysis to drive our data gathering and to analyze our data. Results: We showed that UX integration challenges and success factors are both technical and organizational, however, they mainly belong to the latter category. We found that various decisions that are made outside the authority of UX practitioners have an inevitable impact on enabling or prohibiting UX integration and that the integration is influenced by various changes that organizations undergo over time as well as planned UX initiatives. Our findings underline the similarities between UX integration and organizational change, in general, and Software Process Improvement (SPI) in particular. We also found that the known unique characteristics of UX (subjective, holistic, dynamic, context-dependent, and worthwhile) have implications not only for the day-to-day work of practitioners but also for UX integration. Based on our findings, we propose various UX integration principles and practices to help software companies in their integration efforts. Conclusion: We argue that to prevent a lopsided focus on the pragmatic aspect of UX in the software industry, software practitioners and researchers should explicitly differentiate between UX and other software quality characteristics, in particular, usability and address the unique characteristics of UX in their work. In addition, they should apply the existing body of knowledge in the two fields of organizational change and SPI especially to address the organizational issues concerning UX integration. Although our focus has been on UX, our findings also may shed light on integrating other multidisciplinary and emerging concepts into the complex context of software organizations.
... For instance, one user may perceive particular software features as simple, novel, and admirable while another may perceive them as complicated and old. Other software quality characteristics can be treated more objectively, at least in theory ( Kashfi et al., 2016). The term subjective has also been used in requirements literature to highlight that since quality characteristics are hard to measure, practitioners tend to judge them based on their personal opinion ( Chung et al., 2000;Paech & Kerlow, 2004). ...
Full-text available
User eXperience (UX) is a key factor in the success of software systems. Many software companies face challenges in their work with UX. Existing research does not analyze UX practices and challenges in relation to other software quality characteristics or, in particular, in relation to usability. A better understanding of these challenges can help researchers and practitioners better address them in the future. In this empirical study, we have interviewed 17 practitioners with different backgrounds and occupations from eight software development companies. Their responses are coded, and analyzed with thematic analysis. We report eight themes of challenges that practitioners face in their work with UX. While some of these challenges partly overlap with those reported in existing literature about usability or other software quality characteristics, the participants of our study either view many of the challenges as unique to UX, or more severe in the case of UX. Although at a superficial level challenges of UX and other quality characteristics overlap, we differentiate these challenges at a deeper level through the five main characteristics of UX: subjective , holistic , dynamic , context-dependent and worthwhile . In particular, we identified that these characteristics have at least 20 implications (i.e. additional difficulties) for day-to-day work of practitioners. We found that 11 of these implications have been previously reported in literature. However, to the best of our knowledge, the remaining nine implications are unique to our study. These implications can explain why practitioners perceive the challenges to be more severe than for other quality characteristics. Most importantly, they can explain the industry’s lopsided focus on the pragmatic aspect of UX. Our findings can be useful for researchers in identifying new and industry-relevant research areas and for practitioners to learn from empirically investigated challenges in UX work, and base their improvement efforts on such knowledge. Identifying and investigating the overlaps underlines the importance of these challenges, and can also help finding research areas not only for enhancing UX work but also software quality in general. It also makes it easier for practitioners to spot, better understand as well as find mitigation strategies for UX, through learning from past experiences and developments in the area of software quality.
Full-text available
Management Information Systems researchers rely on longitudinal case studies to investigate a variety of phenomena such as systems development, system implementation, and information systems-related organizational change. However, insufficient attention has been spent on understanding the unique validity and reliability issues related to the timeline that is either explicitly or implicitly required in a longitudinal case study. In this paper, we address three forms of longitudinal timeline validity: time unit validity (which deals with the question of how to segment the timeline – weeks, months, years, etc.), time boundaries validity (which deals with the question of how long the timeline should be), and time period validity (which deals with the issue of which periods should be in the timeline). We also examine timeline reliability, which deals with the question of whether another judge would have assigned the same events to the same sequence, categories, and periods. Techniques to address these forms of longitudinal timeline validity include: matching the unit of time to the pace of change to address time unit validity, use of member checks and formal case study protocol to address time boundaries validity, analysis of archival data to address both time unit and time boundary validity, and the use of triangulation to address timeline reliability. The techniques should be used to design, conduct, and report longitudinal case studies that contain valid and reliable conclusions.
Conference Paper
Full-text available
A central issue in knowledge management and software process improvement is to learn from experience. In software engineering, most experience is gathered in projects, which makes project experience a prime source for learning. Many companies conduct postmortem reviews, but we have found few companies that analyze the outcome of several reviews to facilitate learning on an organizational level. This paper reports an explorative study of what we can learn from analyzing postmortem review reports of twelve projects in a medium-size software company.
Full-text available
In his In the blink of an eye, Walter Murch, the Oscar-awarded editor of the English Patient, Apocalypse Now, and many other outstanding movies, devises the Rule of Six--six criteria for what makes a good cut. On top of his list is "to be true to the emotion of the moment," a quality more important than advancing the story or being rhythmically interesting. The cut has to deliver a meaningful, compelling, and emotion-rich "experience" to the audience. Because, "what they finally remember is not the editing, not the camerawork, not the performances, not even the story--it's how they felt." Technology for all the right reasons applies this insight to the design of interactive products and technologies--the domain of Human-Computer Interaction, Usability Engineering, and Interaction Design. It takes an experiential approach, putting experience before functionality and leaving behind oversimplified calls for ease, efficiency, and automation or shallow beautification. Instead, it explores what really matters to humans and what it needs to make technology more meaningful. The book clarifies what experience is, and highlights five crucial aspects and their implications for the design of interactive products. It provides reasons why we should bother with an experiential approach, and presents a detailed working model of experience useful for practitioners and academics alike. It closes with the particular challenges of an experiential approach for design. The book presents its view as a comprehensive, yet entertaining blend of scientific findings, design examples, and personal anecdotes.
Full-text available
Abstract,Case study is a suitable research methodology,for software engineering,research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims,at providing,an introduction to case study methodology,and,guidelines for researchers,conducting,case studies and,readers studying,reports of such,studies. The content is based on the authors’ own,experience from conducting,and reading case studies. The terminology,and,guidelines are compiled,from,different methodology,handbooks,in other research domains, in particular social science and information systems, and adapted to the needs,in software,engineering. We,present,recommended,practices for software engineering,case studies as well,as empirically,derived,and,evaluated,checklists for researchers and readers of case study research. Keywords,Casestudy.Research methodology.Checklists .Guidelines
Full-text available
This paper presents a framework that draws on structuration theory and dialectical hermeneutics to explicate the dynamics of software process improvement (SPI) in a packaged software organisation. Adding to the growing body of qualitative research, this approach overcomes some of the criticisms of interpretive studies, especially the need for the research to be reflexive in nature. Our longitudinal analysis of the case study shows SPI to be an emergent rather than a deterministic activity: the design and action of the change process are shown to be intertwined and shaped by their context. This understanding is based upon a structurational perspective that highlights how the unfolding/realisation of the process improvement (intent) are enabled and constrained by their context. The work builds on the recognition that the improvements can be understood from an organisational learning perspective. Fresh insights to the improvement process are developed by recognising the role of the individual to influence the improvement through facilitating or resisting the changes. The understanding gained here can be applied by organisations to enable them to improve the effectiveness of their SPI programmes, and so improve the quality of their software.
Project retrospectives can be powerful tools for project teams to collectively identify communication gaps and practices to improve for future projects. However, even if project members take the time for a retrospective, it can be hard to correctly remember and jointly discuss past events in a constructive way. Fact-based timelines that visualize a project's events offer a possible solution.
This study investigates connections between usability efforts and organizational factors. This is an important field of research which so far appears to be insufficiently studied and discussed. It illustrates problems when working with software engineering tasks and usability requirements. It deals with a large company that manufactures industrial robots with an advanced user interface, which wanted to introduce usability KPIs, to improve product quality. The situation in the company makes this difficult, due to a combination of organizational and behavioural factors that led to a “wicked problem” that caused conflicts, breakdowns and barriers. Addressing these problems requires a holistic view that places context in the foreground and technological solutions in the background. Developing the right product requires communication and collaboration between multiple stakeholders. The inclusion of end users, who fully understand their own work context, is vital. Achieving this is dependent on organizational change, and management commitment. One step to beginning this change process may be through studying ways to introduce user-centred design processes.
To remain competitive, software companies must establish practices that enhance quality and advance process management. To this end, they have increasingly turned to software process improvement (SPI) methodologies, of which the ISO 9000 standards and the capability maturity model (CMM) are the best known. The underlying principle of both methodologies is to assess organizational capabilities to produce quality software, but they depend on different underlying processes.Whether the practices advocated by these methodologies lead to high-quality software has been the topic of ongoing debates. Both scholars and practitioners are looking for hard evidence to justify the time and effort required by such guidelines to improve the software-development process and its end product.In this paper, we investigate the impact of SPI methodologies on software quality, first by theoretical comparison and then with empirical data. Our findings reveal that each methodology has had a different level of impact on software quality factors. These findings could help software-development organizations select the methodology or combination that best meets their quality requirement.
This article presents an experience report where we compare 8 years of experience of product related usability testing and evaluation with principles for software process improvement (SPI). In theory the product and the process views are often seen to be complementary, but studies of industry have demonstrated the opposite. Therefore, more empirical studies are needed to understand and improve the present situation. We find areas of close agreement as well as areas where our work illuminates new characteristics. It has been identified that successful SPI is dependent upon being successfully combined with a business orientation. Usability and business orientation also have strong connections although this has not been extensively addressed in SPI publications. Reasons for this could be that usability focuses on product metrics whilst today's SPI mainly focuses on process metrics. Also because today's SPI is dominated by striving towards a standardized, controllable, and predictable software engineering process; whilst successful usability efforts in organisations are more about creating a creative organisational culture advocating a useful product throughout the development and product life cycle. We provide a study and discussion that supports future development when combining usability and product focus with SPI, in particular if these efforts are related to usability process improvement efforts.