A process oriented user test on public e-Services - The Swedish municipality case

Abstract

This paper elaborates on the potential to design and conduct process-oriented user tests on public e-Services. The idea is to use key constructs of business process orientation (BPO) to develop a basis for designing and communicating the value of user tests in a public e-Service context. Our hypothesis is that user tests can provide valuable results for all actors involved in e-Service development, not only in terms of how to conduct user tests per se, but also to provide incentives for a customer driven focus and highlighting the value derived from user tests. In the municipality case, actors and stakeholders are municipalities involved in an inter-organisational collaboration on e-Service development, i.e. different municipalities, locally and regionally, as well as the supplier. Citizens partaking in a university course conduct the user tests with eye tracking technology. In total 71 users performed in test sessions on 11 different e-Services in a standard platform provided by one supplier. In this paper, we elaborate on a BPO test design based on basic constructs of the approach compared with findings from the reconstruction of the test process design used with an implicit approach 2012 and an explicit BPO approach in 2013. Contributions include a generic test-process design for data collection, analysis and delivery to stakeholders in an overall e-Service development. Furthermore, findings from our test sessions and test process design can inspire and guide other universities to collaborate with practitioners. In addition to the strength of a real case for the students, user test sessions in collaboration can be an eye-opener not only for municipalities but also suppliers. Thus, the likelihood that test results will be applied in the further development of e-Services has increased.

Full text

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
A Process Oriented User Test on Public e-Services – The Swedish Municipality Case
Marie-Therese Christiansson, Malin Wik
Information systems, Karlstad Business School, Faculty of Arts and Social Sciences, Karlstad
University, Sweden
marie-therese.christiansson@kau.se
malin.wik@kau.se
Abstract: This paper elaborates on the potential to design and conduct process-oriented user tests on
public e-Services. The idea is to use key constructs of business process orientation (BPO) to develop
a basis for designing and communicating the value of user tests in a public e-Service context. Our
hypothesis is that user tests can provide valuable results for all actors involved in e-Service
development, not only in terms of how to conduct user tests per se, but also to provide incentives for a
customer driven focus and highlighting the value derived from user tests. In the municipality case,
actors and stakeholders are municipalities involved in an inter-organisational collaboration on e-
Service development, i.e. different municipalities, locally and regionally, as well as the supplier.
Citizens partaking in a university course conduct the user tests with eye tracking technology. In total
71 users performed in test sessions on 11 different e-Services in a standard platform provided by one
supplier. In this paper, we elaborate on a BPO test design based on basic constructs of the approach
compared with findings from the reconstruction of the test process design used with an implicit
approach 2012 and an explicit BPO approach in 2013. Contributions include a generic test-process
design for data collection, analysis and delivery to stakeholders in an overall e-Service development.
Furthermore, findings from our test sessions and test process design can inspire and guide other
universities to collaborate with practitioners. In addition to the strength of a real case for the students,
user test sessions in collaboration can be an eye-opener not only for municipalities but also suppliers.
Thus, the likelihood that test results will be applied in the further development of e-Services has
increased.
Keywords: e-Service evaluation, process orientation, test process design, user tests, eye tracking
1 Introduction
e-Service development has grown to become a daily practice in government; a means for realizing the
digital agendas of the local level as well as national and international levels (Magnusson and
Christiansson 2011). Public e-Services are progressively used as a means for governmental agencies
to interact and exchange information with citizens and businesses. These services are typically web
based and are meant to improve citizens’ interactions with the government, to make governmental
organizations more efficient and effective, and to increase the transparency of government and lead to
a more democratic society (Chourabi et al, 2009). User-centred development is central in national
directives (Ministry of Enterprise, 2011). However, there is little advice on how to implement this in
practice. This paper elaborates on the potential to design and conduct process-oriented user tests on
public e-Services. The idea is to use key constructs in business process orientation (BPO) to develop
a basis for communicating the purpose and value of user tests in the public e-Service context. Our
hypothesis is that user tests can provide a powerful basis for all actors involved in e-Service
development, not only in terms of how to conduct user tests per se, but also to provide incentives for a
customer driven focus and to highlight the value derived from user tests.
User needs and expectations of e-Services require municipalities to provide a solution to meet the
expectations. To this end, the user test is here viewed as a business process through the lens of BPO.
The application of the approach means that business is performed horizontally in and between
organisations. Thus, business is viewed and defined in terms of end-to-end processes. Hence, the
customer needs as trigger to value-added actions across organisational boundaries to a produce a
result with of significant value (Davenport 1993). A business process can be explained as a
relationship where actors from different organisations are working together on complementary
activities for the purpose of achieving mutual benefits and the best possible result (cf. Ford et al,
2010). From this point of view, e-Service development, including test and other issues performed in
collaboration, can be referred to as an inter-organisational process. Thus, organisations with needs
and willingness to collaborate with personal investments, commitments and a joint use of resources
can contribute to a win-win relationship (Alter and Hage 1993). The latter may briefly be described as
a practice with activities performed by the organisations with the best capability to provide the

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
resources (persons, competence, time, technology and information) that are required. Possibilities of
collaboration are important to clarify in all kinds of business performance, and the test process in no
exception.
The Swedish municipality in our case, Karlstad municipality, uses an e-Service portal as one part of
the overall e-Service offerings, totalling 70 e-Services. The supplier, Abou, describes the portal as a
standard system that is configurable with the ability to turn the difficult handling of forms into user-
friendly e-Services integrated with the E-ID and My Account. Solutions can be integrated with back-
end systems, enabling faster and easier processes with functions such as transparency and duplicate
signatures in the same case. Multiple channels for citizen contact with municipalities are provided by,
for example, reception, telephone, email, mobile apps, social media and website forms and e-Services
to initiate case handling processes. The website is the main channel in the municipality case and one
challenge is to increase citizen use of e-Services. Since 2010 the municipality collaborates with all of
the 16 municipalities in the county regarding e-Service development through the e-Service portal. The
population ranges from around 3 700 inhabitants in the smallest municipality to 85 000 inhabitants.
Common solutions, test, implementations, training and maintenance are joint issues and decisions. By
using the same standard portal for e-Services on a local, regional and national basis, the overall
benefits are enhanced through joint analysis, requirement and procurement phases. The collaboration
aims for a better economic and functional result by means of a common technical platform. Moreover,
when it comes to improvements, other municipalities in the portal community may have resolved some
functions and features that another municipality has put on hold, i.e. member municipalities can
benefit from development initiatives driven by other municipalities.
It is beyond the scope of this paper to elaborate on other evaluation tools, test processes and
instruments for analysing e-Service initiatives. The structure of the paper is as follows: Section 2
presents the BPO constructs in a conceptual model used in this paper; Section 3 describes our
research design; Section 4 reports on the user tests in the municipality e-Services conducted 2012
and discusses our findings with implications for the BPO test design in 2013; and Section 5 concludes
the paper with reflections on lessons learned and suggestions for further research.
2 BPO Constructs in a Test Design – User Tests on Public e-Services
Business processes are crossing departmental and organisational boarders as a result of e.g.,
digitalisation in terms of e-commerce, e-Services as well as business process outsourcing (Van Looy
2014). E-Services are usually communicated and supported by employees at the municipal contact
centre. As the first line of contact with the citizens, the municipal advisors direct the user to e-Services
and will guide the user through the service if necessary. Thus, providing e-Services as tasks
performed across administrations implies that beside the IT-service department, the municipal
advisors at the contact centre as well as employees working with the web site are stakeholders in the
test process and the result. To make an impact on the design and the use of test result, stakeholders
are necessary to identify. Lindgren (2013) presents a conceptual framework for identifying,
characterizing and involving public e-Service stakeholders in the development and implementation.
However, in a business process view this work can be narrowed down to determining where the
process starts and end, i.e. defining the scope for a specific purpose. The foundation in the business
process orientation is to adopt a horizontal view of the organisation and business processes through
the value chain (Davenport 1993). This requires a management with a customer-focused mindset and
the ability and willingness to facilitate collaboration in cross-functional process teams (Hammer and
Stanton 1999) across administrations, instead of individual efforts. Since business processes should
contribute to a result with a significant value for the external customer (Österle 1995), value added
business performance needs to serve the customer needs and goals (Neubauer 2009). Further on, in
order to align business processes with the organisation strategy, the vision, strategy and goals must
be translated into the purpose of business processes and goals (metrics) to fit with management
directives together with metrics on business performance, i.e. the activities (Davenport 1993;
Kohlbacher 2010). Thus, strategic alignment is achieved when the employees in an organisation act in
direct relations to fit the intentions of the management. In order to succeed in this direction, business
processes need to be identified, visible, measured and monitored, i.e. the business process
management (BPM) work practice (Rohloff 2009). Thus, BPM requires an integrated approach and
holistic perspective. In addition to IT, core factors in terms of strategic alignment, governance,
methods, people, and culture are highlighted (Rosemann and vom Brocke 2010). Van Looy (2014)
uses the funnel structure to define BPO as the broader concept, which implies business process

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
management (BPM) with distinguished focus on the culture (top management support and rewards)
and structure (horizontal or matrix chart) capability including management of modelling, optimisation
and deployment (implemented and working). However, the basis of this paper is the view of BPO as
the effects (BPM paper) following the means in the process oriented approach. Additionally, the view
on business processes as well as BPM and BPO differs in organisations as well as in research. Thus,
it is necessary to define and clarify the scope. Figure 1 illustrates our scope in terms of the test
process design with a purpose and goal, together with a need for test and the performance required.
Different stakeholders affect the test or are affected by the test result.
!
Figure!1:!The!business!process!scope!–!in!relation!to!test!process!design!
The business should be identified, defined and described as business processes in a modelling to
illustrate the test process design based on the presented constructs below. We have given some
examples from the reconstruction in the municipality case, described in section 3 and 4. In order to
use a BPO approach, intended effects should be explicitly based on core ideas in BPO theory as well
as practice. The main point is to make the basis of performance explicit in order to set indicators and
measurements according to the purpose and goals of different stakeholders interest. Performance
indicators and metrics (e.g., time, cost, accessibility, flexibility) based on the business process
constructs are operationalized in business. Hence, a BPO approach can be used in order to plan and
evaluate quality and performance in a systematic and holistic way with the possibility to work on
horizontal end-to-end processes, across organisations, with continuous improvements. The focus of
usability tests is on whether if the system meets specific usability criteria (Rubin and Chisnell 2008) or
on identifying problems which arise in use (Benyon 2014). The focus and purpose of our test were the
possibilities for the user to find, understand and use the e-Service based on provided information.
Core elements in a business process are identified (cf. e.g., Davenport 1993; Goldkuhl 2005; Österle
1995) and used as constructs in this paper by following the means presented above and in previous
work (Christiansson and Wik 2014; Christiansson 2013), see Table 1:
Table!1:!The!conceptual!model!of!constructs!in!a!business!process!–!in!relation!to!our!case!!
Construct
Case 2012
Case 2013
Evaluation object
Effects
Purpose
To evaluate
usability
To evaluate
communicability
Why is the design
appropriate?
Business with a
result to achieve
Goal
User tests to gain
experiences and
apply theory in
practice
User tests with test
results to improve
e-Services
What is the expected
outcome?
A target value in
relation to
organisational
strategies
Stakeholder
The municipality
The university
Municipalities
The e-Service portal
supplier
The university
Who affects the test of e-
Services? Who are
affected by the test
result?
Perspectives on
business
performance
Organisation
Karlstad
municipality
Municipalities in
Värmland
Who has the
resources/skills and is
Where business
performance takes

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
Karlstad university
Abou
Karlstad university
accountable for results?
place or who
provides resources
Actor/Role
Course manager
Researchers
Test moderator
Test administrator
Observer
User/Student
Course manager
Researchers
Test moderator
Test administrator
Observer
User/Citizens
Who will be involved in
the performance?
A person in a
professional role, an
organisation, or a
system/machine?
Someone who
initiate and perform
business process
activities
Customer
Students
Karlstad
municipality
Municipalities
e-Service portal
supplier
Students
Who is the external
reason for performing
business and who
benefits from results in
their internal use?
Someone who has
needs,
expectations and
goals
Trigger
A test assignment
Requests for test
Who or what initiates the
business?
Customer needs,
regulations,
directives
Activity
To identify, prepare,
conduct, analyse
and report user
tests with students
To identify, prepare,
conduct, analyse
and report user
tests with citizens in
the municipality
What action is value
added?
A desired way of
working to reach a
specific result
Flow
Way of performing
actions and take
decisions
Way of performing
actions and take
decisions
What business logic will
be performed?
A structure of
activities and
information
required for
performance
Resource
The municipality
home page/e-
Service portal/ test
environment
Eye tracker, Web
cam, Recorder
The municipality
home page/e-
Service portal/test
environment
Eye tracker, Web
cam, Recorder
Survey & Report
tool, Wireframes
What is necessary to
support the activity in
order to be able to
perform?
Instruments,
information,
knowledge, time,
machines etc.
needed to support
the performance
Input
Preconditions
Test instructions
Template/protocol
Gaze replay
Lessons learned
Preconditions
Test instructions
Template/log notes
Scenario
Role description
Gaze replay
What is necessary to be
able to perform the
activity?
A basis for/a
prerequisite for
conducting a
process activity
Output
Gaze replay
Heat map
Gaze plot
Analysis protocol
Gaze replay
Heat map
Gaze plot
What is the result of
performed activity, what
is necessary as input in
the next one?
Partial result from
a performed
process activity
Result
Test result report
Test result report
based on
wireframes
What is delivered to the
customer?
The purpose of the
process,
something that is
produced and
offered with a
significant value
3 Research Design in the Municipality Case
The Municipality Case is described and analysed based on the conceptual model of the BPO
construct presented above. A reconstruction of our test design of 2012 and 2013 in the university
undergraduate course will serve as the empirical basis of this paper. One course assignment is to
conduct a user test in a laboratory specifically designed for conducting user and usability tests. For the
students, the user test assignment is a possibility to conduct a real case and to apply theory to

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
practice. The lab consists of three rooms: the reception room where users are greeted and on
occasions are interviewed at a pre- or post-stage of the test; the test room where the eye tracker is
situated and the users perform their task and the control room where various stakeholders can
observe the test sessions through a mirror wall and listen to the audio output. At the time of the 2013
testing, students in different test roles and two users (citizens) each conducted a test for 30 minutes.
The following sections present our methodological routine for the user test data collection and analysis
to give a background to our test process design.
3.1 Data collection
In the municipality case, we used the Tobii technology 1750 eye tracker (Tobii 2014) as a data
collection tool to capture and record eye movements as well as the real time dialogue between the
user, test monitor, administrator and observers. Gaze data were collected at a 50 Hz sampling
frequency. The recording and analysis software used was Tobii Studio 2.8, running on Windows XP.
Additionally, audio and video of the test participants were recorded with a Logitech webcam.
Elements in the communicability concept (Christiansson 2013) were used when developing a test
protocol for observations and a template for the students’ test report, to improve the transfer of
findings from the test process result to the municipality. A complementary technique of data collections
used by the researchers was a pre-interview with the users to collect user expectations of an e-
Service per se, handling time and expected results (when and how such result should be delivered). In
test sessions, we asked and encouraged the users to “think aloud” meaning that verbalized their
thoughts, actions, confusions and frustrations (Rubin and Chisnell 2008). The think-aloud technique is
associated with some disadvantages as the user can find it as unnatural and obtrusive to constantly
think out loud (Rubin and Chisnell 2008) and it may affect the interaction and scan paths of the user
(Pernice and Nielsen 2009). Nonetheless, by combining gaze replay with the think-out-loud technique
we were able to see exactly what the user saw, acted upon and says which help in understanding why
users have problems finding e-Services, performing and completing their task(s). In analysis, the eye
tracking data can be visualised in heat maps (still images that show user attention, i.e. where eyes are
focused in terms of length and time); gaze plots (still images that shows where users fix their eyes in
terms of order); and gaze replay, which is a recording of the screen and the user's eye movements
visualise each fixation and action over time. See Figure 2.
Figure!2.!A!heat!map!and!a!gaze!plot!=!shows!one!user!searching!for!the!e=Service!=!Apply!for!Direct!Debit.!
Reasons for not using gaze plots and heat maps in our analysis were that, gaze plots constructed
from lengthy eye-movements recordings (such as ours) are easily overplotted, and heat maps do not
show the order of the user’s fixations (see e.g., Cöltekin et al, 2010; Andrienko et al, 2012).
Furthermore, these static visualizations do not take dynamic elements, which are common on the e-
Service platform, into account (Pernice and Nielsen 2009). This means that if the user opened a
popup window, the static visualization will be displayed as if the user has studied the web site behind
the popup dialogue. Such circumstances can be detected by studying the gaze replay, allowing
affected recordings and/or heat maps and gaze plots to be excluded, but would be a tedious task.
Another option of analysis is to manually draw areas of interest (AOI) on the gaze recordings.
However, this is also a time-consuming task and as we are not conducting a quantitative analysis,
AOIs would not elicit the information we are interested in. Instead, we defined AOIs separately by

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
using wireframes. Wireframes are a commonly used method when outlining the structure of the
content on a website, without focusing on details of the design (Benyon 2014). We used the approach
to extract data to a web survey used in parallel with the gaze replay.
3.2 Gaze replay analysis
Using the eye-tracking technique result in large amount of data to handle. Extracting results and
interpreting the eye tracking data are labor-intensive as well as difficult (Jacob and Karn 2003). To our
best knowledge most eye-tracking studies focus on quantitative measures and analysis, such as
number of fixation per AOI (Poole and Ball 2006) or analysis of static stimuli (Kurzhals and Weiskopf
2013), instead of analysis based on viewing the eye-movement recordings. In studies reporting on
gaze replays, eye-movements recordings have been used in combination with “retrospective think
aloud” (Mazman and Altun 2012; Kostonos et al, 2009). In both studies the test participants used
retrospective think aloud while viewing the recording with eye-movement overlays, to self-assess
performances. Kurzhals and Weiskopf (2013) present means of analysis of eye-tracking data on
dynamic stimuli, e.g., videos. Through this method, multiple recordings of multiple users can be
visualized and summarized. However, in the municipality case it is important to study one user at the
time, because of our context of e-Service use. Besides, our brief search for support in analysing gaze
replay data shows that this is an emerging area of research and we had to work with an inductive
approach. We decided to use the gaze replay technique because it does provide valuable data, such
as how efficiently a user searches for an element and indications of a user’s difficulty in extracting
information from an element and the importance of the element (Jacob and Karn 2003). Eye-
movement analysis is appropriate as it affords seeing what the users actually see, do, react on and act
upon, instead of relying only on what the users say they have done, seen and reacted on. However, it
is not possible to draw conclusions from the users' understanding of what they have seen or not seen.
The recorded user comments and insights, gazes and search patterns, failed actions, action modes
(status in errands) and problems occurred in finding, understanding and using the e-Service can,
however, be observed and extracted from the recordings. To be able to draw correct conclusions on
usability, gaze recordings from at least six users need to be included in the analysis (Pernice and
Nielsen 2009).
Heat maps and gaze plots were not used to draw conclusions, only to visualize results to different
stakeholders (e.g. the municipality, the e-Service platform supplier and our students). In the test
process design of 2012, log notes with empirical data from the visualisations and recorded voices from
the user gaze replay were captured and structured by each of us (one researcher and one master
student) based on our two background references, i.e. a human-computer-interaction lens and a
communicability lens. In a second run we merged our observations in an analysis protocol. The
protocol was further used by the researcher in analysing and structuring our findings into the
characteristics of communicability (Christiansson 2013). Altogether, this way of working was very time
consuming. Therefore, the challenge in the test process design of 2013 was to accomplish a more
effective handling of the extensive data results. We developed a web-based survey in the tool Survey
& Report to help us collect and structure log notes and, at the same time analyse the material faster.
In this work we had to reconstruct our analysis (which steps and in what order according to the gaze
replay) to develop a useful template as a basis for the survey. The survey used in our gaze-replay
analysis was designed in an iterative manner. To get more usable log notes, the first survey included a
template on how to collect observations notes of the e-Service communicability (our purpose and test
instructions) with guideline questions. However, when trying out the survey during a session of gaze-
replay analysis, we found that the survey did not correspond to what we wanted to extract from the
recordings, and in what order we wanted to elicit the information. Furthermore, we noticed that the
focal e-Services could be accessed from different levels of the platform, and by using different
elements. Extracting data from the gaze recordings therefore demanded a shared way of naming the
elements and levels on the e-Service platform.
By working with wireframes in the analysis, we discovered the multiple layers in a web-based e-
Service resulting in five wireframes. Wireframes for each level of the e-Service were therefore
constructed, and the survey was re-designed to correspond to the wireframes. Level 1 representing
the municipality home page, level 2 the e-Services start page, level 3 the focal area of e-Services,
level 4 the focal e-Service and level 5 the e-Service. The areas marked in the wireframes correspond
to where we wanted to capture user gaze and/or user’s actions, in other terms AOIs. However, as the
result of the test process for the municipality was to increase the ability to communicate their e-
Services as providers, it is important to be able to communicate our findings regarding the content,

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
placement, user interpretations, and so on, to the communicator officers in the different administra-
tions in a common and easy way. Thus, we used the wireframes to be able to visualize our comments
on where the usage problem occurred, where information was missing, what areas the users
neglected etc. in the test result report. For two of the wireframes we used in the municipality case, see
Figure 3.
!
Figure!3:!Wireframes!of!the!e=Services!start!page!(level!2)!and!the!focal!area!of!e=Services!page!(level!3)!
The left wireframe shows the structure of the e-Services start page and the right wireframe shows the
structure of the focal area e-Services page. Areas in the frame representing the municipality website
link (1), the municipality logo (2), the search area (3), the global navigation bar (4), the left menu/main
categories (5), the contextual content and (6). Further on, depending of e-Service level, the number
corresponds to e.g., drop-down menus: e-Service categories, right menu/shortcuts, the e-Service
name, information icon, link to the e-Service, link to a .pdf form, information in text right
menu/shortcuts and information in text.
4. User Test Design – The Swedish Municipality Case
Based on the motives and arguments according to the methodological considerations presented in
section 3, this section compares the two different test designs from 2012 and 2013 with a focus on a
test process design with an implicit as well as explicit BPO approach as well as identifying
improvements still to be made to achieve value and performance in an efficient manner.
The purpose of the test process design in 2012 was to conduct a usability test as a case in the
university course. Thus the student was the customer with an experience of testing and applied theory
as a result. By using the eye tracker as a tool for data collection, we deemed that the study of user
experience of e-Service use would yield an idea of the concept of communicability. The test sessions
were conducted with two different tasks (test 1 and test 2). The roles involved in the test were the
course manager (CM), test administrator (TA), test moderator (TM) and observer (O) as well as the
two researchers (R).
The established and well working co-operation with Karlstad municipality was a pre-condition for
getting access to the test environment in the e-Service platform (a standard portal) provided by one
supplier. To prepare the user test, 28 different e-Services in the standard portal were provided in the
test environment with fake E-ID together with the course assignment. Selected e-Services used in the
test were: Apply for Direct Debit; Parking permission for “green cars”; Composting food waste;
Drawing archive; Sign up for food supply business; Food poisoning and Booking civil marriage. The
idea of this selection was that our users could relate to the service provided.
In test 1 the user was asked to find one (without using the search function) of the selected e-Service
from the e-Service start-page (the test environment), use it and determine case status and expected
turnaround time (case handling time). In test 2 the task was to navigate from the municipality home
page to find the requested e-Service and to describe its purpose and expected turnaround time.

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
Overall 31 test sessions were conducted and analysed to understand the concept of communicability
in e-Service use for research and practical purposes. The practical implications for the municipality
however, was limited to a demo of gaze replay from some test sessions, the heat maps and gaze plots
for illustrating and communicating our findings in the municipality. For a more comprehensive report
on lessons learned, confer Christiansson (2013) and Christiansson and Wik (2014).
By using the BPO approach in the 2013 test process design we shifted attention to value added
actions in relation to different stakeholders and the customer. Three different external customers are
identified as the municipalities, the e-Service portal supplier and our students in the university course.
Thus, “two tracks” in the test process are interesting to design in terms of performance and result with
a significant value. In our case, the municipality requested some more hands-on recommendations on
how to communicate their e-Services. Thus, the purpose was changed from testing usability to testing
communicability (including metrics from e.g. usability) and a focus on test result report with guidelines
to where information actions should be taken (based on wireframes). Moreover, new test data
collection could be the basis of further development, a possibility that increased with the supplier
observing the user test. During test sessions in 2013, see Figure 4, the user was directed to a
scenario, to interpret the task and to find an e-Service to handle the errand from the e-Service start-
page (the test environment) and use it. As the users are part of the “Google generation” and some
municipalities expressed their interest in using a more powerful search engine in their e-Service
delivery, we let the user describe intended keywords to use in order to deliver more value in the test
report.
Figure!4:!The!test!process!design!in!the!municipality!case!2013!
5. Conclusions: Lessons Learned
The purpose of this paper is to develop a model for communicating the purpose and value of user
tests in the public e-Service context. Besides contributing a user test case employing the eye tracking
technology and the applied BPO in a test process design, our findings have several implications for
organisations and test managers. In general, principles of a generic test process design are presented
in terms of BPO key constructs presented in Table 1 and the generic test process in Figure 5, as
guidelines in practice as well as to be used in further research for repeated studies and test process
improvements.

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
!
!
!
!
!
!
!
!
!
!
Figure!5:!A!generic!test!process!design!based!on!key!constructs!in!BPO!
This paper elaborates on the potential to design and conduct process-oriented user tests on public e-
Services. The idea is to use key constructs in business process orientation (BPO) to develop a basis
for communicating the purpose and value of user tests in the public e-Service context. A process
oriented user test on public e-Services can provide a powerful basis for all actors involved, not only in
terms of conducting user tests per se, but also to provide incentives for a customer driven focus and to
highlight the value gained from user tests results. By using stakeholders as co-producers in the test
process (e.g. the supplier to act as observer), the learning and customer insights will significantly
increase. With a business process perspective, the end-to-end process will be viewed across
organisations and make use of win-win situations.
The lessons learned from a process-oriented test design is that it provides value added potential
because the customer/stakeholder can view test results in the context of their goals in organisations.
Further on, by collecting user expectations, values in the e-Service per se can be defined as well as a
acting as basis for comparing test results. In addition, the customer process can be mapped and
discussed in terms of the e-Service value in use, in order to gather a more comprehensive view.
We believe in visualisations for communicating ‘what’ to do and ‘why’ when using the generic test
process design as a basic pattern to describe, explain, discuss and adjust. The test process
description can provide a powerful basis for all actors involved in e-Service development, not only in
terms of how to conduct user tests per se, but also to provide incentives for a customer driven focus
and to highlight the value for different stakeholders gained from user tests. In addition, all stakeholders
and actors involved in the inter-organisational evaluation are visualised with important “hand-shakes”
to make sure that the test assignment is prioritized and that test results are implemented by “whom it
may concern” from the e-Service user perspective. With a BPO approach it is easier to design an
appropriate test process by visualising the performance based on the customer perspective and from
the different stakeholder views to identify opportunities to reach win-win situations. Each action in the
inter-organisational process should add value, which means that no time is spent on activities or
documentations that are not relevant to the internal customer or the external customer. Hence, the
stakeholder analysis is crucial in defining the actors who are affected or should participate.
References
Alter, C. and Hage, J. (1993) Organizations working together, Sage, Newbury Park, Calif.
Andrienko, G. Andrienko, N. Burch, M. and Weiskopf, D. (2012) Visual analytics methodology for eye movement
studies, Visualization and Computer Graphics IEEE Transactions on, Vol 18, No.12, pp. 2889-2898.
Benyon, D. (2014) Designing interactive systems: A comprehensive guide to HCI, UX and interaction design, 3rd
Edition, Harlow: Pearson Education Limited.

In: Proceedings of the 8th European Conference on IS Management and Evaluation, ECIME
2014, pp. 39-48
Chourabi, H. Mellouli, S. and Bouslama, F. (2009) Modeling e-government business processes: New approaches
to transparent and efficient performance, Information Polity, Vol 14, pp. 91-109.
Christiansson, M-T. (2013) Improving Citizens’ Ability to Find, Understand and Use e-Services: Communicating
the Social Interaction Dimension, Systems, Signs & Actions Vol 7, pp. 177-204.
Christiansson, M-T. and Wik, M. (2014) Testing Communicability in Public e-Services – Process and Outcomes,
In: Proceedings of EGOV2014 (forthcoming).
Çöltekin, A. Fabrikant, S. I. and Lacayo, M. (2010) Exploring the efficiency of users' visual analytics strategies
based on sequence analysis of eye movement recordings, International Journal of Geographical Information
Science, Vol 24, No. 10, pp. 1559-1575.
Ford, D. Gadde, L.-E. Håkansson, H. and Waluszewski, A. (2010) Analysing Business Interaction. IMP Journal,
Vol 4, No. 1, pp. 82-103.
Goldkuhl, G. (2005) Workpractice Theory – What it is and Why we need it, In: Proceedings of ALOIS, Limerick,
Ireland, pp. 1-6.
Hammer, M. and Stanton, S. (1999) How Process Enterprises Really Work, Harvard Business Review, Vol 77,
No. 6, pp. 108-118.
Jacob R.J.K and Karn K.S. (2003) Eye Tracking in Human-Computer Interaction and Usability Research: Ready
to Deliver the Promises, In: Hyona, Radac, Deubel (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye
Movement Research, Elsevier Science, pp 573- 605.
Kohlbacher, M. (2010) The effects of process orientation: a literature review. Business Process Management
Journal, Vol 16, No. 1, pp. 135-152.
Kurzhals, K. and Weiskopf, D. (2013) Space-Time Visual Analytics of Eye-Tracking Data for Dynamic Stimuli,
Visualization and Computer Graphics IEEE Transactions on, Vol 19, No. 12, pp. 2129-2138.
Lindgren, I. (2013) Public e-service stakeholders - A study on who matters for public e-service development and
implementation, PhD thesis No. 580, Linköping University.
Magnusson, M. and Christiansson, M-T. (2011) Using Goal Modelling to Evaluate Goals for e-Service
Development in Government, In: Proceedings of the 5th ECIME, pp. 312-320.
Mazman, S. and Altun, A. (2012) Modeling Cognitive Strategies during Complex Task Performing Process,
Turkish Online Journal of Qualitative Inquiry, Vol 3, No. 4, pp. 1-27.
Ministry of Enterprise (2011) ICT for Everyone - A Digital Agenda for Sweden, N2011.19, Accessible:
http://www.government.se/sb/d/2025/a/181914 [2014-01-21].
Neubauer, T. (2009) An empirical study about the status of business process management, Business Process
Management Journal, Vol 15, No. 2, pp. 166-183.
Pernice, K. and Nielsen, J. (2009) Eyetracking Methodology: How to Conduct and Evaluate Usability Studies
Using Eyetracking, Accessible: http://www.useit.com/eyetracking/methodology [2014-01-21].
Poole, A. and Ball, L.J. (2006) Eye tracking in HCI and usability research, In: Ghaoui, C. (ed.), Encyclopaedia of
human-computer interaction, Idea Group Inc., Pennsylvania, pp. 211-219.
Rosemann, M. and vom Brocke, J. (2010) The six core elements of business process management, In: vom
Brocke J and Rosemann M (Eds.) Handbook on Business Process Management: Introduction, Methods and
Information Systems, (1), Springer, Berlin, pp. 109-24.
Rubin, J. and Chisnell, D. (2008) Handbook of usability testing: how to plan, design, and conduct effective tests,
Indianapolis, Wiley Pub.
Tobii (2014) Eye tracking studies - Usability and HCI. Accessible: http://www.tobii.com/en/eye-
tracking-research/global/library/customer-cases/usability-hci/#.UtlAL_RDs-X [2014-01-17].
Van Looy, A. (2014) Business Process Maturity – A Comparative Study on a Sample of Business Process
Maturity Models, Springer-Verlag.
Österle, H. (1995) Business in the Information Age – Heading for New Processes, Springer-Verlag, Berlin.