ArticlePDF Available

A Comparative Study of Electronic Performance Support Systems


Abstract and Figures

Electronic performance support systems (EPSS) deliver relevant support information to users while they are performing tasks. The present study examined the effect of different types of EPSS on user performance, attitudes, system use and time on task. Employees at a manufacturing company were asked to complete a procedural software task and received support from either an intrinsic, extrinsic, external performance support system or no system at all. Results revealed significant differences on performance, attitudes and use between several treatment groups. The study suggests that providing any kind of EPSS to support task performance is better than having none at all. In addition, designers can improve user performance, attitudes and use by creating systems that integrate with the primary work interface.
Content may be subject to copyright.
Volume 18, Number 4/2005 71
Performance Improvement Quarterly, 18(4) pp. 0-00
A Comparative Study of Electronic Performance
Support Systems
Frank Nguyen
James D. Klein
Howard Sullivan
Arizona State University
Electronic performance support
systems (EPSS) deliver relevant sup-
port information to users while they are
performing tasks. The present study
examined the effect of different types of
EPSS on user performance, attitudes,
system use and time on task. Employ-
ees at a manufacturing company were
asked to complete a procedural soft-
ware task and received support from
either an intrinsic, extrinsic, external
performance support system or no sys-
tem at all. Results revealed significant
differences on performance, attitudes
and use between several treatment
groups. The study suggests that provid-
ing any kind of EPSS to support task
performance is better than having none
at all. In addition, designers can im-
prove user performance, attitudes and
use by creating systems that integrate
with the primary work interface.
Gloria Gery (1991) introduced the
concept of electronic performance sup-
port systems (EPSS) as a method to
enable human performance through
just-in-time learning and task sup-
port. She asserted that through EPSS
one could generate “day-one perfor-
mance…for novice performers” (Gery,
1995, p. 47). Since its introduction,
Gery has reported that the principles
of EPSS have been applied to a wide
range of work interfaces from appli-
cations that automate tax prepara-
tion to financial planning Web sites
to tools that help consumers plan and
purchase travel (Gery, 2003).
To guide practitioners, authors
and experts have shared their in-
sights on a wide range of EPSS topics
over the past decade. For instance, a
number of experts have focused on
the unique methods and processes
required to develop EPSS. Raybould
(2000) introduced a performance
support mapping methodology. This
approach combines elements from
disparate fields including business
process reengineering, human per-
formance technology and instruc-
tional design into a new field dubbed
performance support engineering.
Huber, Lippincott, McMahon, and
Witt (1999) provided a framework
for the skills, competencies and job
roles that make up an effective EPSS
development team.
Other authors have focused on
the value of performance support.
Chase (1998) asserted that EPSS
could reduce the time and cost
associated with training new em-
ployees. Altalib (2002) provided a
detailed process on how to measure
the return on investment (ROI) for
72 PerformaNce ImProVemeNt Quarterly
EPSS. Based largely on the work of
Davidson (1998), Phillips (1997), and
Hawkins, Gustafson, and Nielson
(1998), Altalib’s comprehensive ROI
approach examined potential ben-
efits derived from EPSS including
hard measures such as increased
sales or manufacturing production
to soft measures such as employee
Some have offered strategic views
on how EPSS relates to and comple-
ments other information interven-
tions such as knowledge manage-
ment and training. Rosenberg (1995)
argued that trainers and others in-
volved in instructional design should
shift to the more holistic views of hu-
man performance technology (HPT)
which embraces EPSS and training
among other types of interventions.
With the advent of HPT, Sherry and
Wilson (1999) predicted the con-
vergence of the traditional roles of
instructional designer, performance
support designer and information
Interestingly, very few of the cur-
rent ideas and principles related to
EPSS are based on any substantive
research. In fact, although EPSS
has been discussed for almost two
decades, very few studies have been
conducted to measure its effective-
ness. One of the few research stud-
ies that examined EPSS compared
the effectiveness of computer-based
and print-based performance aids
in the Army (Morrison & Witmer,
1983). While this study found no
significant differences between the
two delivery media, it did not address
more fundamental issues such as the
overall effectiveness of EPSS or the
specific EPSS designs that may be
better under differing performance
To this end, Gery introduced a
conceptual framework for EPSS that
illustrates key design differences be-
tween potential performance support
systems. She asserted that there are
three fundamental types of EPSS:
external, extrinsic and intrinsic sup-
port (Gery, 1995).
External systems store content
used to support task performance in
an external database. This content is
not integrated within a user’s work
interface. As a result, users are forced
to manually locate relevant informa-
tion in the external EPSS. Common
examples of external performance
support systems include search
engines, frequently asked question
pages, and help indexes. In addition,
external performance support “may
or may not be computer mediated”
(Gery, 1995, p. 53). Job aids or docu-
mentation are common external per-
formance support interventions.
Extrinsic “[p]erformance sup-
port…is integrated with the system,
but is not in the primary workspace”
(Gery, 1995, p. 51). In other words,
extrinsic systems integrate with the
user’s work interface in such a way
that the EPSS can identify the user’s
location in a system or even the exact
task that they may be working on.
With this contextual information,
the extrinsic system can intelligently
locate content that may be relevant
to support the task at hand. Like ex-
ternal performance support systems,
though, the content used to support
a task is external to the work inter-
Intrinsic systems provide users
with task support that is incorporated
directly within their work interface.
Due to this direct integration with
the interface, Gery asserted that in-
trinsic EPSS provides “[p]erformance
Volume 18, Number 4/2005 73
support that is inherent to the system
itself. It’s so well integrated that, to
workers, it’s part of the system” (Gery,
1995, p. 51). Under this rather broad
definition, examples of intrinsic per-
formance support systems can range
from tools that automate tasks and
processes, user-centered design of
work interfaces to reduce complexity
and improve usability, or embedded
knowledge that is displayed directly
in the work interface.
With this dis-
tinction between
external, extrin-
sic and intrinsic
EPSS, Gery (1995)
provided designers
with a guideline
to implement 80%
of their support
systems as intrin-
sic, 10% extrinsic,
and the remain-
ing 10% external.
She argued that
this guideline opti-
mized designer and performer time,
decreased performance development
overhead, and still accomplished the
desired impact: user performance.
However, the impact of the three
types of EPSS has not been empiri-
cally tested. Thus, performance tech-
nologists lack validated principles to
guide them in the selection, design
and development of EPSS. To address
this gap, this article summarizes a
study that focused on four key re-
search questions:
1. Of the three types of EPSS
(external, extrinsic, or intrinsic),
which one is better at enabling user
2. Which type of EPSS do users
3. Which type of EPSS do they use
more often?
4. Which one minimizes the time
it takes for a user to complete a given
Seventy-two employees from a
semiconductor manufacturing com-
pany participated in the study. The
employees were recommended by
their direct man-
agers or identified
by other partici-
pants. All partici-
pants involved in
the study had com-
pleted at least four
years of college:
forty-six obtained
a bachelor’s and
twenty-six ob-
tained a master’s
degree in various
fi eld s. Pot entia l
participants were
screened for prior knowledge of the
corporation’s learning management
system used in the study; any indi-
viduals with previous experience us-
ing the system, which served as the
basis for the task, were not selected
to participate in the study.
The participants represented a di-
verse range of job roles: twenty-eight
software engineers, fourteen train-
ing professionals, twelve managers,
five business analysts, five human
resource professionals, three accoun-
tants, two financial analysts, one de-
sign engineer and one customer sup-
port specialist. The participants were
distributed across four western states:
forty-seven were located in Arizona,
nine in Oregon, eight in Northern
California, and seven in Utah.
lack validated
principles to
guide them in the
selection, design
and development
of EPSS.
74 PerformaNce ImProVemeNt Quarterly
Software application. One portion
of the company’s learning manage-
ment system was adapted for use in
the study. As part of the process to
create an online training course, em-
ployees are required to submit data
that describes their training course.
Such metadata typically includes the
course’s name, description, objectives
and other relevant data. The course
registration software that is normal-
ly used to submit this metadata was
extracted from the learning manage-
ment system and tied to an isolated
database designed specifically for
the study.
As illustrated in Figure 1, the
course registration software screens
included a series of open text fields
that required the user to input rel-
evant data as well as menus that
required the user to select from a
number of pre-defined choices. In
total, the course registration module
required twelve user inputs or selec-
tions. Data entered into the course
registration software were stored in
a database for analysis.
Performance support systems.
The course registration software
was modified to include three dif-
ferent types of performance support
systems. The three treatments were
Figure 1. Software Application.
Volume 18, Number 4/2005 75
based on EPSS categories established
by Gery (1995): including external,
extrinsic and intrinsic support.
External performance support
system. The external system imple-
mented in this study was a search
engine. When users in this treat-
ment clicked a help button located in
the menu of the course registration
software, their request was recorded
in a database and a popup window
opened that prompted them to enter
a keyword as shown in Figure 2. Once
they submitted their keyword, the
external support system searched
through the EPSS content repository
and presented the user with a choice
of help topics based on their query.
Participants then had to select the
appropriate topic to read.
Extrinsic performance support
system. The extrinsic system was a
context-sensitive help system. Help
buttons in the form of a question
mark were inserted throughout the
software application. When users
clicked on the buttons, their request
was recorded in a database and a new
window opened displaying support
information associated with the task
as illustrated in Figure 3.
Intrinsic performance support
system. The intrinsic performance
support was an information-based
system that provided the users with
task-relevant text instructions direct-
ly in the course registration software
screens as shown in Figure 4. In order
to capture the number of times partici-
pants used the intrinsic performance
Figure 2. External Performance Support System.
76 PerformaNce ImProVemeNt Quarterly
support system, help buttons were
inserted throughout the software.
When users clicked the buttons, their
request was recorded in a database
and the information associated with
the support instance was displayed
adjacent to the button. While this
approach deviated somewhat from
typical instances of intrinsic perfor-
mance support where information
may be displayed automatically, the
researchers felt that it was an impor-
tant modification that facilitated the
comparison of EPSS use across the
three treatments.
In addition to the three perfor-
mance support types, the system
was also modified to display no per-
formance support at all to facilitate
a control group. The no performance
support system is illustrated in the
software application screen shown
earlier in Figure 1.
The content across the three per-
formance support systems was identi-
cal and differed only in the manner in
which it was accessed and presented.
When the user accessed the course
registration software, the system ran-
domly assigned the user to one of the
four performance support treatments
described above. As the software was
loaded onto the participant’s comput-
er, the system automatically changed
the interface to show or hide the
appropriate buttons and on-screen
information based on the participant’s
treatment group assignment.
Task scenario. The task scenario
portrayed a realistic issue that a new
Figure 3. Extrinsic Performance Support System.
Volume 18, Number 4/2005 77
employee might face. The scenario
included information that a training
manager might provide to an employ-
ee when registering a new course in
the corporate learning management
system. The following is an excerpt of
the task scenario text:
Imagine that you have been
recently hired as a training adminis-
trator. The message below contains
information that your new manager
emailed you for an online training
course that needs to be published.
Thanks for taking care of this
task for me. Here’s how I’d like to
have the course set up:
q The name of the course is Pro-
gram Life Cycle.
q It is an online learning course
that provides an overview of the eBG
Program Life Cycle, or PLC.
q I would like for this course to
appear in the course catalog under
a category called Project Manage-
q I’ve already saved the course to
the web server, and it is located in a
folder called PLC.
q We’re phasing out the class-
room-based course. This will be an
online, web-based course.
q The course’s test contains 10
questions, and I’d like students
to pass with 90% before they get
q I’d like to have students take
a survey right after they finish the
course. We don’t need to do the fol-
low-up survey that is sent out six
weeks later.
Figure 4. Intrinsic Performance Support System.
78 PerformaNce ImProVemeNt Quarterly
q Go ahead and set up the course
so that students can see it in the
catalog immediately.
q I’d also like to have this course
set up so that the system replicates
it to the other geographies. It looks
like we will have students in China,
Malaysia, Ireland, Israel and the
United States.
Criterion Measures
Four criterion measures were
used in the study: user performance
on the task, user attitude surveys, use
of EPSS, and time on task.
Performance. User performance on
the task was measured by evaluating
the number of correct items the par-
ticipants submitted to the software
application. As mentioned earlier, the
course registration module required
twelve user inputs or selections. Data
entered by the user into the course
registration software was stored in
a database and subsequently evalu-
ated by the lead researcher. Partici-
pants received one point for each cor-
rect input with a maximum of twelve
points possible.
User attitude surveys. A four-item
survey was developed to measure
participant attitudes towards the re-
spective performance support system
provided in the task. Respondents
used a 5-point Likert scale (5=strong-
ly agree, 1=strongly disagree) to rate
their attitudes regarding the effec-
tiveness of the performance support
Use of EPSS. Various en route data
were recorded to measure the number
of access to the performance support
system. When participants in the
external performance support treat-
ment clicked the Help button located
on the software application’s naviga-
tion menu, the participant’s name
and time of access were recorded. In
addition, the performance support
content that was subsequently ac-
cessed by the participant was noted.
Each page of content that was opened
by the participant was considered one
access to the EPSS. Similarly, when
participants in both the extrinsic and
intrinsic performance support treat-
ment clicked the question mark (?)
icon embedded within the software
application, the participant’s name,
time of access and location in the ap-
plication were recorded. Each click on
the question mark icon was consid-
ered one access to the EPSS. Since the
control group was not provided with
a performance support system, no ac-
cesses were recorded for participants
in this group.
Time-on-task. The total amount of
time participants spent completing
the task in the study was measured
by calculating the difference between
the time at which participants logged
into and out of the software applica-
Since the participants in the study
were geographically dispersed, the
researchers arranged approximately
twenty data collection sessions at
various sites over the course of two
weeks. Small groups of 3-5 partici-
pants were directed to prearranged
conference rooms to ensure that they
were not distracted by phone calls, e-
mail or co-workers while completing
the study. The lead researcher gave
participants the task scenario and
instructed them to read it until they
were comfortable with the task. Once
all participants finished reading the
task scenario, the lead researcher
instructed them to complete the task
using only the information provided
by the task scenario and any help
Volume 18, Number 4/2005 79
that may be provided by the soft-
ware application. In addition, the
researcher instructed participants to
log out of the software application as
soon as they felt they had completed
the task.
The participants were then given
the location of the software ap-
plication on the corporate network
and logged in using an automated
authentication system pre-installed
on all corporate-issued computers.
When users accessed the software
application, the system randomly
assigned them into one of four treat-
ment groups (intrinsic, extrinsic,
external or no EPSS) and displayed
the appropriate performance support
system. Participants were not aware
that they had been assigned to a dif-
ferent treatment group or that their
system was configured with a differ-
ent EPSS. The opening screen of the
software application provided a brief
set of instructions demonstrating
how to access the support system.
Participants individually worked
through the task using the software
program and performance support
system. Once the participants com-
pleted the task and logged out of
the system, they were automatically
directed to and completed the user at-
titude survey. One participant in the
control group declined to complete
the survey.
To ensure participant motivation,
the researchers worked with a subset
of managers within the company to
identify all employees in their organi-
zations with no prior experience with
the corporate learning management
system. These managers encouraged
their employees to participate in the
study. In addition, refreshments were
offered to participants upon comple-
tion of the task, and a letter of recog-
nition was sent to the manager and
participant after the study.
Design and Data Analysis
This study used a posttest-only
control-group design. One-way anal-
ysis of variance (ANOVA) was con-
ducted on participants’ performance
on the task, use of EPSS and time on
task. One-way multivariate analysis
of variance (MANOVA) was conduct-
ed on the data from the attitude sur-
vey, followed by univariate ANOVAs
where appropriate.
Results reported in this section
are for performance on the task sce-
nario, user attitudes, use of EPSS,
and time on task.
The first research question inves-
tigated the effect of different types of
support systems on user performance
while completing a procedural soft-
ware task. Table 1 shows the mean
scores and standard deviations for
performance on the task scenario. The
table reveals that the mean scores
were 10.83 (90%) for the extrinsic
group, 10.06 (84%) for the intrinsic
group, 9.61 (80%) for the external
group, and 8.50 (71%) for participants
who were not provided with a perfor-
mance support system. A one-way
analysis of variance conducted on the
performance scores yielded a signifi-
cant overall difference, F (3, 68)=7.74,
p<.01. The strength of the relationship
between the treatments and the per-
formance scores was large, η2=.25.
Post-hoc tests were conducted to
determine significant differences in
mean performance scores. Multiple
comparisons conducted using the
Tukey method revealed that both
80 PerformaNce ImProVemeNt Quarterly
the intrinsic and extrinsic groups
had significantly higher scores on
the task over the group with no
performance support system. The
difference in the performance scores
between the intrinsic, extrinsic and
external groups was not significant.
The difference between the external
and no support group performance
scores was also not significant.
User Attitudes
The second research question in-
vestigated the effect of performance
support systems on the attitudes of the
participants. A four-item, five-point
Likert-type survey was administered
after completion of the task scenario.
The mean attitude scores by treat-
ment are shown in Table 2. The overall
mean score across the four items was
3.33 (5=strongly agree, 1=strongly
disagree), indicating neutral attitudes
towards the statements about the
performance support systems. The
table reveals that the average rating
was 4.37 for the extrinsic group, 3.71
for the intrinsic group, 3.26 for the
external group, and 1.88 for partici-
pants who were not provided with a
performance support system.
A 4 x 4 MANOVA was conducted
on the data to test for significant
differences. The overall means were
significantly different across the four
treatment groups, Wilks’ Λ=.52, F
(12, 170)=3.91, p<.01. The strength
of the relationship between the treat-
ments and user attitude scores was
moderate, η2=.19.
Follow-up univariate analyses of
variance revealed significant differ-
ences between treatment groups on
all four of the items. Post-hoc tests
were conducted to determine signifi-
cant differences between treatment
groups on the four survey items. Pair-
wise comparisons revealed 14 signifi-
cant differences between groups. On
all four questions, the three groups
with a performance support system
(external, extrinsic and intrinsic) had
significantly more positive attitudes
than the participants in the no EPSS
group. In addition, participants in
the extrinsic group had significantly
more positive attitudes than external
performance support users on two of
the questions: “Information in the
help system was easy to find” and “I
felt confident that I could complete
the task using the help system.”
Use of EPSS
The third research question ex-
amined the effect of treatment on
Table 1
User Performance Across EPSS Types
Support Condition M SD
Extrinsic EPSS 10.83 1.20
Intrinsic EPSS 10.06 1.47
External EPSS 9.61 1.54
No EPSS 8.50 1.69
Note: Maximum total correct=12.
Volume 18, Number 4/2005 81
performance support system use.
The number of times each user
accessed the appropriate help sys-
tem during the task scenario was
tracked by the software application.
The mean use scores are shown in
Table 3. The table reveals that par-
ticipants who were provided with an
extrinsic EPSS accessed the system
an average of 6.72 times, intrinsic
2.94 times, and external 2.11 times.
A one-way analysis of variance con-
ducted on the use scores yielded a
significant overall difference, F (2,
51)=11.44, p<.01. The strength of
the relationship between the treat-
ments and the use of EPSS was large,
Post-hoc tests were conducted to
determine significant differences
in mean use scores. Multiple com-
parisons conducted using the Tukey
method revealed that the extrinsic
group used the performance sup-
port system significantly more often
than both the intrinsic and exter-
nal groups. The difference in use
of EPSS between the intrinsic and
external groups was not significant.
The final research question inves-
tigated the effect of treatment on to-
tal time to complete the task scenario.
This was measured by calculating
the difference between the time at
which participants logged into and
out of the software application. The
data revealed that the external group
spent an average of 7.99 minutes on
the task, the extrinsic group spent
8.66 minutes, no EPSS participants
spent 8.77 minutes, and the intrinsic
group spent 11.25 minutes. A one-
way analysis of variance conducted
on the time on task yielded no sig-
nificant overall difference between
the mean scores.
Table 2
User Preference Across EPSS Types
Extrinsic Intrinsic External No EPSS
The help system provided the
appropriate level of detail to aid
in task completion.
4.44 3.78 3.33 2.00
Information in the help system
was easy to find.
4.44 3.44 3.33 1.82
I felt confident that I could
complete the task using the
help system.
4.33 3.89 3.11 1.82
I would use the help system
4.28 3.72 3.28 1.88
4.37 3.71 3.26 1.88
Note: Questionnaire items were measured on a five-point scale from 1 to 5
(Strong Disagree to Strongly Agree)
82 PerformaNce ImProVemeNt Quarterly
This study examined the effect of
various electronic performance sup-
port systems on user performance,
attitudes, use of EPSS, and time on
task. Four treatment groups complet-
ed a task scenario using an intrinsic
EPSS, extrinsic, external, or no sup-
port system at all.
Performance scores for the ex-
trinsic and intrinsic groups were sig-
nificantly higher than scores for the
group that was not provided with a
performance support system. A closer
examination of the data reveals that
performance of the participants who
received the extrinsic EPSS was 19%
more accurate on the task than those
who were not given an EPSS and
the intrinsic group was 13% more
accurate than the no EPSS group.
Furthermore, performance for the
external EPSS group was 9% more
accurate than the no EPSS group,
although the difference between
these two groups was not statistically
significant. Nevertheless, the positive
improvement in performance scores
for all three treatment groups over the
control group indicates that providing
an EPSS to support task performance
is better than having none at all.
It is not surprising that the ex-
ternal group scored the lowest of
the three EPSS treatments. Since
external performance support sys-
tems are not integrated with the
work interface, users are responsible
for locating the support content that
is relevant to the task. Findings for
user attitudes in the current study
suggest that users may perceive ex-
ternal support systems as less useful
than extrinsic support. For example,
the extrinsic group was more likely
to strongly agree with the statement
that the “Information in the help
system was easy to find” and “I felt
confident that I could complete the
task using the help system.” Further-
more, qualitative data reported by
Spool (2001) indicated that when pro-
vided an external EPSS to conduct a
single search, users located relevant
content only 55% of the time. Spool
noted that the “more times users
searched, the less likely they were to
find what they wanted” (p. 1). Users
who searched twice found relevant
content only 38% of the time; those
who searched three or more times
never found the correct support infor-
mation (Spool, 2001). Combined with
the result of the current study, these
observations suggest that designers
can improve user performance by cre-
Table 3
User Access to EPSS
Support Condition M SD
Extrinsic EPSS 6.72 4.01
Intrinsic EPSS 2.94 2.90
External EPSS 2.11 2.00
No EPSS -- --
Volume 18, Number 4/2005 83
ating performance support systems
that are integrated with the primary
work interface.
Given Gery’s assertion that 80%
of a performance support solution
should be comprised of intrinsic
systems, it is somewhat surprising
that users provided with the intrinsic
EPSS did not perform significantly
better than those given the other
types of support systems. The intrin-
sic EPSS in this study used an “em-
bedded knowledge” approach (Gery,
1995, p. 70). As users encountered
issues or questions in the procedural
task, they referred to embedded
help buttons that provided support
information directly in the primary
workspace. A more robust approach
to intrinsic EPSS would be to physi-
cally redesign and optimize the ap-
plication workspace to logically align
with the users’ workflow through the
task and perhaps even automate cer-
tain manual processes. This type of
human factors engineering approach
would likely have a positive effect
on user performance. Software work
interfaces are often built around
systems or databases rather than hu-
man tasks or processes. By designing
logical interfaces around user work-
flows, as one would in human factors
engineering, work interfaces become
more intuitive and user friendly.
Use of EPSS
Participants provided with an
extrinsic EPSS used their support
system significantly more than those
provided with intrinsic or external
systems. Participants accessed ex-
trinsic EPSS on average two times
more than those provided with an
intrinsic system and three times
more than those provided with an
external system. This result suggests
that designers can increase use of
performance support systems by
integrating them more directly into
the primary work interface.
It is not surprising that use of
the external performance support
system was lowest among the three
treatment groups that received an
EPSS. External systems require
users to search for and locate infor-
mation, sometimes in futility. Spool
(2001) noted that less than 25% of
participants searched more than two
times. Furthermore, “those that did
persevere [by searching more] did not
see positive results” (p. 1). Extrinsic
and intrinsic systems integrate with
the work interface allowing them
to understand work contexts and
deliver the appropriate support in-
formation. This integration reduces
the amount of work required by the
user to find support information and
improves the chances of finding the
correct information on the first at-
It is surprising that use of the
intrinsic EPSS was significantly less
than the extrinsic. Once again, this
finding may be due to the fact that
an embedded knowledge approach
was used for the intrinsic EPSS. Most
modern computer displays default to
a resolution of 1024 by 768 pixels.
Software developers are challenged
to fit the necessary components of a
work interface—forms, fields, menus,
icons, and toolbars—in this limited
area. When intrinsic systems employ
an embedded knowledge approach, it
must display support content directly
into the primary workspace. This can
be done through the use of a tooltip,
embedded pane, or resizing the pri-
mary work interface to accommodate
an adjacent window. By doing so, the
intrinsic EPSS must compete with
84 PerformaNce ImProVemeNt Quarterly
the other interface elements for pre-
cious screen real estate. The lack of
space to display the intrinsic EPSS
content may explain the significantly
reduced use of intrinsic support as
compared to extrinsic.
Extrinsic and external perfor-
mance support systems both require
the user to access content stored and
delivered by an outside system. As a
result, they do not compete for limited
space in the primary work interface
as intrinsic systems do.
Performance and Use of EPSS
When considering performance
and use factors together, one may
note that extrinsic performance sup-
port was used significantly more by
participants in the study than in-
trinsic support. However, the greater
use of extrinsic performance support
did not result in a significant perfor-
mance difference between the two
User Attitudes
Participants in the three perfor-
mance support groups had signifi-
cantly more positive attitudes than
the no EPSS group. This finding can
be attributed to the fact that partici-
pants in the control group were not
provided with any on-task support or
guidance. Participants in this study
indicated a strong preference for any
kind of on-task support from an ex-
ternal, extrinsic or intrinsic system.
This finding further validates the
notion that providing any kind of
EPSS to support task performance
is preferable to having none at all.
In addition, the extrinsic group had
significantly more positive attitudes
than the external users on two survey
questions. This finding is consistent
with results for the performance and
use variables. It is likely due to the
fact the extrinsic participants could
immediately locate relevant support
content on the first request rather
than having to search for and locate
support information using the exter-
nal system.
Performance, Use of EPSS and
User Attitudes
The fact that the extrinsic group
rated their performance support
system significantly higher than
the external group corresponds to
the significant increase in use of the
extrinsic system in comparison to
the intrinsic and external groups.
This relationship highlights the fact
that, for interventions like EPSS
that rely on learner control and self-
regulation, it is important to design
the system in such a way that users
prefer the support system, have easy
access to information, and feel that
they will find the answers they need.
More simply, although certain sys-
tems like intrinsic EPSS may have
psychological benefits over other
designs, if the users feel that the
system is annoying or unhelpful, they
will not use it and therefore will not
maximize the benefits it may offer to
aid task performance.
This study focused on a relatively
simple procedural software task.
As a result, the findings may not be
valid when extended to more complex
tasks or work contexts that are not
based in software. While the subjects
in the study had no prior knowledge
of the task, they are all employees
that work in a corporate setting that
requires high computer system use.
Users that may have less computer
experience could perform differently
Volume 18, Number 4/2005 85
when exposed to the performance
support systems offered. In addition,
the data entered by participants as
they completed the task was evaluat-
ed only by the lead researcher in this
study. Future studies that involve
evaluation by a group of observers
may yield different results.
Future Research
Although these findings shed light
on the relative effectiveness of dif-
ferent types of performance support,
many questions remain. The depen-
dent measures used in this study are
a handful of many important factors
to human performance technologists.
Studies that examined a broader
range of measures such as informa-
tion retention, error rate and time-
on-task reduction would be invalu-
able. The task that participants were
asked to complete in this study was
fairly simplistic. A follow-up study
that used a similar but more complex
task would provide better context for
the results of this study. While this
study focused on procedural software
tasks, human performance technolo-
gists are also applying EPSS towards
the improvement of performance that
involves physical tasks. Examples
include aircraft repair, automobile
repair, and manufacturing equip-
ment operations. It would be useful to
extend this study in other settings to
determine if the results can be trans-
ferred to these other work contexts.
Furthermore, a comparative study
that examines the effectiveness of
a broader range of intrinsic EPSS
(embedded knowledge versus hu-
man factors engineering) compared
to other EPSS designs would provide
additional insight into the value
of electronic performance support
Altalib, H. (2002). ROI calculations
for electronic performance support
systems. Performance Improvement,
41(10), 12-22.
Chase, N. (1998). Electronic support cuts
training time [Electronic version].
Quality Magazine. Retrieved January
12, 2005 from http://openacademy.
Davidson, L. (1998). Measure what you
bring to the bottom line. Workforce,
77, 34-40.
Gery, G. (1991). Electronic performance
support systems. Tolland, MA: Gery
Gery, G. (1995). Attributes and behaviors
of performance-centered systems.
Performance Improvement Quarterly,
8(1), 47-93.
Gery, G. (2003). Ten years later: a new
introduction to attributes and behav-
iors and the state of performance-
centered systems. In G.J. Dickelman
(Ed.), EPSS revisited: A lifecycle for
developing performance-centered
systems (pp. 1-3). Silver Spring, MD:
International Society for Performance
Hawkins, C.H. Jr., Gustafson, K.L., &
Nielson, T. (1998). Return on invest-
ment (ROI) for electronic performance
support systems: A web-based system.
Educational Technology, 38, 15-22.
Huber, B., Lippincott, J., McMahon, C., &
Witt, C. (1999). Teaming up for per-
formance support: A model of roles,
skills and competencies. Performance
Improvement, 38(7), 10-14.
Morrison, J.E., & Witmer, B.G. (1983). A
comparative evaluation of computer-
based and print-based job perfor-
mance aids. Journal of Computer-
Based Instruction, 10(3), 73-75.
Phillips, J.J. (1997). Handbook of training
evaluation and measurement methods
(3rd ed.). Houston: Gulf Publishing
Raybould, B. (2000). Building perfor-
mance-centered web-based systems,
information systems, and knowledge
86 PerformaNce ImProVemeNt Quarterly
management systems in the 21st
century. Performance Improvement,
39(6), 69-79.
Rosenberg, M.J. (1995). Performance
technology, performance support, and
the future of training: A commentary.
Performance Improvement Quarterly,
8(1), 94-99.
Sherry, L., & Wilson, B. (1996). Support-
ing human performance across disci-
plines: A converging of roles and tools.
Performance Improvement Quarterly,
9(4), 19-36.
Spool, J.M. (2001). Users don’t learn to
search better [Electronic version]. Re-
trieved April 3, 2005 from http://www.
FRANK NGUYEN is a doctoral
student focusing on performance
support systems. He has managed
the development and deployment
of enterprise e-learning and per-
formance support systems at Intel
Corporation for the last five years.
Nguyen is co-author of Efficiency
in Learning (Jossey Bass, 2005)
and holds a masters in Educa-
tional Technology from Arizona
State University. Telephone: (480)
552-0559. E-mail: frank.nguyen@
JAMES D. KLEIN is a Professor and
Program Leader in the Educational
Technology program at Arizona
State University, Tempe where he
teaches courses on instructional
design, research, and performance
improvement. His most recent
scholarly work includes the book,
Instructor Competencies: Stan-
dards for Face-to-face, Online, and
Blended Settings. Mailing address:
Arizona State University, Box
870611, Tempe, AZ 85287-0611.
Telephone: (480) 965-0349. E-mail:
HOWARD SULLIVAN is a profes-
sor in the Division of Psychology
in Education at Arizona State Uni-
versity where he teaches courses in
instructional design, educational
evaluation and educational re-
search. He was the founding Re-
search Editor of Educational Tech-
nology Research and Development.
He has held positions as a Visiting
Scholar at the UCLA Center for
the Study of Evaluation and as a
Senior Fellow at the University of
Melbourne (Australia) Institute of
Education. He was selected by the
ASU Graduate College as the 2002
ASU Outstanding Doctoral Men-
tor. Mailing address: Arizona State
University, Box 870611, Tempe, AZ
85287-0611. Telephone: (480) 965-
0348. E-mail:
... In addition, some of the results of older studies can now be considered obsolete, because technologies available in the past are very different from those available today (Ley, 2020). Moreover, literature on EPSS is criticized for not being empirical (Mao, 2004;Nguyen et al., 2005;Gal and van Schaik, 2010) but based instead on anecdotal evidence (Mao, 2004;Gal and van Schaik, 2010). The present exploratory studies address this research gap from two perspectives. ...
... As such, users have to turn away from the target system and to break the work context in order to use the external EPSS (Gery, 1995;Mao, 2004;Sumuer and Yildirim, 2015). Early examples of external performance support are help desks, questions and answers Q&A, job aids, manuals, knowledge databases, and search engines (Gery, 1995;Nguyen et al., 2005;Nguyen and Hanzel, 2007;Gal and Nachmias, 2012). More recent examples also include Web 2.0 technologies, such as online forums and communities and the content provided there. ...
... One of the most frequently mentioned benefits of EPSS is its potential to support employee performance (Barker and Banerji, 1995;Gery, 1995;Chang, 2004;Nguyen and Klein, 2008) and as a result different aspects of employee productivity (Bastiaens, 1999;Altalib, 2002). Several empirical studies have reported positive effects of EPSS on various measures of performance (Bastiaens, 1999;van Schaik et al., 2002;Mao and Brown, 2005;Nguyen et al., 2005;Gal and Nachmias, 2011;Lanese and Nguyen, 2012;Rios et al., 2013;Nuss et al., 2014;Yakin and Yildirim, 2016;Gal et al., 2017;Ugur-Erdogmus and Cagiltay, 2019). These were, for instance, positive effects on expertise reports or speed of task completion of police officers in Turkey (Yakin and Yildirim, 2016) and positive effects on time used for and quality of maintenance procedures of the engine air bleed system on a Boeing 737 aircraft (Rios et al., 2013). ...
Full-text available
In the context of office work, learning to handle an Enterprise Resource Planning (ERP) system is important as implementation costs for such systems and associated expectations are high. However, these expectations are often not met because the users are not trained adequately. Electronic Performance Support Systems (EPSS) are designed to support employees’ ERP-related problem-solving and informal learning. EPSS are supposed to enhance employees’ performance and informal workplace learning through task-specific and granular help in task performance and problem-solving. However, there is little empirical research on EPSS. Two survey studies addressed this research gap. In the first study, 301 people working in Human Resource (HR)-related positions and functions evaluated the learning potential of EPSS as well as potential advantages and obstacles concerning the implementation and use of EPSS. Though other measures are currently assessed as more important for learning, HR employees expect a strong increase in the significance of EPSS for employee learning. In the second study, 652 users of ERP software completed a questionnaire on characteristics of their daily work tasks, team characteristics, individual dispositions, their coping with ERP-related problems, and characteristics of EPSS. Findings indicate that the most frequently available and used approach when dealing with an ERP-related problem is consulting colleagues. Three EPSS types can be distinguished by their increasing integration into the user interface and their context-sensitivity (external, extrinsic, and intrinsic EPSS). While external and extrinsic EPSS are available to many users, intrinsic EPSS are less common but are used intensively if available. EPSS availability is identified to be a strong positive predictor of frequency of EPSS use, while agreeableness as well as the task complexity and information-processing requirements show small negative effects. Moreover, more intensive ERP users use EPSS more frequently. In general, ERP users value, features such as context-sensitivity, an integration of the EPSS into the ERP system’s user interface, the option to save one’s own notes, and information displayed in an extra window. It is expected that EPSS will play an important role in workplace learning in the future, along with other measures.
... With this distinction between external, extrinsic and intrinsic EPSS, Gery (1995) provided designers with a guideline to implement 80% of their support systems as intrinsic, 10% extrinsic, and the remaining 10% external. She argued that this guideline optimized designer and performer time, decreased performance development overhead, and still accomplished the desired impact: user performance (Nguyen & al, 2005). ...
Full-text available
This study aims to highlight the importance of adopting the electronic performance support system to enhance organizational learning. The results of this study show a performance gap in companies using the conventional training to their employees. That gap can be filled through giving priority and developing the training by shifting from traditional training to the new method, which is the electronic system to support performance as an innovative approach contributing in the achievement of organizational learning. Keywords: EPSS: Electronic learning support system, Conventional training, Organizational learning.
... • EPSS can also support informal workplace learning (Gery, 1995;Raybould, 1995;van Schaik, Pearson & Barker, 2002) • Prior study with 301 participants working in HR-related roles confirm EPSS' potential to support employee learning • Although companies have been applying EPSS -with varying success -since the 1990s, empirical research on their effectiveness is scarce (Chang, 2004;Gal et al., 2017;Gal & Nachmias, 2012;Mao, 2004;Mao & Brown, 2005;Nguyen & Klein, 2008) • Literature on EPSS is criticized for not being empirical (Gal & van Schaik, 2010;Mao, 2004;Nguyen et al., 2005) but based instead on anecdotal evidence (Gal & van Schaik, 2010;Mao, 2004) → Research gap 4 Leiß, Rausch & Seifried University of Mannheim, Economics and Business Education ...
Full-text available
The Potential of Electronic Performance Support Systems (EPSS) to Promote Employee Learning Presentation Slides, AERA, San Diego, 2022
... et al., 2003;cagiltay, 2006;Nguyen, 2007;Nguyen, et al., 2005;Pahl, 20021 ‫(صغير‬ - ‫فورية)‬ 7 ‫م‬ 3 ‫(متوسط‬ - ‫فورية)‬ 7 ‫م‬ 5 ‫(كبير‬ - ‫فورية)‬ 7 ‫مؤجلة‬ ‫م‬ 2 ‫(صغير‬ - ‫مؤجلة)‬ 7 ‫م‬ 4 ‫(متوسط‬ - ‫مؤجلة)‬ 7 ‫م‬ 6 ‫(كبير‬ - ‫مؤجلة)‬‫س‬ 1 ‫س‬ 1 ‫س‬ 1 ‫س‬ 1 ‫س‬ 1 ‫م‬ 2 ‫(صغير‬ - ‫مؤجلة)‬ ‫م‬ 1 ‫(صغير‬ - ‫فورية)‬ ‫م‬ 3 ‫(متوسط‬ - ‫فورية)‬ ‫م‬ 4 ‫(متوسط‬ - ‫مؤجلة)‬ ‫م‬ 5 ‫(كبير‬ - ‫فورية)‬ ‫م‬ 6 ‫(كبير‬ - ‫مؤجلة)‬ * - 3.‫س‬ 1 ‫دان‬ ‫ا‬ ‫س‬ 3 ‫س‬ 3 ‫س‬ 3 ‫م‬ 4 ‫(متوسط‬ - ‫مؤجلة)‬ ‫م‬ 1 ‫(صغير‬ - ‫فورية)‬ ‫م‬ 2 ‫(صغير‬ - ‫مؤجلة)‬ ‫م‬ 3 ‫(متوسط‬ - ‫فورية)‬ ‫م‬ 5 ‫(كبير‬ - ‫فورية)‬ ‫م‬ 6 ‫(كبير‬ - ‫مؤجلة)‬ - 7.‫س‬ 1 ‫س‬ 2 ‫س‬ 3 ‫س‬ 4 ‫س‬ 5 ‫م‬ 6 ‫(كبير‬ - ‫مؤجلة)‬ ‫م‬ 1 ‫(صغير‬ - ‫فورية)‬ ‫م‬ 2 ‫(صغير‬ - ‫مؤجلة)‬ ‫م‬ 3 ‫(متوسط‬ - ‫فورية)‬ ‫م‬ 4 ‫(متوسط‬ - ‫مؤجلة)‬ ‫م‬ 5 ‫(كبير‬ - ‫فورية)‬ -The effect of the interaction between the volume of micro-e-training content (small -medium -large) and the timing of providing educational support (immediate -postponed) on the mobile training environment to develop some competencies for effective organization of Big Data and improve the level of satisfaction of trainees in digital training at Qassim University. ...
... Electronic Performance Support Systems (EPSS) stellen den Lernenden relevante Informationen unmittelbar beim Ausführen ihrer Aufgaben zur Verfügung (Nguyen et al. 2005 Berücksichtigung dieser Kontexte eben dann die passenden Inhalte angezeigt werden" (Kröpfl 2020 ...
Full-text available
Lerninhalte, die nach dem Gießkannenprinzip auf alle MitarbeiterInnen verteilt werden, sind selten effizient. Für jeden Lernenden eigene Inhalte zu erstellen, stellt aber einen (zu) großen Aufwand da. Dies ist in vielen Fällen allerdings gar nicht erforderlich, da es bereits bestehende Inhalte (News- Artikel, Lernvideos, Whitepapers, Podcast-Interviews etc.) gibt, die man nutzen und ins E-Learning integrieren kann. Content Curation ist eine Möglichkeit, dies umzusetzen. Von Content Curation spricht man, wenn man eine „Kollektion“ an Inhalten zu einem bestimmten Thema für eine definierte Zielgruppe kuratiert. Man greift dabei auf bestehende Inhalte zurück – ähnlich wie ein(e) KuratorIn in einem Museum die Ausstellungsstücke nicht selbst herstellt. Ziel ist es, für die einzelnen Lernenden relevante Inhalte zu kuratieren, die ihnen einen Mehrwert liefern – und damit die Informationsüberlastung zu reduzieren.
... Intrinsic mobile performance support system can automatically detect context through a combination of context-aware computing and ubiquitous and pervasive techniques to adapt to user preferences, location, time and environment (Specht & Kravcik, 2006). Research showed, contrary to initial expectations, that external and extrinsic performance support systems are more effective than intrinsic ones (Nguyen, Klein, & Sullivan, 2005). However, it would be speculation to expect that this would be the case with mobile performance support systems as well. ...
Full-text available
The Mobile Training Methodology for E-business ver. 2.0 has been prepared as part of the “E-business Mobile Training - use of mobile Performance Support System for acquiring e-business management skills” project, co-financed by the Leonardo da Vinci – Lifelong Learning Programme. It is based on “mobile Performance Support System for Vocational Education and Training” project, and results developed during its implementation.
Full-text available
Time is money and a reduction in time should be the first goal of any training program or any employee development initiative. The skills and knowledge acquired today become irrelevant or obsolete quickly. In that kind of dynamics, the organizations have shifted focus on how to shorten time to proficiency of employees and bring them up to speed to the required performance in a shorter time. This book intends to establish scholarly know-how on this construct of Accelerated Proficiency and provide practitioners and researchers with a condensed and coherent knowledge-base on this burning business concept. The book explores concepts, themes, nature, and challenges associated with the process of accelerating proficiency. This book discusses the business metrics of time to proficiency and speed to proficiency. This book is organized around reviews of studies on performance and proficiency from several different disciplines spanning over five decades. In particular, this book will enable you with the answers to some crucial questions - What is the nature of the proficiency in the organization? What is the meaning and goal of accelerating proficiency? Whether or not proficiency and performance of employees can be accelerated? What kind of methods has been researched and proven to accelerate proficiency?
Full-text available
The purpose of the present study was to measure the effectiveness and efficiency of the three methods: training, electronic support system, the combination of training and electronic performance support system, for improving performance and increasing employee efficiency. The research method was semi-experimental, and the post-test with the control group was used to measure the performance and efficiency of the staff. The statistical population in this research was 570 insurance agents of the Day insurance company throughout the country. Among them, 60 insurance agents were selected by cluster sampling method and randomly divided into three groups: "Electronic training," "Electronic Performance Support System" And "Combination of electronic training and electronic performance support system.” A questionnaire was designed with the help of subject specialists to measure the actual performance of the participants. Content validity was used to determine the validity of the questionnaire, and the reliability of the sculptors (correlation coefficient was 0.79) was used to assess the reliability of the questionnaire. To measure the efficiency (amount of time spent completing the job task), the arrival time of the insurance agents to the Electronic Performance Support System and the moment of their departure from this environment is estimated. The collected data were analyzed using multivariate analysis of variance and post hoc tests. The results of the data analysis showed that the insurance agents who participated in the "Combination of Electronic training and the Electronic Performance Support System" group had better performance compared to the insurance agents in the other two groups of "Electronic training " and "Electronic Performance Support System", and They did their job activity in a shorter time.
Work is becoming increasingly com-puter-mediated. Performance development is becoming increasingly difficult and expen-sive due to complexity, change, restructuring, and redefinition of who does work. In the coasumer market place, softw are is emerging with characteristics and behaviors; that actively support performance development while permitting learning. These performance-centered systems are the next step in the evolution of performance support. Performance Technologists must and will play an active part in both their advocacy and design. Ultimately, no consumer or large-scale software system should be considered accep table until day-one performance isgenerated for novice performers while still accommodating the requirements of expert performers.
Electronic Performance Support Systems (EPSSs) are replacing traditional training in many settings. This article describes a package to determine the return on investment (ROI) of an EPSS used by the U.S. Department of Veterans Affairs, applicable as well as to the calculation of ROI for other EPSSs. (Author/AEF)
Performance support is close to the center of a host of related fields and specialties, including human performance technology, electronic performance support systems, technical communications, and instructional design. Because of their common interest in performance support, and common external influences such as cognitive psychology and digital technologies, roles and tools within these fields are beginning to converge, resulting in unprecedented overlap. In times of rapid change, related fields have an opportunity to learn from one another, borrowing useful elements and incorporating them into their own practices. The purpose of this paper is to explore the similarities, differences, and emerging trends among some of these fields and to gain insights into how their evolution affects performance support. Across these fields, we find a continuing tension between designed messages and tools allowing users more flexibility and control. The best performance-support systems include both of these components as well as a strong human support component. We also observe a trend toward greater reliance on users and user communities in defining and controlling support systems.
How close is the relationship between performance technology, performance support, and training? Very close, actually. Both performance technology and electronic performance support (EPS) have their roots in education and training, but in many ways they have transcended their be ginnings and have evolved beyond where training leaves off. This commentary offers a performance technologist's view of EPS and suggests not only that a performance technology perspective is a requirement for the development of EPS, but that involvement in building and implementing an EPS can help people shift their paradigm from training to performance. Implications for the education and training field now and in the future are also proposed.