Content uploaded by Frank Nguyen
Author content
All content in this area was uploaded by Frank Nguyen on Mar 01, 2022
Content may be subject to copyright.
Volume 18, Number 4/2005 71
Performance Improvement Quarterly, 18(4) pp. 0-00
A Comparative Study of Electronic Performance
Support Systems
Frank Nguyen
James D. Klein
Howard Sullivan
Arizona State University
ABSTRACT
Electronic performance support
systems (EPSS) deliver relevant sup-
port information to users while they are
performing tasks. The present study
examined the effect of different types of
EPSS on user performance, attitudes,
system use and time on task. Employ-
ees at a manufacturing company were
asked to complete a procedural soft-
ware task and received support from
either an intrinsic, extrinsic, external
performance support system or no sys-
tem at all. Results revealed significant
differences on performance, attitudes
and use between several treatment
groups. The study suggests that provid-
ing any kind of EPSS to support task
performance is better than having none
at all. In addition, designers can im-
prove user performance, attitudes and
use by creating systems that integrate
with the primary work interface.
Gloria Gery (1991) introduced the
concept of electronic performance sup-
port systems (EPSS) as a method to
enable human performance through
just-in-time learning and task sup-
port. She asserted that through EPSS
one could generate “day-one perfor-
mance…for novice performers” (Gery,
1995, p. 47). Since its introduction,
Gery has reported that the principles
of EPSS have been applied to a wide
range of work interfaces from appli-
cations that automate tax prepara-
tion to financial planning Web sites
to tools that help consumers plan and
purchase travel (Gery, 2003).
To guide practitioners, authors
and experts have shared their in-
sights on a wide range of EPSS topics
over the past decade. For instance, a
number of experts have focused on
the unique methods and processes
required to develop EPSS. Raybould
(2000) introduced a performance
support mapping methodology. This
approach combines elements from
disparate fields including business
process reengineering, human per-
formance technology and instruc-
tional design into a new field dubbed
performance support engineering.
Huber, Lippincott, McMahon, and
Witt (1999) provided a framework
for the skills, competencies and job
roles that make up an effective EPSS
development team.
Other authors have focused on
the value of performance support.
Chase (1998) asserted that EPSS
could reduce the time and cost
associated with training new em-
ployees. Altalib (2002) provided a
detailed process on how to measure
the return on investment (ROI) for
72 PerformaNce ImProVemeNt Quarterly
EPSS. Based largely on the work of
Davidson (1998), Phillips (1997), and
Hawkins, Gustafson, and Nielson
(1998), Altalib’s comprehensive ROI
approach examined potential ben-
efits derived from EPSS including
hard measures such as increased
sales or manufacturing production
to soft measures such as employee
attitudes.
Some have offered strategic views
on how EPSS relates to and comple-
ments other information interven-
tions such as knowledge manage-
ment and training. Rosenberg (1995)
argued that trainers and others in-
volved in instructional design should
shift to the more holistic views of hu-
man performance technology (HPT)
which embraces EPSS and training
among other types of interventions.
With the advent of HPT, Sherry and
Wilson (1999) predicted the con-
vergence of the traditional roles of
instructional designer, performance
support designer and information
technologist.
Interestingly, very few of the cur-
rent ideas and principles related to
EPSS are based on any substantive
research. In fact, although EPSS
has been discussed for almost two
decades, very few studies have been
conducted to measure its effective-
ness. One of the few research stud-
ies that examined EPSS compared
the effectiveness of computer-based
and print-based performance aids
in the Army (Morrison & Witmer,
1983). While this study found no
significant differences between the
two delivery media, it did not address
more fundamental issues such as the
overall effectiveness of EPSS or the
specific EPSS designs that may be
better under differing performance
conditions.
To this end, Gery introduced a
conceptual framework for EPSS that
illustrates key design differences be-
tween potential performance support
systems. She asserted that there are
three fundamental types of EPSS:
external, extrinsic and intrinsic sup-
port (Gery, 1995).
External systems store content
used to support task performance in
an external database. This content is
not integrated within a user’s work
interface. As a result, users are forced
to manually locate relevant informa-
tion in the external EPSS. Common
examples of external performance
support systems include search
engines, frequently asked question
pages, and help indexes. In addition,
external performance support “may
or may not be computer mediated”
(Gery, 1995, p. 53). Job aids or docu-
mentation are common external per-
formance support interventions.
Extrinsic “[p]erformance sup-
port…is integrated with the system,
but is not in the primary workspace”
(Gery, 1995, p. 51). In other words,
extrinsic systems integrate with the
user’s work interface in such a way
that the EPSS can identify the user’s
location in a system or even the exact
task that they may be working on.
With this contextual information,
the extrinsic system can intelligently
locate content that may be relevant
to support the task at hand. Like ex-
ternal performance support systems,
though, the content used to support
a task is external to the work inter-
face.
Intrinsic systems provide users
with task support that is incorporated
directly within their work interface.
Due to this direct integration with
the interface, Gery asserted that in-
trinsic EPSS provides “[p]erformance
Volume 18, Number 4/2005 73
support that is inherent to the system
itself. It’s so well integrated that, to
workers, it’s part of the system” (Gery,
1995, p. 51). Under this rather broad
definition, examples of intrinsic per-
formance support systems can range
from tools that automate tasks and
processes, user-centered design of
work interfaces to reduce complexity
and improve usability, or embedded
knowledge that is displayed directly
in the work interface.
With this dis-
tinction between
external, extrin-
sic and intrinsic
EPSS, Gery (1995)
provided designers
with a guideline
to implement 80%
of their support
systems as intrin-
sic, 10% extrinsic,
and the remain-
ing 10% external.
She argued that
this guideline opti-
mized designer and performer time,
decreased performance development
overhead, and still accomplished the
desired impact: user performance.
However, the impact of the three
types of EPSS has not been empiri-
cally tested. Thus, performance tech-
nologists lack validated principles to
guide them in the selection, design
and development of EPSS. To address
this gap, this article summarizes a
study that focused on four key re-
search questions:
1. Of the three types of EPSS
(external, extrinsic, or intrinsic),
which one is better at enabling user
performance?
2. Which type of EPSS do users
prefer?
3. Which type of EPSS do they use
more often?
4. Which one minimizes the time
it takes for a user to complete a given
task?
Method
Participants
Seventy-two employees from a
semiconductor manufacturing com-
pany participated in the study. The
employees were recommended by
their direct man-
agers or identified
by other partici-
pants. All partici-
pants involved in
the study had com-
pleted at least four
years of college:
forty-six obtained
a bachelor’s and
twenty-six ob-
tained a master’s
degree in various
fi eld s. Pot entia l
participants were
screened for prior knowledge of the
corporation’s learning management
system used in the study; any indi-
viduals with previous experience us-
ing the system, which served as the
basis for the task, were not selected
to participate in the study.
The participants represented a di-
verse range of job roles: twenty-eight
software engineers, fourteen train-
ing professionals, twelve managers,
five business analysts, five human
resource professionals, three accoun-
tants, two financial analysts, one de-
sign engineer and one customer sup-
port specialist. The participants were
distributed across four western states:
forty-seven were located in Arizona,
nine in Oregon, eight in Northern
California, and seven in Utah.
…performance
technologists
lack validated
principles to
guide them in the
selection, design
and development
of EPSS.
74 PerformaNce ImProVemeNt Quarterly
Materials
Software application. One portion
of the company’s learning manage-
ment system was adapted for use in
the study. As part of the process to
create an online training course, em-
ployees are required to submit data
that describes their training course.
Such metadata typically includes the
course’s name, description, objectives
and other relevant data. The course
registration software that is normal-
ly used to submit this metadata was
extracted from the learning manage-
ment system and tied to an isolated
database designed specifically for
the study.
As illustrated in Figure 1, the
course registration software screens
included a series of open text fields
that required the user to input rel-
evant data as well as menus that
required the user to select from a
number of pre-defined choices. In
total, the course registration module
required twelve user inputs or selec-
tions. Data entered into the course
registration software were stored in
a database for analysis.
Performance support systems.
The course registration software
was modified to include three dif-
ferent types of performance support
systems. The three treatments were
Figure 1. Software Application.
Volume 18, Number 4/2005 75
based on EPSS categories established
by Gery (1995): including external,
extrinsic and intrinsic support.
External performance support
system. The external system imple-
mented in this study was a search
engine. When users in this treat-
ment clicked a help button located in
the menu of the course registration
software, their request was recorded
in a database and a popup window
opened that prompted them to enter
a keyword as shown in Figure 2. Once
they submitted their keyword, the
external support system searched
through the EPSS content repository
and presented the user with a choice
of help topics based on their query.
Participants then had to select the
appropriate topic to read.
Extrinsic performance support
system. The extrinsic system was a
context-sensitive help system. Help
buttons in the form of a question
mark were inserted throughout the
software application. When users
clicked on the buttons, their request
was recorded in a database and a new
window opened displaying support
information associated with the task
as illustrated in Figure 3.
Intrinsic performance support
system. The intrinsic performance
support was an information-based
system that provided the users with
task-relevant text instructions direct-
ly in the course registration software
screens as shown in Figure 4. In order
to capture the number of times partici-
pants used the intrinsic performance
Figure 2. External Performance Support System.
76 PerformaNce ImProVemeNt Quarterly
support system, help buttons were
inserted throughout the software.
When users clicked the buttons, their
request was recorded in a database
and the information associated with
the support instance was displayed
adjacent to the button. While this
approach deviated somewhat from
typical instances of intrinsic perfor-
mance support where information
may be displayed automatically, the
researchers felt that it was an impor-
tant modification that facilitated the
comparison of EPSS use across the
three treatments.
In addition to the three perfor-
mance support types, the system
was also modified to display no per-
formance support at all to facilitate
a control group. The no performance
support system is illustrated in the
software application screen shown
earlier in Figure 1.
The content across the three per-
formance support systems was identi-
cal and differed only in the manner in
which it was accessed and presented.
When the user accessed the course
registration software, the system ran-
domly assigned the user to one of the
four performance support treatments
described above. As the software was
loaded onto the participant’s comput-
er, the system automatically changed
the interface to show or hide the
appropriate buttons and on-screen
information based on the participant’s
treatment group assignment.
Task scenario. The task scenario
portrayed a realistic issue that a new
Figure 3. Extrinsic Performance Support System.
Volume 18, Number 4/2005 77
employee might face. The scenario
included information that a training
manager might provide to an employ-
ee when registering a new course in
the corporate learning management
system. The following is an excerpt of
the task scenario text:
Imagine that you have been
recently hired as a training adminis-
trator. The message below contains
information that your new manager
emailed you for an online training
course that needs to be published.
Thanks for taking care of this
task for me. Here’s how I’d like to
have the course set up:
q The name of the course is Pro-
gram Life Cycle.
q It is an online learning course
that provides an overview of the eBG
Program Life Cycle, or PLC.
q I would like for this course to
appear in the course catalog under
a category called Project Manage-
ment
q I’ve already saved the course to
the web server, and it is located in a
folder called PLC.
q We’re phasing out the class-
room-based course. This will be an
online, web-based course.
q The course’s test contains 10
questions, and I’d like students
to pass with 90% before they get
credit.
q I’d like to have students take
a survey right after they finish the
course. We don’t need to do the fol-
low-up survey that is sent out six
weeks later.
Figure 4. Intrinsic Performance Support System.
78 PerformaNce ImProVemeNt Quarterly
q Go ahead and set up the course
so that students can see it in the
catalog immediately.
q I’d also like to have this course
set up so that the system replicates
it to the other geographies. It looks
like we will have students in China,
Malaysia, Ireland, Israel and the
United States.
Criterion Measures
Four criterion measures were
used in the study: user performance
on the task, user attitude surveys, use
of EPSS, and time on task.
Performance. User performance on
the task was measured by evaluating
the number of correct items the par-
ticipants submitted to the software
application. As mentioned earlier, the
course registration module required
twelve user inputs or selections. Data
entered by the user into the course
registration software was stored in
a database and subsequently evalu-
ated by the lead researcher. Partici-
pants received one point for each cor-
rect input with a maximum of twelve
points possible.
User attitude surveys. A four-item
survey was developed to measure
participant attitudes towards the re-
spective performance support system
provided in the task. Respondents
used a 5-point Likert scale (5=strong-
ly agree, 1=strongly disagree) to rate
their attitudes regarding the effec-
tiveness of the performance support
system.
Use of EPSS. Various en route data
were recorded to measure the number
of access to the performance support
system. When participants in the
external performance support treat-
ment clicked the Help button located
on the software application’s naviga-
tion menu, the participant’s name
and time of access were recorded. In
addition, the performance support
content that was subsequently ac-
cessed by the participant was noted.
Each page of content that was opened
by the participant was considered one
access to the EPSS. Similarly, when
participants in both the extrinsic and
intrinsic performance support treat-
ment clicked the question mark (?)
icon embedded within the software
application, the participant’s name,
time of access and location in the ap-
plication were recorded. Each click on
the question mark icon was consid-
ered one access to the EPSS. Since the
control group was not provided with
a performance support system, no ac-
cesses were recorded for participants
in this group.
Time-on-task. The total amount of
time participants spent completing
the task in the study was measured
by calculating the difference between
the time at which participants logged
into and out of the software applica-
tion.
Procedures
Since the participants in the study
were geographically dispersed, the
researchers arranged approximately
twenty data collection sessions at
various sites over the course of two
weeks. Small groups of 3-5 partici-
pants were directed to prearranged
conference rooms to ensure that they
were not distracted by phone calls, e-
mail or co-workers while completing
the study. The lead researcher gave
participants the task scenario and
instructed them to read it until they
were comfortable with the task. Once
all participants finished reading the
task scenario, the lead researcher
instructed them to complete the task
using only the information provided
by the task scenario and any help
Volume 18, Number 4/2005 79
that may be provided by the soft-
ware application. In addition, the
researcher instructed participants to
log out of the software application as
soon as they felt they had completed
the task.
The participants were then given
the location of the software ap-
plication on the corporate network
and logged in using an automated
authentication system pre-installed
on all corporate-issued computers.
When users accessed the software
application, the system randomly
assigned them into one of four treat-
ment groups (intrinsic, extrinsic,
external or no EPSS) and displayed
the appropriate performance support
system. Participants were not aware
that they had been assigned to a dif-
ferent treatment group or that their
system was configured with a differ-
ent EPSS. The opening screen of the
software application provided a brief
set of instructions demonstrating
how to access the support system.
Participants individually worked
through the task using the software
program and performance support
system. Once the participants com-
pleted the task and logged out of
the system, they were automatically
directed to and completed the user at-
titude survey. One participant in the
control group declined to complete
the survey.
To ensure participant motivation,
the researchers worked with a subset
of managers within the company to
identify all employees in their organi-
zations with no prior experience with
the corporate learning management
system. These managers encouraged
their employees to participate in the
study. In addition, refreshments were
offered to participants upon comple-
tion of the task, and a letter of recog-
nition was sent to the manager and
participant after the study.
Design and Data Analysis
This study used a posttest-only
control-group design. One-way anal-
ysis of variance (ANOVA) was con-
ducted on participants’ performance
on the task, use of EPSS and time on
task. One-way multivariate analysis
of variance (MANOVA) was conduct-
ed on the data from the attitude sur-
vey, followed by univariate ANOVAs
where appropriate.
Results
Results reported in this section
are for performance on the task sce-
nario, user attitudes, use of EPSS,
and time on task.
Performance
The first research question inves-
tigated the effect of different types of
support systems on user performance
while completing a procedural soft-
ware task. Table 1 shows the mean
scores and standard deviations for
performance on the task scenario. The
table reveals that the mean scores
were 10.83 (90%) for the extrinsic
group, 10.06 (84%) for the intrinsic
group, 9.61 (80%) for the external
group, and 8.50 (71%) for participants
who were not provided with a perfor-
mance support system. A one-way
analysis of variance conducted on the
performance scores yielded a signifi-
cant overall difference, F (3, 68)=7.74,
p<.01. The strength of the relationship
between the treatments and the per-
formance scores was large, η2=.25.
Post-hoc tests were conducted to
determine significant differences in
mean performance scores. Multiple
comparisons conducted using the
Tukey method revealed that both
80 PerformaNce ImProVemeNt Quarterly
the intrinsic and extrinsic groups
had significantly higher scores on
the task over the group with no
performance support system. The
difference in the performance scores
between the intrinsic, extrinsic and
external groups was not significant.
The difference between the external
and no support group performance
scores was also not significant.
User Attitudes
The second research question in-
vestigated the effect of performance
support systems on the attitudes of the
participants. A four-item, five-point
Likert-type survey was administered
after completion of the task scenario.
The mean attitude scores by treat-
ment are shown in Table 2. The overall
mean score across the four items was
3.33 (5=strongly agree, 1=strongly
disagree), indicating neutral attitudes
towards the statements about the
performance support systems. The
table reveals that the average rating
was 4.37 for the extrinsic group, 3.71
for the intrinsic group, 3.26 for the
external group, and 1.88 for partici-
pants who were not provided with a
performance support system.
A 4 x 4 MANOVA was conducted
on the data to test for significant
differences. The overall means were
significantly different across the four
treatment groups, Wilks’ Λ=.52, F
(12, 170)=3.91, p<.01. The strength
of the relationship between the treat-
ments and user attitude scores was
moderate, η2=.19.
Follow-up univariate analyses of
variance revealed significant differ-
ences between treatment groups on
all four of the items. Post-hoc tests
were conducted to determine signifi-
cant differences between treatment
groups on the four survey items. Pair-
wise comparisons revealed 14 signifi-
cant differences between groups. On
all four questions, the three groups
with a performance support system
(external, extrinsic and intrinsic) had
significantly more positive attitudes
than the participants in the no EPSS
group. In addition, participants in
the extrinsic group had significantly
more positive attitudes than external
performance support users on two of
the questions: “Information in the
help system was easy to find” and “I
felt confident that I could complete
the task using the help system.”
Use of EPSS
The third research question ex-
amined the effect of treatment on
Table 1
User Performance Across EPSS Types
Support Condition M SD
Extrinsic EPSS 10.83 1.20
Intrinsic EPSS 10.06 1.47
External EPSS 9.61 1.54
No EPSS 8.50 1.69
Note: Maximum total correct=12.
Volume 18, Number 4/2005 81
performance support system use.
The number of times each user
accessed the appropriate help sys-
tem during the task scenario was
tracked by the software application.
The mean use scores are shown in
Table 3. The table reveals that par-
ticipants who were provided with an
extrinsic EPSS accessed the system
an average of 6.72 times, intrinsic
2.94 times, and external 2.11 times.
A one-way analysis of variance con-
ducted on the use scores yielded a
significant overall difference, F (2,
51)=11.44, p<.01. The strength of
the relationship between the treat-
ments and the use of EPSS was large,
η2=.31.
Post-hoc tests were conducted to
determine significant differences
in mean use scores. Multiple com-
parisons conducted using the Tukey
method revealed that the extrinsic
group used the performance sup-
port system significantly more often
than both the intrinsic and exter-
nal groups. The difference in use
of EPSS between the intrinsic and
external groups was not significant.
Time-on-Task
The final research question inves-
tigated the effect of treatment on to-
tal time to complete the task scenario.
This was measured by calculating
the difference between the time at
which participants logged into and
out of the software application. The
data revealed that the external group
spent an average of 7.99 minutes on
the task, the extrinsic group spent
8.66 minutes, no EPSS participants
spent 8.77 minutes, and the intrinsic
group spent 11.25 minutes. A one-
way analysis of variance conducted
on the time on task yielded no sig-
nificant overall difference between
the mean scores.
Table 2
User Preference Across EPSS Types
Extrinsic Intrinsic External No EPSS
The help system provided the
appropriate level of detail to aid
in task completion.
4.44 3.78 3.33 2.00
Information in the help system
was easy to find.
4.44 3.44 3.33 1.82
I felt confident that I could
complete the task using the
help system.
4.33 3.89 3.11 1.82
I would use the help system
again.
4.28 3.72 3.28 1.88
4.37 3.71 3.26 1.88
Note: Questionnaire items were measured on a five-point scale from 1 to 5
(Strong Disagree to Strongly Agree)
82 PerformaNce ImProVemeNt Quarterly
Discussion
This study examined the effect of
various electronic performance sup-
port systems on user performance,
attitudes, use of EPSS, and time on
task. Four treatment groups complet-
ed a task scenario using an intrinsic
EPSS, extrinsic, external, or no sup-
port system at all.
Performance
Performance scores for the ex-
trinsic and intrinsic groups were sig-
nificantly higher than scores for the
group that was not provided with a
performance support system. A closer
examination of the data reveals that
performance of the participants who
received the extrinsic EPSS was 19%
more accurate on the task than those
who were not given an EPSS and
the intrinsic group was 13% more
accurate than the no EPSS group.
Furthermore, performance for the
external EPSS group was 9% more
accurate than the no EPSS group,
although the difference between
these two groups was not statistically
significant. Nevertheless, the positive
improvement in performance scores
for all three treatment groups over the
control group indicates that providing
an EPSS to support task performance
is better than having none at all.
It is not surprising that the ex-
ternal group scored the lowest of
the three EPSS treatments. Since
external performance support sys-
tems are not integrated with the
work interface, users are responsible
for locating the support content that
is relevant to the task. Findings for
user attitudes in the current study
suggest that users may perceive ex-
ternal support systems as less useful
than extrinsic support. For example,
the extrinsic group was more likely
to strongly agree with the statement
that the “Information in the help
system was easy to find” and “I felt
confident that I could complete the
task using the help system.” Further-
more, qualitative data reported by
Spool (2001) indicated that when pro-
vided an external EPSS to conduct a
single search, users located relevant
content only 55% of the time. Spool
noted that the “more times users
searched, the less likely they were to
find what they wanted” (p. 1). Users
who searched twice found relevant
content only 38% of the time; those
who searched three or more times
never found the correct support infor-
mation (Spool, 2001). Combined with
the result of the current study, these
observations suggest that designers
can improve user performance by cre-
Table 3
User Access to EPSS
Support Condition M SD
Extrinsic EPSS 6.72 4.01
Intrinsic EPSS 2.94 2.90
External EPSS 2.11 2.00
No EPSS -- --
Volume 18, Number 4/2005 83
ating performance support systems
that are integrated with the primary
work interface.
Given Gery’s assertion that 80%
of a performance support solution
should be comprised of intrinsic
systems, it is somewhat surprising
that users provided with the intrinsic
EPSS did not perform significantly
better than those given the other
types of support systems. The intrin-
sic EPSS in this study used an “em-
bedded knowledge” approach (Gery,
1995, p. 70). As users encountered
issues or questions in the procedural
task, they referred to embedded
help buttons that provided support
information directly in the primary
workspace. A more robust approach
to intrinsic EPSS would be to physi-
cally redesign and optimize the ap-
plication workspace to logically align
with the users’ workflow through the
task and perhaps even automate cer-
tain manual processes. This type of
human factors engineering approach
would likely have a positive effect
on user performance. Software work
interfaces are often built around
systems or databases rather than hu-
man tasks or processes. By designing
logical interfaces around user work-
flows, as one would in human factors
engineering, work interfaces become
more intuitive and user friendly.
Use of EPSS
Participants provided with an
extrinsic EPSS used their support
system significantly more than those
provided with intrinsic or external
systems. Participants accessed ex-
trinsic EPSS on average two times
more than those provided with an
intrinsic system and three times
more than those provided with an
external system. This result suggests
that designers can increase use of
performance support systems by
integrating them more directly into
the primary work interface.
It is not surprising that use of
the external performance support
system was lowest among the three
treatment groups that received an
EPSS. External systems require
users to search for and locate infor-
mation, sometimes in futility. Spool
(2001) noted that less than 25% of
participants searched more than two
times. Furthermore, “those that did
persevere [by searching more] did not
see positive results” (p. 1). Extrinsic
and intrinsic systems integrate with
the work interface allowing them
to understand work contexts and
deliver the appropriate support in-
formation. This integration reduces
the amount of work required by the
user to find support information and
improves the chances of finding the
correct information on the first at-
tempt.
It is surprising that use of the
intrinsic EPSS was significantly less
than the extrinsic. Once again, this
finding may be due to the fact that
an embedded knowledge approach
was used for the intrinsic EPSS. Most
modern computer displays default to
a resolution of 1024 by 768 pixels.
Software developers are challenged
to fit the necessary components of a
work interface—forms, fields, menus,
icons, and toolbars—in this limited
area. When intrinsic systems employ
an embedded knowledge approach, it
must display support content directly
into the primary workspace. This can
be done through the use of a tooltip,
embedded pane, or resizing the pri-
mary work interface to accommodate
an adjacent window. By doing so, the
intrinsic EPSS must compete with
84 PerformaNce ImProVemeNt Quarterly
the other interface elements for pre-
cious screen real estate. The lack of
space to display the intrinsic EPSS
content may explain the significantly
reduced use of intrinsic support as
compared to extrinsic.
Extrinsic and external perfor-
mance support systems both require
the user to access content stored and
delivered by an outside system. As a
result, they do not compete for limited
space in the primary work interface
as intrinsic systems do.
Performance and Use of EPSS
When considering performance
and use factors together, one may
note that extrinsic performance sup-
port was used significantly more by
participants in the study than in-
trinsic support. However, the greater
use of extrinsic performance support
did not result in a significant perfor-
mance difference between the two
groups.
User Attitudes
Participants in the three perfor-
mance support groups had signifi-
cantly more positive attitudes than
the no EPSS group. This finding can
be attributed to the fact that partici-
pants in the control group were not
provided with any on-task support or
guidance. Participants in this study
indicated a strong preference for any
kind of on-task support from an ex-
ternal, extrinsic or intrinsic system.
This finding further validates the
notion that providing any kind of
EPSS to support task performance
is preferable to having none at all.
In addition, the extrinsic group had
significantly more positive attitudes
than the external users on two survey
questions. This finding is consistent
with results for the performance and
use variables. It is likely due to the
fact the extrinsic participants could
immediately locate relevant support
content on the first request rather
than having to search for and locate
support information using the exter-
nal system.
Performance, Use of EPSS and
User Attitudes
The fact that the extrinsic group
rated their performance support
system significantly higher than
the external group corresponds to
the significant increase in use of the
extrinsic system in comparison to
the intrinsic and external groups.
This relationship highlights the fact
that, for interventions like EPSS
that rely on learner control and self-
regulation, it is important to design
the system in such a way that users
prefer the support system, have easy
access to information, and feel that
they will find the answers they need.
More simply, although certain sys-
tems like intrinsic EPSS may have
psychological benefits over other
designs, if the users feel that the
system is annoying or unhelpful, they
will not use it and therefore will not
maximize the benefits it may offer to
aid task performance.
Limitations
This study focused on a relatively
simple procedural software task.
As a result, the findings may not be
valid when extended to more complex
tasks or work contexts that are not
based in software. While the subjects
in the study had no prior knowledge
of the task, they are all employees
that work in a corporate setting that
requires high computer system use.
Users that may have less computer
experience could perform differently
Volume 18, Number 4/2005 85
when exposed to the performance
support systems offered. In addition,
the data entered by participants as
they completed the task was evaluat-
ed only by the lead researcher in this
study. Future studies that involve
evaluation by a group of observers
may yield different results.
Future Research
Although these findings shed light
on the relative effectiveness of dif-
ferent types of performance support,
many questions remain. The depen-
dent measures used in this study are
a handful of many important factors
to human performance technologists.
Studies that examined a broader
range of measures such as informa-
tion retention, error rate and time-
on-task reduction would be invalu-
able. The task that participants were
asked to complete in this study was
fairly simplistic. A follow-up study
that used a similar but more complex
task would provide better context for
the results of this study. While this
study focused on procedural software
tasks, human performance technolo-
gists are also applying EPSS towards
the improvement of performance that
involves physical tasks. Examples
include aircraft repair, automobile
repair, and manufacturing equip-
ment operations. It would be useful to
extend this study in other settings to
determine if the results can be trans-
ferred to these other work contexts.
Furthermore, a comparative study
that examines the effectiveness of
a broader range of intrinsic EPSS
(embedded knowledge versus hu-
man factors engineering) compared
to other EPSS designs would provide
additional insight into the value
of electronic performance support
systems.
References
Altalib, H. (2002). ROI calculations
for electronic performance support
systems. Performance Improvement,
41(10), 12-22.
Chase, N. (1998). Electronic support cuts
training time [Electronic version].
Quality Magazine. Retrieved January
12, 2005 from http://openacademy.
mindef.gov.sg/OpenAcademy/
Learning%20Resources/EPSS/c16.htm
Davidson, L. (1998). Measure what you
bring to the bottom line. Workforce,
77, 34-40.
Gery, G. (1991). Electronic performance
support systems. Tolland, MA: Gery
Associates.
Gery, G. (1995). Attributes and behaviors
of performance-centered systems.
Performance Improvement Quarterly,
8(1), 47-93.
Gery, G. (2003). Ten years later: a new
introduction to attributes and behav-
iors and the state of performance-
centered systems. In G.J. Dickelman
(Ed.), EPSS revisited: A lifecycle for
developing performance-centered
systems (pp. 1-3). Silver Spring, MD:
International Society for Performance
Improvement.
Hawkins, C.H. Jr., Gustafson, K.L., &
Nielson, T. (1998). Return on invest-
ment (ROI) for electronic performance
support systems: A web-based system.
Educational Technology, 38, 15-22.
Huber, B., Lippincott, J., McMahon, C., &
Witt, C. (1999). Teaming up for per-
formance support: A model of roles,
skills and competencies. Performance
Improvement, 38(7), 10-14.
Morrison, J.E., & Witmer, B.G. (1983). A
comparative evaluation of computer-
based and print-based job perfor-
mance aids. Journal of Computer-
Based Instruction, 10(3), 73-75.
Phillips, J.J. (1997). Handbook of training
evaluation and measurement methods
(3rd ed.). Houston: Gulf Publishing
Company.
Raybould, B. (2000). Building perfor-
mance-centered web-based systems,
information systems, and knowledge
86 PerformaNce ImProVemeNt Quarterly
management systems in the 21st
century. Performance Improvement,
39(6), 69-79.
Rosenberg, M.J. (1995). Performance
technology, performance support, and
the future of training: A commentary.
Performance Improvement Quarterly,
8(1), 94-99.
Sherry, L., & Wilson, B. (1996). Support-
ing human performance across disci-
plines: A converging of roles and tools.
Performance Improvement Quarterly,
9(4), 19-36.
Spool, J.M. (2001). Users don’t learn to
search better [Electronic version]. Re-
trieved April 3, 2005 from http://www.
uie.com/articles/learn_to_search
FRANK NGUYEN is a doctoral
student focusing on performance
support systems. He has managed
the development and deployment
of enterprise e-learning and per-
formance support systems at Intel
Corporation for the last five years.
Nguyen is co-author of Efficiency
in Learning (Jossey Bass, 2005)
and holds a masters in Educa-
tional Technology from Arizona
State University. Telephone: (480)
552-0559. E-mail: frank.nguyen@
asu.edu
JAMES D. KLEIN is a Professor and
Program Leader in the Educational
Technology program at Arizona
State University, Tempe where he
teaches courses on instructional
design, research, and performance
improvement. His most recent
scholarly work includes the book,
Instructor Competencies: Stan-
dards for Face-to-face, Online, and
Blended Settings. Mailing address:
Arizona State University, Box
870611, Tempe, AZ 85287-0611.
Telephone: (480) 965-0349. E-mail:
james.klein@asu.edu
HOWARD SULLIVAN is a profes-
sor in the Division of Psychology
in Education at Arizona State Uni-
versity where he teaches courses in
instructional design, educational
evaluation and educational re-
search. He was the founding Re-
search Editor of Educational Tech-
nology Research and Development.
He has held positions as a Visiting
Scholar at the UCLA Center for
the Study of Evaluation and as a
Senior Fellow at the University of
Melbourne (Australia) Institute of
Education. He was selected by the
ASU Graduate College as the 2002
ASU Outstanding Doctoral Men-
tor. Mailing address: Arizona State
University, Box 870611, Tempe, AZ
85287-0611. Telephone: (480) 965-
0348. E-mail: sully@asu.edu
