Content uploaded by Frank Nguyen
Author content
All content in this area was uploaded by Frank Nguyen on Sep 13, 2022
Content may be subject to copyright.
The Effect of Performance Support
and Training as Performance
Interventions
Frank Nguyen and James D. Klein
For decades, training has been one of the most
common interventions used by organizations
to improve the performance of their employees
and to teach them new ideas and skills. Training
interventions typically involve delivery of predesigned
and developed instruction, practice, and assessment
activities with the goal of increasing learner profi-
ciency on desired behaviors or attitudes. Examples of
training include an instructor-led course to train new
hires on organizational culture and policies, an
individualized Web-based course to train existing
employees in use of a new software application, or a
virtual course to teach managers to develop and grow
their employees. The critical commonality among all
training interventions is the fact that employees are
asked to learn and master the desired outcomes prior
to applying the information to their work.
Organizations have come to rely on training
interventions because they can increase user knowl-
edge, performance, and the results these factors exert
on the entire organization. Arthur, Bennett, Edens,
and Bell (2003) conducted a meta-analysis of 162
published training research studies and found a
medium-to-large effect size for training. Burke and
Day (1986) found similar results. They conducted a
meta-analysis of 70 published managerial training
research studies and found that on average manage-
rial training was moderately effective at improving learner achievement.
These training benefits come at a high cost. A study conducted by the
American Society for Training and Development (ASTD) found that
1
PERFORMANCE IMPROVEMENT QUARTERLY, 00(0) PP. 00–00
&2008 International Society for Performance Improvement
Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/piq.20017
For decades, training has been
one of the most common interven-
tions used by organizations to im-
prove the performance of their
employees and teach them new ideas
and skills. But owing to the cost of
developing and delivering training,
organizations have adopted alterna-
tive ways to enable employee perfor-
mance while reducing the cost and
minimizing the time users spend away
from the job. One alternative is
electronic performance support sys-
tems (EPSS). The present study exam-
ined the effect of EPSS and training on
user performance, time on task, and
time in training. Results revealed that
participants receiving only EPSS and
those receiving training and EPSS
performed significantly better on a
tax preparation procedure than parti-
cipants who received only training.
Training-only users also spent signifi-
cantly more time completing the
procedural task than their counter-
parts in other treatment groups, lead-
ing to a negative correlation between
time on task and performance. The
implications of these findings for the
design and development of perfor-
mance support and training interven-
tions are discussed.
piq 20017
employees at top companies spent an average of 39.31 hours in training in
2005 (Sugrue & Rivera, 2005). The average monetary cost spent on this
training was $1,435 per employee (Sugrue & Rivera, 2005). It is estimated
that more than $51 billion is spent on formal training each year
(Dolezalek, 2005). This estimate incorporates only the cost of designing,
developing, and delivering training courses. It does not include the salary
costs or the lost productivity while employees are pulled off the job to
attend training.
To address these persistent costs, another technology has emerged over
the past decade: electronic performance support systems (EPSS). EPSS gives
users ‘‘individualized on-line access to the full range of ysystems to
permit job performance’’ (Gery, 1991, p. 21). In other words, EPSS can offer
users the information and tools they need to do their job, on the job.
Common instances of EPSS include search engines that allow per-
formers to look for information to solve a pressing performance problem,
embedded help systems that deliver relevant and specific information to
assist a performer in completing a task, a printed job aid with clearly defined
steps to complete an infrequent task, or even systems that simplify or
automate complex tasks for the performer. All of these examples share one
key differentiator compared to training: EPSS interventions focus on
supporting performance while the work is being performed rather than at
some arbitrary point in time beforehand as with training.
This capability has led many performance technologists to pursue
EPSS in the hope of reducing or eliminating the costs of attending class
and delivering online training. For example, the eLearning Guild recently
found that 69% of online training developers planned to embed
performance support content directly into users’ work interfaces and
software tools (Pulchino, 2006). To guide performance technologists
pursuing adoption of EPSS, it is important to examine what is currently
known about these systems.
EPSS as a Performance Intervention
Gery (1995) asserted that performance support systems fall into three
categories: external, extrinsic, and intrinsic support.
External systems store content used to support task performance in an
external database. This content is not integrated within a user’s work
interface. As a result, users are forced to manually locate relevant
information in the external EPSS. Common examples of external
performance support systems include search engines, frequently asked
question (FAQ) pages, and help indexes. In addition, external perfor-
mance support ‘‘may or may not be computer mediated’’ (Gery, 1995,
p. 53). Job aids or documentation are common non-computer-based
performance support interventions.
Extrinsic ‘‘performance support yis integrated with the system, but
is not in the primary workspace’’ (Gery, 1995, p. 51). In other words,
2 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
extrinsic systems integrate with the user’s work interface in such a way
that the EPSS can identify the user’s location in a system or even the exact
task that he or she is working on. With this contextual information, the
extrinsic system can intelligently locate content that may be relevant to
support the task at hand. Like external performance support systems,
though, the content used to support a task is external to the work
interface.
Intrinsic systems give users task support that is incorporated directly
within their work interface. Because of this direct integration with the
interface, Gery asserted that intrinsic EPSS offers ‘‘performance support
that is inherent to the system itself. It’s so well integrated that, to workers,
it’s part of the system’’ (Gery, 1995, p. 51). Under this rather broad
definition, examples of intrinsic performance support systems can range
from tools that automate tasks and processes to user-centered design of
work interfaces that reduce complexity and improve usability, and to
embedded knowledge that is displayed directly in the work interface.
A number of authors have asserted that more highly integrated
intrinsic performance support systems are better than those disconnected
from the user’s work interface. From observations conducted during
software usability studies, Carroll and Rosson (1987) argued that the users
who need support the most, such as novices, are least inclined to use
nonintegrated support systems. Raybould (2000) contended that ‘‘as
support moves further from the tool, it becomes less powerful and more
expensive to use’’ (p. 34). On the basis of these assumptions, Gery (1995)
gave designers a guideline to implement 80% of their support systems as
intrinsic, 10% extrinsic, and the remaining 10% external.
However, very little empirical work has been done to examine the
guidelines. Nguyen, Klein, and Sullivan (2005) studied the most effective
types of performance support systems by testing three types that aligned
with Gery’s intrinsic, extrinsic, and external EPSS categories. Results from
the study indicated that users who were given extrinsic and intrinsic
performance support systems performed significantly better on a software
procedure compared to a control group with no EPSS. In addition, all
users who were given an EPSS had significantly more positive attitudes
than the control group.
In addition, authors have expressed their views on the future of
performance support. Laffey (1995) painted a vision of performance
support systems no longer being static systems with fixed access to fixed
support content. Instead, ‘‘performance support systems will be tailored to
the work environment . . . dynamic in their knowledge and expertise’’
(Laffey, 1995, p. 34). In other words, continually evolving technology will
offer ways for performance support systems to recognize the user, identify
what he or she is doing, and adapt content according to the user’s needs.
Although the vast majority of the work done thus far is nonempirical,
performance technologists have access to literature to help them
understand the types of performance support systems that are possible,
suggestions on the best type of EPSS, and a glimpse of where the field of
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 3
piq 20017
performance support may be heading in the future. At the same time,
training professionals can draw from a long-standing body of literature
from instructional design and technology, human resources, and other
disciplines to assist in solving instructional problems. The same cannot be
said for the intersection of these two fields.
EPSS and Training as Performance Interventions
A widely held belief about using EPSS and training is that by
implementing a performance support intervention one can reduce or even
eliminate the amount of training required to address a performance
problem (Chase 1998; Desmerais, Leclair, Fiset, & Talbi, 1997; Foster 1997). Q1
This notion of reducing training through EPSS and enabling ‘‘day-one
performance’’ has been a major attraction for performance technologists.
Two previous studies that attempted to validate this assumption
produced conflicting results. Bastiaens, Nijhof, Stremer, and Q2Abma (1997)
explored the effectiveness of combinations of computer-based and paper-
based performance support with computer-based and instructor-led
training. The researchers conducted a study with insurance agents and
found that they preferred the paper-based forms over the electronic
software tool as well as the instructor-led training over computer-based
training. These researchers found no significant difference on test
achievement scores, performance, or sales results over a one-year period.
There are several issues with the design of this study that complicate
interpretation of the results. The sample size (36) was relatively small,
particularly for a study that included five treatment groups. The lack of
significant differences or unexpected results from the study may be due to
the lack of statistical power associated with such a small sample size.
Another potential limitation is that the performance support system used
was a computer-based training course coupled with a software applica-
tion; an external EPSS design. Other research indicates that this
nonintegrated form of performance support is not as effective as other
integrated systems (Bailey, 2003; Nguyen, Klein & Sullivan, 2005).
Mao and Brown (2005) also examined the relationship between EPSS
and training. One group of novice users were given one hour of
instructor-led training on Microsoft Access while another group was
provided with a wizard-type EPSS that could be configured either as an
intrinsic system intelligently supplying content to users as they performed
tasks or as an external system presenting content only when searched for.
The users then completed self-paced practice activities where they
attempted ‘‘exercises to be completed with the help of’’ the EPSS (Mao &
Brown, 2005, p. 35). Results indicated that users who were given the EPSS
performed significantly better on an achievement test than those who
received training. They found no significant difference between the two
groups on a procedural task.
However, there are several limitations in interpreting these results.
The researchers did not adequately control for the quality of the content
4 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
in the training and EPSS. The authors noted that ‘‘it is possible that the
performance of the instruction group had more to do with the limitations
of the instructor than with the instructional method’’ (Mao & Brown,
2005, p. 43). In addition, the EPSS group was furnished with practice
activities to complete before attempting the performance task; an
instructional method that would be more characteristic of training than
EPSS. The authors admit that ‘‘ordinarily it is unlikely that learners would
be given this type of direction to use online task support in the real world’’
(Mao & Brown, 2005, p.43).
Purpose of the Current Study
Although the literature examining the efficacy of training and
performance support systems as individual performance interventions is
somewhat robust, research examining the intersection between these two
fields yields few clear and satisfying answers. Studies conducted to this
point have produced conflicting results and are laden with procedural
issues that cloud interpretation of findings.
The lack of concrete information on the relationship between
performance support and training is problematic to performance
technologists, instructional designers, and trainers attempting to combine
interventions. As mentioned earlier, practitioners’ reported interest in
implementing EPSS is substantial (Pulchino, 2006). Because of the lack of
substantive research, no evidence-based guidelines can currently be
offered on how these practitioners can best combine performance support
and training to maximize employee performance.
To address this gap, the research study examined the effect of
implementing EPSS, training, and a combination of these interventions.
Some have asserted that it may be possible to abandon training altogether
when a performance support system is properly implemented (Chase
1998; Desmerais et al., 1997; Foster 1997; Sleight 1993). The researchers of
this study sought to explore this assumption by addressing these research
questions:
Q1
1. What combination of performance support and training
maximizes user performance?
2. What combination of performance support and training
minimizes the time to complete a task?
Method
Design and Participants
A posttest-only, control group design was used in this study.
Participants were randomly assigned to one of three treatment groups:
training-only, EPSS-only, and training and EPSS. One group received
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 5
piq 20017
training prior to completing the performance task. Another group had
access to an EPSS while completing the task but did not receive any prior
training. A third treatment group received both the pretask training as
well as access to the EPSS. Dependent measures included user
performance on the task, time to complete the task, and time in training.
Seventy-eight employees from multiple companies completed the
study. Direct managers at various companies identified some participants.
Additional volunteers for the study were also solicited from local chapters
of the ASTD and the International Society for Performance Improvement
(ISPI). Participants involved in the study represented a broad array of
educational backgrounds: 39 had obtained a master’s degree, 28 held a
bachelor’s degree, six obtained a doctoral degree, three were high school
graduates, and two held an associate degree. The participants represented
a diverse range of job roles: 33 were involved in the education and training
industry, 16 identified themselves as software developers or IT profes-
sionals, 15 were in human resources, six were involved in manufacturing,
three worked in customer service, and five worked in other professions.
Materials
Materials in this study included a task software application, Web-
based training course, performance support system, task scenario, and
pretask demographic survey.
All participants in the study used a Web-based software application
based on a corporate tax return form. As part of the process to submit a
tax return, companies are required to submit data regarding revenue,
profit, costs, and other financial information. These data are typically
recorded on paper-based forms, but participants in this study were asked
to record data and calculations into an online tax form.
As illustrated in Figure 1, the tax software application included a series
of open text fields that required the participant to input relevant data
using information supplied in the task scenario. In total, the corporate tax
scenario required 58 participant inputs. Data entered into the tax software
application were stored in an isolated database for analysis at the
conclusion of the study.
In addition to the tax form, a web-based training course was used to
teach processes, procedures, and principles that are required as part of the
corporate tax preparation task. If the participants were assigned to the
training-only or training and EPSS groups, then the tax software
application required them to complete the Web-based training activity
before attempting the corporate tax performance task.
The Web-based training course included in the study contained nine
introductory screens, 49 information screens, 24 practice screens, and five
concluding screens. In total, the course included 87 screens and took
approximately one hour to complete. The course was divided into five
modules; each one addressed a specific instructional objective. Each
module began with an introductory screen informing the learner of the
objective for the module. In addition, this screen referenced a diagram of
6 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
the corporate tax process, which served as an advance organizer for the
content (see Figure 2). One or more instructional screens addressed each
line in the tax form. Instructional screens presented a brief amount of
content, which includes tax concepts, rules, procedures that must be
completed in the tax form, and examples where relevant. After the
instructional sequence, each module featured scenario-based practice
activities, with the exception of module 1, which offered matching and
multiple-choice practice activities for factual objectives. All practice
activities included appropriate feedback for correct responses or
remediation feedback for incorrect responses.
The Web-based training course was authored in Adobe Captivate.
Instructional screens included images, animations, and text. Audio and
video were excluded from the Web-based training course owing to
usability issues when the content was used for performance support and
bandwidth concerns over the Internet. Participants navigated within
modules using a VCR-like toolbar located at the bottom of each screen.
They navigated between modules using a menu located on the upper left
corner of the screen. This navigation sequence was chosen because it is a
user interface design that is common among current learning manage-
ment systems.
The tax software application was also equipped with a performance
support system for participants in the EPSS-only and training and EPSS
treatments. The EPSS is illustrated in Figure 3. The EPSS used was an
extrinsic context-sensitive help system, which was found in previous
studies to be an effective method to deliver on-the-job support (Bailey,
2003; Nguyen, Klein, & Sullivan, 2005). The opening screen of the tax
FIGURE 1.
Software
application.
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 7
piq 20017
software application gave a brief set of instructions demonstrating how to
access the support system. Help buttons in the form of a question mark
were inserted throughout the tax software application. When participants
clicked on the buttons, their request was recorded in a database and a new
window opened displaying support information associated with the task.
To avoid any effects due to content differences between the training
and EPSS, the content used for the EPSS was derived from the training
course. Web-based training courses can be developed into modular,
reusable learning objects. These learning objects are granular components
of a training course, such as individual modules, lessons, screens, practice
activities, and media elements. These objects can exist independently
from the original training course, which then allows them to be accessed
as isolated single learning offerings or combined to create new training
courses.
As mentioned previously, the Web-based training course contained 49
screens where participants received instructional information. These
information objects were linked directly to individual help buttons
embedded in the tax software application. This concept of reusing the
information objects to the EPSS is illustrated in Figure 4. By using this
approach, identical components from the Web-based training course
could be reused for performance support purposes. Because the actual
learning objects were identical between the two treatments, any
differences that were due to the quality of the content could be eliminated.
The task scenario portrayed a realistic issue that a new employee
might face. It included information that a manager might furnish to a new
finance employee in preparing federal tax submission for a company.
FIGURE 2.
Web-based
training course.
8 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
Corporate tax preparation was chosen as the basis of the scenario because
of the complexity of the task.
Figure 5 shows an excerpt of the task scenario text. The first section
prompted the participant to imagine having recently been hired as a
financial analyst for a small manufacturing company. The second portion
of the task scenario contained an e-mail that was sent to the participant
from the imaginary new manager. In the e-mail, the manager asked the
participant to prepare a tax return for the company. To support this task,
the e-mail contained detailed financial information including income,
expenses, payroll, and other company information. The participant used
these data and any training and support information to complete the tax
return using the tax software application.
FIGURE 3.
Electronic
performance
support system.
FIGURE 4.
Use of information
objects for both
EPSS and
web-based
training.
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 9
piq 20017
All participants completed a pretask demographic survey prior to
participating in the study, as shown in Figure 6. The primary goal of the
instrument was to determine the participants’ prior knowledge about the
task used in the study. The survey was also used to profile the participants’
industry and level of education. Participants could also voluntarily submit
themselves to a gift card drawing, which was used as an incentive to
motivate participants to complete the study.
Criterion and Enroute Measures
Three measures were used in the study: user performance on the task,
time in training, and time to complete task.
FIGURE 5.
Excerpt from
the corporate
tax scenario.
FIGURE 6.
Pre-task
demographic
survey.
10 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
User performance on the task was measured by evaluating the number
of correct items the participants submitted to the tax software application.
As mentioned earlier, the tax scenario required 58 user inputs or
selections. Data entered by the user into the tax software were stored in a
database and subsequently evaluated by the researcher. Participants
received one point for each correct input, with a maximum of 58 points
possible.
The total amount of time spent completing the performance task was
measured by calculating the difference between the time at which
participants logged into and out of the tax software application.
The total amount of time participants spent in the training was
measured by calculating the difference between the time at which
participants logged into and out of the Web-based training course.
Procedures
Because the participants in the study worked in different companies
and were geographically dispersed, various corporate training managers
and local chapters of ASTD and ISPI were asked to recruit participants
from their respective organizations. An e-mail invitation was sent to study
participants. The e-mail instructions advised participants to allocate a
two-hour block of time to complete the study. During this time, they were
asked to avoid distractions from phone calls, e-mail, or co-workers once
they had started the study. They were instructed to complete the task
using only the information in the task scenario and any training or
support that may be provided by the system. The participants were
instructed to submit the information as soon as they felt they had
completed the task.
The e-mail instructions directed participants to the location of the
research study on the Internet. Prior to implementation of the treatments,
participants completed a demographic survey that was used to screen for
prior knowledge of corporate tax preparation. Any individual currently
working in a finance-related role, with a finance-related degree, or with
tax or accounting certifications was not selected to participate in the
study.
If the participant did not have any finance background, the system
randomly assigned him or her into one of three treatment groups
(training-only, EPSS-only, and training and EPSS) and displayed the
appropriate intervention. Participants were not aware that they were
assigned to a different treatment group or that their system was
configured with a training or EPSS intervention. If the participants were
part of the training-only group, they were first directed to take the Web-
based training course. If the participants were part of the EPSS-only
group, the opening screen of the tax software application was presented,
with a brief set of instructions demonstrating how to access the support
system. If the participants were part of the training and EPSS group, they
first took the training course and were then given the performance
support system instructions. Once the participants completed the task
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 11
piq 20017
and submitted the tax information, they were automatically directed by
the system to complete the user attitude survey.
Data Analysis
One-way analysis of variance (ANOVA) was conducted on partici-
pants’ performance on the task, time in training, and time to complete the
task, followed by multiple comparisons where appropriate.
Results
Results reported in this section are for performance on the task
scenario, time on task, and time in training.
Performance
The first research question investigated the effect of EPSS, training,
and a combination of these two interventions on user performance while
completing a tax preparation task. Table 1 shows the mean scores and
standard deviations for performance on the task. The table reveals that the
mean scores were 46.54 (80%) for the training and EPSS group, 43.92
(76%) for the EPSS-only group, and 39.92 (69%) for the training-only
group. A Levene’s test of equality of variances was significant, F(2,
75) 54.49, p5.01. To account for the lack of homogeneity of variances
between the treatment groups, a one-way analysis of variance using the
Welch test was conducted on the performance scores. This test revealed a
significant overall difference, F(2, 46) 522.37, po.01. The strength of the
relationship between the treatments and the performance scores was
large, Z
2
5.31.
Post-hoc tests were conducted to determine significant differences in
mean performance scores. Multiple comparisons conducted using the
Tukey method revealed that participants in the training and EPSS group
and those in the EPSS-only group had significantly higher scores on the
task than those in the training-only group. The difference in the
TABLE 1
User performance by treatment
Treatment group MSD
Training & EPSS 46.54
4.93
EPSS-only 43.92
3.99
Training-only 39.92 2.54
Overall means 43.46 4.76
Note: Maximum total correct 558.
Significant difference between Training-only and Training & EPSS at po.01
Significant difference between Training-only and EPSS-only at po.01
12 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
performance scores between the EPSS-only and training and EPSS groups
was not significant.
Time on Task
The second research question investigated the effect of treatment on
total time to complete the task scenario. This was measured by calculating
the difference between the time at which participants logged into and
out of the tax software application. The mean time on task is shown in
Table 2. The table reveals that the EPSS-only group spent an average of 26
minutes, 39 seconds on the task; the training and EPSS group spent 31
minutes, 32 seconds; and the training-only group spent 1 hour, 29
minutes, and 58 seconds. A Levene’s test of equality of variances was
significant, F(2, 75) 549.99, po.01. To account for the lack of
homogeneity of variances between the treatment groups, a one-way
analysis of variance using the Welch test was conducted on the time-on-
task data. This test revealed a significant overall difference, Fo(2,
45) 511.20, po.01. The strength of the relationship between the
treatments for time on task was strong, Z
2
5.32. The correlation between
time on task and performance on the task scenario was significant at .36,
po.01.
Post-hoc tests were conducted to determine significant differences in
mean time on task scores. Multiple comparisons conducted using the
Tukey method revealed that the training-only group spent significantly
more time on the task scenario than both the EPSS-only and training and
EPSS groups. The difference in time on task between the EPSS-only and
training and EPSS groups was not significant.
Time in Training
The amount of time that participants spent in training was recorded
by calculating the difference between the time that the participants logged
into and out of the Web-based training course. The data revealed that the
training and EPSS group spent an average of 42 minutes, 12 seconds in the
Web-based training course while the training-only group spent 34
minutes, 48 seconds. A Levene’s test of equality of variances was not
TABLE 2
Time to complete the tax scenario by treatment
Treatment group MSD
Training-only 1:29:58 1:05:24
Training & EPSS 0:31:32
0:20:44
EPSS-only 0:26:39
0:17:35
Overall means 0:49:23 0:49:40
Significant difference between Training-only and Training & EPSS at po.01
Significant difference between Training-only and EPSS-only at po.01
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 13
piq 20017
significant. As a result, a conventional one-way analysis of variance was
conducted on time in training, which yielded no significant overall
difference between the mean scores.
Discussion
Performance
Findings revealed that performance scores for the training and EPSS
and EPSS-only groups were significantly higher than scores for the group
that received only training prior to completing the task
scenario. Participants who received training and EPSS
on average completed the task with 80% accuracy; the
EPSS-only group averaged 76%, while the training-
only group 69%. Effect size estimates show that the
strength of the treatments was rather strong. These
findings support the notion that performance support
systems can have a significant impact on user
performance.
One potential reason participants who received
only training had significantly lower performance
scores that those in the other treatment groups may
be a transfer gap between the training course and the
task. The Web-based training course included instructional content that
covered portions of the tax preparation procedure. These instructional
sequences were followed by practice activities, which gave participants a
scenario and asked them to complete that portion of the tax procedure.
To minimize participant attrition, no additional practice activity or
assessment was furnished at the end of the training course to tie the entire
tax procedure together into one whole task. This lack of part-task-to-
whole-task transfer could have had some effect on the training-only
participants’ lower performance on the task. Van Merrie
¨nboer (1997)
suggested that designers should begin their instructional sequences with
part-task procedural practice and then evolve into whole-task problem-
solving exercises. In addition, ‘‘more and more complex versions of the
whole cognitive skill’’ should then be introduced (van Merrie
¨nboer, 1997,
p. 8). The use of this instructional design strategy could have increased the
effectiveness of the training treatment.
Another potential reason for the lower performance of the training-
only group is the volume of information that participants were required to
memorize, recall, and apply at task performance. The Web-based training
course used in this study contained 87 navigation, instructional, practice,
and transitional screens, which included facts, concepts, and procedures
on how to complete the corporate tax preparation task. Another
compounding factor could be the participants’ behavior while in training.
Participants spent an average of 38 minutes and 44 seconds reviewing the
content in the 87 screens of the training course prior to completion of the
Findings revealed that
performance scores for
groups that received
performance support
were significantly higher
than scores for the group
that received only
training prior to
completing the task
scenario.
14 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
tax procedure; an average of 27 seconds per screen. This seemingly short
period of time in the training course suggests that participants skimmed
the training and learned the information at only a superficial level. This
practice would put training-only participants at a disadvantage compared
to EPSS-only and training and EPSS participants, who could quickly
reference and apply information at the moment of need.
This highlights a major benefit of EPSS for procedural tasks such as
the one used in this study: performance support aligns with the adult
learners’ preference for personal relevance. As part of his theory of adult
learning, Knowles (1984) asserted that adult learners (1) are self-directed,
(2) draw on their reservoir of experience for learning, (3) are ready to learn
when they assume new roles, (4) want to solve problems and apply new
knowledge immediately, and (5) are self-motivated to learn. These adult
learning principles imply that participants would learn better if they could
tie the information in the training course to previous experience and
could immediately practice and apply their new learning to an immediate
problem. Although all study participants had access to the same
information, just the EPSS-only and training and EPSS participants could
learn while performing in the context of the work.
Time on Task
The training-only group spent significantly more time completing the
task scenario than participants in the training and EPSS and EPSS-only
groups. In fact, participants who received only the training intervention
spent roughly one hour more, or nearly triple their EPSS-equipped
counterparts. A closer examination of the time-on-task data showed that a
sizeable number of training-only participants spent two to three hours
completing the task.
The tax software application was programmed to prevent participants
from moving backward through the research materials. After the
conclusion of the study, two training-only participants reported frustra-
tion with this design. In an attempt to obtain information to help them
complete the task, these training-only participants (and potentially others)
attempted to return to the Web-based training course for reference.
Attempting to move backward in the software resulted in reported loss of
information entered into the tax software application. As a result, certain
training-only participants had to restart the tax scenario from the
beginning even though they might already have spent a considerable
amount of time in the task.
Participants in this research study completed the task scenario in their
work setting, often in a private office, home office, cubicle, or meeting
room. Many participants admitted that, despite the researchers’ instruc-
tions to avoid distractions such as e-mail, phone calls, or co-workers, they
attempted to address issues by phone or e-mail, or even in some cases
complete other tasks while completing the study. This tendency to attend
to other tasks while completing the tax preparation scenario was reported
across the three treatment groups and is a potential hurdle in conducting
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 15
piq 20017
future research in a virtual, asynchronous manner as attempted in this
study.
Time on Task and Performance
In considering the fact that training-only participants spent nearly
three times as much time on the task as those in the EPSS-only and
training and EPSS groups, it is interesting to note that the training-only
participants scored significantly lower on the performance task than
participants in the other two groups respectively. Correlation tests
between the factors revealed a negative correlation between time on task
and performance of -.36. In short, despite any additional time training-
only participants might have invested in completing the tax preparation
activity, they did not perform any better than their counterparts. In fact,
they performed worse.
Recommendations for Training and Performance Support
Interventions
The findings from this study have important implications for
performance technologists who are considering training or EPSS as
performance interventions. The increase in performance for the partici-
pants who received EPSS over their training-only counterparts suggests
that performance technologists should consider performance support
systems to help mitigate information-related performance problems.
Previous authors have suggested that performance technologists could
implement the EPSS and then reduce or even potentially eliminate
training (Chase 1998; Desmerais et al., 1997; Q1Foster 1997; Sleight 1993).
The fact that training-only participants in this study performed worse on
the tax preparation task than the other two groups lends some credence to
this notion. In addition, participants who were given both training and
EPSS interventions did not perform significantly better than those given
only the EPSS.
In considering EPSS and training as performance interventions, it is
important to note that a performance technologist should consider the
amount of time that is available to deliver prescribed interventions and
the desired level of performance. For example, participants in the EPSS-
only group spent an average of 26 minutes in the tax preparation task,
with an accuracy level of 76%. Meanwhile, participants in the training and
EPSS group spent an average of 74 minutes taking training, using the
EPSS, and completing the performance task. Despite the additional time
invested, they achieved a proficiency level of 80%; just slightly more than
the EPSS-only participants. These data suggest that if a performance
problem is critical and business conditions allow sufficient time to
develop and deliver multiple interventions, then it may be worthwhile to
invest heavily in both training and EPSS interventions. If time is a
constraint or incremental increases in performance are not necessary,
then the benefits derived from a robust training intervention may not be
worth the time or cost invested.
16 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
It is also important to note that development of an electronic
performance support system has been demonstrated to be costly from
both time and monetary perspectives
Q1 (Desmerais et al., 1997; Hawkins
et al., 1998). In this particular study, integration of the EPSS into the tax
software application added an additional 40–50 hours of work. This
number does not factor in the cost to develop content for the EPSS.
Because learning objects from the Web-based training course were reused
for the EPSS, no specialized content development was required to deliver
just-in-time support. When performance technologists are considering
EPSS as an intervention, they should weigh the potential benefits on user
performance and attitudes against the time and
monetary investment that will be required.
Limitations
Several limitations should be considered in
interpreting the results of this study. The task used
in this research was tax preparation, which is largely
a procedure supported by requisite background
facts, concepts, and principles. One potential limita-
tion is that these findings may not extend well to
other work contexts; for instance, these findings may
not be valid for other procedural tasks such as
manufacturing operations or software tasks. They
also may not apply well to principle-based tasks such as preparing an
employee development plan or solving a supply chain problem. Additional
research should be conducted to determine if the interventions used in
this study transfer to other tasks, particularly those that are of longer
duration or higher complexity.
Another limitation is that the study employed an extrinsic EPSS to
support task performance. If another type of EPSS was used, such as an
external search engine or an intrinsic automated wizard, the outcome
would likely be affected. However, doing so could fundamentally change
the tax software application and task so dramatically that it would make
any comparisons between EPSS and non-EPSS groups difficult at best.
When this study was designed, a control group was consciously
excluded. The researchers felt that introduction of a control group could
potentially result in low participation and high attrition among the
corporate employees who participated in the study. If a sample group
could be identified that would better facilitate inclusion of a control
group, the relative contribution of EPSS and training as individual
performance interventions could be clarified.
Finally, the virtual asynchronous method that was used to collect data
from participants introduced distractions during the data collection that
were difficult to control. A number of participants reported answering
phone calls or e-mail while completing the Web-based training or
tax preparation task. Having a more controlled environment could
In considering EPSS and
training as performance
interventions, it is
important to note that a
performance technologist
should consider the
amount of time that is
available to deliver
prescribed interventions
and the desired level of
performance.
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 17
piq 20017
yield different results, particularly for the time-on-task and time-in-
training measures.
Future Research
These findings offer insight into the effect of EPSS and training as
performance interventions, but many more practical questions remain
unanswered. The dependent measures used in this study are a handful of
many factors important to human performance technologists. Studies that
examine a broader range of measures such as information retention, error
rate, or transfer of knowledge to a job task would be valuable.
The present study focused on a tax preparation task. Human
performance technologists are also applying training and EPSS to a broad
spectrum of other performance problems. Examples are physical
procedures such as aircraft repair, automobile repair, and manufacturing
equipment operations, as well as soft-skill principles such as employee
development, managerial and leadership skills, and corporate compliance.
It would be useful to extend the current study to other settings
to determine if the results can be transferred to these other work contexts.
This study measured the performance of participants immediately
following completion of a training course or use of an EPSS. In reality,
performance technologists rely on these interventions to inform
and support employees for an extended period of time while they
are on the job. Using a repeated measures design to determine the
performance and attitudes of participants at some period of time after
the conclusion of the initial data collection would permit valuable insight
into the ability of EPSS and training to facilitate long-term retention and
transfer of information.
Increased use of performance support systems in actual practice
requires that human performance technologists conduct empirical
research to determine the best ways to employ these interventions. As
was done in the current study, additional research should examine the
impact of knowledge support interventions such as EPSS and training on
the performance of users in real-world settings.
Reference
Altalib, H. (2002). ROI calculations for electronic performance support systems.
Performance Improvement,41(10), 12–22.
Arthur, W., Bennett, W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in
organizations: A meta-analysis of design and evaluation features. Journal of
Applied Psychology,88(2), 234–245. Q3
Bailey, B. (2003). Linking vs. searching: Guidelines for use [electronic version]. Retrieved
January 17, 2006, from http://www.webusability.com/article_linking_vs_search-
ing_2_2003.htm.
Bastiaens, T. J., Nijhof, W. J., Streumer, J. N., & Abma, H. J. (1997). Working and learning
with electronic performance support systems: An effectiveness study. Interna-
tional Journal of Training and Development,1(1), 72–78.
Burke, M. J., & Day, R. R. (1986). A cumulative study of the effectiveness of managerial
training. Journal of Applied Psychology,71(2), 232–245.
18 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
Carroll, J. M., & Rosson, M. B. (1987). Paradox of the active user. In J. M. Carroll, (Ed.),
Interfacing thought: Cognitive aspects of human-computer interaction (pp. 80–11).
Boston: Bradford Books/MIT Press.
4
Chase, N. (1998). Electronic support cuts training time [electronic version]. Quality
Magazine. Retrieved January 12, 2005, from http://openacademy.mindef.gov.sg/
OpenAcademy/Learning Resources/EPSS/c16.htm.
Desmarais, M. C., Leclair, R., Fiset, J. V., & Talbi, H. (1997). Cost-justifying electronic
performance support systems. Communications of the ACM,40(7), 39–48.
Dolezalek, H. (2005). The 2005 industry report. Training,42(12), 14–28.
Foster, E. (1997). Training when you need it [electronic version]. Info World. Retrieved
November 17, 2004, from http://openacademy.mindef.gov.sg/OpenAcademy/
Learning%20Resources/EPSS/c1.htm
Gery, G. (1991). Electronic performance support systems. Tolland, MA: Gery Associates.
Gery, G. (1995). Attributes and behaviors of performance-centered systems.
Performance Improvement Quarterly,8(1), 47–93.
Hawkins, C. H., Jr., Gustafson, K. L., & Nielson, T. (1998). Return on investment (ROI) for
electronic performance support systems: A Web-based system. Educational
Technology, 38(4), 15–22.
Knowles, M. S. (1984). Andragogy in action: Applying modern principles of adult
education. San Francisco: Jossey-Bass.
Laffey, J. (1995). Dynamism in Electronic Performance Support Systems. Performance
Improvement Quarterly,8(1), 31–46.
Mao, J., & Brown, B. (2005). The effectiveness of online task support versus
instructor-led training. Journal of Organizational and End User Computing, 17(3),
27–46.
Nguyen, F., Klein, J. D., & Sullivan, H. (2005). A comparative study of electronic
performance support systems. Performance Improvement Quarterly,18(4),
71–86.
Pulchino, J. (2006, April 3). Future direction in e-learning 2006 report. Retrieved March
29, 2006, from the eLearning Guild database http://www.elearningguild.org
Raybould, B. (2000). Building performance-centered Web-based systems, information
systems, and knowledge management systems in the 21st century. Performance
Improvement,39(6), 69–79.
Sleight, D. A. (1993). Types of electronic performance support systems: Their
characteristics and range of designs [electronic version]. Retrieved January 12,
2005, from http://openacademy.mindef.gov.sg/OpenAcademy/Learning Re-
sources/EPSS/c7.htm
Sugrue, B., & Rivera, R. J. (2005, December 6). 2005 State of the industry report.
Retrieved March, 15, 2006, from the American Society for Training and
Development database http://www.astd.org
Van Merrie
¨nboer, J.J.G. (1997). Training complex cognitive skills. Upper Saddle River, NJ:
Educational Technology Publications.
FRANK NGUYEN
Frank Nguyen, PhD., is an assistant professor in the Educational
Technology program at San Diego State University. He has managed
learning and performance strategies for various Fortune companies. He is
co-author of Efficiency in Learning (Jossey Bass, 2006) and has written
articles on e-learning, instructional design, and performance support.
Mailing address: San Diego State University, 1520 Mirabelle Lane, Santee,
CA 92071. Phone: (480) 707–1195. E-mail: frank@frankn.net
Volume 00, Number 0 / 2008 DOI: 10.1002/piq 19
piq 20017
JAMES D. KLEIN
James D. Klein, PhD., is a professor and program leader in the
Educational Technology program at Arizona State University, where he
teaches courses on instructional design, research, and performance
improvement. His most recent scholarly work includes the book
Instructor competencies: Standards for face-to-face, online, and blended
settings. Mailing address: Division of Psychology in Education, Arizona
State University, Tempe, AZ 85287–0611. Phone: (480) 965–0349. E-mail:
james.klein@asu.edu
20 DOI: 10.1002/piq Performance Improvement Quarterly
piq 20017
