Content uploaded by Frank Nguyen
Author content
All content in this area was uploaded by Frank Nguyen on Sep 07, 2022
Content may be subject to copyright.
COMPARING THE IMPACT OF ELECTRONIC
PERFORMANCE SUPPORT AND WEB-BASED TRAINING
James D. Klein
Arizona State University-Tempe
Frank Nguyen
American Express
ABSTRACT
This paper reports on a study conducted to examine the effect of an electronic performance support system (EPSS) and
web-based training (WBT) on user performance, use of EPSS, and time on task. Employees from several different multi-
national companies completed a tax preparation procedure using an EPSS, a WBT course, or a combination of the two
interventions. Findings revealed performance scores for participants receiving the EPSS-only and those receiving EPSS
and WBT were significantly higher than scores for participants who received the WBT-only. Results also indicated a
positive correlation between performance and use of the EPSS. Findings further showed that WBT-only users spent
significantly more time completing the task than their counterparts in other treatment groups leading to a negative
correlation between time on task and performance. Implications for the design and implementation of electronic
performance support and web-based training interventions are provided.
KEYWORDS
Performance Interventions, Electronic Performance Support, Web-based Training
1. INTRODUCTION
As a result of the tremendous time and monetary costs that can be incurred with formal training, learning
organizations have increasingly searched for ways to reduce expenditures and to minimize the time users
spend away from their job. The advent of the Internet has led to the introduction of new digital tools such as
eLearning, podcasting, and virtual classrooms. While these technologies can help reduce the overall cost of
training by eliminating travel, classroom facilities, and even the instructor, they still require employees to
take time away from their job to participate in training for hours, days, or even weeks.
To address these persistent costs, another technology has emerged: electronic performance support
systems (EPSS). EPSS provides users with “individualized on-line access to the full range of…systems to
permit job performance” (Gery, 1991, p.21). In other words, EPSS can provide users with the information
and tools that they need to do their job, on the job. This capability has led many performance technologists to
pursue EPSS in the hope of reducing or eliminating the costs of attending in-class and online training.
Although EPSS has been discussed for almost two decades, very few research studies have been
conducted to measure its effectiveness in digital environments. Results of two meta-analyses indicate
significant, positive results for performance interventions such as job aids to train military personnel
(Duncan, 1985) and clinical decision support systems in medicine (Hunt, Haynes, Hanna, & Smith, 1998).
Two studies of EPSS show positive impacts for performance support. Nguyen, Klein, and Sullivan (2005)
found that users provided with extrinsic and intrinsic performance support performed significantly better on a
software procedure compared to a control group with no EPSS. EPSS users also had significantly more
positive attitudes than the control group. Furthermore, a study by Darabi (2004) indicated that using an EPSS
increased student confidence for performing analysis of organizational problems and their causes.
While these findings are important, they do not answer some of the most basic questions that performance
technologists face when choosing to implement an EPSS. For example, one of the most widely held notions
about EPSS is that implementing such on-the-job support interventions can reduce or even eliminate the
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2009)
267
amount of training that is necessary to address a performance problem. This notion of “day-one performance”
has been a major attraction for performance technologists.
2. PURPOSE
This study examined the effect of implementing EPSS, web-based training, and a combination of these two
performance interventions. It has been assumed that implementing performance support alone is preferable to
a training intervention. In fact, some have asserted that it may be possible to abandon training altogether
when a performance support system is properly implemented. The researchers sought to challenge these
assumptions by addressing the following two research questions: (1) What combination of performance
support and training maximizes user performance? (2) What combination of performance support and
training minimizes the time to complete a task?
3. METHOD
3.1 Participants
Seventy-eight employees from various multi-national companies completed the study. Some participants
were identified by their direct managers. Additional volunteers for the study were also solicited from the
American Society for Training & Development (ASTD) and the International Society for Performance
Improvement (ISPI). Participants involved in the study represented a broad array of educational backgrounds:
39 obtained a masters degree, 28 held a bachelors degree, six obtained a doctoral degree, three were high
school graduates, and two held an associates degree. The participants represented a diverse range of job roles:
33 were involved in the education and training industry, 16 identified themselves as software developers or
IT professionals, 15 were in human resources, six were involved in manufacturing, three worked in customer
service, and five worked in other professions.
3.2 Materials
Materials in this study included a task software application, web-based training course, performance support
system, task scenario, and pre-task demographic survey.
A web-based software application based on a corporate tax return form was used by all participants in the
study. As part of the process to submit a tax return, companies are required to submit data regarding revenue,
profit, costs, and other financial information. While these data are typically recorded on paper-based forms,
participants in this study were asked to record data and calculations into an online tax form. The tax software
application included a series of open text fields that required the participant to input relevant data using
information provided in the task scenario. In total, the corporate tax scenario required 58 participant inputs.
Data entered into the tax software application were stored in an isolated database for analysis at the
conclusion of the study.
In addition to the tax form, a web-based training course was used to teach processes, procedures, and
principles that are required as part of the corporate tax preparation task. If the participants were assigned to
the WBT-only or WBT & EPSS groups, then the tax software application required them to complete the
WBT activity before attempting the corporate tax performance task.
The web-based training course included contained nine introductory screens, forty-nine information
screens, twenty-four practice screens, and five concluding screens. In total, the course included 87 screens
and took approximately one hour to complete. The course was divided into five modules; each one addressed
a specific instructional objective. Each module began with an introductory screen informing the learner of the
objective for the module. In addition, this screen referenced a diagram of the corporate tax process which
served as an advance organizer for the content. Each line in the tax form was addressed by one or more
instructional screens. Instructional screens included a brief amount of content which includes tax concepts,
rules, procedures that must be completed in tax form and examples where relevant. After the instructional
ISBN: 978-972-8924-95-9 © 2009 IADIS
268
sequence, each module provided scenario-based practice activities with the exception of Module 1 which
provided matching and multiple-choice practice activities for factual objectives. All practice activities
included appropriate feedback for correct responses or remediation feedback for incorrect responses.
The web-based training course was authored in Adobe Captivate. Instructional screens included images,
animations and text. Audio and video were excluded from the WBT course due to usability issues when the
content was used for performance support and due to bandwidth concerns over the Internet. Participants
navigated within modules using a VCR-like toolbar located at the bottom of each screen. They navigated
between modules using a menu located on the upper left corner of the screen. This navigation sequence was
chosen as it is a user interface design that is common among current learning management systems.
The tax software application was also equipped with a performance support system for participants in the
EPSS-only and WBT & EPSS treatments. The EPSS is illustrated in Figure 1. The EPSS used was an
extrinsic context-sensitive help system which was found in previous studies to be an effective method to
deliver on-the-job support (Nguyen, Klein, & Sullivan, 2005). The opening screen of the tax software
application provided a brief set of instructions demonstrating how to access the support system. Help buttons
in the form of a question mark were inserted throughout the tax software application. When participants
clicked on the buttons, their request was recorded in a database and a new window opened displaying support
information associated with the task.
Figure 1. Electronic Performance Support System
To avoid any effects due to content differences between the WBT and EPSS, the content used for the
EPSS was derived from the WBT course. WBT courses can be developed into modular, reusable learning
objects. These learning objects are granular components of a training course such as individual modules,
lessons, screens, practice activities, or media elements. These objects can exist independently from the
original WBT course, which then allows them to be accessed as isolated single learning offerings or be
combined in different ways to create new training courses.
As mentioned in previously, the WBT course contained 49 screens where participants received
instructional information. These information objects were linked directly to individual help buttons
embedded in the tax software application. By using this approach, identical components from the WBT
course could be reused for performance support purposes. Since the actual learning objects were identical
between the two treatments, any differences due the quality of the content could be eliminated.
The task scenario portrayed a realistic issue that a new employee might face. It included information that
a manager might provide to a new finance employee when preparing federal tax submission for a company.
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2009)
269
Corporate tax preparation was chosen as the basis of the scenario due to the complexity of the task. The first
section prompted the participant to imagine that they have recently been hired as a financial analyst for a
small manufacturing company. The second portion of the task scenario contained an email that was sent to
the participant from their imaginary new manager. In the email, their manager asked the participant to
prepare a tax return for the company. To support this task, the email contained detailed financial information
including income, expenses, payroll, and other company information. The participant used these data and any
training and support information to complete the tax return using the tax software application.
3.3 Criterion and Enroute Measures
User performance on the task was measured by evaluating the number of correct items the participants
submitted to the tax software application. As mentioned earlier, the tax scenario required 58 user inputs or
selections. Data entered by the user into the tax software were stored in a database and subsequently
evaluated by the researcher. Participants received one point for each correct input with a maximum of 58
points possible. The total amount of time participants spent completing the performance task was measured
by calculating the difference between the time at which participants logged into and out of the tax software
application. The total amount of time participants spent in the WBT course was measured by calculating the
difference between the time at which participants logged into and out of the WBT course.
3.4 Procedures
Since the participants in the study worked in different companies and were geographically dispersed, various
corporate training managers and chapters of ASTD and ISPI were asked to recruit participants from their
respective organizations. An email invitation was sent to study participants. The email instructions advised
participants to allocate a two-hour block of time to complete the study. During this time, they were asked to
avoid distractions from phone calls, email, or co-workers once they had started the study. They were
instructed to complete the task using only the information provided by the task scenario and any training or
support that may be provided by the system. The participants were instructed to submit the information as
soon as they felt they had completed the task.
The email instructions directed participants to the location of the research materials on the Internet. Prior
to the implementation of the treatments, participants completed a demographic survey that was used to screen
for prior knowledge of corporate tax preparation. Any individual currently working in a finance-related role,
with a finance-related degree, or with tax or accounting certifications were not selected to participate in the
study.
If the participant did not have any finance background, the system randomly assigned them into one of
three treatment groups (WBT-only, EPSS-only, WBT & EPSS) and displayed the appropriate intervention.
Participants were not aware that they were assigned to a different treatment group or that their system was
configured with a different WBT or EPSS intervention. If the participants were part of the WBT-only group,
they were first directed to take the WBT course. If the participants were part of the EPSS-only group, the
opening screen of the tax software application was presented providing a brief set of instructions
demonstrating how to access the support system. If the participants were part of the WBT & EPSS group,
they first took the WBT course and were then provided with the performance support system instructions.
4. RESULTS
The first research question investigated the effect of EPSS, WBT, and a combination of these two
interventions on user performance while completing a tax preparation task. The mean scores for user
performance were 46.54 (80%) for the WBT & EPSS group, 43.92 (76%) for the EPSS-only group, and
39.92 (69%) for the WBT-only group. A one-way analysis of variance using the Welch test revealed a
significant overall difference, F (2, 46) = 22.37, p < .01. The strength of the relationship between the
treatments and the performance scores was large, η2 = .31.
ISBN: 978-972-8924-95-9 © 2009 IADIS
270
Post-hoc tests were conducted to determine significant differences in mean performance scores. Multiple
comparisons conducted using the Tukey method revealed that participants in the WBT & EPSS group and
those in the EPSS-only group had significantly higher scores on the task than those in the WBT-only group.
The second research question investigated the effect of treatment on total time to complete the task
scenario. This was measured by calculating the difference between the time at which participants logged into
and out of the tax software application. The EPSS-only group spent an average of 26 minutes, 39 seconds on
the task; the WBT & EPSS group spent 31 minutes, 32 seconds; and the WBT-only group spent 1 hour, 29
minutes, and 58 seconds. A one-way analysis of variance using the Welch test was conducted on the time on
task data. This test revealed a significant overall difference, F (2, 45) = 11.20, p < .01. The strength of the
relationship between the treatments for time on task was strong, η2 = .32. The correlation between time on
task and performance on the task scenario was significant at -.36, p < .01.
Post-hoc tests were conducted to determine significant differences in mean time on task scores. Multiple
comparisons conducted using the Tukey method revealed that the WBT-only group spent significantly more
time on the task scenario than both the EPSS-only and WBT & EPSS groups. The difference in time on task
between the EPSS-only and WBT & EPSS groups was not significant.
The amount of time that participants spent in training was recorded by calculating the difference between
the time that the participants logged into and out of the WBT course. The data revealed that the WBT &
EPSS group spent an average of 42 minutes, 12 seconds in the WBT course while the WBT-only group
spent 34 minutes, 48 seconds. A one-way analysis of variance conducted on time in training yielded no
significant overall difference between the mean scores.
5. DISCUSSION
Findings from the current study support the notion that electronic performance support systems can have a
significant impact on user performance. Results revealed performance scores for participants receiving the
EPSS-only and those receiving EPSS and WBT were significantly higher than scores for participants who
received the WBT-only. One potential reason why participants who received WBT-only had significantly
lower performance scores may be due to a transfer gap between the WBT course and the task. The WBT
included instructional content that covered portions of the tax preparation procedure. Instructional sequences
were followed by practice activities which provided participants with a scenario and asked them to complete
that portion of the tax procedure. To minimize participant attrition, no additional practice activity or
assessment was provided at the end of the WBT to tie the entire tax procedure together into one whole task.
This lack of part-task to whole-task transfer could have had some effect on the WBT-only participants’
lower performance on the task. van Merriënboer (1997) suggested that designers should begin their
instructional sequences with part-task procedural practice and then evolve into whole-task problem solving
exercises. In addition, “more and more complex versions of the whole cognitive skill” should then be
introduced (van Merriënboer, 1997, p. 8). The use of this instructional design strategy could have increased
the effectiveness of the WBT treatment.
Another potential reason for the lower performance of the WBT-only group is the volume of information
that participants were required to memorize, recall, and apply at task performance. The WBT course used in
this study included 87 navigation, instructional, practice, and transitional screens which included facts,
concepts, and procedures on how to complete the corporate tax preparation task. Another compounding
factor could be the participants’ behavior while in training. Participants spent an average of 38 minutes and
44 seconds reviewing the content in the 87 screens of the WBT course prior to completion of the tax
procedure – an average of 27 seconds per screen. This seemingly short period of time in the WBT course
suggests that participants “skimmed” the WBT and only learned the information at a superficial level. This
strategy would put WBT-only participants at a disadvantage as compared to EPSS-only and WBT & EPSS
participants who could quickly reference and apply information at the moment of need.
The WBT-only group spent significantly more time completing the task scenario than participants in the
WBT & EPSS and EPSS-only groups. In fact, participants provided only with the WBT intervention spent
roughly one hour more or nearly triple their EPSS-equipped counterparts. A closer examination of the time
on task data showed that a sizeable number of WBT-only participants spent two to three hours completing
the task.
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2009)
271
The tax software application was programmed to prevent participants from moving backwards through
the research materials. After the conclusion of the study, two WBT-only participants reported frustration with
this design. In an attempt to obtain information to help them complete the task, these WBT-only participants
(and potentially others) attempted to return to the WBT course for reference. Attempting to move backwards
in the software resulted in reported loss of information entered into the tax software application. As a result,
certain WBT-only participants had to restart the tax scenario from the beginning even though they may have
already spent a considerable amount of time in the task.
In considering the fact that WBT-only participants spent nearly three times as much time on the task as
those in the EPSS-only and WBT & EPSS groups, it is interesting to note that the WBT-only participants
scored significantly lower on the performance task than participants in the other two groups respectively.
Correlation tests between the factors revealed a negative correlation between time on task and performance
of -.36. In short, despite any additional time WBT only participants may have invested in completing the tax
preparation activity, they did not perform any better than their counterparts. In fact, they performed worse.
The findings from this study have important implications for performance technologists who are
considering web-based training or EPSS as performance interventions. The increase in performance for the
participants who received EPSS over their WBT-only counterparts suggests that performance technologists
should consider performance support systems to help mitigate information-related performance problems.
When considering EPSS and WBT as performance interventions, it is important consider the amount of
time that is available to deliver prescribed interventions and the desired level of performance. For example,
participants in the EPSS-only group spent an average of 26 minutes in the tax preparation task with an
accuracy level of 76%. Meanwhile, participants in the WBT & EPSS group spent an average of 74 minutes
taking training, using the EPSS, and completing the performance task. Despite the additional time invested,
they achieved a proficiency level of 80% – just slightly more than the EPSS-only participants. These data
suggest that, if a performance problem is very critical and conditions allow sufficient time to develop and
deliver multiple interventions, then it may be worthwhile to invest heavily in both web-based training and
EPSS interventions. If time is a constraint or incremental increases in performance are not necessary, then the
benefits derived from a robust training intervention may not be worth the time or cost invested.
It is also important to note that the development of an electronic performance support system has been
demonstrated to be costly from both a time and monetary perspective. In this particular study, the integration
of the EPSS into the tax software application added an additional 40-50 hours of work. This number does not
factor in the cost to develop content for the EPSS. Since learning objects from the WBT course were reused
for the EPSS, no specialized content development was required to deliver just-in-time support. When
performance technologists are considering EPSS as an intervention, they should weigh the potential benefits
on user performance against the time and monetary investment that will be required.
The increased use of electronic performance support systems and web-based training in actual practice
requires that performance technologists conduct empirical research to determine the best ways to employ
these interventions. As was done in the current study, additional research should examine the impact of
knowledge support interventions such as EPSS and web-based training on the performance of users in real
world settings.
REFERENCES
Darabi, A. (2004). Contributions of an electronic performance support system to learning a complex cognitive skill. In K.
Morgan & M. J. Spector (Eds.), The internet society: Advances in learning, commerce, and security. (pp. 215-225).
Billerica, MA: WIT Press.
Duncan, C. S. (1985). Job aids really can work: A study of the military application of job aid technology. Performance &
Instruction, 24(4), 1-4.
Gery, G. (1991). Electronic performance support systems. Tolland, MA: Gery Associates.
Hunt, D.L., Haynes, R.B., Hanna, S.E., & Smith, K. (1998). Effects of computer-based clinical decision support systems
on physician performance and patient outcomes. Journal of the American Medical Association, 280(15), 1339-1346.
Nguyen, F., Klein, J.D., & Sullivan, H. (2005). A comparative study of electronic performance support systems.
Performance Improvement Quarterly, 18(4), 71-86.
van Merriënboer, J. J. G. (1997). Training Complex Cognitive Skills. Englewood Cliffs: Educational Technology
Publications.
ISBN: 978-972-8924-95-9 © 2009 IADIS
272
