Content uploaded by Frank Nguyen
Author content
All content in this area was uploaded by Frank Nguyen on Sep 07, 2022
Content may be subject to copyright.
The Effect of Performance Support
and Training on Performer Attitudes
Frank Nguyen
While training has been a proven and heavily
relied on intervention to impart job-en-
abling information to performers (Arthur,
Bennett, Edens, & Bell, 2003; Burke & Day, 1986), its
ability to have a positive effect on job performance has
been demonstrated to diminish over time. According
to a study conducted by the Research Institute of
America, performers retain only 58% of the content
delivered by training interventions just 33 minutes
after course completion (Snipes, 2005). Retention
continues to decline from there. After the second
day, just 33% of the information learned in training is
retained. Three weeks later, only 15% can be recalled.
Rackham (1979) found that 87% of the learning ac-
quired from training is lost within 30 days if it is not
supported by an additional intervention.
One intervention that has been adopted by perfor-
mance technologists to provide support is an electro-
nic performance support system (EPSS). These systems provide users with
"individualized on-line access to the full range of ysystems to permit job
performance" (Gery, 1991, p. 21). In other words, performance support
systems seek to provide the right information to the right user at the right
time.
Past research has examined the intersection of these two fields. Bastiaens,
Nijhof, Streumer, and Abma (1997) explored the effectiveness of different
combinations of computer-based and paper-based performance support
with computer-based and instructor-led training. They found that partici-
pants preferred paper-based forms of performance support over electronic
versions as well as instructor-led training over computer-based training.
They found no significant difference on test achievement scores, perfor-
mance, or sales results over a one-year period. Mao and Brown (2005)
95
PERFORMANCE IMPROVEMENT QUARTERLY, 22(1) PP.95 –114
&2009 International Society for Performance Improvement
Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/piq.20047
While training has been a proven
and heavily relied on intervention to
impart job-enabling information to
performers, its ability to have a positive
effect on job performance has been
demonstrated to diminish over time.
One intervention that has been
adopted by performance technologists
to provide ongoing support is an elec-
tronic performance support system
(EPSS). The study presented here exam-
ined the effect of EPSS and training on
user attitudes. Results revealed that
participants receiving only EPSS and
those receiving training and EPSS had
significantly higher attitudes than par-
ticipants who received only training.
Recommendations on how to best
combine and implement these perfor-
mance interventions based on these
data are discussed.
conducted a study comparing the effect of instructor-led training and
performance support on learning how to use Microsoft Access. They found
that users provided with an EPSS performed significantly better on an
achievement test than those provided with training. They found no sig-
nificant difference between the two groups on a procedural task. Nguyen and
Klein (2008) examined the effect of training and performance support
individually and as combined interventions. They reported that participants
who received only EPSS and both training and EPSS together performed
significantly better on a procedural task than those who received just
training.
These findings provide some evidence to support the widely held belief
that implementing a performance support intervention can reduce or
perhaps even eliminate the amount of training necessary to address a
performance problem (Chase, 1998; Desmarais, Leclair, Fiset, & Talbi
1997; Foster, 1997). However, they do not provide any evidence-based
guidelines to assist performance technologists to determine when a parti-
cular problem can be safely addressed by just training, EPSS, or a combina-
tion of EPSS and training and what proportion of those two interventions
may be required.
The research study examined here sought to address this gap by
exploring the following research question: What combination of perfor-
mance support and training do users prefer? By revealing performers’
attitudes toward these two interventions, the psychological factors that
influence the use of such systems to support performance should be revealed.
Method
Design and Participants
This research study employed an identical design to the study previously
reported by Nguyen and Klein (2008). A posttest-only control group design
was used. Participants were randomly assigned to one of three treatment
groups: training only, EPSS only, or training and EPSS. One group received
training prior to completing the performance task. Another group had access
to an EPSS while completing the task but did not receive any prior training. A
third treatment group received both the pretask training and access to the
EPSS. The primary dependent measure was user attitudes.
Seventy-eight employees from multiple companies completed the study.
Some participants were identified by their direct managers at various
companies. Additional volunteers for the study were also solicited from
local chapters of the American Society for Training and Development
(ASTD) and the International Society for Performance Improvement (ISPI).
Participants involved in the study represented a broad array of educational
backgrounds: 39 had a master’s degree, 28 held a bachelor’s degree, 6 had
obtained a doctoral degree, 3 were high school graduates, and 2 held an
associate degree. The participants represented a diverse range of job roles: 33
were involved in the education and training industry, 16 identified them-
96 DOI: 10.1002/piq Performance Improvement Quarterly
selves as software developers or information technology professionals, 15
were in human resources, 6 were involved in manufacturing, 3 worked in
customer service, and 5 worked in other professions.
Materials
The materials in this study were a tax software application, a Web-based
training course, a performance support system, a task scenario, a posttask
attitude survey, and an interview questionnaire.
Tax Software Application. A Web-based software application based on a
corporate tax return form was used by all participants in the study. As part of
the process to submit a tax return, companies are required to submit data
regarding revenue, profit, costs, and other financial information. While these
data are typically recorded on paper-based forms, participants in this study
were asked to record data and calculations into an online tax form as
illustrated in Figure 1.
Web-Based Training Course. In addition to the tax form, a Web-based
training course was used to teach processes, procedures, and principles that
are required as part of the corporate tax preparation task. If the participants
were assigned to the training-only or training and EPSS groups, the tax
software application required them to complete the Web-based training
activity before attempting the corporate tax performance task.
The Web-based training course consisted of 9 introductory screens, 49
information screens, 24 practice screens, and 5 concluding screens. In total,
FIGURE 1. SOFTWARE APPLICATION
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 97
the course contained 87 screens and took approximately 1 hour to complete.
The course was divided into five modules, each addressing a specific instruc-
tional objective. Each module began with an introductory screen informing
the learner of the objective for the module. In addition, this screen referenced
a diagram of the corporate tax process, which served as an advance organizer
for the content(see Figure 2). Each line in the taxform was addressed by one or
more instructional screens. Instructional screens included a brief amount of
content about tax concepts, rules, and procedures that must becompleted in
tax form and examples whererelevant. After the instructional sequence, each
module provided scenario-based practice activities, with the exception of
module 1, which provided matching and multiple-choice practice activities
for factual objectives.All practice activities included appropriatefeedback for
correct responses or remediation feedback for incorrect responses.
Performance Support System. The tax software application was also equipped
with a performance support system for participants in the EPSS-only and the
training and EPSS treatments. The EPSS is illustrated in Figure 3. The EPSS
used was a context-sensitive help system that was found in previous studies
to be an effective method to deliver on-the-job support (Bailey, 2003;
Nguyen, Klein, & Sullivan, 2005). The opening screen of the application
provided a brief set of instructions demonstrating how to access the support
system. Help buttons in the form of a question mark were inserted
throughout the tax software application. When participants clicked on the
buttons, their request was recorded in a database, and a new window opened
displaying support information associated with the task.
FIGURE 2. WEB-BASED TRAINING COURSE
98 DOI: 10.1002/piq Performance Improvement Quarterly
To avoid any effects due to content differences between the training and
EPSS, the content used for the EPSS was derived from the training course.
Web-based training courses can be developed into modular, reusable
learning objects. These learning objects are granular components of a
training course, such as individual modules, lessons, screens, practice
activities, or media elements. These objects can exist independent of the
original training course, which then allows them to be accessed as single
learning offerings or be combined in different ways to create new training
courses. These information objects were linked directly to individual help
buttons embedded in the tax software application. With this approach,
identical components from the Web-based training course could be reused
for performance support purposes. Since the actual learning objects were
identical between the two treatments, any differences due to the quality of the
content could be eliminated.
Task Scenario. The task scenario portrayed a realistic issue that a new
employee might face. It included information that a manager might
furnish to a new finance employee in preparing federal tax submission for
a company. Corporate tax preparation was chosen as the basis of the scenario
because of the complexity of the task.
Figure 4 shows an excerpt of the task scenario text. The first section
prompted the participant to imagine having recently been hired as a financial
analyst for a small manufacturing company. The second portion contained
an e-mail sent to the participant from the imaginary new manager. In the
FIGURE 3. ELECTRONIC PERFORMANCE SUPPORT SYSTEM
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 99
e-mail, the manager asked the participant to prepare a tax return for the
company. To support this task, the e-mail contained detailed financial
information about income, expenses, and payroll and other company
information. The participant used these data and any training and support
information to complete the tax return using the tax software application.
Posttask Attitude Survey. An eight-item attitude survey was administered
after participants completed the task. The survey is shown in Exhibit 1. The
survey was divided into three sections: usefulness of training and support,
quantity of learning, and satisfaction with the system.
The first section of the survey examined the usefulness of the training and
support content. The first question asked participants how easy it was to find
the information they needed to perform the task. The second question
examined whether participants felt training and support content had the
appropriate level of detail.
The next section addressed the quantity of learning provided for the task.
The third question asked participants if the amount of training provided was
sufficient to complete the task, and the fourth question asked if the amount of
help during the task was sufficient.
The third set of questions gauged the participants’ satisfaction with the
system. The fifth question asked participants if they felt confident complet-
ing the task. The sixth examined whether the participants would like to use
software applications such as the one demonstrated in the study in the future,
and the seventh asked if they would recommend one for their organization.
The final question prompted participants to rate the amount of time they
spent learning how to perform the task as about right, too much, or too little.
The Cronbach’s alpha reliability coefficient was .81 for the attitude survey.
Posttask Interview Questionnaire. A 10-item questionnaire was used to
interview selected participants 1 to 5 days after they completed the
FIGURE 4. TASK SCENARIO
100 DOI: 10.1002/piq Performance Improvement Quarterly
EXHIBIT 1 POSTTASK ATTITUDE SURVEY
This brief survey is the last part of the study. It will be used to compare your preferences between different
types of learning support.
1. It was easy to find the information I needed to complete the tax scenario.
Strongly agree
Agree
Disagree
Strongly disagree
2. The learning support provided was at the appropriate level of detail to aid in completing the tax scenario.
Strongly agree
Agree
Disagree
Strongly disagree
3. I received enough training to successfully complete the tax scenario.
Strongly agree
Agree
Disagree
Strongly disagree
4. I received enough help to successfully complete the tax scenario.
Strongly agree
Agree
Disagree
Strongly disagree
5. I felt confident that I could successfully complete the tax scenario.
Strongly agree
Agree
Disagree
Strongly disagree
6. In the future, I would like to use learning support such as the one demonstrated in this study.
Strongly agree
Agree
Disagree
Strongly disagree
7. I would recommend the type of learning support used in this study to my organization to help employees
perform better.
Strongly agree
Agree
Disagree
Strongly disagree
8. The total amount of time I spent learning how to perform the tax scenario was:
Too little
About right
Too much
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 101
research study. The questionnaire is shown in Exhibit 2. The interview
questions were divided into three sections: the first three questions were
targeted for participants who completed the study in the training-only and
the training and EPSS groups, questions 4 to 6 targeted participants who
completed the study in the EPSS-only and the training and EPSS treatment
EXHIBIT 2 INTERVIEW QUESTIONNAIRE
Interview questions for treatment groups receiving Training (Training-only, Training & EPSS)
Procedure: Provide interview participants with a link to the web-based training course, ask them to
launch the course, and then ask the following questions.
1. What did you like about the training course?
2. What did you not like about the training course?
3. Explain how you went through the course.
Interview questions for treatment groups receiving EPSS (EPSS-only, Training & EPSS)
Procedure: Provide interview participants with a link to the tax software with EPSS, ask them to launch
the application, and then ask the following questions.
4. What did you like about the online help?
5. What did you not like about the online help?
6. Explain how you used the online help.
Interview questions for all treatment groups
7. Did you feel the learning support help you complete the task? Why?
8. What motivated you to complete the task?
9. What kinds of issues did you run into while trying to complete the research study?
10. What would you recommend changing to encourage people to complete a study like this in
the future?
FIGURE 5. TRAINING AND PERFORMANCE SUPPORT MATRIX
102 DOI: 10.1002/piq Performance Improvement Quarterly
groups, and questions 7 to 10 were common questions for all participants
who completed the research study.
Criterion Measures
The primary measure in this study was the users’ attitudes toward the
performance support and/or training intervention provided.
The eight-item survey shown in Exhibit 1 was administered to measure
participant attitudes toward the effectiveness of the training and support
interventions provided in aiding them to complete the task. For most
questions, respondents used a 4-point Likert scale (4 5strongly agree,15
strongly disagree) to rate their attitudes regarding the effectiveness of the
interventions. On question 8, respondents were asked to rate the amount of
time they spent learning how to perform the task. As a result, respondents
were provided with only three options: "about right," "too much," and "too
little." The internal consistency reliability of this survey was determined
using a Cronbach alpha procedure.
To collect qualitative data regarding user attitudes, seven participants
from each of the three treatment groups were selected to participate in
posttask interviews. Interview participants were identified using interest data
collected using a pretask demographic survey. In an attempt to ensure that
the participants’ comments were recent and relevant, participants were
interviewed 1 to 5 days after they participated in the study. Study participants
were sent hyperlinks to view the Web-based training course or performance
support system based on their treatment group. Participant feedback was
collected by the lead researcher using the posttask interview questionnaire.
These data were compiled and used to inform the data from the user attitude
survey.
FIGURE 6. THE RELATIONSHIP BETWEEN TRAINING AND PERFORMANCE SUPPORT
CONTENT
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 103
Procedures
Since the participants in the study worked in different companies and
were geographically dispersed, various corporate training managers and
local chapters of ASTD and ISPI were asked to recruit participants from their
respective organizations. An e-mail invitation was sent to study participants.
The e-mail instructions directed participants to the location of the research
study on the Internet. Prior to the assignment of treatments, participants
completed a demographic survey that was used to screen for prior knowledge
of corporate tax preparation. Any individual currently working in a finance-
related role, with a finance-related degree, or with tax or accounting
certifications was not selected to participate in the study.
The system randomly assigned participants who did not have any finance
background into one of three treatment groups (training-only, EPSSonly, and
training and EPSS) and displayed the appropriate intervention. Participants
were not aware that they were assigned to a different treatment group or that
TABLE 1 USER ATTITUDE SCORES BY TREATMENT
ITEM
TRAINING-
ONLY EPSS-ONLY
TRAINING
& EPSS
OVERALL
MEANS
(n526) (n526) (n526) (n578)
It was easy to find the information I
needed to complete the tax scenario.
1.73
ab
2.62
ac
3.15
bc
2.50
The learning support provided was at
the appropriate level of detail to aid
in completing the tax scenario.
1.54
ab
2.58
ac
3.12
bc
2.41
I received enough training to success-
fully complete the tax scenario.
2.27
db
1.92d
c
2.77
bc
2.32
I received enough help to successfully
complete the tax scenario.
1.77
ab
2.58
a
2.96
b
2.44
I felt confident that I could successfully
complete the tax scenario.
2.04
ab
2.54
a
2.85
b
2.47
In the future, I would like to use
learning support such as the one
demonstrated in this study.
2.23
ab
2.81
ac
3.35
bc
2.79
I would recommend the type of
learning support used in this study
to my organization to help
employees perform better.
1.77
ab
3.12
a
3.31
b
2.73
Overall Means 1.91 2.59 3.07 2.52
Note. Items were measured on a 4-point Likert scale (1 5strongly disagree,45strongly agree).
a
Significant difference between Training-only and EPSS-only at po.01.
b
Significant difference between Training-only and Training & EPSS at po.01.
c
Significant difference between EPSS-only and Training and EPSS at po.01.
d
Significant difference between Training-only and EPSS-only at po.05.
104 DOI: 10.1002/piq Performance Improvement Quarterly
their system was configured with a different training or EPSS intervention. If
the participants were part of thetraining-onlygroup, they were firstdirected to
take the Web-based training course. Ifthe participants were part of the EPSS-
only group, the opening screen of the tax software application was presented,
providing abrief set of instructions demonstrating how to accessthe support
system. If the participants were partof the training and EPSS group, they first
took thetraining courseand were then provided withthe performancesupport
system instructions. Once the participants completed the task and submitted
the tax information, they were automatically directed by the system to
complete the user attitude survey.
Data Analysis
Multivariate analysis of variance (MANOVA) was conducted on a
portion of the data from the attitude survey, followed by univariate ANOVAs
where appropriate. A chi square test was conducted on nonparametric data
from the attitude survey. A Cronbach alpha procedure was conducted on the
user attitude data to compute the reliability of the posttask survey.
Results
Survey Data
The primary research question investigated the effect of EPSS and
training interventions on the attitudes of participants. An eight-item survey
was administered after completion of the tax procedure task. The first seven
items included a 4-point Likert-type scale for participants to respond to
statements about the training and performance support interventions (4 5
strongly agree,15strongly disagree). Data reported in Table 1 reveal that the
average overall attitude rating was 3.07 for the training and EPSS group, 2.59
for the EPSS-only group, and 1.91 for the training-only group.
A 3–by-7 MANOVA was conducted on these seven items to test for
significant differences. The overall means were significantly different across
the three treatment groups: Wilks’ l5.18, F(18, 134) 513.14, po.01. The
strength of the relationship between the treatments and user attitude scores
was strong: e1215.57.
Follow-up one-way analyses of variance revealed significant differences
among treatment groups on all seven of the survey items. Post hoc tests were
TABLE 2 USER ATTITUDE FREQUENCIES FOR TIME SPENT
LEARNING HOW TO COMPLETE THE TASK
RATING TOO LITTLE ABOUT RIGHT TOO MUCH
Training-only 16 0 10
EPSS-only 4 17 5
Training and EPSS 3 21 2
Total 23 38 17
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 105
TABLE 3 SUMMARY OF PARTICIPANT POST-TASK INTERVIEW RESPONSES
ITEM RESPONSES
Interview questions for training-only and training and EPSS groups
1. What did you like about the training course?
Practice activities 6
Tax process flow diagram 4
Navigation 2
Explanation of navigation 1
Organization 1
2. What did you not like about the training course?
Tax topic 6
Too much detail 5
Did not provide a job aid 2
Could not apply the learning immediately 1
Too much information 1
3. Explain how you went through the training course.
Proceeded through course in a linear fashion 8
Read each topic thoroughly at first, skimmed towards the end 7
Read certain topics thoroughly, skimmed other topics 4
Skimmed all topics 3
Interview questions for EPSS-only and training and EPSS groups
4. What did you like about the online help? 6
Information available at point of need 5
Did not have to remember the information 2
Information chunked 1
Did not have to learn the information in a training course
5. What did you not like about the online help?
Information did not provide enough detail to support successful task completion 8
Information was not clear 2
Tax topic 2
Could not use external sources for information (IRS, Google) 1
Wait time while help content loaded 1
106 DOI: 10.1002/piq Performance Improvement Quarterly
conducted on these items. Pairwise comparisons revealed 20 significant
differences among groups. On six questions, participants in the EPSS-only
group and those in the training and EPSS group had significantly more
positive attitudes than participants in the training-only group. In addition,
participants in the training and EPSS group had significantly more positive
attitudes than those in the EPSS-only group on four questions.
6. Explain how you used the online help.
Used when did not know the answer 8
Used to validate prior knowledge 5
Used for each line item 1
Interview questions for all participants
7. Did you feel the learning support helped you complete the task?
Why?
No, could not remember information from training 8
No, not enough detail in help 5
No, tax content was confusing 5
Yes, prefer to learn by doing 3
8. What motivated you to complete the task?
Want to help lead researcher 15
Want to complete any tasks that are started 5
Want to figure out the task 1
9. What kinds of issues did you run into while trying to complete the research study?
No issues reported 11
Distractions in work environment 5
Information on certain help screens disappeared after 30 seconds 2
Went back to training course and lost data in tax software 2
Popup blocker prevented help screens from opening 1
10. What would you recommend changing to improve this study in the future?
No suggestions provided 15
Select a different topic that is more valuable to participants 4
Change online help to an automated wizard 1
Conduct survey research 1
TABLE 3 SUMMARY OF PARTICIPANT POST-TASK INTERVIEW RESPONSES
(CONTINUED)
ITEM RESPONSES
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 107
The eighth item on the survey presented three choices that measured the
participants’ perception toward the amount of time they spent learning how
to perform the tax scenario. The response frequency for this item is shown in
Table 2. The table reveals that 38 participants (49%) thought that the amount
of time they spent learning was about right, 23 participants (29%) thought
that they had spent too little time, and 17 (22%) thought that they had spent
too much time. Closer examination of the data shows that the majority of
participants in both the training and EPSS group (81%) and in the EPSS-only
group (65%) thought that the time they spent learning how to do the task was
about right. But no participants in the training-only group indicated that the
amount of time they spent learning how to perform the task was about right.
Sixteen participants in the training-only group (62%) thought that they spent
too much time learning, and the remaining 10 participants (38%) thought
that they did not spend enough time. A chi square test revealed that the
difference in proportions between the treatment groups was significant: w12
1(2, N578) 59.00, po05.
Interview Data
Seven participants from each treatment condition were interviewed to
determine their opinions of the program (n521). User interview responses
to the ten questions are summarized and presented in Table 3.
Participants who received training as part of their treatment were first
asked what they liked about the Web-based training course. Six of the 14
training participants indicated that they liked the practice activities. Four
participants identified the tax process flow diagram used at the beginning of
each unit of instruction as a helpful advanced organizer.
When asked about what they did not like about the Web-based course,
six participants who received training indicated that they did not like the tax
topic used as the basis of course. Five participants complained that the course
had too much detailed information.
Training-only and training and EPSS group participants were then asked
to explain how they proceeded through the training course. Seven partici-
pants indicated that they started out reading the pages of the course
thoroughly at first but then skimmed the content as they progressed. Four
participants reported that they selected topics that they felt were important
and read them thoroughly; they skimmed less meaningful topics. Three
participants admitted that they skimmed through the course without reading
the topics thoroughly. In addition, eight participants added that they
proceeded through the course in a linear fashion.
Participants who received the EPSS as part of their treatment were asked
what they liked about the online help system. Six of the 14 EPSS participants
indicated that they liked the fact that support information was available at the
point of need. Five participants liked the fact that they did not have to
remember the information required to complete the tax preparation task.
When asked what they did not like about the EPSS, eight participants felt
that the online help system did not provide information that was detailed
enough to support successful completion of the task. Two participants found
108 DOI: 10.1002/piq Performance Improvement Quarterly
the information unclear, and two others complained that they did not like the
topic of the tax scenario.
EPSS-only and training and EPSS group participants were then asked to
explain how they decided to use the online help system during the task. Eight
participants reported that they used the EPSS when they did not know the
answer. Five indicated that they used the EPSS when they felt they knew the
answer but wanted to validate their assumptions.
All 21 interview participants were then asked if they felt that sufficient
learning support was provided to help them complete the task. Eight of 14
training participants indicated that theycould not remember enough informa-
tion from the training course to successfully complete the tax preparation
scenario. Five of 14 EPSS participants felt that they did not have enough support
because the help content did not provide enough detail. Three of 14 EPSS
participants found the learning support sufficient. Five participants from all 21
interviewees reported that the topic of taxes was confusing to them.
When asked what motivated them to complete the tax scenario, 15 of 21
interviewees across all treatment groups noted that they wanted to help the
lead researcher complete the study. Meanwhile, five indicated that their
preference was to complete any task that they start.
Participants were asked to identify any issues that they may have
experienced while completing the study. While most participants did not
identify any problems, five participants mentioned that they had difficulty
focusing on the tax scenario due to distractions in their work environment
such as e-mail, phone calls, and interruptions from coworkers. Two EPSS
participants pointed out that content in one of the help screens in the EPSS
disappeared after 30 seconds. In addition, two training participants com-
plained that they attempted to return to the Web-based training course for
reference while they were completing the tax preparation task and lost all of
their data in the process. One participant noted that a pop-up blocker
installed on the computer prevented the online help windows from appear-
ing at first, and the participant corrected the issue.
Finally, participants were asked for recommendations to improve the
research study. Fifteen of the 21 interviewees did not offer any suggestions.
Four participants suggested that the research be focused on a topic that
would be more meaningful to potential participants. One participant
suggested that the EPSS be changed from an online help system to a wizard
that automated the tax preparation task. Another participant noted that
research in industry is typically conducted using survey instruments that
participants can spend a short amount of time completing. The participant
did not realize that she would be asked to complete a task.
Discussion
Key Themes
Several key themes can be interpreted from the results of the attitude
survey and participant interviews.
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 109
Too Much Training. It is clear that participants provided only with training
were generally less satisfied than the other groups were. During follow-up
interviews, six participants said they felt overwhelmed by the amount of
information or detail provided in the training course. The Web-based
training course used in this study contained 87 navigation, instructional,
practice, and transitional screens. Participants spent an average of
38 minutes 44 seconds reviewing the content in the training course prior
to completing the tax procedure. This volume of
information likely contributed to the lower
attitudes of the training-only group. One
participant noted, "It was very easy to get lost in
the explanations." Because of the overwhelming
amount of information, some participants found
it difficult to remember the content when they were
asked to complete the tax preparation task. Said
one, "It was difficult for me to fill out all of the information in the form
because I could not remember all the details [from the training course]."
Another participant complained, "This is one of the problems I have with
typical training. Sometimes I miss details that are critical. It’s hard to
differentiate what’s important from what I already know."
A few training-only participants even asked for some type of support
system to provide information during task completion. Typical responses
were: "Itwould have been nice to havesome cheat sheet to findthe things that I
remembered reading,but I didn’t remember what thedetails were" and"I think
the biggestthing [that would have assisted me inthis task] would be some type
of help during the task because the steps can get rather convoluted."
One side effect of the overwhelming amount of information provided
during training was the participants’ tendency to process the information at a
superficial level. Findings revealed that 13 of 21 participants interviewed
admitted that they skimmed through portions of or, in a few cases, all of the
training course. One participant noted, "I started out reading everything and
then towards the end, or for specific sections, I found myself skimming."
Another participant flatly stated, "I didn’t pay attention to the last half of the
training course."
When asked if their course-taking habits in the study were different from
what they do in a normal work or training setting, participants felt that their
behaviors were typical: "I approached this [training course] the way as I would
normally," said one. Others provided more insight into their online training
habits: "In general, I usually go through it fairly quickly. I tend to skim and
read things at a high-level." One participant admitted, "When I take required
courses, I sometimes skip to the end to get credit and meet the requirements."
Another participant felt that the amount of content was problematic: "I
found myself skimming through some of it just because there was no way I
could possibly remember all of the detail."
The Strengths of Training. Despite calls from some authors (Chase, 1998;
Desmarais et al., 1997; Foster, 1997) to reduce or eliminate
It is clear that participants
provided only with training
were generally less
satisfied than the other
groups were.
110 DOI: 10.1002/piq Performance Improvement Quarterly
training, participants did identify attributes of their training intervention that
seemed to add value. Some participants commented that they liked the tax
process flow diagram used at the beginning of each unit of instruction. The
process served not only as an advanced organizer for the training course but
also as a guide or checklist for the participants as they completed the task.
Others indicated that more than any other aspect of the training course, the
practice activities helped to prepare them to perform the tax procedure. Due
to the high amount of user control provided by EPSS, it would be difficult to
provide performers with comparable prior knowledge about the process or
opportunities to practice the task if a training intervention was not provided.
Not Enough Support. While training-only participants complained that they
received an overwhelming amount of information, eight participants who
received EPSS felt that the system did not provide information that was
detailed enough to support successful completion of the task. This finding is
particularly interesting as the content delivered to participants in both the
training and EPSS treatments was identical. This suggests that while the
volume ofinformation delivered duringpretask trainingshould be minimized,
access to support content during job performance should be much broader.
Recommendations for Training and Performance Support
Interventions
Based on the findings of this study, several recommendations can be
offered to assist performance technologists when implementing training and
performance support as performance interventions:
Limit the amount of information delivered during training. Participants
reported feeling overwhelmed by the quantity of content they received and
skimmed through the training course as a result. Performance technologists
and instructional designers should carefully identify the content to be
delivered using a training intervention in the analysis phase. Two potential
factors to consider are frequency and criticality.
As shown in Figure 5, tasks and information that are used frequently and
are mission critical should be delivered primarily through a training interven-
tion. A performer who did not master this content prior to task performance
probablywould not be able to complete thetask correctly, potentially harming
self, others, or even the organization as a whole. Tasks and information that
performersdo not routinely work with and are not critical to the organization
would likely serve as distractersin a training intervention. As such, they can be
safely delivered exclusively through performance support. Objectives that are
either high frequency and low criticality or low frequency and high criticality
should also be trained, although the level of mastery required may be variable.
Since such tasks may not beperformed routinely or the consequences of error
significant, it is also important to provide redundantperformance support. By
doing so, performers can refresh what they may have learned during training
while on the job.
Provide performers with robust practice during training that incorporates
the EPSS. Practice has been regarded as a powerful instructional method
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 111
(Merrill & Salisbury, 1984, Vinsonhaler & Bass, 1972). Participants in this
study confirmed that the practice provided them an opportunity to prepare
for and build confidence toward successfully completing the task. If possible,
performance technologists and instructional designers should incorporate
any performance support systems that will be available to performers on the
job into training practice activities, particularly activities that may require the
performer to solve realistic problems. This will provide the performers the
opportunity to learn how to use the EPSS in a safe environment and
potentially increase adoption as performers may experience firsthand the
value of using the system to solve problems.
Make training materials available for performance support. In this study,
participants became familiar with the support content during the training
treatment but were not able to accurately recall the information that they
learned. Those provided with both training and EPSS could quickly access
the information through the performance support system, allowing them to
refresh or validate what they learned during the training phase. One
participant noted, "I could cross-check whatever I was putting in the item,
I could feel confident that it was hopefully the right stuff." Extending this
practice to other performance problems could benefit performers.
Provide access to a broader range of content for performance support.
While performance technologists should strive to minimize training content
to objectives high in frequency or criticality, they should at the same time
maximize the amount of relevant support content available to performers on
the job. As shown in Figure 6, EPSS content can be considered the superset of
information required for task performance, while training can be a carefully
prescribed subset of that body of information (in addition to instructional
methods that may not be relevant for performance support, such as
introductory materials, practice, and assessment).
It is important to note, however, that providing a dearth of performance
support information without regard to limiting the scope or providing
integrated, intuitive ways to access the information could also have a negative
effect on performance and attitudes. A performance technologist must still
be judicious in selecting the appropriate EPSS type and support content to
address a particular performance problem.
Limitations
Several limitations should be considered when interpreting the results of
this study. The task used in this research was tax preparation, a procedure
largely supported by requisite background facts, concepts, and principles.
One potential limitation is that these findings may not extend well to other
work contexts. In addition, some participants did not like the topic of tax
preparation, which may have had a negative impact on their attitudes from
the outset.
Another limitation is that the study employed an extrinsic context help
EPSS (Gery, 1995). This extrinsic performance support system was selected
for use in this study based on its demonstrated strength in previous research
(Bailey, 2003; Nguyen et al., 2005). If a different type of EPSS was used, such as
112 DOI: 10.1002/piq Performance Improvement Quarterly
an external search engine or intrinsic automated
wizard, participant attitudes could be affected.
Finally, the training intervention used in this
study was a Web-based training course that parti-
cipants completed individually with minimal super-
vision or control. If another method to deliver
training was adopted, such as instructor-led or peer
training, participant attitudes could be affected.
Future Research
Although the results of this study support the
notion that one should strive to rely less on train-
ing and more on performance support interventions for content of certain
frequency and criticality, this practice could create a new problem of
information overload during job performance instead of during training.
To mitigate such issues, additional research should focus on how to best
present EPSS content, techniques to present text-based information, navi-
gation strategies, and the efficacy of rich media such as audio and video for
performance support.
The availability of content originally designed for training to be
used for performance support also creates instructional design and
development challenges. Research on how to best design materials for
reuse across training and EPSS would ensure its effectiveness for both
interventions.
Conclusion
While training and performance support may have been initially con-
ceived to be mutually exclusive interventions, this study provides insight into
performers’ attitudes toward both and how they could potentially be
combined into an effective solution to address problems before and during
job performance.
References
Arthur, W., Bennett, W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in
organizations: A meta-analysis of design and evaluation features. Journal of Applied
Psychology,88(2), 234–245.
Bailey, B. (2003). Linking vs searching: guidelines for use [Electronic version]. Retrieved
January 17, 2006, from http://www.Webusability.com/article_linking_vs_
searching_2_2003.htm
Bastiaens, T. J., Nijhof, W. J., Streumer, J. N., & Abma, H. J. (1997). Working and learning with
electronic performance support systems: An effectiveness study. International
Journal of Training and Development,1(1), 72–78.
Burke, M. J., & Day, R. R. (1986). A cumulative study of the effectiveness of managerial
training. Journal of Applied Psychology,71(2), 232–245.
Chase, N. (1998). Electronic support cuts training time [Electronic version]. Quality
Magazine. Retrieved January 12, 2005, from http://openacademy.mindef.gov.sg/
OpenAcademy/Learning Resources/EPSS/c16.htm
Desmarais, M. C., Leclair, R., Fiset, J. V., & Talbi, H. (1997). Cost-justifying electronic
performance support systems. Communications of the ACM,40(7), 39–48.
While performance
technologists should strive
to minimize training
content to objectives high
in frequency or criticality,
they should at the same
time maximize the amount
of relevant support content
available to performers on
the job.
Volume 22, Number 1 / 2009 DOI: 10.1002/piq 113
Foster, E. (1997). Training when you need it [Electronic version]. Info World. Retrieved
November 17, 2004, from http://openacademy.mindef.gov.sg/OpenAcademy/
Learning%20Resources/EPSS/c1.htm
Gery, G. (1991). Electronic performance support systems. Tolland, MA: Gery Associates.
Gery, G. (1995). Attributes and behaviors of performance-centered systems. Performance
Improvement Quarterly,8(1), 47–93.
Mao, J., & Brown, B. (2005). The effectiveness of online task support versus instructor-led
training. Journal of Organizational and End User Computing,17(3), 27–46.
Merrill, P. F., & Salisbury, D. (1984). Research on drill and practice strategies. Journal of
Computer-Based Instruction,11(1), 19–21.
Nguyen, F., & Klein, J. D. (2008). The effect of performance support and training as
performance interventions. Performance Improvement Quarterly,21(1), 95–114.
Nguyen, F., Klein, J. D., & Sullivan, H. (2005). A comparative study of electronic performance
support systems. Performance Improvement Quarterly,18(4), 71–86.
Rackham, N. (1979). The coaching controversy. Training and Development Journal,33(11),
12–16.
Snipes, J. (2005). Blended learning: Reinforcing results. CLO Magazine. Retrieved June 29,
2008, from http://www.ninthhouse.com/papers/4%20CLO%20Reinforcing%20
Results.pdf
Vinsonhaler, J. F., & Bass, R. K. (1972). A summary of ten major studies on CAI drill and
practice. Educational Technology,12(7), 29–32.
FRANK NGUYEN
Frank Nguyen is an assistant professor at San Diego State University.
Mailing address: San Diego State University, 5500 Campanile Drive, San
Diego, CA 92182. E-mail: frank@frankn.net
114 DOI: 10.1002/piq Performance Improvement Quarterly
