Content uploaded by Helene Metcalfe
Author content
All content in this area was uploaded by Helene Metcalfe on Nov 11, 2019
Content may be subject to copyright.
EVALUATION OF AN ONLINE LEARNING RESOURCE FOR NURSING
STUDENTS PREPARING FOR AN EMERGENCY DEPARTMENT CLINICAL
PLACEMENT USING KIRKPATRICK’S MODEL
Darren Falconer1, Dr Helene Metcalfe2, Professor Jeffrey Hamdorf 3
1Medical School, Faulty of Medicine, Dentistry and Health Sciences, The University of Western
Australia / PhD Candidate (Australia)
2Medical School, Faulty of Medicine, Dentistry and Health Sciences, The University of Western Australia
/ Co-Supervisor (Australia)
3Medical School, Faulty of Medicine, Dentistry and Health Sciences, The University of Western Australia
/ Principal Supervisor (Australia)
Abstract
Clinical placements in the Emergency Department (ED) can offer nursing students a unique learning
experience to develop their knowledge and skills in managing the critical care patient. However, the
complexity of this clinical placement can be daunting leading to increased stress and anxiety. This can
have implications for student learning and clinical performance creating feelings of unpreparedness for
the challenges of emergency nursing. Analysis of students’ concerns were obtained through an
explanatory sequential mixed methods study identifying key areas of concern. Following this, an online
learning resource “Are you PreparED” was developed. This educational resource, is a repository of useful
information relevant to the role of the nursing student in the ED and is focused on the key areas of
concern identified by the students.
Evaluation of the online learning resource was structured using the first two stages of Kirkpatrick’s
model. Following completion of an online survey, students provided feedback on several aspects
including: design, usability, relevance and accuracy of information. In addition, comments were obtained
regarding their perceived learning, knowledge acquisition and intent to apply to the clinical setting.
Perceived levels of preparedness following use of the online learning resource was also explored.
The goal of this study was to provide a valuable online learning resource for nursing students attending a
clinical placement in the ED. Evaluation of the resource was paramount to assess quality and establish
that learning had occurred, and educational goals met. Suggestions offered by the students provided a
valuable source of information for further improvements to this online resource.
Keywords: Online resource, evaluations, student nurses, emergency department, preparedness.
1. Introduction
This paper will explore the evaluation of an online learning resource developed to assist nursing
students in their preparedness for a clinical placement in the Emergency Department (ED). Following the
development of the ARE YOU PREPARED website, student evaluations were undertaken using the
Kirkpatrick’s Model of Evaluation Framework to identify learner satisfaction and perceived learning and
preparedness.
2. Background
The ED clinical placement provides nursing students with a unique opportunity to link theory to
practice in a critical care setting. Students are faced with a number of new challenges in the context of a
busy, fast-paced and rapidly changing acute care environment. Patient care is increasingly complex and
unpredictable with increased patient acuity and high patient turnover. Holbury and Newcombe (2016)
acknowledge that the ED is an unpredictable and often highly emotive environment. They note that some
nurses thrive on this type of clinical setting and need to be expert in the assessment, recognition and
management of patients across the lifespan with life- threatening illness or injury. In addition, nurses are
required to process large amounts of information in order to facilitate clinical decision making in this
time-pressured environment.
Therefore, the ED clinical placement offers nursing students the opportunity to have exposure to a
wealth of learning opportunities, many of which they may not have experienced while on previous
clinical placements (Williams & Palmer, 2014). Nursing students are able to engage with the
multidisciplinary ED clinical team in order to build confidence in their theoretical knowledge and clinical
skills in the care and management of the acutely unwell patient. The aim of this unique clinical placement
is for students to develop a specialised skill set in the areas of patient assessment, fast decision making,
prioritisation of care and responding to the deteriorating patient (Purling & King 2012; Williams &
Palmer, 2014).
However, for some nursing students the anticipation of practicing in this acute care environment can
be intimidating, anxiety provoking and stressful. The literature suggests that students often form
preconceived ideas about what to expect from a clinical placement in the ED. This can lead to students
questioning their abilities with a perceived lack of preparedness (Morrell & Ridgeway 2014). Areas of
concern are wide ranging and include: knowledge and clinical skill deficits, fear of making mistakes,
ineffective communication, feelings of inadequacy, fitting into the team, level of support and mentorship,
dealing with conflict and having the inability to recognise and respond effectively to a deteriorating
patient. Studies show that educational preparation of nursing students in this acute care area is becoming
increasingly challenging with many nursing students failing to meet industry expectations (Purling &
King, 2012).
Students allocated a clinical placement in the ED are generally in the latter stages of their nurse
education, and therefore under pressure to demonstrate sound knowledge and clinical competence (Porter,
Morphet, Missen & Raymond 2013). The expectation from both academic faculty and healthcare industry
is that students will be adequately prepared for the challenges of working in an acute care clinical area
and able to respond effectively to critical care incidents. The overarching expectation is that nursing
students will be well prepared for the realities of practicing as a graduate nurse in an increasing complex
healthcare environment (Dimitriadou et al 2015; Watt & Pascoe 2013). Despite this, it is evident from the
nursing literature that this is not necessarily the case with students often floundering when faced with the
challenges of working in an acute care setting. The evidence suggests that students are lacking confidence
in their knowledge and clinical abilities and experience difficulty in making sound clinical judgements in
the management of the acutely unwell patient (Doody, Tuohy & Deasy 2012; Duchscher 2009; Woods et
al 2014). This has far reaching consequences, as these negative experiences may influence decisions
about whether to pursue a career in emergency nursing. Future projections indicate an increased need for
critical care nurses both nationally and internationally due to a diminishing workforce (Halcomb,
Salamonson, Raymond, & Knox, 2012). Together with the ageing population, unprecedented population
growth and declining nursing workforce there is a need to recruit and retain nurses with critical care
expertise.
In light of this, it is clear that undergraduate nursing education needs to take heed and find innovative
ways to adequately prepare students for acute care clinical placements and the realities of their future
professional role. While this issue is without doubt complex and multidimensional, it is generally agreed
that nursing student unpreparedness may have far reaching consequences when it comes to providing high
quality patient care, especially in specialised areas such as the ED (Halcomb et al, 2012; Williams &
Palmer, 2014).
To address the issue of preparedness, an explanatory sequential mixed methods design study was
undertaken to identify areas of concern by nursing students in attending a clinical placement in the ED
(Creswell & Plano-Clark 2018). Both the quantitative and qualitative findings from this study provided a
platform for the development, implementation and evaluation of on an online learning resource titled ‘Are
you PreparED?’. Whilst there are a number of useful online learning resources available for healthcare
students globally. The focus of these resources is largely aimed at medical students and has an
emphasising on specialised clinical skills. Examples include electrocardiogram (ECG) interpretation,
patient history taking, clinical assessment and laboratory data interpretation. While some of this
information is extremely important, these resources are somewhat limited for nursing students. In
addition, the notion of preparing students for their clinical placement appears to be lacking. Levett-Jones
et al (2015) suggest that nursing students' clinical placement experiences can be a critical turning point
and may even affect their professional aspirations. It is this lack of online resources specifically aimed at
preparing nursing students for clinical practice in the ED that has facilitated the development of this
website. Following development of the online resource, student evaluation is essential in order to ensure
learning effectiveness, knowledge acquisition and intent to apply to the clinical setting (Maycock et al
2012).
3. Design and Development of the Website
As previously identified, the development of the ARE YOU PREPARED online learning
resource was based on the need to increase student preparedness prior to attending a clinical placement in
the ED. Analysis of both qualitative and quantitative data from students at 3 universities in Western
Australia informed the structure and content base of the website (figure 1).
Figure 1 Website development and evaluation process
Following a successful grant application, an expert web designer was engaged to create and develop
a visually appealing and easy to navigate site (Maycock et al 2012). It was essential to ensure that the
website was clear, easily understood, relevant and up to date (Ng, Archbold, Mayer and Mulla 2015). The
website homepage contains an introductory video and instructions on how to navigate the site. The
content was structured using a modular format located within four broad topics: ‘Orientation to the ED’,
‘Patient Care’, ‘Clinical Skills’ and ‘Professional Issues’. As suggested by Elkman (2018) the content is
predominantly text-based and included photographs, videos, helpful tips, nursing blogs, links to external
websites and self-directed learning. Students are given the opportunity to engage with the material and
acquire meaningful knowledge contextualised to nursing practice in the ED. Trademarking of the ARE
YOU PREPARED was obtained to protect copyright of the website.
4. Evaluation
When looking at evaluating a website it is essential to identify what is being evaluated, how it is
expected to work and how the evaluation results will improve the resource. Cook and Elleway (2015)
suggest that evaluation can be seen as the process of judging the value of something, they also note that
this can also include theoretical dimensions. In the context of the ARE YOU PREPARED website, it was
essential to firstly evaluate the design, usability, relevance and accuracy of information offered. An expert
panel consisting of senior academic staff members, researchers and experienced ED clinical staff to assess
the online learning resource for content and construct validity. This involved critical appraisal of the site
and constructive feedback in relation to visual appeal, ease of navigation, currency, relevance and
accuracy of content and perceived usefulness (Ng et al 2015). The expert panel were also asked to
provide suggestions for improvements so that modifications could be made. Following this, the website
was launched and all final year nursing students across the three universities were asked for feedback
using an online survey. The theoretical framework used to guide the evaluation of the online resource was
Kirkpatrick’s Model of Evaluation.
4.1. Kirkpatrick’s Model of Evaluation
Kirkpatrick’s Model of Evaluation has been utilised for well over forty years, and this model has
gained an overwhelming popularity for use as a framework to support systematic evaluation. Primarily
used within the business sector, the model provides strategies for evaluating organisational training
programs. However, use of the model has expanded and been adapted into other disciplines including
tertiary education and clinical practice as a means for assessing and appraising education outcomes and
intervention studies (Praslova 2010; Smidt, Balandin, Sigafoos & Reed 2009).
The model presents four sequential levels which measure impact of the educational intervention. The
1st level ‘reaction’ refers to learner satisfaction with the intervention. Evaluation at this level can generate
valuable information about relevance of content, perceived usefulness, learning tools and modes of
delivery. This can provide an insight into how participants perceive their level of motivation and desire to
engage. While this information may be useful for the purpose of modifying material, this information is
subjective and does not guarantee learning.
At level 2, ‘learning’ emphasis is on determining if the educational intervention has met the learning
outcomes and the extent to which users perceived learning has occurred. This can include increased
knowledge, skills, confidence and commitment to apply into practice
Level 3 ‘behaviour’ and level 4 ‘results’ then move beyond individual satisfaction and learning. The
focus moves to organisational change through positive behaviour and transfer of learning into the
workplace. The ultimate goal of levels 3 and 4 is to achieve a level of workplace performance that has
met individual learning needs, as well as the goals of the organisation (Moreau 2017; Reio, Rocco, Smith
& Chang 2017; Rouse 2011). In relation to the online learning resource evaluation was limited to levels 1
and 2 of Kirkpatrick’s model.
5. Conclusion
Attending a clinical placement in the ED can be an overwhelming and daunting experience for
nursing students. This paper has explored the development of the website ARE YOU PREPARED, an
online resource to assist nursing students with their preparedness for attending a clinical placement in the
ED. Effective evaluation of the resource was undertaken to ensure the design, usability, relevance and
accuracy of information. In addition, evaluation of the resource using the Kirkpatrick’s Model of
Evaluation framework assisted in identifying not only learner satisfaction with the resource, but how the
students perceived their knowledge skills and confidence after using this online learning resource.
Furthermore, students perceived levels of preparedness were appraised. While primarily designed for
nursing students, this online learning resource has the potential to be of benefit in preparing other
healthcare professionals for a clinical placement in the ED.
References
Cook, D. A., & Ellaway, R. H. (2015). Evaluating technology-enhanced learning: A comprehensive
framework. Medical Teacher, 37, 961-970.
Creswell, J. W., & Plano-Clark, V. L. (2018). Designing and conducting mixed methods research (3rd
ed.). Thousand Oaks, CA: Sage Publications.
Dimitriadou, M., Papastavrou, E., Efstathiou, G., & Theodorou, M. (2015). Baccalaureate nursing
students' perceptions of learning and supervision in the clinical environment. Nursing & Health
Sciences, 17 , 236-242. doi:10.1111/nhs.12174
Doody, O., Tuohy, D., & Deasy, C. (2012). Final-year student nurses' perceptions of role transition.
British Journal of Nursing, 21, 684-688. doi:10.12968/bjon.2012.21.11.684
Duchscher, J. E. B. (2009). Transition shock: the initial stage of role adaptation for newly graduated
registered nurses. Journal of Advanced Nursing, 65, 1103. doi:10.1111/j.1365-
2648.2008.04898.x
Elkman L (2018) Developing and evaluating an e-learning resource for nurses in haemato-oncology.
Cancer Nursing Practice. doi: 10.7748/cnp.2018.e1510
Halcomb, E. J., Salamonson, Y., Raymond, D., & Knox, N. (2012). Graduating nursing students’
perceived preparedness for working in critical care areas. Journal of Advanced Nursing, 68(10),
2229-2236. doi:10.1111/j.1365-2648.2011.05911.x
Holbery, N., & Newcombe, P. (Eds.). (2016). Emergency nursing at a glance (At a glance series).
Chichester, West Sussex: John Wiley & Sons.
Levett-Jones, T., Pitt, V., Courtney-Pratt, H., Harbrow, G., & Rossiter, R. (2015). What are the primary
concerns of nursing students as they prepare for and contemplate their first clinical placement
experience? Nurse Education in Practice, 15, 304-309.
Maycock, B., Hallett, J., Hildebrand, J., Brown, G., & Lobo, R. (2012). My-peer toolkit: Developing an
online resource for planning and evaluating peer-based youth programs. Youth Studies Australia,
(31), 53-61.
Moreau, K. A. (2017). Has the new Kirkpatrick generation built a better hammer for our evaluation
toolbox? Medical Teacher, 39, 999-1001. doi:10.1080/0142159X.2017.1337874
Morrell, N., & Ridgway, V. (2014). Are we preparing student nurses for final practice placement? British
Journal of Nursing, 23, 518-523.
Ng, Z. Y., Archbold, S., Mayer, C., & Mulla, I. (2015). SoundSpace online: The development of an
online resource about hearing loss. American Journal of Audiology, 24, 289.
doi:10.1044/2015_AJA-14-0086
Porter, J. Morphet, J. Missen, K., & Raymond, A. (2013). Preparation for high-acuity clinical placement:
Confidence levels of final year nursing students. Advances in Medical Education and Practice,
4, 83-89. doi:10.2147/AMEP.S42157
Praslova, L. (2010). Adaptation of Kirkpatrick's four level model of training criteria to assessment of
learning outcomes and program evaluation in Higher Education. Educational Assessment,
Evaluation and Accountability, 22, 215-225. doi:10.1007/s11092-010-9098-7
Purling, A., & King, L. (2012). A literature review: graduate nurses' preparedness for recognising and
responding to the deteriorating patient. Journal of Clinical Nursing, 21, 3451-3465.
doi:10.1111/j.1365-2702.2012.04348.x
Reio, T. G., Jr., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A Critique of Kirkpatrick's Evaluation
Model. New Horizons in Adult Education & Human Resource Development, 29, 35-53.
doi:10.1002/nha3.20178
Rouse, D. N. (2011). Employing Kirkpatrick's evaluation framework to determine the effectiveness of
health information management courses and programs. Perspectives in health information
management / AHIMA, American Health Information Management Association, 8, 1-5.
Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. A. (2009). The Kirkpatrick model: A useful tool for
evaluating training outcomes. Journal of Intellectual and Developmental Disability, 3, 266-274.
doi:10.1080/13668250903093125
Watt, E., & Pascoe, E. (2013). An exploration of graduate nurses' perceptions of their preparedness for
practice after undertaking the final year of their bachelor of nursing degree in a university‐based
clinical school of nursing. International Journal of Nursing Ppractice, 19, 23-30.
doi:10.1111/ijn.12032
Williams, E., & Palmer, C. (2014). Student nurses in critical care: benefits and challenges of critical care
as a learning environment for student nurses. Nursing in Critical Care, 19, 310-315.
doi:10.1111/nicc.12053
Woods, C., West, C., Mills, J., Park, T., Southern, J., & Usher, K. (2014). Undergraduate student nurses’
self-reported preparedness for practice. Collegian, 272. doi:10.1016/j.colegn.2014.05.003
