Question

Does anyone know of a validated instrument to measure motivation and engagement of students with e-learning materials?

I'm evaluating the usability of a multimedia resources we developed and would like to compare the first version of the application with a revised version which has been optimized for usability. I am looking for something more than questionnaires like the System Usability Scale which measure subjective user satisfaction and want to get at any improvements in motivation or engagement.

Topics

8 / 2  ·  31 Answers  ·  5888 Views

Popular Answers

All Answers (31)

  • Lai Jiang · Institute of Tropical Medicine
    Dear Razeen,
    Probably you can look at this post https://www.researchgate.net/post/Do_you_know_any_rating_scale_evaluating_motivation_for_learning
    where people posted some questionnaires which used to assess various aspects of motivation (goal orientation, self-efficacy). depends on how you define "engagement", the questionnaire on self-regulation may also be useful for you to explore how engaged students are (i.e., how well self-regulated skills students can deploy for their learning).
  • Kiarii Aalto · Aalto University
    Agreed, it depends on how you define engagement and motivation. The Karasek & Theorells model for safe work - the control, demand and support - may also provide input into the definitions and measurements of these
  • Luca Mazzola · Politecnico di Milano
    Dear Razeen,
    despite the fact some researchers tried to propose a scale to measure the student engagement, I think that it depends too much from the specific objective of the tool/resource you are analyzing.
    Maybe also the (socio-psychological) concept of "self-efficacy" can be useful for your analysis, if you think the result of an improvement in your material will provoke an improvement in one or more skill.
    I saw it used for validating the impact of ICT adoption in K-12 teachers in the context of under-developed countries...

    Based on my experiences, I have never seen a validated scale, but I noticed that every research group adopts its own (even thought sometimes they differ just by little details).
  • Jon Mason · Charles Darwin University
    Good question, Razeen. I would also be interested in what other researchers are doing in this space. The notion of a "validated instrument" for motivation sounds challenging, though, & e-Learning has become quite multi-disciplinary. We are all motivated by many things in life & understanding why we do things can sometimes demand intensive reflection. Motivation can have many facets and can often be explained in terms of "reasons why". One facet is "aspiration" & a project I am involved in at the moment is looking into this with a targeted cohort of students over a 2-year period. As for engagement -- while there may well be a correlation between motivation & engagement perhaps there are situations where motivation has nothing to do with it? For example, an impulsive (engaged) interaction with a compelling online resource (that may have been explicitly produced as a learning resource or not).
  • Michael Brückner · Naresuan University
    Engagement could just be the time spent with various parts of your e-learning materials, which you could measure. In a Web environment this leads to a typical Web analytics application where you, e.g., try to improve the effectiveness of your Web site. Offering two versions for comparison could be a starting point.
  • Sheikh Imran · University of Kashmir
    i can send you the 62 point scale for measuring motivation and engagement of students with e-learning materials. You can mail me at smimran@kashmiruniversity.ac.in
  • Rey Segundo Guerrero-Proenza · University of Havana
    This is a good question, among other things, because it generates new questions, like: what is motivation/engagement? In order to measure, one needs to know first what is measuring.
  • Perhaps this publication will help: Kim, K. J. (2005). Adult Learners' Motivation in Self-directed E-learning, Dissertation. Bloomington IN: Indiana University (Accessed 12/21/2012 http://books.google.com/books?id=Z0RJMwAACAAJ).
  • Yaacov Katz · Michlalah Jerusalem Academic College
    Following is a relevant questionnare:

    E-learning Attitude Scale

    1. E-learning does not scare me at all
    2. Working with a e-learning would make me very nervous
    3. I do not feel threatened when others talk about e-learning
    4. I feel aggressive and hostile toward e-learning
    5. It wouldn’t bother me at all to take e-learning courses
    6. E-learning make me feel uncomfortable
    7. I would feel at ease in an e-learning class
    8. I get a sinking feeling when I think of trying to use e-learning
    9. I would feel comfortable working with e-learning
    10. E-learning make me feel uneasy and confused
    11. I’m no good with e-learning
    12. Generally I would feel OK about trying a new problem with e-learning
    13. I don’t think I would do advanced e-learning
    14. I am sure I could learn through e-learning
    15. I’m not the type to do well with e-learning
    16. I am sure I could learn how to handle e-learning
    17. I think using e-learning would be very hard for me
    18. I could get good grades in e-learning courses
    19. I do not think I could handle an e-learning course
    20. I have a lot of self-confidence when it comes to studying through e- learning
    21. I would like to study via e-learning
    22. The challenge of solving problems with e-learning does not appeal to me
    23. I think studying through e-learning would be enjoyable and stimulating
    24. Figuring out e-learning problems does not appeal to me
    25. When there is a problem with e-learning that I can’t immediately solve, I would stick with it until I have the answer
    26. I don’t understand how some people can spend so much time studying through e-learning and seem to enjoy it
    27. Once I start studying with e-learning, I would find it hard to stop
    28. I will do as little study with e-learning as possible
    29. If a problem is left unsolved in an e-learning class, I would continue to think about it afterward
    30. I do not enjoy talking with others about e-learning
  • Yaacov Katz · Michlalah Jerusalem Academic College
    The questionnaire is in an advanced stage of development by Yaacov J Katz and Yaacov B Yablon and has been piloted a number of times in the Hebrew version only. In the last pilot study (2012) the questionnaire was administerd to a sample of 132 first year social science students at an accredited Israeli university. Validity was achieved by evaluation of the original 48 items by a panel of 5 e-learning experts well-versed in the use of attitudinal questionnaires (who recommended that 18 items be discarded) and reliability for the 30 item questionnaire (Cronbach Alpha) was computed at the .87 level.
  • Kasim Koruyan · The Open University (UK)
    Dear Yaacov, If I use some of the items, how can cite this in my work?
    Thanks
  • Yaacov Katz · Michlalah Jerusalem Academic College
    I suggest that you cite the work as follows:

    Katz, Y.J. & Yablon, Y.B. (2012). A reliability study of the e-Learning Attitude Scale. Work in Progress, School of Education, Bar-Ilan University.
  • Panagiotis Zaharias · School of Pedagogical & Technological Education
    Dear Razeen

    during my PhD study I developed an instrument which measured motivation to learn along with e-learning usability. Actually that was the main outcome of my doctoral thesis: the development of a new questionnaire-based usability evaluation method that proposes motivation to learn as a new type of usability measurement for e-learning courses. Two large empirical studies were conducted in authentic business environments, with real learners-employees in corporate settings who took several e-learning courses and evaluated them by using my instrument. Results provided significant evidence for reliability and validity of the instrument. This work was completed in 2004. Since then many scholars have used my theoretical framework and instrument.
    You can find more details about the development and the validity of this questionnaire in the following publications:

    Zaharias, P. and Poulymenakou, A. (2009). Developing a usability evaluation method for e-learning applications: beyond functional usability. International Journal of Human-Computer Interaction Vol 25 (1), January 2009, 75 – 98.

    Zaharias, P. (2006). A Usability Evaluation Method for e-learning: Focus on Motivation to Learn. In Extended Abstracts of Conference on Human Factors in Computing Systems – CHI 2006:

    Fyi, I am attaching a version of the instrument.

    Best regards
    Panagiotis
  • Razeen Davids · Stellenbosch University
    Thank you Panagiotis, and also to everyone else who has responded. I really do appreciate all the very useful contributions.
  • B. Yousif · University of Southern Queensland 
    HI I have done similar work but not completed yet.. You can have a look at this paper which is the first draft of the work.. we have a system to evaluate the involvment of the students in online discussion.. I used Multimedia as well
  • Faris Baker · Arab Open University - Kuwait
    I think the following can indicate motivation and engagement of the student:

    1.The number of times the student comes to the e-learning class.
    2.The number of times and number of weeks the student participate in the e-earning forums.
    3.The numbers of times the student vote on a post reply or start new thread.
    4.The number of times the student active in the forums.
    5.The number of times the student read thread from classmates.
    6.The number of times the student vote on interesting or useful posts.
    7.The number of times the student post good replies (e.g. earn 3 likes or more).
    8.The number of times the student start good threads.
    9.The number of times the student write a post that classmates vote up a lot.
    10.The number of times the student start a thread that classmates vote up a lot.
    11.The number of the times the student looks at the online quizzes and solves them.
    12.The number of times the student do the e-learning course assignments
  • Kiarii Aalto · Aalto University
    How to measure user engagement and gamification: http://touchedspace.wordpress.com/2013/05/08/how-to-measure-user-engagement-and-gamification/

    Engagement, Causes and Effect Matrix: http://touchedspace.files.wordpress.com/2013/05/poster.pdf

    Effects of evaluation on learning: http://touchedspace.wordpress.com/2013/06/25/effects-of-evaluation-on-learning/

    Extrinsic motivation & engagement is measurable. On the contrary, intrinsic may not.
  • Fridolin Wild · The Open University (UK)
    You may like this piece (appeared in JASIST):
    Heather L. O’Brien and Elaine G. Toms (2010): The Development and Evaluation of a Survey to Measure User Engagement, In: JASIST, 61(1):50–69.

    https://www.researchgate.net/publication/220433691_The_development_and_evaluation_of_a_survey_to_measure_user_engagement?ev=srch_pub
  • Greg Voykhansky · Capella University
    the Gallup Organization Q12 study offers an instrument for measuring worker's engagement. Wonder if you can apply it to your study.
    Thackray, J. (2001, March 15). Feedback for real. Gallup Business Journal. Retrieved from http://businessjournal.gallup.com/content/811/Feedback-Real.aspx


    Medlin, B., & Faulk, L. (2001). The relationship between optimism and engagement: The impact on student performance. Research in Higher Education Journal, 13, 1-9. Retrieved from http://search.proquest.com.library.capella.edu/docview/889136489?accountid=27965
  • Isabella Hanudin · Sunway Education Group
    Does anyone have this journal or know how to get it
    The title is: "The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education"
  • Radina Nikolic · British Columbia Institute of Technology
    It is available online:
    http://www.selfdeterminationtheory.org/SDT/documents/1992_VallerandPelletierBlaisBriere_EPM.pdf
  • Isabella Hanudin · Sunway Education Group
    Thanks!
  • Ali Tarhini · Brunel University
    Hi Razeen,

    The following link contains some useful relevant instruments that may be of help for you :http://itestlrc.edc.org/STEM_education_instruments?page=3 and www.inn.colorado.edu.
  • Razeen Davids · Stellenbosch University
    Thank you Ali, looks like a very useful site.
  • Kathy Brennan · University of North Carolina at Chapel Hill
    This paper may be useful to you:
    O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 938-955.
  • Razeen Davids · Stellenbosch University
    Thank you Kathy, this is really useful!
  • Michelle Linder · Georgia Institute of Technology
    Allow self directed participation and track the depth and/or how far into the program your users delve into or participate in. If you find that your users only interact with the first few layers of your program, chances are they weren't motivated to interact with your program. Then ask your users why?

    Another method is to test retention of the material. People who are interested in material tend to be motivated into learning that material. Of course, in your falsification, don't forget to account for any and all of the various learning disabilities, and different learning styles. You might have users that are motivated to learn, but are unable to learn within the parameters of your e-learning program's delivery system.
  • Neila Rjaibi · Institut Supérieur de Gestion de Tunis
    Useful papers :

    1.Modeling the Assessment of Quality Online Course: An empirical Investigation of Key Factors Affecting
    Learner’s Satisfaction, Neila Rjaibi, Latifa Ben Arfa Rabai, IEEE TECHNOLOGY AND ENGINEERING EDUCATION (ITEE). Vol 7, No 1 (2012), edited 23 March 2012, ISSN 1558-7908.

    2 Assessing Quality in E-learning including learner with Special Needs, Latifa Ben Arfa Rabai, Neila Rjaibi, Proceedings of The Fourth National Symposium on Informatics, Technologies for Special Needs, April 23-25, 2013. King Saud University, Riyadh, Saudi Arabia
  • Neila Rjaibi · Institut Supérieur de Gestion de Tunis
    The first paper
  • Razeen Davids · Stellenbosch University
    Thank you, Michelle and Neila, for your suggestions.

    Regards
    Razeen

Question Followers (75) See all