Available via license: CC BY 4.0
Content may be subject to copyright.
Citation: Hellín, C.J.; Calles-Esteban,
F.; Valledor, A.; Gómez, J.;
Otón-Tortosa, S.; Tayebi, A.
Enhancing Student Motivation and
Engagement through a Gamified
Learning Environment. Sustainability
2023,15, 14119. https://doi.org/
10.3390/su151914119
Academic Editors: Ripon Kumar
Chakrabortty and Hao-Chiang
Koong Lin
Received: 10 August 2023
Revised: 5 September 2023
Accepted: 21 September 2023
Published: 24 September 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
sustainability
Article
Enhancing Student Motivation and Engagement through
a Gamified Learning Environment
Carlos J. Hellín , Francisco Calles-Esteban , Adrián Valledor , Josefa Gómez , Salvador Otón-Tortosa
and Abdelhamid Tayebi *
Computer Science Department, University of Alcala, 28801 Alcala de Henares, Spain; carlos.hellin@uah.es (C.J.H.);
adrian.valledor@uah.es (A.V.); josefa.gomezp@uah.es (J.G.); salvador.oton@uah.es (S.O.-T.)
*Correspondence: hamid.tayebi@uah.es
Abstract:
Gamification is a viable strategy used to enhance motivation and engagement in program-
ming classes among students. However, automated evaluation capabilities, which are crucial for
giving students fast and correct feedback, are frequently lacking in currently available gamification
technologies. This study proposes a novel web-based application that combines automated pro-
gramming assessment features with gamification concepts; the aim is to provide students taking a
programming course with an engaging learning environment. A survey conducted with 215 under-
graduate students assessed how the tool affected the motivation and engagement of students. The
findings show that the tool had a beneficial impact on students’ willingness to participate in class,
study, increase their self-confidence, engage in healthy competition with peers, and learn from their
mistakes. The qualitative feedback that students offered regarding the features of the tool that they
liked best is also covered in the study. This paper contributes to the field of programming education
by presenting a detailed gamified tool, incorporating automated evaluation and gamification in a
web-based platform.
Keywords:
gamification; programming courses; student motivation; engagement; automated assessment
1. Introduction
Games can serve as very powerful tools for improving learning processes, from
three distinct yet complementary perspectives: as tools for teaching content or skills, as an
object of the learning project itself, and as a philosophy to be taken into account when
designing the training process [
1
]. These tools, also known as gamification, have been
instrumental in promoting the interests and motivations of students. In addition, student
performances in the course have improved, and engagement with online learning platforms
has increased. Gamification in educational systems has consistently generated profound
interest among educators and researchers. It has been demonstrated that gamification has
an extremely positive effect on learning [
2
]. Other studies have shown that the incentives
and rewards offered by gamification have a theoretical basis in stimulating self-efficacy,
which can either enhance or impede motivation [3].
The state of the art of web-based gamification tools has been extensively studied over
the past few years. Various approaches, such as game mechanics, rewards, and leader-
boards, have been used to successfully implement gamification. In addition, these tools
have been found to be beneficial in educational and health-related contexts [4].
This related work focuses on the effectiveness of these tools in promoting engagement
and motivation in important use cases, such as programming courses or other engineer-
ing subjects, as well as their ability to support learning.
One notable example is a study conducted by Jusas et al. [
2
], which examined the
effects of web-based gamification on students in an object-oriented programming course.
The study found that the use of gamification led to significant improvements in motivation
Sustainability 2023,15, 14119. https://doi.org/10.3390/su151914119 https://www.mdpi.com/journal/sustainability
Sustainability 2023,15, 14119 2 of 20
and engagement and increased student satisfaction with the course. Another study by
Marín et al. [
5
] found that students using a gamified platform for learning C program-
ming obtained better marks than those using a non-gamified compiler. In [
6
], the authors
found that adopting a gamified approach to teaching programming resulted in enhanced
attendance, increased downloading of course materials, and improved final grades. Buck-
ley et al. [
7
] concluded that gamified learning interventions have a positive impact on
student learning and that the effects of gamification on student participation vary, depend-
ing on whether the student is motivated intrinsically or extrinsically.
To further enhance the understanding of gamified learning environments, recent re-
search has explored different aspects of this approach. Gamification has been applied
to other domains and disciplines, such as English vocabulary training [
8
], sustainability
education [
9
], discovery learning [
10
], and palliative care awareness [
11
]. These studies
have shown that gamification can enhance students’ interest, involvement, and perfor-
mance across different learning scenarios and topics. For example, ref. [
12
] conducted a
cluster micro-randomized trial on medical interns to evaluate the effectiveness of gamified
team competition as a mHealth intervention. The study showed that team competition
significantly increased daily physical activity, suggesting its potential as a mobile health
intervention tool.
Moreover, some studies have explored the use of adaptive gamification, which tailors
the game elements to the preferences and needs of each learner. For instance, ref. [
13
]
investigated the effects of adaptive gamification on students’ motivation in science edu-
cation. They found that adaptive gamification increased students’ intrinsic motivation
and self-efficacy more than non-adaptive gamification. Durmaz et al. [
14
] examined the
influence of gamification elements on the explicit motive dispositions of learners. They
found that different types of rewards had different effects on learners’ achievement, power,
and affiliation motives.
Finally, some studies have focused on designing and evaluating gamification tools for
massive open online courses (MOOCs), which pose specific challenges and opportunities
for engaging and motivating learners. For example, Ortega-Arranz et al. [
15
] presented
GamiTool, a web-based tool that supports instructors in the gamification of MOOCs. Their
evaluation with MOOC instructors and gamification designers showed GamiTool’s high
design expressiveness, usability, and potential for adoption. This system can be used by
instructors to improve student engagement, while researchers can employ it to understand
the effects of gamification in MOOC settings. Khaldi et al. [
16
] conducted a systematic
literature review about gamification in e-learning in higher education. They found that
points, badges, and leaderboards (PBL elements) were the most commonly used game
elements for gamifying e-learning systems. The impact of gamification in MOOCs was also
reviewed by [
17
], revealing positive outcomes with increased participation and retention in
gamified MOOCs.
1.1. Gamification
The concept of gamification has gained popularity in recent years as a way to engage
and motivate people to achieve their goals. Gamification involves the integration of game
design elements, such as points, badges, and leaderboards, into non-gaming contexts, to
create a sense of competition and accomplishment. This technique has been applied in
fields such as education, healthcare, and marketing, with the aim of improving learning
outcomes, promoting healthy behaviors, and increasing engagement. The purpose is to
have a methodology that can motivate the user to use it. It is important to correctly apply
gamification elements to maintain user motivation. In this way, users are engaged on
these platforms while acquiring knowledge. In the context of education, gamification
has been the subject of study and debate among researchers over the years. In general,
the implementation of gamification in the educational system has provided two crucial
benefits for students: they are motivated to continue learning, and it results in improved
performance compared to a more traditional educational context. It is noteworthy that
Sustainability 2023,15, 14119 3 of 20
better scores in practical assignments and overall scores were obtained by students who
completed the gamified experience [18].
1.2. Game Design Elements
Game design elements are commonly used in gamification to create engaging and
rewarding experiences for students. Some of the elements presented here were evaluated
by 19 experts on gamification and education [19].
1.
Points: A key component in many video games. In gamification, they are used as
rewards for successfully completing certain tasks within the environment. The user’s
progress is also represented in a numerical way. Different types of points exist, such
as experience points, redeemable points, and reputation points. Points allow the user
to receive instant feedback on their progress within the platform, motivating them to
continue earning more points, which will unlock additional rewards.
2.
Achievements/badges: In a gamified system, users are granted virtual rewards in
the form of achievements and badges after they have completed certain tasks. These
rewards should be consistent with the gamified task, as the aim is to maintain user mo-
tivation. Sometimes, these achievements or badges are divided into smaller rewards
to make them more frequent and provide more motivation to users. Achievements
and badges are represented by visual images, and they are available for users to view
whenever they want.
3.
Leaderboards: The points obtained by each user, along with their usernames and
avatars, are displayed in a list of participants. These tables create a competitive envi-
ronment among students and challenges for those who are not in the top ten positions.
Additionally, a personal list is maintained to show users who their competitors are if
they are not among the top ten positions.
4.
Store: This is another important section in gamification, as rewards can be obtained by
students. These achievements provide redeemable points, which can be used to purchase
various additional items available in the store. These items can include “Change avatar”
and “Change background”, allowing students to customize their user profiles. Other
items include the possibility of purchasing extra points toward the final grade, a feature
that typically ranks among the most popular and motivating for students.
1.3. Hypothesis
According to the state of the art, the implementation of a web-based gamified tool has
been widely recognized as beneficial for students; it significantly enhances their learning
experience and motivation in programming courses. Furthermore, the integration of an
automated assessment system further contributes to the effectiveness and efficiency of the
learning process. By combining these two powerful components, a novel approach to create
a comprehensive tool that integrates gamification principles with automated programming
assessment capabilities is proposed in this paper.
The primary objective of this research is to design, develop, and evaluate a web-
based tool that leverages the advantages of gamification and automated assessment in a
programming course context. The tool aims to provide students with a stimulating and
engaging learning environment.
The central hypothesis of this research is that the implementation of this gamified tool
will significantly increase students’ motivation and engagement in programming activities.
By incorporating game-like elements, such as achievements, leaderboards, and a reward
system, an immersive learning environment is created that fosters a sense of competition,
collaboration, and accomplishment. It is anticipated that this motivational aspect, combined
with the automated assessment system, will enhance students’ learning outcomes.
We aim to address the following research questions by implementing this tool in a
university programming course:
1.
How does the integration of a gamification system impact students’ motivation and
engagement in a programming course?
Sustainability 2023,15, 14119 4 of 20
2.
What functionalities of the web-based gamified tool influence the students’ motivation?
The remaining sections of this paper are organized as follows. Section 2provides an
overview of the methodology applied in this study, including the web-based gamification
tool used. In Section 3, the results obtained from the study are presented in detail. Finally,
Section 4presents the discussion and conclusions drawn from the research and outlines
potential avenues for future work.
2. Materials and Methods
This section presents the details of the methodology carried out and the descriptions
of the main functionalities included in the web-based tool.
2.1. Methodology
To answer the research questions presented in Section 1.3, a comprehensive evaluation
methodology was employed, including quantitative and qualitative data collection tech-
niques. The study consisted of a sample of students enrolled in the programming course,
who interacted with the tool throughout the duration of the course. Their performance,
engagement levels, and feedback were analyzed to assess the effects of the tool on their
learning experiences.
To evaluate the effectiveness of the proposed tool, a comprehensive assessment was
conducted. This evaluation included the quantitative analysis of students’ answers through
a survey. The results of this survey will provide valuable insights into the effects of
gamification and automated assessment on students’ motivation and learning outcomes in
the context of programming courses.
Importantly, it should be noted that the tool was universally accessible and available
to all students enrolled in the course without any restrictions. This unrestricted access was
designed to ensure equal opportunities for engagement and learning experiences.
Nevertheless, the study did not employ a traditional experimental design with dis-
tinct control and experimental groups. Instead, it focused on a single cohort of students
who interacted with the gamified tool, assessing their experiences and perceptions with-
out the comparative framework of a control group. This design decision emerged from
the intention to capture authentic interactions and feedback during the tool’s inaugural
implementation. Recognizing the intrinsic value of comparative studies, featuring both
control and experimental groups, this limitation of the current study is acknowledged.
Future research will endeavor to incorporate both a control group and an experimental
group to yield a more robust assessment of the effects of gamification on student learning
and motivation.
According to the literature, conducting a survey and quantitatively measuring data are
suggested as valid research methodologies; however, there are some limitations and consid-
erations to keep in mind. Boeren [
20
] notes that survey research can gather both qualitative
and quantitative data while also investigating the effects of survey methodologies on the
format of questions. Barabas [
21
] compares survey experiments with natural experiments
and suggests caution when extrapolating from survey experiments. Mills [
22
] investigated
the extent to which researchers accessed quantitative methodology publications and found
that researchers do not frequently access quantitative methodology literature to determine
the best way to analyze their data. Finally, Nardi [
23
] provides a guide to quantitative
methods, including designing a questionnaire, sampling, and analyzing data. These papers
suggest that conducting a survey and quantitatively measuring data are valid research
methodologies; however, researchers should be aware of the limitations and considerations
of this approach.
This study was conducted at a public university in Spain, involving a total of 215 stu-
dents from three distinct computer engineering degree programs: computer engineering,
computer science engineering, and engineering in information systems. These programs
represent specialized areas within the broader field of computer engineering, with each
encompassing different educational focuses and potential career trajectories. The sample
Sustainability 2023,15, 14119 5 of 20
was selected through stratified random sampling, dividing the students into three strata cor-
responding to the different computer engineering degrees. Within each stratum, students
were randomly selected using a computer-generated randomization process to ensure an
unbiased selection. This method was chosen to provide a comprehensive perspective on
the effects of the gamified learning environment across these diverse educational paths
and to ensure that the sample was representative of the broader population of students
within the computer engineering field at the university. Further demographic information,
such as the year of study, gender distribution, and prior experience with gamified learning,
was also collected to provide additional context and nuance to the analysis. This more
detailed description of the participating student groups helps to ensure that the study’s
findings are relevant and applicable to a broad spectrum of students within the field.
The participants were undergraduate students enrolled in an object-oriented programming
course. The course was gamified using web-based software that included features such
as points, badges, and leaderboards. Student performances were assessed using program-
ming assignments, and their motivation and engagement were measured using a survey.
From March 2023 to May 2023, an online survey platform [
24
] was employed to administer
a questionnaire to the students. Student participation was entirely voluntary, and students
had the option to decide whether or not to complete the questionnaire. It is important to
emphasize that the questionnaire ensured anonymity and confidentiality. This voluntary
nature of participation led to a response rate of 52.5%, which accounted for the difference
between the number of students taking the course and the number of students evaluating
the software. Subsequently, the online survey platform provided an Excel file containing
the students’ responses, facilitating the analysis of the collected data. The data analysis for
this study was performed using R 4.1.1 software, supported by the R Core Team and the
R Foundation for Statistical Computing. The analyses were conducted on a Windows 10
×
86-64 machine with an Intel(R) Core(TM) i7-7700 CPU operating at 3.60 GHz with 16 GB
of physical memory.
The gamification process implemented in this study was thoughtfully designed with
the aim of enhancing students’ engagement and motivation in the programming course.
The process, which was rooted in the principles of game-based learning, leveraged elements
such as points, badges, and leaderboards to create a competitive and engaging learning
environment. The development began with an analysis of the learning objectives and the
identification of key motivational factors that could be influenced by gamification. A series
of iterative prototyping and testing cycles ensured that the gamification features were
effectively integrated and resonated with the interests and needs of students. The final
implementation was seamlessly embedded into the existing course structure, allowing
students to participate in gamified activities alongside traditional learning.
To examine the effects of the gamification platform on students’ motivation in the
programming course, the questionnaire focused on assessing the factors that influenced
their motivation and engagement with the platform. Building upon existing research in the
field [
25
–
28
], the survey was derived from established sets of questions that explored the
various elements affecting motivation. Using a 5-point rating scale (5: completely agree;
4: agree; 3: neutral; 2: disagree; 1: completely disagree), the students were asked to rate
their level of motivation when using the gamification platform for the programming course.
The following reasons were evaluated by the students:
• The use of the tool has motivated me to attend class.
• I was more motivated to study and therefore obtained more points.
• Achieving points improved my self-confidence.
• The competitive environment of the leaderboard enhanced my motivation.
• When my code did not pass the tests, I felt motivated to look for errors and fix them.
In addition to rating their motivation, students were provided with an open question,
“Which part did you like the most?”. This question aimed to gather qualitative feedback
and opinions from the students.
Sustainability 2023,15, 14119 6 of 20
In order to gain a comprehensive understanding of the results, the survey aimed to
provide students with the opportunity to express their opinions regarding the research
questions. By addressing these research questions, this study seeks to contribute to the
growing body of knowledge on the effective integration of gamification and automated
assessment in web-based learning environments. The findings will not only inform the
design and development of future educational tools but also provide valuable guidance
to educators and instructional designers, aiming to create engaging and effective learning
experiences in the field of programming.
Additionally, to supplement the student-centered evaluations, informal interviews
were conducted with the professors of the course, who are also co-authors of this paper.
Experts in the field of gamification were invited to use the tool and offer their academic
opinions. Furthermore, participating students were also informally interviewed to gather
firsthand feedback on their experiences with the gamified learning environment. These
interviews serve to triangulate the data, adding depth and context to the findings obtained
through the student surveys.
2.2. Web-Based Tool
The presented tool is a user-friendly and accessible web-based platform designed
to enhance the learning experience and increase student motivation in a programming
course. The tool offers a seamless and intuitive interface, making it easy for students to
navigate and interact with the platform. Through the implementation of a reward system,
including achievements, badges, and points, students are motivated to actively engage in
their learning journey. These rewards not only recognize their progress and achievements
but also provide a sense of accomplishment and satisfaction.
Additionally, a ranking system is incorporated into the tool, allowing students to
compare their performance with their peers. This healthy competition fosters a sense of
challenge and encourages students to strive for excellence in their programming skills. By
integrating gamification elements into the learning process, the goal is to create an enjoyable
and motivating environment for students, instilling feelings of excitement, curiosity, and a
desire to continually improve their abilities.
The tool is available online at URL https://gamificacion.cc.uah.es/ accessed on
9 August 2023, which is the Spanish version that has been used in the present research work.
Furthermore, the design stages of the web-based tool have been meticulously planned
to ensure its effectiveness and user engagement. These stages encompass research and anal-
ysis, objective definition, user experience design, gamification elements design, education
content development, technical development, testing, and finally, launch and deployment.
A comprehensive visualization of these design stages can be found in Figure 1.
During the “Educational Content Development” stage, the course content was care-
fully integrated into the platform. It is pertinent to note that this content has remained
consistent and has been used across multiple academic years. Given the stability and
time-tested nature of the content, there were no immediate requirements to update the
software post-deployment.
In addition, the “Testing” phase was not solely confined to internal evaluations.
Rigorous testing was undertaken, involving not just the development team but also students
from other courses. This approach was adopted to garner a diverse range of feedback and
to ensure that the platform was free from errors and provided an optimal user experience.
Any discrepancies or issues identified during this phase were promptly addressed, ensuring
that the software was both robust and reliable upon its final deployment.
Sustainability 2023,15, 14119 7 of 20
Figure 1.
Illustrative overview of the design stages for the creation of the web-based gamified
educational tool.
In summary, the platform integrates several key features to further enhance the user
experience. It boasts an intuitive user interface, a robust reward system, progressive
challenges and levels, immediate feedback mechanisms, comprehensive evaluation systems,
and personalized content tailored to individual user progress. A notable aspect is the
competitive edge introduced by allowing students to compare their scores with peers
through a ranking system. Subsequent subsections will delve deeper into some of these
features, providing a more detailed exploration of their implementation.
2.2.1. Points
Points serve as the backbone of the gamification aspect of the system, providing a
tangible measure of achievement and progress. The system employs two types of points:
experience points and redeemable points. Each type serves a distinct purpose and is earned
through different mechanisms, fostering a sense of accomplishment and incentivizing
active participation.
Experience points are designed to track and reward users’ progress in the system.
Users accumulate these points as tasks are successfully completed, tests are passed, and pro-
gramming skills are demonstrated. By earning experience points, users are gradually
leveled up, new challenges are unlocked, and access to higher-level tasks is granted.
The leveling system aims to provide a structured learning path that allows users to develop
their skills incrementally.
On the other hand, redeemable points offer users the opportunity to personalize their
learning journey and customize their experience within the system. These points can be
earned by achieving specific milestones, reaching certain levels, or obtaining exceptional
results. Redeemable points can be accumulated by users and then used to purchase items
from the store.
2.2.2. Achievements
Achievements are rewards that users receive for different actions performed, similar
to badges within the platform. An example of these achievements, including a welcome
achievement and a completion achievement for a task group called familiarization, is
shown in Figure 2.
Upon unlocking an achievement, users not only receive recognition but also earn
redeemable points. These points can be accumulated and used in the store section of the
platform, where users can exchange them for a wide range of items and rewards. This incen-
Sustainability 2023,15, 14119 8 of 20
tivizes users to actively engage with the learning process, striving to unlock achievements
and accrue points for further exploration and personal growth within the system.
Figure 2.
The user profile displays the student’s achievements. In this case, the welcome and
completed familiarization task group achievements are shown.
2.2.3. Badges
Badges are rewards given to users based on their actions while using the tool. An ex-
ample of two badges received by a student for completing tasks on the platform is shown
in Figure 3. These badges serve as incentives to encourage users to perform certain actions.
For instance, when students complete all the tasks in a task group on the gamification
platform, they may be awarded a badge. Each badge has a distinct design, making it easy
to identify.
By earning badges, users can showcase their skills and knowledge through a vi-
sually organized collection. This collection serves as a testament to their progress and
achievements within the platform, motivating them to further explore and excel in their
programming journey.
Figure 3.
The user profile displays the student’s badges. In this case, these two badges demonstrate
that the student has completed two task groups.
2.2.4. Leaderboards
In this web-based tool, leaderboards play a significant role in enhancing student
motivation and engagement. It has been expressed by students that the leaderboards have
been one of the most motivating gamification elements as they compete against other
students from different computer science-related degrees.
A global leaderboard is available, showcasing the top ten users, as depicted in
Figure 4.
This leaderboard displays the ranking position, username, university degree
of the student, current level, and experience points. The positions are determined based on
the accumulated experience points, with the leaderboard sorted in descending order.
Sustainability 2023,15, 14119 9 of 20
Figure 4.
The leaderboards are displayed on a separate web page. The image shows the top four,
but by scrolling down, the remaining six can be seen.
To prevent students from feeling demotivated by not being among the top ten positions
in the global leaderboard, another leaderboard has been implemented. This personal
leaderboard, shown in Figure 5, allows students to view their individual position, even if it
is lower. It enables them to identify the students directly above and below their position,
fostering a sense of competition and encouraging them to strive for improvement.
Figure 5.
The personal ranking is also displayed on the same web page as the other leaderboard,
but below it.
The implementation of leaderboards in the tool has proven to be highly effective in
motivating students to actively participate and progress in the programming course. It
provides a visible representation of their achievements and progress, fostering a healthy
competitive spirit among students.
Additionally, the leaderboards promote a sense of accountability and self-improvement.
Students are aware that their performances are being tracked and compared to others,
which instills a desire to surpass their own previous achievements. This intrinsic motiva-
tion drives students to continuously challenge themselves and aim for higher levels and
experience points.
2.2.5. Store
The store page is another important element in a gamification tool. In this web-based
gamification tool, this feature provides students with an exciting opportunity to personalize
their experience and make meaningful choices with their redeemable points.
Sustainability 2023,15, 14119 10 of 20
The store offers a range of items that can be purchased by students to enhance their
profile and overall learning journey. With the redeemable points, different items such as
“Change Avatar”, “Upload Animated Avatar”, “Change Profile Background”, and “+0.2 in
the Final Grade” can be purchased. An example of one of these items is shown in Figure 6.
When a purchase is made, the redeemable points are deducted from the student’s current
points. This section of the web tool aims to offer students a clear objective for completing
tasks, allowing them to earn redeemable points through achievements and subsequently
spend those points in the store.
Figure 6.
A store item. Purchasing this item allows the user to customize their profile by changing
their avatar.
2.2.6. Tests
When a task is accessed by the student, three elements are presented on the same
page. Firstly, there is the exercise statement, which can be an example like the one shown
in Figure 7. Secondly, there is the code editor, which initially displays a skeleton code for
the student to work on, as seen in Figure 8. Lastly, there are unit tests that the code must
pass, as exemplified in Figure 9. These tests display both the input and the expected output.
The output must match the expected output for the test to be considered valid. On the other
hand, hidden tests serve the same purpose, but the input and expected output are unknown.
In this way, the student must write the correct code and cannot cheat. Information is also
provided to the user in case of execution and compilation errors.
In Figure 9, various buttons with different functionalities can also be observed.
The “Compile” button serves to encode the user’s code into base64 format and send
it to the automated programming assessment system solely for the purpose of compilation.
In case of compilation errors, a notification is provided to the user. The “Test” button, on the
other hand, sends both the code and the unit tests for evaluation. Feedback regarding
the success or failure of each test is conveyed to the user. The “Send” button becomes
available only when all tests have been successfully passed, thus allowing the exercise to be
submitted as successfully completed; this status is subsequently recorded in the database.
Figure 7.
One of the initial exercise statements that will be seen by the student. It is displayed on the
left side of the task page, and the text is encoded with Markdown.
Sustainability 2023,15, 14119 11 of 20
Figure 8.
The code editor used to perform the task. It is located on the task page, to the right of the
exercise statement.
Figure 9.
The tests necessary to pass the task. In this case, a regular test can be seen that expects an
output, but there are also hidden tests for which neither the input nor the expected output is known.
The compile, run tests, and submit buttons are also situated here. This section is located just below
the exercise statement and the code editor.
Notably, the feedback mechanism plays a vital role in motivating students to actively
engage with the tests. When their code fails a test, students are encouraged to review the
error message and identify the specific issue in their implementation. This iterative process
of debugging and refinement stimulates the students’ problem-solving skills and enhances
their understanding of programming concepts.
In summary, the suggestions that the software can offer to students to correct their
errors are as follows:
1. Visibility of unit tests:
These tests clearly display both the input and the expected
output. This allows students to have a concrete understanding of what their code is
supposed to achieve. If their code does not produce the expected output, they know
where the issue lies.
2. Hidden tests:
To ensure genuine understanding and to avoid the potential for students
to simply game the system, there are also hidden tests. These tests do not reveal the
input or expected output. As a result, students must rely on the correctness of their
code rather than attempt to fit their solution to a known output.
3. Feedback mechanism:
In the event of a failed test, the software provides an error mes-
sage, guiding the student to the specific issue in their code. This immediate feedback
mechanism encourages the student to iterative debug and refine their code, reinforcing
their problem-solving skills and deepening their grasp of programming concepts.
Thus, the software does not simply ask students to “start all over again” without
guidance. Instead, it offers valuable feedback that facilitates their learning process. This
continuous testing, combined with real-time feedback, is central to the software’s pedagog-
ical approach. It offers students a more engaging, dynamic learning environment and has
the added benefit of boosting their motivation to actively participate and perform well in
the course.
By integrating testing as an integral part of the gamified tool, students are empowered
to track their progress, identify areas for improvement, and develop a growth mindset.
Sustainability 2023,15, 14119 12 of 20
The combination of immediate feedback, continuous testing, and iterative problem-solving
creates a dynamic learning environment that nurtures students’ programming skills and
fuels their intrinsic motivation to excel in the course.
3. Results
This section presents both the quantitative and qualitative results obtained through
the students’ questionnaires. A subsection is also included, containing a discussion of
these results.
3.1. Questionnaire Results
This subsection presents the results obtained from the survey responses regarding the
effects of the gamification tool on the motivation of students in the programming course.
The questionnaire aimed to assess various factors related to motivation and their influence
on students’ learning experiences. Figures 10–14 display the numerical outcomes obtained
from the survey responses, providing an overview of the motivation levels attributed to
the gamification platform.
The use of the gamification platform has positively influenced students’ motivation to
attend class, as can be seen in Figure 10. A significant percentage of participants (52.21%)
expressed agreement or complete agreement in this regard. Conversely, a small fraction
(12.39%) disagreed or completely disagreed with the statement, suggesting that the platform
may not have been as effective in motivating their class attendance.
Figure 10. Motivation to attend class among students.
According to the results presented in Figure 11, it was found that the participants’
motivation to study was increased by the gamification platform, resulting in a higher
accumulation of points. A significant majority (86.72%) either agreed or completely agreed
with this statement, highlighting the crucial role played by the platform in enhancing their
motivation and academic achievements.
Sustainability 2023,15, 14119 13 of 20
Figure 11. Motivation to study among students, based on their responses.
The acquisition of points through the platform has positively impacted students’
self-confidence, as illustrated in Figure 12. The survey results revealed that 75.22% of
participants agreed or completely agreed that earning points enhanced their self-assurance,
indicating that the reward system embedded in the platform played a significant role in
boosting their confidence.
Figure 12.
Impact of earning points on self-confidence among students, as indicated by their feedback.
The competitive environment created by the platform’s leaderboard has served as
a catalyst for student motivation, as shown in Figure 13. A considerable proportion of
participants (38.05%) completely agreed and an additional percentage (15.04%) agreed
that the competitive aspect of the leaderboard heightened their motivation. Furthermore,
a notable proportion of participants (31.86%) remained neutral, indicating a range of
Sustainability 2023,15, 14119 14 of 20
responses to the competitive element. A smaller percentage (12.39%) disagreed, while a
minority (2.66%) strongly disagreed.
Figure 13.
Influence of the leaderboard on motivation among students, based on their perspectives.
Students expressed that encountering errors in their code and failing tests motivated
them to actively seek and rectify these errors. The majority of participants (72.56%) agreed or
completely agreed with this statement, as depicted in Figure 14, indicating that the platform
effectively stimulated students’ motivation to engage with and learn from their mistakes.
Figure 14.
Motivation derived from encountering errors and failing tests among students, according
to their responses.
The survey results provide valuable insights into the effects of the gamification plat-
form on student motivation in the programming course. While the majority of students
reported positive experiences and benefits from using the platform, it is important to
acknowledge that a minority of participants did not find it as motivating. These findings
Sustainability 2023,15, 14119 15 of 20
highlight the need for further investigation and refinement of the gamification elements to
cater to the diverse motivational needs of students (Table 1).
Table 1. Statistical summary of the survey evaluating the gamification platform.
Item Mean Median Mode SD
The use of the tool has motivated me to attend class. 3.58 4.0 3 1.04
I was more motivated to study and therefore obtain
more points. 4.33 5.0 5 0.84
Achieving points improved my self-confidence. 4.12 4.0 5 0.86
The competitive environment of the leaderboard enhances
my motivation. 3.73 4.0 5 1.17
When my code does not pass the tests, I feel motivated to
look for errors and fix them 3.87 4.0 4 0.91
Inferential Statistical Analysis
One-sample T-tests were used to analyze quantitative data. The objective was to reach
a conclusion about whether the use of the gamification tool has had a significant impact on
student motivation to study the subject. The following null hypotheses were posed:
• The use of the tool does not motivate students to attend class (µ≤3).
• The students are not more motivated to study and to obtain more points (µ≤3).
• Achieving points does not improve the students’ self-confidence (µ≤3).
•
The competitive environment of the leaderboard does not enhance the students’
motivation (µ≤3).
•
The students do not feel motivated to look for errors and fix them when their code
does not pass the tests (µ≤3).
Table 2summarizes the results of the hypothesis tests. According to the p-values ob-
tained, every null hypothesis is rejected. Therefore, it can be concluded that the population
mean is greater than three for the five tests. This means that the use of the gamification
tool has a significant effect on the students’ motivation. Specifically, the students are more
motivated to attend class and to study the subject when they achieve points and when they
advance in the leaderboard. This results in an improvement in their self-confidence. Also,
they feel motivated to look for errors and fix them when their code does not pass the tests.
Table 2. Summary of hypothesis tests on student motivation.
Item p-Value
The use of the tool does not motivate students to attend class. 1.475415 ×10−8
The students are not more motivated to study and to obtain more points. 1.026765 ×10−32
Achieving points does not improve the students’ self-confidence. 1.592054 ×10−26
The competitive environment of the leaderboard does not enhance the stu-
dents’ motivation. 5.371024 ×10−10
The students do not feel motivated to look for errors and fix them when their
code does not pass the tests. 8.823050 ×10−18
3.2. Open Question: Motivation Dimension
The open question “Which part did you like the most?” was included in the survey
to gather insights into the factors that have influenced student motivation. The students
provided responses that encompassed multiple topics, indicating a diverse range of interests
and focus areas. The results presented in Figure 15 reveal the following topics and their
corresponding influence percentages: 13.7% for tests/hidden tests, 12.8% for points, 12% for
the final grade, 11.1% for the practical part, 9.4% for rewards/store, 8.5% for other factors,
Sustainability 2023,15, 14119 16 of 20
7.7% for the general experience, 6.8% for ranking, 5.1% for learning, 4.3% for simplicity,
3.4% for fun, 2.6% for code evaluation, and 2.6% for nothing.
Tests/hidden tests, points, final grades, and the practical parts were the criteria that
had the greatest influences; collectively, they accounted for almost 50% of the responses.
These findings imply that students were driven by the challenge and importance of the
course material, as well as by extrinsic and intrinsic rewards. Code evaluation, nothing,
fun, and simplicity, which together accounted for less than 13% of the responses, were the
least significant factors. These findings suggest that other characteristics of the gamified
system were more highly valued by students than the feedback mechanism, entertainment
value, or convenience of use. The remaining factors, which ranged from 5.1% to 9.4% of the
replies, included rewards/store, other factors, general experience, ranking, and learning,
which moderately impacted student motivation. These findings highlight the variety of
student preferences and the necessity to create gamified lessons that accommodate various
learning objectives and styles.
Figure 15. Most influential motivational topics, as perceived by students.
In addition to the quantitative analysis, it is important to consider the qualitative analysis
of comments provided by students. Several students provided interesting answers regarding
their preferences and the factors that motivated them within the gamification platform.
The following are some of the relevant statements made by students in response to
the question:
• “The hidden tests that ensure consideration of all possible cases.”
• “The progression system: points, achievements, etc.”
• “Being motivated by the reward system.”
•
“The leaderboard made us want to outperform other university degrees and have
more classmates in the top 10.”
These statements highlight the positive impact of various gamification elements on
student motivation and engagement. The presence of hidden tests ensures complete
coverage of all possible cases, promoting a sense of thoroughness and mastery. The progress
system, with its points and achievements, serves as a strong motivator for students to work
harder and excel in their performance. The reward system further enhances motivation by
providing tangible incentives for their efforts. The leaderboard, in particular, generates a
competitive spirit among students, driving them to surpass their peers and attain a coveted
position in the top 10. However, it is essential to acknowledge that the leaderboard can also
have negative consequences, as it may demotivate students at the lower end of the rankings.
Sustainability 2023,15, 14119 17 of 20
To address this, an additional personal ranking system was implemented, as explained in
Section 2.2.4.
Overall, the students’ feedback confirms the importance and effectiveness of gamifica-
tion techniques in the programming course. These techniques provide additional motivation,
foster a desire for improvement, and create a more engaging learning environment. It is cru-
cial to strike a balance, ensuring that the gamification elements promote healthy competition
while also promoting collaboration and a supportive learning community.
3.3. Interviews
This subsection aims to elaborate on the qualitative insights gleaned from informal
interviews conducted with course instructors, academic experts in gamification, and partic-
ipating students.
Firstly, the instructors of the course, who are also the co-authors of this paper, conveyed
a highly favorable view of the gamified learning tool. They observed heightened levels of
student motivation and engagement during the course. The consensus among the educators
was that the tool has potential for broader application across various subjects.
Secondly, academic experts specialized in gamification, having utilized the tool over
a period, vouch for its suitability in the context of this study. In their opinion, based on
years of experience in the field, the results generated by this tool have the potential for
generalization across educational contexts in Spain.
Lastly, student participants expressed enthusiastic approval of the gamified learning
environment. Specific attention was drawn to the competitive aspect engendered by the
leaderboard, which pitted students from different degrees against each other. This feature
was noted as particularly motivating and engaging.
4. Discussion and Conclusions
In this section, the implications of the results will be discussed in relation to the
research questions, and the study’s limitations and the verification of the hypotheses will
be addressed. Additionally, a comparison with previous studies will be conducted to assess
the alignment of the findings.
Regarding the first research question, “How does the integration of a gamification sys-
tem impact students’ motivation and engagement in a programming course?”, the results
from the survey indicate that the integration of the gamification system has had a positive
impact on students’ motivation and engagement in the programming course. A significant
percentage of students expressed that the use of the tool had motivated them to attend
class, with 52.21% either completely agreeing or agreeing with this statement. This finding
suggests that the gamification elements implemented in the platform have successfully
incentivized students to actively participate in the course.
Furthermore, a majority of the students (82.72%) reported being more motivated to
study and obtain more points. This high level of agreement demonstrates the influence of
the gamification system in fostering students’ intrinsic motivation to excel in their studies.
The correlation between increased motivation and the accumulation of points suggests that
the point-based system effectively incentivizes students to strive for higher achievement.
These findings are in line with previous studies that have explored the effects of
gamification on student motivation and engagement in educational settings. For instance,
Campillo-Ferrer et al. [
29
] found that using Kahoot! quizzes improved students’ perception
of certain concepts, increased their active participation, and motivated them toward learn-
ing in a more interactive and stimulating environment. Similarly, Ndlovu and Mhlongo [
30
]
found that gamification has the potential to trigger and maintain students’ situational inter-
est, which serves as a foundation for an engaged and motivated student. The consistent
findings across these studies and the gamification tool presented in this paper support the
notion that gamification can be an effective strategy to enhance student motivation and
engagement in programming courses.
Sustainability 2023,15, 14119 18 of 20
Regarding the second research question, “Which functionalities of the web-based
gamified tool influence in the students’ motivation?”, the survey results shed light on
the functionalities of the web-based gamified tool that have a significant influence on
students’ motivation. The competitive environment of the leaderboard was identified as
a strong motivator, with 53.09% of students either completely agreeing or agreeing that
it enhances their motivation. This suggests that the leaderboard effectively simulates a
sense of competition among students, driving them to outperform their peers and achieve
higher ranks.
The acquisition of points was also found to have a positive impact on students’ mo-
tivation, as 75.72% of students either completely agreed or agreed that achieving points
improved their self-confidence. This indicates that the point system is an effective ex-
trinsic motivator, boosting students’ confidence and providing a tangible measure of
their progress.
Additionally, when students’ code did not pass the tests, a majority (72.56%) expressed
feeling motivated to search for errors and rectify them. This finding highlights the intrinsic
motivation that arises from the challenge of overcoming coding obstacles, indicating that
the gamified tool effectively promoted problem-solving skills and perseverance.
The survey also included an open question asking students about the specific parts
of the web-based gamified tool they liked the most. The results provide further insights
into the functionalities of the tool that influence students’ motivation. The high percentage
of students expressing a liking for tests/hidden tests, points, and the final grade suggests
that these elements contribute significantly to their motivation. The practical part, re-
wards/store, and other factors also play a notable role in influencing student motivation.
The rankings and the overall gamified experience are mentioned as influential factors
as well.
These findings complement the previous results regarding the effects of the leader-
board and point system on motivation. The positive influence of tests/hidden tests, prac-
tical components, and other factors align with the gamification principles of challenges,
feedback, and variety, which are known to enhance motivation in educational contexts.
The functionalities identified in this study are consistent with prior research, highlight-
ing the motivational impact of leaderboards, points, and challenges in gamified learning
environments. For example, in [
31
], the authors found that points and leaderboards stim-
ulated structural and trait competitiveness, which increased engagement. Additionally,
ref. [
32
] concluded that leaderboards were successful in motivating participants to perfor-
mance levels similar to that of difficult and impossible goal-setting. Thus, the findings align
with previous research, indicating that these functionalities play a crucial role in driving
student motivation in programming courses.
The study investigated the effects of a web-based gamified tool on students’ motivation
and engagement in a programming course. Results showed that the gamification platform
positively influenced students’ motivation to attend class and study. It also boosted their
self-confidence and fostered a competitive spirit through the leaderboard. Encountering
errors motivated students to learn from their mistakes.
Despite the positive implications of the research, certain limitations must be taken
into account. First, the study was conducted solely within one educational institution,
potentially narrowing the generalization of the findings. Second, the absence of a control
group for comparison restricts the ability to definitively attribute the observed effects solely
to the gamified tool. Both limitations present avenues for future work.
Also, it can be concluded that the research hypotheses have been empirically sub-
stantiated. The integration of a gamification system has demonstrably enhanced student
motivation and engagement in a programming course. While certain limitations are ac-
knowledged, the study contributes valuable insights to the burgeoning body of knowledge
on gamification in educational settings.
In conclusion, the gamified tool significantly increased students’ motivation and en-
gagement in the programming course, combining automated assessment with gamification
Sustainability 2023,15, 14119 19 of 20
principles. Future work could involve measuring student performances by using this gam-
ification tool and assessing whether the implementation of an automated programming
assessment system in the gamification platform contributes to reducing the workload of
teachers. In addition to the mentioned future work, another important aspect to consider is
the application of this gamification tool in various universities, to gather a diverse range of
examples and feedback. By implementing the tool in different educational institutions, it
will be possible to assess its effectiveness across various academic settings and disciplines.
Author Contributions:
Conceptualization, C.J.H. and F.C.-E.; methodology, J.G. and A.T.; software,
C.J.H. and F.C.-E.; validation, C.J.H. and F.C.-E.; resources, A.V.; writing—original draft preparation,
C.J.H.; writing—review and editing, J.G., A.T. and S.O.-T.; supervision, A.T.; funding acquisition, J.G.
All authors have read and agreed to the published version of the manuscript.
Funding:
This work was supported by the program “Programa de Estímulo a la Excelencia para Pro-
fesorado Universitario Permanente” of the Vice-Rectorate for Research and Knowledge Transfer of the
University of Alcala and by the Comunidad de Madrid (Spain) through project EPU-INV/2020/004.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Not applicable.
Conflicts of Interest: The authors declare no conflict of interest.
References
1.
Molina-Carmona, R.; Llorens-Largo, F. Gamification and Advanced Technology to Enhance Motivation in Education. Informatics
2020,7, 20. [CrossRef]
2.
Jusas, V.; Barisas, D.; Janˇciukas, M. Game Elements towards More Sustainable Learning in Object-Oriented Programming Course.
Sustainability 2022,14, 2325. [CrossRef]
3.
Zahedi, L.; Batten, J.; Ross, M.; Potvin, G.; Damas, S.; Clarke, P.; Davis, D. Gamification in education: A mixed-methods study of
gender on computer science students’ academic performance and identity development. J. Comput. High. Educ.
2021
,33, 441–474.
[CrossRef]
4.
van Gaalen, A.; Brouwer, J.; Schönrock-Adema, J.; Bouwkamp-Timmer, T.; Jaarsma, D.; Georgiadis, J. Gamification of health
professions education: A systematic review. Adv. Health Sci. Educ. 2021,26, 683–711. [CrossRef] [PubMed]
5.
Marín, B.; Frez, J.; Cruz-Lemus, J.A.; Genero, M. An Empirical Investigation on the Benefits of Gamification in Programming
Courses. ACM Trans. Comput. Educ. (TOCE) 2018,19, 4. [CrossRef]
6.
Fotaris, P.; Mastoras, T.; Leinfellner, R.; Rosunally, Y. Climbing up the leaderboard: An empirical study of applying Gamification
techniques to a computer programming class. Electron. J. E-Learn. 2016,14, 94–110.
7. Buckley, P.; Doyle, E. Gamification and student motivation. Interact. Learn. Environ. 2016,24, 1162–1175. [CrossRef]
8.
Panmei, B.; Waluyo, B. The Pedagogical Use of Gamification in English Vocabulary Training and Learning in Higher Education.
Educ. Sci. 2023,13, 24. [CrossRef]
9.
Alzahrani, F.K.; Alhalafawy, W.S. Gamification for Learning Sustainability in the Blackboard System: Motivators and Obstacles
from Faculty Members & Perspectives. Sustainability 2023,15, 4613. [CrossRef]
10.
Aldalur, I.; Perez, A. Gamification and discovery learning: Motivating and involving students in the learning process. Heliyon
2023,9, e13135. [CrossRef]
11.
Reigada, C.; Sandgren, A.; Rivas, S.; Carvajal, A.; Hermida-Romero, S.; Benítez, E.; Ripoll, G.; Olza, I.; Centeno, C.; Gómez, B.
Palliative care stay room—Designing, testing and evaluating a gamified social intervention to enhance palliative care awareness.
BMC Palliat. Care 2023,22, 46. [CrossRef] [PubMed]
12.
Wang, J.; Fang, Y.; Frank, E.; Walton, M.A.; Burmeister, M.; Tewari, A.; Dempsey, W.; NeCamp, T.; Sen, S.; Wu, Z. Effectiveness of
gamified team competition as mHealth intervention for medical interns: A cluster micro-randomized trial. npj Digit. Med.
2023
,
6, 4. [CrossRef] [PubMed]
13.
Zourmpakis, A.I.; Kalogiannakis, M.; Papadakis, S. Adaptive Gamification in Science Education: An Analysis of the Impact of
implementation and Adapted game Elements on Students & Motivation. Computers 2023,12, 143. [CrossRef]
14.
Durmaz, T.B.; Fuertes, J.L.; Imbert, R. Influence of Gamification Elements on Explicit Motive Dispositions. IEEE Access
2022
,
10, 118058–118071. [CrossRef]
15.
Ortega-Arranz, A.; Asensio-Pérez, J.I.; Martínez-Monés, A.; Bote-Lorenzo, M.L.; Ortega-Arranz, H.; Kalz, M. GamiTool:
Supporting Instructors in the Gamification of MOOCs. IEEE Access 2022,10, 131965–131979. [CrossRef]
16.
Khaldi, A.; Bouzidi, R.; Nader, F. Gamification of e-learning in higher education: A systematic literature review. Smart Learn.
Environ. 2023,10, 10. [CrossRef]
Sustainability 2023,15, 14119 20 of 20
17.
de Freitas, M.J.; da Silva, M.M. Systematic literature review about gamification in MOOCs. Open Learn. J. Open Distance E-Learn.
2023,38, 73–95. [CrossRef]
18.
Domínguez, A.; de Navarrete, J.S.; de Marcos, L.; Fernández-Sanz, L.; Pagés, C.; Martínez-Herráiz, J.J. Gamifying learning
experiences: Practical implications and outcomes. Comput. Educ. 2013,63, 380–392. [CrossRef]
19.
Toda, A.M.; Oliveira, W.; Klock, A.C.; Palomino, P.T.; Pimenta, M.; Gasparini, I.; Shi, L.; Bittencourt, I.; Isotani, S.; Cristea, A.I. A
Taxonomy of Game Elements for Gamification in Educational Contexts: Proposal and Evaluation. In Proceedings of the 2019 IEEE
19th International Conference on Advanced Learning Technologies (ICALT), Maceio, Brazil, 15–18 July 2019; Volume 2161-377X,
pp. 84–88. [CrossRef]
20.
Boeren, E. Surveys as tools to measure qualitative and quantitative data. In Handbook of Research on Scholarly Publishing and
Research Methods; IGI Global: Hershey, PA, USA, 2015.
21. Barabas, J.; Jerit, J. Are Survey Experiments Externally Valid? Am. Political Sci. Rev. 2010,104, 226–242. [CrossRef]
22.
Mills, L.; Abdulla, E.; Cribbie, R.A. Quantitative Methodology Research: Is it on Psychologists’ Reading Lists? Tutor. Quant.
Methods Psychol. 2010,6, 52–60. [CrossRef]
23. Nardi, P.M. Doing Survey Research: A Guide to Quantitative Methods; Routledge: London, UK, 2005.
24. EUSurvey Platform. Available online: https://ec.europa.eu/eusurvey/home/welcome (accessed on 18 May 2023).
25. Savage, N.; Birch, R.; Noussi, E. Motivation of engineering students in higher education. Eng. Educ. 2011,6, 39–46. [CrossRef]
26.
Rojas-López, A.; Rincón-Flores, E.G.; Mena, J.; García-Peñalvo, F.J.; Ramírez-Montoya, M.S. Engagement in the course of
programming in higher education through the use of gamification. Univers. Access Inf. Soc. 2019,18, 583–597. [CrossRef]
27.
Figueiredo, J.; García-Peñalvo, F.J. Increasing student motivation in computer programming with gamification. In Proceedings of
the 2020 IEEE Global Engineering Education Conference (EDUCON), Porto, Portugal, 27–30 April 2020; pp. 997–1000. [CrossRef]
28.
Fotaris, P.; Mastoras, T.; Leinfellner, R.; Rosunally, Y. From Hiscore to High Marks: Empirical Study of Teaching Programming
Through Gamification. In European Conference on Games Based Learning; Academic Conferences and Publishing Limited:
Manchester, UK, 2015.
29.
Campillo-Ferrer, J.M.; Miralles-Martínez, P.; Sánchez-Ibáñez, R. Gamification in Higher Education: Impact on Student Motivation
and the Acquisition of Social and Civic Key Competencies. Sustainability 2020,12, 4822. [CrossRef]
30.
Ndlovu, T.N.; Mhlongo, S. An investigation into the effects of gamification on students’ situational interest in a learn-
ing environment. In Proceedings of the 2020 IEEE Global Engineering Education Conference (EDUCON) Porto, Portugal,
27–30 April 2020; pp. 1187–1192.
31.
Amo, L.C.; Liao, R.; Kishore, R.; Rao, H.R. Effects of structural and trait competitiveness stimulated by points and leaderboards
on user engagement and performance growth: A natural experiment with gamification in an informal learning environment. Eur.
J. Inf. Syst. 2020,29, 704–730. [CrossRef]
32.
Landers, R.N.; Bauer, K.N.; Callan, R.C. Gamification of task performance with leaderboards: A goal setting experiment. Comput.
Hum. Behav. 2017,71, 508–515. [CrossRef]
Disclaimer/Publisher’s Note:
The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.
