ArticlePDF Available

Machine learning approach for student engagement automatic recognition from facial expressions

Authors:
Vladimir Soloviev at Financial University
Vladimir Soloviev
Download full-text PDF
Read full-text
Download citation

Figures

Figures - uploaded by Vladimir Soloviev
Author content
All figure content in this area was uploaded by Vladimir Soloviev
Content may be subject to copyright.
Content uploaded by Vladimir Soloviev
Author content
All content in this area was uploaded by Vladimir Soloviev on Mar 20, 2019
Content may be subject to copyright.
SCIENTIFIC PUBLICATIONS OF THE STATE UNIVERSITY OF NOVI PAZAR
SER . A: AP PL . MATH . INFORM.A ND ME CH . vol. 10, 2 (2018), 79-86.
Machine Learning Approach for Student Engagement
Automatic Recognition from Facial Expressions
Vladimir Soloviev
Abstract: Digital revolution can significantly improve the quality of education. There have
been already discussions for a long time about the advantages, disadvantages and opportunities
for transforming traditional classroom activities. Modern students use smartphones and tablets
”from birth”, and for the majority of academic subject areas students can often obtain more
complete, accurate and up-to-date information from the Internet than from lectures. Is it in-
teresting for students to learn? Are they in time with the professor? Is the presentation clear?
How deep are students engaged in learning in the classroom? These issues come to the fore-
front in the era of digital education. However, it was almost unrealistic to control the level of
student engagement until recently: for example, only in the Moscow campuses of the Financial
University the classes are held daily from 8.30 to 22.00 in more than 500 classrooms.
Existing information systems for student engagement automatic recognition are focused
on analyzing individual engagement of students and schoolchildren. We propose a system that
constantly analyzes the flow of data from video cameras installed in classrooms, uses machine
learning models to identify students’ faces, recognize their emotions and determine the level
of engagement, and then aggregates engagement data on student groups, faculties, courses, etc.
on interactive dashboards.
The training dataset consisted of 2,000 faces was used for machine learning model iden-
tification with boosted decision trees algorithm (ADABoost). The quality metrics (Accuracy,
Precision, Recall, AUC) on a test dataset of 500 students faces were all above 0,81.
The system is developed as an elastically scalable cloud service that automatically collects
video streams from cameras installed in classrooms and forms the resulting metrics of the
students and groups’ engagement in the Microsoft Azure cloud.
Manuscript received July 3, 2018; accepted October 4, 2018.
Vladimir Soloviev is with Data Analysis, Decision Making, and Financial Technology Department, Finan-
cial University under the Government of the Russian Federation, 38 Shcherbakovskaya St., Moscow 105187,
Russia e-mail: VSoloviev@fa.ru
79
80 Vladimir Soloviev
1 Introduction
Modern students use various computational devices ”from birth”, and on the Internet for
the majority of theoretical and practical academic disciplines students can often get more
complete, more accurate and more relevant information than from the classes. At the same
time, information on the Internet is often delivered more effectively and eye-catching than
in the classrooms.
Is it interesting for students to learn? Are students in time with the professor or the pace
of presentation is too fast or too slow? Is the presentation clear? How much are students
engaged in learning in classroom? These issues come to the forefront in the era of digital
education. However, it was almost impossible to control the level of student engagement
until recently: for example, only in the Moscow campuses of the Financial University the
classes are held daily from 8.30 to 22.00 in more than 500 classrooms.
Methods for measurement and analysis of student engagement in learning have been
actively developed since the 1980s, primarily with the aim to find a possibility to decrease
the number of expelling students. Surveys conducted at various universities and schools
showed, that from 25 to 60% of students are constantly bored in the classroom and dis-
tracted from learning (see, for example, [1, 2]).
Management of students’ engagement level is relevant nowadays for the traditional
classroom teaching, for MOOCs, for educational games, for simulators, for intelligent
teaching systems, etc. [3, 4, 5, 6].
The most common methods for student engagement measurement include self-assessment
by students themselves; external monitoring using control charts and subsequent rating; au-
tomatic measurement using technical means [7]. For example, the most often used method
in Russian studies is self-assessment (see, for example, [8]).
Information systems for automatic measurement of student engagement have been used
for a long time. A significant part of them is based on analysis of tests execution speed and
accuracy [9, 10]. For example, random answers to easy questions or very short lead times
could indicate weak engagement.
Another class of popular techniques for automatic measurement of engagement level
is based on data processing from various electro- and neurophysiological sensors [11, 12].
These methods could not be implemented at a large-scale, for example, at the level of a
whole university, because it is impossible to provide special sensors to every student at the
university.
The third class of techniques for automatic recognition of engagement, which includes
the system described in this paper, is based on the use of computer vision [13, 14, 15, 16,
17, 18]. Such techniques allow to assess a student’s engagement by analyzing the position
and inclination of the head, the view direction, pose, different gestures, and so on. The
major advantage of such systems is that the engagement level is measured unobtrusively,
without diverting students attention to the engagement measurement process itself.
Machine Learning Approach for Student Engagement Automatic Recognition from Facial Expressions81
This paper describes the experience of a cloud service development and implementa-
tion for monitoring student engagement in the classroom based on intelligent analysis of
video streams from cameras placed in the classrooms, and subsequent aggregation of av-
erage engagement for groups, courses, areas of training, education, faculty on interactive
dashboards.
In this case, the supposed engagement is measured, that is the level of student engage-
ment assessed by external experts.
Based on images of students in the classrooms, the system uses machine learning princi-
ples to determine whether or not this student is engaged. Initially, a large number of photos
of students’ faces made by video cameras in classrooms are presented to experts who di-
vide the photos into two classes (engaged and not engaged). Then the classification model
is trained on this dataset, labelled by experts, and after training the classification model is
used to predict the level of student engagement in the pictures, which neither experts nor
the classification model had previously seen.
The engagement recognition service is deployed in the Microsoft Azure cloud. The user
identification is based on the Microsoft Azure Active Directory directory services synchro-
nized with the on-premise university directory services. The identification of students is
based on pictures from the campus access control database, and the identification of classes
and professors is based on information from the on-premise classes schedule database.
Currently, the system is being piloted in two buildings of the Financial University, with
about 60 video cameras in the classrooms connected to it.
In all known systems engagement is measured on the basis of video streams from cam-
eras placed on individual computers. These systems are capable to measure level of en-
gagement of individual students in computer labs or in distance learning systems (including
massive open online courses). Unlike that we propose a system automatically measures the
engagement level not only for individual students but also for academic groups, faculties,
years, and for the whole university.
2 Cloud Solution Architecture
The architecture of the cloud solution is illustrated by Fig. 1. We use video cameras placed
under the ceiling of classrooms as the Internet of Things devices connected to the Microsoft
Azure IoT Hub. Before being sent to IoT Hub, video streams are preprocessed locally:
individual frames are captured at a specified periodicity.
Data on the classes schedule are taken in reference to the classrooms in which the video
cameras are placed from the on-premise classes schedule database. These data for each
class include classroom ID, start and end time, set of academic groups’ IDs (there usually
is one academic group at a seminar or lab, and more then one academic group at a lecture),
academic subject area ID, and the professor’s ID.
82 Vladimir Soloviev
Fig. 1. Engagement Monitoring Service Architecture
When the IoT Hub receives an image containing a snapshot of students, it sends it to the
Microsoft Azure Cognitive Services to recognize students’ faces and emotions (the students’
pictures in the Microsoft Azure Cognitive Services are synchronized with the campus access
control database). For each face in the picture the Microsoft Azure Cognitive Services return
recognized age, gender, Student ID (from the campus information system), head pose, facial
landmarks, indicators of lipstick, glasses, mustache, sideburns, beard, recognized emotions
(anger, contempt, disgust, fear, happiness, neutral, sadness, surprise), occlusion, etc.
For each face, all the features received from the emotion recognition services, as well
as the timestamp of the snapshot, the type of the class (lecture, seminar, computer lab,
etc.), academic subject area ID and professor’s ID are stored in the Microsoft Azure SQL
Database while images are stored in the Microsoft Azure BLOB Storage.
When a new entry appears in the faces and emotions recognition results table, this entry
is automatically submitted to the Microsoft Azure Machine Learning Studio web service
based on the previously trained classification model. This web service returns the scored
probability for classifying the student as engaged, and this probability is stored in the ap-
propriate field in the faces and emotions recognition results table.
We use the Microsoft Azure Stream Analytics service for real time event processing.
Another element of the system is the Microsoft Power BI service which is used to ag-
gregate the results of engagement level recognition from the Microsoft Azure SQL Database
to the dashboards published on the university portal.
3 Image Labelling Methodology
We developed a special application for images labelling. This application is published in
the Microsoft Azure cloud and allows experts to mark each face as engaged on not (Fig. 2).
Machine Learning Approach for Student Engagement Automatic Recognition from Facial Expressions83
Fig. 2. Engagement Labelling
We asked professors to assess the images of their students. Each professor receives
a task to assess engagement level for a certain number of recognized faces. Most of the
faces are automatically selected from the images obtained from the cameras during classes
taught by this professor (the proportion of such persons is a configurable parameter; we
recommend to set its value at the level of 90-95%), and the rest of the faces are taken from
the classes of other professors. This is necessary to ensure the adequacy of assessment.
Experts have also the opportunity not to evaluate the engagement for individual picture
in cases when the student’s engagement evaluation is impossible or unnecessary.
4 Machine Learning Model for Students’ Engagement Autodetection
We tried logistic regression, boosted decision trees and random forest models as the predic-
tors of engagement class. The best classification results are obtained using the ADABoost
Two-Class Boosted Decision Tree model. For this model the following quality metrics were
obtained: Accuracy = 84.8%, : Precision = 0.825, : Recall = 81.5%, : F1 Score = 82.0%, :
AUC = 91.2%. These results indicate the legitimacy of using the model to predict engage-
ment level.
Among the factors that have the greatest positive impact on the engagement level, the
following features are distinguished (in order of decreasing importance): head pose; recog-
nized age; level of sadness; level of surprise; and also some facial landmarks.
84 Vladimir Soloviev
5 Dashboards
The dashboard is placed on a single web page. In the top menu the user chooses the report-
ing period: semester summary, monthly or daily summary, or individual classes summary.
At any level of the hierarchy the information about engagement is displayed by the
universal display unit in the form of a discrete color scale for the selected period.
Initially, the dashboard displays the upper levels available for the particular user (uni-
versity, faculty, major, or academic subject area).
In the detail mode of the month, the level tree is made in the form of a classic explorer;
in front of each level (faculty, major, level of education, year, academic group, student,
professor), the information on average engagement for the selected level is displayed by
different colors.
A fragment of the dashboard is illustrated by Fig. 3.
Fig. 3. Engagement Monitoring Service Dashboard
6 Results and Discussion
Most of the tools traditionally used to measure the level of student engagement are too
complex to measure the dynamics of the engagement of all the students in the university on
an ongoing basis.
Our service developed in the Financial University for monitoring student engagement
based on intelligent analysis of video streams from cameras placed in classrooms and sub-
sequent aggregation of averaged data on dashboards is intended for use by the university
administration in order to obtain an operational feedback on the dynamics of the average
engagement of student groups during the semester, to compare the dynamics of changes
in the engagement between faculties, years, groups, etc., and to support the decisions on
appropriate corrective actions.
Machine Learning Approach for Student Engagement Automatic Recognition from Facial Expressions85
A distinctive feature of the proposed system is that it is built in the form of a cloud
service that can be used to monitor the engagement of arbitrarily large groups of students,
elastically scaling when the number of students changes. Such a service can be used at the
same time by several university or even across the entire education system.
The results of the pilot use of the service demonstrate a sufficient degree of engagement
prediction adequacy.
References
[1] Larson, R. and Richards, M. (1991). Boredom in the middle school years: Blaming schools
versus blaming students, American Journal of Education, 99, 418443.
[2] Dunleavy, J. and Milton, P. (2009). What did you do in school today? Exploring the concept
of student engagement and its implications for teaching and learning in Canada, Canadian
Education Association, 122.
[3] Anderson, J.R. (1982). Acquisition of cognitive skill, Psychological Review, 89 (4), 369406.
[4] Mostow, J., Hauptmann, A., Chase, L., and Roth, S. (1993). Towards a reading coach that lis-
tens: Automated detection of oral reading errors, In: Proceedings of the 11th National Confer-
ence on Artificial Intelligence (AAAI 1993), American Association for Artificial Intelligence,
Palo Alto, USA: AAAI Press, 392397.
[5] Koedinger, K.R. and Anderson, J.R. (1997). Intelligent tutoring goes to school in the big city,
International Journal of Artificial Intelligence in Education, 8, 3043.
[6] VanLehn, K., Lynch, C., Schultz, K., Shapiro, J., Shelby, R. and Taylor, L. (2005). The Andes
physics tutoring system: Lessons learned, International Journal of Artificial Intelligence in
Education, 15 (3), 147204.
[7] Harris, L. (2008). A phenomenographic investigation of professor conceptions of student en-
gagement in learning, Australian Educational Researcher, 5 (1), 5779.
[8] Maloshonok, N.G. (2014). Student engagement in learning in Russian Universities, Higher
Education in Russia, No 1, 3744 (in Russian).
[9] Beck, J. (2005). Engagement tracing: Using response times to model student disengagement,
In: Proceedings of the 2005 Conference on Artificial Intelligence in Education: Supporting
Learning through Intelligent and Socially Informed Technology, Amsterdam, Netherlands:
IOS Press, 8895.
[10] Johns, J. and Woolf, B. (2006). A dynamic mixture model to detect student motivation and
proficiency, In: Proceedings of the 21st National Conference on Artificial Intelligence (AAAI
2006), American Association for Artificial Intelligence, Palo Alto, USA: AAAI Press, 28.
[11] Pope, A., Bogart, E., and Bartolome, D. (1995). Biocybernetic system evaluates indices of
operator engagement in automated task, Biological Psychology, 40, 187195.
[12] Fairclough, S. and Venables, L. (2006). Prediction of subjective states from psychophysiology:
A multivariate approach, Biological Psychology, 71, 100110.
86 Vladimir Soloviev
[13] Kapoor, A., Picard, R. (2005). Multimodal affect recognition in learning environments, Pro-
ceedings of the 13th Annual ACM International Conference on Multimedia (MULTIMEDIA
2005), NY, USA: ACM, 677682.
[14] McDaniel, B., DMello, S., King, B., Chipman, P., Tapp, K. and Graesser, A. (2007). Facial
features for affective state detection in learning environments, In: Proceedings of the 29th
Annual Conference of the Cognitive Science Society, Austin, USA: Cognitive Science Society,
467472.
[15] DMello, S., Craig, S. and Graesser, A. (2009). Multimethod assessment of affective experience
and expression during deep learning, International Journal of Learning Technology, 4 (3),
165187.
[16] DMello, S. and Graesser, A. (2010). Multimodal semi-automated affect detection from con-
versational cues, gross body language, and facial features, User Modeling and User-Adapted
Interaction, 20 (2), 147187.
[17] Grafsgaard, J., Fulton, R., Boyer, K., Wiebe, E., and Lester J. (2012). Multimodal analysis of
the implicit affective channel in computer-mediated textual communication, In: Proceedings
of the 14th ACM international conference on Multimodal interaction (ICMI 2012), NY, USA:
ACM, 145152.
[18] Whitehill, J., Serpell, Z., Lin, Yi-Ch., Foster, A., and Movellan, J.R. (2014). The faces of
engagement: Automatic recognition of student engagement from facial expressions, IEEE
Transactions on Affective Computing, 5 (1), 8698.
[19] How to Detect Faces in Image // MicrosoftDocs on GitHub. URL: https://
github.com/MicrosoftDocs/azure-docs/blob/master/ articles/ cognitive-services/Face/Face-
API-How-to-Topics/HowtoDetectFacesinImage.md
... Moreover, e-learning or an online learning environment differs from an OCE. Some works have also been proposed for student engagement monitoring in an OCE or in similar settings [2,[22][23][24][25][26][27]. However, these proposals utilize dedicated physical devices or sensors [6,22,24], incorporate only behavioral features [7,23,25], test on limited data [8,27], are prone to efficiency issues [26], or use only emotional features [2]. ...
... Some works have also been proposed for student engagement monitoring in an OCE or in similar settings [2,[22][23][24][25][26][27]. However, these proposals utilize dedicated physical devices or sensors [6,22,24], incorporate only behavioral features [7,23,25], test on limited data [8,27], are prone to efficiency issues [26], or use only emotional features [2]. Moreover, some of these proposals utilize a single pretrained transfer-learning (TL) model [8,25,28] or train a machine-learning or convolutional neural network model [2,6,8,[22][23][24]26,27]. ...
... However, these proposals utilize dedicated physical devices or sensors [6,22,24], incorporate only behavioral features [7,23,25], test on limited data [8,27], are prone to efficiency issues [26], or use only emotional features [2]. Moreover, some of these proposals utilize a single pretrained transfer-learning (TL) model [8,25,28] or train a machine-learning or convolutional neural network model [2,6,8,[22][23][24]26,27]. These limitations show that more pretrained and self-trained models may be incorporated for measuring student engagement in order to compare the performance of the underlying models. ...
Measuring Student Engagement through Behavioral and Emotional Features Using Deep-Learning Models
Article
Full-text available
  • Oct 2024
Students’ behavioral and emotional engagement in the classroom environment may reflect the students’ learning experience and subsequent educational outcomes. The existing research has overlooked the measurement of behavioral and emotional engagement in an offline classroom environment with more students, and it has not measured the student engagement level in an objective sense. This work aims to address the limitations of the existing research and presents an effective approach to measure students’ behavioral and emotional engagement and the student engagement level in an offline classroom environment during a lecture. More precisely, video data of 100 students during lectures in different offline classes were recorded and pre-processed to extract frames with individual students. For classification, convolutional-neural-network- and transfer-learning-based models including ResNet50, VGG16, and Inception V3 were trained, validated, and tested. First, behavioral engagement was computed using salient features, for which the self-trained CNN classifier outperformed with a 97%, 91%, and 83% training, validation, and testing accuracy, respectively. Subsequently, the emotional engagement of the behaviorally engaged students was computed, for which the ResNet50 model surpassed the others with a 95%, 90%, and 82% training, validation, and testing accuracy, respectively. Finally, a novel student engagement level metric is proposed that incorporates behavioral and emotional engagement. The proposed approach may provide support for improving students’ learning in an offline classroom environment and devising effective pedagogical policies.
View
... Algoritma deep learning, seperti convolutional neural network (CNN), telah terbukti efektif dalam mendeteksi dan menganalisis ekspresi wajah dengan tingkat akurasi yang tinggi [2]. Teknologi ini memberikan pendekatan objektif dan otomatis untuk mengenali ekspresi wajah siswa yang menunjukkan ketertarikan mereka selama pembelajaran [3]. ...
Pengenalan Ekspresi Wajah untuk Mendeteksi Ketertarikan Siswa Sekolah Dasar dalam Mata pelajaran Menggunakan Metode CNN (Convolutional Neural Network)
Article
Full-text available
  • Mar 2025
Efektivitas pembelajaran sangat dipengaruhi oleh tingkat ketertarikan siswa terhadap materi yang disampaikan. Salah satu indikator ketertarikan siswa yang dapat diamati adalah ekspresi wajah mereka selama proses pembelajaran. Namun, pengamatan manual oleh pendidik sering kali kurang akurat dan tidak konsisten. Oleh karena itu, pengenalan ekspresi wajah berbasis teknologi menjadi alternatif yang potensial untuk memantau dan menganalisis ketertarikan siswa secara objektif dan real-time. Penelitian ini menggunakan algoritma deep learning untuk mendeteksi ekspresi wajah siswa dan mengklasifikasikannya ke dalam kategori ketertarikan. Data diambil dari video pembelajaran yang direkam di lingkungan kelas. Model convolutional neural network (CNN) digunakan untuk ekstraksi fitur wajah, sementara algoritma klasifikasi emosi diterapkan untuk mengenali ekspresi yang relevan. Dataset pelatihan dan pengujian berupa gambar yang di photo kemudian di proses untuk menentukan ekspresi wajah siswa yang dan diberi label berdasarkan tingkat ketertarikan. Hasil penelitian menunjukkan bahwa model deep learning yang dikembangkan memiliki tingkat akurasi yang baik dalam mengenali ekspresi wajah siswa terkait ketertarikan. Model ini berhasil mendeteksi perubahan ekspresi secara konsisten dalam berbagai kondisi pencahayaan dan sudut pandang kamera. Dengan hasil ini, sistem yang diusulkan memiliki potensi untuk diimplementasikan sebagai alat pendukung dalam meningkatkan kualitas pembelajaran berbasis teknologi dapat mendukung dalam menentukan metode mengajar yang tepat.
View
... Akibatnya, metode ini kurang mampu menyesuaikan dengan gaya belajar individu siswa, yang berujung pada rendahnya motivasi dan keterlibatan mereka dalam proses belajar (Dzobelova et al., 2020). Selain itu, metode ini kurang efisien dalam memanfaatkan teknologi yang saat ini menjadi bagian penting dari kehidupan siswa modern, sehingga tidak memberikan pengalaman belajar yang relevan dengan tuntutan dunia kerja dan kehidupan digital saat ini (Soloviev, 2018). (Mitu, 2013). ...
PEMBELAJARAN MASA DEPAN: Transformasi AI dan E-learning di Era Pendidikan Digital
Book
Full-text available
  • Oct 2024
Buku ini menggambarkan bagaimana kecerdasan buatan (AI) dan e-learning mengubah wajah pendidikan masa depan. AI memberikan kemampuan untuk menciptakan pengalaman belajar yang personal, di mana setiap siswa mendapat materi dan metode pengajaran yang disesuaikan dengan kebutuhan unik mereka. Teknologi ini tidak hanya membuat proses belajar lebih menarik dan relevan, tetapi juga lebih inklusif, memungkinkan akses pendidikan yang lebih luas. Namun, di balik semua keunggulan seperti peningkatan motivasi dan hasil belajar, tantangan seperti privasi data dan kemampuan AI untuk beradaptasi secara real-time masih menjadi perhatian utama. Meski begitu, masa depan pendidikan digital dengan AI menjanjikan pengalaman yang lebih efisien, interaktif, dan memberdayakan.
View
... Other applications of EDM and AI for the transformation of the traditional classroom include the analysis of student facial expressions to assess their level of engagement in the classroom. Soloviev et al. [112] proposed a system that analyzes (in real-time) the data feeds from video cameras that are installed in the classroom and applies AI and facial recognition technology to recognize student emotions to determine their level of enjoyment. Although Chua et al. [113] reviewed case studies and technologies developed to collect and analyze educational data. ...
Data-Driven Artificial Intelligence in Education: A Comprehensive Review
Article
  • Jan 2023
As education constitutes an essential development standard for individuals and societies, researchers have been exploring the use of artificial intelligence (AI) in this domain and have embedded the technology within it through a myriad of applications. In order to provide a detailed overview of the efforts, this article pays particular attention to these developments by highlighting key application areas of data-driven AI in education; it also analyzes existing tools, research trends, as well as limitations of the role data-driven AI can play in education. In particular, this article reviews various applications of AI in education including student grading and assessments, student retention and drop-out predictions, sentiment analysis, intelligent tutoring, classroom monitoring, and recommender systems. This article also provides a detailed bibliometric analysis to highlight the salient research trends in AI in education over nine years (2014–2022) and further provides a detailed description of the tools and platforms developed as the outcome of research and development efforts in AI in education. For the bibliometric analysis, articles from several top venues are analyzed to explore research trends in the domain. The analysis shows sufficient contribution in the domain from different parts of the world with a clear lead for the United States. Moreover, students' grading and evaluation have been observed as the most widely explored application. Despite the significant success, we observed several aspects of education where AI alone has not contributed much. We believe such detailed analysis is expected to provide a baseline for future research in the domain.
View
... Data mining and AI technologies are transforming the traditional classroom and how student participation in the class is assessed. Soloviev (2018) proposes a system that constantly analyzes data feeds from video cameras installed in the classroom. The real-time video data were used to train boosted decision tree algorithms to identify students' faces, recognize emotions, determine engagement levels, and assess class participation. ...
Designing, Developing, and Validating a Measure of Undergraduate Students’ Conceptions of Artificial Intelligence in Education
Article
  • Jul 2023
The purpose of this research study is to design, develop, and validate an instrument for measuring undergraduate students’ conceptions of artificial intelligence in education. Following systematic procedures, our team created a conceptual framework through an extant literature review and used it to create an initial item pool of 48-items across eight educational application domains of AI (Ahmad et al. (2020). We conducted think aloud protocols with a small group (N = 4) of undergraduate students and an expert review with researchers experienced in AI, psychometrics, and education (N = 3) to ensure the intended meaning of the language was aligned with our framework. Next, we recruited undergraduate student participants (N = 445) from two public universities and one state college in the southeastern United States to complete the final survey. Data were randomly split into two datasets, one for fitting a Confirmatory Factor Analysis model, and the other one for testing the model. The final model retained 37-items across the eight factors of conceptions of AI in education: (AI1) intelligent tutoring system, (AI2) students’ grading and evaluation, (AI3) students’ retention and dropout, (AI4) sentiment analysis in education, (AI5) recommendation systems, (AI6) classroom monitoring and visual analysis, (AI7) personalized learning, and (AI8) student’s performance prediction. We present the results, provide interpretations and explanations, outline the limitations and delimitations of the study, and highlight recommendations for future research and practice.
View
... The set of around 2,000 labeled images was used to train the ML model which will classify the given image into one of the categories. This system uses boosted decision tree regression which is a supervised learning method to predict the engagement level of un-labeled images (Soloviev, 2018). ...
View
A Survey of Machine Learning and Deep Learning Methods for Estimating Automatic Engagement and Attention in Offline Classroom Environment
Chapter
  • Feb 2024
This paper aims to present a literature review in the area of machine learning (ML) and deep learning (DL) methods used in automatic estimation of attention and engagement in offline classroom. A literature search was carried out with relevant set of keywords in Scopus, IEEE, and Proquest databases with a time line of 2010–2023. A total of twenty-four articles were selected to address the aim of the study. The paper highlights the importance of attention and engagement estimation by highlighting the challenges faced in an offline classroom environment. Further, the ML and DL methods are summarized as traditional, contemporary, and other methods. The input dataset along with the performance metrics used to analyze the data in each study is discussed in detail. This study will enable the machine learning experts and researchers to get an insight on the current research in this area and help them to bring robust technological interventions in the field of automatic estimation of attention and engagement for offline classroom environment.
View
View
Reimagining Education with Artificial Intelligence
Article
Full-text available
  • Sep 2021
Artificial intelligence (AI) technologies have been implemented successfully in many industries, from virtual hospital assistants to algorithm-based warehouse processing. And now that Covid-19 has forced students and teachers to transition to online or hybrid learning, these technologies could offer new and exciting possibilities for education as well. By incorporating AI and machine learning tools into online classrooms, educators can address many of the challenges that have emerged with the recent loss of face-to-face instruction, including the struggle for students to self-regulate their learning, the burden of curriculum planning and administrative work for teachers, and the loss of personalized interaction between students and teachers. This chapter will explore some of the AI technologies that have been used in educational contexts and describe applications of AI in other industries that could be adapted to create more personalized, flexible, inclusive, and engaging learning experiences. If the future of education is going to include online learning as a substantial component, then AI could be the key to maintaining high levels of motivation and engagement from students and teachers alike.
View
The State of Education and Artificial Intelligence After the Pandemic
Article
Full-text available
  • Jun 2021
The neologism of artificial intelligence and machine learning brought a myriad of conveniences and advantages into the education sector. These new and improved technologies have the potential to be the remedies to the predicaments that have emerged from the rapid transition into online learning and the problems that have existed with traditional learning. These predicaments vary from personalized learning experiences, administrative busy works, to increased overall student interaction. This article explores how artificial intelligence became the new normal in education and the recently revealed benefits of artificial intelligence and machine learning in developing automated solutions to these predicaments such as Intelligent Tutoring Systems (ITS), dynamic schedules, automated grading, and increased personalized student interaction through chat-bots. Along with these solutions, some of the economical advancements that these technologies will bring include new research and investments in AI development, data collection and understanding, and increased jobs. These advancements are on a massive scale and will need to be addressed. By accommodating our resources into the advancements in this area we can make our education more versatile, engaging, and inclusive.
View
A phenomenographic investigation of teacher conceptions of student engagement in learning
Article
Full-text available
  • Apr 2008
Internationally, educational stakeholders are concerned with the high levels of student disengagement, evidenced by early school leaving, poor student behaviour, and low levels of academic achievement. The solution, student engagement, is a contested concept, theorised in a variety of different ways within academic literature. To further understand this concept, a phenomenographic study was conducted to map secondary school teachers’ conceptions of student engagement. Six qualitatively different ways of understanding student engagement were found. This research indicates that teachers do not hold similar understandings of what student engagement means. If the concept of engagement is to become educationally fruitful, the term must be more explicitly defined in educational research and government policy documents to promote shared understandings amongst stakeholder groups.
View
Acquisition of Cognitive Skill
Article
Full-text available
  • Jul 1982
Proposes a framework for skill acquisition that includes 2 major stages in the development of a cognitive skill: (1) a declarative stage in which facts about the skill domain are interpreted and (2) a procedural stage in which the domain knowledge is directly embodied in procedures for performing the skill. This general framework has been instantiated in the ACT system in which facts are encoded in a propositional network and procedures are encoded as productions. Knowledge compilation is the process by which the skill transits from the declarative stage to the procedural stage. It consists of the subprocesses of composition, which collapses sequences of productions into single productions, and proceduralization, which embeds factual knowledge into productions. Once proceduralized, further learning processes operate on the skill to make the productions more selective in their range of applications. These processes include generalization, discrimination, and strengthening of productions. Comparisons are made to similar concepts from previous learning theories. How these learning mechanisms apply to produce the power law speedup in processing time with practice is discussed. (62 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
View
A Dynamic Mixture Model to Detect Student Motivation and Proficiency.
Conference Paper
Full-text available
  • Jan 2006
Unmotivated students do not reap the full rewards of using a computer-based intelligent tutoring system. De- tection of improper behavior is thus an important com- ponent of an online student model. To meet this chal- lenge, we present a dynamic mixture model based on Item Response Theory. This model, which simultane- ously estimates a student's proficiency and changing motivation level, was tested with data of high school students using a geometry tutoring system. By account- ing for student motivation, the dynamic mixture model can more accurately estimate proficiency and the proba- bility of a correct response. The model's generality is an added benefit, making it applicable to many intelligent tutoring systems as well as other domains.
View
Multi-method assessment of affective experience and expression during deep learning
Article
Full-text available
  • Jan 2009
Inquiries into the link between affect and learning require robust methodologies to measure the learner's affective states. We describe two studies that utilised either an online or offline methodology to detect the affective states of a learner during a tutorial session with AutoTutor. The online study relied on self-reports for affect judgements, while the offline study considered the judgements by the learner, a peer and two trained judges. The studies also investigated the relationships between facial features, conversational cues and emotional expressions in an attempt to scaffold the development of computer algorithms to automatically detect learners' emotions. Both methodologies showed that boredom, confusion and frustration are the prominent affective states during learning with AutoTutor. For both methodologies, there were also some relationships involving patterns of facial activity and conversational cues that were diagnostic of emotional expressions.
View
Multimodal analysis of the implicit affective channel in computer-mediated textual communication
Conference Paper
  • Oct 2012
Computer-mediated textual communication has become ubiquitous in recent years. Compared to face-to-face interactions, there is decreased bandwidth in affective information, yet studies show that interactions in this medium still produce rich and fulfilling affective outcomes. While overt communication (e.g., emoticons or explicit discussion of emotion) can explain some aspects of affect conveyed through textual dialogue, there may also be an underlying implicit affective channel through which participants perceive additional emotional information. To investigate this phenomenon, computer-mediated tutoring sessions were recorded with Kinect video and depth images and processed with novel tracking techniques for posture and hand-to-face gestures. Analyses demonstrated that tutors implicitly perceived students' focused attention, physical demand, and frustration. Additionally, bodily expressions of posture and gesture correlated with student cognitive-affective states that were perceived by tutors through the implicit affective channel. Finally, posture and gesture complement each other in multimodal predictive models of student cognitive-affective states, explaining greater variance than either modality alone. This approach of empirically studying the implicit affective channel may identify details of human behavior that can inform the design of future textual dialogue systems modeled on naturalistic interaction.
View
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Article
  • Apr 2014
Student engagement is a key concept in contemporary education, where it is valued as a goal in its own right. In this paper we explore approaches for automatic recognition of engagement from students’ facial expressions. We studied whether human observers can reliably judge engagement from the face; analyzed the signals observers use to make these judgments; and automated the process using machine learning. We found that human observers reliably agree when discriminating low versus high degrees of engagement (Cohen’s kappa=0.96kappa = 0.96). When fine discrimination is required (four distinct levels) the reliability decreases, but is still quite high (kappa=0.56kappa = 0.56). Furthermore, we found that engagement labels of 10-second video clips can be reliably predicted from the average labels of their constituent frames (Pearson r=0.85), suggesting that static expressions contain the bulk of the information used by observers. We used machine learning to develop automatic engagement detectors and found that for binary classification (e.g., high engagement versus low engagement), automated engagement detectors perform with comparable accuracy to humans. Finally, we show that both human and automatic engagement judgments correlate with task performance. In our experiment, student post-test performance was predicted with comparable accuracy from engagement labels (r=0.47) as from pre-test scores (r=0.44).
View
Boredom in the Middle School Years: Blaming Schools versus Blaming Students
Article
  • Aug 1991
This article uses time-sampling data to evaluate fifth to ninth graders' experiences of boredom both as a state, related to specific school and nonschool activities, and as a trait, related to individual dispositions that students bring to school. Data come from a study in which 392 youths carried pagers and reported on their activities and emotions at random times over a week when signaled. Results show that, while boredom is reported frequently during schoolwork, it is also prevalent outside school and the same persons report boredom across these contexts. High rates of boredom were correlated with high ability and, when ability is controlled, with oppositional behavior, but not with onset of adolescence. These findings suggest that individual dispositions are an important contributor to boredom. Nonetheless, variations in rates of boredom across school task situations suggest that schools might be structured to reduce, though not eliminate, student boredom.
View
Towards a Reading Coach that Listens: Automated Detection of Oral Reading Errors.
Conference Paper
  • Jan 1993
giving a hint, or explaining unfamiliar vocabulary. What skill is more important to teach than reading? Afterwards, it reviews the passages where the student had Unfortunately, millions of Americans cannot read. difficulty, giving an opportunity to reread them, and Although a large body of educational software exists to help providing appropriate feedback. teach reading, its inability to hear the student limits what it This paper reports on a scaled-down version of such a can do. system. Along the way it points out some current This paper reports a significant step toward using limitations and directions for future improvement. automatic speech recognition to help children learn to read: an implemented system that displays a text, follows as a 2. What Evelyn does (and doesn't) student reads it aloud, and automatically identifies which words he or she missed. We describe how the system Our implemented prototype, named Evelyn, displays a works, and evaluate its performance on a corpus of second page of text on a screen, and listens while someone reads graders' oral reading that we have recorded and transcribed. it. While the user is reading, Evelyn dynamically displays what it thinks is the reader's current position in the text, by
View
Multimodal affect recognition in learning environments
Conference Paper
  • Nov 2005
We propose a multi-sensor affect recognition system and evaluate it on the challenging task of classifying interest (or disinterest) in children trying to solve an educational puz- zle on the computer. The multimodal sensory information from facial expressions and postural shifts of the learner is combined with information about the learner's activity on the computer. We propose a unified approach, based on a mixture of Gaussian Processes, for achieving sensor fusion under the problematic conditions of missing channels and noisy labels. This approach generates separate class labels corresponding to each individual modality. The final classifi- cation is based upon a hidden random variable, which prob- abilistically combines the sensors. The multimodal Gaus- sian Process approach achieves accuracy of over 86%, sig- nificantly outperforming classification using the individual modalities, and several other combination schemes.
View
Engagement tracing: using response times to model student disengagement.
Conference Paper
  • Jan 2005
Time on task is an important predictor for how much students learn. However, students must be focused on their learning for the time invested to be productive. Unfortunately, students do not always try their hardest to solve problems presented by computer tutors. This paper explores student disengagement and proposes an approach, engagement tracing, for detecting whether a student is engaged in answering questions. This model is based on item response theory, and uses as input the difficulty of the question, how long the student took to respond, and whether the response was correct. From these data, the model determines the probability a student was actively engaged in trying to answer the question. The model has a reliability of 0.95, and its estimate of student engagement correlates at 0.25 with student gains on external tests. We demonstrate that simultaneously modeling student proficiency in the domain enables us to better model student engagement. Our model is sensitive enough to detect variations in student engagement within a single tutoring session. The novel aspect of this work is that it requires only data normally collected by a computer tutor, and the affective model is statistically validated against student performance on an external measure.
View