Conference PaperPDF Available

Learning Analytics as AI Extenders in Education: Multimodal Machine Learning versus Multimodal Learning Analytics

Authors:
Mutlu Cukurova at University College London
Mutlu Cukurova
Download full-text PDF
Read full-text
Download citation

Abstract

The nature of the appropriate role for AI is a topic of great interest in many disciplines, and Education is no exception. The initial focus of AI in Education research was on attempts to create systems that are as perceptive as human teachers. Therefore, the majority of early research in the field focused on designing autonomous intelligent tutoring systems. However, more recently, there have also been AI technologies embedded in non-autonomous systems, used by educators to support a broad range of teaching practices. Within the context multimodal AI in Education, there are two paradigms of i) multimodal machine learning that aims to create AI through externalisation and replication of human cognition and ii) multimodal learning analytics that aims to design artefacts that involve AI technology, but also tightly coupled with humans to enable, aid or extend their cognition and enhance their capabilities. For educational research, where the ultimate purpose is to improve education rather than improving the state-of-the-field in AI, AI extenders as exemplified in multimodal learning analytics research should be considered as a field that is related to but distinct from research on fully autonomous AI designs and they require attention from the AI in Education field at least in equal measure to research on fully autonomous systems. To keep the interest of AI in Education research only on the design of systems that can mimic or replace human tutors, would limit the possibilities of AI in Education to supporting tutors to reach a “standard human tutor” level. On the other hand, if the focus is on non-autonomous AI systems that are unlike a human, but human-centred, there are greater opportunities to extend human cognition and enhance our capabilities in both teaching and learning.
Content uploaded by Mutlu Cukurova
Author content
All content in this area was uploaded by Mutlu Cukurova on Jun 13, 2019
Content may be subject to copyright.
Learning Analytics as AI Extenders in Education:
Multimodal Machine Learning versus Multimodal Learning Analytics
Mutlu Cukurova, PhD
University College London, United Kingdom, m.cukurova@ucl.ac.uk
I. INTRODUCTION
The nature of the appropriate role for AI is a topic of great
interest in many disciplines, and Education is no exception.
Perhaps, the initial focus of AI in Education research was on
attempts to create systems that are as perceptive as human
teachers [1]. Therefore, the majority of early research in the field
focused on designing autonomous intelligent tutoring systems.
However, more recently, there have been AI technologies
embedded in non-autonomous systems, used by educators to
support their practice. Non-autonomous systems, in which AI is
used to extend human cognition and enhance teacher and learner
capabilities, differ significantly from approaches that aim to
create fully automated AI systems. These non-autonomous
approaches might even be considered as ‘incomplete’ or
‘inadequate’ in AI research. Here, I argue that, in educational
contexts, AI systems should be considered a continuum with
regards to the extent they are decoupled from humans, rather
than only an approach to provide full-automation. AI can be used
to externalize, internalize or extend human cognition [2], and
these different conceptualizations and implementations of AI
can each have a valuable role to play in the support of learning
and teaching. In this paper, I will briefly describe the distinctions
between AI as a fully autonomous system versus AI as part of a
non-autonomous supportive system, and present examples of
both in educational contexts. I will particularly focus on
multimodal learning analytics where the human cognition is
internalised or extended with AI tools, rather than externalised.
For educational research, where the ultimate purpose is to
improve education rather than improving the state-of-the-field in
AI, AI extenders as exemplified in learning analytics research
should be distinct from research on fully autonomous AI
designs; and they require attention from the field at least in equal
measure.
II. RELEVANT THEORIES
A. AI to Extend Cognition and Enhance Capabilities
The idea of computers as interactive systems to support and
potentially augment human capabilities is not new [3], yet,
recently, it is re-visited in the context of AI systems designed to
augment human intelligence [4]. The general idea of technology
helping humans become more capable through expanding their
cognitive capacities is usually associated with the thesis of
extended cognition [5]. This argues that human cognition can be
partially represented by artefacts that are able to exemplify the
right kinds of computations. It maintains that the artefacts we use
to help us complete cognitive tasks can become integrated into
our biological capacities and can play a functional role in
triggering our cognitive abilities. Therefore, rather than
considering human cognition as limited to intracranial biological
activities, the theory proposes that the wider processes that take
place in the surrounding environments of humans can actually
constitute human cognition [5]. These “cognitive artefacts” [6]
can replace some of the functions of the brain through extended
cognition, but also can move beyond them to potentially enhance
cognitive capabilities. Based on earlier work [6], more recent
researchers defined a cognitive extender as “…an external
physical or virtual element that is coupled to enable, aid,
enhance, or improve cognition, such that all – or more than – its
positive effect is lost when the element is not present” [2, p.3].
Recently, learning analytics research generated valuable
examples of cognitive extenders in various forms (i.e teacher and
learner dashboards) to support and extend teacher and learner
cognition. Positioning AI systems as cognitive extenders is
significantly different than positioning AI systems as fully
autonomous external systems. The first situates “the
intelligence” within the human-artificial coupled systems,
whereas the latter, in the artificial. There is relatively less
research undertaken in non-autonomous AI systems coupled
with humans, however particularly in areas of social sciences
such as Education, focus on these systems might lead to more
productive outcomes.
III. TECHNOLOGICAL ADVANCES AND REAL WORLD
APPLICATIONS
The literature on such theoretical and philosophical
considerations on the different conceptualizations of AI might
appear too abstract for researchers and practitioners of the AI in
Education field. Nevertheless, a significant question to ponder
upon and investigate for the field is what might different
conceptualizations of AI systems as a continuum with regards to
their autonomy mean for AI in Education research and practice?
AI systems can vary considerably in the extent of their autonomy
and symbiosis with humans, and design decisions about the
autonomy of an AI system have significant social and ethical
implications. Colleagues in [2], make a valuable contribution to
this discussion with their arguments suggesting to consider non-
autonomous AI systems distinctively from autonomous, human
decoupled AI systems. However, general recommendations
naturally fall short in terms of their particular implications in
specific research areas. In the next section, I attempt to make
some of the discussed distinctions more concrete within the
context of multimodal AI systems in Education.
A. Multimodal Machine Learning vs Multimodal Learning
Analytics
Modality can be defined as the type of communication channel
used by two agents to convey and acquire information that
defines the data exchange [7]. Some modalities used in AI in
Education research include, but are not limited to, video, audio,
text, click-stream, eye tracking, electroencephalography (EEG),
and electro-dermal data. Multimodality has great potential to
help us understand the complex world around us and it has been
studied for around three decades in the context of social
semiotics. Its potential to interpret complex social phenomena
led AI researchers to try and build models that can process
information from multiple modalities through machine learning
techniques and social signal processing [8]. The literature on
multimodal machine learning is rich with examples of audio-
visual speech recognition; multimedia content indexing and
retrieval, as well as multimodal affect recognition [9]. Learning
from multimodal data provides opportunities to acquire an in-
depth understanding of complex processes and, for AI research
to make progress, it makes sense to focus on multimodal
machine learning models that can process and relate
information from multiple modalities [8]. However, in
multimodal machine learning research, the ultimate aim is to
create fully-autonomous systems that in essence replicate the
human cognition through decoupling humans from the system
by making machines capable of processing multimodal data at
a similar accuracy. In this sense, they can be considered on the
one extreme of the AI continuum towards high autonomy. For
instance, specifically in learning contexts, there are recent
attempts that aim to interpret various modalities of data
including click-stream data, eye-tracking, EEG, video, and
wristband data to automatically predict learning performance in
game contexts [10]. These kinds of studies are great examples
of showcasing the potential of machine learning approaches to
unravel complicated manifolds in complex educational data
including non-linear effects and multivariate interactions. They
are also good examples to present the superiority of multimodal
over unimodal approaches in terms of predicting learning
performance automatically. The full-automation is particularly
useful for the provision of personalised support to learners
through intelligent tutoring systems and adaptive learning
platforms.
On the other hand, in educational contexts, there are also plenty
of multimodal AI technologies that are embedded in non-
autonomous systems used by educators as support tools. Most
of this research is undertaken under the emerging area of
multimodal learning analytics and they provide promising
opportunities for the advancement of educational research and
practice. For instance, researchers recently proposed a
multimodal feedback approach for learner reflections based on
posture, gaze, volume, and performance data [11]. Similarly,
within the context of collaborative learning, data from verbal
and physical interactions of students are used to provide insights
into their collaborative actions around table-top computers [12].
Furthermore, there are examples of similar aimed research
investigating the potential of multimodal data to support
learners’ self-regulation performance [13]; project-based
learning [14], tutor evaluations in debating [15], and classroom
orchestration for teachers [16]. In this line of research, certain
tasks and activities are indeed automated with the help of AI
approaches. Nevertheless, in essence, most multimodal learning
analytics approaches aim to provide explicit and
comprehensible ways of presenting information to learners and
teachers to make them more informed decision makers.
Therefore, these AI technologies are designed to be tightly
coupled with humans to become part of their extended
cognition, and ultimately enhance their capabilities in teaching
and learning. This work requires a greater emphasis on tutor and
learner interfaces of non-autonomous AI systems as they need
to be smoothly internalized by humans. It is important to note
that the interfaces do not necessarily explain how the subsystem
AI works to teachers and learners, but they provide
opportunities for cognitive processes to be instantiated and
internalized by humans to make the most of the multimodal
learning analytics tool. In this sense, they are less autonomous
systems of the AI continuum. Due to the significant difference
in their respective ultimate goals, the design of autonomous AI
systems as exemplified in the multimodal machine learning
research, and the design of multimodal learning analytics
research should be considered as relevant but distinct initiatives
in the AI in Education field.
IV. SUMMARY
In this paper, I argued for the value of considering the autonomy
of AI systems as a continuum in educational research. Within
the context of AI in Education, I presented two paradigms of i)
multimodal machine learning that aims to create AI through
externalisation and replication of human cognition and ii)
multimodal learning analytics that aims to design artefacts that
involve AI technology, but also tightly coupled with humans to
enable, aid or extend their cognition and enhance their
capabilities. The first approach is particularly useful for the
provision of personalised support for learners through
intelligent tutoring systems. However, education systems are
much broader than what intelligent tutoring systems can
provide on their own. Moreover, to keep the interest of AI in
Education research only on the design of systems that can
mimic or replace human tutors, would limit the possibilities of
AI in Education to supporting tutors to reach a “standard human
tutor” level. On the other hand, if the focus is on non-
autonomous AI systems that are not like human, but human-
centred, there are greater opportunities to extend human
cognition and enhance our capabilities in both teaching and
learning.
REFERENCES
[1] Self, J.A. (1998) The defining characteristics of intelligent tutoring
systems research: ITSs care, precisely. International Journal of Artificial
Intelligence in Education, 10, 350-364.
[2] Vold, K., & Hernandez-Orallo, J. (2019). AI Extenders: The Ethical and
Societal Implications of Humans Cognitively Extended by AI.
Proceedings of AAAI / ACM Conference on Artificial Intelligence, Ethics,
and Society https://doi.org/10.17863/CAM.36128
[3] Engelbart, D. C. (1962). Augmenting human intellect: a conceptual
framework. Summary Report AFOSR-3233, Stanford Research Institute,
Menlo Park, CA.
[4] Rouse, W. B., and Spohrer, J. C. (2018). Automating versus augmenting
intelligence. Journal of Enterprise Transformation, 1–21.
[5] Clark, A., and Chalmers, D. (1998). The extended mind. Analysis, 58(1),
7–19.
[6] Hutchins, E. (1999). Cognitive artefacts. The MIT encyclopedia of the
cognitive sciences, 126-127
[7] Kress, G. (2009). Multimodality: A social semiotic approach to
contemporary communication. Routledge.
[8] Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social signal
processing: Survey of an emerging domain. Image and vision
computing, 27(12), 1743-1759.
[9] Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2019). Multimodal machine
learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 41(2), 423-443.
[10] Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E.
(2019). Multimodal data as a means to understand the learning
experience. International Journal of Information Management, 48, 108-
119.
[11] Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., &
Castells, J. (2018). The rap system: Automatic feedback of oral
presentation skills using multimodal analysis and low-cost sensors.
Proceedings of the 8th international conference on learning analytics and
knowledge, 360–364.
[12] Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., &
Yacef, K. (2013). Capturing and analyzing verbal and physical
collaborative learning interactions at an enriched interactive
tabletop. International Journal of Computer-Supported Collaborative
Learning, 8(4), 455-485.
[13] Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., &
Specht, M. (2017). Learning pulse: A machine learning approach for
predicting performance in self-regulated learning using multimodal data.
Proceedings of the seventh international learning analytics & knowledge
conference, 188–197.
[14] Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised
machine learning in multimodal learning analytics for estimating success
in project‐based learning. Journal of Computer Assisted Learning, 34(4),
366-377.
[15] Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial Intelligence and
Multimodal Data in the Service of Human Decision-making: A Case
Study in Debate Tutoring, British Journal of Educational Technology, 1-
22.
[16] Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez-Triana, M. J., &
Dillenbourg, P. (2018). Multimodal teaching analytics: Automated
extraction of orchestration graphs from wearable sensor data. Journal of
Computer Assisted Learning, 34(2), 193–203.
... Specifically, a key focus of Artificial Intelligent in Education (AIED) has been on agents and tutors (Labarthe et al., 2018). As opposed to externalizing and replicating human cognition (as is the case with AI), in MMLA human cognition is often extended by using computational models of multimodal data tools or the models are internalized by humans to help us make better decisions (Cukurova, 2019). The objective of multimodal learning analytics is to design products that may or may not integrate AI technology but are also closely coupled with humans to enhance their cognition, assist them in their decisions, and extend their capabilities (Cukurova, 2019). ...
... As opposed to externalizing and replicating human cognition (as is the case with AI), in MMLA human cognition is often extended by using computational models of multimodal data tools or the models are internalized by humans to help us make better decisions (Cukurova, 2019). The objective of multimodal learning analytics is to design products that may or may not integrate AI technology but are also closely coupled with humans to enhance their cognition, assist them in their decisions, and extend their capabilities (Cukurova, 2019). MMLA is therefore more concerned about the data itself, identifying ways to process learning data from different modalities of sources to find helpful information for giving feedback to learners (Ochoa et al., 2013). ...
... Apart from providing more accurate predictions of learning in single observations, MMLA uses advances in machine learning and sensor technology to monitor factors that are considered highly relevant to learning but are often overlooked due to difficulties in their dynamic measurement and interpretation (Cukurova, Giannakos, & Martinez-Maldonado, 2020). MMLA research mainly focuses on the challenges of multimodal data collection, integration, interpretation, and visualization from digital and physical environments to provide students and teachers with appropriate feedback to improve the learning and teaching process, regardless of any artificial intelligence implied automation of any of these processes (Cukurova, 2019). ...
Navigating the Ethical Landscape of Multimodal Learning Analytics: A Guiding Framework
Preprint
Full-text available
  • Apr 2024
Artificial intelligence (AI) and multimodal data (MMD) are gaining popularity in education for their ability to monitor and support complex teaching and learning processes. This line of research and practice was recently named Multimodal Learning Analytics (MMLA). However, MMLA raise serious ethical concerns given the pervasive nature of MMD and the opaque AI techniques that process them. This study aims to explore ethical concerns related to MMLA use in higher education and proposes a framework for raising awareness of these concerns, which could lead to more ethical MMLA research and practice. This chapter presents the findings of 60 interviews with educational stakeholders (39 higher education students, 12 researchers, 8 educators, and 1 representative of an MMLA company). A thematic coding of verbatim transcriptions revealed nine distinct themes. The themes and associated probing questions for MMLA stakeholders are presented as a draft of the first ethical MMLA framework. The chapter is concluded with a discussion of the emerging themes and suggestions for MMLA research and practice.
View
... AI in education can also be conceptualised to externalise, to be internalised or extend human cognition (Cukurova, 2019). As the first conceptualisation, in the externalisation of cognition, certain human tasks are defined, modelled and replaced by AI as a tool. ...
... At last, AI models can be used to extend human cognition as part of tightly coupled human and AI hybrid intelligence systems. It is important to note that in such systems, changes in both agents are expected to be observed through their interactions and the whole emergent intelligence is synergistic, that is, it is expected to be more than the sum of each agent's intelligence, both human and artificial (Cukurova, 2019). ...
The interplay of learning, analytics and artificial intelligence in education: A vision for hybrid intelligence
Article
Full-text available
This paper presents a multidimensional view of AI's role in education, emphasising the intricate interplay among AI, analytics and human learning processes. Here, I challenge the prevalent narrow conceptualisation of AI as tools in Education, exemplified in generative AI tools, and argue for the importance of alternative conceptualisations of AI for achieving human–AI hybrid intelligence. I highlight the differences between human intelligence and artificial information processing, the importance of hybrid human–AI systems to extend human cognition and posit that AI can also serve as an instrument for understanding human learning. Early learning sciences and AI in Education Research (AIED), which saw AI as an analogy for human intelligence, have diverged from this perspective, prompting a need to rekindle this connection. The paper presents three unique conceptualisations of AI: the externalisation of human cognition, the internalisation of AI models to influence human mental models and the extension of human cognition via tightly coupled human–AI hybrid intelligence systems. Examples from current research and practice are examined as instances of the three conceptualisations in education, highlighting the potential value and limitations of each conceptualisation for human competence development, as well as the perils of overemphasis on approaches that replace human learning opportunities with AI tools. The paper concludes with advocacy for a broader approach to AIED that goes beyond considerations on the design and development of AI and includes educating people about AI and innovating educational systems to remain relevant in an AI ubiquitous world.
View
... Why is that? AI in Education can also be conceptualised to externalize, to be internalised, or extend human cognition (Cukurova, 2019). As the first conceptualisation, in the externalization of cognition, certain human tasks are defined, modelled and replaced by AI as a tool. ...
... At last, AI models can be used to extend human cognition as part of tightly coupled human and AI hybrid intelligence systems. It is important to note that in such systems, changes in both agents are expected to be observed through their interactions and the whole emergent intelligence is synergistic, that is, it is expected to be more than the sum of each agent's intelligence, both human and artificial (Cukurova, 2019). ...
The Interplay of Learning, Analytics, and Artificial Intelligence in Education: A Vision for Hybrid Intelligence
Article
Full-text available
  • Jul 2024
  • BRIT J EDUC TECHNOL
This paper presents a multi-dimensional view of AI's role in learning and education, emphasizing the intricate interplay between AI, analytics, and the learning processes. Here, I challenge the prevalent narrow conceptualisation of AI as tools, as exemplified in generative AI tools and argue for the importance of alternative conceptualisations of AI for achieving human-AI hybrid intelligence. I highlight the differences between human intelligence and artificial information processing, the importance of hybrid human-AI systems to extend human cognition, and posit that AI can also serve as an instrument for understanding human learning. Early learning sciences and AI in Education research (AIED), which saw AI as an analogy for human intelligence, have diverged from this perspective, prompting a need to rekindle this connection. The paper presents three unique conceptualisations of AI: the externalization of human cognition, the internalization of AI models to influence human mental models, and the extension of human cognition via tightly coupled human-AI hybrid intelligence systems. Examples from current research and practice are examined as instances of the three conceptualisations in education, highlighting the potential value and limitations of each conceptualisation for education, as well as the perils of overemphasis on externalising human cognition. The paper concludes with advocacy for a broader approach to AIED that goes beyond considerations on the design and development of AI, but also includes educating people about AI and innovating educational systems to remain relevant in an AI-ubiquitous world.
View
... Mu et al. [52] highlighted that integrating multimodal data via "many-to-one" or "many-to-many" methods is more effective than single-mode data for predicting learning outcomes. For example, Giannakos et al. [23] and Cukurova [15] demonstrated superior skill acquisition and tutor performance predictions using multimodal data fusion (e.g., eye-tracking and facial Manuscript submitted to ACM video) compared to monomodal approaches. Chango et al. [9] categorised fusion techniques into feature-level (early fusion), decision-level (late fusion), and hybrid fusion. ...
From Complexity to Parsimony: Integrating Latent Class Analysis to Uncover Multimodal Learning Patterns in Collaborative Learning
Preprint
Full-text available
  • Nov 2024
Multimodal Learning Analytics (MMLA) leverages advanced sensing technologies and artificial intelligence to capture complex learning processes, but integrating diverse data sources into cohesive insights remains challenging. This study introduces a novel methodology for integrating latent class analysis (LCA) within MMLA to map monomodal behavioural indicators into parsimonious multimodal ones. Using a high-fidelity healthcare simulation context, we collected positional, audio, and physiological data, deriving 17 monomodal indicators. LCA identified four distinct latent classes: Collaborative Communication, Embodied Collaboration, Distant Interaction, and Solitary Engagement, each capturing unique monomodal patterns. Epistemic network analysis compared these multimodal indicators with the original monomodal indicators and found that the multimodal approach was more parsimonious while offering higher explanatory power regarding students' task and collaboration performances. The findings highlight the potential of LCA in simplifying the analysis of complex multimodal data while capturing nuanced, cross-modality behaviours, offering actionable insights for educators and enhancing the design of collaborative learning interventions. This study proposes a pathway for advancing MMLA, making it more parsimonious and manageable, and aligning with the principles of learner-centred education.
View
... In the educational sector, for instance, AI transcends the conventional bounds of automation to offer tailored learning experiences, automated assessments, and dedicated support mechanisms that cater to the unique needs of learners. This nuanced application of AI in education has sparked a rich dialogue about its wider implications and ethical dimensions, highlighting the importance of transparent and explainable AI systems that can inform and drive forward the innovations in AI-enhanced educational practices (Tuomi, 2018;Zhang & Aslan, 2021;Baltrušaitis, Ahuja, & Morency, 2018;DiMitri et al., 2017;Giannakos et al., 2019;Cukurova, 2019). ...
The Potential for Artificial Intelligence Assistance in Funding Research
Thesis
Full-text available
  • Jun 2024
This dissertation investigates the potential of Artificial Intelligence (AI) in transforming decision-making processes within funding agencies, which include governmental, quasi-governmental, and private organizations that finance research and innovation. These agencies are crucial in directing scientific inquiry and innovation by funding projects that meet their strategic and societal goals. The study seeks to determine how AI can improve the efficiency, transparency, and objectivity of funding allocations, posing the question: "How can AI be effectively integrated into the decision-making frameworks of funding agencies to optimize outcomes?"
View
... Dies verhindert eine klare Abgrenzung. Historisch gesehen lassen sich vier inhaltlich grundsätzlich verschiedene Bedeutungen in der Verwendung erkennen: nämlich die Bezeichnung von Systemen die menschlich denken, die menschlich handeln, die rational denken und die die rational handeln (Norvig & Russel, 2019 Jivet et al., 2020, Cukurova, 2019. ...
View
... Alternative narratives can be found under the headings, 'feminist technoscience', 'Crip technoscience' and 'queer STS'. 7 Here Merleau-Ponty raises a theme since taken up and elaborated in a way which has been influential on technology design under the headings of 'extended cognition' (Clark & Chalmers, 1998;Cukurova, 2019) and 'embodied cognition' (Foglia & Wilson, 2013). 8 The Digitalised Dialogues Across the Curriculum (DiDiAC) project was funded by the Research Council of norway (FInnUT/Project no: 254761). ...
View
Emerging trends and future research opportunities in artificial intelligence, machine learning, and deep learning
Chapter
Full-text available
  • Oct 2024
The Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL), each built on a higher level of the proved technology driving innovation and efficiency. There are a few other futuristic trends clearly on the horizon too, such as the incorporation of AI with Internet of Things (IoT) devices to create environments that are smarter and more responsive. Explainable Artificial Intelligence (XAI) is also becoming more important, as is the need for transparency and accountability in AI decision-making. Federated learning has also emerged as an interesting approach towards privacy-preserving model training in ML by training de-centralized models across multiple devices without sharing raw data. Transformer model such as GPT-4 and BERT are transformer models that have revolutionized the field of natural language processing (NLP) in DL, which are capable of more nuanced understanding and generation of human language. Their usage has increased dramatically, and they are used in everything from healthcare diagnostics to automated content creation. Also, the implication of blockchain-enabled AI to develop hack-proof AI applications, largely in finance and supply chain management is increasingly becoming popular. More research arises in the future, that will be around building hybrid AI models that contains both symbolic reasoning and neural networks, where we expect future research, will be focused on building much more stronger and flexible AI systems. Certainly, further study of the ethical issues around AI deployment - especially what is learned about bias and fairness - will remain an important area of investigation.
View
View
Navigating the ethical landscape of multimodal learning analytics: a guiding framework
Chapter
Full-text available
  • Jan 2024
Artificial intelligence (AI) and multimodal data (MMD) are gaining popularity in education for their ability to monitor and support complex teaching and learning processes. This line of research and practice was recently named Multimodal Learning Analytics (MMLA). However, MMLA raise serious ethical concerns given the pervasive nature of MMD and the opaque AI techniques that process them. This study aims to explore ethical concerns related to MMLA use in higher education and proposes a framework for raising awareness of these concerns, which could lead to more ethical MMLA research and practice. This chapter presents the findings of 60 interviews with educational stakeholders (39 higher education students, 12 researchers, 8 educators, and 1 representative of an MMLA company). A thematic coding of verbatim transcriptions revealed nine distinct themes. The themes and associated probing questions for MMLA stakeholders are presented as a draft of the first ethical MMLA framework. The chapter is concluded with a discussion of the emerging themes and suggestions for MMLA research and practice.
View
Multimodal data as a means to understand the learning experience
Article
Full-text available
  • Dec 2019
  • INT J INFORM MANAGE
Most work in the design of learning technology uses click-streams as their primary data source for modelling & predicting learning behaviour. In this paper we set out to quantify what, if any, advantages do physiological sensing techniques provide for the design of learning technologies. We conducted a lab study with 251 game sessions and 17 users focusing on skill development (i.e., user's ability to master complex tasks). We collected click-stream data, as well as eye-tracking, electroencephalography (EEG), video, and wristband data during the experiment. Our analysis shows that traditional click-stream models achieve 39% error rate in predicting learning performance (and 18% when we perform feature selection), while for fused multimodal the error drops up to 6%. Our work highlights the limitations of standalone click-stream models, and quantifies the expected benefits of using a variety of multimodal data coming from physiological sensing. Our findings help shape the future of learning technology research by pointing out the substantial benefits of physiological sensing.
View
Artificial intelligence and multimodal data in the service of human decision‐making: A case study in debate tutoring
Article
Full-text available
The question: “What is an appropriate role for AI?” is the subject of much discussion and interest. Arguments about whether AI should be a human replacing technology or a human assisting technology frequently take centre stage. Education is no exception when it comes to questions about the role that AI should play, and as with many other professional areas, the exact role of AI in education is not easy to predict. Here, we argue that one potential role for AI in education is to provide opportunities for human intelligence augmentation, with AI supporting us in decision‐making processes, rather than replacing us through automation. To provide empirical evidence to support our argument, we present a case study in the context of debate tutoring, in which we use prediction and classification models to increase the transparency of the intuitive decision‐making processes of expert tutors for advanced reflections and feedback. Furthermore, we compare the accuracy of unimodal and multimodal classification models of expert human tutors' decisions about the social and emotional aspects of tutoring while evaluating trainees. Our results show that multimodal data leads to more accurate classification models in the context we studied.
View
Supervised machine learning in multimodal learning analytics for estimating success in project-based learning
Article
Full-text available
  • May 2018
Multimodal learning analytics provides researchers with new tools and techniques to capture different types of data from complex learning activities in dynamic learning environments. This paper investigates the use of diverse sensors, including computer vision, user‐generated content, and data from the learning objects (physical computing components), to record high‐fidelity synchronised multimodal recordings of small groups of learners interacting. We processed and extracted different aspects of the students' interactions to answer the following question: Which features of student group work are good predictors of team success in open‐ended tasks with physical computing? To answer this question, we have explored different supervised machine learning approaches (traditional and deep learning techniques) to analyse the data coming from multiple sources. The results illustrate that state‐of‐the‐art computational techniques can be used to generate insights into the "black box" of learning in students' project‐based activities. The features identified from the analysis show that distance between learners' hands and faces is a strong predictor of students' artefact quality, which can indicate the value of student collaboration. Our research shows that new and promising approaches such as neural networks, and more traditional regression approaches can both be used to classify multimodal learning analytics data, and both have advantages and disadvantages depending on the research questions and contexts being investigated. The work presented here is a significant contribution towards developing techniques to automatically identify the key aspects of students success in project‐based learning environments and to ultimately help teachers provide appropriate and timely support to students in these fundamental aspects.
View
Multimodal Teaching Analytics: Automated Extraction of Orchestration Graphs from Wearable Sensor Data
Article
Full-text available
  • Jan 2018
The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time) on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye-tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time-independent and time-aware models, achieving median F1 scores of about 0.7–0.8 on leave-one-session-out k-fold cross-validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.
View
Multimodal Machine Learning: A Survey and Taxonomy
Article
Full-text available
  • May 2017
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
View
Learning pulse: a machine learning approach for predicting performance in self-regulated learning using multimodal data
Conference Paper
Full-text available
  • Mar 2017
Learning Pulse explores whether using a machine learning approach on multimodal data such as heart rate, step count, weather condition and learning activity can be used to predict learning performance in self-regulated learning settings. An experiment was carried out lasting eight weeks involving PhD students as participants, each of them wearing a Fitbit HR wristband and having their application on their computer recorded during their learning and working activities throughout the day. A software infrastructure for collecting multimodal learning experiences was implemented. As part of this infrastructure a Data Processing Application was developed to pre-process, analyse and generate predictions to provide feedback to the users about their learning performance. Data from different sources were stored using the xAPI standard into a cloud-based Learning Record Store. The participants of the experiment were asked to rate their learning experience through an Activity Rating Tool indicating their perceived level of productivity, stress, challenge and abilities. These self-reported performance indicators were used as markers to train a Linear Mixed Effect Model to generate learner-specific predictions of the learning performance. We discuss the advantages and the limitations of the used approach, highlighting further development points.
View
The RAP system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors
Conference Paper
  • Mar 2018
Developing communication skills in higher education students could be a challenge to professors due to the time needed to provide formative feedback. This work presents RAP, a scalable system to provide automatic feedback to entry-level students to develop basic oral presentation skills. The system improves the state-of-the-art by analyzing posture, gaze, volume, filled pauses and the slides of the presenters through data captured by very low-cost sensors. The system also provides an off-line feedback report with multimodal recordings of their performance. An initial evaluation of the system indicates that the system's feedback highly agrees with human feedback and that students considered that feedback useful to develop their oral presentation skills.
View
Automating versus augmenting intelligence
Article
  • Feb 2018
This article addresses the prospects for automating intelligence versus augmenting human intelligence. The evolution of artificial intelligence (AI) is summarized, including contemporary AI and the new capabilities now possible. Functional requirements to augment human intelligence are outlined. An overall architecture is presented for providing this functionality, including how it will make deep learning explainable to decision makers. Three case studies are addressed, including driverless cars, medical diagnosis, and insurance underwriting. Paths to transformation in these domains are discussed. Prospects for innovation are considered in terms of what we can now do, what we surely will be able to do soon, and what we are unlikely to ever be able to do.
View
The Extended Mind
Article
  • Jan 2012
Where are the borders of mind and where does the rest of the world begin? There are two standard answers possible: Some philosophers argue that these borders are defined by our scull and skin. Everything outside the body is also outside the mind. The others argue that the meanings of our words "simply are not in our heads" and insist that this meaning externalism applies also to the mind. The authors are suggesting a third position, i.e. quite another form of externalism. Their so called active externalism implies an active involvement of the background in controlling the cognitive processes.
View
Multimodality: A Social Semiotic Approach to Contemporary Communication
Article
  • Dec 2009
The 21st century is awash with ever more mixed and remixed images, writing, layout, sound, gesture, speech, and 3D objects. Multimodality looks beyond language and examines these multiple modes of communication and meaning making. Multimodality: A Social Semiotic Approach to Contemporary Communication represents a long-awaited and much anticipated addition to the study of multimodality from the scholar who pioneered and continues to play a decisive role in shaping the field. Written in an accessible manner and illustrated with a wealth of photos and illustrations to clearly demonstrate the points made, Multimodality: A Social Semiotic Approach to Contemporary Communication deliberately sets out to locate communication in the everyday, covering topics and issues not usually discussed in books of this kind, from traffic signs to mobile phones. In this book, Gunther Kress presents a contemporary, distinctive and widely applicable approach to communication. He provides the framework necessary for understanding the attempt to bring all modes of meaning-making together under one unified theoretical roof. This exploration of an increasingly vital area of language and communication studies will be of interest to advanced undergraduate and postgraduate students in the fields of English language and applied linguistics, media and communication studies and education.
View