Conference PaperPDF Available

What is Learning Analytics about? A Survey of Different Methods Used in 2013-2015

Authors:

Abstract and Figures

The area of Learning Analytics has developed enormously since the first International Conference on Learning Analytics and Knowledge (LAK) in 2011. It is a field that combines different disciplines such as computer science, statistics, psychology and pedagogy to achieve its intended objectives. The main goals illustrate in creating convenient interventions on learning as well as its environment and the final optimization about learning domain's stakeholders (Khalil & Ebner, 2015b). Because the field matures and is now adapted in diverse educational settings, we believe there is a pressing need to list its own research methods and specify its objectives and dilemmas. This paper surveys publications from Learning Analytics and Knowledge conference from 2013 to 2015 and lists the significant research areas in this sphere. We consider the method profile and classify them into seven different categories with a brief description on each. Furthermore, we show the most cited method categories using Google scholar. Finally, the authors raise the challenges and constraints that affect its ethical approach through the meta-analysis study. It is believed that this paper will help researchers to identify the common methods used in Learning Analytics, and it will assist by establishing a future forecast towards new research work taking into account the privacy and ethical issues of this strongly emerged field.
Content may be subject to copyright.
ISSN 2414-6099
294
What is Learning Analytics about? A Survey of Different Methods
Used in 2013-2015
Mohammad Khalil
Graz University of Technology,Graz, Austria
Martin Ebner
Graz University of Technology,Graz, Austria
Abstract
The area of Learning Analytics has developed enormously since the first International Conference
on Learning Analytics and Knowledge (LAK) in 2011. It is a field that combines different
disciplines such as computer science, statistics, psychology and pedagogy to achieve its intended
objectives. The main goals illustrate in creating convenient interventions on learning as well as its
environment and the final optimization about learning domain’s stakeholders (Khalil & Ebner,
2015b). Because the field matures and is now adapted in diverse educational settings, we believe
there is a pressing need to list its own research methods and specify its objectives and dilemmas.
This paper surveys publications from Learning Analytics and Knowledge conference from 2013 to
2015 and lists the significant research areas in this sphere. We consider the method profile and
classify them into seven different categories with a brief description on each. Furthermore, we show
the most cited method categories using Google scholar. Finally, the authors raise the challenges
and constraints that affect its ethical approach through the meta-analysis study.
It is believed that this paper will help researchers to identify the common methods used in Learning
Analytics, and it will assist by establishing a future forecast towards new research work taking into
account the privacy and ethical issues of this strongly emerged field.
Keywords: Learning Analytics, survey, publications, literacy, techniques
Introduction
Since the first Learning Analytics and Knowledge (LAK) conference in 2011 as well as the Horizon
Report in 2013 (Johnson et al., 2013), learning analytics is considered to be an emerging field that
would be applied in the different educational settings. This field provides tools and technologies
that offer the potentials to do proper interventions and improve education in general. The Society
for Learning Analytics and Research (SoLAR) defined it as “the measurement, collection, analysis
and reporting of data about learners and their contexts, for purposes of understanding and
optimizing learning and the environments in which it occurs”. Several studies exchanged views
about learning analytics goals. For instance, Khalil and Ebner (2015) introduced learning analytics
lifecycle and listed the main surveyed objectives of the past four years of the LAK conferences.
ISSN 2414-6099
295
They listed interventions, predictions, reflection, awareness, personalization, recommendation and
benchmarking as the main goals. These goals conformed to Siemen’s defined techniques back in
2012 (Siemens, 2012). In addition to that, different frameworks have been introduced to define both
key objectives and dilemmas of this field. In his paper “The Learning Analytics Cycle: Closing the
loop effectively”, Clow argued that a successful learning analytics should be winding up into
feeding back the product to learners in order to make effective intervention(s) (Clow, 2012). Whiles
Ferguson indexed remarkable challenges of ethics, distinct perspectives from stakeholders’ field of
vision and the methods to use in order to make these goals achievable (Ferguson, 2012).
For the time being, there is a large variety of educational environments such as MOOC-
platforms, LMS, virtual environments, etc. These educational information systems hold “Big Data”
of learners that create huge data repositories. According to learning analytics definition, the data
need to be analysed by typical methodologies in order to reflect benefits on learning and teaching.
The beginning of lively discussions on the differences between learning analytics and educational
data mining were mainly residing to the opposing opinions of using tools and methodologies in both
fields (Baker et al., 2012). Nevertheless, educational data mining and learning analytics are
enriched by the methods of data mining and analytics in general (Baker & Siemens, 2013). In its
first stages, researchers of learning analytics frameworks and structure were discussing methods
such as visualizations, data mining techniques (Elias, 2011), social network analysis (Ferguson,
2012), and sentiment analysis (Siemens, 2012), in addition to statistics which was also mentioned as
a required tool to build learning prediction models (Campbell, DeBlois & Oblinger, 2007).
SoLAR brought to success the annual organization of LAK conferences since 2011.
Accordingly, several categories of methods to analyse educational datasets were used. Most of these
methods tend to process data quantitatively and qualitatively to discover interesting hidden patterns.
Baker and Siemens (2013) mentioned that educational data is what drives new methods to be used
in learning analytics. They said: The specific characteristics of educational data have resulted in
different methods playing a prominent role in EDM/LA than in data mining in general, or have
resulted in adaptations to existing psychometrics methods”. In this paper, we survey publications
from LAK conference from 2013 to 2015. The purpose is to list the most common methods used in
the field of learning analytics in the last three years. We believe this paper provides different
benefits because learning analytics becomes an important field by itself and is now completely
matured into being adapted in different educational institutions and applications. The main
advantages are:
1. It helps learning analytics researchers to identify common methods in use in order to reach
intended goals.
2. It determines methods that are highly cited, e.g. by Google scholar (http://scholar.google.com),
and establish a future forecast towards new research work.
3. Finally, it assists to compare the beginning view about learning analytics methods and the on-
going current version.
In addition, the paper aims to guide future researchers into further advances in this field and
meets the “Smart Learning Excellence” theme of Innovation Arabia 9 conference which is “The
next wave of innovations in Smart-Learning” that mainly considers Big Data and Learning
Analytics as a new wave in educational technology.
ISSN 2414-6099
296
We have organized this paper into the following sections. First, we list the methodology
employed to extract methods used in learning analytics publications. We then show statistical data
and describe methods in detail with remarks about their types. Finally, we discuss and summarize
the main conclusions and list the constraints and the ethical issues of learning analytics.
Research Design
As mentioned before, the conference of learning analytics and knowledge is considered to be the
first and the largest repository of learning analytics publications. We mainly focused on it, and
surveyed 91 papers from LAK 2013 (Suthers et al., 2013), LAK 2014 (Pistilli, Willis & Koch,
2014), and LAK 2015 (Baron, Lynch & Maziarz, 2015), with an additional supplementary literature
from other sources. We excluded papers with topics philosophy, frameworks and conceptual studies
of learning analytics for the reason that they address structures and do not accommodate a
mechanism for revealing patterns. We also faced papers with unclear methods, and these were
excluded too. At the end, 78 publications were set for examination. This study was influenced by
the work of Romero and Ventura (Romero & Ventura, 2010), and Dawson et al. (Dawson et al.,
2014). The classification of methods was based on reading the abstract, keywords, general terms,
methodology section and the conclusion of each paper. In some publications, we paid more analysis
into examining literature and the reference list. Furthermore, we collected the total number of
citations for each analysed paper from Google scholar and observed the trending topics.
Learning Analytics Methods
Learning analytics is a combination of different disciplines like computer science, statistics,
psychology, and education. As a result, we realized different analysis methods that do not only tend
to be too technical but rather pedagogical. Before classifying the analysis methods, we have been
gravitated towards the beginning topics of the emergence of learning analytics, which briefly
described methods and tools for collecting data and analyzing them (Ferguson, 2012; Siemens,
2012). However, the survey reveals more methods being used to examine learners’ data. Our main
methods categories, which will be explained in detail in section 3.1, are: (a) data mining techniques;
(b) statistics and mathematics; (c) text mining, semantics and linguistic analysis; (d) visualization;
(e) social network analysis; (f) qualitative analysis; and (g) gamification. Figure 1 shows grouping
of the methods used in learning analytics for LAK publications with the number of papers in each
category. It should be noted that some publications might be referenced in a different category.
Moreover, a paper could be referenced in multiple methods category.
The bar plot in the figure shows that researchers of 31 publications used data mining techniques
and 26 research studies used statistics and mathematics to analyse their data. This makes both of
these two methods as the top most employed techniques of analysis. We also see that “Text Mining,
Semantics and Linguistic” analyses as well as visualizations are being used in 13 LAK publications
equally. However, social network analysis and qualitative analysis as well as gamification were the
least used techniques.
ISSN 2414-6099
297
Categories
This section lists the methods categories in more detail and states relevant publications under
each method.
Data Mining Techniques: Data mining tends to make sense out of data. The definition of learning
analytics cited the similar idea, namely understanding the data, but in this case, about learners. The
survey shows that data mining techniques are the most used method for analyzing and interpreting
the learners’ log data. The decision tree algorithm was used to predict the performance drop and the
final outcome of students in a Virtual Learning Environment (VLE) (Wolff et al., 2013). Other
researchers used several classification techniques such as step regression, Naive Bayes and REP-
Trees to study students’ behavior and detect learners who game the system (Pardos et al., 2013).
While clustering was used to propose an approach for the purposes of enhancing educational
process mining based on the collected data from logs and detecting students at risk (Bogarín et al.,
2014). Discovering relations between two factors was observed by using multiple linear regression
analysis to forecast the relation between studying time and learning performance (Jo, Kim & Yoon,
2014). Moreover, data mining is used for assessment such as the work at the University of
Missouri-Columbia, which proposed an automated tool to enable teachers assess students in online
environments (Xing, Wadholm & Goggins, 2014). It was remarked that regression analysis was the
common mechanism among data mining techniques.
Statistics and Mathematics: Statistics is the science of measuring, controlling, communicating and
understanding the data (Davidian & Louis, 2012). Publications show that researchers have been
using descriptive statistics and mathematics, such as the mean, median and standard deviation to
signify their results. In addition, inferential statistics was used side by side with data mining in
Figure 1. Number of the examined LAK papers grouped by methods. Some papers share
more than single category
ISSN 2414-6099
298
some of the publications. Markov chain was used to study school students behavior in solving
multiplication (Taraghi et al., 2014). Different statistical techniques were operated to build a
grading system (Vogelsang & Ruppertz, 2015). Additionally, statistical discourse analysis with
Markov chain was employed to study online discussions and summarize demographics (Chiu &
Fujita, 2014), as well as examining student problem solving behavior and adapting it into tutoring
systems (Eagle et al., 2015).
Text mining, Semantics & Linguistic Analysis: Publications which refer to ontologies, mining
texts, discourse analysis, Natural Language Processing, or study of languages are set to be in this
category. Some studies refer to text analysis for assessment purposes of short answer questions
(Leeman-Munk, Wiebe & Lester, 2014), or to enhance collaborative writing between students
(Southavilay et al., 2013). Contextualizing user interactions based on ontologies to illustrate a
learning analytics approach (Renzel & Klamma, 2013). Linguistic analysis was clearly used in
parsing posts from students for prediction purposes (Joksimović et al., 2015). Finally, online
discussion forums were analysed to pioneer an automatic dialogue detection system in order to
develop a self-training approach (Ferguson et al., 2013).
Visualization: When the information is visually presented to the field experts, efficient human
capabilities rise to perceive and process the data (Kapler & Wright, 2015). Visual representations
take the advantages into expanding human decisions within a large amount of information at once
(Romero & Ventura, 2010). There are several studies that cited visualization as a method to analyse
the data and deliver information to end users, such as: building a student explorer screen to prepare
meetings and identifying at-risk students by the teachers (Aguilar, Lonn & Teasley, 2014). Studying
MOOC’s attrition rate and learners’ activities (Santos et al., 2014); Building an awareness tool for
teachers and learners (Martinez-Maldonado et al., 2015), and a dashboard for self-reflection goals
(Santos et al., 2013). Information can be interpreted into heat maps, scatterplots, diagrams, and
flowcharts which were observed in most of the statistical, mathematical and data mining based
publications.
Social Network Analysis: Abbreviated as SNA. It focuses on relationships between entities. In
learning analytics, SNA can be used to promote collaborative learning and investigate connections
between learners, teachers and resources (Ferguson, 2012). Moreover, it can be employed in
learning environments to examine relationships of strong or weak ties (Khalil & Ebner, 2015). This
category includes network analysis in general and Social Learning Analytics (SLA). The survey
observed researchers who: built a collaborative learning environment by visualizing relationships
between students about the same topic (Schreurs et al., 2013). A two-mode network was used to
study students’ patterns and to classify them into particular groups (Hecking, Ziebarth & Hoppe,
2014). It was also used with a grading system in a PLE to examine the centrality of students and
grades (Koulocheri & Xenos, 2013). Again, not so far from this survey study, a network analysis
was done to analyse citations of LAK conference papers (Dawson et al., 2014). The authors studied
the degree centrality and pointed out the emergence and isolated disciplines in learning analytics.
SNA was used to analyse data of connectivist MOOCs by examining interactions of learners from
social media websites (Joksimović et al., 2015).
ISSN 2414-6099
299
Qualitative Analysis: This category is related to the decisions based on explained descriptions of
the analysts. For instance: i) a qualitative evaluation with data mining techniques was made to
understand the nature of discussion forums of MOOCs (Ezen-Can et al., 2015); ii) Usage of
qualitative interviews, which are answered by words to build a learning analytics module of
understanding fractions for school children (Mendiburo, Sulcer & Hasselbring, 2014); iii)
Qualitative meta-analysis to investigate teachers’ needs in technology enhanced learning covered by
the umbrella of learning analytics (Dyckhoff et al., 2013).
Gamification: It is the use of game mechanics and tools to make learning and instruction attractive
and fun (Kapp, 2012). This method is considered as a technique on its own because of its relevant
appearance in educational workshops and the requests to make learning entertaining. Some
examples are using rewards points and progress bar to enhance the retention rate and building a
gamified grading system (Holman, Aguilar & Fishman, 2013), or presenting a competency map
with progress bars, pie charts, labels and hints to improve students’ performance (Grann &
Bushway, 2014). A significant study on monitoring students in a 3D immersive environment was
also advised as another type of gamification techniques (Camilleri et al., 2013).
Prominent Methods and Discussion
In this section, we consider learning analytics methods that have been frequently cited. We used the
Google scholar as a foundation to check methods’ popularity. All the data were collected recently
and retrieved before the submission date. Figure 2 shows Google scholar citations for the analysed
LAK conference papers based on the methods category. The publications with the method type
Data Mining and Techniques, were the most cited articles (452 citations). The ultimate number of
citations in this survey belongs to the paper of (Kizilcec, Piech & Schneider, 2013) with 236
citations. Although we took into consideration the time span of publications, we see that articles
that belong to MOOCs are the most cited papers. Statistics and Mathematics publications were cited
363 times. Qualitative analysis and gamification publications were the least cited articles. In figure
3, we show a density plot of publications’ citations grouped by year. The x-axis records number of
citations converted into logarithmic scale to ease the reading. The y-axis records the density of
publications per year. Since we did not survey a fair number of publications per year, we intended
to use this plot instead of histogram plot, which is highly sensitive to bin size.
Figure 2. Number of Google scholar citations of the examined LAK papers based on
methods category. Retrieved on 26
th
October, 2015.
ISSN 2414-6099
300
Figure 3. Density plot of citations grouped by year
ISSN 2414-6099
301
Some of 2013 publications attracted numerous citations which exceed the expectations such as
(Kizilcec, Piech & Schneider, 2013; Pardos et al., 2013). A descriptive analysis of the articles in
2013 leads to: median=8, mean=24.37, max=236; articles in 2014: median=5, mean=5.56, max=17;
and articles of 2015: median =1, mean=1.42, max=4. The low number of citations for 2015 is
reasonable as the time span between this survey and 2015 LAK publication is around six months.
Challenges & Constraints
The data collection and analysis through learning analytics methods lead to questions related to
ownership, privacy and ethical issues (Khalil & Ebner 2015; Khalil & Ebner, 2016). We summarize
our experience and previous research studies as the following: A) Privacy, in which the learning
analytics specialists need to carefully deliberate the potential privacy issues while collecting,
analyzing, and intervene of the students’ data. B) Transparency of disclosing information of
learners and the needs to proclaim consent by the students. C) Assuring security and achieving the
CIA which is an acronym that refers to Confidentiality, Integrity and Accessibility such as storing
of learners records. D) The ownership of the collected and analysed data.
Conclusion
Learning Analytics is a promising area which provides the adequate tools and methods to optimize
the learning mechanism in the different environments of educational technology platforms. In this
paper, we did a meta-analysis study on publications in the last two years and classified seven
different categories of techniques that have been used in that period. We noticed that learning
analytics researchers adopt data mining and statistics more often than other techniques.
Additionally, 2013 was a stimulating year in showing MOOCs as a desirable article by a distinct
number of citations. Moreover, we also see that some publications have had a high impact on
education with their peak Google scholar score. In fact, the upcoming learning analytics events
might show extinction of methods and an uprising appearance or emergence of new techniques,
which can be allocated in our defined categories. Finally, we summarized our experience in this
field and listed some of the constraints and dilemmas that negatively affect learning analytics
approaches.
References
Aguilar, S., Lonn, S., & Teasley, S. D., 2014. Perceptions and use of an early warning system during a
higher education transition program. In Proceedings of the fourth international conference on learning
analytics and knowledge (pp. 113-117). ACM.
Baker, R. S. J. D., Duval, E., Stamper, J., Wiley, D., & Buckingham Shum, S., 2012. Panel: educational data
mining meets learning analytics. In Proceedings of 2nd International Conference on Learning Analytics &
Knowledge (LAK'12), New York, USA (p. 20).
Baker, R. S., & Siemens, G., 2013. Educational Data Mining and Learning Analytics.
Baron, J., Lynch, G., Maziarz, N., 2015. Proceedings of the Fifth International Conference on Learning
Analytics and Knowledge. ACM, New York, USA.
Bogarín, A., Romero, C., Cerezo, R., & Sánchez-Santillán, M., 2014. Clustering for improving educational
process mining. In Proceedings of the Fourth International Conference on Learning Analytics and
Knowledge (pp. 11-15). ACM.
ISSN 2414-6099
302
Camilleri, V., de Freitas, S., Montebello, M., & McDonagh-Smith, P., 2013. A case study inside virtual
worlds: use of analytics for immersive spaces. In Proceedings of the Third International Conference on
Learning Analytics and Knowledge (pp. 230-234). ACM.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G., 2007. Academic analytics: A new tool for a new
era. EDUCAUSE review, 42(4), 40.
Chiu, M. M., & Fujita, N., 2014. Statistical discourse analysis of online discussions: informal cognition,
social metacognition and knowledge creation. In Proceedings of the Fourth International Conference on
Learning Analytics And Knowledge (pp. 217-225). ACM.
Clow, D., 2012. The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd
international conference on learning analytics and knowledge (pp. 134-138). ACM.
Davidian, M., & Louis, T. A., 2012. Why statistics?. Science, 336(6077), 12-12.
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S., 2014. Current state and future trends: A citation
network analysis of the learning analytics field. In Proceedings of the Fourth International Conference on
Learning Analytics and Knowledge (pp. 231-240). ACM.
Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U., 2013. Supporting action research
with learning analytics. In Proceedings of the Third International Conference on Learning Analytics and
Knowledge (pp. 220-229). ACM.
Eagle, M., Hicks, D., Peddycord III, B., & Barnes, T., 2015. Exploring networks of problem-solving
interactions. In Proceedings of the Fifth International Conference on Learning Analytics And
Knowledge (pp. 21-30). ACM.
Elias, T., 2011. Learning Analytics: Definitions, Processes and Potential. Available from:
http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf. [25 October 2015].
Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S., 2015. Unsupervised modeling for understanding
MOOC discussion forums: a learning analytics approach. In Proceedings of the Fifth International
Conference on Learning Analytics And Knowledge (pp. 146-150). ACM.
Ferguson, R., 2012. Learning analytics: drivers, developments and challenges. In International Journal of
Technology Enhanced Learning, 4(5-6), 304-317.
Ferguson, R., Wei, Z., He, Y., & Buckingham Shum, S., 2013. An evaluation of learning analytics to identify
exploratory dialogue in online discussions. In Proceedings of the Third International Conference on
Learning Analytics and Knowledge (pp. 85-93). ACM.
Grann, J., & Bushway, D., 2014. Competency map: visualizing student learning to promote student success.
In Proceedings of the fourth international conference on learning analytics and knowledge (pp. 168-172).
ACM.
Hecking, T., Ziebarth, S., & Hoppe, H. U., 2014. Analysis of dynamic resource access patterns in a blended
learning course. In Proceedings of the Fourth International Conference on Learning Analytics and
Knowledge. ACM.
ISSN 2414-6099
303
Holman, C., Aguilar, S., & Fishman, B., 2013. GradeCraft: what can we learn from a game-inspired learning
management system?. In Proceedings of the Third International Conference on Learning Analytics and
Knowledge (pp. 260-264). ACM.
Jo, I. H., Kim, D., & Yoon, M., 2014. Analyzing the log patterns of adult learners in LMS using learning
analytics. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp.
183-187). ACM.
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H., 2013. NMC
Horizon Report: 2013 Higher Education Edition. Texas: The New Media.
Joksimović, S., Dowell, N., Skrypnyk, O., Kovanović, V., Gašević, D., Dawson, S., & Graesser, A. C., 2015.
How do you connect?: analysis of social capital accumulation in connectivist MOOCs. In Proceedings of the
Fifth International Conference on Learning Analytics And Knowledge (pp. 64-68). ACM.
Kapler, T., & Wright, W., 2005. GeoTime information visualization. Information Visualization, 4(2), 136-
146.
Kapp, K. M., 2012. The gamification of learning and instruction: game-based methods and strategies for
training and education. John Wiley & Sons.
Khalil, M., & Ebner, M., 2015. Learning Analytics: Principles and Constraints. In Proceedings of World
Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1326-1336).
Khalil, M., & Ebner, M., 2016. De-Identification in Learning Analytics. Journal of Learning Analytics.
Accepted, in Print.
Kizilcec, R. F., Piech, C., & Schneider, E., 2013. Deconstructing disengagement: analyzing learner
subpopulations in massive open online courses. In Proceedings of the third international conference on
learning analytics and knowledge (pp. 170-179). ACM.
Koulocheri, E., & Xenos, M., 2013. Considering formal assessment in learning analytics within a PLE: the
HOU2LEARN case. In Proceedings of the Third International Conference on Learning Analytics and
Knowledge (pp. 28-32). ACM.
Leeman-Munk, S. P., Wiebe, E. N., & Lester, J. C., 2014. Assessing elementary students' science
competency with text analytics. In Proceedings of the Fourth International Conference on Learning
Analytics And Knowledge(pp. 143-147). ACM.
Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A., 2015. The LATUX
workflow: designing and deploying awareness tools in technology-enabled learning settings. In Proceedings
of the Fifth International Conference on Learning Analytics And Knowledge (pp. 1-10). ACM.
Mendiburo, M., Sulcer, B., & Hasselbring, T., 2014. Interaction design for improved analytics.
In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 78-82).
ACM.
Pardos, Z. A., Baker, R. S., San Pedro, M. O., Gowda, S. M., & Gowda, S. M., 2013. Affective states and
state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 117-124).
ACM.
ISSN 2414-6099
304
Pistilli, M., Willis, J., Koch, D., 2014. Proceedings of the Fourth International Conference on Learning
Analytics And Knowledge. ACM, New York, USA.
Renzel, D., & Klamma, R., 2013. From micro to macro: Analyzing activity in the ROLE sandbox.
In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 250-254).
ACM.
Romero, C., & Ventura, S., 2010. Educational data mining: a review of the state of the art. Systems, Man,
and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 40(6), 601-618.
Santos, J. L., Klerkx, J., Duval, E., Gago, D., & Rodríguez, L., 2014. Success, activity and drop-outs in
MOOCs an exploratory study on the UNED COMA courses. In Proceedings of the Fourth International
Conference on Learning Analytics and Knowledge (pp. 98-102). ACM.
Santos, J. L., Verbert, K., Govaerts, S., & Duval, E., 2013. Addressing learner issues with StepUp!: an
Evaluation. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp.
14-22). ACM.
Schreurs, B., Teplovs, C., Ferguson, R., De Laat, M., & Buckingham Shum, S., 2013. Visualizing social
learning ties by type and topic: Rationale and concept demonstrator. In Proceedings of the Third
International Conference on Learning Analytics and Knowledge (pp. 33-37). ACM.
Siemens, G., 2012. Learning analytics: envisioning a research discipline and a domain of practice.
In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 4-8). ACM.
Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A., 2013. Analysis of collaborative writing processes
using revision maps and probabilistic topic models. In Proceedings of the Third International Conference on
Learning Analytics and Knowledge (pp. 38-47). ACM.
Suthers, D., Verbert, K., Duval, E., & Ochoa, X., 2013. Proceedings of the Third International Conference
on Learning Analytics and Knowledge. ACM, New York, USA.
Taraghi, B., Ebner, M., Saranti, A., & Schön, M., 2014, March. On using markov chain to evidence the
learning structures and difficulty levels of one digit multiplication. In Proceedings of the Fourth
International Conference on Learning Analytics And Knowledge (pp. 68-72). ACM.
Vogelsang, T., & Ruppertz, L., 2015. On the validity of peer grading and a cloud teaching assistant system.
In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 41-50).
Wolff, A., Zdrahal, Z., Nikolov, A., & Pantucek, M., 2013. Improving retention: predicting at-risk students
by analysing clicking behaviour in a virtual learning environment. In proceedings of the third international
conference on learning analytics and knowledge (pp. 145-149). ACM.
Xing, W., Wadholm, B., & Goggins, S., 2014. Learning analytics in CSCL with a focus on assessment: an
exploratory study of activity theory-informed cluster analysis. In Proceeding of LAK’14. ACM.
... Learning analytics is an emerging field and has been growing as more practitioners, institutions, and researchers see its potential [5]. The Society for Open Learning Analytics Research (SoLAR) has defined it as "the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs" [6]. So-LAR has also been contributing towards this field "to guide learners, educators, administrators, and funders in making learning-related decisions" [7]. ...
... (2) statistics and mathematics; (3) text mining, semantics, and linguistic analysis; (4) visualization; (5) social network analysis; (5) qualitative analysis; and (6) gamification [6]. Of these categories, it was found that both data mining techniques (such as classification, clustering, and predictive analytics) and studies involving statistics and mathematics for data analysis were the two most employed methods [6]. ...
... (2) statistics and mathematics; (3) text mining, semantics, and linguistic analysis; (4) visualization; (5) social network analysis; (5) qualitative analysis; and (6) gamification [6]. Of these categories, it was found that both data mining techniques (such as classification, clustering, and predictive analytics) and studies involving statistics and mathematics for data analysis were the two most employed methods [6]. In another paper, the authors state that "one of the main applications of learning analytics is tracking and predicting learners' performance as well as identifying potential problematic issues and students at work" [8]. ...
... Understanding the users' online behaviour i.e. how they learn, what they need for pursuing their study, and what is the best way for providing materials, requires extensive interdisciplinary research including from the computer science and statistics to psychology and pedagogy (Khalil and Ebner, 2016). The users' activities on educational platforms are collected as log data which contains valuable information about the users' behaviour. ...
Conference Paper
Full-text available
The availability of open educational resources is growing at an increasingly fast pace since its first promotion by UNESCO in 2002. Today, large variability of opportunities for free and online educational resources are available and accessible by everyone from all around the world who has access to the Internet. An Internet user may exploit numbers of different platforms to find what they are looking for, where one platform may fit their study goal while another platform suits their learning approach. Finding the appropriate content and platform could be like searching for a needle in the haystack where users desperately need help from personalised recommendations. Many platforms aim to transform to a more personalised learning environment, mostly by recommending a content or a peer to study with, providing timely feedback, or a gamified learning environment within the platform. We expect that in the next decade it will be necessary to provide user guidance to the Open Educational Resources not only in a single domain but in cross-domain, cross-site, and cross-cultural nature of the Internet. In this paper, we investigate the users’ learning behaviour by analysing their clickstream data across different learning platforms. The results indicate that most of the users tend to stay on a website for a short duration. Also, the design of materials on different websites affect the number of clicks and the pattern of engagement.
... The aim of this workshop paper is to discuss our proposal of a research area which we call Mobile Multimodal Learning Analytics Methodology (MOLAM) to trace, interpret and support students' development of self-regulated learning (SRL) strategies, skills and knowledge. While the focus of the learning analytics research varies (Khalil & Ebner, 2016;Viberg et al., 2018), increasing research attempts have recently targeted the area of self-regulated learning (SRL; Viberg, Khalil & Baars, 2020). ...
Conference Paper
Full-text available
The aim of this workshop paper is to propose Mobile Multimodal Learning Analytics Methodology (MOLAM). The methodology is suggested to be developed through the lenses of multidisciplinary and multichannel data research approaches, based on the theoretical foundations of Self-Regulated Learning (SRL). MOLAM is theory supported, driven by learning analytics, learner-centered focused, and mobile technology utilized. We argue that MOLAM will have a potential to support learners, teachers and researchers in their understanding and their further fostering of student SRL in formal and informal learning environments.
... Our DCA are based on data mining, tracking and collection techniques, which seem to be the most popular form of LA methods (Khalil & Ebner, 2016). ...
Article
Full-text available
Nowadays learning analytics has been growing as a science, and at the University of Turin we are interested in its potential to enhance both the teaching and the learning experience. In the last few years we have gathered data from two projects: Orient@mente, and start@unito, with the latter offering open online university courses in various disciplines. In addition, we have also studied and analysed the results of the teacher training experience carried out for the start@unito project, as well as those obtained from a survey involving secondary school teachers and the possible employment of the start@unito OERs in their everyday teaching. Our sources of data are students' activity online, the results of formative automatic assessment, and the questionnaires given to the learners; the types of questions range from Likert scale evaluations to multiple choice, yes/no and a few open questions. In this paper we discuss the different tasks we completed in our projects and evaluate their adherence with the learning analytics techniques for citations: Journal of e-Learning and Knowledge Society Je-LKS Je-LKS in terms of structure, availability, statistics, outcomes, interventions and, in general, their usefulness and effectiveness. In this way, the insights gained from both usage tracking and questionnaires can be used whenever possible to make interventions to improve the teaching and the learning experience; at the same time, when such interventions were not possible, we reflected on why this happened and how we can change and improve our approach.
... Finally, this research contributed to an understanding of how learning analytics information may be used to execute interventions, predictions, reflection, awareness, personalization, recommendation and benchmarking (Khalil & Ebner, 2016). The ways in which online learners engaged in educational dialogues among themselves provide indications of how knowledge building was generated. ...
Article
Full-text available
An advantage of online discussion and interaction for learning is that text-based communication allows time for reflection. Such text based communications and reflections provide a rich source of data for research. This research employed learning analytics to understand online learner knowledge building through discussion forums. The activity for such text-based communication is a global MOOC offering. The intention was to understand how the learners build their repertoire of knowledge through the online discussion forums. Online discussion forums are essential elements of MOOC and online learning. The transcripts of the discussion were compiled and the discourse analyzed. The research analyzed the contents of forum discussions using Atlas.ti, which is a qualitative data analysis software. This research revealed that knowledge building was mainly formed through the community knowledge that was generated by the learners. Knowledge building was also a result of posing authentic problems or questions that elicited real ideas connected to the actual situation that the learners were experiencing. It is recommended that relevant learning discussions should incorporate practices that encourage the development of meaningful learning dialogue. A study of this nature is significant to understand learner behaviors and to take actions to support and improve learning outcomes and retention.
... It is recommended that relevant learning discussions should incorporate practices that encourage the development of meaningful learning dialogue. This research contributed to an understanding of how learning analytics information may be used to execute interventions, predictions, reflection, awareness, personalization, recommendation and benchmarking (Khalil & Ebner, 2016). ...
Cover Page
Full-text available
Editorial Team: THANG Siew Ming, HELP University, Kuala Lumpur, Malaysia NG Lay Shi, Universiti Kebangsaan Malaysia, Malaysia This special issue is a compilation of selected articles presented at the GLoCALL 2018 Conferences which were organised jointly by PacCALL and ChinaCALL at Xi"an Jiatong Liverpool University from 16 th to 18 August 2018. This conference is part of the GLoCALL series of conference organised yearly since 2007 to share knowledge, research and experiences on how to use technology to enhance language learning and to explore how technology can be adapted to meet the needs of language learners in a variety of contexts. This special issue offers a selection of papers covering a range of topics that explore the multiple roles of technology in learning and teaching in diverse educational settings. The first four papers focus on multiple roles of technology in language learning. The fifth, sixth and seven articles describe studies that use technology tools to investigate student learning. The last two articles describe two studies that explore the effects of the use of technology on teachers. In the first paper entitled "The Use of Mobile Devices in Language Learning: A Survey on Chinese University Learners' Experiences, Gavin WU Junjie explored the experiences and perceptions of 235 Chinese EFL students in a foreign language school at an Eastern Chinese university towards the use of mobile devices in their learning (MALL). He believed that students' views and prior learning experiences "represent a valuable resource of information for teachers and policy-makers" and their voices have "a 'transformational potential' for school practices" (Manca & Grion, 2017). A questionnaire was conducted on 235 Chinese university students followed by a nine-student text-based group discussion to collect data for the study. The results of questionnaire revealed that over 85 percent of the Chinese learners reported a preference for using their smartphones for learning. They were also aware that they could use their smartphones and multiple online resources to facilitate their informal learning. More than half of them used their mobile devices to enhance their English listening, reading and writing skills, sometimes independently and sometimes directed by their teachers but over 65% of them did not use their mobile devices to improve their spoken English. Thus although the Chinese students appreciated the value of mobile devices in their learning however usage was still limited. The overall conclusion was that students should be encouraged and supported by their teachers in order to facilitate the development of their language skills through mobile learning. The second paper in this collection entitled "Virtual Reality in the Language Classroom: Theory and Practice" was written by Mehrasa Alizadeh. This paper begins with a brief review of the basic concepts of virtual reality in comparison with augmented reality, as well as ways to experience VR and how this new technology aligns with learning theories. The she introduced a free VR mobile application named Expeditions which is a VR educational platform that can be easily set up to allow
Thesis
Full-text available
Learning Analytics (LA) is a new promising field that is attracting the attention of education providers and a range of stakeholders including teachers, learning designers, academic directors and data scientists. Researchers and practitioners are interested in learning analytics as it can provide insights from student data about learning processes, learners who may need more help, and learners’ behaviours and strategies. However, problems such as low educator satisfaction, steep learning curves, misalignment between the analytics and pedagogical approaches, lack of engagement with learning technologies and other barriers to learning analytics development have already been reported. From a human-centred design perspective, these problems can be explained due to the lack of stakeholders’ involvement in the design of the LA tools. In particular, learners and teachers are commonly not considered as active agents of the LA design process. Including teachers, learners, developers and other stakeholders as collaborators in the 𝘤𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 of LA innovations can bring promising benefits in democratising the LA design process, aligning analytics and pedagogy, and meeting stakeholders’ expectations. Yet, working in collaboration with stakeholders to design LA innovations opens a series of questions that are addressed in this thesis in order to contribute to closing the gap for effective co-design of LA innovations. The questions addressed in this thesis are the following: 1. How can co-design techniques assist in the integration of diverse stakeholders in the LA design process? 2. What are the roles of the co-design practitioner/researcher in the LA design process? 3. What are the challenges in engaging stakeholders in the LA design process? Based on co-design principles, and following a Design-Based Research process, this thesis explores the critical challenge of engaging educators and students, the non-technical stakeholders who are often neglected, but who should ultimately be the main beneficiaries of LA innovations. In this research work, three case studies have been used to test, analyse and verify various co-design techniques in diverse learning contexts across a university to generate a co-design toolkit and recommendations for other co-design practitioners: i) learners and educators engaged in simulation-based healthcare scenarios, ii) learners, educators and other stakeholders in a Data Science Masters program , and iii) educators interested in providing personalised feedback at scale. This thesis presents three contributions to knowledge for effectively collaborating with educational stakeholders in the LA co-design process: 1. Inspired by archetypal challenges reported in classic and contemporary co-design literature, and in current LA research, the thesis identifies, exemplifies and reflects on five key challenges for LA co-design: power relationships, surveillance, learning design dependencies, asymmetric teaching/learning expertise, and data literacy. 2. By adopting and adapting well established co-design techniques, across the three case studies, the thesis provides empirical evidence of how these techniques can be used in LA co-design, reflecting on their affordances, and providing guidance on their usage. These detailed findings are distilled into a 𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘈𝘯𝘢𝘭𝘺𝘵𝘪𝘤𝘴 𝘊𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 𝘗𝘭𝘢𝘺𝘣𝘰𝘰𝘬, published under an open license to assist adoption and improvements. 3. Recognising the importance of the co-design practitioner in ensuring that the design process is participatory, the thesis documents and discusses the key functions and skills that this position requires. The role is further complicated when the practitioner is not only a 𝘧𝘢𝘤𝘪𝘭𝘪𝘵𝘢𝘵𝘰𝘳 serving a project, but also a 𝘳𝘦𝘴𝘦𝘢𝘳𝘤𝘩𝘦𝘳 of co-design. This motivates guidelines on the role of the co-design practitioner/researcher when working with stakeholders, and simultaneously studying the LA co-design process, tools and methods.
Chapter
Full-text available
There are new discoveries in the field of educational technologies in the 21st century, which we can also call the age of technology. Learning Analytics (LA) has given itself an important research field in the area of Technology Enhanced Learning. It offers analysis, benchmarking, review and development techniques for example in online learning platforms such as those who host Massive Open Online Course (MOOC). MOOCs are online courses addressing a large learning community. Among these participants, large data is obtained from the group with age, gender, psychology, community and educational level differences. These data are gold mines for Learning Analytics. This paper examines the methods, benefits and challenges of applying Learning Analytics in MOOCs based on a literature review. The methods that can be applied with the literature review and the application of the methods are explained. Challenges and benefits and the place of learning analytics in MOOCs are explained. The useful methods of Learning Analytics in MOOCs are described in this study. With the literature review, it indicates: Data mining, statistics and mathematics, Text Mining, Semantics-Linguistics Analysis, visualization, Social network analysis and Gamification areas are implementing Learning Analytics in MOOCs allied with benefits and challenges.
Conference Paper
Full-text available
Human-centred software design gives all stakeholders an active voice in the design of the systems that they are expected to use. However, this is not yet commonplace in Learning Analytics (LA). Co-design techniques from other domains therefore have much to offer to LA, in principle, but there are few detailed accounts of exactly how such sessions unfold. This paper presents the rationale driving a card-based co-design tool specifically tuned for LA, called LA-DECK. In the context of a pilot study with students, educators, LA researchers and developers, we provide qualitative and quantitative accounts of how participants used the cards. Using three different forms of analysis (transcript-centric design vignettes, card-graphs and time-on-topic), we characterise in what ways the sessions were “participatory” in nature, and argue that the cards succeeded in playing very similar roles to those documented in the literature on successful card-based design tools.
Article
Full-text available
Learning analytics has reserved its position as an important field in the educational sector. However, the large-scale collection, processing, and analyzing of data has steered the wheel beyond the borders to face an abundance of ethical breaches and constraints. Revealing learners’ personal information and attitudes, as well as their activities, are major aspects that lead to identifying individuals personally. Yet, de-identification can keep the process of learning analytics in progress while reducing the risk of inadvertent disclosure of learners’ identities. In this paper, the authors discuss de-identification methods in the context of the learning environment and propose a first prototype conceptual approach that describes the combination of anonymization strategies and learning analytics techniques.
Conference Paper
Full-text available
Within the evolution of technology in education, Learning Analytics has reserved its position as a robust technological field that promises to empower instructors and learners in different educational fields. The 2014 horizon report (Johnson et al., 2014), expects it to be adopted by educational institutions in the near future. However, the processes and phases as well as constraints are still not deeply debated. In this research study, the authors talk about the essence, objectives and methodologies of Learning Analytics and propose a first prototype life cycle that describes its entire process. Furthermore, the authors raise substantial questions related to challenges such as security, policy and ethics issues that limit the beneficial appliances of Learning Analytics processes.
Conference Paper
Full-text available
Designing, deploying and validating learning analytics tools for instructors or students is a challenge requiring techniques and methods from different disciplines, such as software engineering, human-computer interaction, educational design and psychology. Whilst each of these disciplines has consolidated design methodologies, there is a need for more specific methodological frameworks within the cross-disciplinary space defined by learning analytics. In particular there is no systematic workflow for producing learning analytics tools that are both technologically feasible and truly underpin the learning experience. In this paper, we present the LATUX workflow, a five-stage workflow to design, deploy and validate awareness tools in technology-enabled learning environments. LATUX is grounded on a well-established design process for creating, testing and re-designing user interfaces. We extend this process by integrating the pedagogical requirements to generate visual analytics to inform instructors' pedagogical decisions or intervention strategies. The workflow is illustrated with a case study in which collaborative activities were deployed in a real classroom.
Conference Paper
Full-text available
Connections established between learners via interactions are seen as fundamental for connectivist pedagogy. Connections can also be viewed as learning outcomes, i.e. learners' social capital accumulated through distributed learning environments. We applied linear mixed effects modeling to investigate whether the social capital accumulation interpreted through learners' centrality to course interaction networks, is influenced by the language learners use to express and communicate in two connectivist MOOCs. Interactions were distributed across the three social media, namely Twitter, blog and Facebook. Results showed that learners in a cMOOC connect easier with the individuals who use a more informal, narrative style, but still maintain a deeper cohesive structure to their communication.
Article
Recently, interest in how this data can be used to improve teaching and learning has also seen unprecedented growth and the emergence of the field of learning analytics. In other fields, analytics tools already enable the statistical evaluation of rich data sources and the identification of patterns within the data. These patterns are then used to better predict future events and make informed decisions aimed at improving outcomes (Educause, 2010). This paper reviews the literature related to this emerging field and seeks to define learning analytics, its processes, and its potential to advance teaching and learning in online education.
Chapter
During the past decades, the potential of analytics and data mining - methodologies that extract useful and actionable information from large datasets - has transformed one field of scientific inquiry after another (cf. Collins, Morgan, & Patrinos, 2004; Summers et al., 1992). Analytics has become a trend over the past several years, reflected in large numbers of graduate programs promising to make someone a master of analytics, proclamations that analytics skills offer lucrative employment opportunities (Manyika et al., 2011), and airport waiting lounges filled with advertisements from different consultancies promising to significantly increase profits through analytics. When applied to education, these methodologies are referred to as learning analytics (LA) and educational data mining (EDM). In this chapter, we will focus on the shared similarities as we review both parallel areas while also noting important differences. Using the methodologies we describe in this chapter, one can scan through large datasets to discover patterns that occur in only small numbers of students or only sporadically (cf. Baker, Corbett, & Koedinger, 2004; Sabourin, Rowe, Mott, & Lester, 2011); one can investigate how different students choose to use different learning resources and obtain different outcomes (cf. Beck, Chang, Mostow, & Corbett, 2008); one can conduct fine-grained analysis of phenomena that occur over long periods of time (such as the move toward disengagement over the years of schooling - cf. Bowers, 2010); and one can analyze how the design of learning environments may impact variables of interest through the study of large numbers of exemplars (cf. Baker et al., 2009). In the sections that follow, we argue that learning analytics has the potential to substantially increase the sophistication of how the field of learning sciences understands learning, contributing both to theory and practice.
Conference Paper
Intelligent tutoring systems and other computer-aided learning environments produce large amounts of transactional data on student problem-solving behavior, in previous work we modeled the student-tutor interaction data as a complex network, and successfully generated automated next-step hints as well as visualizations for educators. In this work we discuss the types of tutoring environments that are best modeled by interaction networks, and how the empirical observations of problem-solving result in common network features. We find that interaction networks exhibit the properties of scale-free networks such as vertex degree distributions that follow power law. We compare data from two versions of a propositional logic tutor, as well as two different representations of data from an educational game on programming. We find that statistics such as degree assortativity and the scale-free metric allow comparison of the network structures across domains, and provide insight into student problem solving behavior.
Conference Paper
We introduce a new grading system, the Cloud Teaching Assistant System (CTAS), as an additional element to instructor grading, peer grading and automated validation in massive open online courses (MOOCs). The grading distributions of the different approaches are compared in an experiment consisting of 476 exam participants. 25 submissions were graded by all four methods. 451 submissions were graded only by peer grading and automated validation. The results of the experiment suggest that both CTAS and peer grading do not simulate instructor grading (Pearson's correlations: 0.36, 0.39). If the CTAS and not the instructor is assumed to deliver accurate grading, peer grading is concluded to be a valid grading method (Pearson's correlation: 0.76).
Conference Paper
Massively Open Online Courses (MOOCs) have gained attention recently because of their great potential to reach learners. Substantial empirical study has focused on student persistence and their interactions with the course materials. However, most MOOCs include a rich textual dialogue forum, and these textual interactions are largely unexplored. Automatically understanding the nature of discussion forum posts holds great promise for providing adaptive support to individual students and to collaborative groups. This paper presents a study that applies unsupervised student understanding models originally developed for synchronous tutorial dialogue to MOOC forums. We use a clustering approach to group similar posts, compare the clusters with manual annotations by MOOC researchers, and further investigate clusters qualitatively. This paper constitutes a step toward applying unsupervised models to asynchronous communication, which can enable massive-scale automated discourse analysis and mining to better support students' learning.