Conference PaperPDF Available

Abstract and Figures

While Enterprise MOOCs have been established alongside academic MOOCs in recent years, there is still only limited evidence on learner behavior in MOOCs with a clear focus on job-related training and professional development. This short paper addresses this gap by analyzing learner behavior in openSAP Enterprise MOOCs. By means of lag sequential analysis, data from 13 MOOCs from the topic areas business, design, and technology with a total number of N = 72, 668 learners have been analyzed. Consistent high-level behavioral patterns over all three topic areas could be identified. Implications for future research and development are being discussed.
Content may be subject to copyright.
Behavioral Patterns in Enterprise MOOCs at openSAP
Muhittin ¸Sahin1, Marc Egloffstein2, Max Bothe3, Tobias Rohloff3, Nathanael
Schenk4, Florian Schwerer4, and Dirk Ifenthaler5
1Ege University and University of Mannheim
musahin@mail.uni-mannheim.de
2University of Mannheim
eglostein@uni-mannheim.de
3Hasso Plattner Institute, Potsdam
max.bothe,tobias.rohlo@hpi.de
4SAP SE, Walldorf
nathanael.schenk,florian.schwerer@sap.com
5University of Mannheim and Curtin University
ifenthaler@uni-mannheim.de
While Enterprise MOOCs have been established alongside academic
MOOCs in recent years, there is still only limited evidence on learner
behavior in MOOCs with a clear focus on job-related training and pro-
fessional development. This short paper addresses this gap by analyzing
learner behavior in openSAP Enterprise MOOCs. By means of lag sequen-
tial analysis, data from 13 MOOCs from the topic areas business, design,
and technology with a total number of
N
=72,668 learners have been an-
alyzed. Consistent high-level behavioral patterns over all three topic areas
could be identified. Implications for future research and development are
being discussed.
1Introduction
Massive Open Online Courses (MOOCs) have been a growing element in higher
education for more than ten years. Especially the advantage of reaching large num-
bers of learners worldwide seems to be attractive for universities and educational
organizations [5,10]. Over the recent years, MOOCs have also become a viable al-
ternative for corporate training and professional development [6]). One of the most
advanced implementations is openSAP, an open learning platform related to the
Tech/IT-sector. While many companies do not seize the full potential of MOOCs
for training and development [4] or even lack adequate support [8], openSAP im-
plements so-called Enterprise MOOCs [18] to successfully convey knowledge about
new technologies and business topics within the organization as well as to external
stakeholders throughout the enterprise ecosystem [15]. Against the background of
281
¸Sahin et al.: Behavioral Patterns in Enterprise MOOCs at openSAP
the common criticism of MOOCs in terms of instructional quality [7] or completion
rates [13], openSAP seeks to constantly optimize its offering and thus to improve
the learning experience. Therefore, the existing R&D partnership with the Hasso
Plattner Institute (technical expertise) has been extended with the University of
Mannheim, Chair of Learning, Design and Technology (instructional design and
learning analytics expertise). As part of the partnership, several research activities
seek implications for learning design of MOOCs to further advance the openSAP
offering, and its learning experience [9]. This paper reports an initial case study of
this R&D partnership focusing on the behavior of learners in openSAP Enterprise
MOOCs. More specifically, the research project seeks to (1) identify typical behav-
ioral patterns in openSAP Enterprise MOOCs and (2) find out if such patterns
differ between courses from different topic areas. The remainder of this paper
introduces the openSAP University and describes how the R&D partnership ad-
dressed the research questions using a Lag Sequential Analysis (LSA) approach.
Then, the context and findings are presented. The paper closes with a discussion
of implications and an outlook for future research.
2The openSAP University
As part of SAP’s digital education strategy, the openSAP learning platform (avail-
able at open.sap.com) was established in 2013 to meet the increasing demands of
partners, customers and suppliers for knowledge on corporate strategy, business
innovations and product releases in a timely manner [16]. openSAP delivers knowl-
edge via scalable online courses, thus suitable for larger audiences. The courses
are open to everyone and free of charge, providing videos, quizzes, and interaction
in a digital classroom over a fixed period of time. The main topic areas are tech-
nology and software, business, or design; while some additional courses provide
insights on corporate social responsibility-related topics. The technical infrastruc-
ture is based on the HPI MOOC platform developed at the Hasso Plattner Institute
in Potsdam, Germany. In early 2021, the platform counts more than 1.1million
unique registrants from over 200 countries with more than five million enrollments
in around 250 different courses.
With respect to instructional design (ID), openSAP courses follow an elaborate
xMOOC model, providing a structured and well-organized offering [2]. Course
completion can be achieved upon two kinds of certificates. Learners receive a so-
called Confirmation of Participation (CoP) by accessing at least 50% of the overall
course content. In addition, the participants will obtain a Record of Achievement
(RoA) when achieving at least 50% of the points available in the weekly assign-
ments and the final exam.
282
3Sequential Analysis of Online Learning Behavior
3Sequential Analysis of Online Learning Behavior
Sequential analysis is a well-established method of inferential statistics [19], which
can be employed for investigating behavior of learners in online learning systems
[17]. It is regarded as a suitable approach when investigating behavior within an
ongoing interaction [1] and thus has been applied to various MOOC settings (e.g.
[3]. Sequential relationships of observations and events with each other are also
considered in sequential analysis [1]. Log-linear models, lag sequential methods,
z-scores and sequential pattern mining can be used to carry out sequential analysis
and determine sequential patterns. In order to identify typical learning behaviors
of learners, transition probabilities are used to identify significant patterns [1].
The stochastic models provide the mathematical basis for precisely computing
learning-dependent changes in learning environments such as MOOCs [12]. The
analytics process of LSA for this project consists of six distinctive steps: (1) develop
event sequence, (2) map out transitional frequency matrix, (3) derive transitional
probability matrix, (4) calculate z-scores and carry out test of significance, (5) draw
state transition diagram.
4Learner Behavior in Enterprise Moocs
4.1Sample, Data Collection, and Procedure
User events from 13 openSAP courses from the topic areas Business, Design and
Technology have been analyzed with regard to patterns in learner behavior. The
courses in the sample show variations in terms of length, effort, and design pa-
rameters like assessment configuration or additional ID elements (e.g. reflection
prompts or coding exercises). Table 1provides on overview of the courses in the
sample.
The data used to conduct LSA consists of learners’ interactions with the digital
learning environment on the basis of traceable system states and events. In a
preliminary step of data preparation, the event data generated by interactions with
the platform was coded into delineable sessions. A session is based on a sequence
of events whose interval does not exceed 60 minutes, i.e. sessions expire after an
hour of inactivity.
Learners’ interactions with digital learning environments can be classified into
three categories: learner-content, learner-discussion (learner-learner), and learner-
instructor [14]. Following the HPI MOOC platform’s overall structure, the learner
events in a course can be assigned into four main categories: L – Learning (e.g.
video playbacks, self-test submissions, visits to learning items), D – Discussion (e.g.
283
¸Sahin et al.: Behavioral Patterns in Enterprise MOOCs at openSAP
Table 1:Descriptive information on the courses in the sample
Topic area Course Course
length
(weeks)
Max
effort
(hours)
Assess-
ment
config.a
Add.
ID
elements
Enroll-
mentsb
Number
of active
learners
b
Number
of inter-
actions
Business xm1 1 3 w0 4609 2597 125147
leo2 2 8 w+f 1 10542 5626 534548
pa1-tl 3 12 w+f 1 6904 4070 415258
s4h15 4 16 w+f 0 18265 12277 2023627
sbw1 6 24 w1 11664 6270 731436
Design build1 4 16 w+f 2 7749 4350 355387
cwr1-1 3 12 w+p 2 1810 1005 82072
dafie1 5 20 w+p 2 5283 2678 204060
sps3 5 20 w+p 1 6629 3143 380989
Technology ieux1 1 4 w0 13431 6784 205950
java1 5 30 w+e+f 3 21693 11757 3866382
mobile3
5 25 w+f 1 10374 5928 807607
sps2 3 12 w+f 1 10940 5783 721967
Note. aw: weekly assignment; f: final exam; e: graded exercise; p: peer assessment; bat course end.
post comments), P – Progress (e.g. visits of the progress page) and A – Announce-
ment (e.g. visits of the announcement page).
In the first step of LSA, event sequences were created session by session for
each learner based on the interactions with the learning platform. An example of
a simple event sequence would be: LLLLDDLLLPDAALLLL. In the second step,
transitional frequency matrices were created. Then, the transitional probability
matrix was mapped out. Transitional probability is a conditional probability; events
occur in different times and “lag” is used to express these time differences [1]. In
order to test the statistical significance of the transitions, z-scores were calculated,
together with a Bonferroni adjustment to determine the z-score threshold. In the
last step, a state transition diagram was generated for displaying the results.
4.2Results
4.2.1Behavioral Patterns in openSAP Courses
Over all 13 courses in the sample, significant transitions between the four main
categories could be traced. Table 2shows the respective z-scores.
284
4Learner Behavior in Enterprise Moocs
Table 2:z-scores based on the interaction categories
z-score
Announcements
Discussion Learning Progress
Announcement
587.92*41.20*203.42 135.13*
Discussion 30.26*2274.35*1814.73 75.90*
Learning –237.32 1799.57 1572.44*223.21
Progress 208.72*17.38*191.24 227.52*
Note. z-score threshold: 2.96; * statistically significant transitions
The respective state transition diagram for the high-level interactions is shown
in Figure 1.
Created in Master PDF Editor
L
D
P
A
Figure 1:State transition diagram for the overall sample
The state transition diagram shows significant transitions between all the main
categories except for the learning category. Looking at high-level interactions, the
biggest category in terms of events captured is rather isolated.
4.2.2Differences in Behavioral Patterns According to Topic Area
In order to tackle the second research question, LSA was carried for each “course
bucket” (set of courses from one topic area) separately. The underlying assumption
is that learners from the topic areas Business, Design und Technology are using dif-
ferent learning elements in a different frequency through different learning paths,
to different dates in time, with a difference in effort and thus show differences in
content consumption. Results, again, are illustrated in state transition diagrams
(Figure 2a–c):
As the learning category, again, remains isolated from the others, the data show
a consistent pattern on this high level of analysis. Apart from that, behavioral
patterns are similar but there are some minor differences. For example, learners
interact within the discussion category and then interact with the progress.
285
¸Sahin et al.: Behavioral Patterns in Enterprise MOOCs at openSAP
Created in Master PDF Editor
L
D
P
A
L
D
P
A
L
D
P
A
Figure 2:(a) Business courses (b) Design courses (c) Technology courses
5Discussion and Outlook
This study sought to investigate typical behavioral patterns in openSAP Enterprise
MOOCs and if such patterns differ between courses from different topic areas.
Findings indicate that (1) there are consistent patterns and that (2) many charac-
teristics of those patterns also apply when a differential perspective is adopted
with respect to topic areas. The learning category, for example, which contains the
majority of system interactions, remains isolated from the other categories, at least
from the high-level perspective employed in this research. This might be due to a
clear learner focus on working through the contents and towards the assignments,
while the announcement, progress and discussion categories are more likely to
be addressed at the beginning or the ending of a learning session. Moreover, an-
nouncements are also communicated via additional channels (e.g. e-mail), and the
learner progress is evident in the learning area, too. So if there really is a need
to better connect learning activities to collaborative (discussion) or meta-cognitive
(announcements, progress) activities, cannot yet be decided at this stage.
Thus, there is a need for further analyses, on a more granular level, related to
system interactions, as this is also the level on which possible interventions have
to be designed to. Likewise, progress and performance data must be combined
with the more granular interaction data, in order to discern successful and possibly
misleading patterns with regard to learning success. Learner context data collected
in accordance with applicable data protection guidelines will add an additional
layer of detail here. Eventually, the goal is to develop an analytics-driven behavioral
process model that might serve as a baseline for learning design [11]. The high-
level behavioral patterns identified mark the necessary initial step to contextualize
and interpret user-generated data to identify, understand, and cater different user
groups and their learning behavior in an instructional setting, and to ultimately
improve the overall learning experience and success.
286
References
References
[1]
R. Bakeman and J. M. Gottman. Observing interaction: An introduction to
sequential analysis. Cambridge: University press, 1997.
[2]
C. J. Bonk, M. M. Lee, T. C. Reeves, and T. H. Reynolds, editors. MOOCs
and Open Education Around the World. New York, NY: Routledge, 2015.doi:
10.4324/9781315751108.
[3]
M. S. Boroujeni and P. Dillenbourg. “Discovery and temporal analysis of
MOOC study patterns”. In: Journal of Learning Analytics 6.1(2019), pages 16
33.doi:10.18608/jla.2019.61.2.
[4] J. Condé and M. Cisel. “On the Use of MOOCs in Companies: A Panorama
of Current Practices”. In: Digital Education: At the MOOC Crossroads Where
the Interests of Academia and Business Converge. Edited by M. Calise, C. Kloos,
J. Reich, J. A. Ruiperez-Valiente, and M. Wirsing. Cham: Springer, 2019,
pages 3746.doi:10.1007/978-3-030- 19875-6_5.
[5]
J. R. Corbeil, B. H. Khan, and M. E. Corbeil. “MOOCs revisited: Still transfor-
mative or passing fad?” In: Asian Journal of University Education 14.2(2018),
pages 112.
[6]
M. Egloffstein and D. Ifenthaler. “Employee perspectives on MOOCs for
workplace learning”. In: TechTrends 61.1(2017), pages 6570.doi:10.1007 /
s11528-016-0127-3.
[7]
M. Egloffstein, K. Koegler, and D. Ifenthaler. “Instructional quality of busi-
ness MOOCs: Indicators and initial findings”. In: Online Learning 23.4(2019),
pages 85105.doi:10.24059/olj.v23i4.2091.
[8]
M. Hamori. “MOOCs at work: what induces employer support for them?”
In: The International Journal of Human Resource Management (2019). doi:10.
1080/09585192.2019.1616593.
[9]
D. Ifenthaler. “Learning analytics design”. In: The sciences of learning and
instructional design. Constructive articulation between communities. Edited by
L. Lin and J. M. Spector. New York, NY: Routledge, 2017, pages 202211.
doi:10.4324/9781315684444.
[10]
D. Ifenthaler, N. Bellin-Mularski, and D.
-
K. Mah. “Internet: Its impact and
its potential for learning and instruction”. In: The SAGE encyclopedia of edu-
cational technology. Edited by J. M. Spector. Volume 1. Thousand Oaks, CA:
Sage, 2015, pages 416422.doi:10.4135/9781483346397.
[11]
D. Ifenthaler, D. C. Gibson, and E. Dobozy. “Informing learning design
through analytics: Applying network graph analysis”. In: Australasian Journal
of Educational Technology 34.2(2018), pages 117132.doi:10.14742/ajet.3767.
287
¸Sahin et al.: Behavioral Patterns in Enterprise MOOCs at openSAP
[12]
D. Ifenthaler and N. M. Seel. “A longitudinal perspective on inductive reason-
ing tasks. Illuminating the probability of change”. In: Learning and Instruction
21.4(2011), pages 538549.doi:10.1016/j.learninstruc.2010.08.004.
[13]
N. Li, Ł. Kidzi´nski, P. Jermann, and P. Dillenbourg. “MOOC video interaction
patterns: What do they tell us?” In: Design for teaching and learning in a
networked world. Edited by G. Conole, T. Klobuˇcar, C. Rensing, J. Konert, and
É. Lavoué. Volume 9307. Amsterdam: Springer, 2015, pages 197210.doi:
10.1007/978-3-319-24258- 3_15.
[14]
M. G. Moore. “Three types of interaction”. In: American Journal of Distance
Education 3.2(1989), pages 16.
[15]
J. Renz, C. Meinel, and C. Link. “openSAP: Why Are Enterprise MOOCs
Working?” In: International Journal of Advanced Corporate Learning 12.3(2019),
pages 5969.doi:10.3991/ijac.v12i3.11262.
[16]
T. Rohloff, F. Schwerer, N. Schenk, and C. Meinel. “openSAP: Learner Behav-
ior and Activity in Self-Paced Enterprise MOOCs”. In: International Journal
of Advanced Corporate Learning 13.2(2020), pages 3040.doi:10.3991/ijac.v13i2.
16531.
[17]
M. ¸Sahin, S. Keskin, and H. Yurdugül. “Sequential Analysis of Online Learn-
ing Behaviors According to E-Learning Readiness”. In: Online Teaching and
Learning in Higher Education. Edited by P. Isaias, D. G. Sampson, and D. Ifen-
thaler. Cham: Springer, 2020, pages 117131.doi:10.1007/978-3- 030-48190-2_7.
[18]
F. Schwerer and M. Egloffstein. “Participation and Achievement in Enterprise
MOOCs for Professional Learning”. In: Proceedings of the 13th International
Conference on Cognition and Exploratory Learning in the Digital Age (CELDA
2016). Edited by D. G. Sampson. 2016, pages 269276.
[19] A. Wald. Analysis. New York, NY: Dover Publications, 1973.
288
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
The concept of instructional quality is central to the design and evaluation of massive open online courses (MOOCs). As MOOCs from the field of business and management are gaining importance both in academia and professional learning, questions on how to determine and improve the quality of these offerings arise. In this paper, we introduce an instrument for evaluating MOOCs against a set of theoretically grounded instructional design principles. After an overview of related research, we describe the concise course scan rubric and its application in detail. A pilot study with N = 101 business MOOCs reveals their rather low overall instructional quality. While most aspects of structuredness and clarity are rated high, the implementation of instructional design principles falls notably behind. The implications from our study point toward a learner-oriented notion of instructional quality and individualized learning and increased learner support in business MOOCs.
Article
Full-text available
Learning design has traditionally been thought of as an activity occurring prior to the presentation of a learning experience or a description of that activity. With the advent of near real-time data and new opportunities of representing the decisions and actions of learners in digital learning environments, learning designers can now apply dynamic learning analytics information on the fly in order to evaluate learner characteristics, examine learning designs, analyse the effectiveness of learning materials and tasks, adjust difficulty levels, and measure the impact of interventions and feedback. In a case study with 3550 users, the navigation sequence and network graph analysis demonstrate a potential application of learning analytics design. Implications based on the case study show that integration of analytics data into the design of learning environments is a promising approach.
Article
Massive Open Online Courses (MOOCs) have been a subject of research since 2012, especially in the context of professional development and workplace learning due to their flexible schedule and format, which is a prerequisite for on-the-job learning. But MOOCs often do not fulfill the promise of flexible learning as it is only possible to achieve a certificate during the course runtime. An unpredictable workload and thus a lack of time often results in not showing up to a course or dropping out during the course runtime. Therefore, some platform content remains accessible even after the course runtime in self-paced mode. These courses differ from live courses as participants still can access all of the content and the discussion forum in read-only mode, but are not able to take the graded assignments and exams which are a prerequisite to achieving a certificate at the end of a course. Even though it is only possible by paying a fee to earn a graded certificate for these self-paced courses, we identified a high share of additional enrollments after the course end that suggests there is still interest from participants. Nevertheless, learning behavior in self-paced courses has not been a major subject of research, yet. This work contributes to closing this research gap by exploring the learner behavior in self-paced courses. The results show tendencies of more time-efficiency and engagement of self-paced learners under certain conditions and pave the way for further research and practical applications.
Chapter
An important advantage of e-learning environments is the numerical observation of the learning behaviors of learners. The use of e-learning environments by learners creates a learner data (log data). From these learner data, the navigation patterns obtained by using educational data mining have a very important in learning and teaching design. Studies have shown that learners’ learning behaviors in online learning environments may vary according to the characteristics of learners. Studies on the differentiation of the navigation patterns according to the psycho-educational characteristics of the learners provide very strong inputs to the design of the learning environment appropriate to the characteristics of the learners, which is named as adaptive learning environments. According to these inputs, learning environment designs can be developed according to the individual characteristics of the learners. Online learners’ readiness (OLR) for e-learning is an important psycho-educational structure. The aim of this study is to investigate learners’ navigations in the e-learning environment according to the level of readiness for e-learning. Self-directed learning, learner control, motivation sub-dimensions were used in this study as online readiness sub-dimensions. The consecutive analysis was used to reveal the model of human behavior and communication patterns. For this purpose, lag sequential analysis was used when learners’ system interactions were analyzed sequentially. According to the results of the analysis, it has been found that the sequential navigation patterns of the learners differ according to the OLR structure. The findings of this research are expected to provide important information and suggestions to online learning environment designers.
Article
openSAP is one of the first and leading companies offering free online courses. After more than five years of offering openSAP.com this paper at hand takes a look at insight gained during this time and aims to identify success factors for enterprise MOOC providers.
Article
Massive open online courses (MOOCs) are the learning technology with the fastest adoption rates in recent years, and they have the potential to transform corporate development practices. There is, however, only fragmented evidence on how employers use MOOCs. This paper relies on human capital theory to formulate hypotheses about the antecedents of employer support for MOOCs. It tests the hypotheses using data from a survey of MOOC learners and secondary data on the courses and on learners’ countries. The results confirm that small employers that lack the scale to invest in in-company formal training are more likely to provide support for MOOCs than their larger counterparts, and that some organizations grant learners time off from work for MOOCs with not only core, but also non-core but still job-relevant content. Overall, however, employers fail to capitalize on low-cost MOOCs to compensate for a lack of formal training. On the contrary, what little MOOC support they offer goes to executives and full-time, rather than part-time employees.
Chapter
MOOCs represent an opportunity for companies to either save money, by asking their employees to follow a free course instead of using paying services, or to freely increase the proportion of workers who benefit from training opportunities. Some companies go beyond providing mere encouragements to follow these online courses. They can set common time slots for employees to collaborate on the course, allow them to follow the MOOC during working hours, discharge them of some tasks, or even reward, to some extent, those who manage to complete the course. In this article, we study these practices through a survey that was answered by 1847 users of Unow, a French platform that used to design MOOCs that targeted companies. It is uncommon for employees to be allowed to follow the course during working hours. MOOCs are typically integrated in an informal way, since they do not fit in the traditional frameworks structuring professional training in companies. It comes at a risk for employees, who may have to negotiate in an interpersonal way in what conditions the course is followed, without the protection of negotiated company agreements.
Article
The large-scale and granular interaction data collected in online learning platforms such as massive open online courses (MOOCs) provide unique opportunities to better understand individuals’ learning processes and could facilitate the design of personalized and more effective support mechanisms for learners. In this paper, we present two different methods of extracting study patterns from activity sequences. Unlike most of the previous works, with post hoc analysis of activity patterns, our proposed methods could be deployed during the course and enable the learners to receive real-time support and feedback. In the first method, following a hypothesis-driven approach, we extract predefined patterns from learners’ interactions with the course materials. We then identify and analyze different longitudinal profiles among learners by clustering their study pattern sequences during the course. Our second method is a data-driven approach to discover latent study patterns and track them over time in a completely unsupervised manner. We propose a clustering pipeline to model and cluster activity sequences at each time step and then search for matching clusters in previous steps to enable tracking over time. The proposed pipeline is general and allows for analysis at different levels of action granularity and time resolution in various online learning environments. Experiments with synthetic data show that our proposed method can accurately detect latent study patterns and track changes in learning behaviours. We demonstrate the application of both methods on a MOOC dataset and study the temporal dynamics of learners’ behaviour in this context.
Article
Massive Open Online Courses (MOOCs) can be considered a rather novel method in digital workplace learning, and there is as yet little empirical evidence on the acceptance and effectiveness of MOOCs in professional learning. In addition to existing findings on employers’ attitudes, this study seeks to investigate the employee perspective towards MOOCs in professional contexts and to further explore the acceptance of MOOCs for workplace learning. In a survey study, N = 119 employees from a wide range of enterprises were questioned with regard to motivation, credentials, and incentives related to participation in MOOCs. Findings indicate a high importance of on-the-job and career development learning purposes as well as a general interest in MOOC topics. Credentials are deemed necessary, yet their acceptance among the relevant stakeholders is considered rather low. Implications for design and implementation of MOOCs in digital workplace learning and for further research are discussed.