Content uploaded by Stylianos Sergis
Author content
All content in this area was uploaded by Stylianos Sergis on Oct 09, 2015
Content may be subject to copyright.
IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, MANUSCRIPT ID 1
Learning Object Recommendations For
Teachers Based On Elicited ICT Competence
Profiles
Stylianos Sergis, Student Member, IEEE and Demetrios G. Sampson, Golden Core and Senior
Member, IEEE
Abstract — Recommender Systems (RS) offer personalized services for facilitating the process of appropriate item selection.
To perform this task, user profiling mechanisms should be implemented to automatically construct and update meaningful user
profiles. These profiles can drive the RS in providing informed recommendations suited to the unique characteristics of each
user. In the context of Technology enhanced Learning (TeL) Recommender Systems, the majority of research focus directly on
learners’ profiling and ignore the potential benefits of profiling teachers’ professional capacities too. As a result, limited previous
works exist on effectively capturing and utilizing individual teachers’ particular professional characteristics, such as their Digital
Competences (commonly referred to as ICT Competences) and exploiting these in systems that support their teaching
preparation and practice, for example in the selection of appropriate educational resources. This paper proposes a RS which
targets to support teachers in selecting Learning Objects (LO) from existing LO Repositories (LORs) in a unified manner,
namely by (a) automatically constructing their ICT Competence Profiles based on their actions within these LORs and (b)
exploiting these profiles for more efficient LO selection. Experiments with data from 3 real-life LORs are presented and
evaluation results are discussed to demonstrate the benefits of the proposed system.
Index Terms— Learning Objects, Personalized E-Learning, Web Services, Recommender Systems
—————————— ——————————
1 INTRODUCTION
ECOMMENDER Systems (RS) are software tools de-
signed to assist users in tackling the information over-
load problem by highlighting suitable items in a personal-
ized manner [1]. They have been used in a wide range of
application contexts, spanning from the movie industry
[2] to e-commerce [3] and Technology enhanced Learning
(TeL) [4].
RS have been implemented based on an increasing
number of techniques, the most prominent and common-
ly used being the content-based filtering, the collaborative
filtering, the demographic filtering and the hybrid ap-
proaches [1]. The content-based filtering systems generate
recommendations based on users' past choices and the
content similarity between the items that were favored in
the past to novel ones that have not yet been discovered
[5]. The collaborative filtering systems utilize the users'
ratings over the available items for providing recommen-
dations based on information provided by the "like-
minded" neighbors of the active user (i.e., the user cur-
rently using the system and receiving the generated rec-
ommendations) [6]. The demographic filtering relies on
the assummed commonalities that users with similar de-
mographic backgrounds will have [7]. Finally, hybrid
approaches combine techniques from the other approach-
es in order to reap the benefits of all, while tackling their
individual drawbacks [8].
In order to provide their personalized services, RS are
usually implemented to automatically gather data from
the users towards capturing their unique attributes, creat-
ing and updating individual profiles [9]. The reason for
this is that users tend to either not manually provide per-
sonal data, or they do not provide them in an accurate
manner, at a great cost to the RS prediction capacity in
both occassions [9]. Therefore, and in-line with the gen-
eral need for constructing and exploiting highly granulat-
ed user profiles in RS applications [10], intelligent mecha-
nisms should be employed in order to collect and process
meaningful user data for infering essential user character-
istics that could provide added value in the quality of
recommendations.
In the context of TeL, RS have been primarily utilized
for the recommendation of different types of Learning
Objects (LO) based on individual teachers’ and learners’
profiles. Nevertheless, although learners’ profiling has
been extensively considered, teachers’ profiling is almost
neglected [11]. Taking into consideration, the important
role of teachers in adopting technology-supported school-
based educational innovations [12], the complexity of
technology-supported teaching practice [13] and the cur-
rent diversity of individual teachers’ digital competences
[14], it is safe to assume that individual teachers’ profes-
xxxx-xxxx/0x/$xx.00 © 200x IEEE Published by the IEEE Computer Society
————————————————
S.Sergis is with the Department of Digital Systems, University of Pi-
raeus, Piraeus, GR-18532, Greece, and the Information Technologies In-
stitute, Centre for Research and Technology Hellas, Greece. E-mail: ste-
liossergis@iti.gr
D. G. Sampson is with the School of Education, Curtin University
(since October 2015) and the Information Technologies Institute, Centre
for Research and Technology Hellas, Greece. Until September 2015 he
was with the Department of Digital Systems, University of Piraeus,
Greece, E-mail: sampson@iti.gr
Manuscript received 5 January 2015.
R
2 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
sional ICT competences profiling is an important element
to be considered in educational RSs and, thus, it is worthy
to investigate this topic.
In our previous work, we have proposed and evaluat-
ed a LO recommendation approach that aimed to support
teachers in identifying and selecting educational re-
sources for their course design and delivery, taking into
consideration the individual teacher’s current ICT compe-
tence status [15]. Furthermore, we have proposed and
evaluated a mechanism for eliciting Fuzzy teachers’ ICT
Competence Profile (ICT-CP) based on their behavior
patterns within Learning Object Repositories (LOR) [16].
Following these initial works, in this paper we pro-
pose and study a Recommender System that supports
teachers, in a holistic manner, to select appropriate Learn-
ing Objects from existing LO Repositories (LORs) based
on their dynamically updated ICT competence profile.
The contribution of the proposed system is twofold. First,
the RS utilizes the teachers' ICT Competence profiles,
which are automatically elicited and dynamically updat-
ed based on the teachers' continuous usage behavior pat-
terns within LORs. Second, the recommendation process
itself adopts a new method for generating the LO recom-
mendations by exploiting the previously mentioned elic-
ited ICT Competence profiles, i.e. (a) by creating an ICT
Competence-based neighborhood and (b) by promoting
the LOs that the active teacher is currently competent in
using.
The proposed system is presented and evaluated fol-
lowing a two-layer presentation in line with the common-
ly accepted layered evaluation methodology [17] of adap-
tive learning systems, which has been extended to RS in
[18]. More specifically, a separate presentation and evalu-
ation of the "Teacher ICT Competence Profile Elicitation"
Layer and the "Learning Object Recommendation Generation"
Layer is performed.
The remainder of this paper is structured as follows.
Section 2 presents the background of the present study, as
well as the identified research problem. Section 3 presents
the proposed approach to teacher ICT Competence-based
LO recommendations. This approach includes both a
method for eliciting the teachers' ICT Competence Profile
(ICT-CP) based on their usage behavior within Learning
Object Repositories as well as the manner in which these
profiles are exploited towards providing more informed
ICT Competence-based LO recommendations, compared
to existing, commonly used RS methods (described in
Section 4). Section 4 presents a detailed analysis of the
evaluation methodology employed. Section 5 presents the
evaluation results for both Layers of the proposed RS.
Finally, Section 6 contains the conclusions of the present
work and the future work in this research agenda.
2 BACKGROUND
2.1 User Profiling
User profiling is a technique that has been widely applied
in a range of software applications, including Adaptive
Web Systems [19] and Recommender Systems [20]. It in-
volves gathering data from the users in order to create a
profile for each of them, depicting their unique attributes
in the context of the system's application [21]. The latter
could involve, for example, movie genre preferences in a
movie recommender system [2], or learner preferences in
an educational RS [22].
The process of creating individual user profiles in
software applications is essential because each user has
his/her own characteristics and needs. Therefore, captur-
ing these user attributes is necessary for enabling the pro-
vision of personalized services [23]. A user profile, there-
fore, presents the system’s full interpretation of the users'
preferences and personal characteristics.
In many cases, user profiles are not provided by the
users themselves but they are being elicited automatically
through the identification, collection and processing of
relevant user actions’ data [24]. The reason for this is that
users are usually either unwilling to provide such infor-
mation or when they do, the validity and completeness of
the provided data cannot be ensured [25]. Therefore, the
elicitation process is usually based on the users' relevance
feedback data [26].
Relevance feedback data can be attained in two ways,
namely explicit feedback and implicit feedback [27]. The
former requires users to perform specific actions that will
inform and update their profile attributes, e.g., assign
ratings to items or download items. This approach pro-
vides a set of benefits for the hosting RS system, such as
increased development simplicity and enhanced accuracy
in the profile update process [28]. Users' implicit feedback
refers to mechanisms that monitor the users' interaction
with the system in an unobtrusive manner. Such ap-
proaches have been developed in order to completely
detach the user from the explicit feedback providing pro-
cess and maximize the amount of data that are being har-
vested by the system [29]. Examples of user actions that
are being monitored for profiling purposes include
browsing time in each item and type of items accessed,
uploaded or ignored [21].
2.2 User Profiling in Teacher-Oriented
Recommender Systems
In the context of Technology-enhanced Learning (TeL),
the majority of the implemented RS targets the learners
and aims to provide them with personalized learning
material and sequences of learning activities towards spe-
cific educational goal attainment [30]. The main learner
attributes that are being used in such processes include
their prior knowledge, learning preferences/styles, indi-
vidual goals and other cognitive characteristics [31]. Sub-
sequently, user profiling approaches have been primarily
focused on accommodating the attributes of the learners
[32].
However, despite the fact that learners are indeed the
main focus of learning processes, other actors are also
important for the successful implementation of effective
learning procedures. More specifically, teachers also play
a vital part in the educational processes and, amongst
other reasons, their ICT competences [12], [14] and per-
sonal attitudes towards ICT use [33] can greatly affect the
level and the quality of their technology-supported teach-
AUTHOR ET AL.: TITLE 3
ing practice. Therefore, systematic accommodation of
teachers’ professional characteristics should be an im-
portant design consideration for educational RS. Within
this context, our previous work [11] included a literature
review of existing teacher-oriented educational RS. The 22
identified approaches are summarized in Table 1. Table 1
also contains information on the types of relevance feed-
back data that are being harvested. These relevance feed-
back data include Explicit and Implicit data. The former
include Social data (e.g., ratings and bookmarks) and user
Demographic data, while the latter include, for example
number of views of LO. Moreover, Table 1 presents in-
formation on whether these data are being utilized only
for ad-hoc similarity calculations (i.e., they do not incor-
porate profile creation) or if they are being automatically
processed and exploited to create dynamic and adaptive
user profiles for providing personalized recommenda-
tions (i.e., they do incorporate profile creation).
TABLE 1
EXISTING TEACHER-ORIENTED TEL RECOMMENDER SYSTEMS
RS
Relevance Feedback Data
Profile Creation
1
[34]
Explicit: Demographic
Explicit: Social
Implicit
no
2
[35]
Explicit: Social
no
3
[36]
-
no
4
[37]
Explicit: Demographic
Explicit: Social
no
5
[38]
Explicit: Social
no
6
[39]
Explicit: Social
no
7
[40]
Implicit
yes
8
[41]
Implicit
Explicit: Social
yes
9
[42]
Explicit: Social
yes
10
[43]
-
no
11
[44]
-
no
12
[45]
Explicit: Social
no
13
[46]
Explicit: Social
no
14
[47]
Explicit: Social
no
15
[48]
-
no
16
[49]
Explicit: Social
no
17
[50]
Explicit: Social
Implicit
yes
18
[51]
Explicit: Social
no
19
[52]
Explicit: Demographic
Explicit: Social
yes
20
[53]
Explicit: Social
no
21
[54]
Explicit: Demographic
Explicit: Social
no
22
[22]
Explicit: Social
Implicit
yes
Only (completed) systems utilizing the above techniques
were considered. Table values designated as "no" either a
lack of user profiling mechanism in the corresponding
system or lack of specific presentation of relevant data in
the presenting paper.As Table 1 depicts, existing teacher-
oriented RS utilize a wide range of "raw" relevance feed-
back data types for creating user profiles. More specifical-
ly, most RS utilize "Explicit: Social" relevance feedback
data (N=17, x=77%). "Implicit" (N=5, x=23%) and "Explic-
it: Demographic" (N=4, x=18%) relevance feedback data
are used less often. However, regardeless of the type of
the relevance feedback utilized, the Table 1 data signify
that only a small portion of existing teacher-oriented RS
(N=6, x=27%) create, maintain and exploit user profiles
for providing enhanced recommendations to teachers,
based on these relevance feedback data. This can be con-
sidered as potential drawback, since systematic and effi-
cient user modeling is an essential element of successful
RSs [10].
More specifically, Shelton et al. [40] describe a system
that utilizes the teachers' clicks and time on each webpage
to detect their preferences and alter the recommendations
accordingly. Brusilovsky et al. [41] propose a social navi-
gation approach for assisting teachers to identify useful
educational resources within web-based repositories,
through manipulation of their usage history and explicit
feedback. Schoeffeger et al. [42] (and [50]) proposed a
method for identifying emergent topics that teachers are
dealing with and are interested in. Fazeli et al. [52] pro-
posed a method for collecting both demographic and so-
cial data from teachers towards utilizing them for build-
ing trust networks. Finally, Ferreira-Satler et al. [22] pro-
pose an ontology-based fuzzy teacher profile inference
system. The main idea is to elicit the preferences of the
teachers based on the semantic importance of each LO
that they create or browse. This importance is derived
from lexical analysis of the LOs and is calculated against
the existing teacher profile.
A careful analysis of the existing approaches highlights
the fact that key teachers’ individual professional capacity
characteristics, such as their aforementioned ICT Compe-
tences, are not being currently exploited for providing
personalized recommendations to support their daily
teaching practice of course design and delivery. Incorpo-
rating such information can lead to more focused recom-
mendations by forming better teacher neighborhoods and
by filtering candidate LOs to identify the most appropri-
ate for each individual teacher's ICT competences. Our
previous work [15] has presented initial evidence that
incorporating this type of teacher characteristics in the LO
recommendation process depicted in a formal manner
(i.e., the UNESCO ICT Competency Framework for
teachers [55]) and appropriately mapping them (e.g., [11])
to the specific LO metadata schema that each dataset em-
ploys (e.g., the Open Discovery Space Repository LO
Metadata Application Profile [56]) can have a significant
positive impact on the predictive accuracy of the RS.
Moreover, due to the fact that (a) competences are not
reified characteristics, but evolve over time, and (b) that
such data are rarely (much less correctly) provided by the
teachers themselves, profiling mechanisms should strive
to construct, and maintain up-to-date, highly granulated
ICT Competence profiles for teachers based on their rele-
vance feedback data [26].
In the light of the above, a research problem can be
identified, namely how can a teacher ICT Competence
profile be (a) elicited based on the implicit and explicit
relevance feedback data of the teachers and (b) utilized
4 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
for delivering more informed LO recommendations. To
the best of our knowledge, such an approach has not been
previously considered in the field of RS. Such a system
could provide a unified solution for providing personal-
ized LO recommendations to individual teachers, since it
would both automatically create and update teacher ICT
competence profiles and exploit them towards informed
LO suggestions.
The contribution of this paper, therefore, is the presen-
tation of a proposed solution towards tackling this identi-
fied research problem. The first step, namely, teachers'
ICT-CP elicitation, has already been analysed in our pre-
vious work [16]. In the context of this paper, however an
additional evaluation of this step was performed in order
to further enhance validattion of its robustness. The se-
cond step of the research problem, namely the exploita-
tion of the generated ICT-CP for enhanced LO recom-
mendations, is the original contribution of this paper.
3 ICT COMPETENCE-BASED LEARNING OBJECT
RECOMMENDATIONS FOR TEACHERS
This section presents the proposed solution to the
identified research problem, i.e. a new RS for providing
enhanced LO recommendations to teachers based on their
elicited ICT-CP. The proposed system is schematically
depicted in Fig. 1.
Fig. 1. Proposed Recommender System overview
It essentially comprises two main layers, as follows:
1. Teacher ICT Competence Profile Elicitation Lay-
er. The first Layer, which is described in detail in
Section 3.1, relates to the elicitation of the active
teacher's ICT-CP based on their relevance feed-
back data. More specifically, the system creates a
unified representation of the active teacher by
harvesting his/her "Usage Data" from the LOR.
Following that, this unified representation is fed to
the ICT-CP Elicitation mechanism and the fuzzy
ICT-CP of the teachers is generated.
2. ICT Competence-based Learning Object Rec-
ommendation Generation Layer. The second Lay-
er of the proposed RS is the LO recommendation
generation. This is performed by utilizing the out-
put data generated from the first Layer. More spe-
cifically, as it will be further described in Section
3.2, it utilizes these output data in a dual manner,
i.e., (a) for the selection of the active teacher's
neighbors, in terms of their ICT-CP similarity to
the active teacher and (b) for weighting each can-
didate LO in terms of its appropriateness for the
active teacher's ICT-CP. This is a promising ap-
proach, which was introduced in our previous
work [15] and was shown to result to more accu-
rate LO recommendations.
The presentation of the RS will be performed in two steps
in accordance to the two Layers of the RS, i.e. the teacher
ICT Competence Profile Elicitation Layer and the ICT
Competence-based LO Recommendation Generation
Layer.
3.1 Teachers' ICT Competence Profile Elicitation
Layer
3.1.1 Teacher Relevance Feedback Data
The first Layer of the proposed RS relates to the elicitation
of the teachers' ICT Competence profiles based on their
relevance feedback data from their interaction within any
LOR.
Towards tackling this issue, a specific and appropriate
set of relevance feedback data had to be selected. More
specifically, this set should provide a valid proxy of
teachers' ICT competence over the the available LO type
in the LOR. More specifically, the LO types referred to the
distinct metadata attributes which are used to character-
ize LO in LORs. It should be noted that these attributes
are flexible and can be adapted to meet the IEEE LOM
Application Profile that each LOR adopts, in line with the
official IANA MIME type extensions
(http://tinyurl.com/ol2lv34).
A set of four types of relevance feedback data (from
now on referred to as "Usage Data"), commonly used in
the literature, was selected, as follows:
1. Rating History. This category included the ratings
teachers had provided over the available LO. To
accommodate the unique rating patterns of each
teacher (e.g., some tend to rate high by default), all
ratings were first normalized by subtracting the
rating mean of the active teacher. Rating data
could provide a solid proxy of ICT competence
due to the fact that a high rating could imply that
the user has either actually used the LO or, at least,
is comfortable with it [5]. A low rating on the other
hand, was not be considered as an indicator of low
competence, because a person would probably not
rate an item low just because they would not be
able to use it.
2. Bookmarking History. In the same vein as before,
bookmarking a specific LO type repeatedly could
provide insight on the fact that the active teacher
utilizes this type of resources frequently in their
daily teaching practice. Therefore, it could provide
a solid proxy for inferring the level of relevant ICT
competence of teachers.
AUTHOR ET AL.: TITLE 5
3. Learning Object Access History. The access pat-
terns of teachers could be exploited for capturing
their ICT competences in the accessed LO types.
More specifically, if specific LO types had been re-
peatedly accessed from a teacher, they could be
regarded as commonly used by him/her. This
could infer useful information about the teacher's
ICT competence profile.
4. Learning Object Creation History. The last "Us-
age Data" category referred to active teachers'
sharing history, i.e. the number of LOs that each
teacher had uploaded to the LOR. These data
could provide insight on the teachers’ ICT Compe-
tence based on the assumption that a teacher, who
had created (and shared) a specific LO type, would
be competent in actually using them.
These "Usage Data" types were utilized by the system
for eliciting the teachers' ICT Competence profiles. More
specifically, for each teacher, this process included an
initial harvesting of all LOs that any of the "Usage Data"
were available for. This information was then fed to the
ICT Competence Profile Elicitation mechanism, towards
inferring the teachers' ICT-CP.
3.1.2. The Teacher ICT Competence Profile Elicitation
mechanism
The ICT-CP Elicitation mechanism comprised two
phases, namely the "Aggregation" Phase and the "Fuzzifi-
cation" Phase (Fig. 2).
Fig. 2. Overview of the proposed RS Layer 1
1. The Aggregation Phase. The aim of this phase is
to create a unified representation of the teachers'
interactions within the Learning Object Reposito-
ries (LOR). More specifically, for each teacher, all
four "Usage Data" information per LO metadata
attribute are merged in one "Interactions" metric.
This aggregated metric provides an overview of
the level of interaction that each teacher had with
the diverse LO metadata types, by considering all
four "Usage Data".
In order to calculate this metric, each LO metadata
attribute type received a weight representing the
level of significance that it had for each teacher,
i.e., the level of preference that the active teacher
demonstrated to this particular LO attribute type
compared to the rest within the same "Usage Data"
category.
The weights are calculated using the following
formula for each teacher :
where refers to each metadata attribute type
within the same "Usage Data" category and re-
fers to the overall cardinality of the metadata at-
tribute set (which can vary between different LOR)
within the same "Usage Data" category. The possi-
ble values for each weight range from [0 ... 1]. A
weight value of "zero" for a specific LO metadata
type signifies that the teacher has interacted with
no LO of this type regarding the specific "Usage
Data" category (e.g, no ratings provided for any
LO of type "video"). On the other hand, a weight
value of "one" for a specific LO metadata type sig-
nifies that the teacher has interacted solely with
LO of this particular type regarding a specific "Us-
age Data" category (e.g., the teacher has provided
bookmarks only for LO of type "video").
Moreover, (1) also normalizes the available "Usage
Data", since the different "Usage Data" categories
would probably not have the same cardinality of
LO (i.e., there would probably be more "Access"
data than "Bookmark" data).
Therefore, building on the weights calculated in
(1) for each LO metadata type, the combined "In-
teractions" metric is computed using the following
formula, for each teacher :
where, is the resulting "Interactions"
metric for each LO metadata attribute type , is
the cardinality of the “Usage Data” set and is
the weight of the LO metadata attribute, as calcu-
lated by (1).
Therefore, for each teacher , the "Interactions"
metric presents a unified and combinative depic-
tion of their usage pattern in a LOR, for each LO
metadata attribute type by considering all four
"Usage Data". This is done by considering together
(a) the significance that each LO metadata attrib-
ute type had to each teacher (i.e., the weights) and
(b) the number of LO per metadata attribute type
that the teacher had interacted with.
2. The Fuzzification Phase. After the completion of
the Aggregation Phase process, and the construc-
tion of the active teacher's unified "Interactions"
metric, the Fuzzification Phase is activated. This
step is responsible for inferring the level of the
teacher's ICT competence (their ICT-CP) by trans-
lating the aggregated "Interactions" metrics for
each teacher to their fuzzy equivalents. The latter
would be the final proxy of the teachers' ICT
Competence. The reason for employing fuzzy logic
is due to the fact that it is considered as an appro-
priate means of depicting data that are "vague"
and difficult to assign to crisp categories [57].
6 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
Therefore, apart from the extensive use of this ap-
proach for creating user profiles in different appli-
cation contexts [22], [58], fuzzy logic is selected
due to the "fluid" nature of competences. More
specifically, competences are not crisp characteris-
tics, but can be attained in a continuum of profi-
ciency levels and can be constantly updated [59].
The adopted fuzzification process uses a set of five
linguistic variables depicting levels of competence,
i.e., Very Low, Low, Medium, High and Very
High. The use of five linguistic variables is a typi-
cal method in the literature (e.g., [29]). Addition-
aly, a triangular membership function was uti-
lized, which is presented in Fig. 3. The latter was
used due to the fact that it is a commonly used
membership function type for depicting character-
istics similar to ours [24].
Fig. 3. Membership functions of the proposed RS
where μ(α) represents the membership function
value and it is calculated for all five linguistic var-
iables each time,
represents the "Interactions"
metric value and the
and
depict the mini-
mum and maximum marginal values for each
Fuzzy linguistic variable.
The membership func-
tion is schematically depicted in Fig. 4.
Fig. 4. Schematic Representation of the Membership func-
tion of the proposed RS
To give a brief example of the above, supposing
that a teacher had a resulting aggregated "Interac-
tions" value of "0.9" (from the Aggregation Phase
of the ICT Competence Elicitation mechanism) re-
garding a specific type of LO (e.g., "inquiry-based"
Lesson Plans), their generated fuzzy ICT-CP for
this LO type would be:
VL
L
M
H
VH
0
0
0
0.4
0.6
More specifically, this means that the particular
teacher is 40% Highly Competent in utilizing in-
quiry-based lesson plans and 60% Very Highly
competent. Therefore, upon completion of the pro-
cess of the Fuzzification Phase, each teacher's ICT-
CP has been modeled in a fuzzy manner for all LO
metadata types, based on his/her "Usage Data"
within the LOR. The utilization of fuzzy logic for
capturing and representing teachers' ICT Compe-
tences allows for more granulated construction of
profiles. In turn, this enhanced level of teacher
profile granularity can assist in providing more in-
formed recommendations [22].
The exact manner in which this is realized in the
proposed RS is described in the next section.
3.2 ICT Competence-based Recommendations
Layer
After the completion of the processes of the first Layer,
the active teacher has been assigned with a unique fuzzy
ICT Competence profile depicting their personal compe-
tences in each of the distinct LO metadata attribute types
stored in the LOR. At this stage, the second Layer of the
RS is activated, i.e. the LO recommendation generator
(Fig. 5).
Fig. 5. Overview of the proposed RS Layer 2
As aforementioned, the contribution in the proposed
approach in this Layer related to the novel manner in
which these recommendations were generated. More spe-
cifically, the elicited fuzzy ICT-CPs were utilized (a) as a
means for alternative neighbor selection and (b) as a
weighting factor for determining the appropriateness of
each candidate LO to the active teacher's level of ICT
competence.
The second Layer is divided in three phases, namely:
(a) the Neighbor Selection Phase, (b) the ICT Competence
Defuzzification Phase and (c) the Recommendation Gen-
eration Phase.
1. The Neighbor Selection Phase: this phase aims at
selecting the most suitable set of neighbors for the
active teacher based on the similarity of their
fuzzy ICT competence profiles instead of the
commonly used rating similarity. In this way, the
active teacher is provided with recommendations
based on the opinions of colleagues that have the
same ICT competence background with him/her,
and thus (s)he is recommended with LO which
will potentially be both useful and appropriate for
them.
The method employed for calculating this similari-
ty is the Euclidean Distance, which is commonly
used for similar processes [60].
2. The Defuzzification Phase: this phase aims to
translate the unified fuzzy teacher ICT-CP in a
single factor depicting the combined level of ICT
AUTHOR ET AL.: TITLE 7
competence of the teacher and incorporating all
fuzzy levels. This factor will be utilized for
weighting each candidate LO in terms of its suita-
bility for the active teacher. The method employed
for this Phase is the commonly used Center of
Gravity method [61], which computes the center of
gravity of the area under the membership func-
tion. The resulting factor of the Defuzzification
Process is utilized in the second Phase of the ICT
Competence-based Recommendations Layer of the
RS.
3. The Recommendation Generation Phase: this
phase aims to generate and deliver the ICT Com-
petence-based LO recommendations. As afore-
mentioned, the proposed recommendation meth-
od utilizes the teachers' ICT-CP profiles in a dual
manner.
First, the active teacher's neighbors are selected in
Phase 1 ("Neighbor Selection Phase"), based on the
similarity between their fuzzy ICT-CP. Second, the
defuzzified ICT-CP factor is utilized as a filtering
method of candidate LOs in terms of their appro-
priateness for the active teacher's level of compe-
tence.
Based on the above, for each LO that is being
considered for recommendation, the system as-
sesses its appropriateness score (AS) for the active
teacher t using the following formula:
where N is the amount of neighbors that had rat-
ed the specific LO, rj i is the rating provided for
LO by neighbor teacher j and wj is the Euclidean
Distance between the active teacher and neighbor
j. The Euclidean Distance is, therefore, utilized as
a weighting factor in order to assign more gravi-
ty to the opinions of "closer" neighbors. Finally,
the COGtype of (i) is the output of the defuzzification
process of the active teacher t. The resulting AS
value represents the predicted rating for the ac-
tive LO for the active teacher.
This section presented the proposed solution to the
identified research problem, i.e. a unified approach for
eliciting and exploiting teacher ICT-CP towards more
informed LO recommendations. The next section presents
the methodology employed and results generated for
evaluating the performance of the proposed system, in
terms of (a) the accuracy of the ICT-CP elicitation mecha-
nism and (b) the predictive accuracy of the RS for gener-
ating LO recommendations.
4 EVALUATION
4.1 Methodology
The evaluation methodology adopted in this paper fol-
lows the layered evaluation approach [17]. More specifi-
cally, monolithic evaluation of RS has been identified to
lead to limited remedying potential in case of low RS ac-
curacy. Essentially, this means that a potentially low ac-
curacy of the RS cannot be linked to the specific element
of the RS that is causing this level of performance. To al-
leviate this issue, the layered evaluation process proposes
separate evaluation Phases for each of the RS Layers,
namely Phase 1 refers to the evaluation of the Teacher
Profiling Layer and Phase 2 refers to the evaluation of the
ICT Competence-based Recommendation Layer. In this
way, any underperforming Layer can be highlighted and
focused remedying actions can be performed, towards
increasing the accuracy of the specific Layer, and by ex-
tension, of the overall RS.
In the context of the present study, the Phase 1 evalua-
tion focuses on the proposed system's accuracy in re-
creating existing teacher ICT-CP. More specifically, the
system generates a set of teacher ICT-CP based on their
relevance feedback data. The generated ICT-CPs will then
be contrasted against the existing ICT-CPs, which have
already been provided voluntarily by the teachers them-
selves in the context of the ODS Project. Moreover, it
should be mentioned that the existing ICT-CPs were pro-
vided by the teachers in a manual manner, using a web-
form and following the well-known UNESCO ICT Com-
petency Profile for teachers
1
. The metric used for imple-
menting this benchmark was the Jaccard co-efficient [62].
The Phase 1 evaluation has already been performed
and reported in our previous work [16]. In the context of
this paper, however, we retrieved an updated version of
the same dataset and further validated the results, to-
wards enhancing the robustness of the proposed method.
The updated evaluation results of Phase 1, as well as a
comparison to the initial evaluation results described in
[16], are discussed in Section 5.1 of this paper.
Regarding the Phase 2 evaluation, the main focus was
assessing the accuracy of the Layer 2 of the proposed ap-
proach to predict the ratings of the active teacher on LO
they had not interacted with, using the process described
in (3). This evaluation phase aims at providing evidence
on the positive added value and the increased accuracy of
the proposed RS (depicted as Fuzzy Hybrid, abbreviated
as FH) compared to existing approaches. The commonly
used Root Mean Squared Error (RMSE) metric was select-
ed for measuring the predictive accuracy [63]. This metric
is calculated based on the formula:
where is the set of generated predicted ratings for
users on items , is the set of known ratings and is
the set of users and items for which the ratings are
known. It should be noted that since this metric aims to
capture errors in the predictions of the RS, lesser values of
RMSE designate a better predictive accuracy of the RS.
As aforementioned, our proposed FH approach was
benchmarked against a set of "control" recommendation
methods, which are (a) commonly used in the literature
and (b) have been reported to provide high levels of accu-
racy. More specifically, the control set included two types
of recommendation approaches, namely user-based col-
1
http://unesdoc.unesco.org/images/0021/002134/213475e.pdf [Ac-
cessed 10 May 2015]
8 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
laborative filtering (U) and item-based collaborative filter-
ing (I) [62]. The former computes similarities between
users to find the most similar users and predicts ratings
based on how the item was rated by the most like-minded
users. The latter approach shares the same idea but it is
based on similarity between items rather than between
users. For each of these two recommendation approaches
one similarity measure was implemented, namely the
Pearson correlation coefficient (PCC) for the user-based
collaborative filtering [64] and the Adjusted Cosine corre-
lation coefficient (CCC) for the item-based collaborative
filtering [65]. Both similarity measures have been selected
due to their reported high performance in terms of pre-
dictive accuracy in the context of their specific recom-
mendation approach [64], [65].
The Pearson correlation coefficient is calculated based
on the following formula:
where is the set of items that both users and have
rated, and denote the ratings of the users and
on the item i¸ while
and
denote the average ratings
of the two users respectively.
The Adjusted Cosine correlation coefficient is calculat-
ed based on the following formula:
where is the set of users that have rated both items
and j, and denote the ratings of the user on items
and j respectively while
denotes the average ratings
of the user.
Therefore, the overall control recommendation method
set consisted of two alternatives, i.e. User-based Pearson
Correlation Coefficient (UPCC) and the Item-based Ad-
justed Cosine Correlation Coefficient (ICCC).
For all three cases (two control approaches plus the
proposed approach), the predictive accuracy evaluation
process was based on the standard "Leave-N-out" tech-
nique [66]. More specifically, this method includes split-
ting the available dataset in two sub-sets, namely the
"training" set and the "test" set. The former contained 70%
of the overall data and was used for training the RS and
generating the recommendations. The latter contained the
remaining 30% of the overall data and was used for eval-
uating the system accuracy, i.e., the recommendations
generated from the training set [62]. Finally, the evalua-
tion experiment was run for an increasing number of
neighborhood size (one through twenty) in order to mon-
itor the behavior of each approach in each occasion.
4.2 Datasets
The present study utilized three educational datasets for
supporting the two evaluation Phases, all of which were
retrieved from existing Learning Object Repositories
which are currently in use. More specifically, the LORs
used in our experiments were the Open Discovery Space
(ODS) Repository [http://portal.opendiscoveryspace.eu],
the Discover the Cosmos (DtC) Repository
[http://portal.discoverthecosmos.eu], and the Open Sci-
ence Resources (OSR) Repository
[http://www.osrportal.eu/en/repository].
The ODS dataset, an earlier version of which was used
in previous work [11], [15], [16], contained (a) existing,
manually provided teachers' ICT Competence profiles,
depicted using the UNESCO ICT Competency Frame-
work for teachers (ICT-CFT) [55], (b) LO metadata records
characterized using a specific IEEE LO Metadata Applica-
tion Profile [56] and (c) "Usage Data" of the teachers as
they were defined in Section 3.1.1. Due to the unique
availability of existing teachers' ICT-CP, the ODS dataset
was utilized for the Phase 1 evaluation process, for
benchmarking the proposed teacher ICT Competence
elicitation method. As aforementioned, this evaluation
process was initially described in [16], where it was per-
formed on an earlier instance of the ODS dataset. In the
context of the work presented in this paper, an up-to-date
version of the ODS dataset was retrieved towards further
validating these initial promising evaluation results. In
both experiments, towards the evaluation of the re-
creation accuracy of the proposed method, mapping rules
were utilized for connecting ICT Competences as they
were described in ICT-CFT and LO metadata attributes as
they were described in the IEEE LOM Application Profile
of the ODS dataset [56]. These mapping rules were pre-
sented and evaluated in previous work [11].
The ODS dataset was not utilized in the Phase 2 evalu-
ation process, in order to fully adhere to the layered eval-
uation framework adopted, namely to not only evaluate
the different layers of the proposed RS individually, but
also to utilize unique datasets for evaluating each layer.
The DtC dataset [http://portal.discoverthecosmos.eu]
and the OSR dataset [67] contained (a) LO metadata rec-
ords and (b) "Usage Data" of teachers that had been regis-
tered to the corresponding portals. The DtC and OSR da-
tasets did not contain existing teacher ICT Competence
profiles. Therefore, they were used only in the Phase 2
evaluation process, i.e. for evaluating the predictive accu-
racy of the proposed ICT Competence-based RS for
teachers, by exploiting teachers’ ICT-CP elicited by the
first Layer of the proposed RS.
An overview of all three datasets is provided in the fol-
lowing Table 2. Aggregation Level (AL) 1 LOs refer to
standalone Educational Resources (e.g., flash simulations,
educational games, text documents), while Aggregation
Level 2 LOs refer to Lesson Plans and/or Educational
Scenarios (i.e., flows of learning activities supported by
Educational Resources).
Overall Sample Size (N) refers to the total number of
LOs in each AL category. The number of unique users
refers to the cardinality of the set of teachers that had con-
tributed at least in one "Usage Data" category. The re-
maing four data categories, namely Rating Data, Access
Data, Creation Data and Bookmark Data, refer to the
number of LOs that the corresponding "Usage Data" were
provided for. As aforementioned, these four "Usage Data"
categories were exploited by the proposed RS for eliciting
the teachers' ICT-CP.
AUTHOR ET AL.: TITLE 9
TABLE 2
EDUCATIONAL DATASETS OVERVIEW
ODS Dataset
DtC Dataset
OSR Dataset
AL1
AL2
AL1
AL2
AL1
Overall Sample
Size (N)
794
519
92709
629
1545
Rating Data
240
173
835
7469
1148
Access Data
3308
392
183
708
5026
Creation Data
794
519
92709
629
1545
Bookmark Data
177
45
47
39
345
Unique Users
686
209
281
829
The OSR dataset only contained usable data for Ag-
gregation Level 1 Learning Objects. Finally, as "unique
users" for the ODS dataset were only considered those
who had contributed their ICT-CP, since the purpose of
this dataset was to evaluate the first Layer of the pro-
posed RS.
Furthermore, regarding the Phase 2 evaluation process, in
order to allow for a more informed overview of the eval-
uation results (presented in Section 5), an analysis of spe-
cific characteristics of the datasets utilized (i.e. DtC AL1,
DtC AL2 and OSR) was performed. More specifically, this
analysis was performed due to the reported high depend-
ence of all RS's performance to the dataset to which they
are fed [62],[68] and the resulting need to robustly under-
stand the shift in performance of the selected RS bench-
mark approaches between the different datasets. Based on
the works of [68] and [69], we selected a set of two dataset
characteristics that have been reported to greatly influ-
ence the level of performance of RS. These are:
1. Rating Density. It is defined as the ratio of known
ratings against the number of all possible ratings
that can be provided (i.e., ,
where U and I are the number of users and items
respectively). A logarithmic transformation was
performed in order to normalize its values. Rating
Density has been attributed with a high positive
correlation to collaborative filtering RS perfor-
mance, i.e., denser datasets can allow for more ac-
curate recommendations [68].
2. Rating Standard Deviation. It is defined as the
standard deviation of the ratings provided by the
users in the dataset. Rating Standard Deviation
has been attributed with a high negative correlation
to collaborative filtering RS performance, i.e., low-
er levels of Rating Standard Deviation can lead to
more accurate recommendations [68].
The instantiations of the abovementioned dataset char-
acteristics for the selected datasets for the Phase 2 Evalua-
tion are depicted in Table 3.
TABLE 3
EDUCATIONAL DATASET CHARACTERISTICS
DtC Dataset
OSR Dataset
AL1
AL2
AL1
Rating Density
-4,365
-1,386
-3,047
Rating Standard
Deviation
0,966
0,875
0,901
As Table 3 depicts, the densest dataset is the DtC AL2,
followed by the OSR and the DtC AL1. Therefore, based
on the conclusions of [68], we should expect that evalua-
tion result accuracy in each of the three datasets should
follow the same correspondingly descreasing order. This
intuition is supported by the Rating Standard Deviation
data, which (given its negative correlation to RS accuracy)
also signify the same correspondingly descreasing order
in terms of expected evaluation accuracy.
Table 4 presents an overview of the evaluation meth-
odology and datasets described in Sections 4.1 and 4.2.
TABLE 4
OVERVIEW OF EVALUATION METHODOLOGY
RS Layer
Evaluation
Dataset
Benchmark
Evaluation
Focal Point
Evaluation
Metric
Layer 1
ODS
Existing
teacher ICT-
CP
ICT-CP
Re-creation
accuracy
Jaccard
Coefficient
Layer 2
DtC,
OSR
UPCC
Rating
Predictive
Accuracy
RMSE
ICCC
The following section presents the results for Phase 1 and
Phase 2 of the evaluation methodology described in this
section.
5 RESULTS
Following the layered evaluation approach, the results for
each of the two Layers of the proposed RS are presented
separately in the following sections.
5.1 Teacher ICT Competence Profile Elicitation
Method Evaluation
As aforementioned, an initial evaluation of the Phase 1
was conducted in our previous work [16]. These prelimi-
nary evaluation results (Experiment #1) are presented in
Fig. 6 along with the new Phase 1 evaluation results gen-
erated from the updated version of the ODS dataset (Ex-
periment #2). Both sets of results provide evidence of the
proposed method's high efficiency to elicit the ICT-CP of
teachers based on their "Usage Data".
More specifically, for Educational Resources (AL1
LOs) the proposed system re-created the existing ICT-CP,
which, as aforementioned, where provided by the teach-
ers themselves in a manual manner through a web-form,
with 70% accuracy in the initial Experiment #1 and with
73% accuracy in the updated Experiment #2. For Lesson
Plans (AL2 LOs) the ICT-CP re-creation accuracy level
was 81% in the initial evaluation Experiment #1 and 79%
accuracy in the new evaluation Experiment #2.
Regarding the evaluation results' accuracy levels in
their own regard, they are very promising for both evalu-
ation experiments. The reported deviations from the orig-
inal, user-provided ICT-CPs can be mainly attributed to
the prominent reason that teachers could have provided
incorrect descriptions of their ICT competences to begin
with.
10 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
Fig. 6. Accuracy results for the two evaluation experiments of the
Phase 1 of the teacher ICT Competence Elicitation Mechanism
This is a very common issue in the user profiling litera-
ture, where users provide incomplete or false profiling
data [21], [25]. Therefore, the manually-provided data
would not always correctly represent the current level of
ICT competence of the teacher that provided them, as
opposed to the automatic elicitation method, which cap-
tures the teachers' actual usage patterns in using LOs and
can therefore provide a more solid indicator of preference
and competence [29]. Nonetheless, the re-creation accura-
cy results are significantly high in all experiments and for
both Aggregation Levels despite this known shortcoming.
Moreover, regarding the result differences between
Experiment #1 and Experiment #2, despite their small
size, they could be attributed to the updated sets of “Us-
age Data" that the ODS dataset contained. These potential
data alterations could have triggered these minor accura-
cy changes of the system, since the actual teachers’ usage
patterns could have been slightly altered since the initial
version, without a corresponding manual change in the
ICT-CP by the teacher. In any case, however, as Fig. 6
depicts, the evaluation result deviations between the two
Experiments are very small and the absolute evaluation
results are consistent between Experiments, indicating the
proposed elicitation method's robustness.
Overall, the evaluation results for the first Layer of the
proposed system were promising and provided with evi-
dence that the teacher ICT-CP Elicitation Method could
be utilized within Learning Object Repositories, in order
to elicit the teachers ICT-CP. As aforementioned, this in-
formation could allow for enhanced LO recommenda-
tions. The evaluation results for the latter, which repre-
sents the second Layer of the proposed system, are pre-
sented in the following section.
5.2 ICT Competence-based Learning Object
Recommendations Evaluation
The second Phase in the layered evaluation of the pro-
posed approach to LO recommendations focused on the
added value it could provide in terms of predictive accu-
racy, by utilizing the generated teacher ICT-CP from the
first Layer. As aforementioned, the DtC (for AL1 and AL2
LO) and OSR (for AL1 LO) datasets were utilized in this
process. The evaluation results for each dataset will be
presented in separate.
1. Discover the Cosmos Dataset. Fig. 7 and 8 depict
the evaluation results of the DtC dataset for the
two Aggregation Levels of Learning Objects that
the dataset contained. As the Fig. 7 depicts, the
proposed approach outperfoms both control rec-
ommendation methods for the AL1 LOs.
Fig. 7. Preditive accuracy evaluation results for AL1 Learn-
ing Objects in the DtC dataset.
Even though the accuracy of the two benchmark
methods increases considerably as the neighbor-
hood size (NS) grows, they are outperformed by
the proposed FH approach for all NS values.
Moreover, in the user-based control method, the
threshold neighborhood size is around 10, mean-
ing that increasing the neighbors above that
threshold does not offer significant increase in ac-
curacy.
This is due to the fact that relatively few ratings
and users were available in the DtC dataset for the
AL1 LOs. Therefore, the user neighborhoods had
limited data to be built on. This is closely related
to the influence of dataset characteristics described
in Section 4.2, and more specifically, the Rating
Density [68]. For the ICCC method this occurs at a
smaller degree, since, regarding candidate neigh-
bors at least, there is a larger pool to choose from
(i.e., the AL1 LOs of the dataset). The ICCC accu-
racy (as well as the proposed FH's accuracy),
therefore, continues to improve for a larger NS
span, and begins to plateu at around 15 neighbors.
Therefore, the findings from this dataset conclude
that the proposed FH approach provides more ac-
curate LO recommendations, throughout the NS
AUTHOR ET AL.: TITLE 11
span.
Regarding AL2 LOs (Fig. 8), the proposed ap-
proach similarly outperfoms both control recom-
mendation methods for all NS values. Moreover, it
is important to notice that all three methods' pre-
dictive accuracy is considerably increased, com-
pared to their AL1 equivalents.
Fig. 8. Preditive accuracy evaluation results for AL2 Learn-
ing Objects in the DtC dataset
The reason for this universal performance im-
provement can be attributed to the dataset charac-
teristics of the DtC AL2 dataset and its comparison
to its AL1 equivalent. More specifically, as Table 3
depicted, the DtC AL2 dataset was much denser
(Rating Density) and had a lesser value of Rating
Standard Deviation. Therefore, building on (and
confirming) the findings of [68], we mainly attrib-
ute the difference of the results of Fig. 7 and Fig. 8
(and the enhanced predictive accuracy of the RS in
the DtC AL2 dataset), to these two dataset charac-
teristics' values.
Moreover, the significant improvement of Rating
Density (compared to the DtC AL1 dataset) in-
creased the NS threshold that provided better pre-
dictive accuracy for all three recommendation
methods. More specifically, the NS threshold for
the UPCC control method was increased to
around 17 and for the proposed FH, as well as for
the ICCC control method, the NS threshold is
around 18.
Overall, despite the significant improvement in all
three RS methods in the DtC AL2 datasets (com-
pared to the DtC AL1), the proposed FH still out-
performed the two benchmark methods and, es-
sentially, delivered more accurate LO recommen-
dations.
2. Oper Science Resources Dataset. As aforemen-
tioned, the OSR dataset only contained usable data
for AL1 LOs. The accuracy evaluation results are
presented in Fig. 9.
Fig. 9. Preditive accuracy evaluation results for AL1 Learn-
ing Objects in the OSR dataset
As the Fig. 9 depicts, the proposed approach again
ourperforms both benchmark methods for all
neighborhood sizes.
Moreover, the predictive accuracy of all approach-
es is decreased compared to the DtC AL2 dataset,
but are better that the DtC AL1 dataset. This find-
ing can be explained by considering the dataset
characteristics presented in Table 3. More specifi-
cally, as the Table 3 data depict, the values for both
OSR dataset charasteristics (i.e., Rating Density
and Rating Standard Deviation) are better that the
DtC AL1 and worse than the DtC AL2. Therefore,
the predictive accuracy results depicted in Fig. 9
(and their connection to the results of Fig. 7 and
Fig. 8) can be attributed to these facts. Regarding
the NS thresholds, the results follow a similar pat-
tern, i.e., they are increased compared to the DtC
AL1 dataset and decreased compared to the DtC
AL2 dataset. More specifically, the UPCC control
method plateus at around 16 neighbors, and the
ICCC control method, as well as the proposed FH
approach plateu at around 17 neighbors.
Therefore, the OSR dataset re-validated the prom-
ising results of the two DtC datasets, i.e. that the
proposed FH approach can deliver more accurate
LO recommendations to teachers, for all NS.
Overall, a consolidated overview of the predictive ac-
curacy of the proposed FH approach provides with evi-
dence of the added value that it can offer. More specifical-
ly, the reported promising results from Phase 2 evalua-
tion experiments in the DtC (for both AL1 and AL2 LOs)
and OSR datasets show that the proposed approach can
deliver more accurate LO recommendations to teachers
based on their elicited ICT Competence profiles. This is in
line with the findings of our previous work [15] which
presented initial evidence that incorporating (existing)
teachers' ICT-CP in the LO recommendation process can
provide with increased accuracy and, essentially, better
results.
12 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
6 CONCLUSIONS AND FUTURE WORK
This paper introduced and evaluated a new RS for
providing LO recommendations to teachers based on
their ICT-CP elicited from their relevance feedback data.
More specifically, the proposed system was divided in
two Layers, namely (a) the teacher ICT Competence Pro-
file Elicitation Layer, which was targeted at eliciting and
constructing the teachers' ICT-CP based on their rele-
vance feedback data, and (b) the ICT Competence-based
LO Recommendation Layer, which utilized the elicited
teacher ICT-CP from Layer 1 for providing more in-
formed LO recommendations.
The RS evaluation followed the same layered approach
and provided with evaluation results for each system
Layer. Regarding the Layer 1 evaluation, the expreriment
presented in this paper was the second one being con-
ducted and the consistency between the two Experiments
provides with additional evidence of the proposed elicita-
tion's method robustness. Overall, both individual Layer
evaluations, as well as in combination (as a holistic sys-
tem), provided with evidence that the proposed approach
can generate LO recommendations to teachers, which are
personalized to their level of ICT competence. Moreover,
the increased granularity offered by the fuzzy depiction
of ICT competences was shown to have provided an add-
ed value in the overall RS performance.
Therefore, the evaluation results show that the pro-
posed RS has the potential to be used for assisting teach-
ers in their everyday tasks of course design, lesson plan-
ning as well as their implementation and delivery, by
facilitating the process of selecting and retrieving appro-
priate LOs that match the teachers' ICT competence lev-
els.
Future work in this research agenda could include the
investigation in a granulated manner of the level of influ-
ence that the different levels of teachers’ ICT competence
can have on the predictive accuracy of the proposed sys-
tem. Moreover, it would be interesting to investigate the
potential of adaptive neighbor selection. More specifical-
ly, this strand will aim to extend the work presented in
this paper which included selecting the active teacher
neighbors based on their ICT-CP similarity. The next step
will be to select neighbors in an adaptive manner depend-
ing on (a) the specific metadata attribute type of candi-
date LO to be recommended and (b) the specific ICT
competences of the candidate neighbors in this particular
LO type. By performing this dynamic neighborhood crea-
tion, the "gravity" of each neighbor in the recommenda-
tion process will be properly adjusted to reflect not only
their similarity to the active teachers' ICT-CP, but also
their "credibility" to recommend this particular LO
metadata attribute type. Put simply, from the initial pool
of ICT competence-based neighbors (proposed in the pre-
sent paper), those that are more highly competent in the
particular LO type (that is to be recommended) will have
an increased "gravity" of opinion.
Finally, further research should be focused on incorpo-
rating additional contextual factors in the LO recommen-
dation process for teachers, such as the ICT Competences
of their school [70], [71]. The reason for this is that such
contextual factors (e.g., school infrastructure and school
leader ICT competences) have been repeatedly reported
as significant in the level of uptake of ICT in the formal
learning processes [72]. Therefore, accommodating them
within the LO selection process would potentially extend
the scope of the proposed RS and improve its capacity to
facilitate teachers in selecting appropriate LOs for their
unique context of work and, therefore, to optimize their
instructional design and delivery efforts.
ACKNOWLEDGMENTS
The work presented in this paper has been partially supported by (a)
the “Open Discovery Space: A socially-powered and multilingual
open learning infrastructure to boost the adoption of eLearning Re-
sources” Project that is funded by the European Commission's CIP-
ICT Policy Support Programme (Project Number: 297229) and (b)
“Inspiring Science: Large Scale Experimentation Scenarios to Main-
stream eLearning in Science, Mathematics and Technology in Prima-
ry and Secondary Schools” Project that is funded by the European
Commission's CIP-ICT Policy Support Programme (Project Number:
325123). Moreover, the authors would like to acknowledge the Dis-
cover the Cosmos and the Open Science Resources Projects for
providing the corresponding educational datasets.
REFERENCES
[1] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez,
"Recommender Systems Survey," Knowledge-Based Systems, vol. 46,
pp. 109-132, 2013.
[2] J. Bobadilla, F. Ortega, A. Hernando, and J. Alcala, "Improving
Collaborative Filtering Recommender System Results And
Performance Using Genetic Algorithms," Knowledge-Based Systems,
vol. 46, no. 8, pp. 1310-1316, 2011.
[3] Z. Huang, D. Zeng, and H. Chen, "A Comparison Of Collaborative-
Filtering Recommendation Algorithms For e-Commerce," IEEE
Intelligent Systems, vol. 22, no. 5, pp. 68-78, 2007.
[4] N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, "Introduction
and Background," Recommender Systems for Learning, N.
Manouselis, H. Draschler, K. Verbert, and E. Duval, eds., New York:
Springer, pp. 1-20, 2013.
[5] P. Lops, M. de Gemmis, and G. Semeraro, "Content-Based
Recommender Systems: State Of The Art And Trends," Recommender
systems handbook, F. Ricci, L. Rokach, B. Shapira and P.B Kantor,
eds., US: Springer, pp. 73-105, 2011.
[6] J.B. Schafer, D. Frankowski, J. Herlocker, and S. Sen, "Collaborative
Filtering Recommender Systems," The adaptive web, P. Brusilovsky,
A. Kobsa, W. Nejdl, eds., Berlin Heidelberg: Springer, pp. 291-324,
2007.
[7] R. Burke, "Hybrid Recommender Systems: Survey And
Experiments," User modeling and user-adapted interaction, vol. 12,
no. 4, pp. 331-370, 2002.
[8] R. Burke, "Hybrid Web Recommender Systems, " The adaptive web,
P. Brusilovsky, A. Kobsa, W. Nejdl, Eds., Berlin Heidelberg:
Springer, pp. 377-408, 2007.
[9] L. Marin, D. Isern, and A. Moreno, "Dynamic Adaptation of
Numerival Attributes in a User Profile," Applied intelligence, vol. 39,
no. 2, pp. 421-437, 2013.
[10] G. Adomavicius, and A. Tuzhilin, "Toward The Next Generation Of
Recommender Systems: A Survey Of The State-Of-The-Art And
Possible Extensions," Knowledge and Data Engineering, IEEE
Transactions on, vol. 17, no. 6, pp. 734-749, 2005.
[11] S. Sergis, P. Zervas, and D. G. Sampson, "Towards Learning Object
Recommendations based on Teachers' ICT Competence Profiles,"
Proc. of the 14th IEEE International Conference on Advanced
Learing Technologies (ICALT2014), pp. 534-538, 2014.
AUTHOR ET AL.: TITLE 13
[12] Y. Goktas, N. Gedik, and O. Baydas, "Enablers and barriers to the use
of ICT in primary schools in Turkey: A comparative study of 2005–
2011," Computers & Education, vol. 68, pp. 211-222, 2013.
[13] M. Solar, J. Sabattin and V. Parada, "A Maturity Model for Assessing
the Use of ICT in School Education," Educational Technology &
Society, vol. 16, no.1, pp. 206–218, 2013.
[14] G. Sang, M. Valcke, J. van Braak, and J. Tondeur, "Student teachers’
thinking processes and ICT integration: Predictors of prospective
teaching behaviors with educational technology," Computers &
Education, vol. 54, no. 1, pp. 103-112, 2010.
[15] S. Sergis, P. Zervas, and D.G. Sampson, "ICT Competence-Based
Learning Object Recommendations For Teachers," Proc. of the IADIS
11th International Conference on Cognition and Exloratory Learning
in Digital Age (CELDA2014), pp. 150-157, 2014.
[16] S. Sergis, and D.G. Sampson, "Eliciting Teachers' ICT Competence
Profiles Based On Usage Patterns Within Learning Object
Repositories," Proc. of the 6th IEEE International Conference on
Technology in Education 2014 (T4E2014), pp. 99-105, 2014.
[17] P. Brusilovsky, C. Karagiannidis, and D. Sampson, "Layered
Evaluation of Adaptive Learning Systems," International Journal of
Continuing Engineering Education and Lifelong Learning, Special
issue on Adaptivity in Web and Mobile Learning Services, vol. 14, no.
4/5, pp. 402-421, 2004.
[18] N. Manouselis, C. Karagiannidis, and D. Sampson, "Layered
Evaluation in Recommender Systems: A Retrospective Assessment,"
Computer Science and Information Systems, vol. 10, no.1, pp 109-
127, 2014.
[19] P. Brusilovsky, and E. Millan, "User Models for Adaptive
Hypermedia and Adaptive Educational Systems," The Adaptive Web
Lecture Notes in Computer Science Volume 4321, P. Brusilovsky, A.
Kobsa, and W. Nejdl, eds., Berlin Heidelberg: Springer, pp. 3-53,
2007.
[20] F. Ricci, L. Rokach, and B. Shapira, "Introduction To Recommender
Systems Handbook," Recommender Systems Handbook, F. Ricci, L.
Rokach, B. Shapira, and P.B. Kantor, eds., US: Springer, pp. 1-35,
2011.
[21] S. Schiaffino, and A. Amandi, "Intelligent User Profiling," Artificial
Intelligence An International Perspective, M. Bramer, ed., Berlin
Heidelberg : Springer, pp. 193-216, 2009.
[22] M. Ferreira-Satler, F.P. Romero, V.H. Menendez-Dominguez, A.
Zapata, and M.E. Prieto, "Fuzzy Ontologies-Based User Profiles
Applied To Enhance E-Learning Activities," Soft Computing, vol. 16,
no. 7, pp. 1129-1141, 2013.
[23] X. Qi, "Research On User Profiling Technology For Personalized
Demands," Proc. of the 2010 International Conference on Intelligent
Computation Technology and Automation, vol. 3, pp. 198-201, 2010.
[24] M.Y.H. Al-Shamri, and K.K. Bharadwaj, "Fuzzy-Genetic Approach
To Recommender Systems Based On A Novel Hybrid User Model,"
Expert systems with applications, vol. 35, no. 3, pp. 1386-1399, 2008.
[25] M. Belk, E. Papatheocharous, P. Germanakos, and G. Samaras,
"Modeling Users On The World Wide Web Based On Cognitive
Factors, Navigation Behavior And Clustering Techniques," Journal of
Systems and Software, vol. 86, no. 12, pp. 2995-3012, 2013.
[26] L. Marin, A. Moreno, and D. Isern, "Automatic Preference Learning
On Numeric And Multi-Valued Categorical Attributes," Knowledge-
Based Systems, vol. 56, pp. 201-215, 2014.
[27] M. de Gemmis, L. Iaquinta, P. Lops, C. Musto, F. Narducci, and G,
Semeraro, "Learning Preference Models In Recommender Systems,"
Preference Learning, F. Fürnkranz, and E. Hüllermeier, eds.,
Springer Berlin Heidelberg, pp. 387-407, 2011.
[28] M. Zanker, and M. Jessenitschnig, "Case-Studies On Exploiting
Explicit Customer Requirements In Recommender Systems," User
Modeling and User-Adapted Interaction, vol. 19, no. 1-2, pp. 133-
166, 2009.
[29] L. Marin, D. Isern, A. Moreno, and A. Valls, "On-Line Dynamic
Adaptation Of Fuzzy Preferences," Information Sciences, vol. 220, pp.
5-21, 2013.
[30] N. Manouselis, H. Drachsler, K. Verbert, and E. Duval, "TEL As A
Recommendation Context," Recommender Systems for Learning, N.
Manouselis, H. Drachsler, K. Verbert, and E. Duval, eds. New York:
Springer, pp. 21-36, 2013.
[31] K. Chrysafiadi, and M. Virvou, "Student Modeling Approaches: A
Literature Review For The Last Decade," Expert Systems with
Applications, vol. 40, no. 11, pp. 4715-4729, 2013.
[32] S. Garcia-Martinez, and A. Hamou-Lhadj, "Educational
Recommender Systems: A Pedagogical-Focused Perspective,"
Multimedia Services in Intelligent Environments, G.A. Tsihrintzis, M.
Virvou and L.C. Jain, eds., International Publishing: Springer, pp.
113-124, 2013.
[33] J. Tondeur, M. Cooper, and C.P. Newhouse, "From ICT Coordination
To ICT Integration: A Longitudinal Case Study," Journal of
Computer Assisted Learning, vol. 26, no. 4, pp. 296–306, 2010.
[34] A. Zapata, V.H., Menéndez, M.E. Prieto, and C. Romero, "A
Framework For Recommendation In Learning Object Repositories:
An Example Of Application In Civil Engineering," Advances in
Engineering Software, vol. 56, pp. 1-14, 2013.
[35] N. Manouselis, R. Vuorikari, and F. Van Assche, "Collaborative
Recommendation of E‐Learning Resources: An Experimental
Investigation," Journal of Computer Assisted Learning, vol. 26, no. 4,
pp. 227-242, 2010.
[36] C. Limongelli, A. Miola, F. Sciarrone, and M. Temperini, "Supporting
Teachers to Retrieve and Select Learning Objects for Personalized
Courses in the Moodle_LS Environment," Proc. of the 12th
International Conference Advanced Learning Technologies
(ICALT2012), pp. 518-520, 2012.
[37] J. Bozo, R. Alarcón, and S. Iribarra, "Recommending Learning
Objects According to a Teachers’ Contex Model". Sustaining TEL:
From Innovation to Learning and Practice, M. Wolpers, P.A.
Kirschner, M. Scheffel, S. Lindstaedt, and V. Dimitrova, eds., Berlin
Heidelberg: Springer, pp. 470-475, 2010.
[38] A. Walker, M.M. Recker, K. Lawless, and D. Wiley, "Collaborative
Information Filtering: A Review And An Educational Application,"
International Journal of Artificial Intelligence in Education, vol. 14,
no. 1, pp. 3-28, 2004.
[39] S. Rafaeli, M. Barak, Y. Dan-Gur, and E. Toch, "QSIA–A Web-Based
Environment For Learning, Assessing And Knowledge Sharing In
Communities," Computers & Education, vol. 43, no. 3, pp. 273-289,
2004.
[40] B.E. Shelton, J. Duffin, Y. Wang, and J. Ball, "Linking Open Course
Wares And Open Education Resources: Creating An Effective Search
And Recommendation System," Procedia Computer Science, vol. 1,
no. 2, pp. 2865-2870, 2010.
[41] P. Brusilovsky, L.N. Cassel, L.M. Delcambre, E.A. Fox, R. Furuta, D.
D. Garcia, ... and M. Yudelson, "Social Navigation For Educational
Digital Libraries," Procedia Computer Science, vol. 1, no.2, pp. 2889-
2897, 2010.
[42] K. Schoefegger, P. Seitlinger, and T. Ley, "Towards A User Model
For Personalized Recommendations In Work-Integrated Learning: A
Report On An Experimental Study With A Collaborative Tagging
System," Procedia Computer Science, vol. 1, no. 2, pp. 2829-2838,
2010.
[43] Y. Wang, and K. Sumiya, "Semantic Ranking Of Lecture Slides
Based On Conceptual Relationship And Presentational Structure,"
Procedia Computer Science, vol. 1, no. 2, pp. 2801-2810, 2010.
[44] V.A. Zaldivar, and D. Burgos, "Meta-Mender: A Meta-Rule Based
Recommendation System For Educational Applications," Procedia
Computer Science, vol. 1, no. 2, pp. 2877-2882, 2010.
[45] H. Avancini, and U. Straccia, "User Recommendation For
Collaborative And Personalised Digital Archives," International
Journal of Web Based Communities, vol. 1, no.2, pp. 163-175, 2005.
[46] S. El Helou, C. Salzmann, and D. Gillet, "The 3A Personalized,
Contextual and Relation-based Recommender System," Journal of
Universal Computer Science, vol. 16, no.16, pp. 2179-219, 2010.
[47] K.H. Tsai, T.K. Chiu, M.C. Lee, and T.I. Wang, "A Learning Objects
Recommendation Model Based On The Preference And Ontological
Approaches," Proc. of the Sixth International Conference on
Advanced Learning Technologies, pp. 36-40, 2006.
[48] G. Sielis, C. Mettouris, A. Tzanavari, and G. Papadopoulos, "Context-
Aware Recommendations Using Topic Maps Technology For The
Enhancement Of The Creativity Process," Educational Recommender
Systems and Technologies: Practices and Challenges, O. Santos and
J. Boticatio, eds. Hershey, PA: IGI-Global, pp. 43-66, 2012.
14 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID
[49] M. García-Valdez, and B. Parra, "A Hybrid Recommender System
Architecture for Learning Objects," Evolutionary Design of Intelligent
Systems in Modeling, Simulation and Control, O. Castillo and W.
Pedrycz, eds., Berlin Heidelberg: Springer, pp. 205-211, 2009.
[50] R. Schirru, S. Baumann, M. Memmel, and A. Dengel, "Extraction of
Contextualized User Interest Profiles in Social Sharing Platforms,"
Journal of Universal Computer Science, vol. 16, no.16, pp. 2196-
2213, 2010.
[51] H. Drachsler, D. Pecceu, T. Arts, E. Hutten, L. Rutledge, P. Van
Rosmalen, ... and R. Koper, "Remashed–Recommendations For
Mash-Up Personal Learning Environments," Learning in the synergy
of multiple disciplines, U. Cress, V. Dimitrova and M. Specht, eds.,
Berlin Heidelberg: Springer, pp. 788-793, 2009.
[52] S. Fazeli, H., Drachsler, F., Brouns, and P. Sloep, "Towards a Social
Trust-aware Recommender for Teachers," Recommender Systems for
Technology Enhanced Learning, N. Manouselis, H. Drachsler, K.
Verbert and O.C. Santos, eds., New York: Springer, pp. 177-194,
2014.
[53] J. Dron, R. Mitchell, P. Siviter, and C. Boyne, "Cofind—An
Experiment In N-Dimensional Collaborative Filtering," Journal of
Network and Computer Applications, vol. 23, no. 2, pp. 131-142,
2000.
[54] C. Cobos, O. Rodriguez, J. Rivera, J. Betancourt, M. Mendoza, E.
Leon, and E. Herrera-Viedma, "A Hybrid System Of Pedagogical
Pattern Recommendations Based On Singular Value Decomposition
And Variable Data Attributes," Information Processing &
Management, vol. 49, no. 3, pp. 607-625, 2013.
[55] P. Zervas, K. Chatzistavrianos, and D. Sampson, "Towards Modeling
Teachers’ ICT Competence Profile in Europe," in R. Huang, Kinshuk
and J.K. Price eds., ICT in Education in Global Context, International
Publishing: Springer, pp. 163-184, 2014.
[56] N. Athanasiadis, S. Sotiriou, P. Zervas, and D.G. Sampson, "The
Open Discovery Space Portal: A Socially-Powered and Open
Federated Infrastructure," Digital Systems for Open Access to Formal
and Informal Learning, D.G. Sampson, D. Ifenthaler, J.M. Spector, P.
Isaias eds., International Publishing: Springer, pp. 11-23, 2014.
[57] T.-C. Hsieh, T.-I. Wang, C.-Y. Su, and M.-C. Lee, "A Fuzzy Logic-
Based Personalized Learning System For Supporting Adaptive
English Learning," Educational Technology & Society, vol. 15, no. 1,
pp. 273–288, 2012.
[58] D. Anand, and B.S. Mampilli, "Folksonomy-Based Fuzzy User
Profiling For Improved Recommendations," Expert Systems with
Applications, vol. 41, no. 5, pp. 2424-2436, 2014.
[59] D. Sampson, and D. Fytros, "Competence Models In Technology-
Enhanced Competence-Based Learning," Handbook on Information
Technologies for Education and Training, H.H. Adelsberger, Kinshuk
and J.M. Pawlowski, eds. Berlin Heidelberg: Springer, pp. 155-177,
2008.
[60] L. Candillier, F. Meyer, and M. Boullé, "Comparing State-Of-The-
Art Collaborative Filtering Systems," Machine Learning and Data
Mining in Pattern Recognition, P. Perner ed., Berlin Heidelberg:
Springer, pp. 548-562, 2007.
[61] W.V. Leekwijck and E.E. Kerre, "Defuzzification: Criteria And
Classification," Fuzzy sets and systems, vol. 108, no. 2, pp. 159-178,
1999.
[62] K. Verbert, H. Drachsler, N. Manouselis, M. Wolpers, R. Vuorikari,
and E. Duval, "Dataset-Driven Research For Improving
Recommender Systems For Learning," Proc. of the 1st International
Conference on Learning Analytics and Knowledge, pp. 44-53, 2011.
[63] G. Shani, and A. Gunawardana, "Evaluating Recommendation
Systems," Recommender systems handbook, F. Ricci, L. Rokach, B.
Shapira and P.B Kantor, eds., US: Springer, pp. 257-297, 2011.
[64] J. Herlocker, J.A. Konstan and J. Riedl, ―An Empirical Analysis Of
Design Choices In Neighborhood-Based Collaborative Filtering
Algorithms,‖ Information Retrieval, vol. 5, no. 4, pp. 287–310, 2002.
[65] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, "Item-Based
Collaborative Filtering Recommendation Algorithms," ACM Proc. of
the 10th international conference on World Wide Web, pp. 285-295,
2001.
[66] J.L. Herlocker, J.A. Konstan, L.G. Terveen, and J.T. Riedl,
"Evaluating Collaborative Filtering Recommender Systems," ACM
Transactions on Information Systems, vol. 22, no. 1, pp. 5-53, 2004.
[67] D.G. Sampson, P. Zervas, and S. Sotiriou, "Learning Object
Repositories for Science Education: The Case of the Open Science
Resources Repository," 11th IEEE International Conference
Advanced Learning Technologies (ICALT2011), pp. 358-359, 2011.
[68] G. Adomavicius, and J. Zhang, "Impact Of Data Characteristics On
Recommender Systems Performance," ACM Transactions on
Management Information Systems, vol 3, no. 1, pp. 3:1-3:17, 2012.
[69] J, Bobadilla, and F. Serradilla, "The Effect of Sparsity on
Collaborative Filtering Metrics," Proc. of the Twentieth Australasian
Conference on Australasian Database-Volume 92, pp. 9-18, 2009.
[70] S. Sergis, I. Sholla, P. Zervas, and D.Sampson, "Supporting School
ICT Uptake: The ASK School ICT Competence Management
System," Proc. of the International Conference on Interactive Mobile
Communication Technologies and Learning, pp. 359 - 363, 2014.
[71] S. Sergis, P. Zervas, and D. Sampson, ―A Holistic Approach For
Managing School ICT Competence Profiles Towards Supporting
School ICT Uptake,‖ International Journal of Digital Literacy and
Digital Competence, vol 5, no 4, pp. 33-46, 2014.
[72] S. Sergis, and D.G. Sampson, "From Teachers’ to Schools’ ICT
Competence Profiles," Digital Systems for Open Access to Formal
and Informal Learning, D. Sampson, D. Ifenthaler, J. M. Spector and
P. Isaias, Eds., International Publishing: Springer, pp. 307-327, 2014.
Stylianos Sergis received a B.Sc. in "Informatics
and Telecommunications" (June 2010) from the
Department of Informatics and
Telecommunications of the National and
Kapodistrian University of Athens, Greece and a
M.Sc. in "Informatics in Education" (June 2012)
from the Faculty of Primary Education of the
National and Kapodistrian University of Athens,
Greece. Since July 2013 he is a Ph.D. student at the Department of
Digital Systems, University of Piraeus. His PhD research focuses on
Recommender Systems for Technology enhanced Education. He is
an IEEE Student Member.
Demetrios G Sampson (SM’04), has received a Diploma in Electrical
Engineering from the Democritus University of
Thrace, Greece in 1989 and a Ph.D. in Electronic
Systems Engineering from the University of Essex,
UK in 1995. From October 2015 he is a Professor of
Learning Technologies at the School of Education,
Curtin University, Western Australia. He was a
Professor of Digital Systems for Learning and
Education at the Department of Digital Systems, University of Pirae-
us, Greece (2003-2015) and a Research Fellow at the Information
Technologies Institute (ITI), Centre of Research and Technology
Hellas (CERTH) (since 2000). He is the Founder and Director of the
Advanced Digital Systems and Services for Education and Learning
(ASK) since 1999. His main scientific interests are in the area of
Learning Technologies. He is the co-author of 345 publications in
scientific books, journals and conferences with at least 1620 known
citations (h-index: 21). He has received 8 times Best Paper Award in
International Conferences on Advanced Learning Technologies. He
is Co-Editor-in-Chief of the Educational Technology and Society
Journal (5-year impact factor 1.34). He has served or serves as Mem-
ber of the Steering Committee of the IEEE Transactions on Learning
Technologies, Member of the Advisory Board of the Journal of King
Saud University - Computer and Information Sciences and the Inter-
national Journal of Digital Literacy and Digital Competence, Mem-
ber of the Editorial Board of 23 International/National Journals and
a Guest Co-Editor in 28 Special Issues of International Journals. His
participation in the organization of scientific conferences involves:
General and/or Program Committee Chair in 40 International Con-
ferences, Program Committees Member in 400 Internation-
al/National Scientific Conferences. He has been a Keynote/Invited
Speaker in 65 International/National Conferences. He has been pro-
ject director, principle investigator and/or consultant in 65 R&D
projects with external funding at the range of 14 Million € (1991-
2016). He is a Senior and Golden Core Member of IEEE and was the
elected Chair of the IEEE Computer Society Technical Committee on
Learning Technologies (2008-2011). He is the recipient of the IEEE
Computer Society Distinguished Service Award (July 2012).
