Critical Thinking through a Reflexive Platform

Full text

!"#$%#&'(%)*'+,*-'./*-0%1/*2+,+*3+%/*%%
)*2/,4-'./*%5+3(*/0/67%8-9+:%;.6(+,%<:=3-'./*%-*:%5,-.*.*6%>)5;<5?%
)5;<5%!"#$%
@A,.0%!BC!$D%!"#$D%E0(-/D%F/,'=6-0%
(''AGHHIIIJ.'(+'JK/=*J+:=J',H%
!"""#$%&%'()#*+,-./0#$1234564-ART#
!9:*0#978-1-5386-4623-6#
2/(@+A.@#-B0#
:()%CDAD#8EDF./GD&B#
H.I%/&,.E&#(J#"'.A&/DA%'#K#"'.A&/(EDAG#"E)DE../DE)#
<><>?#:.-.LM#!G&%E-+'M#N+/L.B#
N.'0O;P7?3?7<5;=>36#1%Q0#O;P7?3?7?46?>=5#
"#$%&'()*! +,-! ./$&',*! 0/&1'22'#,3! 452*&+6*',(! '2! $/&1'**/-! 7'*)!
6&/-'*!*#!*)/!2#8&6/9!:'5&+&'/2!+&/!$/&1'**/-!*#!$)#*#6#$%!5/%#,-!*)/!
;'1'*!#<!=9>9!6#$%&'()*!;+7!<#&!$&'?+*/!82/!#<! $+*&#,2!*)#2/!+&*'6;/2!',!
*)'2!?#;81/!*)+*!6+&&%!+!6#-/!+*!*)/!5#**#1!#<!*)/!<'&2*!$+(/@!$&#?'-/-!
*)/! $/&A6#$%! <//! ',-'6+*/-! ',! *)/! 6#-/! '2! $+'-! *)&#8()! "#$%&'()*!
";/+&+,6/! "/,*/&@! BBB! .#2/7##-! C&'?/@! C+,?/&2@! D4! EFGBH9! I#&!
&/$&',*! #&! &/$85;'6+*'#,! $/&1'22'#,@! /1+';! *#! JKKK! "#$%&'()*2!
D+,+(/&! +*! $852A$/&1'22'#,2L'///9#&(9! 4;;! &'()*2!&/2/&?/-9!
"#$%&'()*!MBEFN!5%!JKKK9!

Critical Thinking through a Reflexive Platform
Matthew Montebello
Department of Artificial Intelligence
University of Malta
Malta
matthew.montebello@um.edu.mt
Bill Cope, Mary Kalantzis, Anastasia Olga Tzirides,
Samaa Haniya, Tabassum Amina, Duane Searsmith,
Naichen Zhao, Min Chen
University of Illinois, USA
billcope@illinois.edu
Abstract—Massive numbers of online learners have opted to
enrol in MOOC platforms such as Coursera, EdX and Udacity,
where a plethora of topical courses are offered. However, a
limitation of all those platforms is that they cannot effectively
support teaching higher order thinking, such as critical thinking,
due to logistical issues in assessing such skills. This paper
discusses a pedagogically designed online system that employs a
set of purposely devised techniques based on a reflexive ideology,
whereby complex reasoning skills such as critical and creative
thinking form part of the process. Additionally, peer- and self-
reviewing together with feedback upon feedback enables scalable
assessment of complex critical thinking assignments. We
investigate the effect of such processes as we analyse assessment
data collected from a pilot project in a single online course with
just 150 students that made full use of our system, providing over
3.5 million essential data points. A pre- and post- survey also
provided additional quantitative and qualitative data as we close
the paper with a series of conclusions and recommendations.
Keywords—critical thinking; creative thinking; reflexive
pedagogy; e-learning; affordances.
I. INTRODUCTION
A combination of three phenomena: Political, Commercial
and Scientific, has led to a greater demand for higher education
courses, programmes and certification. Effective globalization
and an overall increase in Internet and Web access has
facilitated and supported the proliferation of e-learning courses.
The collapse of political barriers and the attractiveness of a free
market economy is the first aspect of the motivating triad that
paved the way for e-learning to gain popularity and become an
education portal at international level. A second motivation is
the commercialization of universities and escalation of other
private higher institutions [1]. Finally, the widespread
pervasiveness of technology in all sectors of society, least of all
education, paved the way for higher education to become easily
and conveniently available to a wider audience [2]. As a result
of these phenomena, Massive Open Online Courses (MOOCs)
emerged, and universities showed interest as they provided a
greater exposure, an opportunity to generate revenue, and
potentially cut down costs on faculty expenses [3]. MOOCs
became a trend in higher education and by 2012 they reached
an all-time maximum [4] providing tuition from a plethora of
topics while employing the online network to connect learners
across the globe in a way that is not possible to achieve within
a physical learning space. However, a number of issues diluted
the overall success of MOOCs as student numbers dwindled
and retention rates were as low as 4% [5]. Amongst the
numerous criticisms of MOOCs, assessment and feedback
stood out, as hundreds of learners enrolled in a typical MOOC.
It is highly unrealistic and physically impossible to cater for
large number of learners providing the required feedback to
every individual piece of work submitted by each student. Such
concerns exacerbate themselves even further as one considers
the feedback and evaluation required to matters concerning
critical thinking skills and creative competences. The platform
presented in this paper, called Common Ground Scholar
(CGScholar), embodies a reflexive pedagogy based on
Bloom’s theoretical suggestions [6], that a high majority of
learners can master a topic through differentiated instruction
and various formative assessment methods. The platform is
purposely logistically unrestrictive and academically
permissive to allow the individual learners to achieve the
mastery required in their own time and in their own personal
scholarship way. We argue that our philosophical reasoning of
employing a reflexive pedagogy through a set of new learning
affordances [7] is made possible by the advent of Web 2.0
technologies and the escalation of digital media.
The rest of the paper will be organised as follows. In the
next section we will position our e-learning platform amongst
the numerous others within an evolving online environment.
This is followed by a thorough depiction of every element
within the CGScholar in an attempt to justify our
epistemological stance through the different implementation
decisions deployed, and the various capabilities that mirror a
reflexive pedagogy. Section IV is all about data collected
through the platform while real online courses were and are
still being held. In this section we demonstrate the extent and
enormity of the task at hand as we analyse and interpret
millions of data-points that CGScholar generates automatically
through the interaction of the numerous learners and tutors,
expert reviewers, as well as, teaching assistants. Amongst other
issues we go through a comprehensive analytics functionality
tool that enables all the players involved to visualise in a single
interactive chart all the progress achieved by the individual
learners and the class as a whole. Finally, the paper comes to a
closure with an overall summarisation of the different
conclusions inferred, together with beneficial recommendation
to future online learning environments.
978-1-5386-4623-6/18/$31.00 ©2018 IEEE

II. RELATED WORK
Learning enables one to incorporate new information into
existing knowledge and participate successfully in the
environments that are significant to them. E-learning is a form
of learning that is facilitated by the massive penetration of
Internet as a form of communication. E-learning can be
defined as the ubiquitous transfer of information to as many
people as needed with the means of Internet and web
technologies. Garrison [8] defined e-learning as “the
utilization of electronically mediated asynchronous and
synchronous communication for the purpose of thinking and
learning collaboratively.” (p. 2) A critical advantage to e-
learning is the ubiquitous nature that allows for schedule
flexibility and opportunity to learn any time as long as Internet
access is available. Some other advantages compared to face-
to-face learning are cost effectiveness, actuality, use of
multimedia, interactive opportunities, and automated
evaluation [9,10,11]. On the contrary, an important
disadvantage is that students are required to use the
technology efficiently and inability to do so will hamper
learning and increase frustration. Among other disadvantages
are security vulnerabilities in e-exams, increasing anti-
sociality in users, and the limitation of using learning
management systems (LMS) without adequate technical skills
(Zhang et al., 2004).
The major components of an e-learning system are the
course content and learning management systems (LMS).
LMS is a platform that manages courses, course content and
resources, students, instructors and the reporting mechanism.
Since the 1990s, instructors have used technology to meet
their pedagogical goals. The flow of technological innovations
can be overwhelming and require careful consideration to
identify the most useful and effective ones. One such
technology in education is LMS and currently they are created
both from proprietary software manufacturers and open source
projects. Canvas, Moodle, Blackboard, Edmodo and Google
Classroom are all examples of LMS. A LMS is a software
application designed to assist instructors in meeting their
pedagogical goals to deliver learning content to students [12].
This platform not only provides a space to deliver content, but
also allows instructors to share course content, make class
announcements, submit assignments and communicate with
other students [13]. Research shows that using Blackboard,
positively affects mathematical problem-solving skills [14]
and is preferred for student participation in discussions and
submitting assignments [15]. However, some drawbacks of
this platform are difficulty in using the software, inefficiency
in bandwidth usage when having to download every time to
access, and cost of the software itself [16]. Moodle, another
popular LMS, was designed to emphasize collaboration,
inquiry and discovery-based learning [17]. Blackboard,
Moodle and many other LMS practice didactic pedagogy
where all required contents and assignments are provided.
Although there are discussion boards for interaction, it does
not create opportunity for critical thinking and learning.
Critical thinking has been a desired skill for students in order
not only to perform better in their courses in an educational
setting, but also to perform in a more effective way in their
workplace later and to be active citizens of the complex
society that we live in [18]. Thus, the development and use of
such a skill has been one of the goals of traditional education
for years [18]. According to the American Philosophical
Association [19] critical thinking is defined as “purposeful,
self-regulatory judgment that uses cognitive tools such as
interpretation, analysis, evaluation, inference, and explanation
of the evidential, conceptual, methodological, criteriological,
or contextual considerations on which judgment is based.”
Thus, the use of critical thinking in educational settings could
be translated as a higher quality understanding of theories,
evidence and significant issues [20] through scientific work
and applications of various subjects in real world contexts.
However, the use of technology and e-learning has
transformed the way that critical thinking is cultivated and
used in education [18]. It is more challenging to develop
critical thinking using the current technology, because it does
not support easily higher-level learning and creative teaching
strategies, such as active learning, team-based learning, and
discussion [18]. In addition, the present generation of
technological learning and assessment systems prevent the
instruction of critical thinking in an effective way due to the
lack of support in formative assessment of complex
assignments. The current support for assessment in e-learning
is mainly summative and primarily limited to multiple choice
questions and short answers [21,22,23,24] and simple essay
scoring [25,26,25]. Therefore, it is burdensome for the current
generation of e-learning to provide complex and practical
assignments that are required for the instruction of complex
concepts and skills, like critical thinking. Nevertheless, a way
that this kind of skills could be targeted in an e-learning
setting is through the application of interactive elements
(activities and resources) in the learning process [18], such as
calibrated peer review [25,26,25,27,28] and formative
assessment, the important element of which is that happens
directly during the learning process [29,30,31]. These
instructional techniques allow students to modify and adjust
their works and knowledge to the appropriate real-world
context. This is what will prepare them for the society that
awaits, and it will make them the active citizens and
professionals that we aim to.
III. THE ONLINE ENVIRONMENT
Every part of our e-learning platform CGScholar will be
presented, examined and analysed in detail based on the
learning theories they are grounded on. We will revisit
Bloom’s learning for mastery suggestions [6], and our own
theoretical rendition [32] as we argue in favour of a reflexive
pedagogy in contrast to a restrictive and conservative didactic
one. A number of subsections will follow in this section to
clearly distinguish between the different essential parts of the
online platform in an attempt to assist the reader visualise how
we transform our philosophical reasoning into a practical and
functional portal that has been extensively employed by
students and trainers numerous times.

A. Design
The design of the paradigm within the CGScholar online
learning platform is based on Bloom’s mastery learning
method and reflexive pedagogy. Benjamin Bloom’s mastery
learning method [6] highlights feedback and corrective
procedure. Instead of using assessments as evaluation devices
that marks the end of a learning unit, Bloom recommend using
them as learning tools, which are deemed as part of the
instructional process to diagnose individual difficulties
confronted in the learning process, i.e. feedback and give
suggestions on remediation procedures, i.e. corrective. Bloom’s
mastery learning helps learners identify what they have learned
well and what they need to learn better. Reflexive pedagogy
pushes Bloom’s mastery learning further by not only
accentuates feedback and correctives, but also creates a
dialogue in which students move between different knowledge
processes, a to-and-fro dialogue among learners and teachers,
peers, parents, experts and critical friends [7]. The design of
CGScholar platform aims to represent the main characteristics
of learning activities in a reflexive pedagogy. The
characteristics are listed as follows [7]:
a. Position the learner as the knowledge creator.
Learners are the agent in the knowledge-making process.
Reflexive pedagogy focuses more on linking personal and local
experience to more general bodies of human knowledge.
b. Encourage the learner to undertake activities that are
meaningful and realistically complex.
Under reflexive pedagogy, tasks for learners are highly
related to real life because most effectively acquisition of deep,
disciplinary knowledge are realized when situated in the
context of socially realistic tasks.
c. Challenge the learner to develop increasingly
sophisticated and deeply perceptive conceptual schemas.
Contrary to traditional pattern of understanding, knowledge
making and knowledge communication, Reflexive pedagogy
engages learners as co-constructor of concepts by building
increasingly powerful intellectual schemas for the benefit of
intellectual development.
d. Prompt learners to make their thinking or knowledge
processes explicit.
Metacognition, or thinking about the thinking, and meta
knowledge is the capacity to reflect upon and articulate the
process of one’s knowing, which makes thinking more efficient
and effective.
e. Deploy a variety of knowledge media, representing
knowledge in many ways.
Reflexive pedagogy uses synaesthesia, or mode of shifting
as a pedagogical device which enable the application of multi-
literacies theory [33], i.e. the learner activities should involve a
wide variety of representational modes: written, oral, visual,
audio, tactile, gestural and spatial.
f. Encourage dialogue and group collaboration.
Peers and the broader community in the construction of
knowledge generate powerful learning by benefiting from
various sources of expertise.
g. Offer a broad range of task options to cater for the
diversity of learners.
Reflexive pedagogy caters tremendously for variations and
diversities in the knowledge created and the way in which it
created, from one learner to the next. In collaborative work,
reflexive pedagogy builds on the complementarity of
differences, or the knowledge that is constructed by the group,
which is greater than the sum of its parts.
h. Create a learning environment that gives learners
continuous feedback on their learning.
Reflexive pedagogy attaches great significance on
formative assessment. Contrary to summative assessment,
formative assessment occurs during the learning process
instead of the end of learning process, providing direct and
specific feedback that supports student learning.
i. Offer a mix of activities that represent different
knowledge processes.
Reflexive pedagogy is a process of moving forward and
backward between different kinds of knowledge processing,
the different kinds of things learners can do in order to know.
Each sequence of knowledge processes needs to be designed to
cater to a group of learners, and area of subject matter and the
pedagogical orientation of the teacher or the school, instead of
a list of things learners have to do, or a set order.
The design of CGScholar online learning platform revolves
around reflexive pedagogy with different functions, students
and admins using Updates and Creator with peer review
function in the comment areas.
B. Pedagogical Principles
The pedagogical principles of “seven affordances” are
followed in the curriculum design of courses through
CGScholar online platform. Cope and Kalantzis [34]
operationalize the idea of e-learning ecologies, a term served as
a metaphor to understand the learning environment as an
ecosystem which incorporates the complex interactions among
human, textural, discursive and spatial dynamics, by
heuristically segmented them into seven “new learning”
affordances, ie. e-Learning Affordances: ubiquitous learning,
active knowledge making, multimodal meaning, recursive
feedback, collaborative intelligence, metacognition and
differentiated learning. In the CGScholar environment, the
seven affordances represent an “agenda for new learning and
assessment” that redefine the relations among knowledge and
learning, recalibrating traditional modes of pedagogies in an
effort to create learning ecologies which better suits the
educational needs and goals of our time.
C. Main Features
CGScholar online platform is built in an attempt to reframe
the relations of knowledge and learning and the functions
within the system and accelerate the development of critical
thinking for learners. Critical thinking as defined by Elder is

“self-guided, self-disciplined thinking which attempts to reason
at the highest level of quality in a fair-minded way” [35]. The
learning process in CGScholar system starts from the
perspective of individual learner—their voice, interests and
localized relevance. One of the founding principle of seven
affordances, active knowledge production, upholds that
learners are the knowledge designers instead of the ones that
reproduce knowledge. Moreover, according to another
affordance, recursive feedback, formative assessment is
accentuated in the CGScholar platform, offering four on-the-fly
assessment mechanisms: review, annotations, checker and
survey. Learning interactions as well as learning artifacts are
being assessed. CGScholar also build recursive feedback—
feedback whose value is weighted by feedback on feedback,
and ratings that are moderated by inter-rater reliability
calculations. The affordance of metacognitive reflection is
realized in CGScholar environment. One way is in the
explicitness of a semantically framed knowledge-
representation space, where learners learn about information
architectures in an analytical way – in the organizational logic
of the structure tool, or the tagging tool, and the markup
requirements of ‘emphasis’ or ‘block quote’. Another way is
the requirement to self- and peer-assess against explicit criteria
that are presented before the work even begins, a meta-
reflective process which is now shared with the teacher.
Students and teachers necessarily engage in dialogue about the
fundamental nature of the task, as well as the specifics of task
performance [34].
CGScholar provides tools for fully multimodal knowledge
representation. Learning is deepened as students shift from one
mode to another, making their meanings one way then another
complementary way [34]. Learners in CGScholar environment
employ Updates and Creator to complete weekly and term
assignments. Within every community, Updates, partly
functions as a blog, can include embedded video, audio, data
and external links, e.g. for admins to share courses contents, or
notify a deadline or agenda, and for learners to present their
topics, and provides comment areas below to facilitate peer-
and-admin discussion. Creator, usually for term papers, is a
semantic editor and multimodal working space, which
transcends beyond the handwriting book or the word processor.
Digital objects, including image, audio, video, text, math, live
links, dataset and embedded external media, from Youtube
videos to GitHub code, all of which can be inserted within the
body of the text. With the completion of a course, learners
could review on their performance via visualized learning
analytics with three main measures of knowledge: Know! i.e.
the quality of the knowledge work; Focus! i.e. the effort
contributed, and Help! i.e. community contributions.
CGScholar provides a new generation of analytics that could
replace traditional tests with embedded formative assessments.
The paradigm of the overall design of CGScholar serves as an
agent that helps the development of learners’ critical thinking
with a focus of learner-oriented perspective and learning
process.
IV. DATA COLLECTION AND ANALYSIS
This section discusses data collection and the analysis that
we performed in an attempt to evaluate the overall performance
of our platform in relation to the use of feedback as a form of
critical thinking, along with the educational inferences that
such an analysis might render. Amongst other analyses we
study and endeavor to answer the following question. How
does the learners’ performance change as a result of employing
critical thinking strategies based on the use of recursive
feedback through the CGScholar platform in the duration of the
course?
Before presenting the findings, it is important to understand
and appreciate why employing critical thinking skills via the
form of recursive feedback of peers and experts helps to
improve the students’ final results. A number of studies have
shown that students value feedback from their peers [36] as
they recognized the significance and usefulness of such
information [37]. The power of tutor feedback [38] is
considered to be a strategic pedagogical technique within
higher education as academics, more than students, believe it to
be a main factor of encouraging critical thinking and thus lead
to academic success [39]. Even though studies [40] have shown
that students expect such crucial feedback in order to improve
their work, a number of educational researchers have reported
issues with feedback given by academics at a number of higher
education institutions around the world amongst which are
Surridge [41], Krause, et. al., [42], and Sadler [43].
Predominant concerns include criticisms regarding the nature
and timing of feedback given by educators [44], the
terminology employed [45], as well as the feedback that is not
fit for purpose as it fails to specify improvements [46], making
is hard to actually adopt any advice given [47].
For this reason, the need to further research and investigate
the effect of feedback on the final student outcome is critical
and essential, especially as higher education institutions and
academics are not so sure about such issues [48], which creates
even more friction between students and educators [49]. Moss
& Brookhart [50] claim that feedback, as part of a formative
assessment pedagogy, needs to feed forward to be effective. If
students feel and consider that the feedback given to them by
the experts is not contributing, or is not clear how it feeds
forward, then it is rejected and ignored. William [51] also
agrees about the critical role of experts’ feedback as a crucial
component of formative assessment and concludes that this
only holds when the student is convinced and cognizant that
the feedback received will improve the performance. Sadler
[43] states that for students to truly appreciate, accept and
fruitfully employ feedback in order to enhance and enrich their
work they need to fully understand and conceptually appreciate
three things. What exactly is the feedback given asking them to
do? Feedback provided by an expert impinges directly upon the
subject matter and student are aware that such help can assist in
raising the quality of their work and a better final assessment.
Secondly, is the feedback specifically suggesting on how to
perform a quality leap? Students need to be instructed and
possibly shown of how to excel, and where their work is
lacking. In the absence of such realization the final outcome
will not be superior to the previous work. Finally, are the
students fully aware of which criteria have been employed
when the feedback was given? Experts will need to ensure that
each piece of advice and each recommendation are provided
against the established performance indicator that was initially

set for the piece of work. With the conceptual knowledge of
these three areas students will be in a position to utilize and
take full advantage of the experts’ feedback and will positively
manifest itself in the final outcome.
Figure 2 - Comparison between initial and final expert
scores
Data Collection:
The case study here is based on student interactions using
the CGScholar platform environment in one graduate Summer
course, which has 157 graduate students, offered in 2017 at the
University of Illinois, Urbana-Champaign. Over the eight-week
course, the CGScholar platform enabled collection of highly
granular machine and peer feedback in two peer, self and
expert reviewed projects, textual annotations, NLP academic
language analysis, and structured online discussions. The
review data of Figure 2 was collected as part of one writing
assignment where peers reviewed submitted work versions
prior to further revisions. Reviews were guided by a detailed
rubric with five criterions related to the assignment. For each
criterion reviewers provided written feedback and a rating from
0 to 4. Three expert reviews were also created for the peer
reviewed work versions and the final work versions. Figure 2
shows the relationship between expert review ratings of the
peer reviewed and final versions.
Findings:
The results of our study comprise two sets of review results
submitted by experts at the beginning and at the end of the
project, giving us the possibility to analytically investigate any
resulting changes. Such changes could have potentially
occurred due to a number of factors, apart from the obvious
initial feedback given by the experts themselves, amongst
which are peer reviews discussed earlier, feedback on
feedback, and further reading while also attending class
sessions. The data collected was analysed and a visual of one
of the performance indicators can be seen in Figure 2. The final
assessments can clearly be seen to follow the initial ones, and
once more with an average percentage increase of
approximately 30% on average. What is also evident from this
chart is the wider range of distribution of the initial scores,
compared to a more compact and cohesive set of values
ranging within the final scores. This aligns with the “mastery”
notion of Bloom’s, with a wider range of course participants
achieving mastery objectives than would be the case if their
papers had simply been submitted, without opportunity for
engaging students in critical thinking activities of peer review
and revision.
V. CONCLUSIONS
In this paper we have presented a series of big data learning
analytics that resulted from a number of authentic online
courses in our e-learning platform, strongly grounded within a
reflexive pedagogy. The education methodologies employed
focus on how to better assess complex reasoning skills such as
critical and creative thinking, through peer, expert and self-
reviews. Additionally, pre- and post- surveys also provided
quantitative and qualitative data to supplement our conclusions
following the interpretation of the results. We strongly believe
that the functionality provided through such a platform, based
on Bloom’s learning for mastery theory, together with our new
learning affordances model, learners are in a better position to
achieve their academic goals through their own pace and in
tune with their personal learning needs.
ACKNOWLEDGMENT
The authors would like to thank all the students and expert
reviewers who participated in this intervention, as well as the
members of the wider research team who wrote and provided
helpful comments on previous versions of this. Additionally,
this works has been supported by research grants from the US
Department of Education, Institute of Education Sciences:
“The Assess-as-You-Go Writing Assistant” (R305A090394);
“Assessing Complex Performance” (R305B110008); “u-
Learn.net: An Anywhere/Anytime Formative Assessment and
Learning Feedback Environment” (ED-IES-10-C-0018); “The
Learning Element” (ED-IES-lO-C-0021); and “InfoWriter: A
Student Feedback and Formative Assessment Environment”
(ED-IES-13-C-0039). Bill and Melinda Gates Foundation:
“Scholar Literacy Courseware.” National Science Foundation:
“Assessing 'Complex Epistemic Performance' in Online
Learning Environments” (Award 1629161).
REFERENCES
[1]
J.W. Alstete and N.J. Beutell, "Performance indicators in online distance
learning courses: a study of management education," Quality Assurance
in Higher Education, vol. 12, no. 1, pp. 6-14, 2004.
[2]
A. Stella and A. Gnanam, "Quality assurance in distance education: The
challenges to be addressed," Higher Education, vol. 47, pp. 143-160,
2004.
[3]
G. Schrank. (2012) Infighting at Univer- sity of Virginia re: MOOC
education. [Online].http://edlab.tc.columbia.edu/index. php?q=node/7863
[4]
V. Camilleri, L. Busuttil, and M. Montebello, "MOOCs: Exploiting
Networks for the Education of the Masses or Just a Trend?," in The
Social Classroom: Integrating Social Network Use in Education.
Hershey, PA: IGI Global, 2014, pp. 348-366.
[5]
D. Koller, A. Ng, C. Do, and Z. Chen, "Retention and intention in
Massive open online Courses," Educase Review, pp. 62-63, 2013.
[6]
B.S. Bloom, "Learning for mastery," UCLA-CSEIP, Evaluation
Comment, vol. 1, no. 2, pp. 1-12, 1968.
[7]
M. Kalantzis and B. Cope, New Learning: Elements of a Science of

Education. Illinois: Cambridge University Press, 2012.
[8]
R. D. Garrison, E-Learning in the 21st century a community of inquiry
framework for research and practice. New York: Routledge, 2017.
[9]
C. Dean, Technology Based Training and On-line Learning. Sheffield,
UK: Peak-Dean Interactive, 2002.
[10]
M.J. Rosenberg, E-learning: Strategies for delivering knowledge in the
digital age. (Volume 3). New York: McGraw-Hill, 2001.
[11]
D. Zhang, J.L. Zhao, L. Zhou, and J.F. Nunamaker, "Can E-Learning
Replace Classroom Learning?," vol. 47, no. 5, pp. 75-79, 2004.
[12]
M. Machado and E. Tao, "Blackboard vs. moodle: Comparing user
experience of learning management systems," in 37th Annual Frontiers
In Education Conference - Global Engineering: Knowledge Without
Borders, Opportunities Without Passports, 2007.
[13]
S. Lonn and S.D. Teasley, "Saving time or innovating practice:
Investigating perceptions and uses of Learning Management Systems,"
Computers & Education, vol. 53, no. 3, pp. 686-694, 2009.
[14]
S.M. Jacob and H.K. Sam, "Effect of online discussion forums to
improve mathematics problem solving and critical thinking -A
comparative study," in International Conference on Innovations in
Information Technology, Innovations in Information, 2008.
[15]
S. N. Hamade, "Student Perceptions of Learning Management Systems
in a University Environment: Yahoo Groups vs Blackboard," in 2012
Ninth International Conference On Information Technology - New
Generations, Information Technology, 2012.
[16]
P. Bradford, M. Porciello, N. Barkin, and D. Bakus, "The Blackboard
Learning System: The Be All and End All," Educational Instruction,
Journal of Educational Technology Systems, vol. 35, no. 3, pp. 301-314,
2006.
[17]
K. Brandl, "Are you ready to “Moodle”? ," Language Learning &
Technology, vol. 9, no. 2, pp. 16–23, 2005.
[18]
R.G. Saadé, D. Morin, and J.D.E. Thomas, "Critical thinking in E-
learning environments," Computers in Human Behavior, vol. 28, no. 5,
pp. 1608–1617, 2012.
[19]
P. A. Facione, Executive summary: Critical thinking: A statement of
expert consensus for purposes of educational assessment and instruction
– The Delphi Report. Millbrae, CA: The California Academic Press,
1990.
[20]
J. Guiller, A. Durndell, and A. Ross, "Peer interaction and critical
thinking: Face-to-face or online discussion? ," Learning and Instruction,
vol. 18, no. 2, pp. 187–200, 2008.
[21]
M. Brooks, S. Basu, C. Jacobs, and L. Vanderwende, "Divide and
correct: Using clusters to grade short answers at Scale," in First ACM
Conference on Learning @ Scale Conference - L@S 14, 2014.
[22]
C. Leacock and M. Chodorow, "C-Rater: Automated Scoring of Short-
Answer Questions," Computers and the Humanities, vol. 37, no. 4, pp.
389-405, 2003.
[23]
T. Mitchell, N. Aldridge, and P. Broomhead, "Computerised Marking of
Short-Answer Free-Text Responses," in 6th International Computer
Assisted Assessment (CAA) Conference, 2002.
[24]
M. Mohler and R. Mihalcea, "Text-to-text semantic similarity for
automatic short answer grading," in 12th Conference of the European
Chapter of the Association for Computational Linguistics on - EACL '09,
2009.
[25]
Y. Chen, "Investigating MOOCs through blog mining," International
Review of Research in Open and Distance Learning, vol. 15, no. 2, 2014.
[26]
S. P. Balfour, "Assessing writing in MOOCs: Automated essay scoring
and calibrated peer review," Research and Practice in Assessment, vol. 8,
no. 1, pp. 40–48, 2013.
[27]
H. Suen, "Peer assessment for massive open online courses (MOOCs),"
The International Review of Research in Open and Distance Learning,
vol. 15, no. 3, 2014.
[28]
C. Sandeen, "Assessment's place in the new MOOC world," Research &
Practice in Assessment, vol. 8, pp. 5-12, 2013.
[29]
N. Chudowsky, R. Glaser, and J. W. Pellegrino, Knowing what Students
Know: The science and design of educational assessment. Washington,
DC: National Acad. Press, 2001.
[30]
P. Black and D. Wiliam, "Assessment and classroom learning,"
Assessment in Education: Principles, Policy and Practice, vol. 5, no. 1,
pp. 7–74, 1998.
[31]
B. Cope, M. Kalantzis, S. McCarthey, C. Vojak, and S. Kline,
"Technology-Mediated Writing Assessments: Principles and Processes,"
Computers and Composition, vol. 28, no. 2, pp. 79-96, 2011.
[32]
B. Cope and M. Kalantzis, e-Learning Ecologies. New York: Routledge,
2017.
[33]
M. Kalantzis, B. Cope, E. Chan, and L. Dalley-Trim, Literacies, 2nd ed.
Cambridge, UK: Cambridge University Press, 2016.
[34]
B. Cope and M. Kalantzis, "Towards a new learning : the Scholar social
knowledge workspace, in theory and practice," E-Learning and Digital
Media, vol. 10, no. 4, pp. 332-356, 2013.
[35]
L. Elder. (2007) Defining Critical Thinking. [Online].
http://www.criticalthinking.org/pages/defining-critical-thinking/766
[36]
D. S. Ackerman and B.L. Gross, "Instructor Feedback: How Much Do
Students Really Want?," Journal of Marketing Education, vol. 32, no. 2,
pp. 172–181, 2010.
[37]
D. Hounsell, "Towards a More Sustainable Feedback to Students," in
Rethinking Assessment in Higher Education. London: Routhledge, 2007,
pp. 101–113.
[38]
J. Hattie, Visible learning: A synthesis of over 800 meta-analyses
relating to achievement. New York, USA: Routledge, 2009.
[39]
D. Carless, D. Salter, M. Yang, and J. Lam, "Developing Sustainable
Feedback Practices," Studies in Higher Education, vol. 36, no. 4, pp.
395-407, 2011.
[40]
R. Higgins, P. Hartley, and A. Skelton, "The Conscientious Consumer:
Reconsidering the Role of Assessment Feedback in Student Learning,"
Studies in Higher Education, vol. 27, no. 1, pp. 53-64, 2002.
[41]
P. Surridge, "The National Student Survey 2005-2007: Findings and
Trends," Bristol, 2008.
[42]
K.L. Krause, R. Hartley, R. James, and C. McInnis, "The first year
experience in Australian universities: Findings from a decade of national
studies," Centre for the Study of Higher Education, Higher Education
Innovation Programme, Dept. of Education, Science and Training,
Melbourne, 2005.
[43]
D. R. Sadler, "Beyond feedback: Developing student capability in
complex appraisal," Assessment & Evaluation in Higher Education, vol.
35, no. 5, pp. 535-550, 2010.
[44]
HEA, A Marked Improvement: Transforming Assessment in Higher
Education. York: Higher Education Academy, 2012.
[45]
M. Weaver, "Do students value feedback? Student perceptions of tutors’
written responses," Assessment & Evaluation in Higher Education, vol.
31, no. 3, pp. 379–394, 2006.
[46]
R. Higgins, P. Hartley, and A. Skelton, "Getting the message across: The
problem of communicating assessment feedback," Teaching in Higher
Education, vol. 6, no. 2, pp. 269–274, 2001.
[47]
A. Poulos and M.J. Mahony, "Effectiveness of feedback: The students’
perspective," Assessment & Evaluation in Higher Education, vol. 33, no.
2, pp. 143–154, 2008.
[48]
S. Bohnacker-Bruce, "Effective Feedback: The Student Perspective,"
Capture, vol. 4, no. 1, pp. 25–36, 2013.
[49]
R. O’Brien and L. Sparshatt, "Exploring Staff and Students’ Mental
Maps: Creating Narratives for Successful Assessment Feedback," in
Transforming the student experience, Harrogate, 2008.
[50]
C.M. Moss and S.M. Brookhart, Advancing Formative Assessment in
Every Classroom: A Guide for Instructional Leaders. Alexandria, Va:
Association for Supervision and Curriculum Development, 2009.
[51]
D. William, Embedded Formative Assessment. Bloomington, IN:
Solution Tree Press, 2011.