Content uploaded by Vera Baum
Author content
All content in this area was uploaded by Vera Baum on Jul 16, 2014
Content may be subject to copyright.
Assessing Backstage—A Backchannel for
Collaborative Learning in Large Classes
A Formative Study on its Usability and Influence on Students‘ Questioning
Vera Gehlen-Baum
Educational Technology
Saarland University
Saarbrücken, Germany
Alexander Pohl, François Bry
Institute for Informatics
University of Munich
Munich, Germany
Abstract—To reduce the passivity of students in large lectures, we
propose to enhance the classroom with the digital backchannel
Backstage, which aims at promoting awareness and
participation. We tested Backstage in a preliminary user study at
a relatively early stage of development. Besides gaining first
impressions of usability, we were particularly interested in the
kinds of questions raised on Backstage and how they correlate to
the participants' attitudes towards questioning. The study
confirms known issues regarding questioning that are not fully
overcome with the tested prototype. Thus it is concluded that the
concept of Backstage needs to be extended in order to embrace
remedial assistance in how to ask adequate questions during
lectures.
Collaborative Learning, Enhanced Classroom, Digital
Backchannel, Questioning
I. INTRODUCTION
Maybe everyone in her academic life has attended some
instructor-centered lectures with large and anonymous
audiences and only few students actively participating in the
lecture. Although findings suggest that active participation
contributes to better learning (e.g. [1,2]), large class lectures do
not provide much space for an actively participating and
collaborating audience. Certainly, social mediation, i.e., the
feasibility of social interaction between a teacher and her
audience as well as among the students, is a limiting factor: the
larger the class, the fewer social interaction takes place. In
groups with more than 30 members, a manager is needed to
facilitate discourse. In many cases, the teacher plays this role
[3].
Discourse between persons often follows certain triadic
communication structures or patterns. For example, a discourse
pattern frequently observed in classrooms is the IRE (Initiation,
Response, and Evaluation) pattern: the teacher initiates
discourse by posing a question; a student responds to that
question that is finally evaluated by the teacher. Besides
evaluation, the discourse can also follow the IRF (Initiation,
Response, and Follow up) pattern: the teacher may follow up
on student‘s response with a further comment [3].
Yet, in such a discourse usually only few persons, i.e. the
teacher and a few students, are involved. The large part of the
audience basically remains passive. According to the
constructivist‘s perspective, however, learners construct their
own knowledge and understanding, which requires active
participation in the learning process [4]. Furthermore,
educational scientists have developed the theory of
communities of learners, according to which learning is not
understood as a transfer of knowledge between experts, nor an
acquisition of experts‘ knowledge by novices, but rather as a
transformation of the learner‘s participation in the learning
process [5].
One way to increase active participation of and social
interaction among students may be to employ textual
computer-mediated communication in the classroom, since
―Text-only CMD [Computer Mediated Discourse] is a
surprisingly effective way to ‗do‘ interactional work […]‖ [6]
(p. 11). Besides, CMD is effective in encouraging students to
participate in the discourse. For example, in the field of second
language learning, it has been observed how CMD is adopted
by students for their learning and how it affects their
participation [7]. It is reported on the observation that every
student participated in CMD, including those students who are
shy and more self-conscious about the mistakes they make than
other students. Furthermore, the balanced nature of CMD has
been highlighted: students that dominate classroom discourse,
e.g. by answering every question posed by the teacher, do not
dominate CMD. Every student gets a chance to speak without
being interrupted.
We propose to enhance the classroom with what are called
digital backchannels, i.e., computer-mediated synchronous
(textual) communication tools, making non-disruptive
information exchange among the audience during the lecture
possible. In particular, we hypothesize that providing a social
environment in which questions can be raised during the
lecture is a major benefit of using backchannels in large class
lectures. However, we contend that current digital
backchannels do not sufficiently meet the conditions and
requirements imposed by large class lectures.
A. Using Backstage in Large Classes
In a joint effort of computer and educational scientists, we
developed the digital backchannel Backstage, which provides a
rich set of interactions specifically tailored for the use in large
lecture classes. Its functionalities comprise fast information
exchange, various kinds of feedback dedicated to both the
student and the lecturer, and incentives to partake. In the
remainder of this section, we provide a short overview of
Backstage. For a more detailed description the interested reader
is referred to [8].
By means of a microblog, students may exchange ideas and
comments through short messages, comprising only a few
words. The most prominent example of a microblog is
Twitter
1
, in which messages comprise a maximum of 140
characters. Microblogs allow messages to be written and read
quickly, and hence are well suited for the use as digital
backchannels in large class lectures. Backstage provides
different modes of communication: messages can be visible
either to the public or to certain users only (private
communication). Besides, the author‘s username may be
shown along with each of her message or be hidden
(anonymous communication). Indeed, one can argue for or
against the usefulness of all these modes of inter-personal
communication. For example, anonymous communication
certainly lowers the barrier to partake, especially during the
first turns at the beginning of a semester, but may likewise
simplify the dissemination of offending messages. We think
that whether certain modes of communication should be
available largely depends on the compilation of the audience
and the topic of the lecture. Thus, Backstage is highly
customizable by the lecturer, meaning that the lecturer, even
during lecturing, can easily change the availability of the
different modes of communication provided by Backstage.
Students may assign messages to certain predefined
categories, e.g. ―Question‖ or ―Lecturing Pace‖. According to
these categories, messages can be filtered out, aggregated and
be displayed at the lecturer‘s dashboard. This provides her with
a concise and simple topic-related overview of the backchannel
discourse. Thus, categories can be used by students to give
immediate feedback to the lecturer related to contents (e.g.
―Question‖, ―Remark‖) or to lecturing style (e.g. ―Lecturing
Pace‖).
Messages can furthermore be rated by students to assess the
relevance of a message for the lecture. This is provided on
Backstage by means of a simple rating scheme allowing
students to quickly express approval or rejection of a message.
Rating in the context of Backstage serves two important
purposes. First, it makes possible to determine a ranking of the
top k messages that can be displayed at the lecturer‘s
dashboard. Combined with the topic-related aggregation of
messages described in the previous paragraph the lecturer
obtains a very concise, yet meaningful overview of the
backchannel discourse. Second, rating provides feedback to
students given by their peers. This feedback can furthermore be
accumulated to form a kind of status for a student. The better
her messages are rated the higher the status she gains.
To promote active participation the lecturer can conduct
short quizzes, reminiscent to Audience Response Systems
2
(ARS; e.g. [9-11]) via Backstage. A student‘s concentration
drastically declines after 20 minutes of continuous lecture [2].
Quizzes allow the instructor to break up the lecture into smaller
1
http://www.twitter.com
2
Audience Response Systems are also referred to as Student Response
Systems, Personal Response Systems, and Clicker Systems, among others.
sections and to assess factual knowledge and students‘
retention. In addition to fostering active participation, quizzes
provide feedback about the student‘s performance relative to
their peers. When the lecturer starts a quiz, the students‘
dashboards display a multiple-choice item. The students‘
responses are displayed and updated in real-time at the
lecturer‘s dashboard. As soon as the lecturer closes the voting
phase, the responses are aggregated and displayed at both the
lecturer‘s and the students‘ dashboards for further review.
B. Usability—A Tradeoff between Economy of Effort and
Learning
In [12], usability is defined as ―a quality attribute that
assesses how easy user interfaces are to use―, and is
operationalized by five variables. While some of these, e.g. the
number and severity of errors the user makes (errors) or the
time necessary to rebuild the knowledge acquired after some
time of not using the system (memorability), are important to
consider for all kind of software, the remaining three
variables—learnability, efficiency, and satisfaction—are of
particular importance for computer-based learning
environments.
In most cases it is rational to simplify user interactions by
means of a clear and concise user interface (learnability) that
enables users to quickly accomplish their tasks (efficiency).
However, the resulting economy of effort, especially when
considering user interfaces of applications for learning, might
not always be the best way to improve learning processes and
results. Studies on expertise have shown that many activities
are quickly learned to an extent sufficient for daily life. In
order to become an expert, however, it is necessary to
perpetually reflect, and possibly deviate from, the activities as
being taught [13]. Therefore it could be fruitful to provide user
interactions that initiate some cognitive activity in the learning
environment.
Learning environments often adopt user interactions from
well-known applications such as Facebook
3
or Twitter which
are familiar to most learners. This allows users to quickly grasp
the way to interact with the application (learnability) and to
quickly accomplish the tasks without much need for settling in.
However, besides mere usability, further—educational—design
parameters need to be taken into account in order to ease
interactions as well as provide for occasions in which the user
can reflect on her interactions. Similar to Piaget‘s ―Conceptual
Change‖ [14] that can be observed in children being stretched
to their limits due to insufficient mental models of the world,
adult learners should be stretched to their limits when carrying
out learning tasks in a sub-optimal manner. Likewise, they
should be provided with instructions supporting them in
optimizing their learning. In Computer Supported
Collaborative Learning (CSCL) this is frequently achieved by
the provision of collaboration scripts: learners obtain roles
according to which the tasks and the orders in which to carry
out these tasks are specified. Several studies could substantiate
3
http://www.facebook.com
the superiority of this approach compared to respective controls
(e.g. [15-17]).
Undoubtedly, usability is an essential aspect of the design
of learning platforms. However, it might prove reasonable—
with sound judgement—to attach more importance on
educational claims (e.g. the conveyance of competencies in
questioning) rather than on mere usability principles.
C. Asking the Right Kinds of Questions to Improve
Understanding
In several articles (e.g. [18,19]) it has been argued that
questioning is an important strategy to elaborate new
knowledge. Unfortunately, students often find it difficult to ask
for help and further explanation [20]. This is not surprising,
considering that formulating questions already requires some
understanding of, and former knowledge about, the topic (e.g.
[18]). Questioning is thus subject of various fields of research.
For example, reciprocal teaching [21] and script research (e.g.
[22,23]) deal with questioning or at least highlights its
importance [24].
Not only the quantity, but also the quality, of questions is
important for learning [23]. In ibid. a distinction is made
between three kinds of questions: factual, comprehension, and
integration questions. The Table I summarizes the descriptions
of questions and their distinctions regarding knowledge.
Tab. I Kinds of Questions (from [23], p. 350, with minor adaptations).
Question
Description
Factual
Asks for recall of facts or other information
explicitly covered in the lesson.
Encompasses knowledge restating: Simple
statements of fact or information gleaned directly
from the lesson or prior knowledge.
Comprehension
Asks for a process or term to be described or
defined.
Encompasses knowledge assimilation:
Definitions, descriptions, and other material
paraphrased in student‘s own words.
Integration
Goes beyond what was explicitly stated in the
lesson, connects two ideas together, or asks for an
explanation, inference, justification, etc.
Encompasses knowledge integration: Makes new
connections or goes beyond what was provided in
the lesson — explanations, inferences,
relationships between ideas, justifications,
statements, linking session content to material
from outside the lesson (prior knowledge and
personal experience).
In order to achieve better learning results, it is especially
important to formulate integration questions as these are
connected to a deeper elaboration, though it is quite uncommon
that integration questions are raised without being fostered by
appropriate guidance. These were the findings of the study
presented in [23], were it has been shown that integration
questions could be trained and lead to a ―more complex
knowledge construction‖ (p. 338).
D. Research Question
We tested the Backstage prototype in a preliminary user
study at an early stage of development. We focused on the
following questions. (1) What are the participants‘ impressions
of the usability of Backstage? (2) What kinds of questions were
asked on Backstage in comparison to traditionally held
presentations without any backchannel support? (3) How do
these questions correlate with the participants‘ attitudes
towards asking questions on Backstage?
II. METHOD
A. Participants and Design
Nineteen participants from different subjects (mainly
Educational Science and Informatics but also Psychology and
German as foreign language) participated in this study. The
mean age of the participants was M = 26.32 (SD = 3.33). Five
(26.32 %) participants were female and fourteen (73.68 %)
were male. Since few can be said about students‘ prior
knowledge in large lectures, we decided to omit a pretest for
this preliminary study and thus chose an experimental design
with one control group. The participants were picked randomly
from the different fields of study. Every participant in the
experimental group was assigned a working place equipped
with a PC, resembling traditional computer-based learning
environments. Although unnecessary, each participant of the
control group was also assigned a PC working place.
To assure that the prototypical implementation of
Backstage was able to cope with the amount of data that was to
be processed we subdivided the experimental group (fourteen
participants) into three subgroups with five and four
participants, respectively. For the control group, we used the
equal group size of five participants in order to ensure the
participants weren‘t inhibited by a larger group size.
B. The Learning Environment
A 30-minutes presentation was held in front of each group
in a room with PC workplaces. The presentation was about two
educational subjects which were presented for ten to fifteen
minutes each. The first talk was about the notion of transfer—
the definition, the reasons why it occasionally fails to take
place and the ways to foster it. The second part was about the
research on expertise—the definition and a short overview of
this field. We tried to eliminate additional talk in the
classroom. The participants of the control group listened to the
presentation in a traditional way and thus were invited to raise
hands.
Prior to each trial, every participant in the experimental
group was already logged into Backstage using pseudonyms.
The experimental group could only use certain features of
Backstage. As the study was conducted at an early stage of
development, several features were not completely
implemented or unstable. The participants were informed about
the state of implementation and the features that could be used.
The Figure 1 shows the layout of the students‘ dashboards.
The microblog communication capabilities as described in
Section I.A were completely functioning and available. The
participants could also rate messages, although the rating
scores of the messages were not aggregated and rendered along
with each message. The participants could also assign
messages to categories that were already aggregated properly
and were rudimentarily displayed at the presenter‘s dashboard.
The top-k ranking of messages was also not available but the
presenter strived to respond to the questions that were raised on
the backchannel. The quiz module was basically available,
although the rendering of quiz results was still in need for
improvements.
C. Instruments and Dependent Variables
The dependent variables were (1) usability, (2) the
participants‘ attitudes towards asking questions, and (3) the
number and kinds of questions asked during the presentation.
The first two variables were mainly measured by questionnaire
items and by analysis of the backchannel communication.
We tested the usability with fourteen multiple choice
questions (Likert scale
4
from 1 to 6) and three open items.
Since the participants were using Backstage for the first time,
we considered more important to measure the time needed to
get acquainted with the application, than other usability
aspects, e.g. the number of errors the participants made. Thus,
the learnability was measured by the time needed to get
acquainted with Backstage and usability was measured by
multiple-choice questions like ―The application‘s user interface
is clear and concise‖. The reliability of the usability questions
were measured using Cronbach‘s α [25] that yields α = 0.91. In
4
The linguistic grades corresponding to the numeric grades 1 to 6 are:
―definitively disagree‖ (1), ―disagree‖,‖ tend to disagree‖, ‖tend to agree‖,
―agree‖, and ―definitively agree‖ (6).
addition, the three open items were meant to give a deeper
insight in what the participants liked or missed. The answers
were sighted by two coders and certain categories like
―questions to the presenter‖ were counted. Cohen‘s Kappa, by
which the consensus between the two coders is measured [26],
yields κ = 0.92.
The attitude towards questions was measured with seven
multiple choice questions like ―I felt unsure when raising a
question‖. The reliability yields α = 0.77.
The backchannel communication was exported to log files
and the statements of the control group were transcribed
accordingly. The statements were classified twice by distinct
coders into ―question‖, ―response to question‖, and ―feedback
to the presenter‖. Since no other kinds of statements ended
with questions marks, questions could be easily identified. A
response to a question could be determined by its adjacency to
the respective question or by context. Feedback to the presenter
could be extracted in a similar way. In a second step,
statements belonging to ―question‖ were further subdivided
according to their content into the classes given in the Table I
(κ = 0.74).
Since the control group was not to answer questions
regarding the usability of the application, we provided a similar
questionnaire that focused on questioning. The outcomes of
this questionnaire, however, are not presented in this article
which focuses on the analysis of the process data.
D. Procedure
We conducted each trial during a 30-minutes presentation.
The participants were assigned to the prepared places. For the
control group, the places were additionally equipped with pen
Figure 1 A participant's dashboard on Backstage. The picture shows the microblog editor and timeline (left) and the results of a quiz that has just been
conducted (right). The dashboard shown is in German, since the user study was conducted in German.
and paper and for the experimental group the login to
Backstage had already been accomplished. The experimental
group was provided with a brief introduction to Backstage and
how to use the different functions. Furthermore, both
experimental and control group were provided with a short
summary about the outline of the study. The control group was
particularly instructed to raise hands in order to pose a question
during the presentation. The presentation was subdivided into
two parts—the first part was about transfer, the second about
expertise. After each section both groups were invited to raise
questions. Immediately after the presentations, which were
held comparably in contents and style, the participants were
asked to fill out a questionnaire.
E. Statistical Analysis
Due to the small group sizes we decided to forgo inductive
statistical analysis and to restrict ourselves to descriptive
analysis. However, in most cases the heterogeneous t-tests (not
reported on in this paper) reveal significant differences
between traditional presentations and presentations with
Backstage running.
III. RESULTS
A. Usability
The mean M = 4.42 (SD = 0.78) on usability we measured
is between ―tend to agree‖ and ―agree‖, thereby indicating that
the participants found it easy and intuitive to work with
Backstage: every participant posted at least one message on the
backchannel during the presentation. Also, the participants‘
comments point in this direction. Notably, the time the
participants stated to be necessary in order to get acquainted
with Backstage highly varied. While the average was M = 8.79
(SD = 8.34) minutes, the minimum time stated was one minute
and the maximum was 30 minutes.
By means of the open items we could figure out that the
participants appreciated the possibility to directly send
feedback to the presenter (mentioned eight times). The process
data showed that the experimental groups gave feedback to the
presenter more often than the control group (M = 5.43) which
gave only short feedback, e.g. ―ok‖ for acknowledging
(M = 0.40). Also, Backstage‘s quiz functionality to assess
current knowledge was appreciated by the participants of the
experimental groups (mentioned five times).
However, they missed the easy-to-use communication
known from other communication tools. Similar statements
were made regarding the lack of presentations slides integrated
in Backstage (mentioned twice). Three participants also noted
the risk of getting distracted by the use of Backstage.
B. Questions
With Backstage the participants gave approximately nine
times more remarks during the presentation than without the
backchannel (Figure 2A).
Among the messages on the backchannel, only a small
fraction was off-topic (M = 2.5, SD = 2.82). A considerable
amount of the messages were feedback (M = 5.42, SD = 5.57).
The largest part was questions (Figure 2A).
Participants of the experimental groups raised questions
more frequently than of the control group. This is also reflected
by the positive attitude towards asking questions on Backstage
(M = 4.56; SD = .87). While few questions are about using
Backstage, the large part of questions refer to the presentation
contents. The questions of participants of the control group did
not refer to the use of Backstage, but to the presentation‘s
content only. The distribution of questions regarding the
classification of questions as given in the Table I is illustrated
in the Figure 2B.
A)
B)
Figure 2 A) A participant‘s average number of contributions and questions, in
particular. The numbers in brackets indicate the respective standard variances.
B) A participant‘s average number of questions split in the kinds described in
the Table I. The numbers in brackets indicate the respective standard
variances.
IV. DISCUSSION
Since this article reports on a preliminary study mainly
focusing on first impressions of the usability of Backstage,
only few statements can be made regarding the usefulness of
Backstage from an educational point of view. However, further
studies to investigate the educational usefulness of Backstage
are currently in the planning process.
A. Usability
The average time of nine minutes needed to get acquainted
with Backstage is rather short, which can be attributed to the
similarity to well-known platforms such as Twitter. However,
the variability around the average could be a ramification of the
participants‘ different notions of ―becoming acquainted with
Backstage‖, their backgrounds, or their levels of experience
with new media. For clarification, however, further
investigation is necessary.
Although the analysis of the questionnaires suggests that
the participants liked working with Backstage, in the open
items, the usability has been remarked to be in need of
improvements. Keeping in mind that on Backstage the
interaction design aims at triggering reflective thinking of
users, this remark did not come up unexpectedly. Furthermore,
the responds to the multiple choice questions suggest that most
participants did not find the deficiencies too disturbing. The
observation that every participant posted at least one message
during the presentation indicates that Backstage is convenient
to use and positively influences participation.
Whether learners are content with such a ―rectified‖ user
interaction design (satisfaction) or reject them, but at the same
time achieve better learning outcomes, is an essential aspect
that needs to be carefully considered. The learner may,
similarly to motivation [27], develop satisfaction due to
improved learning outcomes and positive feedback, despite
initial rejection stemming from potential cuts in usability.
However, one needs to clarify whether satisfaction and
impressions of usability may change in the long run.
The remarks regarding the quiz and the feedback
functionalities of Backstage suggest that the effects of ARSs on
active participation also apply to Backstage. To sum up, we
may conclude that the users get along well with Backstage,
even with the first (yet unfinished) prototypical
implementation. No fundamental criticism was given, but
further functionalities encouraged, e.g. the integration of
presentation slides in Backstage. For the next stage of
development we seized on this suggestion. The integration of
presentation slides, however, needed careful consideration to fit
the design rationale behind Backstage. This will be reported on
in a companion paper in the near future.
B. Questions
Comparing the experimental to the control group, we could
find indications for positive effects on the activity of the
participants when using Backstage. Thus, more questions were
raised on Backstage than being asked by the control group. The
same applies to feedback addressed to the presenter. It should
be noted that the participants of the control group only asked
questions after invitation, maybe to avoid interrupting the
presenter. In how far the increase of questions on Backstage
has impact on the learning results needs to be investigated in a
further study. However, we may assume that an increase of
activity positively influences learning success and, therefore,
motivation (e.g. [28]). However, as being stated by the
participants, the raising of more questions may also contribute
to distraction. Further investigation of this issue is required.
Regarding questioning, the findings of [23] could be
confirmed. In the experimental group the majority of questions
were factual questions, lesser were comprehension questions,
and the least integration questions. We consider this outcome
as an indication that questioning on Backstage needs guidance.
In a further step in Backstage‘s development we thus
reconsidered the interactions and the way they need to be made
feasible in order to account for the need of guidance. This issue
will also be presented in a forthcoming companion paper.
The high number of integration questions of the control
group neither confirms the findings of [23], nor does it meet
our expectations. However, this can be explained by the
following two circumstances: (1) each remark in the smaller
control group (compared to the experimental group) is in
average of a greater influence (compared to the experimental
group). (2) In the control group, most participants, i.e. three out
of five, were students of Psychology and Educational Sciences
and about to finish their studies. Since integration questions
highly depend on prior knowledge, the described constellation
of the group could explain the unexpected outcome. We cannot
rule out that the topics chosen for the presentation have
impinged on the participants‘ quality of questioning.
In summary, the findings of this preliminary study suggest
that Backstage has positive influence on the activation of
learners during presentations. Another aspect of active
participation that deserves investigation, besides the support in
asking questions, is the use of Backstage in large lecture
classes. Participants noted the risk of distraction when using
Backstage in a real lecture setting. Findings on the basis of the
Cognitive Load Theory support the participants‘ appraisal [29].
If predominant distraction can be verified in forthcoming
studies, then it is necessary to adapt lectures appropriately in
order to leverage Backstage. For example, this can be achieved
by conceding breaks for questioning on the backchannel. A
more thorough structuring of the lecture by means of macro-
scripts [30] may additionally support the versatile use of
Backstage. In using these macro-scripts we can also include
additional activities like collaboration in the lecture and
support them in various ways.
ACKNOWLEDGMENT
The authors would like to thank Jochen Rick at Educational
Technology, Saarland University, for the fruitful discussions.
REFERENCES
[1] A.W. Chickering and Z.F. Gamson, ―Seven Principles for Good
Practice in Undergraduate Education,‖ AAHE Bulletin, 1987, pp. 3-7.
[2] J. Cuseo, ―The Empirical Case against Large Class Size: Adverse
Effects on the Teaching, Learning, and Retention of First-Year
Students,‖ Journal of Faculty Development, vol. 21, 2007, pp. 5-21.
[3] H. Nassaji and G. Wells, ―What‘s the Use of ‗Triadic Dialogue‘?: An
Investigation of Teacher-Student Interaction,‖ Applied Linguistics, vol.
21, 2000, pp. 376-406.
[4] G. Mietzel, Pädagogische Psychologie des Lehrens and Lernens
[Educational Psychology of Teaching and Learning], Göttingen:
Hogrefe, 2003.
[5] B. Rogoff, ―Developing understanding of the idea of communities of
learners,‖ Mind, Culture, and Activity, vol. 1, 1994, pp. 209-229.
[6] S.C. Herring, ―Computer-Mediated Discourse,‖ Handbook of
Discourse Analysis, D. Tannen, D. Schiffrin, and H. Hamilton, eds.,
Oxford: Blackwell, in press.
[7] O.R. Kelm, ―The Use of Synchronous Computer Networks in Second
Language Instruction: A Preliminary Report,‖ Foreign Language
Annals, vol. 25, 1992, pp. 441-454.
[8] F. Bry, V. Gehlen-Baum, and A. Pohl, ―Promoting Awareness and
Participation in Large Class Lectures: The Digital Backchannel
Backstage,‖ Proceedings of the IADIS International Conference e-
society, Spain, Avila: 2011, pp. 27–34.
[9] R.H. Kay and A. LeSage, ―Examining the Benefits and Challenges of
using Audience Response Systems: A Review of the Literature,‖
Computers & Education, vol. 53, 2009, pp. 819 - 827.
[10] C. Fies and J. Marshall, ―Classroom Response Systems: A Review of
the Literature,‖ Journal of Science Education and Technology, vol. 15,
2006, pp. 101-109.
[11] R.W. Preszler, A. Dawe, C.B. Shuster, and M. Shuster, ―Assessment
of the Effects of Student Response Systems on Student Learning and
Attitudes over a Broad Range of Biology Courses,‖ CBE Life Science
Education, vol. 6, 2007, pp. 29-41.
[12] J. Nielsen, ―Usability 101: Introduction to Usability,‖ 2003.
[13] K.A. Ericsson, ―The Influence of Experience and Deliberate Practice
on the Development of Superior Expert Performance,‖ The Cambridge
Handbook of Expertise and Expert Performance, K.A. Ericsson, N.
Charness, P.J. Feltovich, and R.R. Hoffman, eds., Cambridge:
Cambridge University Press, 2006, pp. 683-703.
[14] J. Piaget and R. Fatke, Meine Theorie der geistigen Entwicklung [My
Theory about Intellectual Development], Beltz, 2003.
[15] L. Kobbe, A. Weinberger, P. Dillenbourg, A. Harrer, R. Hämäläinen,
P. Häkkinen, and F. Fischer, ―Specifying Computer-Supported
Collaboration Scripts,‖ International Journal of Computer-Supported
Collaborative Learning, vol. 2, 2007, pp. 211-224.
[16] I. Kollar, F. Fischer, and J.D. Slotta, ―Internal and External
Collaboration Scripts in Web-based Science Learning at Schools,‖
Proceedings of the Conference on Computer Support for Collaborative
Learning: Learning 2005: The Next 10 Years!, International Society of
the Learning Sciences, 2005, pp. 331–340.
[17] A. Weinberger, F. Fischer, and K. Stegmann, ―Computer-Supported
Collaborative Learning in Higher Education: Scripts for Argumentative
Knowledge Construction in Distributed Groups,‖ Proceedings of the
Conference on Computer Support for Collaborative Learning:
Learning 2005: The Next 10 Years!, Taipei, Taiwan: International
Society of the Learning Sciences, 2005, pp. 717–726.
[18] B. Ertl and H. Mandl, Kooperationsscripts als Lernstrategie
[Cooperation Scripts as a Learning Strategy], München: 2004.
[19] B. Rosenshine and C. Meister, ―Reciprocal Teaching: A Review of the
Research.,‖ Review of Educational Research, vol. 64, 1994, pp. 479-
530.
[20] V. Aleven, E. Stahl, S. Schworm, F. Fischer, and R. Wallace, ―Help
Seeking and Help Design in Interactive Learning Environments,‖
Review of Educational Research, vol. 73, 2003, pp. 277 -320.
[21] A.S. Palinscar and A. Brown, ―Reciprocal Teaching of
Comprehension-Fostering and Comprehension-Monitoring Activities,‖
Cognition and Instruction, vol. 1, 1984, pp. 117-175.
[22] A. King, ―Scripting Collaborative Learning Processes: A Cognitive
Perspective,‖ Scripting Computer-Supported Collaborative Learning,
F. Fischer, I. Kollar, H. Mandl, and J.M. Haake, eds., Boston, MA:
Springer US, 2007, pp. 13-37.
[23] A. King, ―Guiding Knowledge Construction in the Classroom: Effects
of Teaching Children How To Question and How To Explain,‖
American Educational Research Journal, vol. 31, 1994, pp. 358-68.
[24] D. Dansereau and A. O‘Donnell, ―Scripted Cooperation in Student
Dyads: A Method for Analyzing and Enhancing Academic Learning
and Performance.,‖ Interactions in Cooperative Groups. The
Theoretical Anatomy of Group Learning, New York: Cambridge
University Press, 1992, pp. 120-141.
[25] L.J. Cronbach, ―Coefficient Alpha and the Internal Structure of Tests,‖
Psychometrika, vol. 16, 1951, pp. 297-334.
[26] J. Cohen, Statistical Power Analysis for the Behavioral Sciences,
Routledge, 1988.
[27] E.L. Deci and R.M. Ryan, ―The Support of Autonomy and the Control
of Behavior,‖ Journal of Personality and Social Psychology, vol. 53,
1987, pp. 1024-37.
[28] E. Deci and R. Ryan, ―Die Selbstbestimmungstheorie der Motivation
und ihre Bedeutung für die Pädagogik [Self Theories on Motivation
and their Implications for Pedagogy],‖ Zeitschrift für Pädagogik, vol. 2,
1993.
[29] J. Sweller, ―Cognitive Load Theory, Learning Difficulty, and
Instructional Design,‖ Learning and Instruction, vol. 4, 1994, pp. 295-
312.
[30] P. Dillenbourg and P. Jermann, ―Designing Integrative Scripts,‖
Scripting Computer-Supported Collaborative Learning, F. Fischer, I.
Kollar, H. Mandl, and J.M. Haake, eds., Boston, MA: Springer US,
2007, pp. 275-301.
