Content uploaded by Charles Elavsky
Author content
All content in this area was uploaded by Charles Elavsky on Nov 18, 2019
Content may be subject to copyright.
Putting twitter to the test: Assessing outcomes for student
collaboration, engagement and success_1284 1..15
Reynol Junco, C. Michael Elavsky and Greg Heiberger
Reynol Junco is a professor in the Department of Academic Development and Counseling at Lock Haven University.
His research investigates the effects of social media on student development, engagement and success. C. Michael
Elavsky is an assistant professor in the Department of Media Studies at the Pennsylvania State University. His
research addresses a wide range of issues including media studies, new media, pedagogy, the cultural industries and
music as cultural/political communication. Greg Heiberger is coordinator, advisor and instructor of Pre-Health
Professional Programs in the Biology & Microbiology Department at South Dakota State University. His main
research interests are innovative interventions which increase student engagement, success and retention. Address for
correspondence: Dr Reynol Junco, 104 Russell Hall, Lock Haven University, Lock Haven, PA 17745, USA. Email:
rey.junco@gmail.com
Abstract
Herein, we present data from two studies of Twitter usage in different postsecondary
courses with the goal of analyzing the relationships surrounding student engagement
and collaboration as they intersect learning outcomes. Study 1 was conducted with 125
students taking a first-year seminar course, half of who were required to use Twitter
while the other half used Ning. Study 2 was conducted with 135 students taking a large
lecture general education course where Twitter participation was voluntary. Faculty in
Study 1 engaged with students on Twitter in activities based on an a priori theoretical
model, while faculty in Study 2 only engaged students sporadically on the platform.
Qualitative analyses of tweets and quantitative outcomes show that faculty participation
on the platform, integration of Twitter into the course based on a theoretically driven
pedagogical model and requiring students to use Twitter are essential components of
improved outcomes.
Introduction
Twitter, a microblogging and social networking platform that allows users to post 140-character
updates, has revolutionized the social media landscape. In the 5 years since its introduction, Twitter
has garnered over 200 million users who send an estimated 155 million messages (“tweets”) per day
(Twopblog, 2011). While Facebook has been the most popular social networking site for college
students, educators have been more willing to use Twitter as part of their college courses possibly
because Twitter is primarily a microblogging platform and therefore more amenable to ongoing
public dialogue (Antenos-Conforti, 2009; Ebner, Lienhardt, Rohs & Meyer, 2010; Grosseck &
Holotescu, 2009; Junco, Heiberger & Loken, 2011; Schroeder, Minocha & Schneider, 2010; Smith &
Caruso, 2010). Indeed, a study of nearly 1400 faculty members found that 56% of faculty who were
Twitter users used Twitter as a learning tool in the classroom (Faculty Focus, 2010). Another study
of 1920 university faculty members found that even though more faculty were Facebook users, an
equal percentage used Facebook andTwitter in their courses (Moran, Seaman & Tinti-Kane, 2011).
Twitter as an educational intervention
Little research exists examining the efficacy of Twitter as a classroom learning tool. Mirvis, Sales
and Hackett (2006) found that the efficacy of new educational interventions, especially those
British Journal of Educational Technology (2012)
doi:10.1111/j.1467-8535.2012.01284.x
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA. Published by Blackwell Publishing, 9600 Garsington Road, Oxford
OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
involving technologies, are contingent upon a variety of factors including context, choice
of technology, methods for implementation and how the particular platforms intersect with
students’ technology acumen/familiarity. Additionally, Johnson (2011) examined the effect that
different types of Twitter posts had on instructor credibility, while Junco et al (2011) examined
the effect of educational uses of Twitter on student engagement and grades.
Student engagement
The construct of student engagement is defined as the time and effort students invest in edu-
cational activities that are empirically linked to desired college outcomes (Kuh, 2009) and encom-
passes various factors, including investment in the academic experience of college, interac-
tions with faculty, involvement in cocurricular activities and interaction with peers (Kuh, 2009;
Pascarella & Terenzini, 2005). Besides a study by Gunawardena et al (2009), which found that
student engagement and learning was enhanced by web 2.0 collaboration, few studies explore
the specific ways in which online collaboration is linked to engagement. However, a number of
studies have examined the links between the use of web 2.0 technologies like Twitter, collabora-
tion and student engagement.
Twitter, collaboration and student engagement
While little research exists examiningTwitter, a few studies have discovered correlations between
Facebook use and student engagement (see Heiberger & Harper, 2008; HERI, 2007; Junco,
2012a). The Heiberger and Harper (2008) and HERI (2007) studies found positive correlations
between social networking website use and single-item measures of college student engagement.
Conversely, the Junco (2012a) study found that while time spent using Facebook was positively
Practitioner notes
What is already known about this topic
• Student use of social media is integrally related to how students engage the world.
• Little research exists on how social media use is linked to college student engagement
in relation to academic outcomes.
• One study using a controlled design demonstrated a relationship between Twitter use
and student engagement.
What this paper adds
• An empirical comparison of two ways in which Twitter was differently integrated into
college courses.
• The utilization of quantitative and qualitative data to assess real-world academic
outcomes related to Twitter use.
• Evidence-based best practices for using Twitter in educationally relevant and produc-
tive ways.
Implications for practice and/or policy
• If integrating Twitter in their courses, faculty should require and structure its use
along educationally relevant criteria.
• To achieve the most effective results, faculty should have a theoretically driven
pedagogical basis for incorporating Twitter.
• Faculty should actively engage with students on the platform to obtain maximum
benefit.
2British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
related to time spent in cocurricular activities, it was negatively related to scores on a scale
designed to measure student engagement. Additionally, four studies have examined the relation-
ship between Facebook use and student learning; however, the results have been mixed with
two studies showing no relationship and two showing a negative relationship (Junco, 2012b;
Kirschner & Karpinski, 2010; Kolek & Saunders, 2008; Pasek, More & Hargittai, 2009).
Only the study by Junco et al (2011) used a controlled design to evaluate the effects of Twitter use
on student engagement, based on Chickering and Gamson’s (1987) seven principles for good
practice in undergraduate education: (1) student/faculty contact; (2) cooperation among stu-
dents; (3) active learning; (4) prompt feedback; (5) emphasizing time on task; (6) communicating
high expectations; and (7) respecting diversity. The findings from that study were noteworthy:
students in the Twitter group had significantly increased engagement and higher overall semester
grade point averages (GPAs) than the control group. Junco et al (2011) conclude that Twitter can
be repurposed for educationally relevant activities that have impacts on real-world academic
outcomes, namely student engagement (offline) and grades.
Research questions
RQ1: How does explicit encouragement of Twitter usage (through course design) impact the
relationship between student engagement and grades?
RQ2: Are there differences in collaboration between a class that requiresTwitter use and one that
does not?
RQ3: What are the effective elements of integrating Twitter into college courses?
Study 1: Examining the effects of requiring students to use Twitter in educationally relevant ways
In this study, we required students taking a first-year seminar course to use Twitter in the ways
that we indicated.
Method
Sample
We used a controlled experimental design to test the causal relationship between Twitter use and
both student engagement and grades. Four sections of a one-credit first-year seminar course for
pre-health professional majors were randomly assigned to the experimental group and three to
the control group. The experimental group used Twitter as part of the class, while the control
group used Ning, a service that allows users to create their own social networking site. None of
the students had used Twitter before participating in this study. Students were asked to participate
in the study by taking a pre- and posttest (the survey containing the engagement instrument).
Although participation was voluntary, participants could enter to win drawings of cash deposits
to their university card accounts throughout the semester.
Of the 132 students in the seven sections, 118 completed the study by taking both the pretest and
the posttest for an overall 89% participation rate in both groups. The final sample sizes were 65
students in the experimental group and 53 in the control group. The final sample was 92%
Caucasian, 5% Latino and 3% Native American. Sixty-two percent of the final sample were
female and 38% were male. The mean age of the sample was 18.2 with a standard deviation of
0.445. The age of the participants ranged from 17 to 20, although over 98% were between 18
and 19 years old. Thirty-two percent of the sample had at least one parent with a bachelor’s
degree.
Procedure
During the second week of the semester, sections received an hour-long training on how to use
either Twitter or Ning, supplemented by question-and-answer periods over the next few class
meetings. Right after the training sessions, both the experimental and control groups were sent
Twitter collaboration & engagement 3
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
links to an online survey that included the 19-item engagement scale as well as demo-
graphic items and items inquiring about a student’s technology use. The survey was
hosted on SurveyMonkey.com. The posttest instrument was sent during the last week of the
study.
The Twitter and Ning accounts were administered by two of the researchers, and activity on
both platforms happened exclusively outside of scheduled class time. Both Twitter and Ning were
used for the educationally relevant activities delineated in Junco et al (2011) that were developed
based on Chickering and Gamson’s (1987) seven principles for good practice in undergraduate
education:
1. Continuity for class discussions: Since the first-year seminar met only once a week for an
hour, Twitter and Ning were used to continue conversations begun in class. For instance,
students were asked to discuss the role of altruism in the helping professions.
2. Giving students a low-stress way to ask questions: Oftentimes, first-year and/or introverted
students are less comfortable asking questions in class. The dynamics of Twitter and Ning
allow students to feel more comfortable asking questions given the psychological barriers
inherent in online communication (Kruger, Epley, Parker, & Ng, 2005).
3. Book discussion: All first-year students read the same book as part of their first-year reading
program. The book Mountains Beyond Mountains (Kidder, 2004) focuses on Dr Paul Farmer’s
medical relief work in Haiti and was used to stimulate discussion about altruism and the
helping professions.
4. Class reminders: Since students all took a similar sequence of courses, we were able to
remind them of due dates for assignments and dates for exams in multiple classes.
5. Campus event reminders: At the beginning of the semester, we used SocialOomph to sched-
ule tweet reminders for the entire semester. These reminders included campus events, speak-
ers, concerts and volunteer opportunities.
6. Providing academic and personal support: We regularly posted information about academic
enrichment opportunities on campus (for instance, the location and hours for the tutoring
center), both periodically and in response to student requests for help. Additionally, we
provided encouragement and support when students reported things such as feeling
“stressed out” or being worried about exams.
7. Helping students connect with each other and with instructors: The “cohort effect”
or the intentional creation of learning communities is an important concept in ensuring
student persistence (Keup, 2005–2006). Additionally, student/faculty interaction is a
National Survey of Student Engagement (NSSE) factor related to student success (Kuh,
2002).
8. Organizing service-learning projects: As part of this course, students needed to participate
in a service learning volunteer opportunity. Students used Twitter or Ning to coordinate
volunteer times with each other.
9. Organizing study groups: With only a little encouragement from the authors via the
Twitter feed, students organized study groups for two of their more difficult courses, Chem-
istry and Biology.
10. Optional assignments: Students had the option of completing two assignments via Twitter
or Ning. The two assignments were:
a. Attend an upper-class student panel and post two questions they had for panelists.
b. Post reactions to their shadowing experience (where they shadowed a healthcare
professional in the community for a day).
11. Required assignments: Students in all sections had four required assignments during the
final 4 weeks of the semester. They were:
4British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
a. Students were required to post two statements and two replies to other students, dis-
cussing how reading Mountains Beyond Mountains has changed their ideas about people
who are less fortunate than they are.
b. Students were asked to watch a video of the Hurst family’s medical volunteer work at
the Pine Ridge Indian Reservation, read an online article about the Hursts, read the
article 100 People: A world portrait, and discuss their reactions by posting two statements
and two responses to other students’ reactions.
c. Students were asked to react to the statement that what Paul Farmer was doing in
Mountains Beyond Mountains was only a band-aid for the problem by posting two state-
ments and posting two responses to other students’ posts.
d. Students were asked to discuss their service project in the context of their future career.
They were also asked to compare and contrast their experience to that of Paul Farmer
and to use examples from their assigned readings.
Instrument and measures
The NSSE is an established instrument that was developed to measure engagement in education-
ally relevant activities and the desired outcomes of college (Kuh, 2009; Pascarella & Terenzini,
2005). Data from the NSSE exhibit acceptable psychometric properties (see Kuh, 2002), and
items focusing on good practices in undergraduate education consistently predict development
during the first year of college, based on multiple objective measures (Pascarella, Seifert & Blaich,
2009).
We developed an engagement instrument that uses 19 items from the NSSE (Appendix). These
items were selected because of their focus on academic and cocurricular engagement. We kept
the original coding of Likert scales from the NSSE; therefore, engagement scale items 1–14 were
coded using a 4-point Likert scale from “Never” (1) to “Very often” (4). Questions 15–17 were
presented as a 7-point Likert scale coded with responses 1 or 2 as “1,” 3 or 4 as “2,” 5 or 6 as “3,”
and 7 as “4.” Responses for question 18 were coded from “Very little” (1) to “Very much” (4).
Lastly, responses for question 19 were coded 1 for “Poor” through 4 for “Excellent.” An aggregate
engagement score was created using the sum of the individual items. The minimum score possible
on the instrument was 19 and the maximum was 76.
Students gave the researchers permission to access their academic records to obtain semester
GPAs as well as high school GPAs, to examine the differences in grades between the experimental
and control groups. Grades were measured on a 4.0 scale ranging from 0 for “F” to 4.0 for “A.” In
this scale, the lowest grade possible was 0–0.99 (an “F”), the next highest was 1.00–1.99 (a “D”),
the next highest was 2.00–2.99 (a “C”), the next highest was 3.00–3.99 (a “B”) and the highest
grade possible was 4.0 (an “A”).
Engagement instrument reliability and validity
Reliability analyses found that the data from both administrations of the survey were internally
consistent. Cronbach’s afor the pretest administration was 0.75, and for the posttest adminis-
tration, it was 0.81. The engagement instrument’s internal consistency was similar to the aof
0.85 reported by Hytten (2010) and the aof 0.82 reported by Kuh, Cruce, Shoup, Kinzie and
Gonyea (2008) using a different 19-item scale from the NSSE. Also, our instrument’s reliability
was similar to the aof 0.85 obtained by examining data on the 22 college activity items (Kuh,
2002). Lastly, the internal consistency estimates for these administrations were similar to the
0.80 found by Junco (2012a) using the same 19 items and a large sample.
Evidence was collected to support the construct validity of the 19-item engagement scale by
correlating the total score on the scale to the number of minutes students reported spending in
cocurricular activities on campus in a typical week. Because, theoretically, students who are more
Twitter collaboration & engagement 5
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
engaged in general spend more time on cocurricular activities, one way to show evidence of
construct validity of the engagement instrument would be if the scores on the engagement
instrument correlated somewhat (ie, shared some of the variance) with the amount of time
students spent in cocurricular activities. Indeed, there was a weak yet significant correlation
between scores on the engagement instrument and average minutes per week students reported
spending in cocurricular activities (Pearson’s r=0.26, p<0.01 at the pretest, and Pearson’s
r=0.33, p<0.001 at the posttest). Still, the correlation coefficients were modest, indicating that
our instrument measures more than just cocurricular engagement. This is congruent with
similar analyses conducted by Junco (2012a).
Statistics and qualitative analyses
To assess differences in engagement and grades, we used mixed-effects analysis of variance
(ANOVA) models with class sections nested within treatment groups. In order to assess changes
between the pre- and posttest measurement of engagement, we used difference scores as the
dependent variable, calculated by subtracting the total pretest score on the engagement instru-
ment from the total posttest score. We used PASW (SPSS) Statistics Version 17.0 for all analyses.
To evaluate how students interacted on Twitter, we used Leximancer software to extract the
themes in the corpus of tweets that was collected over the entire semester. Leximancer uses
algorithms that automatically analyze semantic and relational information in natural language
databases and creates top-level themes. Furthermore, Leximancer visualizes relationships
between major themes in a dataset by producing concept maps and assigning strength values (as
percentages) to each theme. Leximancer output shows the major themes, the concepts used to
make up those themes and how they are related.
Results
Results from the mixed-effects ANOVA model revealed that the Twitter group (M=5.12,
SD =6.69) had significantly higher difference scores than the control group (M=2.29,
SD =7.67) with F(1, 4.9) =12.12, p<0.05. We also conducted a mixed-effects ANOVA model
with pretest engagement scores as the dependent variable and found no preexisting differences in
engagement between the Twitter group (M=35.49, SD =6.84) and the control group
(M=38.17, SD =7.78; F(1, 4.9) =2.80, p=0.16). Therefore, the Twitter group’s engagement
score increased significantly more than the control group’s over the course of the semester and
this difference cannot be explained by preexisting engagement differences.
To examine the effect of Twitter use on student grades, we also used a mixed-effects ANOVA model
with sections nested within the treatment group. The dependent variable was overall first semes-
ter GPA. The semester GPAs of the Twitter group (M=2.79, SD =0.85) were significantly higher
than those of the control group (M=2.28, SD =1.08) with F(1, 4.9) =8.01, p<0.05. We also
conducted a mixed-effects ANOVA model with high school GPA as the dependent variable and
found no preexisting differences between the Twitter group (M=3.56, SD =0.48) and the
control group (M=3.43, SD =0.45; F(1, 4.9) =1.24, p=0.32). Like engagement score, the
Twitter group had higher overall semester GPAs than the control group, and this difference
cannot be explained by preexisting academic ability. Therefore, the Twitter intervention promoted
both student engagement and academic achievement.
Figure 1 shows the average number of tweets students sent each week of the semester. Figure 2
shows the themes extracted from the corpus of tweets with the Leximancer analyses. Students in
this study used Twitter primarily for cognitive matters; however, there were frequent conversa-
tions about affective matters (ie, being stressed about upcoming exams). The most prevalent
themes in the tweet data were, in order of strength:
6British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
1. Farmer: These were tweets where students examined issues raised by their discussion of a
common reading, Mountains Beyond Mountains (Kidder, 2004). This book describes Dr Paul
Farmer’s medical relief work in Haiti and was chosen to help students entering the helping
professions reflect on altruism. An example of a tweet from this theme was: “How does Dr
Farmer continue his work without feeling overwhelmed?”
2. People: Another theme that was an intended focus of this course was how students will help
others in their chosen professions. An example tweet was: “@User so in essence, if we can
experience that selflessness to help other countries, we will further our own drive to help
people here.”
3. Others: This theme was closely related to the People theme in that students discussed their
responsibility to others and their community. An example was “@User makes me realize how
much more I care about myself than others, makes me feel selfish.”
Study 2: Examining the effects of allowing students the option of using Twitter to collaborate through
methods of their choice
In Study 2, we allowed students to choose whether they wanted to use Twitter to collaborate on
course content, did not impose a framework of use and engaged with them intermittently on the
platform.
Method
Sample
One section of a large lecture general education communications course on media and demo-
cracy participated in this study. Students were given the option to utilize Twitter for the class.
Unlike Study 1 where participants in the control group used Ning, students opting not to use
Figure 1: Average number of tweets sent per student in each study over the course of the semester
Twitter collaboration & engagement 7
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
Twitter in Study 2 had no other formal alternative medium for virtual participation. No incen-
tives were offered. At the start of the class, 61% of students said they were familiar with Twitter,
43% said they had explored or used Twitter before and 4% said they had used it as part of a
previous class. These numbers are congruent with and representative of Twitter’s growth since
Study 1 was conducted.
Of the 179 students in the class, 135 completed the study by taking both the pretest and the
posttest for an overall 75% participation rate. Throughout the course of this study, 66 of the
participants used Twitter as part of the class, while 69 did not. The sample was 79% Caucasian,
4% African American, 10% Latino, 11% Asian and 1% Native American. Forty-seven percent of
the sample was female and 53% male. The mean age of the sample was 19.7 with a standard
deviation of 1.6. The age of the participants ranged from 18 to 28, although over 97% were
between 18 and 22 years old. Thirty-seven percent of the sample had at least one parent with a
bachelor’s degree. Because this course was open to students at all levels, there was variability in
class rank. Specifically, 30% were first-year students, 24% were sophomores, 29% were juniors
and 17% were seniors.
Figure 2: Leximancer concept map of themes, theme summary and list of ranked concepts from the Twitter
corpus of Study #1. The circles and their labels in the concept map represent the higher-level themes based on the
concepts in the dataset. The concepts are represented by the darker-colored words in the circles. The themes are
heat-mapped to indicate importance (ie, the most important theme appears in red, and the next most important in
orange, etc). The theme summary shows a list of all themes as well as their importance ranked by connectivity
and relevance. The ranked concept list shows the name of each concept, the number of times it was found in the
corpus of tweets and its relevance in the dataset
8British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
Procedure
During the third week of the semester, students were formally introduced across two 75-minute
class sessions to the technologies we were employing in the course (Twitter and Google docu-
ments), the reasons behind their implementation and the expectations the instructor had for
student participation (in the case of Twitter, voluntary contributions). The Twitter feed was
projected in class on the lecture hall screen once a week during full-class discussion of a topic,
although instructors made little explicit reference to the Twitter feed during discussions. No
specific directions or limitations were given regarding what the students should post, meaning the
open-forum nature of Twitter was fully embraced, including how they engaged one another in
this medium. Attempts were made (at least one class period per week) to fold commentary from
the feed back into the classroom discussions. While the instructors found it both stimulating and
an important addition to what is traditionally a unidirectional format in the large lecture hall, the
results were nonetheless mixed, most often depending on the quality of commentary posted to the
feed by the students.
Instrument and measures
For this study, we used the same 19-item scale based on the NSSE used in Study 1, included the
same additional questions in the online survey and were granted permissions to access academic
records as in Study 1.
Engagement instrument reliability and validity
Reliability analyses found that the data from both administrations of the survey were internally
consistent. Cronbach’s afor the pretest administration was 0.83, and 0.81 for the posttest
administration. This is congruent with the results from Study 1 as well as with data reported by
Hytten (2010), Kuh et al (2008) and Kuh (2002). Like in Study 1, we found that scores on the
engagement instrument both at the pretest and posttest correlated significantly with the hours
per week students reported spending in cocurricular activities (Pearson’s r=0.28, p<0.001 at
the pretest, and Pearson’s r=0.23, p<0.01 at the posttest). These correlations showed the same
pattern as in Study 1, suggesting that the instrument is an omnibus measure of both academic
and cocurricular engagement.
Statistics and qualitative analyses
To assess differences in engagement and course grades, we used one-way ANOVAs. In order to
assess changes between the pre- and posttest measurement of engagement, we used difference
scores calculated by subtracting the pretest from posttest score as the dependent variable. To
evaluate how students interacted on Twitter, we used Leximancer software to code the corpus of
tweets collected over the entire semester.
Results
To examine the effect of Twitter use on student engagement, we used a one-way ANOVA model
with whether students used Twitter as the independent variable. The dependent variable was the
difference score between the posttest administration of the engagement instrument and the
pretest administration. There was no difference between Twitter users (M=0.80, SD =6.22) and
nonusers (M=0.43, SD =6.52) on engagement difference scores with F(1, 133) =0.11,
p=0.74. We also conducted a one-way ANOVA with pretest engagement scores as the dependent
variable and found that there were no preexisting differences in engagement between Twitter
users (M=45.53, SD =7.28) and nonusers (M=47.06, SD =8.84; F(1, 133) =1.2, p=0.28).
To examine the effect of Twitter use on student grades, we also used a one-way ANOVA model
with whether students used Twitter as the independent variable.The dependent variable was final
course grade. Like with the analysis of engagement scores, there was no difference between
Twitter collaboration & engagement 9
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
Twitter users (M=94.27, SD =3.83) and nonusers (M=92.79, SD =7.77) on course grade
F(1, 133) =1.95, p=0.17. We also conducted a one-way ANOVA with high school GPA as the
dependent variable in order to assess for preexisting academic ability differences. There were no
differences between Twitter users (M=3.62, SD =0.49) and nonusers (M=3.57, SD =0.46;
F(1, 121) =0.36, p=0.56).
Figure 1 shows the average number of tweets students sent each week of the semester. Students
in Study 1 began tweeting at week 2 and tweeted through week 14, while students in Study 2
began at week 3 and tweeted through week 13. While students in Study 1 tweeted over a longer
period, students in Study 2 sent a slightly higher number of tweets each week with the exception
of weeks 12–14 when students in Study 1 were completing required assignments via Twitter.
Figure 3 shows the themes extracted from the corpus of tweets for Study 2. Students in this study
used Twitter almost exclusively for cognitive matters. The three most prevalent themes in the
tweet data were, in order of strength:
Figure 3: Leximancer concept map of themes, theme summary and list of ranked concepts from the Twitter
corpus of Study #2. The circles and their labels in the concept map represent the higher-level themes based on
the concepts in the dataset. The concepts are represented by the darker-colored words in the circles. The themes are
heat-mapped to indicate importance (ie, the most important theme appears in red, and the next most important
in orange, etc). The theme summary shows a list of all themes as well as their importance ranked by connectivity
and relevance. The ranked concept list shows the name of each concept, the number of times it was found in the
corpus of tweets and its relevance in the dataset
10 British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
1. People: The People theme was also one of the top three themes in Study 2; however, students
tweeted about people differently than in Study 1. An example tweet illustrates the difference: “I
don’t think most young people realize that non-presidential elections are important as well.”
In this corpus of tweets, the word People was used as a synonym for society or different
subsections of society (as illustrated by the example tweet). It was clear that these tweets were
less critically reflective and focused than the People theme in Study 1; however, the People
theme in this study was congruent with the focus of the course—modern society.
2. Real: In this study, students spent a good amount of time talking about connections between
media and “real” life. For instance, one student tweeted: “Does anyone think that violence in
video games can lead to violence in real life?” This theme was also congruent with a central
theme of the course—exploring the relationship (or lack thereof) between media consumption
and personal, psychological, and societal effects.
3. Class: Students tweeted about two things related to the Class theme: (1) Commentary about
the class as evidenced in this tweet: “favorite class of the semester,” and (2) arranging meet-
ings with groups such as “community 14 want to meet up after class?” Interestingly, this
theme was not related to the issue of social class, a theme that was discussed in a number of
the offline class sessions.
General discussion
RQ1: How does explicit encouragement of Twitter usage (through course design) impact the
relationship between student engagement and grades?
When students are required to use Twitter for a course and faculty engage with them regularly on
the platform, there is an increase in student engagement and grades that was not seen when
students were allowed to choose whether or not to use Twitter and when faculty rarely interacted
with them on the platform. Study 1 established particular parameters for collaboration that
implicitly facilitated and motivated students in ways that were not reproduced in Study 2, pro-
ducing positive outcomes related to student learning. Whether these outcomes are linked to
student motivations regarding grades is not clear. What is clear is that the different results
between Study 1 and 2 cannot be explained by preexisting differences in engagement and aca-
demic ability in the groups that were required to use Twitter (Study 1) and who chose to use
Twitter (in Study 2). This finding is of particular interest as, at least in these two samples,
preexisting differences in engagement and academic ability cannot explain either Twitter adop-
tion or positive outcomes derived through Twitter use.
RQ2: Are there differences in collaboration between a class that requires Twitter use and one that
does not?
While qualitative analyses of tweets show that students in both studies discussed and collaborated
on course content, such collaboration was not directly related to improved outcomes in Study 2.
While only two of the educationally relevant activities in Study 1 specifically called for collabo-
ration (organizing service learning projects and study groups), the amount of collaboration went
beyond these activities. In Study 2, the collaboration among students was incidental as there were
no requirements to do so. This finding provides evidence to support the idea that how instructors
use Twitter (for example, to engage with students by answering questions, encouraging discus-
sions and providing support) is an important factor in engagement and achievement gains seen
with the intervention in Study 1. Specifically, faculty who are more engaged on the platform with
their students will see greater gains in academic outcomes.
Junco et al (2011) suggested that future research take steps to evaluate the proportion of the
variance that is due to the technology and the proportion due to the instructor in outcomes—we
have done so here and found evidence, albeit limited, that both how Twitter is integrated into a
Twitter collaboration & engagement 11
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
college course and how faculty interact with students on Twitter are important elements in such
a design. This could be attributable to many factors including course requirements (students feel
more compelled to participate in the forum in relation to achieving higher grades, the instructor’s
presence, etc), discursive novelty (students are able to engage with one another—not least of
which, the instructor—in compelling ways), self-disclosure (perceived learning is linked to the
perspectives one contributes), and, perhaps most important, a reconfigured sense of empower-
ment in relation to shaping the course design and meaningful outcomes.
Our data suggest that there may also be pedagogical affordances to Twitter as compared to Ning
and other technologies. For instance, Twitter lends itself to a more engaging and continuous
conversation than Ning, and this factor alone could have drawn students in the Twitter group in
Study 1 to be more engaged. Ning updates resembled static posts such as those found on learning
management system discussion boards and they received few student responses, while Twitter
updates garnered multiple responses. Further research will want to evaluate the dynamics of
each platform and include additional control groups to evaluate the pedagogical potential of
each, separate from how they are used.
RQ3: What are the effective elements of integrating Twitter into college courses?
The data from these studies show three effective elements of integrating Twitter into college
courses that can be considered best practices:
1. Requiring students to use Twitter as part of the course is important in affecting academic
outcomes. Students in Study 2 who were not required to use Twitter did not see the engage-
ment and academic benefits experienced by students in Study 1.
2. Twitter should be integrated into the course in educationally relevant ways. For Study 1, we
used a theoretical framework (Chickering & Gamson’s (1987) seven principles for good prac-
tice in undergraduate education) that guided Twitter integration. We did not use this frame-
work for Study 2 instead allowing students to use Twitter in emergent and natural ways.
Therefore, having a theoretical reason to use Twitter and implementing that reason into the
course pedagogy will maximize the benefits achieved.
3. Faculty engagement on the platform is essential in order to impact student outcomes. In Study
1, faculty were actively engaging students on Twitter, while in Study 2, faculty maintained a
more laissez-faire attitude. Interestingly, the students who used Twitter in Study 2 showed a
trend toward higher grades that may have been significant with a larger sample size. Future
research should examine if there are additional benefits received by students who are not
required to use Twitter and whose faculty do not engage with them on the platform.
Limitations
The findings of both studies need to be expanded and replicated with larger samples, more diverse
student populations and a variety of courses. Using a single model of student engagement is an
additional limitation of these studies. Specifically, the Astin (1984) model is but one way to think
about student engagement, and the NSSE is but one way to measure it (see Finn, 1993). A
potential confounding factor for future research may be differences in adoption rates. In these
studies conducted in two different classrooms at two separate times, we saw none of the students
in the earlier study report prior Twitter use, while 43% in the later study reported either prior use
or exploration. These differences may be because more students will adopt Twitter as time passes,
differences in institutional cultures or both.
A further limitation relates to the challenges of framing/assessing collaborative learning out-
comes. For example, while Kuh (2009) delineates engagement as occurring in class and out of
class, the parameters of Twitter’s virtual space clearly challenge such distinctions and any easy
definition for understanding the educational praxis occurring across these delineations. Lastly,
12 British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
providing incentives for students in Study 1 may have increased their motivation to be engaged
as compared to students in Study 2. Conversely, the extent of one’s investment in the activity
related to Twitter might also have motivated positive feedback mechanisms extending beyond the
instrumentality of using the platform simply to successfully achieve expected grading outcomes.
Future studies should incorporate into their design structural considerations to offset the poten-
tial reproduction of the Hawthorne effect (McCarney et al 2007).
Conclusion
The incorporation of new technologies into the contemporary classroom remains an important
and compelling development with regard to producing more effective learning strategies and
outcomes. This study demonstrates that the design of teaching strategies and practices related to
virtual engagement and collaboration is instrumental to achieving positive educational outcomes. It
also underscores the need for contemporary students to improve their capacity to initiate self-
directed, collaborative practices as a means to more effectively take ownership of their learning.
References
Antenos-Conforti, E. (2009). Microblogging on Twitter: social networking in intermediate Italian classes. In
L. Lomicka & G. Lord (Eds), The next generation: Social networking and online collaboration in foreign language
learning Calico Monograph Series, No. 9. (pp. 59–90). San Marcos, TX: CALICO Publications.
Astin, A. (1984). Student involvement: a developmental theory for higher education. Journal of College
Student Personnel,25, 4, 297–308.
Chickering, A. W. & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education.
AAHE Bulletin, 3–7.
Ebner, M., Lienhardt, C., Rohs, M. & Meyer, I. (2010). Microblogs in higher education—A chance to facilitate
informal and process oriented learning? Computers & Education,55, 92–100.
Faculty Focus (2010). Twitter in Higher Education 2010: Usage Habits and Trends of Today’s College
Faculty. Retrieved January 20, 2012, from http://www.facultyfocus.com/free-reports/twitter-in-higher-
education-2010-usage-habits-and-trends-of-todays-college-faculty/
Finn, J. D. (1993). School engagement and student at risk. Washington, DC: National Center for Education
Statistics. Retrieved July 21, 2011, from http://nces.ed.gov/pubs93/93470.pdf
Grosseck, G. & Holotescu, C. (2009). Can we use Twitter for educational activities? Proceedings of the 4th
International Scientific Conference: eLearning and Software for Education, Bucharest, Romania. Retrieved July
6, 2011, from http://adlunap.ro/eLSE_publications/papers/2008/015.-697.1.Grosseck%20Gabriela-
Can%20we%20use.pdf
Gunawardena, C., Hermans, M. B., Sanchez, D., Richmond, C., Bohley, M. & Tuttle, R. (2009). A theoretical
framework for building online communities of practice with social networking tools. Educational Media
International,46, 1, 3–16.
Heiberger, G. & Harper, R. (2008). Have you Facebooked Astin lately? Using technology to increase student
involvement. In R. Junco & D. M. Timm (Eds), Using emerging technologies to enhance student engagement.
New directions for student services issue #124 (pp. 19–35). San Francisco, CA: Jossey-Bass.
Higher Education Research Institute (HERI) (2007). College Freshman and Online Social Networking Sites.
Retrieved November 12, 2007, from http://www.gseis.ucla.edu/heri/PDFs/pubs/briefs/brief-091107-
SocialNetworking.pdf
Hytten, K. J. (2010). Facebook’s contribution to educationally purposeful activities and college student
engagement (Unpublished PhD Thesis, The University of Vermont, Burlington).
Johnson, K. (2011). The effect of Twitter posts on students’ perceptions of instructor credibility. Learning,
Media and Technology,36, 1, 21–38.
Junco, R. (2012a). The relationship between frequency of Facebook use, participation in Facebook activities,
and student engagement. Computers & Education,58, 1, 162–171. doi: 10.1016/j.compedu.2011.
08.004.
Junco, R. (2012b). Too much face and not enough books: the relationship between multiple indices of
Facebook use and academic performance. Computers in Human Behavior,28, 1, 187–198. doi:10.1016/
j.chb.2011.08.026.
Junco, R., Heiberger, G. & Loken, E. (2011). The effect of Twitter on college student engagement and grades.
Journal of Computer Assisted Learning,27, 2, 119–132. doi:10.1111/j.1365-2729.2010.00387.x.
Keup, J. R. (2005–2006). The impact of curricular interventions on intended second year re-enrollment.
Journal of College Student Retention,7, 61–89.
Twitter collaboration & engagement 13
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
Kidder, T. (2004). Mountains beyond mountains: the quest of Dr. Paul Farmer, a man who would cure the world.
New York, NY: Random House.
Kirschner, P. A. & Karpinski, A. C. (2010). Facebook and academic performance. Computers in Human
Behavior,26, 1237–1245.
Kolek, E. A. & Saunders, D. (2008). Online disclosure: an empirical examination of undergraduate Facebook
profiles. NASPA Journal,45, 1, 1–25.
Kruger, J., Epley, N., Parker, J., & Ng, Z. (2005). Egocentrism over e-mail: can we communicate as well as we
think? Journal of Personality and Social Psychology,89, 6, 925–936.
Kuh, G. D. (2002). The national survey of student engagement: conceptual framework and overview of psychometric
properties. Bloomington: Center for Postsecondary Research, Indiana University. Retrieved January 20,
2012, from http://nsse.iub.edu/pdf/psychometric_framework_2002.pdf
Kuh, G. D. (2009). What student affairs professionals need to know about student engagement. Journal of
College Student Development,50, 6, 683–706.
Kuh, G. D., Cruce, T. M., Shoup, R., Kinzie, J. & Gonyea, R. M. (2008). Unmasking the effects of student
engagement on first-year college grades and persistence. Journal of Higher Education,79, 5, 540–563.
McCarney, R., Warner, J., Iliffe, S. S., van Haselen, R., Griffin, M. & Fisher, P. (2007). The Hawthorne effect:
a randomised, controlled trial. BMC Medical Research Methodology,7, 30, doi:10.1186/1471-2288-7-30.
Mirvis, P., Sales, A. & Hackett, E. (2006). The implementation and adoption of new technology in organi-
zations: the impact on work, people, and culture. Human Resource Management,30, 1, 113–139.
Moran, M., Seaman, J. & Tinti-Kane, H. (2011). Teaching, learning, and sharing : how today’s higher education
faculty use social media. Pearson Learning Solutions, the Babson Survey Research Group, and Converseon
Research Report. Retrieved July 6, 2011, from: http://www.pearsonlearningsolutions.com/educators/
pearson-social-media-survey-2011-bw.pdf
Pascarella, E. T., Seifert, T. A. & Blaich, C. (2009). Validation of the NSSE benchmarks and deep approaches to
learning against liberal arts outcomes. Iowa City: University of Iowa Center for Research on Undergraduate
Education.
Pascarella, E. T. & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco,
CA: Jossey-Bass.
Pasek, J., More, E. & Hargittai, E. (2009). Facebook and academic performance: reconciling a media sensa-
tion with data. First Monday,14,5.
Schroeder, A., Minocha, S. & Schneider, C. (2010). The strengths, weaknesses, opportunities, and threats of
using social software in higher and further education teaching and learning. Journal of Computer Assisted
Learning,26, 3, 159–174.
Smith, S. D. & Caruso, J. B. (2010). ECAR study of undergraduate students and information technology. Research
Study, Vol. 6. Boulder, CO: EDUCAUSE Center for Applied Research. Retrieved July 6, 2011, from: http://
www.educause.edu/Resources/ECARStudyofUndergraduateStuden/217333
Twopblog (2011). What does 300 million registered Twitter accounts mean? Retrieved July 6, 2011, from:
http://www.twopblog.com/2011/05/what-does-300-million-registered.html
Appendix
Engagement Instrument
In your experiences at ______________ University during the current school year, how often have
you done each of the following?
Very often, Often, Sometimes, Never
1. Asked questions in class or contributed to class discussions.
2. Participated in a community-based project (e.g., service learning) as part of a regular course.
3. Discussed grades or assignments with an instructor.
4. Talked about career plans with a faculty member or advisor.
5. Discussed ideas from your readings or classes with faculty members outside of class.
6. Worked with faculty members on activities other than coursework (committees, orientation,
student life activities, etc).
7. Discussed ideas from your readings or classes with others outside of class (students, family
members, co-workers, etc).
8. Had serious conversations with students of a different race or ethnicity than your own.
14 British Journal of Educational Technology
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
9. Had serious conversations with students who are very different from you in terms of their
religious beliefs, political opinions, or personal values.
During the current school year, about how often have you done each of the following?
Very often, Often, Sometimes, Never
10. Attended an art exhibit, play, dance, music, theater, or other performance.
11. Exercised or participated in physical fitness activities.
12. Participated in activities to enhance your spirituality (worship, meditation, prayer, etc).
13. Tried to better understand someone else’s views by imagining how an issue looks from his or
her perspective.
14. Have you done or plan to do community service or volunteer work before you graduate from
______________ University?
Done, Plan to do, Do not plan to do, Have not decided
Mark the response that best represents the quality of your relationships with people at
______________ University.
15. Relationships with other students.
Unfriendly, Unsupportive, Sense of Alienation ...... Friendly, Supportive, Sense of Belonging
16. Relationships with faculty members
Unavailable, Unhelpful, Unsympathetic ...... Available, Helpful, Sympathetic
17. Relationships with administrative personnel and offices
Unhelpful, Inconsiderate, Rigid ...... Helpful, Considerate, Flexible
18. To what extent does ______________ University emphasize attending campus events and
activities (special speakers, cultural performances, athletic events, etc)
Very much, Quite a bit, Some, Very Little
19. How would you evaluate your entire educational experience at ______________ University?
Excellent, Good, Fair, Poor
Items used with permission from The College Student Report, National Survey of Student
Engagement, Copyright 2001-11 The Trustees of Indiana University
Twitter collaboration & engagement 15
© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.
