Conference PaperPDF Available

The Unexpected Pedagogical Benefits of Making Higher Education Accessible

Authors:

Abstract

Many ongoing efforts in online education aim to increase accessibility through affordability and flexibility, but some critics have noted that pedagogy often suffers during these efforts. In contrast, in the low-cost for-credit Georgia Tech Online Masters of Science in Computer Science (OMSCS) program, we have observed that the features that make the program accessible also lead to pedagogical benefits. In this paper, we discuss the pedagogical benefits, and draw a causal link between those benefits and the factors that increase the program's accessibility.
The Unexpected Pedagogical Benefits
of Making Higher Education Accessible
Abstract
Many ongoing efforts in online education aim to
increase accessibility through affordability and
flexibility, but some critics have noted that pedagogy
often suffers during these efforts. In contrast, in the
low-cost for-credit Georgia Tech Online Masters of
Science in Computer Science (OMSCS) program, we
have observed that the features that make the program
accessible also lead to pedagogical benefits. In this
paper, we discuss the pedagogical benefits, and draw a
causal link between those benefits and the factors that
increase the program's accessibility.
Author Keywords
Online education; accessibility; higher education.
ACM Classification Keywords
K.3.2. Computer and Information Science Education.
Introduction
Recently, there have been several efforts to make
higher education more accessible, with massive open
online courses—MOOCs—leading this charge [1];
however, MOOCs have proved to be divisive in higher
education, with some educators praising their potential
to make educational content accessible [7], while other
researchers have noted that MOOCs have not yet
delivered on many of their original promises [3, 8].
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all other
uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
L@S 2016, April 25-26, 2016, Edinburgh, Scotland UK
ACM 978-1-4503-3726-7/16/04.
http://dx.doi.org/10.1145/2876034.2893383
David Joyner
Udacity, Inc.
2465 Latham Street NW
Mountain View, CA 94040
david.joyner@udacity.com
Georgia Institute of Technology
85 5th Street NW
Atlanta, GA 30332
david.joyner@gatech.edu
Ashok K. Goel
Georgia Institute of Technology
School of Interactive Computing
85 5th Street NW
Atlanta, GA 30332
ashok.goel@cc.gatech.edu
Charles Isbell
Georgia Institute of Technology
School of Computer Science
801 Atlantic Dr. NW
Atlanta, GA 30332
isbell@cc.gatech.edu
L@S 2016 · Work in Progress
April 25–26, 2016, Edinburgh, UK
117
In the two years since the Georgia Tech online Masters
of Science in Computer Science debuted, we have
noted that in some ways, this program has delivered a
superior pedagogical experience to the traditional
residential program, specifically due to the factors that
made the program accessible. Here, we describe and
connect that accessibility, to the pedagogical benefits
we have observed. We conclude that with the right
planning, the relationship between accessibility and
pedagogy can be mutualistic, not parasitic.
OMSCS Program
The Georgia Tech OMSCS program was developed
through a partnership with AT&T and Udacity, and has
since grown to over 3000 students in two years. The
experience is similar to residential students: students
apply for admission, pay tuition, earn grades, and
complete the same assignments, projects, and exams.
The chief difference is that class material is delivered
via Udacity.com in lieu of classroom lectures.
Each OMSCS class is built around a MOOC developed by
Udacity. However, the for-credit offering of the class
includes individual instructor involvement, expert
human grading, complex projects, and other factors
that are uncommon in MOOCs. The courses themselves
are similar to distance learning classes, with the MOOC
as a textbook. The OMSCS, however, is built from the
ground-up to take advantage of the internet; it is
remarkably affordable compared to residential and
distance learning degrees; it is capable of admitting a
greater number of students; and it is an equivalent
degree to the residential program. Thus, the OMSCS is
large, flexible, affordable, and partially pre-produced
like MOOCs, but is also rigorous, accredited, and
subject to admissions and tuition like distance learning.
Increased Accessibility
The OMSCS program addresses four traditional barriers
to access: scheduling, geography, cost, and exclusivity.
Schedule Flexibility. OMSCS classes require no
synchronous activities and present most class
materials on the first day of the semester, allowing
students to succeed without taking time off work.
Geographic Flexibility. The OMSCS has no
residency requirement. Students may succeed
without uprooting their family or leaving their job
for two years. One student wrote, "I wrote and
submitted Assignment 1 from Liberia. Project 1 was
coded in Israel, and submitted in Palestine.
Assignment 4 was written and submitted from
Mongolia. And Project 2, from home in Portland."
Economic Affordability. The program costs
approximately $8000 including books and fees. The
low tuition and flexibility noted above dramatically
reduce the true costs of the program. As a result, a
plurality of students enroll in the program not for
career benefits, but for personal enjoyment.
Enhanced Inclusivity. The program has no
physical barrier to create an artificial student
capacity. As a result, while the on-campus program
fills up with only 10% of applicants, the OMSCS
accepts all qualified students. This creates a much
more diverse student body, with backgrounds and
experiences from outside computing.
Student Demographics
Table 1 compares the demographic information of the
online and residential sections of CS7637 in Fall 2014
[2]. Notably, online students tend to be older, more
educated, and more experienced. They are also more
likely to be male, domestic, and employed, although
Tools of the Program
Piazza: The forum and
de facto "classroom" for
OMSCS classes, for class
discussions, questions,
announcements, and
other virtual analogues of
the in-class experience.
Every OMSCS course
heavily leverages Piazza.
Peer Feedback: A tool
developed at Georgia
Tech for pairing students
for peer reviews. Peer
review is used as a
learning activity, not to
generate grades,
although experiments
have been performed on
using peer review to
support expert grading
[4]. Roughly one third of
the classes in the OMSCS
use Peer Feedback.
T-Square: The Sakai-
based learning
management system
used for assignment
submission and expert
evaluation.
Udacity: The host of the
program's prepared
lecture material, as well
as interactive exercises
and some class projects.
L@S 2016 · Work in Progress
April 25–26, 2016, Edinburgh, UK
118
the different gender ratios are due to the drastically
different international and domestic ratios between the
programs. Within each location, the gender ratios are
the same between the online and residential programs.
Pedagogical Benefits
The accessibility of the program has led to a student
population that is more intrinsically motivated to learn,
more experienced, and more professionally diverse. We
argue this unique student body and the structure of the
OMSCS combine to create unique pedagogical benefits.
Student-Initiated Discussions
In traditional residential classes, time is a finite
resource that must be budgeted carefully. Online, there
does not exist a finite amount of synchronous class
time to budget, and thus student-initiated discussions
do not compete with planned activities. In CS6460 in
Fall 2015, for example, the class concluded with over
500 discussions initiated by students.
Parallel Discussions
In residential classes, students must take turns and the
class (or groups within the class) may only discuss one
topic at a time. This makes individualization difficult: a
particular discussion may confuse the low-achieving
students or bore the high-achieving students. Online,
discussions may proceed in parallel, allowing students
to self-select which discussions are appropriate for their
ability. This also allows the volume of discussion to
expand; in CS6460 in Fall 2015, ~100 students and
instructors generated 11,000 posts in 17 weeks.
Extended and Self-Documenting Discussions
Because discussions in residential classrooms are time-
boxed within the assigned class time and conducted
verbally, the discussion must end with little left behind
except students' notes and memories. Online,
discussions have no need to come to an artificial end,
and all discussions inherently self-document. In
CS6460, for example, class projects are captured in
single long-running threads that document the project
team's progress across the entire semester.
Help-Seeking and Student-to-Student Interactions
In the OMSCS, students ask any questions on Piazza.
While the question may be implicitly targeted to the
instructor, students may enter and answer or follow-up.
Thus, student-to-student interaction dominates the
class experience. As a result, students receive rapid
answers and feedback from that diverse and
experienced student body instead of just instructors.
The average time to receive an answer is 30 minutes.
De Facto Teaching Assistants
With most classes enrolling anywhere from 200 to 500
students, there are reliably certain students who are
excessively helpful. To an outside observer, these
students would appear to be teaching assistants with
the extent to which they help their classmates.
Anecdotally, it is not unusual for students to note that
certain classmates dramatically improved their class
experience. This has also allowed instructors to identify
candidates for future teaching assistant positions.
Expert-Level Peer Feedback
As noted previously, many OMSCS use peer review [4]
to allow students to review classmates' assignments.
Although MOOCs often use peer feedback to generate
assignment grades [6], the OMSCS program instead
leverages the known pedagogical benefits of both
giving and receiving peer review [5]. The unique
Student Profiles
Online
Residential
Age
<24 15% 82%
25-34 47% 17%
>35 39% 1%
Gender
Female 10% 24%
Male 90% 76%
Location
Domestic 87% 32%
International 13% 68%
Level of Prior Education
Bachelor's 87% 94%
Master's 11% 6%
Ph.D. 2% 0%
Employment Status
Full-Time 90% 5%
Part-Time 5% 15%
None 5% 80%
Prior Coding Experience
<8 years 45% 90%
9-18 years 41% 10%
>18 years 14% 0%
Table 1: Comparison of online
and residential students in the
KBAI class in Fall 2014.
L@S 2016 · Work in Progress
April 25–26, 2016, Edinburgh, UK
119
student body present in the OMSCS means that experts
are participating as peers in these exercises as well.
Organic Student-Instructor Interaction
As instructors, we have been struck with how the online
forum structure creates a 24-hour classroom where we
may come, view discussions in their entirety, and
participate in every conversation, with everyone able to
listen. This has led to the most common positive
feedback we have received: students have commented
that the student-instructor interaction in these classes
exceeds any residential class they have taken.
Conclusion
In this paper, we have worked as participant-observers,
noting pedagogical benefits to students in the OMSCS
program not previously seen in residential classes.
OMSCS students agree: in Summer 2015, 90% of
online CS7637 students rated the course as better than
residential courses. CS6460, a course that relies on the
benefits outlined above, drew a similar assessment;
80% rated it as better than other courses. Students
frequently praise the teaching teams' involvement, the
chance to review classmates' work, and the active
student communities, all of which come from efforts to
increase accessibility. With the right circumstances and
planning, the relationship between accessibility and
pedagogy can be mutualistic, not parasitic.
Acknowledgements
The authors are instructors who have taught both
OMSCS and residential classes. We thank Joe Gonzales,
the creator of Peer Feedback and the staffs of the
Georgia Tech College of Computing, Georgia Tech
Professional Education, and Udacity for their support.
References
1. Daniel, J. (2012). Making sense of MOOCs: Musings
in a maze of myth, paradox and possibility. Journal
of Interactive Media in Education, 2012(3), Art-18.
2. Goel, A., & Joyner, D. (2016). An Experiment in
Teaching Artificial Intelligence Online. In D Haynes
(Ed.) International Journal for the Scholarship of
Technology-Enhanced Learning(1) 1.
3. Guzdial, M. (2014). Ubiquity symposium: MOOCs
and technology to advance learning and learning
research: limitations of MOOCs for computing
education: addressing our needs. Ubiquity,
2014(July), 1.
4. Joyner, D., Ashby, W., Irish, L., Lam, Y., Langston,
J., Lupiani, I., … Goel, A. (2016). Graders as Meta-
Reviewers: Simultaneously Scaling and Improving
Expert Evaluation for Large Online Classrooms. In
Proceedings from the Third ACM Conference on
Learning @ Scale.
5. Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor
or assessee: How student learning improves by
giving and receiving peer feedback. British Journal
of Educational Technology, 41(3), 525-536.
6. Lu, J., & Law, N. (2012). Online peer assessment:
effects of cognitive and affective feedback.
Instructional Science, 40(2), 257-275.
7. Milheim, W. D. (2013). Massive open online
courses (MOOCs): Current applications and future
potential. Educational Technology: The Magazine
for Managers of Change in Education, 53(3), 38-42.
8. Popenici, S. (2015). Deceptive Promises: The
Meaning of MOOCs. In McKay, E. (Ed.) Macro-Level
Learning Through Massive Open Online Courses
(MOOCs): Strategies and Predictions for the Future,
158-167.
L@S 2016 · Work in Progress
April 25–26, 2016, Edinburgh, UK
120
... In this study, we examine the above hypotheses in the context of the online course in knowledge-based artificial intelligence (AI) Joyner, Goel & Isbell 2016). We estimate that the popular AI course has been taken by more than 4000 students in the computer science program; thus, about half of all students in the program have taken the AI course, making the course a good testbed for our cognitive explanations. ...
... Undergraduates who enroll in the class are required to have taken several preliminary courses prior to their enrollment in the course due to the advanced class content. Additional information can be found in earlier works Joyner, Goel & Isbell 2016;Goel 2019). ...
... The survey results suggest that students regularly used self-regulation practices and had positive self-beliefs such as self-efficacy and intrinsic value. A majority of the students in the two terms analyzed in this study self-identified as mature adults, and we know from previous research (Joyner, Goel & Isbell 2016) that many of the students enrolling in the online program have external obligations, such as family and fulltime jobs, that impact the amount of time and attention they can dedicate to their education. ...
Preprint
Massive Open Online Courses (MOOCs) once offered the promise of accessibility and affordability. However, MOOCs typically lack expert feedback and social interaction, and have low student engagement and retention. Thus, alternative programs for online education have emerged including an online graduate program in computer science at a major public university in USA. This program is considered a success with over 9000 students now enrolled in the program. We adopt the perspective of cognitive science to answer the question why do only some online educational courses succeed? We measure learner motivation and self-regulation in one course in the program, specifically a course on artificial intelligence (AI). Surveys of students indicate that students self-reported assessments of self-efficacy, cognitive strategy use, and intrinsic value of the course are not only fairly high, but also generally increase over the course of learning. This data suggests that the online AI course might be a success because the students have high self-efficacy and the class fosters self-regulated learning.
... Dealing with the increasing costs of education is critical, given that lots of learners need access to vastly more affordable and quality education opportunities, including tuition-free course options. For instance, Mohapatra and Mohanty (2017) emphasized that the affordability of the offering is one of the prime predictors of the learners' perception, while Joyner et al. (2016) uncovered that providing more affordable courses has led to the learner population that is more intrinsically motivated to learn, more experienced, and more professionally diverse in some contexts. Institutions and platforms are thus under increasing pressure to provide more affordable learning without sacrificing optimal learning outcomes. ...
Article
Full-text available
Online education platforms play an increasingly important role in mediating the success of individuals’ careers. Therefore, while building overlying content recommendation services, it becomes essential to guarantee that learners are provided with equal recommended learning opportunities, according to the platform principles, context, and pedagogy. Though the importance of ensuring equality of learning opportunities has been well investigated in traditional institutions, how this equality can be operationalized in online learning ecosystems through recommender systems is still under-explored. In this paper, we shape a blueprint of the decisions and processes to be considered in the context of equality of recommended learning opportunities, based on principles that need to be empirically-validated (no evaluation with live learners has been performed). To this end, we first provide a formalization of educational principles that model recommendations’ learning properties, and a novel fairness metric that combines them to monitor the equality of recommended learning opportunities among learners. Then, we envision a scenario wherein an educational platform should be arranged in such a way that the generated recommendations meet each principle to a certain degree for all learners, constrained to their individual preferences. Under this view, we explore the learning opportunities provided by recommender systems in a course platform, uncovering systematic inequalities. To reduce this effect, we propose a novel post-processing approach that balances personalization and equality of recommended opportunities. Experiments show that our approach leads to higher equality, with a negligible loss in personalization. This paper provides a theoretical foundation for future studies of learners’ preferences and limits concerning the equality of recommended learning opportunities.
... However, it may be the case that in these at-scale programs, a different trend takes hold; scale itself may facilitate the emergence of a student community even in the absence of a deliberate design by an instructor. Joyner, Goel & Isbell [13] posit this development as among the "unexpected" benefits of scaling degree programs. They note that student activity is far higher in these large classes, allowing a student community to develop more emergently. ...
... While Coetzee et al. (2014) found that peer-to-peer assessment in chatrooms was not effective in supporting learning outcomes, Kulkarni et al. (2015) suggest pedagogical strategies for making collaboration in large online classes more effective. Our experience with online learning aligns more closely with the more positive view (Joyner, Isbell & Goel 2016). However, we are unaware of any AI technology for enhancing social interactions by connecting students to form micro-communities as Jill Watson SA does. ...
Preprint
We have developed an AI-powered socio-technical system for making online learning in higher education more accessible, affordable and achievable. In particular, we have developed four novel and intertwined AI technologies: (1) VERA, a virtual experimentation research assistant for supporting inquiry-based learning of scientific knowledge, (2) Jill Watson Q&A, a virtual teaching assistant for answering questions based on educational documents including the VERA user reference guide, (3) Jill Watson SA, a virtual social agent that promotes online interactions, and (4) Agent Smith, that helps generate a Jill Watson Q&A agent for new documents such as class syllabi. The results are positive: (i) VERA enhances ecological knowledge and is freely available online; (ii) Jill Watson Q&A has been used by >4,000 students in >12 online classes and saved teachers >500 hours of work; (iii) Jill Q&A and Jill Watson SA promote learner engagement, interaction, and community; and (iv). Agent Smith helps generate Jill Watson Q&A for a new syllabus within ~25 hours. Put together, these innovative technologies help make online learning simultaneously more accessible (by making materials available online), affordable (by saving teacher time), and achievable (by providing learning assistance and fostering student engagement).
... Professors in this online program have noted the significant impact that the online platform allows individual students to have [14]. The forum-based environment allows every student to effectively have unlimited time "in front of" the class to share their thoughts without taking time away from the planned lecture. ...
Conference Paper
Full-text available
In 2014, Georgia Tech launched an online campus for its Master of Science in Computer Science program. The degree, equal in stature and accreditation to its on-campus counterpart, offered a notably lower cost of attendance. Its design emphasized flexibility in both geography and time, allowing students from around the world to earn a highly-ranked MSCS without taking time off work or moving to campus. Five years later, the program enrolls over 8000 students per semester and has graduated 1500 alumni. It is believed to be the largest program of its kind and has received recognition from national organizations on professional education. Existing research on the pro-gram has focused on challenges and opportunities to scale that are agnostic to the content itself. In this reflection, we look at the creation and growth of the program as it relates to graduate-level CS instruction. In particular, we note a unique and powerful unity of content and platform: the online delivery of the program dovetails with the technical skillsets of the professors and students that it draws, putting both in the position to contribute and innovate.
... The guiding mission has always been that the Institute would not offer such programs if we could not be certain that their quality would be at least as high as our on-campus programs. The evidence has demonstrated that not only do students in the online programs have the same assessment and project scores as students in the traditional, face-to-face programs, they sometimes do even better (Goel and Joyner, 2017;Joyner et al., 2016). Further, there are tangential benefits. ...
Preprint
Full-text available
We outline how we detected programming plagiarism in an introductory online course for a master's of science in computer science program, how we achieved a statistically significant reduction in programming plagiarism by combining a clear explanation of university and class policy on academic honesty reinforced with a short but formal assessment, and how we evaluated plagiarism rates before SIGand after implementing our policy and assessment.
Article
Full-text available
MOOCs (Massive Open Online Courses) are the educational buzzword of 2012. Media frenzy surrounds them and commercial interests have moved in. Sober analysis is overwhelmed by apocalyptic predictions that ignore the history of earlier educational technology fads. The paper describes the short history of MOOCs and sets them in the wider context of the evolution of educational technology and open/distance learning. While the hype about MOOCs presaging a revolution in higher education has focussed on their scale, the real revolution is that universities with scarcity at the heart of their business models are embracing openness. We explore the paradoxes that permeate the MOOCs movement and explode some myths enlisted in its support. The competition inherent in the gadarene rush to offer MOOCs will create a sea change by obliging participating institutions to revisit their missions and focus on teaching quality and students as never before. It could also create a welcome deflationary trend in the costs of higher education. Explanatory Note During my time as a Fellow at the Korea National Open University (KNOU) in September 2012 media and web coverage of Massive Open Online Courses (MOOCs) was intense. Since one of the requirements of the fellowship was a research paper, exploring the phenomenon of MOOCs seemed an appropriate topic. This essay had to be submitted to KNOU on 25 September 2012 but the MOOCs story is still evolving rapidly. I shall continue to follow it. 'What is new is not true, and what is true is not new'. Hans Eysenck on Freudianism This paper is published by JIME following its first release as a paper produced as part of a fellowship at the Korea National Open University (KNOU). Both the original and this republication are available non-exclusively under Creative Commons Attribution (CC-BY). Apart from this note and minor editorial adjustments the paper is unchanged. Normal 0 false false false EN-GB X-NONE X-NONE
Chapter
Full-text available
Since 2011, massive open online courses (MOOCs) fired the imagination of the general public as well as the academics, university administrators and investors alike. This chapter is an analysis of the main promises and expectations associated with MOOCs in higher education. This analysis is largely informed by a literature review of new extensive research reports, press releases, media articles, scholarly blogs and academic papers. Considering costs and benefits, ethical aspects and the impact on the landscape of higher education, we explore if MOOCs stay consistent with their initial promises and rhetoric. This chapter continues the discussion on the book section ‘RIA and education practice of MOOCs,’ with the particular focus on the topic of ‘educational training design.’
Chapter
Since 2011, massive open online courses (MOOCs) fired the imagination of the general public as well as the academics, university administrators and investors alike. This chapter is an analysis of the main promises and expectations associated with MOOCs in higher education. This analysis is largely informed by a literature review of new extensive research reports, press releases, media articles, scholarly blogs and academic papers. Considering costs and benefits, ethical aspects and the impact on the landscape of higher education, the author explores whether MOOCs stay consistent with their initial promises and rhetoric. This chapter continues the discussion on the book section ‘RIA and education practice of MOOCs,’ with the particular focus on the topic of ‘educational training design.’
Conference Paper
Large classes, both online and residential, typically demand many graders for evaluating students' written work. Some classes attempt to use autograding or peer grading, but these both present challenges to assigning grades at for-credit institutions, such as the difficulty of autograding to evaluate free-response answers and the lack of expert oversight in peer grading. In a large, online class at Georgia Tech in Summer 2015, we experimented with a new approach to grading: framing graders as meta-reviewers, charged with evaluating the original work in the context of peer reviews. To evaluate this approach, we conducted a pair of controlled experiments and a handful of qualitative analyses. We found that having access to peer reviews improves the perceived quality of feedback provided by graders without decreasing the graders' efficiency and with only a small influence on the grades assigned.
Article
Computing education has some significant education challenges today. We aren't diverse enough, and we need to be able to develop more teachers. Despite popular opinion, the current generations of MOOCs don't meet those needs.
Article
This study investigated the relationship between the quality of peer assessment and the quality of student projects in a technology application course for teacher education students. Forty-three undergraduate student participants completed the assigned projects. During the peer assessment process, students first anonymously rated and commented on two randomly assigned peers' projects, and they were then asked to improve their projects based on the feedback they received. Two independent raters blindly evaluated student initial and final projects. Data analysis indicated that when controlling for the quality of the initial projects, there was a significant relationship between the quality of peer feedback students provided for others and the quality of the students' own final projects. However, no significant relationship was found between the quality of peer feedback students received and the quality of their own final projects. This finding supported a prior research claim that active engagement in reviewing peers' projects may facilitate student learning.
Article
This study reports the effects of online peer assessment, in the form of peer grading and peer feedback, on students’ learning. One hundred and eighty one high school students engaged in peer assessment via an online system—iLap. The number of grade-giving and grade-receiving experiences was examined and the peer feedback was coded according to different cognitive and affective dimensions. The effects, on both assessors and assessees, were analyzed using multiple regression. The results indicate that the provision by student assessors of feedback that identified problems and gave suggestions was a significant predictor of the performance of the assessors themselves, and that positive affective feedback was related to the performance of assessees. However, peer grading behaviors were not a significant predictor of project performance. This study explains the benefits of online peer assessment in general and highlights the importance of specific types of feedback. Moreover, it expands our understanding of how peer assessment affects the different parties involved. KeywordsPeer assessment–Online assessment–Peer feedback–Peer grading–Cognitive feedback–Affective feedback
Massive open online courses (MOOCs): Current applications and future potential. Educational Technology: The Magazine for Managers of Change in Education
  • W D Milheim
Milheim, W. D. (2013). Massive open online courses (MOOCs): Current applications and future potential. Educational Technology: The Magazine for Managers of Change in Education, 53(3), 38-42.
An Experiment in Teaching Artificial Intelligence Online
  • A Goel
  • D Joyner
Goel, A., & Joyner, D. (2016). An Experiment in Teaching Artificial Intelligence Online. In D Haynes (Ed.) International Journal for the Scholarship of Technology-Enhanced Learning(1) 1.
Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms
  • D Joyner
  • W Ashby
  • L Irish
  • Y Lam
  • J Langston
  • I Lupiani
  • A Goel
Joyner, D., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., … Goel, A. (2016). Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings from the Third ACM Conference on Learning @ Scale.