Figure 7 - uploaded by Neil T. Heffernan
Content may be subject to copyright.
The cycle of perpetual evolution that stems from use of an educational platform as a collaborative research tool. An initial hypothesis comparing new methods to best known practices grows into a series of ideas that improve system content while benefiting students and advancing knowledge in the field. These ideas continue to grow until limited by the platform's capabilities. Infrastructure improvements validated by previous findings and inspired by research demand can then be made to return the cycle to a fresh starting point, where new hypotheses can be formed.
Source publication
Background/Context
Large-scale randomized controlled experiments conducted in authentic learning environments are commonly high stakes, carrying extensive costs and requiring lengthy commitments for all-or-nothing results amidst many potential obstacles. Educational technologies harbor an untapped potential to provide researchers with access to ext...
Context in source publication
Context 1
... growth and adaptation exemplifies perpetual evolution. Essentially, a simple hypothesis acts as the seed for an expanse of research that germinates through related ideas, eventually pushing the limits of the system until infrastructure improvements must be made to accommodate further questions -a cycle depicted in Figure 7. As the cycle begins, researchers form novel hypotheses that compare manipulations within the platform to best (known) practices (either comparable traditional classroom practices or previous versions of the platform's material). ...
Citations
... that made this study possible. If you want more information about the ASSISTments platforms, we suggest Heffernan and Heffernan (2014); for more about E-TRIALS, we suggest Ostrow et al. (2017). We would also like to thank Mia Farinella and Alyssa Glynn for assisting in coding for the study. ...
Background
Providing students with worked out problem solutions is a beneficial instructional technique in STEM disciplines, and studying examples that have been worked out incorrectly may be especially helpful for reducing misconceptions in students with low prior content knowledge. However, past results are inconclusive and the effects of incorrect worked examples alone or in combination with correct examples remains unclear.
Objectives
We aim to address whether studying incorrect examples alone or in combination with correct examples can support the reduction of students' fraction misconceptions, operationalized as errors made with high confidence.
Methods
After incorrectly solving a sampling problem, 130 students in 4th through 11th grade in the U.S. were randomly assigned to a condition in an online problem set focused on fraction equivalence. Students studied either single‐type worked examples (i.e., correct or incorrect; n = 49) or combination‐type worked examples (correct and incorrect; n = 41) or engaged in a problem‐solving control (n = 50).
Results
Studying a combination of correct and incorrect worked examples was as effective as the problem‐solving control with feedback at improving fraction equivalence knowledge and reducing the rate of high‐confidence errors. Students in both the combination condition and the problem‐solving with feedback condition outperformed those who studied either correct or incorrect worked examples alone.
Conclusions
Results support the inclusion of a combination of correct and incorrect worked examples when teaching students with low prior content knowledge. Studying a combination of example types within an online tutor helps to reduce misconceptions about fractions, a topic students commonly struggle with. A problem‐solving task with corrective feedback worked equally well.
... As part of this development, ASSISTments introduced AXIS [27], the E-TRIALS TestBed [19] and the TeacherASSSIST system [20] to allow educators and researchers to create and evaluate the effectiveness of different problem sets and on-demand assistance materials. In the context of massive open online courses (MOOCs), DynamicProblem [28] was introduced as a proof-ofconcept system that supports bandit algorithms [14] to collect feedback from students regarding the helpfulness of individual assistance materials. ...
We describe a fielded online tutoring system that learns which of several candidate assistance actions (e.g., one of multiple hints) to provide to students when they answer a practice question incorrectly. The system learns, from large-scale data of prior students, which assistance action to give for each of thousands of questions, to maximize measures of student learning outcomes. Using data from over 190,000 students in an online Biology course, we quantify the impact of different assistance actions for each question on a variety of outcomes (e.g., response correctness, practice completion), framing the machine learning task as a multi-armed bandit problem. We study relationships among different measures of learning outcomes, leading us to design an algorithm that for each question decides on the most suitable assistance policy training objective to optimize central target measures. We evaluate the trained policy for providing assistance actions, comparing it to a randomized assistance policy in live use with over 20,000 students, showing significant improvements resulting from the system’s ability to learn to teach better based on data from earlier students in the course. We discuss our design process and challenges we faced when fielding data-driven technology, providing insights to designers of future learning systems.
Keywordsintelligent tutoring systemsmulti-armed bandits
... Educators and researchers using these platforms can be solicited to create content for students, and the impact of this new content on students' learning can be evaluated empirically with data collected from students using the platforms. ASSISTments, an online learning platform that has been running since 2006, not only has evidence that the platform is effective at increasing students' learning gains (Roschelle et al., 2009), but has also been engaging with educators and researchers to create tutoring for struggling students (Ostrow et al., 2017;Patikorn & Heffernan, 2020). Within ASSISTments, students are assigned mathematics problem sets from one of multiple open access curricula, which they complete online through the ASSISTments Tutor. ...
... In recent years, the ASSISTments learning platform has developed a research platform that allows automatic deployment of studies across the web. This platform has been used by dozens of external researchers to carry out their studies in thousands of math classrooms [27]. Increased support for A/B studies has also been incorporated into MOOC platforms [30], leading to large-scale studies such as [18], which tested an intervention in over 200 courses with millions of enrolled learners. ...
Recent years have seen a surge in research conducted on intelligent online learning platforms, with a particular expansion of research conducting A/B testing to decide which design to use, and research using secondary platform data in analyses. This scientometric study aims to investigate how scholarship builds on these two different types of research. We collected papers for both categories-A/B testing, and educational data mining (EDM) on log data-in the context of the same learning platform. We then collected a randomized stratified sample of papers citing those A/B and EDM papers, and coded the reason for each citation. On comparing the frequency of citation categories between the two types of papers, we found that A/B test papers were cited more often to provide background and context for a study, whereas the EDM papers were cited to use past specific core ideas, theories, and findings in the field. This paper establishes a method to compare the contribution of different types of research on AIED systems such as interactive learning platforms.
... Machine learning can assess students' competencies, find weaknesses, and recommend additional materials. [17] AI could be implemented in colleges through the application of chatbots. Chatbots are available 24 hours a day and can help students in answering their questions instead of professors, no matter the time of day or night. ...
One year has passed since the world declared a state of a pandemic caused by COVID-19. The main objective was to limit the number of people in public places to mitigate the spread of the virus. The Serbian government also decided to suspend teaching in higher education institutions, secondary and primary schools, and the regular operation of preschool education institutions. The habitats of all the inhabitants of the world have completely changed in all spheres of life. Offices and classrooms have moved into family homes and the question now arises as to how much digitization has made it easier or harder to complete everyday responsibilities. However, they all had to come to terms with the fact and adapt in a possible way to the current situation. Teachers and parents faced a difficult challenge, but perhaps the most difficult was for pupils and students, among whom many have not had direct contact with their professors for a year and they master most of the material on their own. The aim of the research in this paper was to determine how much the way of conducting online teaching in Serbia has changed, compared to last year. An attempt was made to determine whether more positive or negative grades were given by students for the approach and manner of teaching in high schools and faculties.
... This makes innovative educational approaches or programs very hard to evaluate. Unless you can implement a program and sustain it at some scale, you cannot even study whether, and how, it is effective (Ostrow, Heffernan, & Williams, 2017). And the more innovative a program is, the less likely it is to fit with existing cultural routines, and thus the more difficult it is to get it up, running, and implemented at scale. ...
... Second, because at least part of the instruction is delivered to students at their computers or mobile devices, individual students-even those who are members of the same face-to-face class-can be randomly assigned to get different versions of the content. This makes it possible to do experimental research in the context of real educational settings, something that was nearly impossible to do before using research designs that required random assignment of schools and classrooms, not individual students, to different instructional conditions (Ostrow, Heffernan, & Williams, 2017). ...
Background/Context
Despite advances in the learning sciences, a persistent gap remains between research and practice.
Purpose/Objective/Research Question/Focus of Study
In this project, we develop and try out a new approach to education research and development in which researchers, designers/ developers, and instructors collaborate to continuously improve an online interactive textbook.
Intervention/Program/Practice
Using a “learn by doing” strategy, we first created a highly instrumented online textbook for introductory statistics. The design of our online book is based on the practicing-connections hypothesis: Instead of learning individual “bits” of information and then hoping that learners end up with transferable knowledge, we designed a curriculum to engage students in repeated practice of the connections—between core concepts, representations, and the world—that make knowledge transferable. The textbook includes more than 1,200 formative assessments, generating large amounts of data relevant to both the process and outcomes of college students learning of statistics. Using the affordances of technology, we then began working to apply routines and practices from open software development (Git) and improvement science (Toyota Kata) to build an improvement community focused on continuous improvement of the online book. We also are building a technology platform (CourseKata) to publish the book from markdown files stored on GitHub; distribute the book through widely used learning management systems; collect detailed student data and deliver it back to instructors and, in a de-identified form, researchers; and manage experiments that randomly assign different versions of content to different students within a single class, and then assess the effects on students’ learning.
Research Design
Our research design is a mixed-methods design research and improvement study. We gauge success through measures of process, outcome, and transfer.
Conclusions/Recommendations
We are at only the beginning of what we see as a lengthy project. We are encouraged, however, by our progress, and invite others—including researchers, designers/developers, and instructors—to join us in our improvement community focused on improving the transferable learning of basic statistical concepts at scale.
... The learning process will determine the learning outcomes (Miharja, Hindun, & Fauzi, 2019;Tauhid et al., 2014). One of the factors causing the low learning outcomes and student understanding occurs because it does not develop the appropriate learning characteristics (Ostrow et al., 2017). Science, especially biology as a life-science of learning, is carried out by contextualizing material content (Amin, 2016). ...
The purpose of this research was to develop a research-based reference book and applied it through the problem-based learning (PBL) using reference books in learning activities. The method for the reference book used the ADDIE development models. The reference book's effectiveness test results were conducted using a purposive sampling technique, with a total sample of 55 students and analyzed by t-test. The research results showed that the reference books of the development results were validated by media experts and material experts. The integration of the development of reference books in learning activities has an impact on increasing student understanding as demonstrated through the pre-test and post-test scores that differ significantly with t-value> t-table (4.149> 2.045) at p <0.05. Therefore, the development of the insect's immune system book with the PBL model increasthe student learning outcomes significantly. Also, this research has been able to improve students' ability and competence in solving problems in insects' immune system subject matter.
... AI is considered to be the most promising (breakthrough) digital technology of industry 4.0. The principles of its work and perspectives of its application in the modern economic practice are studied in the following works: Bogoviz, (2019), Boyd and Holton, (2018) (Burch and Miglani, 2018;Ginsberg et al., 2017;Ostrow et al., 2017;and Thomas and Nedeva, 2018). ...
Purpose
The purpose of the paper is to determine the perspectives of diversification of educational services in the conditions of industry 4.0 on the basis of artificial intelligence (AI) training, determine the consequences of this process for academic and teaching staff and to develop recommendations for its practical implementation.
Design/methodology/approach
The methods of horizontal, trends and regression analysis are used for studying social consequences of digital modernization of the markets of higher education (for academic and teaching staff). The research is performed by the example of modern Russia on the basis of the statistical data of Federal State Statistics Service and the International Telecommunication Union. The timeframe of the research covers academic years 2000/2001-2018/2019.
Findings
It is determined that digital modernization of the sphere of higher education stimulates the reduction of the universities’ need for academic and teaching staff and growth of their unemployment. However, further digital modernization of economy on the basis of breakthrough technologies of industry 4.0 will lead to creation of a new type of educational services that are provided within entrepreneurship of universities – AI training of business. This will ensure development of university entrepreneurship (and reduction of dependence of universities on state financing), as well as growth of the employment opportunities for experts (academic and teaching staff) in the sphere of AI, which will not depend on the number of students, but will be connected to demand for AI training from digital business.
Originality/value
The role of AI training in the structure of production business processes of a university in the conditions of industry 4.0 is determined. The necessity for state stimulation of development of digital business in the modern economic systems is substantiated. It is shown that government has to pay close attention to the issues of support in the sphere of AI and mass distribution of their results. Because of this, it will be possible to control social risks in the sphere of higher education.
... AS-SISTments (www.assistments.org), an online learning environment known for its embrace of educational research at scale [21,22], is currently hosting a series of randomized controlled trials examining learning interventions that target the innate needs defined by SDT with the goal of promoting integrated learning and thereby improving student performance. In support of this research, the present work attempts to validate four subscales of the IMI measuring students' perceptions of autonomy, belonging (or relatedness), competence, and interest/enjoyment within ASSISTments. ...
Online learning environments allow for the implementation of psychometric scales on diverse samples of students participating in authentic learning tasks. One such scale, the Intrinsic Motivation Inventory (IMI) can be used to inform stakeholders of students’ subjective motivational and regulatory styles. The IMI is a multidimensional scale developed in support of Self-Determination Theory [1, 2, 3], a strongly validated theory stating that motivation and regulation are moderated by three innate needs: autonomy, belonging, and competence. As applied to education, the theory posits that students who perceive volition in a task, those who report stronger connections with peers and teachers, and those who perceive themselves as competent in a task are more likely to internalize the task and excel. ASSISTments, an online mathematics platform, is hosting a series of randomized controlled trials targeting these needs to promote integrated learning. The present work supports these studies by attempting to validate four subscales of the IMI within ASSISTments. Iterative factor analysis and item reduction techniques are used to optimize the reliability of these subscales and limit the obtrusive nature of future data collection efforts. Such scale validation efforts are valuable because student perceptions can serve as powerful covariates in differentiating effective learning interventions.
