No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Using Artifacts to Characterize Reform-Oriented Instruction: The Scoop Notebook and Rating Guide. CSE Technical Report 707
Abstract
This document includes the final data collection and scoring tools created by the "Scoop" project, a five-year project funded through the Center for Evaluation, Standards,and Student Testing (CRESST), to develop an alternative approach for characterizing classroom practice. The goal of the project was to use artifacts and related materials to represent classroom practice well enough that a person unfamiliar with the teacher or the lessons can make valid judgments about selected features of practice solely on the basis of those materials. The artifacts and other materials were collected in a binder called the Scoop Notebook. Thus, the project sought to answer the question, "Can accurate judgments about reform-oriented instructional practice be made based on the classroom artifacts and teacher reflections assembled in the Scoop Notebook?" This document describes the Scoop Notebook and the rating guides, gives instructions for assembling the materials and explaining the process to teachers, and discusses two potential uses of the Scoop Notebook--as a tool to characterize classroom practice or as a tool for teacher professional development. The appendices present the final versions of the Scoop Notebook and rating guide for both mathematics and science. (Contains 3 footnotes.)
... This research required a systematic approach and consistent protocol that could accommodate case studies of pedagogical practice across multiple sites, to allow for the pilot to be scaled up if conditions were amenable. 1 We drew on the artefactled design of US scholars Borko, Martinez, Stecher and others-variously called "Scoop", "e-portfolios" and "artefact packages"-developed for large-scale evaluation of middle school Science and Mathematics pedagogies (Borko et al. 2005(Borko et al. , 2007Martinez et al. 2012). Their approach was compatible with the intention to inquire into the minutiae of practice and to foreground the agency of teachers as designers of teaching and learning. ...
... Whilst Borko et al. collected artefacts across 5-7 days of instruction in Maths or Science, we asked teachers to upload artefacts across a sequence of ten lessons in English. Borko et al. (2007) asked teachers to select three categories of artefact in every lesson: materials produced before class (e.g. tasks, rubrics), materials generated during class (e.g. board notes) and materials generated outside class (e.g. ...
... Our study offered these as examples, but asked teachers to choose one artefact each lesson of any of these types that reflected their approach to teaching writing. Reflecting the times, Borko et al. (2007) initially used hard copy binders, and later e-portfolios in a purpose-built online portal (Martinez et al. 2012). For this study, password-protected folders were created in Cloudstor, the online storage service owned by Australian and New Zealand universities. ...
In secondary schools, English teachers are often made responsible for writing results in national testing. Yet there have been few studies that focussed on this key group, or on how pedagogical practices have been impacted in the teaching of writing in their classrooms. This study investigated practices of English teachers in four secondary schools across different states, systems and regions. It developed a novel method of case study at a distance that required no classroom presence or school visits for the researchers and allowed a multi-sited and geographically dispersed design. Teachers were invited to select classroom artefacts pertaining to the teaching of writing in their English classes, compile individualised e-portfolios and reflect on these items in writing and in digitally conducted interviews, as well as elaborating on their broader philosophies and feelings about the teaching of writing. Despite and sometimes because of NAPLAN, these teachers held strong views on explicit teaching of elements of writing, but approached these in different ways. The artefacts that they created animated their teaching practices, connected them to their students and their subject, suggested both the pressure of externally driven homogenising approaches to writing and the creative individualised responses of skilled teachers within their unique contexts. In addition to providing granular detail about pedagogical practices in the teaching of writing in the NAPLAN era, the contribution of this paper lies in its methodological adaptation of case study at a distance through teacher-curated artefact portfolios that enabled a deep dive into individual teachers’ practices.
... Despite the sustained efforts and calls to reform teaching practices, national survey data show that U.S. science teachers are still using traditional and typical teaching practices. The 2012 national survey of Science and Mathematics Education (3,701 science teachers from 1,403 schools from grades [1][2][3][4][5][6][7][8][9][10][11][12] showed that roughly 96% of the middle school teachers explained science ideas to the entire class and 92% of them had the entire class engage in discussions [1]. Small group work in science classrooms was implemented by approximately 79% of middle school science teachers [1]. ...
... Ref. [6] also valued "hands-on activities" using "scientific resources" such as lab materials and technology as an effective science teaching practice. Ref. [9] proposed ten elements of science teaching practices based on the vision of inquiry-oriented instruction in the NSES [8] and the Mosaic-II project [10]. Table 1 shows the detailed descriptions of each component, adapted from [9]. ...
... Ref. [9] proposed ten elements of science teaching practices based on the vision of inquiry-oriented instruction in the NSES [8] and the Mosaic-II project [10]. Table 1 shows the detailed descriptions of each component, adapted from [9]. ...
... The extensive collection of a wide variety of artifacts, both teacher-created (lesson plans, reflections, etc.) and teacher-collected (photos of classroom set-up, classroom teaching videos and completed surveys, peer analysis, assessments, etc.) provided a rich description of teachers' practices needed to recognize emerging or existing patterns of the classroom community, conceptualize underlying social meaning or importance of those patterns, and contextualize the observed actions in terms of existing literature, similar practices, or historical events (McCutcheon, 1981;Parker, Bonney, & Schamberg, 2013). Borko et al., (2007) described the goal of their work: "To use artifacts and related materials to represent classroom practice well enough that a person unfamiliar with the teacher or the lessons can make valid judgments about selected features of practice solely based on those materials" (p. 7). ...
... While the Scoop has been found to be a valid and reliable means for classroom observation (Borko et al., 2007), the integration of the Scoop Notebook with the online instructional strategies course resulted in a process that in reality engaged the teachers in an action research process. As systematic inquiry conducted by the teachers into their own classroom instruction, the primary beneficiaries of the research action were the teachers, a characteristic of action research (Mertler, 2009). ...
... Scoop: A method for gathering a 'scoop' of an educator's work in the classroom setting (much like a scientist wishing to analyze soil might gather from a particular plot of land) (Borko et al., 2007). ...
As teacher education leverages online instructional environments, an important research focus is how best to re-conceptualize mathematics inservice teacher classroom observations. This chapter proposes an alternative methodology to the structure of traditional observations, where the Scoop Notebook provides a window into mathematics teachers' Technological Pedagogical Content Knowledge (TPACK), specifically their TPACK-of-practice. In this descriptive, cross-case study, a re-designed Scoop process frames teachers ' engagement in classroom practices, putting into practice their scholarly formal knowledge and developing in-depth, rich reflections from their classroom actions and artifacts. This study illustrates how embedding the Scoop process into a graduate instructional strategies course as part of a Master's degree curriculum results in engaging teachers in action research where they use artifacts as objects to think with for transforming their TPACK for integrating technologies in teaching mathematics.
... Artifact protocols can be thought of as a particular type of instructional collection that is much narrower. The protocols most widely researched are designed to measure the intellectual rigor and quality of the assignments teachers give students as well as the student work that is produced in response to those assignments (e.g., Borko, Stecher, & Kuffner, 2007;Newmann, Bryk, & Nagaoka, 2001). These protocols are designed to be independent of the academic difficulty of a particular course of study. ...
... Both artifact and instructional collection instruments have been used for various purposes, ranging from formative feedback for the improvement of teaching practice to licensure and high-stakes advanced certification decisions. For example, the Scoop Notebook is an instructional collection protocol that has been used to improve professional practice (Borko et al., 2007;Borko, Stecher, Alonzo, Moncure, & McClam, 2005). The portfolio protocol for NBPTS certification is used as a part of a voluntary high-stakes assessment for advanced certification status (e.g., Cantrell, Fullerton, Kane, & Staiger, 2008;National Research Council, 2008;Szpara & Wylie, 2007). ...
... For almost all protocols, raters are required to pass a certification test before scoring. Although the quality of the training as judged by interrater agreement varies across protocols and studies, the literature suggests it is possible to train raters to acceptable levels of agreement (more than 70%) with significant effort (Borko et al., 2007;Gitomer, 2008b;Ingvarson & Hattie, 2008;Matsumura et al., 2006;M. Wilson, Hallam, Pecheone, & Moss, 2006). ...
... As they consider the strategies, they also examine their teaching of mathematics and/or science topics with technology. Through the development of a "Scoop" portfolio, participants gather (or scoop) their instruction over a short period of time, analyze their work using recommendations from national standards, and reflect on their effectiveness in guiding learning in technology-enhanced learning environments (Borko, Stecher, Kuffner , 2007). Their scoop portfolio collection contains in depth analyses and reflections (before, during and after the instruction) over the scoop period displayed electronically. ...
... The provision of feedback to assist others in improving their designs of the course projects was an important attribute of the Scoop Electronic Portfolio project in the instructional strategies course (Borko, et. al, 2007). This project required participants to gather (or scoop) and examine their teaching with technology. After the collection and initial analysis of the instruction, participants submitted their work to their assigned partners for review, assessment, and recommendations as to the quality of the teaching and the presentation. Once the autho ...
This qualitative descriptive study examines strategies for fostering a community of learners in an online Master's program in mathematics education and science education. Multiple cases analyses describe in-service teachers' work in the community of learners the process of their development of course assignments and projects. A cross case analysis distinguishes factors impacting cognitive, social, and teaching presences in the virtual community of learners. Specific contextual factors (e.g., extensive clarity in the assignment expectations, shift of learning management to the participants, and assess community interaction as well as the assignment content) contribute to higher levels of learning during the first year of the program. Results provide recommendations for online course environments in science and mathematics education for meeting the educational needs of teachers in geographically diverse and often rural locations as they learn to integrate technologies as learning tools.
... In online discussions, participants addressed strategies they were using to make their instruction more inquiry based, including the adapting of regular text-based science lessons to a more inquiry focus. The primary evidence for the teachers' use of inquiry was supplied by the use of Scoop notebooks (Borko, Stecher, & Kuffner, 2007). Teachers were asked to compile a notebook of evidences from their classroom over a period of three months during their second year in the project. ...
Statements of outcomes for 21st century learners typically include inquiry-based learning as a major goal. In the PRISM Project, 62 elementary teachers in Montana were selected to receive professional development using inquiry science instruction in their classrooms. Participants attended workshops designed to model inquiry lessons, participated in online discussions to help them make their lessons more inquiry-based, and prepared Scoop notebooks containing three lessons demonstrating how they were implementing inquiry in their classrooms. Based on analysis of these data, participants were judged to have met the goal of the project to increase their use of inquiry in the science classroom.
... As it was originally conceived, the scoop notebook is a teacher-generated, artefact-based collection method that combines aspects of portfolios and self-reports and that can be completed in as little as one week (Martínez et al., 2012). One goal of the notebooks is to collect enough artefacts to describe a teacher's practice, perhaps without actually observing it (Borko et al., 2007). They have been found to provide a standardised, valid, and reliable instrument (Stecher, 2005) that allows researchers to unobtrusively collect data about pedagogical actions for use with qualitative, quantitative, or mixed methodologies. ...
Electronic texts have become ubiquitous in the United States over the past two decades, even frequently displacing traditional physical texts in classrooms. Electronic textbooks, or e-textbooks, are sometimes adopted as a cost-saving measure or a way to engage reluctant students. But integrating e-textbooks into pedagogical and learning practices is rarely a simple swap of paper for screen. Students who read academic texts from screens would often benefit from additional support from instructors, many of whom are underprepared to provide effective literacy instruction to students. Set within the framework of activity theory, this research took a multi-level approach to explore how teachers and students engaged with e-textbooks in one American high school. Four biology teachers and 159 of their students participated in this multiple case study that spanned two academic years and three learning environments due to Covid-related disruptions. Each of the learning environments – face-to-face, online, and hybrid – served as an individual case because it provided distinct spatial and temporal boundaries for the teaching and learning activities. Data sources included questionnaires, naturally occurring teaching and learning artefacts, and observation. Findings from this study suggest that these biology teachers engaged with the e-textbook in much the same way as they would a physical textbook during their pedagogical actions. Additionally, students did not have opportunities to engage with the e-textbook during class time; however, they did engage with it to complete homework assignments. The students’ potential to engage with the e-textbook was constrained by the teachers’ engagement practices as well as a rule that ensured students only engaged with the e-textbook when outside of class.
... Classroom artifacts. We used an approach modeled after the Scoop Notebook developed by Hilda Borko and colleagues (Borko et al., 2007;Borko & Stecher, 2012) to gather classroom artifacts as well as teachers' reflections on those artifacts. The artifact collection-scoops-occurred twice per year from each participating teacher and included lesson plans, assignments, scoring guides or rubrics, marked-up student work (assignments or assessments; six samples per lesson), and teacher's responses to reflection questions about the lesson and student work. ...
Background/Context
The emphasis on scientific practices articulated by the National Research Council framework and the Next Generation Science Standards requires significant pedagogical shifts for U.S. science teachers.
Purpose/Objective/Research Question/Focus of Study
This study provides a rare window into the challenges and opportunities teachers encounter as they introduce argument writing into their science classrooms with support from the National Writing Project's Inquiry into Science Writing project. The purpose of this study is to better understand the teacher-change process so as to inform the development of future professional development efforts.
Population/Participants/Subjects
Case studies were drawn from a professional development network led by the National Writing Project to support teachers in studying and improving their practice while sharing knowledge and benefiting from the expertise of others. The network included 28 middle school teachers at five writing project sites around the United States; the case studies presented in this article are based on the experiences of three of these teachers.
Intervention/Program/Practice
The Inquiry into Science Writing Project was a 2-year practitioner-driven professional learning experience seeking to better understand and support student practice around evidence-based science writing. During the duration of the project, teachers taught at least one lesson series culminating in written arguments by students each semester, and participated in two summer institutes, an ongoing national professional learning community, and monthly meetings of their local teacher research group.
Research Design
The study uses a qualitative comparative case study approach.
Data Collection and Analysis
The case studies draw on interviews, lesson artifacts, written teacher reflections, and samples of student work.
Conclusions/Recommendations
The study findings reinforce the complexity of the change process: The relationship between teachers’ knowledge, beliefs, and attitudes and their practice was not linear and unidirectional (i.e., change in attitude leads to change in practice) but rather iterative and mediated by both student work and the external supports they received. These findings confirm the need for sustained learning environments with features that promote enactment and reflection on student work to support teacher change. Further, they suggest that professional development providers should think about how to build habits of reflection into their own design processes, allowing space for feedback and learning from practitioners.
... Similar results were observed using the Scoop Notebook to assess teachers' use of reform-oriented practices. The Scoop Notebook is used to collect data about classroom instruction without the labor and cost demands of typical class room observations (Borko, Stecher, & Kuffner, 2007). The study included 19 middle and high school science and math teachers, 11 of whom were former LAs. ...
Much of modern education reform is focused on implementation of evidenced-based teaching, but these techniques are sometimes met with trepidation from faculty, due to inexperience or lack of necessary resources. One near-peer teaching model designed to facilitate evidenced-based teaching in Science, Technology, Engineering, and Mathematics classrooms is the Learning Assistant (LA) model. Here, we describe the details of the LA model, present a scoping review of literature using the four original goals of the LA model as a framework, and suggest future areas of research that would deepen our understanding of the impact that the LA model may have on education. We summarize how the LA model improves student outcomes and teacher preparation and identify a relative deficiency of literature that addresses how the LA model impacts faculty and departmental/institutional change. Additionally, of the 39 papers reviewed, 11 are strictly pre-experimental study designs, 28 use quasi-experimental designs or a combination of quasi and pre-experimental, and none of them included a true experimental design. Thus, we conclude that current studies suggest that LA model positively impacts education, but more refined assessment would improve our understanding of the model. Furthermore, despite the encouraging research on the impact of the LA model and the proliferation of LA programs at institutions across the world, the study of the LA model has been, for the most part, limited to a small group of education researchers. Therefore, a major objective of this review is to introduce the LA model to a new group of instructors and researchers who can further our understanding of this promising model.
... The instrument items used for their study were developed from videotaped classroom observations of science teachers who were involved with STS groups (Yager et al., 2013). Focusing on the concept of inquiry-oriented teaching, Borko, Stecher, and Kuffner (2007) Other researchers have focused on developing a reliable and valid instrument that can help to overcome conceptual and technical challenges associated with measuring classroom instruction. Significantly, Martinez, Borko, and Stecher (2012) designed an instrument that combines artifact collection (lesson plans, handouts, rubrics, readings, worksheets, assignments, student homework, projects, portfolios, vignettes) and teachers" self-reporting (logs) and called it the Scoop Notebook. ...
Research on science-education reform affirms the importance of taking into consideration teachers’ pedagogical beliefs in relation of the constructivist perspective to ensure successful implementation of a reform-oriented curriculum. In addition, prominent studies pinpoint the need for teachers to have sufficient pedagogical content knowledge (PCK) to adapt a reform-oriented science curriculum to meet students’ abilities and interests. This study focused on the reform-oriented science curriculum in Oman, and in particular, the grade 12 biology curriculum. The new biology curriculum emphasizes constructivism and encourages student-centered instruction, inquiry-based learning, cooperative learning, problem-solving, and critical thinking. However, since its implementation in 2008, various obstacles and challenges have been reported by teachers. These include a mismatch between the planned curriculum and the implemented curriculum. This mismatch has been attributed to a lack of motivation and PCK among teachers that would allow them to shift their teaching practices. Consequently, the purpose of this study was to understand how teachers’ pedagogical beliefs about student-centered learning have shaped their implementation of the curriculum. The study also sought to identify how biology teachers perceive the relationship between their PCK and their pedagogical decisions. An ethnographic approach to data analysis was employed, with multiple data sources including classroom observations, learning artifacts, reflections, and semi-structured interviews used. The findings of this ethnographic study indicate that Omani teachers’ beliefs and PCK shape the implementation of the reform-oriented curriculum by influencing their conceptualization of the curriculum, their identification of students’ misconceptions, their decisions about classroom teaching practices, and the level of their students’ engagement. The results of this study corroborate and expand upon previous research that suggests that teachers’ beliefs and PCK should be taken into consideration when designing and planning for new curriculum materials, teacher-education programs, and professional development opportunities. Keywords: pedagogical beliefs, reform-oriented curriculum, pedagogical content knowledge (PCK), constructivism, curriculum implementation.
... The instrument items used for their study were developed from videotaped classroom observations of science teachers who were involved with STS groups (Yager et al., 2013). Focusing on the concept of inquiry-oriented teaching, Borko, Stecher, and Kuffner (2007) Other researchers have focused on developing a reliable and valid instrument that can help to overcome conceptual and technical challenges associated with measuring classroom instruction. Significantly, Martinez, Borko, and Stecher (2012) designed an instrument that combines artifact collection (lesson plans, handouts, rubrics, readings, worksheets, assignments, student homework, projects, portfolios, vignettes) and teachers" self-reporting (logs) and called it the Scoop Notebook. ...
Research on science-education reform affirms the importance of taking into consideration teachers’ pedagogical beliefs in relation of the constructivist perspective to ensure successful implementation of a reform-oriented curriculum. In addition, prominent studies pinpoint the need for teachers to have sufficient pedagogical content knowledge (PCK) to adapt a reform-oriented science curriculum to meet students’ abilities and interests. This study focused on the reform-oriented science curriculum in Oman, and in particular, the grade 12 biology curriculum. The new biology curriculum emphasizes constructivism and encourages student-centered instruction, inquiry-based learning, cooperative learning, problem-solving, and critical thinking. However, since its implementation in 2008, various obstacles and challenges have been reported by teachers. These include a mismatch between the planned curriculum and the implemented curriculum. This mismatch has been attributed to a lack of motivation and PCK among teachers that would allow them to shift their teaching practices. Consequently, the purpose of this study was to understand how teachers’ pedagogical beliefs about student-centered learning have shaped their implementation of the curriculum. The study also sought to identify how biology teachers perceive the relationship between their PCK and their pedagogical decisions. An ethnographic approach to data analysis was employed, with multiple data sources including classroom observations, learning artifacts, reflections, and semi-structured interviews used. The findings of this ethnographic study indicate that Omani teachers’ beliefs and PCK shape the implementation of the reform-oriented curriculum by influencing their conceptualization of the curriculum, their identification of students’ misconceptions, their decisions about classroom teaching practices, and the level of their students’ engagement. The results of this study corroborate and expand upon previous research that suggests that teachers’ beliefs and PCK should be taken into consideration when designing and planning for new curriculum materials, teacher-education programs, and professional development opportunities. Keywords: pedagogical beliefs, reform-oriented curriculum, pedagogical content knowledge (PCK), constructivism, curriculum implementation.
... Moreover, it was found that these instruments did not have a characteristic that was revealed to be essential to our work-a given focus and structure-to allow more rigorous, precise, and powerful comparability between different situations and cases (Lopes et al., 2014, p. 416). Several approaches have to studying the teaching practice of teachers in the classroom been identified: inferring teaching characteristics by linking them to the school culture, curriculum material, teacher knowledge and beliefs, and students' learning (Borko, Stecher, & Kuffner, 2007;Tytler, 2003) and using artefacts in the classroom combined with self-reporting (Martínez, Borko, & Stecher, 2012). ...
It is acknowledged that to improve the value of the learning process and outcomes in areas such as science, technology, engineering and math, the teaching quality needs to be enhanced. Therefore, it is crucial to have access to real teaching practices. The multimodal narrative (MN) tool allows teaching practices to become public, sharable, and usable (open science perspective), preserving their holistic, complex, and ecological nature. This tool has characteristics and a structure that enable an in-depth study of teaching practices, in different contexts, with several purposes (e.g., teacher education, professional development, and research). This chapter presents MNs and the necessary steps involved in collecting multimodal data, structuring the narrative, and validating the document. MNs can be used by teachers and researchers, or other professionals, with multiple specific objectives, globally contributing to improving professional practices.
... Recent research suggests that effective technology integration has the potential to promote STEM learning through facilitated implementation of RTPs. 6-8 Borko, Stecher, and Kuffner 9 have proposed the ten dimensions of RTPs for teaching science, including the "Use of scientific resources" and "Hands-on" activity. Literature review additionally reveals that educational robotics as a scientific resource has emerged as a learning tool that has tremendous potential in offering fun hands-on activities in an attractive learning environment. ...
... 52 Pilot studies of a tool called the "Scoop Notebook" have also shown promise. 53 More research is needed, but artifact analysis may be an informative part of a broader assessment system. ...
Given the experience to date with an overwhelming focus on student achievement scores as a basis for high-stakes decisions, policymakers would do well to pause and carefully examine the issues that make teacher assessment so complex before implementing an assessment plan. To facilitate such examination, this brief reviews credible research exploring: the feasibility of combining formative assessment (a basis for professional growth) and summative assessment (a basis for high-stakes decisions like dismissal); the various tools that might be used to gather evidence of teacher effectiveness; and the various stakeholders who might play a role in a teacher assessment system. It also offers a brief overview of successful exemplars.
... This fourth course, a blend of online discussions and practical teaching experiences, directed the participants' attention toward examining instructional strategies for teaching with technologies. They concurrently engaged in developing a Scoop portfolio [22], where they designed a five-day sequence of lessons, taught the lessons (reflecting daily and cumulatively on the instruction), and assessed their students' progress in meeting the objectives. The teacher-participants consistently engaged in two reflective thought processes: reflection-in-action and reflection-on-action [7] as they engaged in the practical experiences in this course. ...
... Matsumura and colleagues found that a reliable estimate of ELA classroom assignment quality could be attained with three assignments, that there was overlap among the scales, and that there was a relationship between assignment quality and other measures of teaching quality (Clare, 2000;Clare & Aschbacher, 2001;Clare, Vald es, Pascal, & Steinberg, 2001;Matsumura & Pascal, 2003;Matsumura et al., 2008). Similar work looking at middle school mathematics and science classes found that a reliable estimate of classroom practice could be based on teacher assignments and student work (Borko, Stecher, & Kuffner, 2007;Borko et al., 2005). ...
We investigate classroom assignments and resulting student work to identify important characteristics of assignments in terms of instructional quality and their validity as measures of teaching quality. We examine assignment quality within a large-scale project exploring multiple measures including classroom observations, teacher knowledge measures, and value-added estimates based on student achievement scores. Analyses included descriptive statistics, multivariate analyses to understand factors contributing to score variance, and correlational analyses exploring the relationship of assignment scores to other measures. Results indicate relatively low demand levels in all teacher assignments, a marked difference in score distributions for mathematics (math) and English language arts (ELA), and a substantial relationship between what was asked of and produced by students. Relationships between assignments scores, classroom characteristics, and other measures of teaching quality are examined for both domains. These findings help us understand the nature of and factors associated with assignment quality in terms of intellectual demand.
... Therefore, the program design identified a third course (SED 594) to follow these two courses with which to involve the participants in designing, implementing, analysing, and reflecting on teaching and learning in a 5-day sequence of lessons that integrated technologies as learning tools in their classrooms. In this third course, the participants developed electronic portfolios following guidelines for the development of a "Notebook" (Borko, Stecher, & Kuffner, 2007) or electronic portfolio. Table 2 describes the expectations the teachers needed to complete and present through their electronic notebooks by the end of the course. ...
A systems approach provides insight for expanding teachers’ pedagogical reasoning for integrating multiple technologies in inquiry, communication, and collaboration. An online learning trajectory supports the integration of a systems pedagogical approach for guiding teachers in developing their technological pedagogical thinking and reasoning so they in turn are able to implement a systems pedagogical approach with their own students. Specific instructional strategies guide teachers in refining their mental models for integrating multiple technologies in teaching mathematics through their increasingly complex technological pedagogical understanding as they learn about the technologies and teaching with those technologies. This study focuses on the impact that a system of multiple technologies as pedagogical tools has on teachers’ technological pedagogical reasoning as they integrate multiple technologies in their classrooms. A systems pedagogical understanding is at the core of teachers’ enhancement of their technological pedagogical reasoning, and supports the transformation of their knowledge called technological pedagogical content knowledge.
... A hands-on measure consists of teachers' reports of the number of minutes the class spent on hands-on science. Borko, Stecher, and Kuffner (2007) in the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) reported on this in a clearer and more concise manner by summarizing the ten dimensions of RTPs for teaching science. These dimensions were created, guided by the vision of inquiry-oriented teaching described in the National Science Education Standards (NRC, 1996) and the Mosaic-II project . ...
Growing evidence from recent curriculum documents and previous research suggests that reform-oriented science teaching practices promote students’ conceptual understanding, levels of achievement, and motivation to learn, especially when students are actively engaged in constructing their ideas through scientific inquiries. However, it is difficult to identify to what extent science teachers engage students in reform-oriented teaching practices (RTPs) in their science classrooms. In order to exactly diagnose the current status of science teachers’ implementation of the RTPs, a valid and reliable instrument tool is needed. The principles of validity and reliability are fundamental cornerstones in developing a robust measurement tool. As such, this study was motivated by the desire to point out the limitations of the existing statistical and psychometric analyses and to further examine the validation of the RTP survey instrument. This paper thus aims at calibrating the items of the RTPs for science teachers using the Rasch model. The survey instrument scale was adapted from the 2012 National Survey of Science and Mathematics Education (NSSME) data. A total of 3701 science teachers from 1403 schools from across the USA participated in the NSSME survey. After calibrating the RTP items and persons on the same scale, the RTP instrument well represented the population of US science teachers. Model-data fit determined by Infit and Outfit statistics was within an appropriate range (0.5–1.5), supporting the unidimensional structure of the RTPs. The ordered category thresholds and the probability of the thresholds showed that the five-point rating scale functioned well. The results of this study support the use of the RTP measure from the 2012 NSSME in assessing usage of RTPs.
... In this practice-based, hybrid course, they collaborate online as a community of learners in the identification and development of technologyenhanced instructional strategies for promoting inquiry and higher order thinking in a discourse-oriented learning experience. The course project adapted from the Scoop Research Project by Borko, Stecher, and Kuffner (2007) guides participants in compiling an Electronic Portfolio containing classroom artifacts and related materials. The participants gather data, documenting their instruction using technologies in teaching mathematics over a series of lessons with analysis of videos of two lessons, analysis of student work, and multiple reflections on their practice. ...
This study examines an online learning progression that advances K-12 technological pedagogical content knowledge (TPACK) for integrating technologies as mathematics learning tools. Using a social metacognitive constructivist learning trajectory, the courses activate inquiry learning communities in specific tasks and reflections to challenge and support their thinking about teaching mathematics with a diversity of technologies as learning tools. The online technology coursework is interwoven with a hybrid course that blends community inquiries with teacher practice and reflection as they design, implement, and assess the impact of teaching with technologies. This descriptive case study reveals the impact of the designed learning progression and concludes by proposing an online instructional progression for transforming teachers' knowledge for teaching mathematics with technologies.
... Two existing measures served as a foundation for M-Scan measurement development. Dimensions were selected, defined, and adapted from the SCOOP measure; which used classroom artifacts such as tests, observations, and instructional materials to measure the quality of mathematics instruction (Borko et al., 2005;Borko, Stecher, & Kuffner, 2007). The SCOOP measure included an observational component to understanding the quality of instruction. ...
... This course incorporated the idea of advancing the participants' TPACK for integrating technology (T) in mathematics and science (the content or C) with classroom pedagogical practices (P), the academic emphasis was on TPK (technological pedagogical knowledge). Participants gathered evidence of their instruction with technology through the design of an electronic portfolio modified from the research-based Scoop project (Borko, Stecher, & Kuffner, 2007) that was found to be a valid and reliable means for classroom observation through artifacts and reflections incorporated in the completed Scoop portfolio. For this course, this portfolio demonstrated the participant's developing TPACK knowledge and skills for strategic planning, implementation, and critical analysis when teaching mathematics/science with appropriate technologies. ...
... In online discussions, participants addressed strategies they were using to make their instruction more inquiry based, including the adapting of regular text-based science lessons to a more inquiry focus. The primary evidence for the teachers' use of inquiry was supplied by the use of Scoop notebooks (Borko, Stecher, & Kuffner, 2007). Teachers were asked to compile a notebook of evidences from their classroom over a period of three months during their second year in the project. ...
... When a teacher uses game examples to help bridge the game-based implicit learning to STEM content discussed explicitly in the classroom, students have higher gains on pre/post tests about related science content than students in the Control classes. To unpack these findings further, we have been analyzing logs of teacher activity modeled after the SCOOP Notebook developed by CRESST (Borko, Stecher & Kuffner, 2007). The coding system developed for this study focuses on the type of classroom activities (e.g., direct instruction, hands-on activities, etc.); the science content taught (e.g., Newton's Laws, forces, etc.), and game-based pedagogies used (e.g., using game examples during instruction, modeling game play, discussing the game, etc.). ...
Games provide an important vehicle for educators to promote and study learning. This symposium examined research on measuring implicit game-based learning and teachers leveraging its relationship for explicit (e.g. school-based) STEM learning. The authors have developed a series of learning games that simulate authentic scientific phenomena, providing a learning mechanic for players to dwell in that phenomena and build their implicit understandings. The data logs generated through digital gameplay were mined to understand the patterns of play that may be related to implicit learning—the development of knowledge that is not yet explicitly formalized. Teachers used examples from games to help bridge implicit game-based learning to explicit STEM concepts taught in class.
... Another approach is to focus on particular aspects of teaching practice, such as interactions (Martin and Hand 2009). A further general approach to teaching practices is to collect indicators about the classroom environment (Borko et al. 2007;Scott and Fisher 2004) or instructional artefacts combined with selfreporting (Martínez et al. 2012). ...
This study deals with the problem of how to collect genuine and useful data about science classroom practices, and preserving the complex and holistic nature of teaching and learning. Additionally, we were looking for an instrument that would allow comparability and verifiability for teaching and research purposes. Given the multimodality of teaching and learning processes, we developed the multimodal narrative (MN), which describes what happens during a task and incorporates data such as examples of students’ work, photos, diagrams, etc. Also, it describes teachers’ intentions, preserving the nature of teaching practice in natural settings and it is verifiable and comparable. In this paper, we show how the MN was developed and present the protocol that was used for its construction. We identify the main characteristics of the MN and place it in the context of international research. We explore the potential of the MN for research purposes, illustrating its use in a research study that we carried out. We find that the MN provides a way to gather, organize and transform data, avoiding confusing and time-consuming manipulation of data, while minimizing the natural subjectivity of the narrator. The same MN can be used by the same or by different researchers for different purposes. Furthermore, the same MN can be used with different analysis techniques. It is also possible to study research practices on a large scale using MNs from different teachers and lessons. We propose that MNs can also be useful for teachers’ professional development.
... The M-Scan offers a list of important features to consider when planning and implementing a mathematics lesson or instructional unit. The M-Scan was adapted from a data collection tool, the Scoop notebook (Borko et al. 2005;Borko, Stecher, and Kuffner 2007;and Stecher et al. 2003). The research team used eight dimensions from the Scoop scoring guide to link the NCTM Standards to dimensions that can be observed in the classroom. ...
Evaluate the quality of your instruction by using the eight dimensions of M-Scan, an observation tool that links math standards with day-to-day practice.
... Like direct observation, the portfolio entries displayed important details of classroom lessons; similar to survey data, the portfolio entries permitted access to the teacher's perspective. Although the NBPTS portfolio data might appear to overly limited as a source of information about teaching practice because the records do not include direct observation of actual teaching, the data in the NBPTS portfolio submissions are in many ways quite similar to those that have been used and validated by other researchers to study classroom practice using alternatives to direct observation and survey methods, such as "scoop" sampling of instructional artifacts (e.g., lesson plans, student work) to characterize instructional activity (Borko, Stecher, & Kuffner, 2007) and using classroom assignments to judge instructional quality (Clare & Aschbacher, 2001;Matsumura, Garnier, Pascal, & Valdés, 2002). Researchers interested in alternatives to direct observation methods (which are invasive, labor intensive, expensive, and impractical on a large scale) and survey methods (which involve questions susceptible to multiple interpretations, have questionable validity, and provide little information about the details of instructional lessons) might be wise to consider data like those collected in the NBPTS portfolio process to open another window on classroom instructional practice. ...
We present an analysis that probed empirically the relationship among three different views of exceptional mathematics teaching:
(a) the operational definition of “highly accomplished teaching” of mathematics used by the National Board for Professional
Teaching Standards (NBPTS) in the United States, (b) the effective use of cognitively demanding tasks in the mathematics classroom,
and (c) the use of innovative pedagogical strategies. We analyzed samples of instructional practice—lesson artifacts and teachers’
commentaries on lessons—submitted by candidates seeking NBPTS certification in the area of Early Adolescence/Mathematics.
The instructional samples were systematically probed for evidence of mathematical and pedagogical features associated with
the views of cognitive demand and innovative pedagogy, and the features found in the submissions of applicants who were awarded
NBPTS certification are contrasted with those who were not awarded certification. Our analyses detected a fairly strong interaction
between the NBPTS view of accomplished teaching and the view of effective mathematics instruction associated with cognitively
demanding tasks. Nevertheless, even in these lessons that teachers selected for display as “best practice” examples of their
mathematics teaching, innovative pedagogical approaches were not systematically used in ways that supported students’ engagement
with cognitively demanding mathematical tasks.
KeywordsMathematics teaching-Teaching quality-Cognitively demanding tasks-Pedagogical innovation
Advances in game-based learning and educational data mining enable novel methods of formative assessment that can reveal implicit understandings that students may demonstrate in games but may not express formally on a test. This chapter explores a framework of bridging in game-based learning classes, where teachers leverage and build upon students' game-based implicit learning experiences to support science classroom learning. Bridging was studied with two physics learning games in about 30 high-school classes per game. Results from both studies show that students in bridging classes performed better on external post-tests, when accounting for pre-test scores, than in classes that only played the game or did not play the game at all. These findings suggest the teachers' role is critical in game-based learning classes. Effective bridging includes providing teachers with common game examples along with actionable discussion points or activities to connect game-based learning with classroom content.
This chapter describes the analysis of 729 daily teacher logs from a 2013-14 national classroom implementation study with hundreds of high school physics students using the game, Impulse, finding classrooms using materials to bridge implicit and explicit science learning performed significantly better than control classrooms (Rowe et al., 2014). This effect was moderated by whether or not the class was a Honors/AP class. The authors examine the student and teacher demographics, science content, instructional materials and methods, and game-based pedagogies as potential explanations for those findings. The largest difference among Honors/AP vs. non-Honors/AP classrooms using any Bridge activities was their use of formal, teacher-led discussion.
The design and empirical support for the online TPACK learning trajectory emerged through a multi-year research process that provided a thorough, in-depth description of how the tools (community of learners and reflection) and processes (shared/individual knowledge development and inquiry) support the scaffolding of TPACK content as an integration of subject matter content, pedagogy, and technologies, thus modeling the knowledge teachers need for teaching with technology. The learning trajectory, framed with a social metacognitive constructivist lens, engaged inservice teachers in knowledge-building communities using inquiry-based, problem-based learning, guiding them in reframing their knowledge for designing student-directed, problem-based learning with the integration of technologies. Limitations and future research extend the understanding of TPACK through online teacher education continued learning in graduate programs and other professional development programs designed to support teachers in rethinking and reframing their knowledge for teaching with technologies. Multiple factors frame the thinking about future designs for these online programs aimed at transforming inservice teachers' TPACK. Future challenges include whether and how online programs might be designed for developing preservice teachers' TPACK.
Advances in game-based learning and educational data mining enable novel methods of formative assessment that can reveal implicit understandings that students may demonstrate in games but may not express formally on a test. This chapter explores a framework of bridging in game-based learning classes, where teachers leverage and build upon students' game-based implicit learning experiences to support science classroom learning. Bridging was studied with two physics learning games in about 30 high-school classes per game. Results from both studies show that students in bridging classes performed better on external post-tests, when accounting for pre-test scores, than in classes that only played the game or did not play the game at all. These findings suggest the teachers' role is critical in game-based learning classes. Effective bridging includes providing teachers with common game examples along with actionable discussion points or activities to connect game-based learning with classroom content.
Inservice teacher preparation balances theory with practical experiences to support teachers in integrate their theoretical knowledge into their teaching practice. Online instruction holds potential for this education but questions how classroom observations are conducted in the teachers ' classroom practices, particularly where the teachers are geographically dispersed. This multiple case descriptive study examines an online analogue to traditional classroom observations, where the Scoop Notebook (Borko, Stecher, & Kuffner, 2005) reveals inservice teachers ' Technological Pedagogical Content Knowledge (TPACK), more specifically their TPACK-of-practice (Cochran-Smith & Lytle, 1999). The Scoop Electronic Portfolio development process describes teachers' engagement in classroom practices, transitioning their scholarly theoretical knowledge to practical knowledge through in-depth, rich reflections from classroom actions and artifacts. This course blends the practical experiences of the Scoop process with asynchronous community of learners ' explorations of instructional strategies. The results describe teachers engaged in action research using Scoop artifacts as objects to think with for transforming their TPACK for integrating technologies in teaching their content, ultimately transforming their TPACK-of-practice.
This qualitative, design-based research identifies innovative instructional practices for teacher professional development that support an online community of learners in reconstructing their technological pedagogical content knowledge (TPACK) for teaching mathematics. This analysis describes instructional practices that guide inservice teacher participants in inquiring and reflecting to confront their knowledge- of-practice conceptions for integrating multiple technologies as learning tools. The research program describes an online learning trajectory and instructional strategies supporting the tools and processes in steering the content development in a social metacognitive constructivist instructional framework towards moving from "informal ideas, through successive refinements of representation, articulation, and reflection towards increasingly complex concepts over time" (Confrey & Maloney, 2012). The results provide recommendations for online professional development learning environments that engage the participants as a community of learners.
The design and empirical support for the online TPACK learning trajectory emerged through a multiyear research process that provided a thorough, in-depth description of how the tools (community of learners and reflection) and processes (shared/individual knowledge development and inquiry) support the scaffolding of TPACK content as an integration of subject matter content, pedagogy, and technologies, thus modeling the knowledge teachers need for teaching with technology. The learning trajectory, framed with a social metacognitive constructivist lens, engaged inservice teachers in knowledge-building communities using inquiry-based, problem-based learning, guiding them in reframing their knowledge for designing student-directed, problem-based learning with the integration of technologies. Limitations and future research extend the understanding of TPACK through online teacher education continued learning in graduate programs and other professional development programs designed to support teachers in rethinking and reframing their knowledge for teaching with technologies. Multiple factors frame the thinking about future designs for these online programs aimed at transforming inservice teachers' TPACK. Future challenges include whether and how online programs might be designed for developing preservice teachers' TPACK.
The Mathematics Scan (M-Scan), a content-specific observational measure, was utilized to examine the extent to which standards-based mathematics teaching practices were present in three focal lessons. While previous studies have provided evidence of validity of the inferences drawn from M-Scan data, no prior work has investigated the affordances and limitations of the M-Scan in capturing standards-based mathematics teaching. We organize the affordances and limitations into three categories: the operationalization of the M-Scan, the organization of the M-Scan, and the M-Scan within the larger ecology of instruction. Our analysis indicates the M-Scan differentiates among lessons in their use of standards-based mathematics teaching practices by operationalizing the M-Scan dimensions at the lesson level, sometimes at the expense of capturing the peaks and valleys within a single lesson. Simultaneously, the analysis revealed how the application of the rubrics may be impacted by lesson transcripts. We discuss the theoretical organization of the M-Scan and its implications for researchers and practitioners applying the rubrics. Finally, we point to the affordances and limitations of the M-Scan within the larger ecology of instruction by considering curricular issues and two dimensions of instruction not highlighted by the M-Scan.
This study reports on the results of research aimed at exploring the usefulness and feasibility of evaluating the teaching of mathematics in Mexican elementary schools. An evaluation was made of forty class sessions in twenty classrooms in three schools in Mexico City. The instrument used was the Instructional Quality Assessment. The results show that this type of evaluations has great potential in determining the design of actions in educational improvement, at both the systemic and specific school level. The conclusion is that they are feasible, since their educational value would compensate for the significant costs and limitations that are implicit in their use.
In an attempt to foster inquiry-oriented learning in middle grades (grades 6 - 8), a technology mediated pedagogy integrating science and mathematics was promoted through Project SMILE (Science and Mathematics Integration for Literacy Enhancement). It involved in-service teachers in professional learning and classroom implementation over a period of two academic years, with the explicit goal of enhancing teachers' ability to foster more authentic inquiry in their classes. This chapter describes the design of Project SMILE in the context of recent reform efforts in science and mathematics education, along with the theoretical underpinnings for the design. Project activities, followed by the research methodology employed to investigate the impact on both teachers and students are described next. Finally, research results and their implications are discussed with an eye toward the usefulness of integrating science and mathematics and involving specific technological tools to foster greater inquiry-oriented learning in school science and mathematics.
The Paleo Exploration Project (PEP) was a University of Montana (UM) professional development program serving K-12 teachers from eastern Montana. Two cohorts of 25 teachers each completed the program. Each cohort was engaged in the training for 12-18 months. The program began with several 2-day teachers' weekend workshops during the spring semester. The following summer, teachers attended a weeklong summer research institute with middle-school-aged students. Over the next academic year, teachers took part in a final weekend workshop and developed, and in most cases implemented, their own learning activities with their students. Using a design experiment framework, we learned that teachers needed (1) additional hands-on practice with the technologies, (2) a curriculum component that was targeted more directly on scientific inquiry, and (3) more practice with project design.
In an attempt to foster inquiry-oriented learning in middle grades (grades 6 - 8), a technology mediated pedagogy integrating science and mathematics was promoted through Project SMILE (Science and Mathematics Integration for Literacy Enhancement). It involved in-service teachers in professional learning and classroom implementation over a period of two academic years, with the explicit goal of enhancing teachers' ability to foster more authentic inquiry in their classes. This chapter describes the design of Project SMILE in the context of recent reform efforts in science and mathematics education, along with the theoretical underpinnings for the design. Project activities, followed by the research methodology employed to investigate the impact on both teachers and students are described next. Finally, research results and their implications are discussed with an eye toward the usefulness of integrating science and mathematics and involving specific technological tools to foster greater inquiry-oriented learning in school science and mathematics.
In an attempt to foster inquiry-oriented learning in middle grades (grades 6 - 8), a technology mediated pedagogy integrating science and mathematics was promoted through Project SMILE (Science and Mathematics Integration for Literacy Enhancement). It involved in-service teachers in professional learning and classroom implementation over a period of two academic years, with the explicit goal of enhancing teachers' ability to foster more authentic inquiry in their classes. This chapter describes the design of Project SMILE in the context of recent reform efforts in science and mathematics education, along with the theoretical underpinnings for the design. Project activities, followed by the research methodology employed to investigate the impact on both teachers and students are described next. Finally, research results and their implications are discussed with an eye toward the usefulness of integrating science and mathematics and involving specific technological tools to foster greater inquiry-oriented learning in school science and mathematics.
Technology, pedagogy, and content knowledge (TPACK) is a dynamic lens that describes teacher knowledge required for designing, implementing, and evaluating curriculum and instruction with technology. TPACK strategic thinking incorporates knowing when, where, and how to use domain-specific knowledge and strategies for guiding students' learning with appropriate digital, information, and communication technologies. This chapter maps historical responses to the question of the knowledge that teachers need for teaching amid the emerging views of and challenges with TPACK. A review of empirical progress serves to illuminate potential insights, values, and challenges for directing future research designed to identify a teacher's learning trajectory in the development of a more robust and mature TPACK for teaching with current and emerging information and communication technologies.
Powerful and modern digital technologies have significantly impacted mathematics teaching - both what is to be learned and how it should be learned. Technology, pedagogy, and content knowledge (TPACK) is the knowledge that teachers rely on for teaching content with appropriate digital technologies. What preparation do mathematics teachers need in order to develop this knowledge needed for integrating appropriate digital technologies as teaching and learning tools? The challenges of understanding TPACK and identifying appropriate educational programs for pre-service mathematics teachers call for thoughtful attention toward the development of the knowledge, skills, and dispositions that support the dynamic nature embedded within the TPACK construct. The design of appropriate pre-service teacher learning trajectories for developing a rigorous TPACK emphasizes that both how and where they learn to teach mathematics are fundamental to what is learned about teaching and learning mathematics. Redesign ideas and models support re-thinking and re-designing pre-service mathematics teacher preparation programs.
Technology applications aligned with science, technology, engineering, and math (STEM) workplace practices can engage students in real-world pursuits but also present dramatic challenges for classroom implementation. We examined the impact of teacher professional development focused on incorporating these workplace technologies in the classroom. Because existing measures primarily use only presence or type of technology as proxies for implementation quality, we developed an expanded framework that incorporated (a) the type of technology used; (b) the degree of alignment to STEM practices; (c) the use of student-centered pedagogical practices; and (d) the degree of relevance to real-world contexts. While our framework successfully described the variation in technology implementation in our study group, we found no statistically significant difference between teachers with and without extensive training on STEM workplace technologies. Our results provide evidence that the framework captures quality of technology use and point to the need for additional research on effective teacher education around technology applications.
This chapter investigates the challenges and needs that should be addressed in teacher preparation for educational technology. It presents a study which analyzed 30 preservice elementary school teachers’ lesson plans, representing their first attempts to design a web-based lesson. The analysis focused on the types of activities they had designed, the characteristics of the web resources they had selected, and the scaffolding techniques they had planed to use to support their students. Most preservice teachers designed either open-research or over-structured student activities that focused on information retrieval. Overall, they demonstrated a limited repertoire of techniques to scaffold elementary students with web learning beyond the level of information search and factual information extraction. Also, many students did not utilize the full range of online information resources and selected inappropriate websites. Study findings highlight that, in order to use the Internet productively and creatively, teachers need to develop complex forms of knowledge that require the integration of knowledge about technology, pedagogy, and content.
Since the publication of the National Science Education Standards in 1996, learning science through inquiry has been regarded as the heart of science education. However, the TIMSS 1999 Video Study showed that inquiry-based teaching has been taking place less in the United States than in Japan. This study examined similarities and differences in how Japanese and American middle-school science teachers think and feel about inquiry-based teaching. Teachers' attitudes toward the use of inquiry in science teaching were measured through a survey instrument (N=191). Teachers' understanding of inquiry-based teaching was examined through interviews and classroom observations in the United States (N=9) and Japan (N=15). The results show that in spite of the variations in teachers' definitions of inquiry-based teaching, teachers in both countries strongly agree with the idea of inquiry-based teaching. However, little inquiry-based teaching was observed in either of the countries for different reasons. The data indicate that Japanese teachers did not generally help students construct their own understanding of scientific concepts in spite of well-planned lesson structures and activity set-ups. On the other hand, the observational data indicate that American teachers often lacked meaningful science content in spite of their high level of pedagogical knowledge. The need for addressing the importance of scientific concepts in teacher preparation programs in higher education institutions in the US is advocated. To the Japanese science education community, the need for teachers' acquisition of instructional strategies for inquiry-based teaching is strongly addressed.
Even as research has begun to document that teachers matter, there is less certainty about what attributes of teachers make the most difference in raising student achievement. Numerous studies have estimated the relationship between teachers' characteristics, such as work experience and academic performance, and their value-added to student achievement; but, few have explored whether instructional practices predict student test score gains. In this study, we ask what classroom practices, if any, differentiate teachers with high impact on student achievement in middle school English Language Arts from those with lower impact. In so doing, the study also explores to what extent value-added measures signal differences in instructional quality. Even with the small sample used in our analysis, we find consistent evidence that high value-added teachers have a different profile of instructional practices than do low value-added teachers. Teachers in the fourth (top) quartile according to value-added scores score higher than second-quartile teachers on all 16 elements of instruction that we measured, and the differences are statistically significant for a subset of practices including explicit strategy instruction.
ResearchGate has not been able to resolve any references for this publication.