ArticlePDF Available

Data in Practice: Conceptualizing the Data-Based Decision-Making Phenomena

Authors:

Abstract

"Data use" and "data-based decision making" are increasingly popular mantras in public policy discourses and texts. Policy makers place tremendous faith in the power of data to transform practice, but the fate of policy makers' efforts will depend in great measure on the very practice they hope to move. In most conversations about data use, however, relations between data and practice have been under conceptualized. In this essay, I identify and discuss some conceptual and analytical tools for studying data in practice by drawing on work from various theoretical traditions. I explore some ways in which we might frame a research agenda in order to investigate data in everyday practice in schools. My account is centered on schoolhouse work practice, but the research apparatus I consider can be applied to practice in other organizations in the education sector and indeed to interorganizational practice, a critical consideration in the education sector.
A preview of the PDF is not available
... Additionally, as educational change processes in schools are related to data-based learning processes, routines in data use can be identified (Coburn & Turner, 2011;Spillane, 2012). Coburn and Turner defined routines for data use "as the modal ways that people interact with data and each other in the course of their ongoing work" (2011, p. 181). ...
... 182). According to Spillane (2012), organizational routines are a useful unit of analysis for studying data use because they help to analyze "standard ways of doing things in the school and how, if at all, these standard ways of doing things change in response to data-use initiatives" (p. 117). ...
Article
Full-text available
Routines play a major role in educational change in schools. But what happens if the routines performed by school staff fail to deal successfully with current challenges? What strategies aid adaptation of the routines in a specific situation? Up to now, there exists no comprehensive concept for understanding why and at what points the adapting of routines in schools in a specific situation takes a favorable or unfavorable direction. To address this gap, we propose extending theories on routines by considering theories on self-regulated and collectively regulated learning. We consider these theories to be a beneficial complement because of their broad theoretical, methodological, and empirical research base. We argue that these theories enhance the understanding of adapting routines to specific challenging situations in schools. We present a newly developed theoretical framework for dealing with specific challenging situations in schools as an interplay between routines and regulation processes. Finally, important research questions regarding the suggested approach are discussed.
... The Shanghai model will provide empirically based understandings about how expert teachers network within and across schools. As such it also contributes to an understanding of distributed instructional leadership, which Spillane (2012) argues is urgently needed to improve the quality of learning and teaching in schools. Findings of this study therefore have the potential to inform understanding of the conditions necessary to empower expert teachers to be boundary spanners who connect learning among different constituencies (Coldren & Spillane, 2007;Wenger, 1998). ...
... Indeed, during the last decade, several tools (e.g., educational dashboards, teaching augmentation tools) were designed within this conceptual framework (see for example [3,43]). But despite these efforts, teachers often do not adopt such tools either because they lack training in data-driven teaching practices and skills, or because the tools either do not fit classroom reality or are not suitable for the teachers' pedagogical preferences and habits [15,36]. In addition, there is also the issue of self-efficacy: to adopt educational technology, teachers should trust both the technology and their own ability to master it [16,17,25]. ...
... Over the past decade, the education research field has increasingly expanded in the area of data use practices within schools and school districts, particularly in Anglo-Saxon research (e.g. Coburn & Turner, 2011;Fenwick & Edwards, 2016;Huber & Skedsmo, 2016;Jennings, 2012;Kelly and Downey, 2012;Little, 2012;Racherbäumer et al., 2013;Schildkamp et al., 2014;Spillane, 2012;Sun et al., 2016;Mausethagen et al., 2018Mausethagen et al., , 2019. A general finding is that data use depends on the factors related to organisational routines, such as access to data, time, financial resources, leadership and the norms of interaction (Farrell & Coburn, 2017;Selwyn, 2016). ...
Article
Full-text available
The use of data for governance purposes has been widely recognised as a way for national authorities to coordinate their activities across administrative levels and improve educational quality. This places the mid-central authority-in many countries the municipal level-in the midst of modern education governing. This article reports a case study analysis of the particular uses of performance data and numbers by mid-central municipal authorities in the daily work of governing schools in Nor-way. The three empirical case studies combine an analysis of policy document and fieldwork interviews with municipal administrators. The article contributes important insights into the role of municipal administrators as interpreters of policy goals at a crucial yet understudied level of the education system. In contrast to the dominant perspective in the data use literature, which often addresses implementation and the effectiveness of how numbers and data can be ideally designed and used, the results provide grounds for a more nuanced understanding of the institutional processes related to setting performance goals.
Thesis
This dissertation consists of three papers. "Online Learning, Offline Outcomes: Online Course Taking and High School Student Performance." This paper uses fixed effects models to estimate differences in contemporaneous and downstream academic outcomes for students who take courses virtually and face-to-face, both for initial attempts and for students taking courses for credit recovery. We find that while contemporaneous outcomes are positive for virtual students in both settings, downstream outcomes vary by attempt type. For first-time course-takers, virtual course-taking is associated with decreases in the likelihood of taking and passing follow-on courses, and of a measure designed to proxy for graduation readiness. For credit recovery students, virtual course-taking is associated with increased likelihood of taking and passing follow-on courses, and of being in line for graduation. Supplemental analyses suggest that selection on unobservables would have to be substantial to render these results null. "Bridging School Data Use in Principle and in Practice." This paper seeks to better understand the practice of data use for improvement in schools, as well as how well research literature on school data use captures the process as it unfolds in practice. To do so, I first synthesize leading research literature to create a framework reflecting the “in principle” guidance for school data use. This framework is then used to describe and understand the practice of data use in a high-data-use school over a two-year case study. My primary finding is that while the “in principle” guidance for data use captures many important components of the process, it understates the complexity in the relationships between these components. The consequence of this understatement is a bifurcation of the supports required for data use and the process of school data use, belying the reciprocal interdependence that I argue better describes their connectedness. "Allocations and Expectations: Comparing Teacher Reports of Virtual and Face-to-Face Instruction." This paper relies on a teacher survey asking teachers to describe their experiences teaching both in face-to-face classrooms and in a well-established statewide virtual learning system. The responses are used to create two groups, face-to-face and virtual, made up of the same set of individuals. I then compare the similarities and differences between the two learning modalities. Statistically significant differences are found in the time allocated to several common teaching tasks, in several aspects of the work environment, in expectations for common materials and practices, and in the overall impression of the curriculum and materials. Ultimately, my findings point to a need for more research of this type to better understand the practice of virtual instruction and how it differs from traditional face-to-face instruction. For policy and practice, this work also suggests a need for greater recognition of these differences, and the strengthening of systems to support and prepare teachers and administrators to be effective in virtual instruction.
Article
Formal achievement data such as test scores and school performance feedback from standardised assessments can be a powerful tool for data‐based decision making and school improvement. However, teachers’ and school leaders’ usage of these data is not necessarily straightforward or predictable. In order to illuminate how educational professionals engage with data in their daily practice, from their own subjective backgrounds and within their own contexts, data use researchers increasingly adopt a sensemaking perspective. Sensemaking, a theoretical construct grounded in psychological and organisational scholarship, offers a framework and a vocabulary to explain how cues such as educational output data are processed in real‐life educational settings. As such, sensemaking research sheds light on reasons why educational professionals’ use of these formal achievement data may deviate from normative expectations. The present study is a conceptual review of how sensemaking is conceived and applied in literature on educational professionals’ use of formal achievement data. In total, 25 empirical and theoretical studies were selected and subjected to thematic analysis. Findings include that sensemaking is used as a lens to study data use, as well as a label for interpretive micro‐processes of data analysis and interpretation, and that formal achievement data can be regarded as sensemaking resources. An integrated conceptual framework on educational professionals’ sensemaking of formal achievement data is presented, including a discussion of critical insights that may inspire future research on data‐based decision making in education.
Article
Background : COVID-19 had a major global impact on education, prompting concerns about its unequal effects and some impetus to reboot equity strategies. Yet, policy processes exhibit major gaps between such expectations and outcomes, and similar inequalities endured for decades before the pandemic. Our objective is to establish how education researchers, drawing on policy concepts and theories, explain and seek to address this problem. Methods : A qualitative systematic review (2020-21), to identify peer reviewed research and commentary articles on education, equity, and policymaking, in specialist and general databases (ERIC, Web of Science, Scopus, Cochrane/ Social Systems Evidence). We did not apply additional quality measures. We used an immersive and inductive approach to identify key themes. We use these texts to produce a general narrative and explore how policy theory articles inform it. Results : 140 texts (109 articles included; 31 texts snowballed) provide a non-trivial reference to policymaking. Limiting inclusion to English-language produced a bias towards Global North articles. Our comparison with a review of health equity research highlights distinctive elements in education. First, education equity is ambiguous and contested, with no settled global definition or agenda (although some countries and international organisations have disproportionate influence). Second, researchers critique ‘neoliberal’ approaches that dominate policymaking at the expense of ‘social justice’. Third, more studies provide ‘bottom-up’ analysis of ‘implementation gaps’. Fourth, more studies relate inequity to ineffective policymaking to address marginalised groups. Conclusions : Few studies use policy theories to explain policymaking, but there is an education-specific literature performing a similar role. Compared to health research, there is more use of critical policy analysis to reflect on power and less focus on technical design issues. There is high certainty that current neoliberal policies are failing, but low certainty about how to challenge them successfully.
Article
Full-text available
Background/Context: Though change is constant in organizations, determining how to successfully implement planned change has been a perennial challenge for both organizational scholars and practitioners. While the empirical knowledge base on planned change in schools and other organizations offers numerous insights, the inattention to activity, or the practice of leading and managing change, remains. Organizational change theory, and specifically organizational routines, offers a useful lens with which to examine planned change in work practice in schools. Purpose: The purpose of this study is to understand the role organizational routines play in changing school work practice. Research Design: We employed a longitudinal case study of leadership practice at Adams, a K-8 urban school, over the course of four years. We spent 90 days collecting observations, interviews, and artifacts from a wide range of individuals (e.g., teachers and administrators). We identified one key routine, the Five Week Assessment Routine, and analyzed the ostensive and performative aspects of the routine, identifying similarities and differences in the routine over time. In addition, we identified two different kinds of change, planned and unplanned, as well as the reasons why participants changed the routine. Conclusions: Organizational routines structure work practice, stabilizing it over time, even in the face of considerable change. They represent one mechanism for sustaining leadership. In addition, routines can be a source of both episodic and continuous change in the work place. New routines can serve as a mechanism to build instructional coherence, internal accountability, and professional community. This account suggests that one way to change norms and culture in an organization is through the design and implementation of new routines. Leaders can create opportunities for change in school practice. By designing and supporting an organizational routine, the leaders at Adams focused the practice of the faculty on improving teaching and learning in ways they had never done before. The case of Adams demonstrates that in the enactment of a routine, school staff can also create change in practice. How they make sense of the routine, and integrate it into their practice, can create shifts in teaching and learning. Change happens in the interplay between individual agency and the structure of the routine. Through a case study of one organizational routine in one school, we analyzed how organizational routines can enable both change and constancy in school practice.
Article
The situative perspective shifts the focus of analysis from individual behavior and cognition to larger systems that include behaving cognitive agents interacting with each other and with other subsystems in the environment. The first section presents a version of the situative perspective that draws on studies of social interaction, philosophical situation theory, and ecological psychology. Framing assumptions and concepts are proposed for a synthesis of the situative and cognitive theoretical perspectives, and a further situative synthesis is suggested that would draw on dynamic-systems theory. The second section discusses relations between the situative, cognitive, and behaviorist theoretical perspectives and principles of educational practice. The third section discusses an approach to research and social practice called interactive research and design, which fits with the situative perspective and provides a productive, albeit syncretic, combination of theory-oriented and instrumental functions of research. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
What is the shoulder-to-the grindstone work of transforming underperforming schools into higher performing schools? What makes this work so difficult? This book sheds light on these questions from the perspective of the Success for All Foundation (SFAF), an organization that has collaborated with thousands of elementary schools to enact a common strategy for comprehensive school reform, all in an effort to improve the reading achievement of millions of students. This story of SFAF spans twenty turbulent years. It begins in 1987, with the strategy of improving reading achievement by improving students' cooperative learning in classrooms. It stretches through 2008, with efforts to influence federal education policy to support that strategy. There is nothing in the story to suggest a quick fix. Rather, the theme that emerges is that the problems and possibilities of effective, large-scale, sustainable education reform lie in the complexity of public education: in interdependencies among underperforming schools, programs of reform, the organizations that advance those programs, and the environments in which all operate. The story ultimately locates the problems of education reform not in schools but, instead, in reformers, themselves. By tracing SFAF's deep push into public education, the purpose of the book is to assist a wide array of reformers in seeing, understanding, and ultimately confronting its complexity.
Article
Many formal organizational structures arise as reflections of rationalized institutional rules. The elaboration of such rules in modern states and societies accounts in part for the expansion and increased complexity of formal organizational structures. Institutional rules function as myths which organizations incorporate, gaining legitimacy, resources, stability, and enhanced survival prospects. Organizations whose structures become isomorphic with the myths of the institutional environment-in contrast with those primarily structured by the demands of technical production and exchange-decrease internal coordination and control in order to maintain legitimacy. Structures are decoupled from each other and from ongoing activities. In place of coordination, inspection, and evaluation, a logic of confidence and good faith is employed.