No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
A framework for evaluating systems initiatives
Abstract
In April 2007, more than 60 evaluators, funders, state leaders, and national experts came together in Pittsburgh for a symposium on evaluating systems initiatives — efforts to build or reform health, education, or human service systems for purposes of improving individual or community well-being. The symposium was sponsored by The Build Initiative, a foundation-funded multi-state effort to ensure that children from birth through age fi ve are safe, healthy, eager to learn, and ready to succeed in school. 2 The Build Initiative supports states' efforts to build comprehensive and coordinated early childhood systems of programs, policies, and services that work together to achieve positive outcomes for young children and their families. 3 Evaluating systems efforts like The Build Initiative in ways that both capture their impact and inform their ongoing development can be a signifi cant challenge. Systems initiatives are complex and notoriously "hard to measure." They involve multiple programs and players and feature outcomes at multiple levels (individual, family, community, and state). They involve numerous public or private funding streams administered through different agencies and decision-making structures. They require aligning goals and coordinating actions across programs with different political cultures. And, they tackle diffi cult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language. Finally, all efforts to improve systems are long-term efforts that evolve over time in response to the political, economic, and social contexts around them. These many complexities place systems initiatives directly outside of the more familiar and more traditional program evaluation comfort zone. Consequently, less consensus exists about how to assess them. Still, systems initiative evaluation is not uncharted territory. Systems initiatives have been around for decades and various evaluation approaches have been tried. Of particular note are "theory of change" evaluation approaches that have gained substantial momentum since the mid 1990s when the Aspen Institute's Roundtable on Comprehensive Community Initiatives for Children and Families introduced them as a promising approach for evaluating complex initiatives. Theories of change are now the cornerstone of many, if not most, systems initiative evaluations. 4 But while theories of change have added much to evaluation practice in this area, they are not (and did not promise to be) a panacea for all evaluation dilemmas that systems initiatives present. . In practice they have been more a way of describing system elements and systems initiative complexities than an evaluation methodology that spells out initiative assumptions and ways of testing whether they are valid.
... Instead, programmatic diversity efforts must be orchestrated to sustain systemic change. We propose using a systems change framework to critically evaluate, develop, and coordinate equity and inclusion efforts across system modes (Coffman, 2007). ...
... The theory of systems change is designed to reform the underlying conditions in a system as they relate to social change, diversity, and inclusion, and was originally conceived in activist pedagogy (Coffman, 2007). Its early applications centered around access to resources related to physical and mental health in early childhood development, and recently has become more frequently utilized in corporate management areas and social organizations (Kania et al., 2018;Seelos & Mair, 2018). ...
... The systems change framework itself is a descriptive set of interconnected spheres or categories of influence of a "system"-for example, a program, department, school, business, organization, or initiative. Systems operate on many organizational levels (e.g., individual, community, state), often have a variety of funding sources, and must "tackle difficult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language" (Coffman, 2007). The framework allows these complexities to be dissected and evaluated without losing sight of the system as a whole (Foster-Fishman & Watson, 2018). ...
While academia is moving forward in terms of diversifying recruitment of undergraduate and graduate students, diverse representation is still not found across the academic hierarchy. At the graduate level, new discussions are emerging around efforts to improve the experiences of women and underrepresented minorities through inclusive graduate programming. Inclusive graduate programs are that which actively center and prioritize support for diverse experiences, identities, career goals, and perspectives, from recruitment through graduation. Establishing regular and rigorous evaluation of equity and inclusion efforts and needs is a critical component of this work. This is recognized by funding agencies that increasingly require reporting on inclusion efforts; here, we suggest use of a systems change framework for these evaluations.
A systems change approach emphasizes three levels: explicit change (e.g., policies), semi‐explicit change (e.g., power dynamics), and implicit change (e.g., biases). We use the Ecology, Evolution, and Behavior (EEB) PhD Program at the University of Texas at Austin in an exercise to (a) identify areas of concern regarding inclusive programming voiced by graduate students, (b) categorize efforts to address these concerns, and (c) integrating and evaluating which areas of the systems change framework show the greatest progress or potential for progress. We argue this framework is particularly useful for academic systems as they are complex, composed of variable individuals, and must address diverse stakeholder needs.
... A system can be defined as "a group of interacting, interrelated, and interdependent components that form a complex and unified whole." 3 The term complex adaptive systems (CAS) acknowledges that programs and activities are not conducted in vacuums, but are part of larger networks, with histories and evolving dynamics. 4 The Health Resources and Services Administration and ACF operate from the idea that state MIECHV programs and home visiting should be regarded as a component rooted within a comprehensive, well-functioning early childhood system that fosters long-term maternal, infant, and early childhood outcomes and strong parent-child relationships. ...
... To define the Missouri MIECHV system, Coffman's five areas of focus were defined: (1) context, (2) components, (3) connections, (4) infrastructure and (5) scale. 3 Primary context is a mission directly related to improving the early childhood (pregnancy to birth to age 5) outcomes in Missouri, or one of MIECHV's benchmarks. Secondary context is a parent organization of T a primary context organization. ...
... The definition of a CAS 4 was then applied against the visual representation of the MIECHV system designed with Coffman's framework, 3 to validate its fit. Because publicly available documents were reviewed, University of Missouri Health Sciences Institutional Review Board approval was not required. ...
Background: The Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program was created by the 2010 Patient Protection and Affordable Care Act. MIECHV provides comprehensive services to at-risk families through evidence-based home visiting programs. Purpose: The following question is addressed: Does the Missouri MIECHV system meet the definition of a complex adaptive system (CAS)? Methods: A systematic review was conducted of documents related to MIECHV programs (federal, state, and local levels), and to affiliated programs with a home visiting and early childhood (aged birth to 5 years) scope. The organizations’ fit was identified for the scope of early childhood home visiting programs, and then its relationship extracted to MIECHV and its affiliates. Results: MIECHV meets the definition of a CAS, being dynamic, massively entangled, scale independent, transformative, and emergent. Over 250 organizations were identified; 19 federal and 79 state organizations; 24 nonprofits at the federal level, 31 at the state; over 150 community-level agencies; and 13 home visiting models implemented in Missouri. Implications: A considerable amount of organizational complexity exists within the MIECHV system and among its affiliates with a home visiting and early childhood scope. The complexity of the system challenges its potential for effective and efficient implementation, coordination, sustainability, and evaluation, and increases the potential for redundancy, overlap, and fragmentation. Evaluating a CAS requires acknowledgement of its complexity, beyond traditional approaches to evaluation. Creating visualization tools of federal, state, and local stakeholders and their relationships is a practical approach for aligning, organizing, and communicating the work flow.
... Context includes improving the political context that surrounds the systems building by developing a shared vision and goals and by educating policymakers and the public so that the changes needed to create systems are sustainable (Schaack et al., 2012, Coffman, 2012). This requires political will in changing the political environments that surround the success of systems change, and tends to come early in the process but should be continuously shifting depending on the political environment (Coffman, 2007). The eight critical elements involved in changing the political climate for early childhood systems, and therefore, addressing changes to " context " in Coffman's framework for change include (a) recognition of need, (b) creating a shared early learning vision, (c) developing political leadership, (d) building capacity and expertise, (e) developing programs, actions, and policy successes, (f) expanding public awareness and support, (g) mobilization and advocacy, and (g) strengthening alignment and readiness for change (Coffman, 2007). ...
... This requires political will in changing the political environments that surround the success of systems change, and tends to come early in the process but should be continuously shifting depending on the political environment (Coffman, 2007). The eight critical elements involved in changing the political climate for early childhood systems, and therefore, addressing changes to " context " in Coffman's framework for change include (a) recognition of need, (b) creating a shared early learning vision, (c) developing political leadership, (d) building capacity and expertise, (e) developing programs, actions, and policy successes, (f) expanding public awareness and support, (g) mobilization and advocacy, and (g) strengthening alignment and readiness for change (Coffman, 2007). Coffman's description of " components " includes establishing high performing programs and services and expanding and increasing access to them within the system (Schaack et al., 2012; Coffman, 2012). ...
... Coffman's description of " components " includes establishing high performing programs and services and expanding and increasing access to them within the system (Schaack et al., 2012; Coffman, 2012). According to Coffman (2007) a challenge to systems work is not necessarily the lack of connections, but the lack of parts to connect in the first place, therefore, ensuring that an adequate supply of services and programs exists is an important foundation of systems change that should not be overlooked. Coffman's third factor, " connection " includes creating strong linkages across system components that assist in fostering cross sector planning, promoting collaboration and referrals, and aligning standards that improve results and make a system a system (Schaack et al., 2012; Coffman, 2012). ...
The purpose of this qualitative interview study was to examine complexities that emerged throughout the process of implementing California’s Race to the Top Early Learning Challenge (RTTT:ELC) grant in Alameda County. Data was collected through individual semi-structured interviews with six distinct stakeholder groups including individuals working on the implementation of the RTTT:ELC at state, regional, county, local, and provider levels. Data was primarily analyzed using descriptive coding and analytic memoing to identify the complexities participants reported and to examine the various responses stakeholders had to the policy implementation process. Additionally, deductive coding was completed using concepts drawn from Coffman’s (2012) functional analysis, Spillane et al.’s (2002) cognitive sense-making framework, and Heifetz’s (1994) adaptive leadership theory. Findings of the study indicated the identification of similar complexities to policy implementation activities (i.e., resources, messaging of the policy intent, collaboration) represented across all stakeholder groups. The findings also indicated stakeholders perceived the implementation challenges as adaptive, versus technical, in nature. This study makes an important contribution to the literature by documenting the sense-making process of policy implementation in early childhood and implications for systems building at state and national levels.
... The EBHV grantees operate in complex systems, conceptualized as groups of interrelated and interdependent agents (individuals and organizations) that are working together in various settings on activities that influence the prevention of child maltreatment (Holland 1995;Foster-Fishman et al. 2007). These webs of agents form a complex whole that changes as the parts interact (Wheatley 1992;Kauffman 1995;Coffman 2007). ...
... A better alternative is to specify a theory of change that articulates the alignment of activities that are critical to system change (Walker and Kubish 2008). It is important to begin the evaluation process with a "clear map of what initiatives are accomplishing compared to what they are trying to achieve, and how the change process is expected to occur" (Coffman 2007). A developmental evaluation approach, in which evaluators work closely with evaluation stakeholders to understand and support developmental or emergent changes in program design, may also be appropriate (Coffman 2007;Patton 2008). ...
... It is important to begin the evaluation process with a "clear map of what initiatives are accomplishing compared to what they are trying to achieve, and how the change process is expected to occur" (Coffman 2007). A developmental evaluation approach, in which evaluators work closely with evaluation stakeholders to understand and support developmental or emergent changes in program design, may also be appropriate (Coffman 2007;Patton 2008). Another response is to shift evaluation designs over time or to use multiple designs simultaneously or sequentially in order to capture multiple dynamics operating within grantees' initiatives (Parsons 2007). ...
This report grounds the cross-site systems domain evaluation approach in the system literature. It focuses on three aspects: (1) the system-based evaluation approach and theory of change, (2) core infrastructure concepts, and (3) system-based evaluation methods.
... Systems thinking is an approach for solving complex problems, that emphasises looking at the whole rather than the isolated parts, and highlighting the relationships between the parts, their causal linkages and feedback loops [1,2]. The application of systems thinking to prevention is experienced at two levels [3]. ...
... Systems thinking is drawn from the overlapping sciences of systems theory and complexity theory [19], where system dynamics and soft systems methodology are considered to be the relevant approaches to health promotion [5,20,21] that are currently being used in community-based prevention [22][23][24][25][26]. These methods shift the focus of health promotion activities away from a reductionist perspective to a holistic one [27], recognising that the essence of structure is not the parts themselves, but the relationships between the parts, their causal connections [2]. This is a fundamental shift in thinking for most health promotion professionals moving from objectives-based planning and considering 'Where do I implement a project?', to problem-solving processes that view the complexity of an issue by asking 'What needs to change to improve the situation?' [28]. ...
Prevention systems improve the performance of health promotion interventions. This research describes the establishment of the Australian state government initiative, Healthy Together Victoria's (HTV) macro infrastructure for the delivery of large-scale prevention interventions.
Methods:
This paper reports on findings of 31 semi-structured interviews about participants' understanding of systems thinking and their reflections of the strengths and weaknesses of the HTV prevention system. A chronic disease prevention framework informed the coding that was used to create a causal loop diagram and a core feedback loop to illustrate the results.
Results:
Findings highlighted that HTV created a highly connected prevention system that included a sizeable workforce, significant funding and supportive leadership. Operating guidelines, additional professional development and real-time evaluation were significant gaps, which hindered systems practice. For inexperienced systems thinkers, these limitations encouraged them to implement programs, rather than interact with the seemingly ambiguous systems methods.
Conclusions:
HTV was an innovative attempt to strengthen health promotion infrastructure, creating a common language and shared understanding of prevention system requirements. However, the model was inadequate for HTV to achieve population-level reductions in chronic disease as system oversight was missing, as was an intervention delivery focus. Clarity was needed to define the systems practice that HTV was seeking to achieve. Importantly, the HTV prevention system needed to be understood as complex and adaptive, and not prioritized as individual parts.
... All of this fits within an overall integrated marine risk assessment and risk management framework which has been previously identified and refined as DAPSI(W)R(M) (pronounced dap-see-worm, derived from DPSIR, Elliott et al., 2017a;EEA, 1999; details given below). Coffman (2007) emphasised the understanding, mapping and evaluating change in a system as "a group of interacting, interrelated, and interdependent components or not that form a complex and unified whole". Coffman (2007) used a social system to link the context, components, connections, infrastructure and scale for each of the activities, outcomes and impacts in an area. ...
... Coffman (2007) emphasised the understanding, mapping and evaluating change in a system as "a group of interacting, interrelated, and interdependent components or not that form a complex and unified whole". Coffman (2007) used a social system to link the context, components, connections, infrastructure and scale for each of the activities, outcomes and impacts in an area. The overall purpose or goal of the system is achieved through the actions and interactions of its components, its values, its actors (stakeholders) and their relationships, its policies and its resources, and techniques and technologies, thereby showing the complexity of coupled socio-ecological systems (Liu et al., 2007;Ostrom, 2009). ...
Marine and estuarine management requires an excellent understanding of the interacting, interrelated and interdependent sub-systems comprising ecological, societal and management complexity. Managing such a complex system sustainably relies on knowing what aspects can be managed, and conversely what aspects are outside the control of the manager. Accordingly, by taking elements from existing environmental management approaches, especially in Europe and Canada, here we propose an integrated systems analysis approach which links 14 component sub-systems. Using these cases shows that while all elements exist, they have hitherto not been combined into a holistic decision support system. These components are linked here in a cycle of three Parts - (A) defining the policy problems facing the seas, (B) obtaining the relevant and fit-for-purpose natural and social sciences data and information, and (C) creating an input for policy and decision-making which involves stakeholders. The component sub-systems are: an Underpinning Framework Sub-system (1), which then leads to the Issue Sub-system (2), which is vision-related and includes causes and/or consequences of pressures to be managed. The Ecological Sub-system (3) links the biota and its environment to the Socio-ecological Sub-system (4) and the Socio-economic subsystem (5), which considers the macroeconomic aspects. The Resources and Delivery Sub-system (6) considers which scientists do what and how do they do it and the Provenance Sub-system (7) checks that there is a fit-for-purpose and defendable science evidence base. The Governance Sub-system (8), incorporates policies and politics as well as horizontally and vertically integrating the Legislative (8A) and the Administrative Sub-systems (8B). The Communication (8C) and Stakeholder Sub-systems (8D) ensure involvement across the stakeholder typology (of formal and informal actors). Finally, the Achievement Sub-system (9) and the Feedback Sub-system (10) ensures that all of these actions achieve successful and sustainable marine resource management.
... • local system conditions aligned with change goals, including supportive policies and practices, power dynamics, network exchanges, and resource access (Coffman, 2007;Foster-Fishman & Watson, 2017); ...
... identified as critical to system functioning (e.g., Coffman, 2007;Foster-Fishman, et al., 2007): mindsets, program components, connections, regulations, resources, and power. By making the system the focus of inquiry, the system scan engages diverse stakeholders in a critical analysis of the local community, helping to move the conversation away from victim blaming to a recognition that the community system propagates and maintains poor outcomes. ...
The act of transforming community outcomes requires diverse stakeholders across an array of settings to become actors of change. Drawing from the ABLe Change Framework systemschange model, this article presents four processes used in numerous communities across the United States to effectively engage diverse stakeholders in taking actions to improve local systems. This article introduces the ABLe Change Framework tools, which are used to promote these action-oriented habits, and then discusses how foundations can use them to create the conditions that promote inclusive community change. © 2018 Dorothy A. Johnson Center for Philanthropy, Grand Valley State University.
... Efforts to improve such systems are necessarily long-term and evolve in response to surrounding political, economic and social conditions (Coffman, 2007). Systemlevel change is difficult and depends on a good understanding of national needs and untapped potentials; it also requires early engagement of key stakeholders in pursuit of shared priorities and an agreed vision (Kadletz et al., 2021). ...
This article reports the findings from a phenomenographic study of career experts’ conceptions of systems development in lifelong guidance settings. The results show that conceptions of systems development in lifelong guidance varied from minimal, aspirational, strategic to systemic. By exploring the logical relationship between qualitatively different conceptions, it provides policymakers and other stakeholders with a way of holistically viewing the varying levels of lifelong guidance systems development. The matrix presented in this article may serve as a catalyst for reflection on crucial elements, such as legislation, leadership and cooperation, that have the potential to improve systems development in lifelong guidance.
... An entity of interacting, interrelated and interdependent components that form a complex whole (Coffman, 2007) To trace the dynamic relationships between the actors, establishments, and infrastructures of the neighbourhood -its residents, socioeconomic organisations, public spaces, establishments, heritage, and culture -in reference to the municipality of Leuven wherein it is located, and to the broader world. ...
Cities and dense urban areas are dynamic environments, always adapting to changing circumstances and shocks, such as the recent COVID-19 pandemic. Vaartkom, a neighbourhood in Leuven, provides an interesting case-study, having undergone a drastic transformation in the past two decades, from dilapidated industrial zone to residential quarter and cultural hot-spot. This has introduced a demographic shift, which inevitably influences the use of public and private space in the neighbourhood, creating new areas for inclusion and exclusion. Our research focuses on how the use of public space has changed under COVID-19, and how community members envision their neighbourhood in a post-COVID context. We employed various methods – such as interviews, site visits, stakeholder and physical mapping exercises – and worked with the community to identify the different areas of in-and exclusivity. Some findings relate to the conflicting expectations about the use and future of public space and the link between the location of public engagement and the level of inclusiveness. This illustrates the magnitude and consciousness of the effort required to be truly inclusive. Above all, our own understanding of inclusivity broadened significantly over the duration of the project, illustrating the clear advantage of using a transdisciplinary approach in research. Our findings have been summarised in a small video.
... Second, those working on system change evaluations identify shared challenges, tensions, and uncertainties all regarding the lack of a fixed, stable intervention model and, instead, a focus on adaptation (Beer, 2017;Patton, 2011;Patton et al., 2015) and "continuous, intentional learning" (Beer, 2017, p. 5;Knight et al., 2017). Third, there have been a series of papers providing guidance for systems-oriented and system change evaluation (Beer, 2017;Coffman, 2007;Dichter, 2008;Hargreaves, 2010Hargreaves, , 2018Latham, 2014;Poth et al., 2016). These outline the following considerations: capturing interim outcomes (Beer, 2017); treating learning as an outcome from the outset (Beer, 2017); developing a conceptual theory of change that gives stakeholders flexibility to account for their contexts (Dichter, 2008); creating shared understanding of measures (Poth et al., 2016); and employing systems methods (e.g., system mapping, system dynamics modeling) (Hargreaves, 2010). ...
This chapter explores evaluation about and for system change within a case study of The Rippel Foundation's ReThink Health initiative. Their work involves field building to understand and shift what constrains current systems influencing health and well‐being and place‐based efforts to change systems carried out with regionally focused stewards. Guiding questions addressed how ReThink Health envisions, supports, and evaluates system change. Data included document review and group and individual interviews with staff members followed by iterative cycles of descriptive pattern coding and analysis. Seven, interrelated findings depict system change as (1) reframing problems and (2) cultivating stewardship supported by (3) adapted and new tools. Additionally, evaluation (4) fuels learning, (5) uses a developmental theory of system change, (6) assesses changes to system conditions, and (7) seeks to continuously develop value. Authors identify considerations for others engaged in system change evaluation.
... Second, those working on system change evaluations identify shared challenges, tensions, and uncertainties all regarding the lack of a fixed, stable intervention model and, instead, adaptation (Beer, 2017;Patton, 2011Patton, , 2016 requiring "continuous, intentional learning" (Beer, 2017, p. 5;Knight et al., 2017). Third, there have been a series of papers providing guidance for systems-oriented and system change evaluation (Beer, 2017;Coffman, 2007;Dichter, 2008;Hargreaves, 2010Hargreaves, , 2018Latham, 2014;Poth et al., 2016). These outline the following considerations: ...
Pressing social problems and inequities call attention to the need for interventions that target root causes and change systems. A significant challenge facing system change efforts is the lack of systems-based evaluation theory and methods. To address this challenge, we conducted an instrumental case study of evaluation within an exemplary philanthropy engaged in system change work for over a decade. Research questions address how the foundation envisions, supports, and evaluates system change. Data collection included document review and group and individual interviews with staff members followed by two cycles of descriptive coding and analysis. Seven, interrelated findings depict system change as (1) reframing problems and (2) cultivating stewardship supported by (3) adapted and new tools. Evaluation (4) fuels learning, (5) uses a developmental theory of system change, (6) assesses changes to system conditions, and (7) seeks to continuously develop value. We conclude with implications for system change evaluation.
... All of this fits within an overall integrated marine risk assessment and risk management framework which has been previously identified and refined as DAPSI(W)R(M) (pronounced dap-seeworm, derived from DPSIR, Elliott et al., 2017a;EEA 1999; details given below). Coffman (2007) emphasised the understanding, mapping and evaluating change in a system as "a group of interacting, interrelated, and interdependent components or not that form a complex and unified whole". Coffman (2007) used a social system to link the context, components, connections, infrastructure and scale for each of the activities, outcomes and impacts in an area. ...
Marine and estuarine management requires an excellent understanding of the interacting, interrelated and interdependent sub-systems comprising ecological, societal and management complexity. Managing such a complex system sustainably relies on knowing what aspects can be managed, and conversely what aspects are outside the control of the manager. Accordingly, by taking elements from existing environmental management approaches, especially in Europe and Canada, here we propose an integrated systems analysis approach which links 14 component sub-systems. Using these cases shows that while all elements exist, they have hitherto not been combined into a holistic decision support system. These components are linked here in a cycle of three Parts - (A) defining the policy problems facing the seas, (B) obtaining the relevant and fit-for-purpose natural and social sciences data and information, and (C) creating an input for policy and decision-making which involves stakeholders. The component sub-systems are: an Underpinning Framework Sub-system (1), which then leads to the Issue Sub-system (2), which is vision-related and includes causes and/or consequences of pressures to be managed. The Ecological Sub-system (3) links the biota and its environment to the Socio-ecological Sub-system (4) and the Socio-economic subsystem (5), which considers the macroeconomic aspects. The Resources and Delivery Sub-system (6) considers which scientists do what and how do they do it and the Provenance Sub-system (7) checks that there is a fit-for-purpose and defendable science evidence base. The Governance Sub-system (8), incorporates policies and politics as well as horizontally and vertically integrating the Legislative (8A) and the Administrative Sub-systems (8B). The Communication (8C) and Stakeholder Sub-systems (8D) ensure involvement across the stakeholder typology (of formal and informal actors). Finally, the Achievement Sub-system (9) and the Feedback Sub-system (10) ensures that all of these actions achieve successful and sustainable marine resource management.
... It is becoming a preferred health promotion approach (Allender et al., 2015;Brennan, Sabounchi, Kemner, & Hovmand, 2015;House of Commons, 2018;Sims & Aboelata, 2019) for managing complex communitybased problems that brings systemic solutions to prevention that aim to alter the underlying structure by challenging values, policies and procedures, norms, routines, rules and regulations, capacity and practices, resources, and power that drive systems (Abercrombie, Harries, & Wharton, 2015;Foster-Fishman, Nowell, & Yang, 2007;Mitchell & Baur, 2019). Systems thinking recognizes that the essence of structure is not the parts themselves but the relationships between the parts, their causal connections (Coffman, 2007). Best (2011) warns that this is a fundamental shift in thinking for most health promotion practitioners as this is not the how they design interventions. ...
Background. A systems mindset is the ability to see problems in their wider context and in terms of their underlying structure. This research describes how a systems mindset was understood and applied by prevention practitioners in a large-scale community-based initiative that employed a systems thinking approach. Method. This qualitative research included 31 primary semistructured interviews. Deductive thematic analysis was based on Braun and Clarke’s analysis framework and was guided by Senge and Scharmer’s knowledge-creating system. Results. The practitioners had been introduced to systems theory and were aware of complex problems and the need for equally sophisticated solutions. Their knowledge was not in-depth, although this may be adequate, as a theoretical overview seemed to be sufficient to support practice. A range of tools was available to practitioners to guide their systems mindset; however, none were preferred. Practitioners’ awareness of the tools varied, as did their feelings toward them as some found them helpful and others did not. A narrower focus on tools could have benefited those who had not yet grasped systems theory. The use of projects within a systems approach confused some practitioners, yet others saw them as platforms to leverage change from. Implications for practice. With a systems mindset practitioners are able to develop systemic solutions to difficult problems. To do this, they require an overview of complex adaptive systems theory, an applied understanding of systems tools, and an experiential learning opportunity to shift their knowledge into practical know-how.
... The Build Framework (Coffman, 2007) was used as an analytical construct to examine ECD systems change process and impact and to connect the county's diverse efforts to improve early childhood systems. Build is a research-based framework for evaluating initiatives that have systems change as a key goal and outcome. ...
The Neil and Louise Tillotson Fund - a donor-advised fund of the New Hampshire Charitable Foundation - invests in early childhood development in Coös County, New Hampshire's largest and most rural and economically disadvantaged county. Local community members joined forces with the fund to create an integrated early childhood development system for Coös' children and families. The evaluation documented increased capacity and quality and surfaced lessons for funders and others pursuing systems change efforts in early learning, and in rural areas more broadly. © 2018 Dorothy A. Johnson Center for Philanthropy, Grand Valley State University.
... LADDERS is similar to other frameworks most often published in the grey literature by consulting companies in preparing reports for clients such as the CDC, AHRQ, and the United Way, or those used by change institutes and consultants like the Tamarack Institute and the 5 core conditions in their Collective Impact concept 7 and the Build Initiative's framework for evaluating systems initiatives. 8 The description of LADDERS invariably generates "face validity"-everyone nods theirs heads and acknowledges these elements. Thus, it is probably not necessary to fully describe the actual LADDERS elements individually as there is ample organizational and systems change literature for that. ...
The science of learning health systems borrows and adapts models from many fields. One in particular is implementation science which has been experiencing a flourishing of new theories, models, and frameworks, some of which are generating sufficient evidence as to their effectiveness and applicability to emerge as candidates for wide adoption as useful tools for the field. In reviewing these, a common paradigm can be described which is a synthesis of those elements regularly cited by health systems implementing successful transformational change activities. As a paradigm, it offers a practical bridge to these models, concepts, and frameworks that are often hard to operationalize and are used with varying degrees of completeness. These elements can be arranged in a memorable acronym—LADDERS—Leadership, Alignment, Data, Demonstration, Evaluation, Replication, and Sustainability. LADDERS identifies the multiple elements and dimensions recognized by persons involved in leading health system change activities. It provides a simple, useful way to assess progress by health systems in planning, implementing, evaluating, and sustaining change. There is ample organizational and systems change literature to fully describe the actual LADDERS elements individually; therefore, this article describes characteristics and functions of each element and the dynamics represented in a DNA image to reflect that in learning health systems change is recursive, constant, and happens in complex environments that are always readjusting to new stimuli and directions, and this is often not accounted for in a framework, model, or theory. It concludes with several examples of application of the LADDERS paradigm and suggests how it is a complementary approach to accomplishing Institute of Medicine Learning Health Systems goals.
... A few existing frameworks describe outcome areas related to advocacy and policy-oriented work, and describe the areas of infrastructure and other interim outcomes that reflect enabling conditions or otherwise signal progress for long-term policy change or social change. See Reisman, et al., 2007;Coffman, 2007;Reisman, Gienapp, & Kelly, 2015; Alliance for Justice, 2013; and Klugman, 2010. ...
Key Points
As its seven-year Nuclear Security Initiative wound down in late 2014, the William and Flora Hewlett Foundation engaged ORS Impact to conduct a summative evaluation. That evaluation yielded insights pertinent to future work on nuclear security and other fields where policy-related investments, strategies, and goals are prioritized, as well as insights regarding Hewlett’s approach to the initiative exit.
During the life of the initiative, significant changes in the geopolitical landscape influenced both the relevance and the expected pace of advancement of its established goals and targets. Rather than focusing on whether identified targets had been achieved in a narrow “success/failure” framework, the evaluation explored where and how Hewlett’s investments and actions made a difference and where meaningful progress occurred over the seven years of investment. Evaluation findings highlighted contributions and areas of progress that had not been explicitly anticipated or specifically identified in the initiative’s theory of change.
This article describes the initiative and its theory of change, evaluation methods and approaches, findings, and how these informed the foundation’s planning for initiative exits and approach to measurement of time-bound investments. Although time-bound philanthropic initiatives are a well-established practice, the approach merits closer examination in order to discern effective ways to implement, evaluate, and wind down these types of investments.
... System mapping, also referred to as concept mapping, is the process of creating a visual tool that depicts a given system or relationships. A system is a group of interacting, interrelated, and interdependent components that form a complex and unified whole (Coffman, 2007). A system's purpose is achieved through the actions and interactions of its components. ...
The objective of this paper is to introduce a pedagogical tool, the Comparative Political Economy System (CPES) Matrix, that which can be used to develop a students’ understanding of the global business environment and how that environment affects a firm’s business decisions. To gain such understanding, students can use a concept matrix to plot several countries’ political economies into a single, compelling image. The resulting comparative system matrix often provides an “Aha” moment for students, because they have now a visual image and can begin to understand how the various components in the matrix interact together.
... Interventions that are designed with complexity in mind attempt to facilitate self-organization by creating the conditions for it to flourish, in part through the development of relationships between actors within subsystems and between actors across systems (Catsambas et al. 2008;Parsons 2009). The importance of understanding self-organization is a common feature of system approaches to evaluation (Cabrera et al. 2008;Catsambas et al. 2008;Coffman 2007;Leischow and Milstein 2006;Williams and Hummelbrunner 2010). ...
... By bringing evidence based practices to the field via a simple to use checklist, the authors hope to help contribute to the development of a comprehensive early childhood system and that supports programs in the provision of high quality services to children and families. Coffman (2007) has proposed a framework for building early childhood systems that reflects an intentional, organized approach to creating or improving both the system and the outcomes it is designed to produce. Coffman's typology addresses the need for a common understanding about systems-building efforts and describes five areas of focus, presented in Table 1. ...
The prevalence of preschool expulsion, coupled with racial disparities in expulsion rates and the potential long term negative effects of challenging behaviors in the early years, has created an urgent need to build early childhood systems to address these issues. The teaching and guidance policy essentials checklist (TAG-PEC) has been developed to assess nine essential features of high quality early childhood discipline policies and can be used by early childhood programs in a variety of settings to evaluate, refine, and revise existing policies or guide in the development of new policies. Using data from 282 guidance policies assessed using the TAG-PEC, the authors use a framework for systems building developed by Coffman (A framework for evaluating systems initiatives, 2007) to position the TAG-PEC as a powerful tool that can be used to help build and support an effective and cohesive early childhood system. Findings indicate that, overall, discipline policies were not high quality and there was a great deal of variability in the TAG-PEC scores. By bringing evidence based practices to the field via a simple to use checklist, the authors hope to help contribute to the development of a comprehensive early childhood system and that supports programs in the provision of high quality services to children and families.
... To implement the strategy and provide a conceptual framework for the councils, The Trust adopted a systems building framework developed by Julia Coffman (2007). This framework provided the councils with a common language to use during their planning phase and in their implementation grant applications and helped to ensure that they remained focused on systems building. ...
Key Points
· Systems that provide services to children tend to operate in silos; foundations can play a role in helping bridge these silos by supporting “systems building” efforts.
· Using examples from two foundations and two communities, this article explores the challenges and lessons learned in systems building work.
· Educating grantees and other community members about systems and systems building is a critical first step in the process.
· Supporting systems building requires an iterative process and foundations should continuously reinforce the importance of systems building activities.
... In addition, a systems-oriented evaluation evolves as the program changes in response to its internal and external conditions, and to the relationships between and among different actors. Consequently, an evaluation's design must be responsive to shifts in the initiative's focus, resources, needs, opportunities, and challenges (Coffman 2007;Parsons 2007;Williams and Iman 2007;W.K. Kellogg Foundation 2007). ...
Abstract In this article, I invite readers to think outside of evaluation’s current boundaries and to see the deep connectedness between what museums hope to achieve and how we evaluate the extent to which these aspirations may be realized. To do this, I present four imperatives for making museum evaluation more relevant, credible, and useful: 1) Link program activities with intended outcomes and hoped-for impact. 2) Take a systems-oriented evaluation approach. 3) Use affirmative data collection approaches based on assets and strengths. 4) Engage in courageous conversations.
... Table II.5 Infrastructure Capacity Categories by Types of Activities Infrastructure Capacity Categories Types of Activities Planning Strategic planning, tactical planning, decision making Operations Outreach, intake, screening, assessment, referral procedures Fiscal Strategies Fiscal partnering, fundraising, researching funding sources, leveraging dollars to support direct services Communications Information sharing, dissemination of lessons learned, policy advocacy, marketing, public awareness, disseminating information through the media Collaboration Leadership, alignment of goals and strategies, development of relationships, working through existing partnerships Community and Political Support Building community awareness and support, building political buy-in and support Workforce Capacity Training, technical assistance, coaching, supervision, retaining staff Evaluation Capacity Data collection, storage, retrieval, and analysis for quality assurance, quality improvement, epidemiology, surveys, or program evaluation Sources: Flaspohler et al. 2008; Coffman 2007; October 2008 ...
This report summarizes key activities conducted during the initial planning year and provides reflections on experiences and lessons learned using a participatory approach to evaluation planning.
The role of education staff in supporting social, emotional, and mental health (SEMH) difficulties is critical, however access to specialist mental health input can be difficult. Digitally‐mediated service delivery was implemented during the COVID‐19 pandemic and may be integrated in a post‐COVID‐19 world to support education staff. Using a systems change evaluation framework, this study explored staff perceptions of a digitally‐mediated, multidisciplinary model for SEMH needs in primary school settings. Participants were education staff ( n = 6) from five participating schools in a rural county in the UK and the multidisciplinary team ( n = 7). Focus groups were conducted at baseline, 12 months, and 24 months during a 2‐year pilot project. Thematic analysis was used to analyse the data. Education staff and the multidisciplinary team perceived progress over time in relation to increased capacity (for early intervention) and improved coordination (between relevant children's services). Digitally‐mediated service delivery facilitated changes in practice that were empowering for education staff and supportive in meeting communication needs. This study provides preliminary support for acceptability of digitally‐mediated team communication as an enhanced model of SEMH service provision in primary school settings in the UK.
BACKGROUND
Service delivery organizations are advancing the provision of trauma-informed care (TIC) for youth to improve outcomes. However, currently there are no validated, reliable evaluation measures to capture the voices of adolescent clients and how well they perceive TIC implementation.
AIMS
The purpose of this project was to create an evaluation measure with strong content validity for adolescent health and service users to give feedback to organizations about their implementation of TIC. This article outlines Step 1 of our instrument development, by discussing our process creating the measure and affirming content validity. Psychometric testing of this measure (Step 2) is described in a companion paper.
METHOD
We combined deductive theory substruction with an inductive participatory process to create, revise, and finalize the measure. The National Center for Trauma-Informed Care’s framework of four practices and six principles was substructed into an 18-item draft measure. A four-member community youth advisory board (CYAB) then worked to inductively modify our draft to provide age-appropriate clarity and ensure a nontriggering respondent experience. Finally, the CYAB members conducted cognitive interviews with 10 other adolescents in local evening data collection events, refining the measure for future psychometric testing.
RESULTS
The process resulted in a 20-item form based on CYAB feedback. Refinements included providing an accessible definition of trauma, asking questions about trauma history, and asking whether trauma affected the client’s visit on the day of service use. The CYAB involvement enhanced content validity and ensured a trauma-informed instrument development approach.
While academia is moving forward in terms of diversifying recruitment of undergraduate and graduate students, diverse representation is still not found across the academic hierarchy. At the graduate level, new discussions are emerging around efforts to improve the experiences of women and underrepresented minorities through inclusive graduate programming. Inclusive graduate programs are that which actively center and prioritize support for diverse experiences, identities, career goals, and perspectives, from recruitment through graduation. Establishing regular and rigorous evaluation of equity and inclusion efforts and needs is a critical component of this work. This is recognized by funding agencies that increasingly require reporting on inclusion efforts; here we suggest use of a systems change framework for these evaluations.
A systems change approach emphasizes three levels: explicit change (e.g. policies), semi-explicit change (e.g. power dynamics), and implicit change (e.g. biases). We use the Ecology, Evolution, and Behavior (EEB) PhD Program at the University of Texas at Austin in an exercise to (1) identify areas of concern regarding inclusive programming voiced by graduate students, (2) categorize efforts to address these concerns, and (3) integrating and evaluating which areas of the systems change framework show the greatest progress or potential for progress. We argue this framework is particularly useful for academic systems as they are complex, composed of variable individuals, and must address diverse stakeholder needs.
In this paper, we present the process of developing and evaluating an instrument designed to measure the extent to which a complex community system has changed as a result of a community initiative, and for purposes of this research, doing this within the content area of developing local trauma-informed child welfare systems in specific communities in Michigan. The instrument was designed for the Southwest Michigan Children’s Trauma Assessment Center’s (CTAC) Substance Abuse and Mental Health Services Administration (SAMHSA)-funded initiative to bring a trauma-informed perspective to professionals working with children in child welfare. The Trauma Informed System Change Instrument was developed with the input of experts in trauma-informed system change. Two parts of the instrument were analyzed separately using confirmatory factor analysis. A two factor model was fit for Community Characteristics and a three factor model for Individual Characteristics. Although adequate factorial validity was obtained for the instrument, specific items on the instrument that were problematic in fitting the model were identified, and suggestions for revising the instrument for improved functionality are offered, as are other potential uses of the instrument.
This report focuses on approaches used by QRISs in Indiana and Pennsylvania to connect with and build on the programs and resources that exist within the early childhood care and education system.
This toolkit is an informational resource for state administrators, child care and early education practitioners, and other stakeholders on how QRISs work, how to plan and design QRIS evaluations, and why such evaluations are important.
This paper presents a new approach to the design and implementation of community change efforts like a System of Care. Called the ABLe Change Framework, the model provides simultaneous attention to the content and process of the work, ensuring effective implementation and the pursuit of systems change. Three key strategies are employed in this model to ensure the integration of content and process efforts and effective mobilization of broad scale systems change: Systemic Action Learning Teams, Simple Rules, and Small Wins. In this paper we describe the ABLe Change Framework and present a case study in which we successfully applied this approach to one system of care effort in Michigan.
OCIAL SCIENCE HAS PROVEN ESPECIALLY inept in offering solutions for the great problems of our time—hunger, violence, poverty, hatred. There is a pressing need to make headway with these large chal- lenges and push the boundaries of social inno- v a t ion t o m a ke re a l prog re s s . T he ver y possibility articulated in the idea of making a major difference in the world ought to incorpo- rate a commitment to not only bring about sig- nificant social change, but also think deeply about, evaluate, and learn from social innova- tion as the idea and process develops. However, because evaluation typically carries connota- tions of narrowly measuring predetermined out- comes achieved through a linear cause-effect intervention, we want to operationalize evalua- tive thinking in support of social innovation through an approach we call developmental evaluation. Development al evaluation is designed to be congruent with and nurture developmental, emergent, innovative, and trans-
Build evaluation symposium: Evaluating early learning systems building initiatives
- C Bruner
Bruner, C. (2006). Build evaluation symposium: Evaluating early learning systems building initiatives. Unpublished manuscript.
New approaches to evaluating systems: The Gwen R. Iding Brogden Distinguished Lecture Series
- H Weiss
46 Weiss, H. (1995). New approaches to evaluating systems: The Gwen R. Iding Brogden Distinguished Lecture Series (pp. 411-416).
(in press) Chapter 10: Conceptualizing the intervention: Alternatives for evaluating theories of change
- M Q Patton
Patton, M.Q. (in press). Chapter 10: Conceptualizing the intervention: Alternatives for evaluating
theories of change. In Patton, M.Q. (in press), Utilization-focused evaluation (4th ed.). Thousand
Oaks, CA: Sage Publications.