Methods and Metrics Challenges of Delivery-System Research

School of Public Health, Department of Health Management and Policy, The University of Michigan, Ann Arbor, MI, USA.
Implementation Science (Impact Factor: 4.12). 03/2012; 7(1):15. DOI: 10.1186/1748-5908-7-15
Source: PubMed


Many delivery-system interventions are fundamentally about change in social systems (both planned and unplanned). This systems perspective raises a number of methodological challenges for studying the effects of delivery-system change--particularly for answering questions related to whether the change will work under different conditions and how the change is integrated (or not) into the operating context of the delivery system.
The purpose of this paper is to describe the methodological and measurement challenges posed by five key issues in delivery-system research: (1) modeling intervention context; (2) measuring readiness for change; (3) assessing intervention fidelity and sustainability; (4) assessing complex, multicomponent interventions; and (5) incorporating time in delivery-system models to discuss recommendations for addressing these issues. For each issue, we provide recommendations for how research may be designed and implemented to overcome these challenges.
We suggest that a more refined understanding of the mechanisms underlying delivery-system interventions (treatment theory) and the ways in which outcomes for different classes of individuals change over time are fundamental starting points for capturing the heterogeneity in samples of individuals exposed to delivery-system interventions. To support the research recommendations outlined in this paper and to advance understanding of the "why" and "how" questions of delivery-system change and their effects, funding agencies should consider supporting studies with larger organizational sample sizes; longer duration; and nontraditional, mixed-methods designs.A version of this paper was prepared under contract with the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services for presentation and discussion at a meeting on "The Challenge and Promise of Delivery System Research," held in Sterling, VA, on February 16-17, 2011. The opinions in the paper are those of the author and do not represent the views or recommendations of AHRQ or the US Department of Health and Human Services.1.

Download full-text


Available from: · License: CC BY
  • Source
    • "A systems-focused approach is recommended to articulate interdependent components that contribute to or compromise the effectiveness of health care interventions or programs. It provides insight into the questions of ‘why’ , ‘how’ and the ‘what’ of contexts [16]. ‘The parts do not have to be working well, the purpose may be irrational, but it is a system none the less’ [14], p. 2. The boundary of a system need not correspond with recognised departmental, institutional or other physical boundaries [11]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Insight into local health service provision in rural communities is limited in the literature. The dominant workforce focus in the rural health literature, while revealing issues of shortage of maldistribution, does not describe service provision in rural towns. Similarly aggregation of data tends to render local health service provision virtually invisible. This paper describes a methodology to explore specific aspects of rural health service provision with an initial focus on understanding rurality as it pertains to rural physiotherapy service provision. Method A system theory-case study heuristic combined with a sequential mixed methods approach to provide a framework for both quantitative and qualitative exploration across sites. Stakeholder perspectives were obtained through surveys and in depth interviews. The investigation site was a large area of one Australian state with a mix of rural, regional and remote communities. Results 39 surveys were received from 11 locations within the investigation site and 19 in depth interviews were conducted. Stakeholder perspectives of rurality and workforce numbers informed the development of six case types relevant to the exploration of rural physiotherapy service provision. Participant perspective of rurality often differed with the geographical classification of their location. The numbers of onsite colleagues and local access to health services contributed to participant perceptions of rurality. Conclusions The complexity of understanding the concept of rurality was revealed by interview participants when providing their perspectives about rural physiotherapy service provision. Dual measures, such as rurality and workforce numbers, provide more relevant differentiation of sites to explore specific services, such rural physiotherapy service provision, than single measure of rurality as defined by geographic classification. The system theory-case study heuristic supports both qualitative and quantitative exploration in rural health services research.
    Full-text · Article · Jul 2014 · BMC Medical Research Methodology
  • Source
    • "Although organizational readiness for change in healthcare settings has been identified as an important issue [4], research on the topic is hindered by the absence of a brief, reliable, and valid measure of the construct. Until recently, the primary focus in the literature has been on individual readiness for change, not on organizational readiness for change [2]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Organizational readiness for change in healthcare settings is an important factor in successful implementation of new policies, programs, and practices. However, research on the topic is hindered by the absence of a brief, reliable, and valid measure. Until such a measure is developed, we cannot advance scientific knowledge about readiness or provide evidence-based guidance to organizational leaders about how to increase readiness. This article presents results of a psychometric assessment of a new measure called Organizational Readiness for Implementing Change (ORIC), which we developed based on Weiner's theory of organizational readiness for change. We conducted four studies to assess the psychometric properties of ORIC. In study one, we assessed the content adequacy of the new measure using quantitative methods. In study two, we examined the measure's factor structure and reliability in a laboratory simulation. In study three, we assessed the reliability and validity of an organization-level measure of readiness based on aggregated individual-level data from study two. In study four, we conducted a small field study utilizing the same analytic methods as in study three. Content adequacy assessment indicated that the items developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished the facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field studies revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field studies showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis. This article provides evidence in support of the ORIC measure. We believe this measure will enable testing of theories about determinants and consequences of organizational readiness and, ultimately, assist healthcare leaders to reduce the number of health organization change efforts that do not achieve desired benefits. Although ORIC shows promise, further assessment is needed to test for convergent, discriminant, and predictive validity.
    Full-text · Article · Jan 2014 · Implementation Science
  • Source
    • "It would be helpful to have a framework for reporting these studies, based on the SQUIRE guidelines [53], adapted to cover the reporting of the patient identification processes, the numbers of patients eligible for specific DESIs (initial denominator), the inevitable attrition in numbers along the delivery pathway, the delivery mechanism, the evaluation of use by the patient and the impact on decision outcomes (process and quality). In addition, approaches not previously used in this field should be considered as a means to investigate and measure the challenges of implementing new delivery-systems [54]. For instance, methods such as cognitive task analysis, ethnography and action research, tools to assess the “adaptive reserve’” of teams [55] or their “readiness for change” [56], are approaches that would pay more attention to the role of the participants in shaping and using the technologies [57], and how they fit into the demands of other technologies, such as the electronic medical record and demands for performance metrics. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Two decades of research has established the positive effect of using patient-targeted decision support interventions: patients gain knowledge, greater understanding of probabilities and increased confidence in decisions. Yet, despite their efficacy, the effectiveness of these decision support interventions in routine practice has yet to be established; widespread adoption has not occurred. The aim of this review was to search for and analyze the findings of published peer-reviewed studies that investigated the success levels of strategies or methods where attempts were made to implement patient-targeted decision support interventions into routine clinical settings. An electronic search strategy was devised and adapted for the following databases: ASSIA, CINAHL, Embase, HMIC, Medline, Medline-in-process, OpenSIGLE, PsycINFO, Scopus, Social Services Abstracts, and the Web of Science. In addition, we used snowballing techniques. Studies were included after dual independent assessment. After assessment, 5322 abstracts yielded 51 articles for consideration. After examining full-texts, 17 studies were included and subjected to data extraction. The approach used in all studies was one where clinicians and their staff used a referral model, asking eligible patients to use decision support. The results point to significant challenges to the implementation of patient decision support using this model, including indifference on the part of health care professionals. This indifference stemmed from a reported lack of confidence in the content of decision support interventions and concern about disruption to established workflows, ultimately contributing to organizational inertia regarding their adoption. It seems too early to make firm recommendations about how best to implement patient decision support into routine practice because approaches that use a 'referral model' consistently report difficulties. We sense that the underlying issues that militate against the use of patient decision support and, more generally, limit the adoption of shared decision making, are under-investigated and under-specified. Future reports from implementation studies could be improved by following guidelines, for example the SQUIRE proposals, and by adopting methods that would be able to go beyond the 'barriers' and 'facilitators' approach to understand more about the nature of professional and organizational resistance to these tools. The lack of incentives that reward the use of these interventions needs to be considered as a significant impediment.
    Full-text · Article · Nov 2013 · BMC Medical Informatics and Decision Making
Show more