[Show abstract][Hide abstract] ABSTRACT: Practice pattern data demonstrate regional variation and lower than expected rates of adherence to practice guideline (PG) recommendations for the treatment of stage II/IIIA resected and stage IIIA/IIIB unresected non-small cell lung cancer (NSCLC) patients in Ontario, Canada. This study sought to understand how clinical decisions are made for the treatment of these patients and the role of PGs.
Surveys and key informant interviews were undertaken with clinicians and administrators.
Participants reported favorable ratings for PGs and the evidentiary bases underpinning them. The majority of participants agreed more patients should have received treatment and that regional variation is problematic. Participants estimated that up to 30% of patients are not good candidates for treatment and up to 20% of patients refuse treatment. The most common barrier to implementing PGs was the lack of organizational support by clinical administrative leadership. There was concern that the trial results underpinning the PG recommendations were not generalizable to the typical patients seen in clinic. The qualitative analysis yielded five themes related to physicians' decision making: the unique patient, the unique physician, the family, the clinical team, and the clinical evidence. A dynamic interplay between these factors exists.
Our study demonstrates the challenges inherent in (i) the complexity of clinical decision making; (ii) how quality of care problems are perceived and operationalized; and (iii) the clinical appropriateness and utility of PG recommendations. We argue that systematic and rigorous methodologies to help decision makers mitigate or negotiate these challenges are warranted.
[Show abstract][Hide abstract] ABSTRACT: Inter-professional collaborative care (ICC) for cancer leads to multiple system, organizational, professional, and patient benefits, but is limited by numerous challenges. Empirical research on interventions that promote or enable ICC is sparse so guidance on how to achieve ICC is lacking. Research shows that ICC for diagnosis could be improved. Diagnostic assessment programs (DAPs) appear to be a promising model for enabling ICC. The purpose of this study was to explore how DAP structure and function enable ICC, and whether that may be associated with organizational and clinical outcomes.
A case study approach will be used to explore ICC among eight DAPs that vary by type of cancer (lung, breast), academic status, and geographic region. To describe DAP function and outcomes, and gather information that will enable costing, recommendations expressed in DAP standards and clinical guidelines will be assessed through retrospective observational study. Data will be acquired from databases maintained by participating DAPs and the provincial cancer agency, and confirmed by and supplemented with review of medical records. We will conduct a pilot study to explore the feasibility of estimating the incremental cost-effectiveness ratio using person-level data from medical records and other sources. Interviews will be conducted with health professionals, staff, and referring physicians from each DAP to learn about barriers and facilitators of ICC. Qualitative methods based on a grounded approach will be used to guide sampling, data collection and analysis.
Findings may reveal opportunities for unique structures, interventions or tools that enable ICC that could be developed, implemented, and evaluated through future research. This information will serve as a formative needs assessment to identify the nature of ongoing or required improvements, which can be directly used by our decision maker collaborators, and as a framework by policy makers, cancer system managers, and DAP managers elsewhere to strategically plan for and implement diagnostic cancer services.
[Show abstract][Hide abstract] ABSTRACT: Realist reviews offer a rigorous method to analyze heterogeneous data emerging from multiple disciplines as a means to develop new concepts, understand the relationships between them, and identify the evidentiary base underpinning them. However, emerging synthesis methods such as the Realist Review are not well operationalized and may be difficult for the novice researcher to grasp. The objective of this paper is to describe the development of an analytic process to organize and synthesize data from a realist review.
Clinical practice guidelines have had an inconsistent and modest impact on clinical practice, which may in part be due to limitations in their design. This study illustrates the development of a transparent method for organizing and analyzing a complex data set informed by a Realist Review on guideline implementability to better understand the characteristics of guidelines that affect their uptake in practice (e.g., clarity, format). The data organization method consisted of 4 levels of refinement: 1) extraction and 2) organization of data; 3) creation of a conceptual map of guideline implementability; and 4) the development of a codebook of definitions.
This new method is comprised of four steps: data extraction, data organization, development of a conceptual map, and operationalization vis-a-vis a codebook. Applying this method, we extracted 1736 guideline attributes from 278 articles into a consensus-based set of categories, and collapsed them into 5 core conceptual domains for our guideline implementability map: Language, Format, Rigor of development, Feasibility, Decision-making.
This study advances analysis methods by offering a systematic approach to analyzing complex data sets where the goals are to condense, organize and identify relationships.
BMC Medical Research Methodology 09/2013; 13(1):112. · 2.21 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Integrated knowledge translation (IKT) interventions may be one solution to improving the uptake of clinical guidelines. IKT research initiatives are particularly relevant for breast cancer research and initiatives targeting the implementation of clinical guidelines and guideline implementation initiatives, where collaboration with an interdisciplinary team of practitioners, patients, caregivers, and policy makers is needed for producing optimum patient outcomes. The objective of this paper was to describe the process of developing an IKT strategy that could be used by guideline developers to improve the uptake of their new clinical practice guidelines on breast cancer screening. An interprofessional group of students as well as two faculty members met six times over three days at the KT Canada Summer Institute in 2011. The team used all of the phases of the action cycle in the Knowledge to Action Framework as an organizing framework. While the entire framework was used, the step involving assessing barriers to knowledge use was judged to be particularly relevant in anticipating implementation problems and being able to inform the specific KT interventions that would be appropriate to mitigate these challenges and to accomplish goals and outcomes. This activity also underscored the importance of group process and teamwork in IKT. We propose that an a priori assessment of barriers to knowledge use (i.e., level and corresponding barriers), along with the other phases of the Knowledge to Action Framework, is a strategic approach for KT strategy development, implementation, and evaluation planning and could be used in the future planning of KT strategies.
Journal of Cancer Education 06/2013; · 0.88 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We evaluated clinical practice guideline (cpg) recommendations from Cancer Care Ontario's Program in Evidence-Based Care (pebc) for molecularly targeted systemic treatments (tts) and subsequent funding decisions from the Ontario Ministry of Health and Long-Term Care.
We identified pebc cpgs on tt published before June 1, 2010, and extracted information regarding the key evidence cited in support of cpg recommendations and the effect size associated with each tt. Those variables were compared with mohltc funding decisions as of June 2011.
From 23 guidelines related to 17 tts, we identified 43 recommendations, among which 38 (88%) endorsed tt use. Among all the recommendations, 38 (88%) were based on published key evidence, with 82% (31 of 38) being supported by meta-analyses or phase iii trials. For the 38 recommendations endorsing tts, funding was approved in 28 (74%; odds ratio related to cpg recommendation: 29.9; p = 0.003). We were unable to demonstrate that recommendations associated with statistically significant improvements in overall survival [os: 14 of 16 (88%) vs. 8 of 14 (57%); p = 0.10] or disease- (dfs) or progression-free survival [pfs: 16 of 21 (76%) vs. 3 of 5 (60%); p = 0.59] were more likely to be funded than those with no significant difference. Moreover, we did not observe significant associations between funding approvals and absolute improvements of 3 months or more in os [6 of 6 (100%) vs. 3 of 6 (50%), p = 0.18] or pfs [6 of 8 (75%) vs. 10 of 12 (83%), p = 1.00].
For use of tts, most recommendations in pebc cpgs are based on meta-analyses or phase iii data, and funding decisions were strongly associated with those recommendations. Our data suggest a trend toward increased rates of funding for therapies with statistically significant improvements in os.
Current Oncology 02/2013; 20(1):e21-33. · 1.63 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: PURPOSEThe objectives of this article are (1) to examine the similarities and differences between comparative effectiveness research (CER) and evidence-based medicine (EBM); (2) to describe the implications of CER for systematic review methodologies in oncology; and (3) to address the transition from systematic reviews to guideline development and the implications of CER in this process.Method
An analysis of the principles and methods of CER was undertaken in light of EBM, systematic reviews, and guidelines.Results and CONCLUSION
There is considerable overlap between the principles and methods of the two paradigms. The focus on best care options in the context of routine practice is a more central tenet of the CER paradigm. Thus, its value is not that it is the first paradigm to recognize the importance of a patient-focused approach in the research community, but rather, given the attention it has garnered, the CER paradigm may be precisely the reminder and push required to: one, influence how systematic questions are framed so that a more patient-relevant perspective is achieved; two, broaden the types of study designs that are valued and to include those, such as pragmatic trials and observational studies, that are better able to answer effectiveness questions; three, accelerate the development and application of statistical methods that enable indirect comparisons of cancer care options; and four, create clinical practice guidelines that are better positioned to improve quality of care and system performance. Over time, we will see if the CER paradigm lives up to its potential.
Journal of Clinical Oncology 10/2012; · 18.04 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Guidelines are important tools that inform healthcare delivery based on best available research evidence. Guideline use is in part based on quality of the guidelines, which includes advice for implementation and has been shown to vary. Others hypothesized this is due to limited instructions in guideline development manuals. The purpose of this study was to examine manual instructions for implementation advice.
We used a directed and summative content analysis approach based on an established framework of guideline implementability. Six manuals identified by another research group were examined to enumerate implementability domains and elements.
Manuals were similar in content but lacked sufficient detail in particular domains. Most frequently this was Accomodation, which includes information that would help guideline users anticipate and/or overcome organizational and system level barriers. In more than one manual, information was also lacking for Communicability, information that would educate patients or facilitate their involvement in shared decision making, and Applicability, or clinical parameters to help clinicians tailor recommendations for individual patients.
Most manuals that direct guideline development lack complete information about incorporating implementation advice. These findings can be used by those who developed the manuals to consider expanding their content in these domains. It can also be used by guideline developers as they plan the content and implementation of their guidelines so that the two are integrated. New approaches for guideline development and implementation may need to be developed. Use of guidelines might be improved if they included implementation advice, but this must be evaluated through ongoing research.
[Show abstract][Hide abstract] ABSTRACT: The American Society of Clinical Oncology (ASCO) guidelines program employs a systematic review-based methodology to produce evidence-based guidelines. This is consistent with the stance of the Institute of Medicine on guideline development, which is that high-quality evidence syntheses form the basis for recommendation development. In the absence of high-quality evidence, recommendation development becomes more complex. One option is to provide no recommendations or withdraw a guideline topic. However, it is often the areas of greatest uncertainty in which the evidentiary base is incomplete, and thus, guidelines are needed most. To provide recommendations in such circumstances, an explicit methodology is needed to ensure that a credible process is undertaken, and rigorous, reliable advice is provided. In 2010, the ASCO Board of Directors approved development of guideline recommendations using consensus methodology. A modified Delphi approach to recommendation development, based on the best available data identified in a systematic review, was piloted with an ASCO guideline. Consensus was achieved through the rating of a series of recommendations by a large group of clinicians, including academic and community-based content and methodology experts. A prespecified threshold of agreement was determined to indicate when consensus was achieved. Consensus was defined as agreement by ≥ 75% of raters. The formal consensus methodology used by ASCO enabled development of guideline recommendations on a challenging clinical issue based on limited evidence using a rigorous, transparent, and explicit method. This methodology is proposed for development of future ASCO guidelines on topics for which limited evidence is available.
Journal of Clinical Oncology 07/2012; 30(25):3136-40. · 18.04 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We conducted a systematic review and bibliometric network analysis (SeBriNA) of rituximab for non-Hodgkin's lymphoma.
We searched three primary data sources (1997-2003) for five document types: original research, reviews, guidelines, editorials, and media reports. We conducted cumulative meta-analysis on three outcomes (mortality, tumor response, safety) and used GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) to classify evidence quality. Direct citation relationships between original research documents and other documents were analyzed and visually represented.
Of 6,798 documents, 757 met inclusion criteria. The 317 original research documents represented 209 study clusters and 8,483 evaluated patients. Of 209 study clusters, 2.9% were randomized controlled trials (RCTs) and reported data on outcomes of interest. The quality of evidence was moderate. We identified 1,571 direct citations to the 317 original research documents. The first RCT reporting relevant outcomes appeared in 2000, whereas the first guidelines appeared in 1999. Of 212 media reports, 92% cited no original research.
Of 757 rituximab documents, RCTs of comparisons and outcomes represented <3% of original research. In contrast, review articles, guidelines, editorials, and media reports each outnumbered the relevant original research. The SeBriNA review facilitated the analysis, contextualization, and interpretation of these complex relationships.
Journal of clinical epidemiology 06/2012; 65(9):996-1009. · 5.33 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We describe a new methodology, the systematic review and bibliometric network analysis (SeBriNA), to contextualize the quality and quantity of patient-centered outcomes evidence relative to complementary documents such as reviews, practice guidelines, editorials, and media reports.
The SeBriNA is informed by systematic review and bibliometric analysis methodologies. It focuses on two key concepts: 1) quality of evidence for patient-centered outcomes using cumulative meta-analysis and the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) appraisal approach; 2) quantity of original research and its citation relationships to related documents. It includes four steps: 1) research questions and document selection; 2) data extraction and analysis; 3) document network relationships; and 4) document network visualization.
The primary output from the SeBriNA is an analysis of 1) evidence-the annual cumulative meta-analysis estimate of effect juxtaposed against quality of evidence by patient-centered outcomes (GRADE), and 2) context-the network of relationships between related documents and original research. This analysis can be represented as a single figure.
The SeBriNA may help decision makers conceptualize, interpret, and visualize the quantity, quality, and relevance of original research within a network of related documents. Applications include prospective support for clinical and policy decisions and identification of research gaps.
Journal of clinical epidemiology 06/2012; 65(9):1010-5. · 5.33 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Evidence-based guidelines are important tools and common pathways for translating evidence into clinical practice. It is most urgently needed when significant heterogeneity in practice exist. Actively engaging opinion leaders in the process of evidence-based guidelines development is important for several reasons. These include allowing the collective views of the practice communities to be represented, resolving heterogeneity in practice through discussion, and allowing credible recommendations to be formulated. Most importantly, the process itself is a tool for facilitating dissemination and implementation. Recognizing the gap between practice pattern and guideline recommendations, and devising strategies to address it represent an important step toward maximizing concordance between guideline and practice. Evidence-based recommendations serve as important reference points, against which we can measure, debate, and innovate from.
Current Colorectal Cancer Reports 06/2012; 8(2):83-89.
[Show abstract][Hide abstract] ABSTRACT: BACKGROUND: Social network analysis is an approach to study the interactions and exchange of resources among people. It can help understanding the underlying structural and behavioral complexities that influence the process of capacity building towards evidence-informed decision making. A social network analysis was conducted to understand if and how the staff of a public health department in Ontario turn to peers to get help incorporating research evidence into practice. METHODS: The staff were invited to respond to an online questionnaire inquiring about information seeking behavior, identification of colleague expertise, and friendship status. Three networks were developed based on the 170 participants. Overall shape, key indices, the most central people and brokers, and their characteristics were identified. RESULTS: The network analysis showed a low density and localized information-seeking network. Interpersonal connections were mainly clustered by organizational divisions; and people tended to limit information-seeking connections to a handful of peers in their division. However, recognition of expertise and friendship networks showed more cross-divisional connections. Members of the office of the Medical Officer of Health were located at the heart of the department, bridging across divisions. A small group of professional consultants and middle managers were the most-central staff in the network, also connecting their divisions to the center of the information-seeking network. In each division, there were some locally central staff, mainly practitioners, who connected their neighboring peers; but they were not necessarily connected to other experts or managers. CONCLUSIONS: The methods of social network analysis were useful in providing a systems approach to understand how knowledge might flow in an organization. The findings of this study can be used to identify early adopters of knowledge translation interventions, forming Communities of Practice, and potential internal knowledge brokers.
BMC Health Services Research 05/2012; 12(1):118. · 1.77 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Modifying the format and content of guidelines may facilitate their use and lead to improved quality of care. We reviewed the medical literature to identify features desired by different users and associated with guideline use to develop a framework of implementability and found that most guidelines do not contain these elements. Further research is needed to develop and evaluate implementability tools.
We are launching the Guideline Implementability Research and Application Network (GIRAnet) to enable the development and testing of implementability tools in three domains: Resource Implications, Implementation, and Evaluation. Partners include the Guidelines International Network (G-I-N) and its member guideline developers, implementers, and researchers. In phase one, international guidelines will be examined to identify and describe exemplar tools. Indication-specific and generic tools will populate a searchable repository. In phase two, qualitative analysis of cognitive interviews will be used to understand how developers can best integrate implementability tools in guidelines and how health professionals use them for interpreting and applying guidelines. In phase three, a small-scale pilot test will assess the impact of implementability tools based on quantitative analysis of chart-based behavioural outcomes and qualitative analysis of interviews with participants. The findings will be used to plan a more comprehensive future evaluation of implementability tools.
Infrastructure funding to establish GIRAnet will be leveraged with the in-kind contributions of collaborating national and international guideline developers to advance our knowledge of implementation practice and science. Needs assessment and evaluation of GIRAnet will provide a greater understanding of how to develop and sustain such knowledge-exchange networks. Ultimately, by facilitating use of guidelines, this research may lead to improved delivery and outcomes of patient care.
[Show abstract][Hide abstract] ABSTRACT: Effective implementation strategies are needed to optimize advancements in the fields of cancer diagnosis, treatment, survivorship, and end-of-life care. We conducted a review of systematic reviews to better understand the evidentiary base of implementation strategies in cancer control.
Using three databases, we conducted a search and identified English-language systematic reviews published between 2005 and 2010 that targeted consumer, professional, organizational, regulatory, or financial interventions, tested exclusively or partially in a cancer context (primary focus); generic or non-cancer-specific reviews were also considered. Data were extracted, appraised, and analyzed by members of the research team, and research ideas to advance the field were proposed.
Thirty-four systematic reviews providing 41 summaries of evidence on 19 unique interventions comprised the evidence base. AMSTAR quality ratings ranged between 2 and 10. Team members rated most of the interventions as promising and in need of further research, and 64 research ideas were identified.
While many interventions show promise of effectiveness in the cancer-control context, few reviews were able to conclude definitively in favor of or against a specific intervention. We discuss the complexity of implementation research and offer suggestions to advance the science in this area.
[Show abstract][Hide abstract] ABSTRACT: To explore the role of a four-item Global Rating Scale (GRS) that could be used in place of the Appraisal of Guidelines, Research and Evaluation II (AGREE II).
A mixed four-factor design was used (User Type, Evaluation Type, Clinical Topic, Guideline). Participants were asked to read and evaluate a guideline using both the AGREE II draft and GRS or GRS only and to complete a series of questions regarding overall guideline quality, adoption, utility, and acceptability.
One GRS item varied as a function of User Type. Each item was a significant predictor of participants' outcome measures. All items were rated as useful by stakeholders. The GRS rating scores, outcome measures, and usefulness scores did not vary between the two Evaluation Type conditions. Correlations between the GRS and the outcome measures were stronger compared with those between the AGREE II draft and these measures.
Although the GRS is less sensitive than the AGREE II in detecting differences in guideline quality as a function of User Type, its items did predict important outcome measures related to guideline adoption. The GRS may play a role in guideline evaluation, although further study is warranted.
Journal of clinical epidemiology 12/2011; 65(5):526-34. · 5.33 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Globally, healthcare systems are attempting to optimize quality of care. This challenge has resulted in the development of implementation science or knowledge translation (KT) and the resulting need to build capacity in both the science and practice of KT.
We are attempting to meet these challenges through the creation of a national training initiative in KT. We have identified core competencies in this field and have developed a series of educational courses and materials for three training streams. We report the outline for this approach and the progress to date.
We have prepared a strategy to develop, implement, and evaluate a national training initiative to build capacity in the science and practice of KT. Ultimately through this initiative, we hope to meet the capacity demand for KT researchers and practitioners in Canada that will lead to improved care and a strengthened healthcare system.
[Show abstract][Hide abstract] ABSTRACT: Appropriate screening may reduce the mortality and morbidity of colorectal, breast, and cervical cancers. Several high-quality systematic reviews and practice guidelines exist to inform the most effective screening options. However, effective implementation strategies are warranted if the full benefits of screening are to be realized. We developed an implementation guideline to answer the question: What interventions have been shown to increase the uptake of cancer screening by individuals, specifically for breast, cervical, and colorectal cancers?
A guideline panel was established as part of Cancer Care Ontario's Program in Evidence-based Care, and a systematic review of the published literature was conducted. It yielded three foundational systematic reviews and an existing guidance document. We conducted updates of these reviews and searched the literature published between 2004 and 2010. A draft guideline was written that went through two rounds of review. Revisions were made resulting in a final set of guideline recommendations.
Sixty-six new studies reflecting 74 comparisons met eligibility criteria. They were generally of poor to moderate quality. Using these and the foundational documents, the panel developed a draft guideline. The draft report was well received in the two rounds of review with mean quality scores above four (on a five-point scale) for each of the items. For most of the interventions considered, there was insufficient evidence to support or refute their effectiveness. However, client reminders, reduction of structural barriers, and provision of provider assessment and feedback were recommended interventions to increase screening for at least two of three cancer sites studied. The final guidelines also provide advice on how the recommendations can be used and future areas for research.
Using established guideline development methodologies and the AGREE II as our methodological frameworks, we developed an implementation guideline to advise on interventions to increase the rate of breast, cervical and colorectal cancer screening. While advancements have been made in these areas of implementation science, more investigations are warranted.
[Show abstract][Hide abstract] ABSTRACT: ABSTRACT:
Appropriate screening may reduce the mortality and morbidity of colorectal, breast, and cervical cancers. However, effective implementation strategies are warranted if the full benefits of screening are to be realized. As part of a larger agenda to create an implementation guideline, we conducted a systematic review to evaluate interventions designed to increase the rate of breast, cervical, and colorectal cancer (CRC) screening. The interventions considered were: client reminders, client incentives, mass media, small media, group education, one-on-one education, reduction in structural barriers, reduction in out-of-pocket costs, provider assessment and feedback interventions, and provider incentives. Our primary outcome, screening completion, was calculated as the overall median post-intervention absolute percentage point (PP) change in completed screening tests.
Our first step was to conduct an iterative scoping review in the research area. This yielded three relevant high-quality systematic reviews. Serving as our evidentiary foundation, we conducted a formal update. Randomized controlled trials and cluster randomized controlled trials, published between 2004 and 2010, were searched in MEDLINE, EMBASE and PSYCHinfo.
The update yielded 66 studies new eligible studies with 74 comparisons. The new studies ranged considerably in quality. Client reminders, small media, and provider audit and feedback appear to be effective interventions to increase the uptake of screening for three cancers. One-on-one education and reduction of structural barriers also appears effective, but their roles with CRC and cervical screening, respectively, are less established. More study is required to assess client incentives, mass media, group education, reduction of out-of-pocket costs, and provider incentive interventions.
The new evidence generally aligns with the evidence and conclusions from the original systematic reviews. This review served as the evidentiary foundation for an implementation guideline. Poor reporting, lack of precision and consistency in defining operational elements, and insufficient consideration of context and differences among populations are areas for additional research.
[Show abstract][Hide abstract] ABSTRACT: ABSTRACT:
Practice guidelines (PGs) are systematically developed statements intended to assist in patient and practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the AGREE II.
Participants were randomized to one of three training conditions. 'Tutorial'--participants proceeded through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. 'Tutorial + Practice Exercise'--in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, participants received feedback on how their scores compared to expert norms and formative feedback if scores fell outside the predefined range. 'AGREE II User's Manual PDF (control condition)'--participants reviewed a PDF copy of the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II.
No differences emerged between training conditions on any of the outcome measures.
We believe these results can be explained by better than anticipated performance of the AGREE II PDF materials (control condition) or the participants' level of health methodology and PG experience rather than the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new to this field; however, this requires further study.