[Show abstract][Hide abstract] ABSTRACT: Objectives
New evidence continues to emerge and requires attention after the release of a clinical practice guideline (CPG). The objective of this article is to describe the Document Assessment and Review (DAR) strategy designed to ensue that the CPGs remain current and their quality maintained and to present the results of two iteration of its implementation.
Study Design and Setting
The DAR process involves an annual assessment of our CPGs and a review of documents that require an update search. Two questionnaires are used to conduct the annual assessment and the review. The review involves evidence search, evidence review, and review approval.
In 2011, 109 documents were assessed; 22 (20%) were archived, 1 (1%) was deferred for assessment in 2012, 24 (22%) were considered special cases and 62 (57%) needed a new systematic review of the evidence. Of those 62, 19 (31%) were categorized as urgent, 16 (26%) as high, and others as medium or low priority. In 2012, 88 total documents were assessed; 15 (17%) were archived, 32 (36%) deferred, 3 (3%) were considered special cases, and 38 (43%) were prioritized for review.
Assessment and prioritization of existing CPGs are effective ways of ensuring that resources are directed toward the upkeep of those that are relevant and of highest priority.
Journal of Clinical Epidemiology 09/2014; · 5.48 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Background
The original AGREE (Appraisal of Guidelines for REsearch and Evaluation) Instrument was published in 2003, and its revision, the AGREE II, in 2009. Together, they filled an important gap in the guideline and quality of care fields. Ten years later, the AGREE Enterprise reflects on a trajectory of projects and international collaboration that have contributed to advancing the science and quality of practice guidelines and the uptake of AGREE/AGREE II.FindingsThe AGREE Enterprise has undertaken activities to improve the tool and to develop resources to support its use. Since 2003, the uptake and adoption of AGREE by the international community has been swift and broad. A total of 33 language translations of the original AGREE Instrument and the current AGREE II are available and were initiated by the international community. A recent scan of the published literature identified over 600 articles that referenced the AGREE tools. The AGREE tools have been widely received and applied, with several organizations having incorporated the AGREE as part of their formal practice guideline programs. Since its redevelopment in 2010, the AGREE Enterprise website (www.agreetrust.org) continues to experience steady increases in visitors per month and currently has over 10,000 registered users.Conclusions
The AGREE Enterprise has contributed to the advancements of guidelines through research activities and international participation by scientific and user communities. As we enter a new decade, we look forward to ongoing collaborations and contributing to further advancements to improve quality of care and health care systems.
[Show abstract][Hide abstract] ABSTRACT: Background
Guidelines are the foundation for healthcare planning, delivery and quality improvement but are not consistently implemented. Few guidelines are accompanied by guideline implementation tools (GItools). Users have requested GItools, and developers have requested guidance on how to develop GItools. First it is necessary to characterize GItools. The purpose of this research was to generate a framework of desirable features of GItools.Methods
Items representing desirable GItool features were generated by a cross-sectional survey of the international guideline community. Items were confirmed by 31 guideline developers, implementers and researchers in a two-round Delphi survey administered on the Internet. The resulting GItool framework was applied with a sample of GItools accompanying guidelines identified in the National Guideline Clearinghouse.ResultsThe cross-sectional survey was completed by 96 respondents from Australia, Canada, the United Kingdom, the United States, The Netherlands, and various other countries. Seven of nine items were rated by the majority as desirable. A total of 31 panelists from 10 countries including Australia, Canada, Germany, New Zealand, Peru, Saudi Arabia, Spain, the United Kingdom, and the United States took part in a two-round Delphi survey. Ten items achieved consensus as desirable GItool features in round #1, and two additional items in round #2. A total of 13 GItools for Resource Planning, Implementation and Evaluation were identified among 149 guidelines on a variety of clinical topics (8.7%). Many GItools did not possess features considered desirable.Conclusions
Inclusion of higher quality GItools in guidelines is needed to support user adoption of guidelines. The GItool framework can serve as the basis for evaluating and adapting existing GItools, or developing new GItools. Further research is needed to validate the framework, develop and implement instruments by which developers can apply the framework, and specify which guidelines should be accompanied by GItools.
[Show abstract][Hide abstract] ABSTRACT: Practice pattern data demonstrate regional variation and lower than expected rates of adherence to practice guideline (PG) recommendations for the treatment of stage II/IIIA resected and stage IIIA/IIIB unresected non-small cell lung cancer (NSCLC) patients in Ontario, Canada. This study sought to understand how clinical decisions are made for the treatment of these patients and the role of PGs.
Surveys and key informant interviews were undertaken with clinicians and administrators.
Participants reported favorable ratings for PGs and the evidentiary bases underpinning them. The majority of participants agreed more patients should have received treatment and that regional variation is problematic. Participants estimated that up to 30% of patients are not good candidates for treatment and up to 20% of patients refuse treatment. The most common barrier to implementing PGs was the lack of organizational support by clinical administrative leadership. There was concern that the trial results underpinning the PG recommendations were not generalizable to the typical patients seen in clinic. The qualitative analysis yielded five themes related to physicians' decision making: the unique patient, the unique physician, the family, the clinical team, and the clinical evidence. A dynamic interplay between these factors exists.
Our study demonstrates the challenges inherent in (i) the complexity of clinical decision making; (ii) how quality of care problems are perceived and operationalized; and (iii) the clinical appropriateness and utility of PG recommendations. We argue that systematic and rigorous methodologies to help decision makers mitigate or negotiate these challenges are warranted.
[Show abstract][Hide abstract] ABSTRACT: Inter-professional collaborative care (ICC) for cancer leads to multiple system, organizational, professional, and patient benefits, but is limited by numerous challenges. Empirical research on interventions that promote or enable ICC is sparse so guidance on how to achieve ICC is lacking. Research shows that ICC for diagnosis could be improved. Diagnostic assessment programs (DAPs) appear to be a promising model for enabling ICC. The purpose of this study was to explore how DAP structure and function enable ICC, and whether that may be associated with organizational and clinical outcomes.
A case study approach will be used to explore ICC among eight DAPs that vary by type of cancer (lung, breast), academic status, and geographic region. To describe DAP function and outcomes, and gather information that will enable costing, recommendations expressed in DAP standards and clinical guidelines will be assessed through retrospective observational study. Data will be acquired from databases maintained by participating DAPs and the provincial cancer agency, and confirmed by and supplemented with review of medical records. We will conduct a pilot study to explore the feasibility of estimating the incremental cost-effectiveness ratio using person-level data from medical records and other sources. Interviews will be conducted with health professionals, staff, and referring physicians from each DAP to learn about barriers and facilitators of ICC. Qualitative methods based on a grounded approach will be used to guide sampling, data collection and analysis.
Findings may reveal opportunities for unique structures, interventions or tools that enable ICC that could be developed, implemented, and evaluated through future research. This information will serve as a formative needs assessment to identify the nature of ongoing or required improvements, which can be directly used by our decision maker collaborators, and as a framework by policy makers, cancer system managers, and DAP managers elsewhere to strategically plan for and implement diagnostic cancer services.
[Show abstract][Hide abstract] ABSTRACT: Realist reviews offer a rigorous method to analyze heterogeneous data emerging from multiple disciplines as a means to develop new concepts, understand the relationships between them, and identify the evidentiary base underpinning them. However, emerging synthesis methods such as the Realist Review are not well operationalized and may be difficult for the novice researcher to grasp. The objective of this paper is to describe the development of an analytic process to organize and synthesize data from a realist review.
Clinical practice guidelines have had an inconsistent and modest impact on clinical practice, which may in part be due to limitations in their design. This study illustrates the development of a transparent method for organizing and analyzing a complex data set informed by a Realist Review on guideline implementability to better understand the characteristics of guidelines that affect their uptake in practice (e.g., clarity, format). The data organization method consisted of 4 levels of refinement: 1) extraction and 2) organization of data; 3) creation of a conceptual map of guideline implementability; and 4) the development of a codebook of definitions.
This new method is comprised of four steps: data extraction, data organization, development of a conceptual map, and operationalization vis-a-vis a codebook. Applying this method, we extracted 1736 guideline attributes from 278 articles into a consensus-based set of categories, and collapsed them into 5 core conceptual domains for our guideline implementability map: Language, Format, Rigor of development, Feasibility, Decision-making.
This study advances analysis methods by offering a systematic approach to analyzing complex data sets where the goals are to condense, organize and identify relationships.
BMC Medical Research Methodology 09/2013; 13(1):112. · 2.21 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Integrated knowledge translation (IKT) interventions may be one solution to improving the uptake of clinical guidelines. IKT research initiatives are particularly relevant for breast cancer research and initiatives targeting the implementation of clinical guidelines and guideline implementation initiatives, where collaboration with an interdisciplinary team of practitioners, patients, caregivers, and policy makers is needed for producing optimum patient outcomes. The objective of this paper was to describe the process of developing an IKT strategy that could be used by guideline developers to improve the uptake of their new clinical practice guidelines on breast cancer screening. An interprofessional group of students as well as two faculty members met six times over three days at the KT Canada Summer Institute in 2011. The team used all of the phases of the action cycle in the Knowledge to Action Framework as an organizing framework. While the entire framework was used, the step involving assessing barriers to knowledge use was judged to be particularly relevant in anticipating implementation problems and being able to inform the specific KT interventions that would be appropriate to mitigate these challenges and to accomplish goals and outcomes. This activity also underscored the importance of group process and teamwork in IKT. We propose that an a priori assessment of barriers to knowledge use (i.e., level and corresponding barriers), along with the other phases of the Knowledge to Action Framework, is a strategic approach for KT strategy development, implementation, and evaluation planning and could be used in the future planning of KT strategies.
Journal of Cancer Education 06/2013; · 0.88 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We evaluated clinical practice guideline (cpg) recommendations from Cancer Care Ontario's Program in Evidence-Based Care (pebc) for molecularly targeted systemic treatments (tts) and subsequent funding decisions from the Ontario Ministry of Health and Long-Term Care.
We identified pebc cpgs on tt published before June 1, 2010, and extracted information regarding the key evidence cited in support of cpg recommendations and the effect size associated with each tt. Those variables were compared with mohltc funding decisions as of June 2011.
From 23 guidelines related to 17 tts, we identified 43 recommendations, among which 38 (88%) endorsed tt use. Among all the recommendations, 38 (88%) were based on published key evidence, with 82% (31 of 38) being supported by meta-analyses or phase iii trials. For the 38 recommendations endorsing tts, funding was approved in 28 (74%; odds ratio related to cpg recommendation: 29.9; p = 0.003). We were unable to demonstrate that recommendations associated with statistically significant improvements in overall survival [os: 14 of 16 (88%) vs. 8 of 14 (57%); p = 0.10] or disease- (dfs) or progression-free survival [pfs: 16 of 21 (76%) vs. 3 of 5 (60%); p = 0.59] were more likely to be funded than those with no significant difference. Moreover, we did not observe significant associations between funding approvals and absolute improvements of 3 months or more in os [6 of 6 (100%) vs. 3 of 6 (50%), p = 0.18] or pfs [6 of 8 (75%) vs. 10 of 12 (83%), p = 1.00].
For use of tts, most recommendations in pebc cpgs are based on meta-analyses or phase iii data, and funding decisions were strongly associated with those recommendations. Our data suggest a trend toward increased rates of funding for therapies with statistically significant improvements in os.
Current Oncology 02/2013; 20(1):e21-33. · 1.63 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: PURPOSEThe objectives of this article are (1) to examine the similarities and differences between comparative effectiveness research (CER) and evidence-based medicine (EBM); (2) to describe the implications of CER for systematic review methodologies in oncology; and (3) to address the transition from systematic reviews to guideline development and the implications of CER in this process.Method
An analysis of the principles and methods of CER was undertaken in light of EBM, systematic reviews, and guidelines.Results and CONCLUSION
There is considerable overlap between the principles and methods of the two paradigms. The focus on best care options in the context of routine practice is a more central tenet of the CER paradigm. Thus, its value is not that it is the first paradigm to recognize the importance of a patient-focused approach in the research community, but rather, given the attention it has garnered, the CER paradigm may be precisely the reminder and push required to: one, influence how systematic questions are framed so that a more patient-relevant perspective is achieved; two, broaden the types of study designs that are valued and to include those, such as pragmatic trials and observational studies, that are better able to answer effectiveness questions; three, accelerate the development and application of statistical methods that enable indirect comparisons of cancer care options; and four, create clinical practice guidelines that are better positioned to improve quality of care and system performance. Over time, we will see if the CER paradigm lives up to its potential.
Journal of Clinical Oncology 10/2012; · 18.04 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Guidelines are important tools that inform healthcare delivery based on best available research evidence. Guideline use is in part based on quality of the guidelines, which includes advice for implementation and has been shown to vary. Others hypothesized this is due to limited instructions in guideline development manuals. The purpose of this study was to examine manual instructions for implementation advice.
We used a directed and summative content analysis approach based on an established framework of guideline implementability. Six manuals identified by another research group were examined to enumerate implementability domains and elements.
Manuals were similar in content but lacked sufficient detail in particular domains. Most frequently this was Accomodation, which includes information that would help guideline users anticipate and/or overcome organizational and system level barriers. In more than one manual, information was also lacking for Communicability, information that would educate patients or facilitate their involvement in shared decision making, and Applicability, or clinical parameters to help clinicians tailor recommendations for individual patients.
Most manuals that direct guideline development lack complete information about incorporating implementation advice. These findings can be used by those who developed the manuals to consider expanding their content in these domains. It can also be used by guideline developers as they plan the content and implementation of their guidelines so that the two are integrated. New approaches for guideline development and implementation may need to be developed. Use of guidelines might be improved if they included implementation advice, but this must be evaluated through ongoing research.
[Show abstract][Hide abstract] ABSTRACT: The American Society of Clinical Oncology (ASCO) guidelines program employs a systematic review-based methodology to produce evidence-based guidelines. This is consistent with the stance of the Institute of Medicine on guideline development, which is that high-quality evidence syntheses form the basis for recommendation development. In the absence of high-quality evidence, recommendation development becomes more complex. One option is to provide no recommendations or withdraw a guideline topic. However, it is often the areas of greatest uncertainty in which the evidentiary base is incomplete, and thus, guidelines are needed most. To provide recommendations in such circumstances, an explicit methodology is needed to ensure that a credible process is undertaken, and rigorous, reliable advice is provided. In 2010, the ASCO Board of Directors approved development of guideline recommendations using consensus methodology. A modified Delphi approach to recommendation development, based on the best available data identified in a systematic review, was piloted with an ASCO guideline. Consensus was achieved through the rating of a series of recommendations by a large group of clinicians, including academic and community-based content and methodology experts. A prespecified threshold of agreement was determined to indicate when consensus was achieved. Consensus was defined as agreement by ≥ 75% of raters. The formal consensus methodology used by ASCO enabled development of guideline recommendations on a challenging clinical issue based on limited evidence using a rigorous, transparent, and explicit method. This methodology is proposed for development of future ASCO guidelines on topics for which limited evidence is available.
Journal of Clinical Oncology 07/2012; 30(25):3136-40. · 18.04 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We conducted a systematic review and bibliometric network analysis (SeBriNA) of rituximab for non-Hodgkin's lymphoma.
We searched three primary data sources (1997-2003) for five document types: original research, reviews, guidelines, editorials, and media reports. We conducted cumulative meta-analysis on three outcomes (mortality, tumor response, safety) and used GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) to classify evidence quality. Direct citation relationships between original research documents and other documents were analyzed and visually represented.
Of 6,798 documents, 757 met inclusion criteria. The 317 original research documents represented 209 study clusters and 8,483 evaluated patients. Of 209 study clusters, 2.9% were randomized controlled trials (RCTs) and reported data on outcomes of interest. The quality of evidence was moderate. We identified 1,571 direct citations to the 317 original research documents. The first RCT reporting relevant outcomes appeared in 2000, whereas the first guidelines appeared in 1999. Of 212 media reports, 92% cited no original research.
Of 757 rituximab documents, RCTs of comparisons and outcomes represented <3% of original research. In contrast, review articles, guidelines, editorials, and media reports each outnumbered the relevant original research. The SeBriNA review facilitated the analysis, contextualization, and interpretation of these complex relationships.
Journal of clinical epidemiology 06/2012; 65(9):996-1009. · 5.48 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: We describe a new methodology, the systematic review and bibliometric network analysis (SeBriNA), to contextualize the quality and quantity of patient-centered outcomes evidence relative to complementary documents such as reviews, practice guidelines, editorials, and media reports.
The SeBriNA is informed by systematic review and bibliometric analysis methodologies. It focuses on two key concepts: 1) quality of evidence for patient-centered outcomes using cumulative meta-analysis and the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) appraisal approach; 2) quantity of original research and its citation relationships to related documents. It includes four steps: 1) research questions and document selection; 2) data extraction and analysis; 3) document network relationships; and 4) document network visualization.
The primary output from the SeBriNA is an analysis of 1) evidence-the annual cumulative meta-analysis estimate of effect juxtaposed against quality of evidence by patient-centered outcomes (GRADE), and 2) context-the network of relationships between related documents and original research. This analysis can be represented as a single figure.
The SeBriNA may help decision makers conceptualize, interpret, and visualize the quantity, quality, and relevance of original research within a network of related documents. Applications include prospective support for clinical and policy decisions and identification of research gaps.
Journal of clinical epidemiology 06/2012; 65(9):1010-5. · 5.48 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Evidence-based guidelines are important tools and common pathways for translating evidence into clinical practice. It is most urgently needed when significant heterogeneity in practice exist. Actively engaging opinion leaders in the process of evidence-based guidelines development is important for several reasons. These include allowing the collective views of the practice communities to be represented, resolving heterogeneity in practice through discussion, and allowing credible recommendations to be formulated. Most importantly, the process itself is a tool for facilitating dissemination and implementation. Recognizing the gap between practice pattern and guideline recommendations, and devising strategies to address it represent an important step toward maximizing concordance between guideline and practice. Evidence-based recommendations serve as important reference points, against which we can measure, debate, and innovate from.
Current Colorectal Cancer Reports 06/2012; 8(2):83-89.
[Show abstract][Hide abstract] ABSTRACT: BACKGROUND: Social network analysis is an approach to study the interactions and exchange of resources among people. It can help understanding the underlying structural and behavioral complexities that influence the process of capacity building towards evidence-informed decision making. A social network analysis was conducted to understand if and how the staff of a public health department in Ontario turn to peers to get help incorporating research evidence into practice. METHODS: The staff were invited to respond to an online questionnaire inquiring about information seeking behavior, identification of colleague expertise, and friendship status. Three networks were developed based on the 170 participants. Overall shape, key indices, the most central people and brokers, and their characteristics were identified. RESULTS: The network analysis showed a low density and localized information-seeking network. Interpersonal connections were mainly clustered by organizational divisions; and people tended to limit information-seeking connections to a handful of peers in their division. However, recognition of expertise and friendship networks showed more cross-divisional connections. Members of the office of the Medical Officer of Health were located at the heart of the department, bridging across divisions. A small group of professional consultants and middle managers were the most-central staff in the network, also connecting their divisions to the center of the information-seeking network. In each division, there were some locally central staff, mainly practitioners, who connected their neighboring peers; but they were not necessarily connected to other experts or managers. CONCLUSIONS: The methods of social network analysis were useful in providing a systems approach to understand how knowledge might flow in an organization. The findings of this study can be used to identify early adopters of knowledge translation interventions, forming Communities of Practice, and potential internal knowledge brokers.
BMC Health Services Research 05/2012; 12(1):118. · 1.77 Impact Factor
[Show abstract][Hide abstract] ABSTRACT: Modifying the format and content of guidelines may facilitate their use and lead to improved quality of care. We reviewed the medical literature to identify features desired by different users and associated with guideline use to develop a framework of implementability and found that most guidelines do not contain these elements. Further research is needed to develop and evaluate implementability tools.
We are launching the Guideline Implementability Research and Application Network (GIRAnet) to enable the development and testing of implementability tools in three domains: Resource Implications, Implementation, and Evaluation. Partners include the Guidelines International Network (G-I-N) and its member guideline developers, implementers, and researchers. In phase one, international guidelines will be examined to identify and describe exemplar tools. Indication-specific and generic tools will populate a searchable repository. In phase two, qualitative analysis of cognitive interviews will be used to understand how developers can best integrate implementability tools in guidelines and how health professionals use them for interpreting and applying guidelines. In phase three, a small-scale pilot test will assess the impact of implementability tools based on quantitative analysis of chart-based behavioural outcomes and qualitative analysis of interviews with participants. The findings will be used to plan a more comprehensive future evaluation of implementability tools.
Infrastructure funding to establish GIRAnet will be leveraged with the in-kind contributions of collaborating national and international guideline developers to advance our knowledge of implementation practice and science. Needs assessment and evaluation of GIRAnet will provide a greater understanding of how to develop and sustain such knowledge-exchange networks. Ultimately, by facilitating use of guidelines, this research may lead to improved delivery and outcomes of patient care.
[Show abstract][Hide abstract] ABSTRACT: Effective implementation strategies are needed to optimize advancements in the fields of cancer diagnosis, treatment, survivorship, and end-of-life care. We conducted a review of systematic reviews to better understand the evidentiary base of implementation strategies in cancer control.
Using three databases, we conducted a search and identified English-language systematic reviews published between 2005 and 2010 that targeted consumer, professional, organizational, regulatory, or financial interventions, tested exclusively or partially in a cancer context (primary focus); generic or non-cancer-specific reviews were also considered. Data were extracted, appraised, and analyzed by members of the research team, and research ideas to advance the field were proposed.
Thirty-four systematic reviews providing 41 summaries of evidence on 19 unique interventions comprised the evidence base. AMSTAR quality ratings ranged between 2 and 10. Team members rated most of the interventions as promising and in need of further research, and 64 research ideas were identified.
While many interventions show promise of effectiveness in the cancer-control context, few reviews were able to conclude definitively in favor of or against a specific intervention. We discuss the complexity of implementation research and offer suggestions to advance the science in this area.
[Show abstract][Hide abstract] ABSTRACT: To explore the role of a four-item Global Rating Scale (GRS) that could be used in place of the Appraisal of Guidelines, Research and Evaluation II (AGREE II).
A mixed four-factor design was used (User Type, Evaluation Type, Clinical Topic, Guideline). Participants were asked to read and evaluate a guideline using both the AGREE II draft and GRS or GRS only and to complete a series of questions regarding overall guideline quality, adoption, utility, and acceptability.
One GRS item varied as a function of User Type. Each item was a significant predictor of participants' outcome measures. All items were rated as useful by stakeholders. The GRS rating scores, outcome measures, and usefulness scores did not vary between the two Evaluation Type conditions. Correlations between the GRS and the outcome measures were stronger compared with those between the AGREE II draft and these measures.
Although the GRS is less sensitive than the AGREE II in detecting differences in guideline quality as a function of User Type, its items did predict important outcome measures related to guideline adoption. The GRS may play a role in guideline evaluation, although further study is warranted.
Journal of clinical epidemiology 12/2011; 65(5):526-34. · 5.48 Impact Factor