Article

Statistical considerations in a systematic review of proxy measures of clinical behaviour.

Institute of Health and Society, Newcastle University, 21 Claremont Place, Newcastle upon Tyne, NE2 4AA, UK.
Implementation Science (Impact Factor: 2.37). 02/2010; 5:20. DOI: 10.1186/1748-5908-5-20
Source: PubMed

ABSTRACT Studies included in a related systematic review used a variety of statistical methods to summarise clinical behaviour and to compare proxy (or indirect) and direct (observed) methods of measuring it. The objective of the present review was to assess the validity of these statistical methods and make appropriate recommendations.
Electronic bibliographic databases were searched to identify studies meeting specified inclusion criteria. Potentially relevant studies were screened for inclusion independently by two reviewers. This was followed by systematic abstraction and categorization of statistical methods, as well as critical assessment of these methods.
Fifteen reports (of 11 studies) met the inclusion criteria. Thirteen analysed individual clinical actions separately and presented a variety of summary statistics: sensitivity was available in eight reports and specificity in six, but four reports treated different actions interchangeably. Seven reports combined several actions into summary measures of behaviour: five reports compared means on direct and proxy measures using analysis of variance or t-tests; four reported the Pearson correlation; none compared direct and proxy measures over the range of their values. Four reports comparing individual items used appropriate statistical methods, but reports that compared summary scores did not.
We recommend sensitivity and positive predictive value as statistics to assess agreement of direct and proxy measures of individual clinical actions. Summary measures should be reliable, repeatable, capture a single underlying aspect of behaviour, and map that construct onto a valid measurement scale. The relationship between the direct and proxy measures should be evaluated over the entire range of the direct measure and describe not only the mean of the proxy measure for any specific value of the direct measure, but also the range of variability of the proxy measure. The evidence about the relationship between direct and proxy methods of assessing clinical behaviour is weak.

0 Bookmarks
 · 
156 Views
  • [Show abstract] [Hide abstract]
    ABSTRACT: Objective To advance research on depression communication and treatment by comparing assessments of communication about depression from patient report, clinician report, and chart review to assessments from transcripts.DataOne hundred sixty-four primary care visits from seven health care systems (2010–2011).Study DesignPresence or absence of discussion about depressive symptoms, treatment recommendations, and follow-up was measured using patient and clinician postvisit questionnaires, chart review, and coding of audio transcripts. Sensitivity and specificity of indirect measures compared to transcripts were calculated.Principal FindingsPatient report was sensitive for mood (83 percent) and sleep (83 percent) but not suicide (55 percent). Patient report was specific for suicide (86 percent) but not for other symptoms (44–75 percent). Clinician report was sensitive for all symptoms (83–98 percent) and specific for sleep, memory, and suicide (80–87 percent), but not for other symptoms (45–48 percent). Chart review was not sensitive for symptoms (50–73 percent), but it was specific for sleep, memory, and suicide (88–96 percent). All indirect measures had low sensitivity for treatment recommendations (patient report: 24–42 percent, clinician report 38–50 percent, chart review 49–67 percent) but high specificity (89–96 percent). For definite follow-up plans, all three indirect measures were sensitive (82–96 percent) but not specific (40–57 percent).Conclusions Clinician report and chart review generally had the most favorable sensitivity and specificity for measuring discussion of depressive symptoms and treatment recommendations, respectively.
    Health Services Research 05/2014; · 2.29 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: The Centers for Disease Control and Prevention (CDC), Division for Heart Disease and Stroke Prevention (DHDSP), commissioned an Institute of Medicine (IOM) report to identify the highest priority action areas for CDC, state health departments, and other public health partners in their efforts to reduce and control hypertension. To assess the dissemination and adoption of the IOM report recommendations, DHDSP developed an evaluation based on the Interactive Systems Framework for Dissemination and Implementation (ISF). The evaluation incorporates data collection at critical points across 3 years. In this article, we focus on the ISF systems to describe the role of funded state partners and their relationship with CDC in implementing public health recommendations. We describe baseline results for three data collection activities: (1) key informant interviews, (2) a Web-based survey, and (3) content analysis of state workplans to determine the degree of alignment with IOM recommendations. For example, currently 30 % of surveyed programs are implementing most (or all) of the recommendations in the IOM report, however 76 % intend to change hypertension program priorities based on the recommendations of the IOM report. Qualitative data suggest that there are several facilitators and barriers in implementing public health policy recommendations. DHDSP will use these baseline results to provide additional technical assistance and support to state health departments in their efforts to implement the IOM report's recommendations.
    American Journal of Community Psychology 04/2012; · 1.74 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: It is difficult to foster research utilization among allied health professionals (AHPs). Tailored, multifaceted knowledge translation (KT) strategies are now recommended but are resource intensive to implement. Employers need effective KT solutions but little is known about; the impact and viability of multifaceted KT strategies using an online KT tool, their effectiveness with AHPs and their effect on evidence-based practice (EBP) decision-making behavior. The study aim was to measure the effectiveness of a multifaceted KT intervention including a customized KT tool, to change EBP behavior, knowledge, and attitudes of AHPs. This is an evaluator-blinded, cluster randomized controlled trial conducted in an Australian community-based cerebral palsy service. 135 AHPs (physiotherapists, occupational therapists, speech pathologists, psychologists and social workers) from four regions were cluster randomized (n = 4), to either the KT intervention group (n = 73 AHPs) or the control group (n = 62 AHPs), using computer-generated random numbers, concealed in opaque envelopes, by an independent officer. The KT intervention included three-day skills training workshop and multifaceted workplace supports to redress barriers (paid EBP time, mentoring, system changes and access to an online research synthesis tool). Primary outcome (self- and peer-rated EBP behavior) was measured using the Goal Attainment Scale (individual level). Secondary outcomes (knowledge and attitudes) were measured using exams and the Evidence Based Practice Attitude Scale. The intervention group's primary outcome scores improved relative to the control group, however when clustering was taken into account, the findings were non-significant: self-rated EBP behavior [effect size 4.97 (95% CI -10.47, 20.41)(p = 0.52)]; peer-rated EBP behavior [effect size 5.86 (95% CI -17.77, 29.50)(p = 0.62)]. Statistically significant improvements in EBP knowledge were detected [effect size 2.97 (95% CI 1.97, 3.97(p < 0.0001)]. Change in EBP attitudes was not statistically significant. Improvement in EBP behavior was not statistically significant after adjusting for cluster effect, however similar improvements from peer-ratings suggest behaviorally meaningful gains. The large variability in behavior observed between clusters suggests barrier assessments and subsequent KT interventions may need to target subgroups within an organization.Trial registration: Registered on the Australian New Zealand Clinical Trials Registry (ACTRN12611000529943).
    Implementation Science 11/2013; 8(1):132. · 2.37 Impact Factor

Full-text (3 Sources)

View
49 Downloads
Available from
May 22, 2014