Article

Statistical considerations in a systematic review of proxy measures of clinical behaviour.

Institute of Health and Society, Newcastle University, 21 Claremont Place, Newcastle upon Tyne, NE2 4AA, UK.
Implementation Science (Impact Factor: 2.37). 02/2010; 5:20. DOI: 10.1186/1748-5908-5-20
Source: PubMed

ABSTRACT Studies included in a related systematic review used a variety of statistical methods to summarise clinical behaviour and to compare proxy (or indirect) and direct (observed) methods of measuring it. The objective of the present review was to assess the validity of these statistical methods and make appropriate recommendations.
Electronic bibliographic databases were searched to identify studies meeting specified inclusion criteria. Potentially relevant studies were screened for inclusion independently by two reviewers. This was followed by systematic abstraction and categorization of statistical methods, as well as critical assessment of these methods.
Fifteen reports (of 11 studies) met the inclusion criteria. Thirteen analysed individual clinical actions separately and presented a variety of summary statistics: sensitivity was available in eight reports and specificity in six, but four reports treated different actions interchangeably. Seven reports combined several actions into summary measures of behaviour: five reports compared means on direct and proxy measures using analysis of variance or t-tests; four reported the Pearson correlation; none compared direct and proxy measures over the range of their values. Four reports comparing individual items used appropriate statistical methods, but reports that compared summary scores did not.
We recommend sensitivity and positive predictive value as statistics to assess agreement of direct and proxy measures of individual clinical actions. Summary measures should be reliable, repeatable, capture a single underlying aspect of behaviour, and map that construct onto a valid measurement scale. The relationship between the direct and proxy measures should be evaluated over the entire range of the direct measure and describe not only the mean of the proxy measure for any specific value of the direct measure, but also the range of variability of the proxy measure. The evidence about the relationship between direct and proxy methods of assessing clinical behaviour is weak.

0 Bookmarks
 · 
172 Views
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Background This paper aims to assess the sensitivity and specificity of exit interviews as a measure of malaria case management practice as compared to direct observation.Methods The malaria case management of 1654 febrile patients attending 110 health facilities from across Papua New Guinea was directly observed by a trained research officer as part of a repeat cross sectional survey. Patient recall of 5 forms of clinical advice and 5 forms of clinical action were then assessed at service exit and statistical analyses on matched observation/exit interview data conducted.ResultsThe sensitivity of exit interviews with respect to clinical advice ranged from 36.2% to 96.4% and specificity from 53.5% to 98.6%. With respect to clinical actions, sensitivity of the exit interviews ranged from 83.9% to 98.3% and specificity from 70.6% to 98.1%.Conclusion The exit interview appears to be a valid measure of objective malaria case management practices such as the completion of a diagnostic test or the provision of antimalarial medication, but may be a less valid measure of low frequency, subjective practices such as the provision of malaria prevention advice.
    BMC Health Services Research 12/2014; 14(1):628. · 1.77 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: It is difficult to foster research utilization among allied health professionals (AHPs). Tailored, multifaceted knowledge translation (KT) strategies are now recommended but are resource intensive to implement. Employers need effective KT solutions but little is known about; the impact and viability of multifaceted KT strategies using an online KT tool, their effectiveness with AHPs and their effect on evidence-based practice (EBP) decision-making behavior. The study aim was to measure the effectiveness of a multifaceted KT intervention including a customized KT tool, to change EBP behavior, knowledge, and attitudes of AHPs. This is an evaluator-blinded, cluster randomized controlled trial conducted in an Australian community-based cerebral palsy service. 135 AHPs (physiotherapists, occupational therapists, speech pathologists, psychologists and social workers) from four regions were cluster randomized (n = 4), to either the KT intervention group (n = 73 AHPs) or the control group (n = 62 AHPs), using computer-generated random numbers, concealed in opaque envelopes, by an independent officer. The KT intervention included three-day skills training workshop and multifaceted workplace supports to redress barriers (paid EBP time, mentoring, system changes and access to an online research synthesis tool). Primary outcome (self- and peer-rated EBP behavior) was measured using the Goal Attainment Scale (individual level). Secondary outcomes (knowledge and attitudes) were measured using exams and the Evidence Based Practice Attitude Scale. The intervention group's primary outcome scores improved relative to the control group, however when clustering was taken into account, the findings were non-significant: self-rated EBP behavior [effect size 4.97 (95% CI -10.47, 20.41)(p = 0.52)]; peer-rated EBP behavior [effect size 5.86 (95% CI -17.77, 29.50)(p = 0.62)]. Statistically significant improvements in EBP knowledge were detected [effect size 2.97 (95% CI 1.97, 3.97(p < 0.0001)]. Change in EBP attitudes was not statistically significant. Improvement in EBP behavior was not statistically significant after adjusting for cluster effect, however similar improvements from peer-ratings suggest behaviorally meaningful gains. The large variability in behavior observed between clusters suggests barrier assessments and subsequent KT interventions may need to target subgroups within an organization.Trial registration: Registered on the Australian New Zealand Clinical Trials Registry (ACTRN12611000529943).
    Implementation Science 11/2013; 8(1):132. · 2.37 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Objective To advance research on depression communication and treatment by comparing assessments of communication about depression from patient report, clinician report, and chart review to assessments from transcripts.DataOne hundred sixty-four primary care visits from seven health care systems (2010–2011).Study DesignPresence or absence of discussion about depressive symptoms, treatment recommendations, and follow-up was measured using patient and clinician postvisit questionnaires, chart review, and coding of audio transcripts. Sensitivity and specificity of indirect measures compared to transcripts were calculated.Principal FindingsPatient report was sensitive for mood (83 percent) and sleep (83 percent) but not suicide (55 percent). Patient report was specific for suicide (86 percent) but not for other symptoms (44–75 percent). Clinician report was sensitive for all symptoms (83–98 percent) and specific for sleep, memory, and suicide (80–87 percent), but not for other symptoms (45–48 percent). Chart review was not sensitive for symptoms (50–73 percent), but it was specific for sleep, memory, and suicide (88–96 percent). All indirect measures had low sensitivity for treatment recommendations (patient report: 24–42 percent, clinician report 38–50 percent, chart review 49–67 percent) but high specificity (89–96 percent). For definite follow-up plans, all three indirect measures were sensitive (82–96 percent) but not specific (40–57 percent).Conclusions Clinician report and chart review generally had the most favorable sensitivity and specificity for measuring discussion of depressive symptoms and treatment recommendations, respectively.
    Health Services Research 05/2014; · 2.29 Impact Factor

Full-text (3 Sources)

Download
57 Downloads
Available from
May 22, 2014