Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour

Institute of Health and Society, Newcastle University, 21 Claremont Place, Newcastle upon Tyne, NE2 4AA, UK.
Implementation Science (Impact Factor: 4.12). 02/2010; 5(1):20. DOI: 10.1186/1748-5908-5-20
Source: PubMed


Studies included in a related systematic review used a variety of statistical methods to summarise clinical behaviour and to compare proxy (or indirect) and direct (observed) methods of measuring it. The objective of the present review was to assess the validity of these statistical methods and make appropriate recommendations.
Electronic bibliographic databases were searched to identify studies meeting specified inclusion criteria. Potentially relevant studies were screened for inclusion independently by two reviewers. This was followed by systematic abstraction and categorization of statistical methods, as well as critical assessment of these methods.
Fifteen reports (of 11 studies) met the inclusion criteria. Thirteen analysed individual clinical actions separately and presented a variety of summary statistics: sensitivity was available in eight reports and specificity in six, but four reports treated different actions interchangeably. Seven reports combined several actions into summary measures of behaviour: five reports compared means on direct and proxy measures using analysis of variance or t-tests; four reported the Pearson correlation; none compared direct and proxy measures over the range of their values. Four reports comparing individual items used appropriate statistical methods, but reports that compared summary scores did not.
We recommend sensitivity and positive predictive value as statistics to assess agreement of direct and proxy measures of individual clinical actions. Summary measures should be reliable, repeatable, capture a single underlying aspect of behaviour, and map that construct onto a valid measurement scale. The relationship between the direct and proxy measures should be evaluated over the entire range of the direct measure and describe not only the mean of the proxy measure for any specific value of the direct measure, but also the range of variability of the proxy measure. The evidence about the relationship between direct and proxy methods of assessing clinical behaviour is weak.

Download full-text


Available from: Susan Hrisos
    • "Data were self-reported, and may not fully reflect actual behaviour. Measuring clinical practice behaviour is highly complex and evidence for the accuracy of clinician self-reported behaviour is inconclusive[60]. The survey was limited to publicly employed physiotherapists. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Evidence-based practice is becoming increasingly important in primary care physiotherapy. Clinical practice needs to reflect current best evidence and be concordant with evidence-based clinical guidelines. There is limited knowledge about therapeutic interventions used in primary care physiotherapy in Sweden. The objectives were to examine preferred treatment interventions reported by publicly employed physiotherapists in primary care for three common musculoskeletal disorders (low back pain, neck pain and subacromial pain), the extent to which these interventions were supported by evidence, and associations with demographic variables. 419 physiotherapists in primary care in western Sweden were surveyed using a validated web-based questionnaire. The survey was completed by 271 respondents (65%). Median number of interventions reported was 7 (range 1-16). The most common treatment interventions across the three conditions were advice on posture (reported by 82-94%), advice to stay active (86-92%), and different types of exercise (65-92%). Most of these interventions were supported by evidence. However, interventions with insufficient evidence, such as advice on posture, TENS and aquatic exercise, were also used by 29-96%. Modalities such as laser therapy and ultrasound were sparingly used (<5%), which is in line with evidence. For neck pain, use of evidence-based interventions was associated with gender and for subacromial pain, with work experience. Advice and exercise therapy were the interventions most frequently reported across the three diagnoses, illustrating an active treatment strategy. While most reported interventions are supported by evidence, interventions with unclear or no evidence of effect were also used to a high extent. © 2015 John Wiley & Sons, Ltd.
    No preview · Article · May 2015 · Journal of Evaluation in Clinical Practice
  • Source
    • "Second, the large differences observed between clusters suggest that we potentially should have tailored the KT intervention to each cluster rather than the whole organization. Third, the evidence base regarding whether proxy behavior measures represent actual behavior is not firmly established, but with preferred rival direct measures also lacking validity and reliability [41,58]. Moreover, direct measurement was not affordable in our study given the geography involved, and indirect measurement tools were therefore used [43,59]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: It is difficult to foster research utilization among allied health professionals (AHPs). Tailored, multifaceted knowledge translation (KT) strategies are now recommended but are resource intensive to implement. Employers need effective KT solutions but little is known about; the impact and viability of multifaceted KT strategies using an online KT tool, their effectiveness with AHPs and their effect on evidence-based practice (EBP) decision-making behavior. The study aim was to measure the effectiveness of a multifaceted KT intervention including a customized KT tool, to change EBP behavior, knowledge, and attitudes of AHPs. This is an evaluator-blinded, cluster randomized controlled trial conducted in an Australian community-based cerebral palsy service. 135 AHPs (physiotherapists, occupational therapists, speech pathologists, psychologists and social workers) from four regions were cluster randomized (n = 4), to either the KT intervention group (n = 73 AHPs) or the control group (n = 62 AHPs), using computer-generated random numbers, concealed in opaque envelopes, by an independent officer. The KT intervention included three-day skills training workshop and multifaceted workplace supports to redress barriers (paid EBP time, mentoring, system changes and access to an online research synthesis tool). Primary outcome (self- and peer-rated EBP behavior) was measured using the Goal Attainment Scale (individual level). Secondary outcomes (knowledge and attitudes) were measured using exams and the Evidence Based Practice Attitude Scale. The intervention group's primary outcome scores improved relative to the control group, however when clustering was taken into account, the findings were non-significant: self-rated EBP behavior [effect size 4.97 (95% CI -10.47, 20.41)(p = 0.52)]; peer-rated EBP behavior [effect size 5.86 (95% CI -17.77, 29.50)(p = 0.62)]. Statistically significant improvements in EBP knowledge were detected [effect size 2.97 (95% CI 1.97, 3.97(p < 0.0001)]. Change in EBP attitudes was not statistically significant. Improvement in EBP behavior was not statistically significant after adjusting for cluster effect, however similar improvements from peer-ratings suggest behaviorally meaningful gains. The large variability in behavior observed between clusters suggests barrier assessments and subsequent KT interventions may need to target subgroups within an organization.Trial registration: Registered on the Australian New Zealand Clinical Trials Registry (ACTRN12611000529943).
    Full-text · Article · Nov 2013 · Implementation Science
  • Source
    • "Finally, we plan to undertake analyses comparing the outcomes (e.g., 'x-ray referral,' 'imaging referral excluding x-ray') measured via different methods of data collection (practitioner completed checklists, patient completed checklists, file audit). In addition, we will undertake statistical analyses comparing proxy (e.g., clinical decisions in response to vignettes) and direct methods of measuring clinical behaviour [86]. These analyses will be undertaken to add to our knowledge about the design of future trials, but will not be considered part of the trial effectiveness analyses, and will inform a separate publication. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Variability between clinical practice guideline recommendations and actual clinical practice exists in many areas of health care. A 2004 systematic review examining the effectiveness of guideline implementation interventions concluded there was a lack of evidence to support decisions about effective interventions to promote the uptake of guidelines. Further, the review recommended the use of theory in the development of implementation interventions. A clinical practice guideline for the management of acute low-back pain has been developed in Australia (2003). Acute low-back pain is a common condition, has a high burden, and there is some indication of an evidence-practice gap in the allied health setting. This provides an opportunity to develop and test a theory-based implementation intervention which, if effective, may provide benefits for patients with this condition. Aims This study aims to estimate the effectiveness of a theory-based intervention to increase allied health practitioners' (physiotherapists and chiropractors in Victoria, Australia) compliance with a clinical practice guideline for acute non-specific low back pain (LBP), compared with providing practitioners with a printed copy of the guideline. Specifically, our primary objectives are to establish if the intervention is effective in reducing the percentage of acute non-specific LBP patients who are either referred for or receive an x-ray, and improving mean level of disability for patients three months post-onset of acute LBP. Methods The design of the study is a cluster randomised trial. Restricted randomisation was used to randomise 210 practices (clusters) to an intervention or control group. Practitioners in the control group received a printed copy of the guideline. Practitioners in the intervention group received a theory-based intervention developed to address prospectively identified barriers to practitioner compliance with the guideline. The intervention primarily consisted of an educational symposium. Patients aged 18 years or older who visit a participating practitioner for acute non-specific LBP of less than three months duration over a two-week data collection period, three months post the intervention symposia, are eligible for inclusion. Sample size calculations are based on recruiting between 15 to 40 patients per practice. Outcome assessors will be blinded to group allocation. Trial registration Australian New Zealand Clinical Trials Registry ACTRN12609001022257 (date registered 25th November 2009)
    Full-text · Article · Nov 2010 · Implementation Science
Show more