No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Considering supervision measures: a commentary on Li et al.’s (2024) scoping review
Background
Valid and reliable measurement of implementation strategies is essential to advancing implementation science; however, this area lags behind the measurement of implementation outcomes and determinants. Clinical supervision is a promising and highly feasible implementation strategy in behavioral healthcare for which pragmatic measures are lacking. This research aimed to develop and psychometrically evaluate a pragmatic measure of clinical supervision conceptualized in terms of two broadly applicable, discrete clinical supervision techniques shown to improve providers’ implementation of evidence-based psychosocial interventions—(1) audit and feedback and (2) active learning.
Methods
Items were generated based on a systematic review of the literature and administered to a sample of 154 outpatient mental health clinicians serving youth and 181 community-based mental health providers serving adults. Scores were evaluated for evidence of reliability, structural validity, construct-related validity, and measurement invariance across the two samples.
Results
In sample 1, confirmatory factor analysis (CFA) supported the hypothesized two-factor structure of scores on the Evidence-Based Clinical Supervision Strategies (EBCSS) scale (χ²=5.89, df=4, p=0.208; RMSEA=0.055, CFI=0.988, SRMR=0.033). In sample 2, CFA replicated the EBCSS factor structure and provided discriminant validity evidence relative to an established supervisory alliance measure (χ²=36.12, df=30, p=0.204; RMSEA=0.034; CFI=0.990; SRMR=0.031). Construct-related validity evidence was provided by theoretically concordant associations between EBCSS subscale scores and agency climate for evidence-based practice implementation in sample 1 (d= .47 and .55) as well as measures of the supervision process in sample 2. Multiple group CFA supported the configural, metric, and partial scalar invariance of scores on the EBCSS across the two samples.
Conclusions
Scores on the EBCSS provide a valid basis for inferences regarding the extent to which behavioral health providers experience audit and feedback and active learning as part of their clinical supervision in both clinic- and community-based behavioral health settings.
Trial registration
ClinicalTrials.gov NCT04096274. Registered on 19 September 2019.
To rigorously measure the implementation of evidence-based interventions, implementation science requires measures that have evidence of reliability and validity across different contexts and populations. Measures that can detect change over time and impact on outcomes of interest are most useful to implementers. Moreover, measures that fit the practical needs of implementers could be used to guide implementation outside of the research context. To address this need, our team developed a rating scale for implementation science measures that considers their psychometric and pragmatic properties and the evidence available. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) can be used in systematic reviews of measures, in measure development, and to select measures. PAPERS may move the field toward measures that inform robust research evaluations and practical implementation efforts.
What do clinical supervision research reviews across the last 25 years tell us? That question is subsequently examined. Based on database and literature searches, 20 reviews appearing from 1995 through 2019 were identified for survey examination; consistencies, inconsistencies and other defining features were determined across reviews; and the survey findings and their implications are considered. Primary findings are as follows: (a) ‘proof’ for supervision appears to be more ‘proof by association’ than otherwise, being primarily a product of ex post facto, cross‐sectional, correlational study; (b) evidence supporting supervision impact of any type is weak at best, especially so for worker and client outcomes; (c) supervision models generally lack empirical foundation; (d) evidence‐based supervision appears to be more a hope and dream than supervision‐based reality at present; and (e) the primary methodological problems that plagued supervision research in the 1990s are still all too frequent problems that plague supervision research now. Some questions to entertain about supervision going forward, and some remedies for improving its research, are proposed.
Supervision of staff has a rich history within the social work profession, and is widely valued internationally for the support, knowledge, and skill it is perceived to provide. Moreover, quality supervision is championed for ensuring good client care. However, supervisors practicing within the social work profession have not typically had access to best practice information, nor accessed the parallel research related to the supervision of staff. A supervision primer provides an orientation to the supervision literature, including definitions and disciplinary perspectives. This sets the stage for a scoping review of social work supervision articles for the purpose of supporting supervisors in keeping current, locating, understanding, and applying literature to their practice. Seventy-nine conceptual and empirical articles first-authored by social workers were published in 36 journals between 2013 and 2017. The volume, location, context and nature of these articles are presented, pointing to an increased social work presence in the literature. Implications for supervisory practice, scholarship, and research are discussed.
Scoping reviews, a type of knowledge synthesis, follow a systematic approach to map evidence on a topic and identify main concepts, theories, sources, and knowledge gaps. Although more scoping reviews are being done, their methodological and reporting quality need improvement. This document presents the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) checklist and explanation. The checklist was developed by a 24-member expert panel and 2 research leads following published guidance from the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network. The final checklist contains 20 essential reporting items and 2 optional items. The authors provide a rationale and an example of good reporting for each item. The intent of the PRISMA-ScR is to help readers (including researchers, publishers, commissioners, policymakers, health care providers, guideline developers, and patients or consumers) develop a greater understanding of relevant terminology, core concepts, and key items to report for scoping reviews.
Background
Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria’s clarity and importance.
Methods
Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data.
Findings
The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented.
Conclusions
This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.
Two studies were conducted to revise and empirically test Ellis's framework for inadequate and harmful supervision, and to determine the occurrence of inadequate and harmful clinical supervision from the supervisees' perspective. For Study 1, we delineated 10 criteria for minimally adequate clinical supervision and defined inadequate and harmful supervision by differentiating self-identified and de facto supervision for each. Ratings from 34 supervision experts were used to generate a taxonomy of 16 de facto inadequate and 21 de facto harmful supervision descriptors. Because harmful supervision was distinct from, yet subsumed by, inadequate supervision, we revised the taxonomy and definitions accordingly. In Study 2, the occurrence of inadequate and harmful supervision was assessed for 363 supervisees; 93.0% were currently receiving inadequate supervision and 35.3% were currently receiving harmful supervision. Over half of the supervisees had received harmful clinical supervision at some point. Implications for research, training, and practice are discussed.
Although modern mental health systems increasingly require evidence-based practice (EBP), clinicians continue to object to
restrictions on their professional independence, which they may perceive as counterproductive, divisive and even demeaning.
In the face of such deep-seated objections, do we really need to strive for an evidence-based approach to supervision? We
believe so, and begin our argument by noting a colourful illustration of the problems that can exist in the absence of EBP,
those of poor training, therapist drift and weak professional regulation. Next, we present EBP as a mechanism to reduce such
unacceptable variability in the way that treatments are delivered. We then apply the lessons from this divisive situation
to the practice of clinical supervision, outlining a rationale for an inclusive, synergistic way forward. A number of fundamental
psychological motives for this integration are noted, including supervisor training that is informed by practice-based research,
corrective feedback on supervision, and a supportive professional system. Illustrative examples are provided in each of these
areas. It is concluded that the traditional gulf between research and practice can and should be bridged in relation to supervision.
KeywordsEvidence-based practice (EBP)–Clinical supervision
The aim of the present paper is to present the development of the second version of the Copenhagen Psychosocial Questionnaire (COPSOQ II).
The development of COPSOQ II took place in five main steps: (1) We considered practical experience from the use of COPSOQ I, in particular feedback from workplace studies where the questionnaire had been used; (2) All scales concerning workplace factors in COPSOQ I were analyzed for differential item functioning (DIF) with regard to gender, age and occupational status; (3) A test version of COPSOQ II including new scales and items was developed and tested in a representative sample of working Danes between 20 and 59 years of age. In all, 3,517 Danish employees participated in the study. The overall response rate was 60.4%; (4) Based on psychometric analyses, the final questionnaire was developed; and (5) Criteria-related validity of the new scales was tested.
The development of COPSOQ II resulted in a questionnaire with 41 scales and 127 items. New scales on values at the workplace were introduced including scales on Trust, Justice and Social inclusiveness. Scales on Variation, Work pace, Recognition, Work-family conflicts and items on offensive behaviour were also added. New scales regarding health symptoms included: Burnout, Stress, Sleeping troubles and Depressive symptoms. In general, the new scales showed good criteria validity. All in all, 57% of the items of COPSOQ I were retained in COPSOQ II.
The COPSOQ I concept has been further developed and new validated scales have been included.
Objectives:
Clinical supervision is essential for ensuring effective service delivery. International imperatives to demonstrate professional competence has increased attention on the role of supervision in enhancing client outcomes. Although supervisor competency tools are recognised as important components in effective supervision, there remains a shortage of tools that are evidenced-based, applicable across workforces and freely accessible.
Design:
An expert multidisciplinary group developed the Generic Supervision Assessment Tool (GSAT) to assess supervisor competencies across a range of professions. Initially the GSAT consisted of 32 items responded to by either a supervisor (GSAT-SR) or supervisee (GSAT-SE). The current study, using surveys, employed a cross-sectional design to test the reliability and construct validity of the GSAT.
Methods:
The study consisted of two phases and included 12 professional groups across Australasia. In 2018, exploratory factor analysis (EFA) was undertaken with survey data from 479 supervisors and 447 supervisees. In 2019 survey data from 182 supervisors and 186 supervisees were used to conduct confirmatory factor analysis (CFA). The results were used to refine and validate the GSAT.
Results:
The final GSAT-SR has four factors with 26 competency items. The final GSAT-SE has two factors with 21 competency items. The EFA and CFA confirmed that the GSAT-SR and the GSAT-SE are psychometrically valid tools that supervisors and supervisees can utilise to assess competencies.
Conclusion:
As a non-discipline specific supervision tool, the GSAT is a validated, freely available tool for benchmarking the competencies of clinical supervisors across professions, potentially optimising supervisory evaluation processes and strengthening supervision effectiveness.
Practitioner points:
Supervisor competency tools are recognised as important components of safe and effective supervision provision yet there is a dearth of valid, reliable and effective measures. The Generic Supervision Assessment Tool (GSAT-SR and GSAT-SE) are unique psychometrically valid, and reliable measures of supervisor competence. The GSAT-SR and the GSAT-SE can enhance translation of evidence-based supervision competency skills into regular practice. Validated with a broad cross section of professionals in diverse practice settings the GSAT provides a comprehensive conceptualization of supervisor competence.
The implementation of recovery-oriented and evidence-based practices has become a major challenge for mental health systems and front-line practitioners. This study developed an instrument that would assess the benefits or results that accrue from supervision, including client-centered supervision. The Perceptions of Supervisory Support Scale was administered to 262 case managers. Analyses (including factor analyses and repeated measures analysis of variance) confirmed content validity and reliability. Sub-scales included: (1) emotional support; (2) support for client goal achievement; and (3) professional development support. The scale could guide support for front-line practitioners in delivering client-centered care and could be useful for future research.
Previously published accounts of the evaluation of the effects of clinical supervision, a structured system to support health service staff, have been mainly contained to small scale qualitative studies. Over the past decade, the 36-item Manchester Clinical Supervision Scale (MCSS) has transformed the evaluation landscape and has been used as a quantitative outcome measure in upward of 90 licensed studies in 12 countries worldwide. The factor structure has been replicated by other researchers and the psychometric properties have been found robust. However, it had not been previously tested empirically using newly available and sophisticated statistical analyses.
This study tested the original factor structure and response format of the MCSS for goodness of fit to the Rasch model, using Rasch Unidimensional Measurement Model (RUMM) 2030 software, and investigated the validity of the questionnaire for both nursing and allied health (AH) staff.
A series of Rasch analyses were conducted on the seven subscales of the MCSS. The default procedure for RUMM software uses the partial credit model, which allows items to have varying numbers of response categories and does not assume the distance between response thresholds is uniform.
Detailed Rasch analyses indicated that the 36-item version of the MCSS could be reduced to 26 items and result in improved fit statistics for six subscales rather than seven.
This study reconfirmed the established psychometric properties of the MCSS, now renamed the MCSS-26.
State of the art in the assessment, measurement, and evaluation of clinical supervision
- M V Ellis
- N Iuso
- N Ladany
- Ellis M. V.