Selecting performance measures by consensus: An appropriate extension of the Delphi method?

RAND, 1776 Main Street, P.O. Box 2138, Santa Monica, California 90407-2138, USA.
Psychiatric Services (Impact Factor: 1.99). 01/2006; 56(12):1583. DOI: 10.1176/
Source: PubMed
  • Source
    The Journal of Rehabilitation Research and Development 01/2007; 44(6). DOI:10.1682/JRRD.2006.09.0110 · 1.69 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: The purpose of study was to identify a list of performance measures for schizophrenia treatment services and to assemble a multistakeholder group to reach consensus on a core list. The study was conducted in two stages: first, a systematic review of the literature was conducted to identify a comprehensive list of measures; second, a consensus-building technique, the Delphi process, was used with participants from six groups of stakeholders: schizophrenia experts, mental health clinicians, mental health administrators, the payer (the Alberta Ministry of Health and Wellness), patients, and family members. Thirty stakeholders participated in three rounds of self-completed questionnaires. The degree of consensus achieved in the Delphi process was defined as the semi-interquartile range for each measure. Ninety-seven measures were identified in the literature review. The Delphi method reduced the list to 36 measures rated as essential. The measures address eight domains of service-level evaluation: acceptability, accessibility, appropriateness, competence, continuity, effectiveness, efficiency, and safety. Despite the diversity in backgrounds of the stakeholder groups, the Delphi technique was effective in moving participants' ratings toward consensus through successive questionnaire rounds. The resulting measures reflected the interests of all stakeholders. Several further steps are required before these measures are implemented and include working toward reliability and validity of specific measures, assessing the feasibility and cost-effectiveness of collecting the data, and finally, undertaking risk adjustment for outcome measures.
    Psychiatric services (Washington, D.C.) 04/2012; 63(6):584-91. DOI:10.1176/ · 1.99 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: 1-PAGE KEY MESSAGES: Refining and expanding the national performance measurement system to include a set of core measures for public reporting, is one explicit goal of the Norwegian government. Calculation of many such measures may require access to patient-level data from e.g. administrative files, medical records or quality registries. We performed a model-based evaluation to give advice on whether valid, reliable and evidence-based quality indicators may be developed with data from existing national quality registries. There are approximately 30 such registries in Norway, mainly covering hospital care and a range of clinical areas. Diabetes was chosen as a model for the evaluation. We designed the evaluation process in accordance with acknowledged international methods, in order to demonstrate how quality indicators can be developed and tested with a scientific approach and in a transparent manner. Process and outcome measures for diabetes, chosen in ongoing collaborative projects in which the Nordic countries participate (OECD, Nordic Council of Ministers, WHO), as well as the ones used for public reporting in Denmark and Sweden, formed the main basis for our model. The indicators for diabetes currently in use in the evaluated measurement systems seem to be based on broad professional consensus, as expressed in evidence-based international guidelines and a systematic review prepared by the Danish indicator project. Norwegian clinical guidelines also give support to the validity of these indicators. The set of measures varied somewhat between the systems, probably due to variations in data availability, but also because of differences in purpose and scope of the reporting systems. We also found that the Norwegian quality registries for adult and childhood diabetes enter pertinent information necessary for calculation of all selected measures. However, data from these registers can not be further validated with regard to precision and minimum bias, until earliest in 2009, when both registers have been in operation for one year. Partly due to legal requirements to obtain patients' permission, and voluntary cooperation from health personnel to enter data manually, we judge data quality in the registries to be suboptimal for indicator report retrieval, until the registries can supplemented and quality assured by linkage with hospital administrative files. Even a better option will be the development of a web-based information technology for interactive data entry from electronic medical records; however this technology will probably not be available in the near future. In the meantime, we propose to base further development of existing and new indicators for diabetes, on an alternative dataset extracted for the period 2003-2007 from hospital administrative files, linked and matched with laboratory data and relevant public registries. This project will also have relevance for problems which have to be addressed if other quality registries are to be evaluated for indicator report retrieval.


Available from