No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Delphi Method
Abstract
This article lays out the basic structure of the Delphi method, along with common modifications, and recently developed mixed-model forecasting techniques. Delphi offers a basic structure that can be easily adapted to fit a group's needs, with respect to specific judgmental forecasting efforts.
... The Delphi method was established in the 1950s by Norman Dalkey and Olaf Helmer in an effort to gain reliable expert consensus 59 . Its main purpose is to facilitate and gather a consensus of expert opinions in the face of complex problems 60 . ...
... Synthesizing the initial set of assessment criteria Previous methodological guidance and best practice frameworks have suggested that the quality of information provided to the panelists, such as synthesis of the available evidence, may influence and possibly bias their judgement if not done objectively 60,62 . Therefore, to minimize bias and avoid an initial set of criteria that was impacted by the subjective opinions of the research team, the authors conducted a systematic literature review to identify the relevant criteria used to assess patientfacing eHealth tools in a rigorous manner that builds on previous efforts 18 . ...
Among the thousands of eHealth tools available, the vast majority do not get past pilot phases because they cannot prove value, and only a few have been systematically assessed. Although multiple eHealth assessment frameworks have been developed, these efforts face multiple challenges. This study aimed to address some of these challenges by validating and refining an initial list of 55 assessment criteria based on previous frameworks through a two-round modified Delphi process with in-between rounds of interviews. The expert panel ( n = 57) included participants from 18 countries and 9 concerned parties. A consensus was reached on 46 criteria that were classified into foundational and contextual criteria. The 36 foundational criteria focus on evaluating the eHealth tool itself and were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The 10 contextual criteria focus on evaluating the factors that vary depending on the context the tool is being evaluated for and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability. The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool, but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the strengths and weaknesses of the tool being evaluated. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.
... The Delphi methodology was followed to validate the questionnaire [32,33]. As a minimum number of experts is not defined [32,33], the questionnaire was sent to 7 experts: 2 public health physicians with a management background, 2 directors of two LHDs of the Roma 1 LHA, 2 GPs of the Roma 1 LHA who contributed to devising the DOCs and 1 GP affiliated with a different LHA. ...
... The Delphi methodology was followed to validate the questionnaire [32,33]. As a minimum number of experts is not defined [32,33], the questionnaire was sent to 7 experts: 2 public health physicians with a management background, 2 directors of two LHDs of the Roma 1 LHA, 2 GPs of the Roma 1 LHA who contributed to devising the DOCs and 1 GP affiliated with a different LHA. The directors of the two LHDs and the two GPs working in the Roma 1 LHA were chosen because they joined the LHA COVID-19 primary health care response task force, in charge to organise and plan primary care services delivery to face this extraordinary situation [34]. ...
Background
COVID-19 pandemic represented a shock for healthcare systems. Italy was one of the first country to deal with a huge number of patients to be diagnosed, isolated, and treated with scarce evidence-based guidelines and resources. Several organizational and structural changes were needed to face the pandemic at local level. The article aims at studying the perceived impact of the newly implemented District Operation Centres (DOCs) of Local Health Authority (LHA) Roma 1 in managing active surveillance and home care of COVID-19 patients and their close contacts in cooperation with general practitioners (GPs).
Methods
A questionnaire, developed according to Delphi methodology, was validated by 7 experts and administered to a randomized sample of GPs and family paediatricians (FPs). All medical doctors selected received a phone interview between December 2020 and January 2021. The questionnaire investigated general characteristics of the sample, relations with DOC and its usefulness, and potential developments. A descriptive analysis was performed and inferential statistical tests were used to assess differences.
Results
In April 2020 the LHA Roma 1 implemented one DOCs in each local health district. 215 medical doctors were interviewed, reaching the sample target for health districts (80% CL and 10% MOE) and the whole LHA (90% CL and 5% MOE). Several aspects in the management of COVID-19 cases and close contacts of COVID-19 cases, and of the support of DOCs to GPs/FPs were investigated. More than 55% of the GPs and FPs interviewed found the DOCs useful and more than 78% would recommend a service DOC-like to other LHAs. The medical professionals interviewed would use DOCs in the future as support in treating vulnerable patients, utilizing digital health tools, enlisting specialist doctors, establishing networks, and facilitating professional counselling by nurses.
Conclusions
This study is an attempt to evaluate an organizational change happened during COVID-19 pandemic. DOCs were created to support GPs and FPs as a link between primary healthcare and public health. Although several difficulties were disclosed, DOCs’ experience can help to overcome the fragmentation of the systems and the duality between primary care and public health and make the system more resilient.
... Se realizó un proceso de consenso Delphi de dos rondas. Esta es una técnica particularmente útil para recopilar opiniones representativas de juicios basados en expertos, utilizada a menudo para identificar consensos 16 . Este trabajo sigue los estándares propuestos en la orientación sobre la realización y notificación de estudios Delphi (CREDES) en CP 17 . ...
Introducción:
La insuficiencia cardíaca, enfermedad con alta prevalencia y mortalidad, presenta baja incorporación de cuidados paliativos (CP) a pesar de su necesidad. En Colombia, la falta de indicadores para monitorear la integración de CP en el tratamiento de la insuficiencia cardíaca dificulta la implementación de estrategias efectivas. Se propone utilizar un modelo de clasificación para monitorear la integración de CP en el sistema de salud colombiano.
Objetivo:
Consensuar un conjunto de indicadores evaluados por su relevancia y factibilidad para monitorear la integración de cuidados paliativos en insuficiencia cardíaca en el sistema de salud colombiano.
Método:
Se realizó un proceso de consenso Delphi en dos rondas, incluyendo expertos de las distintas regiones del país. Se utilizó una encuesta en línea sobre nueve indicadores relacionados con la integración de cuidados paliativos en insuficiencia cardíaca en el sistema de salud colombiano. Se valoró en cada indicador su relevancia y factibilidad.
Resultados:
Participaron 17 expertos de todas las regiones de Colombia, el 76% eran médicos y el 24% profesionales de enfermería; en su gran mayoría (65%) pertenecían a sociedades científicas como la Asociación Colombiana de Cuidados Paliativos. Todos los indicadores alcanzaron un puntaje > 7.0 con respecto a su relevancia y factibilidad. El indicador Voluntad anticipada en personas con insuficiencia cardíaca fue considerado el más relevante y factible, seguido de Cuidados paliativos domiciliarios para personas con insuficiencia cardíaca.
Conclusiones:
Se consensuaron ocho de los nueve indicadores propuestos para evaluar la integración de los cuidados paliativos en el tratamiento de insuficiencia cardíaca en Colombia.
... The content of the final version of the scale has been validated through the Delphi method. 20 For the content validation of H-LEFS, 12 experts with a minimum of 5 years of experience in the field were invited, who were also kept blinded with the aim of the study. A Google form (https://forms.gle/ eCSCsCwZdq7ce5DW9) was prepared containing each item of the scale and responses were recorded with the options of whether the translated item was 'valid' or 'invalid' and sent to all the experts. ...
BACKGROUND: Lower Extremity Function Scale (LEFS) serves as a patient-reported outcome measure, used to evaluate the functional status of lower extremity in different musculoskeletal conditions. It was originally developed in English language (E-LEFS) and numerous translations in different languages are available to measure its usability among different cultures. OBJECTIVES: To translate the English version of the Lower Extremity Function Scale (E-LEFS) into Hindi language and to evaluate its content validity, cross-cultural adaptation along with the concurrent validity and reliability. METHOD: Process of forward and backward translation was carried out by two bilingual translators and a physiotherapist after obtaining developer’s approval, for which the Beaton guidelines were used. For the content validation, Delphi method of was employed in which 12 field experts of more than 5 years of experience participated. Cultural adaptation procedures were then carried out with 12 patients, who reported in out-patient department of the hospital through convenience sampling who were asked about whether the H-LEFS was well understood by the patients. Their responses were recorded as positive and negative responses. This process was followed by concurrent validation which was conducted by comparing the scores with the SF-36 scale, involving 30 patients with activity limitation in their daily life. Lastly, 50 patients were recruited for intra-rater reliability testing. RESULT: The scale was successfully translated and verified by the bi-lingual language experts. Content validity came out to be significant with each item scoring more than 0.92. Nearly all items, excluding item no. 8, 16, 17, 18, and 19 received universal agreement with a content validity ratio of 1. The average scale-level validity stood at 0.98, indicating excellent level of validity and the scale was found to be well adapted by the native people as out of 12 patients, 10 patients provided 100% positive response. Concurrent validation using Pearson’s correlation coefficient came out excellent with the values of 0.993 for E-LEFS and 0.890 for SF-36. Cronbach's alpha and ICC values were calculated as 1.00 for Intra-rater reliability. CONCLUSION: LEFS was well translated into Hindi language with an excellent content and concurrent validity and is found to be highly reliable. Hence, it is recommended for the usage of native Hindi speaking population.
... Content validity was evaluated using the Delphi method, whose main virtue lies in its ability to reach consensus among experts [36,37]. Although there are no clear guidelines on the adequate number of experts, Almanasreh et al. suggest that a Delphi panel should be composed of five to 10 [38]. ...
Background
Robust tools to assess self-reported adolescent functional health literacy are lacking. In Portugal, the only available tool is the Newest Vital Sign for Portuguese adolescents (NVS-PTeen), though presenting modest validity and reliability properties. A new instrument–the Functional Literacy Questionnaire (FLiQ)–was developed, inspired by the NVS-PTeen, but following the European Regulation for food labeling and targeting a balanced assessment of numeracy and verbal comprehension skills. This study aimed to evaluate several psychometric properties of the FLiQ when administered to Portuguese adolescents.
Methods
We conducted a longitudinal observational study with three phases: (1) Delphi panel with health literacy experts; (2) self-administration of FLiQ and NVS-PTeen to adolescents in 7th to 9th grades; and (3) re-administration of FLiQ four weeks after baseline, to the same group of participants.
Results
FLiQ’s content validity was excellent, with an Average-Content Validity Index of .95. Overall, 372 adolescents (50.3% girls) aged between 12–17 years (median age: 13) participated in the study. Of these, 150 completed the test-retest assessment. Internal consistency was good (Kuder-Richardson Fornula-20 = .70), as well as test-retest reliability (Intraclass Coefficient Correlation = .82). FLiQ total score was weakly correlated with the school year (rho = .174), and moderately with Portuguese (rho = .348) and Mathematics grades (rho = .333). Factor analysis indicated a two-dimension structure, reflecting numeracy and verbal comprehension skills. Item response theory analysis revealed differences in difficulty and discrimination capacity among items, all with adequate fit values.
Conclusion
FLiQ is a valid and reliable tool. It can be used to monitor functional health literacy levels in Portuguese adolescents.
... Through the Delphi method, the opinion of a panel of experts is surveyed regarding a specific topic. It is featured by merging individual research and practical knowledge from selected experts [13,14]. A practical model framework for risk assessment is created. ...
The goal of this research is an analysis of risk factors that occur with the installation of wastewater treatment plants. Another purpose of this paper is to demonstrate the clear identification of risk factors during the construction of wastewater treatment plants. Also, this study aims to develop models for risk factors assessment as a basis for the analysis of risks associated with the installation of wastewater treatment plants. The paper will point out the identification of risk factors during the installation of wastewater treat-ment plants in a project conducted in the Republic of Serbia. The analysis identified risk factors that brought the project being conducted in the city of Krusevac. The purpose of research and identification of risk factors is to relax managing high-risk situations and to ensure an approach for eliminating the negative effects that risks create. The expected result is a developed model with defined risk factors for the estimation of their negative impact on wastewater treatment plants.
... This Delphi study was conducted and reported according to published guidance; detailed methods and results are reported separately. 19,[25][26][27][28] Stakeholder panel Stakeholders were recruited by the research team based on a matrix describing characteristics of anticipated users of an EOH decision framework and consumers of EOH recommendations. ...
... ;Grime & Wright, 2016;Landeta, Matey de Antonio, Ruiz Herrán & Villarreal Larrinaga, 2002;Lian, Liu & Chen, 2020). ...
Purpose: To identify the skills that, in the processes of digitalization and Industry 4.0, contribute to improving the professional behaviours of the employees involved in these processes.Design/methodology/approach: For this purpose, a Delphi study was carried out, collecting the opinions of 38 teaching and professional experts, through face-to-face meetings or by videoconference, and analysing their opinions using descriptive statistics and structural equation models.Findings: The results obtained show that the key skills for improving professional behaviours are teamwork, conduct, management, and problem-solving, which influence most of the behaviours studied.Originality/value: In practice, these results allow companies to be guided in terms of employee training in relation to digitalization skills and assist their selection processes. Additionally, educational institutions can review their study plans, increasing the relevance of the skills that companies value the most.
... Through the Delphi method the opinion of a panel of experts is surveyed regarding a specific topic. It is featured by merging individual research and practical knowledge from selected experts (Grime & Wright, 2016;Loo, 2002). Commonly, the Delphi method is used to forecast possible developments through discovering consensus between the respondent experts (Diamond et al., 2014;Hallowell & Gambatese, 2010;Shariff, 2015). ...
This paper proposes an approach to capture mutual learning within transdisciplinary research to develop future-oriented sustainability solutions to land use conflicts. While quality criteria for transdisciplinary research projects are increasingly discussed, there is still ample methodological potential to enable and better understand mutual learning as a key component of transdisciplinarity. Changes in the specific developments of perceptions of the participants are difficult to be tamed and tracked. To address this shortcoming we propose to apply the Delphi method for longitudinal tracking of mutual learning within transdisciplinary research.
We present the application of the methodological approach as part of a transdisciplinary project on sustainable land use in the district of Lüneburg (Lower-Saxony, Germany). Evaluation of semi-structured interviews with seven participants translated into a three-round Delphi survey. A concluding round of semi-structured interviews was performed to validate and deepen the findings. The results showcase that the Delphi method can facilitate the operationalization of the convergence of understandings as a relevant outcome of mutual learning within transdisciplinary projects.
... The main purpose of the Delphi method, which we used to complement the N-AHP method in our study, is to reveal and refine group judgments [45]. For explanations on how to use the Delphi method, see [46]. The Delphi method is a method used to assist decision-makers, which enables the most appropriate decision to emerge as a result of the consensus of experts. ...
The existence of natural gas and rare mineral reserves, energy transmission lines, and sacred places within its borders makes that geography a target for other countries, whether neighboring or not. These countries spend most of their budgets on war technologies and good defense. There are many factors to consider when choosing the location of a military base, which is vital in terms of both defense and logistic support. This study aimed to determine the criteria that should be taken into account in determining the borderline security and selecting the location of military bases of great strategic importance by getting rid of the disadvantages of classical decision-making processes. For this purpose, a solution to the problem was sought with the method obtained by combining the AHP method, one of the latest approaches in the decision-making process, with neutrosophic logic. In order to enable the experts to cope with uncertain information and to prevent errors in preference values due to differences in individual approaches, three expert opinions were obtained and the Delphi method was used to increase the advantages of the neutrosophic analytic hierarchy process (N-AHP) method by utilizing the degree of consensus. Expert opinions were received to determine, prioritize, and group the criteria using the Delphi method, and after these criteria were analyzed, their importance levels were determined by weighting the criteria using the N-AHP method. Thus, an important study in which these two compatible methods were used together for the establishment of a military base was presented to researchers. When the criteria weights of the 12 sub-criteria are analyzed, it was concluded that ease of logistics access is the most important criterion for base location selection.
... 11,12 Anonymity, iteration, statistical group response and controlled feedback are key feature of this method. 17,18 The benefits of this method included larger number of potential participants, filling of questionnaire remotely and anonymity in the expert panel group which further avoid undue influence between participants in decision making. 19 Thorough preparation is critical to ensure the validity and accuracy of a Delphi study. ...
Purpose
To prepare a module on ‘Digital Dentistry”, which can be used to sensitize dental undergraduate students during their final year Prosthodontics curriculum.
Methods
Ethical approval (167/Ethics/2022) was taken from the Institution before starting the study. Need assessment was achieved using focused group discussion (Online Zoom meeting) of various stakeholders of the Department of Prosthodontics of the institution. A preliminary module content was prepared in google questionnaire form after thorough literature review by the research team and final content was developed using modified Delphi technique in 3 questionnaire rounds with the help of 27 experts.
Result
Thematic analysis of Need Assessment focussed group discussion led to the emergence of both theoretical and practical aspect of learning in digital dentistry. The final module was developed with the process followed in modified Delphi technique.
Conclusion
Within the limitations of the study the module of Digital dentistry was developed which can be implemented in most of the dental institutes depending on the resources and available infrastructure.
... The usage of the Delphi technique to facilitate group discussion and promote agreement among a panel of experts on a chosen issue has considerably risen during the past fifty years. The element with the biggest impact on the Delphi process is probably the human element (Grime, Wright, 2016). Recently, thanks to the advancements in science and information systems, the possibilities of this method have substantially increased. ...
A well-liked and reputable technique for gathering data from experts in a field of expertise is the Delphi method. This approach may be used to conduct surveys with qualified respondents from a wide geographic area because face-to-face interaction is not necessary. The Delphi method survey is a practical tool for obtaining expert opinions through repetitive rounds of controlled feedback. To have the maximum response rate possible, investigators must actively engage in this area. Participant cooperation is essential for the Delphi process to be carried out successfully. The main goal of this paper is to offer insights which can be useful to other researchers and practitioners preparing to apply Delphi methodology. Moreover, the article contributes to the methodological debate by taking into consideration the introduction of new practices that can be used to overcome certain classical issues of the Delphi methodology: optimal panel size, objectivity, controlled feedback, participant response rate, etc.
Introducción:
La gastrectomia en manga (GM) se ha transformado en la técnica de cirugía bariátrica y metabòlica más realizada en Chile y el mundo. A pesar la gran cantidad de literatura publicada acerca de esta técnica, aún existen aspectos controversiales sobre los cuales es necesario llegar a acuerdos entre expertos. El objetivo de este trabajo es presentar los resultados del primer consenso chileno de GM obtenido usando el método Delphi.
Materiales y Métodos:
Se realizó una encuesta nacional que consideró aspectos relevantes acerca de la GM. Se logró reunir un total de 60.208 casos. A partir de estos resultados, un comité de consenso realizó un grupo de aseveraciones controversiales definidas a partir de una encuesta nacional de GM. Estas aseveraciones se presentaron a un grupo de expertos reconocidos y de diferentes centros de todo Chile. Se utilizó la escala Likert para establecer consensos de acuerdo o desacuerdo con un punto de corte en 67%.
Resultados:
Se realizaron dos reuniones virtuales donde los 36 expertos votaron 30 aseveraciones. En total, el comité de consenso estuvo de acuerdo en 24 de las 30 aseveraciones propuestas. Hubo consenso de desacuerdo en una de ellas y no se logró consenso en cinco.
Conclusión:
En el primer consenso chileno de GM con el método Delphi, se logró reunir a expertos de todo el país para dirimir temas controversiales y poder servir de referencia para la práctica diaria de la cirugía bariátrica y metabólica en el presente y el futuro.
Maankäyttö- ja elinkeinotyön välisen yhteyden problematiikka sekä näkökulmia sen ratkaisuyrityksiin erityisesti Tampereen ja Turun kaupunkiseuduilla.
Youth with visual impairments (VI) who are physically active are more likely to have higher quality of life (QoL) outcomes, however, typically have higher levels of sedentary time and decreased motor competence, health-related fitness, and physical activity levels than their sighted peers. A major challenge in examining the relationship between QoL and physical activity among children with VI is a lack of psychometrically vetted assessment instruments. The purpose of this Delphi study was to explore the preliminary content and face validity for two measures of QoL for youth with VI by achieving expert consensus on the items of the PedsQL™ and the revised HEAR-QL (now named VISION-QL) survey. A panel of 11 experts in VI assessed each item of the PedsQL™ and VISION-QL on the appropriateness for youth with VI and overall clarity using a 4-point Likert-type scale ranging from 1 ( completely disagree) to 4 ( completely agree). In addition, experts provided feedback on whether the revised questions maintained the meaning from the original question. Panelists were instructed to provide feedback for scores below 3 regarding how to improve the clarity or appropriateness of the item. Items with a median score below 3.25 and 70% agreement were revised and then distributed to the expert panel for a second assessment using the same Likert-type scale and instructions to leave comments for scores below 3. Thirty-four questions were revised for the VISION-QL survey from the HEAR-QL survey. Both instruments reached consensus following two rounds. The results of this study feature the first initial vettings for two separate QoL measures evaluated specifically for ecological validity for individuals with VI. Practitioners can now choose depending upon who their target audience is (e.g., child vs. caregiver) and can do so with confidence.
Mass personalization and shorter product life cycles complicate the assembly of manufacturing companies. To handle the complexity, a high number of employees execute varying assembly tasks. Their work is characterized by learning new tasks frequently to produce different variants. To achieve good learning results in a short amount of time, a suitable individual assembly workers’ competence development process is crucial. Therefore, this publication shows the development of a user methodology to plan the competence development process focusing on selecting learning measures that fit both the work situation and learning employees. The presented user methodology is based on a standardized description and selection procedure for learning measures by using results of a Domain Mapping Matrix, Delphi study, and expert interviews. The publication contributes to improved competence development to support the strategic empowerment of the workforce and to increase the flexibility in manual assembly.
Objective: Older adult patients often face challenges in managing their medication regimens after discharge. Clinical information technologies have the potential to improve medication self-management. However, successful implementation requires addressing various factors impacting healthcare providers' adherence. This study aims to devise the most relevant, acceptable, and feasible strategies to support the implementation of an information system for older adults post-discharge management.
Design: A modified Delphi study
Settings and Participants: A diverse panel of 12 experts, including representatives from program committees, pilot hospitals, and the Hospital Authority.
Methods: The study team utilized the Behavior Change Wheel approach to systematically formulate implementation strategies. Two rounds of rating were conducted, and feedback was collected. Consensus was predefined as a threshold of 75% agreement. Experts rated the relevance, acceptability, and feasibility of each statement on a 5-point Likert scale. Quantitative data were analyzed using descriptive statistics, and qualitative data were analyzed by content analysis.
Results: Ten strategy statements achieved consensus in Round 1, and the remaining 5 reached consensus in Round 2 after refinement, established as high relevance, acceptability, and feasibility. The strategies encompassed environmental redesign, communication and marketing, training, and implementation guidelines. The results showed a strong agreement among the experts, with minimal variations across different expert groups. The main themes leading to experts’ variation included digitalization of the process, interrelations between the strategies, and clinical environment constraints.
Conclusions: This study provides tailored strategies for implementing technology-based services for older adults in acute care hospitals. The strategies, derived through expert consensus, address critical implementation determinants to further enhance medication self-management and patient-provider communication. The findings contribute to the existing knowledge on approaches to change healthcare providers’ behaviors and provide practical guidance for healthcare organizations seeking to improve post-discharge care for older adults.
Objective
While antiretroviral therapy (ART) is highly effective, detection of low levels of HIV-1 RNA in plasma is common in treated individuals. Given the uncertainties on the topic, we convened a panel of experts to consider different clinical scenarios, producing a Delphi consensus to help guide clinical practice.
Methods
A panel of 17 experts in infectious diseases, virology and immunology rated 32 statements related to four distinct scenarios: (1) low-level viremia during stable (≥6 months) first-line ART (≥2 consecutive HIV-1 RNA measurements 50–500 copies/mL); (2) a viral blip during otherwise suppressive ART (a HIV-1 RNA measurement 50–1000 copies/mL with adjacent measurements <50 copies/mL); (3) low-level viral rebound during previously suppressive ART (≥2 consecutive HIV-1 RNA measurements 50–500 copies/mL); (4) residual viremia during suppressive ART (persistent HIV-1 RNA quantification below 50 copies/mL). A systematic review, conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis statement, informed the 32 statements. The Delphi procedure was modified to include two voting rounds separated by a moderated group discussion. Grading of Recommendations, Assessment, Development, and Evaluations-based recommendations were developed.
Results
Overall, 18/32 statements (56.2%) achieved a strong consensus, 3/32 (9.4%) achieved a moderate consensus and 11/32 (34.4%) did not achieve a consensus. Across the four scenarios, the panel unanimously emphasised the importance of implementing specific interventions prior to considering therapy changes, including assessing adherence, testing for genotypic drug resistance and scheduling more frequent follow-up visits. Strategies indicated in selected circumstances included therapeutic drug monitoring, quantifying total HIV-1 DNA and evaluating concomitant chronic infections.
Conclusions
While acknowledging the many uncertainties about source, significance and optimal management of low-level viremia during ART, the findings provide insights to help harmonise clinical practice. There is a need for well-designed randomised studies assessing different interventions to manage low-level viremia and future research regarding its definition.
Information hiding in images has gained popularity. As image steganography gains relevance, techniques for detecting hidden messages have emerged. Statistical steganalysis mechanisms detect the presence of hidden secret messages in images, rendering images a prime target for cyber-attacks. Also, studies examining image steganography techniques are limited. This paper aims to fill the existing gap in extant literature on image steganography schemes capable of resisting statistical steganalysis attacks, by providing a comprehensive systematic literature review. This will ensure image steganography researchers and data protection practitioners are updated on current trends in information security assurance mechanisms. The study sampled 125 articles from ACM Digital Library, IEEE Explore, Science Direct, and Wiley. Using PRISMA, articles were synthesized and analyzed using quantitative and qualitative methods. A comprehensive discussion on image steganography techniques in terms of their robustness against well-known universal statistical steganalysis attacks including Regular-Singular (RS) and Chi-Square (X²) are provided. Trends in publication, techniques and methods, performance evaluation metrics, and security impacts were discussed. Extensive comparisons were drawn among existing techniques to evaluate their merits and limitations. It was observed that Generative Adversarial Networks dominate image steganography techniques and have become the preferred method by scholars within the domain. Artificial intelligence-powered algorithms including Machine Learning, Deep Learning, Convolutional Neural Networks, and Genetic Algorithms are recently dominating image steganography research as they enhance security. The implication is that previously preferred traditional techniques such as LSB algorithms are receiving less attention. Future Research may consider emerging technologies like blockchain technology, artificial neural networks, and biometric and facial recognition technologies to improve the robustness and security capabilities of image steganography applications.
The Delphi method was used to gather assessments from 93 experts about drivers of future angling participation by locals and tourist anglers in Denmark, Finland, Iceland, Norway, and Sweden. The main drivers of future angling participation related to habitat and fish populations, and access to and information about fishing. For Norway and Finland, the predicted future decline in local angler numbers was consistent with a life-cycle model of recreational fisheries, while the anticipated increase for the three other countries contradicted the model. For tourist anglers, growth was expected for both domestic and foreign tourists. Long-term and societal drivers, such as urbanization , sociocultural changes, and climate change were not seen as strong drivers, and may be considered out of reach by managers, but should be included with information and conservation drivers in angler recruitment and retention strategies. K E Y W O R D S constraints, facilitators, life-cycle of recreational fisheries, outdoor recreation, recruitment, tourist
Introducción: La gastrectomía en manga (GM) se ha transformado en la técnica de cirugía bariátrica y metabólica más realizada en Chile y el mundo. A pesar la gran cantidad de literatura publicada acerca de esta técnica, aún existen aspectos controversiales sobre los cuales es necesario llegar a acuerdos entre expertos. El objetivo de este trabajo es presentar los resultados del primer consenso chileno de GM obtenido usando el método Delphi.Materiales y Métodos: Se realizó una encuesta nacional que consideró aspectos relevantes acerca de la GM. Se logró reunir un total de 60.208 casos. A partir de estos resultados, un comité de consenso realizó un grupo de aseveraciones controversiales definidas a partir de una encuesta nacional de GM. Estas aseveraciones se presentaron a un grupo de expertos reconocidos y de diferentes centros de todo Chile. Se utilizó la escala Likert para establecer consensos de acuerdo o desacuerdo con un punto de corte en 67%.Resultados: Se realizaron dos reuniones virtuales donde los 36 expertos votaron 30 aseveraciones. En total, el comité de consenso estuvo de acuerdo en 24 de las 30 aseveraciones propuestas. Hubo consenso de desacuerdo en una de ellas y no se logró consenso en cinco. Conclusión: En el primer consenso chileno de GM con el método Delphi, se logró reunir a expertos de todo el país para dirimir temas controversiales y poder servir de referencia para la práctica diaria de la cirugía bariátrica y metabólica en el presente y el futuro.
Smart cities are characterized by the integration of various technologies and the use of data to achieve several objectives. These objectives include the creation of efficiencies, boosting economic development, expanding sustainability, and improving the overall quality of life for individuals residing and working within the urban environment. The aim of this study is to analyze the future of smart cities with respect to developing countries, specifically Jordan as the case. This analysis is based on the opinions and feedback from the field experts. In this study, we are tapping into multiple domains of smart cities such as smart governance, education, healthcare, communication, transportation, security, energy, and sustainability. The field experts' consensus was developed with the Delphi method. The Delphi survey comprises eight questions to assess the views about smart city adoption and development with respect to Jordan. The results and findings of this study revealed specific challenges and opportunities in smart city adoption with respect to Jordan. The experts' opinions have validated the study of the 2023 Smart City Index report. They have offered crucial input and future guidance for the adoption of smart cities in Jordan. Additionally, they have indicated which domains of smart cities should be prioritized during the implementation in Jordan.
A key aspect of transferrable oil and gas expertise to Subsurface CO 2 Storage (SCS) relates to risk assessment. While initial subsurface risk and volume assessments for SCS projects are similar to oil and gas prospect evaluations, full lifecycle risk assessment requires evaluation of the current knowledge of the storage complex and future potential events that may have impacts over long timeframes.
It is important to learn from past water, gas and CO 2 injection and storage projects, examples of which are reviewed here. The concepts of aleatory and epistemic risk and uncertainty are discussed and the use of standard risk matrices for evaluation of long-term, low frequency but potentially high impact events is challenged. Drawing on the large volume of previous work, this paper highlights key elements for development of a robust risking framework, including thorough project framing and implementation of a staged approach to project execution.
The paper assesses methods for ensuring all potential hazards are captured and addresses the challenges of defining a quantitative risking scale suitable for long-term SCS projects. Example quantitative risk profiles can be used to calculate the timing and duration of ‘Peak Risk’, augment monitoring and mitigation planning for management of risk and capital exposure and help ensure successful project outcomes.
Thematic collection: This article is part of the Geoscience workflows for CO 2 storage collection available at: https://www.lyellcollection.org/topic/collections/geoscience-workflows-for-CO2-storage
El planteamiento principal del aprendizaje por competencias es que los estudiantes deben egresar con un saber, un saber hacer y un saber ser. Desde hace décadas, la traductología busca determinar cuáles son los conocimientos, las capacidades y las actitudes que un traductor debe aprender. Si bien son varios los modelos que existen y que, pese a sus divergencias, coinciden en la creciente complejidad de la competencia traductora, no son tantos los estudios sobre cómo se adquiere esta competencia ni las guías para conseguirlo. Al mismo tiempo, el mercado de la traducción avanza a pasos agigantados, sobre todo a raíz de la introducción de las herramientas de traducción asistida por ordenador, la traducción automática o, más recientemente, la inteligencia artificial. En este contexto, resulta pertinente reflexionar sobre qué y de qué manera enseñar en Traducción. Este artículo presenta un listado de prácticas docentes consensuadas entre un conjunto de expertos anónimos, que incluye profesores, traductores en activo y otros con cargos de responsabilidad en empresas de traducción, a través de la metodología Delphi. Además del listado propuesto por la investigadora, creado a partir de las conceptualizaciones de distintos autores, modificado y enriquecido por dichos expertos, y de las discrepancias detectadas entre las valoraciones sobre las prácticas, se presentan las conclusiones, que apuntan a la conveniencia de establecer una mayor coordinación entre asignaturas de traducción en los grados, así como a la pertinencia de lograr una colaboración entre expertos de las tres categorías para aprovechar mejor su sabiduría colectiva.
Industry 4.0, whose effects have been more and more noticeable in recent years, and the digital change it brings call for a new educational model that aligns university instructional processes and curricula with the demands of business. This approach, known as University 4.0, intends to promote more technology-based applications, the power of innovation, the training of skilled specialists to satisfy industrial needs, and the development of competences that can achieve multidisciplinary projects. It is of utmost importance to adapt late-industrializing nations’ educational systems, such as Turkey’s, to this change process. The aim of this research, which takes the Turkish context as a reference, is to reveal how the curricula and educational processes of business schools should be shaped with a perspective that takes into account the human resources requirements of industry 4.0. A decision-making technique that integrates QFD, Delphi, and DEMATEL methods was employed in the study. The results clearly show that the most important expectation for social stakeholders is to provide university-industry cooperation. However, some of the most pressing problems include applied learning through collaborative educational experiences, teamwork skills, changing roles and competencies of academics. The findings point out that the educational activities of business schools must be linked with industry expectations through technology-based training activities and teamwork.
This study aims to determine the factors that cause burnout in nurses, who are critical stakeholders in health service delivery, and to prioritize these factors. For this purpose, Delphi and SWARA methods were used. The research was conducted in a private hospital serving Black Sea Region Türkiye. 10 unit manager nurses who are experts in their fields participated in the Delphi method. As a result of the three-stage Delphi implementation, 16 burnout causes were determined. The 16 identified causes were prioritized with the SWARA method. 47 nurses participated in the SWARA method. As a result of the research, it was determined that the most critical reason for burnout was not being able to get money for the work done/low salary. It was concluded that obscure patient relatives were the least important cause of burnout. It is thought that the financial and moral reward of nurses who work hard will be beneficial. It may be suggested to make satisfactory remuneration for the labor and service provided or to support and increase the raw wage with various contributions. It can be suggested to raise awareness among nurses about burnout and coping.
This study aimed to identify the specific home environment factors that were judged to support or hinder the development of children’s critical reading skills. Using a Delphi method, 32 experts in Finland listed a set of home-related factors that can either hinder or support the development of children’s critical reading skills. The experts then evaluated and ranked the factors according to their perceived importance. A large set of home-related factors was produced. Out of these, we identified 13 supportive and nine hindering factors. The factors highlighted the importance of having a space for the child to be heard and involved in family discussions, having a space for differing viewpoints and critical thinking, parental competencies to support critical reading skills, and positive parental attitudes towards schooling and learning. The findings can be used for measurement and intervention development purposes.
The purpose of this research was to develop an understanding of how information literacy (IL) research is operationalized by means of the Delphi method, the current state of the method’s usage in IL research and its key features. A systematic review of IL research studies utilizing Delphi was undertaken in April and December 2022, using studies retrieved from five databases. The main findings of the analysis are that Delphi was not a common method for IL studies; nevertheless, it was used to study various issues, including digital and health (information) literacy, and in various contexts, mainly those of education, health care, and librarianship, leading to a variety of findings, most often relating to IL competence or skill framework. Delphi was used highly flexibly, utilizing diverse groups of experts with various signifiers of expertise. If applied rigorously, with other methods, Delphi may contribute to IL theory and practice.
The Knowledge Economy is sustained by a large and varied range of type of libraries (public-state or university library, private library…) whose missions and models of delivering services are increasingly challenging due to change in readings habits since many years, and more recently by the Covid-19 crisis. In such context, we look at this sector as an organizational field whose future we question in the light of social (in the habits of access and use of knowledge), technological and relational changes, and raising the following question: “how is the organizational field of libraries reconfiguring internationally and externally in 2030? ". We conduct a prospective study to look ahead to the expected evolution of libraries and the role they could continue to play to support education, culture, and economy (in 2030). We use the Delphi method in three steps (data collection, formulation of an initial set of projections from discussions with experts, interviews and focus-groups). We propose four scenarios, two of which are considered trend scenarios, extensions of current trends in which libraries are adapting to a changing environment. Our results show that, despite significant consequences, COVID-19 did not change the course of these two scenarios. The two other projections are seen as “disruptive” scenarios: one anticipates the marginalization of libraries, the other the breakdown of the library concept itself. We finally offer contributions to the organizational field literature.
The Real-Time Geo-Spatial Consensus System (RT-GSCS) is a web-based open-source tool designed for the
administration of real-time spatial surveys. One distinguishing feature of this system lies in the adoption of the RealTime Spatial Delphi logic as the main computational algorithm to achieve a convergence of opinions within a group of
individuals. The system is available at http://rtgscs.com/ and is particularly instrumental in supporting the decisionmaking process when spatial data is absent or when user judgments are solicited for territorial actions. Since the
release of the beta version, the system has undergone multiple iterations and significant changes, and for this reason,
an updated assessment should be conducted. In this paper, we present an in-depth exploration of the latest version of
RT-GSCS (v3.0), which includes novel algorithms, statistical analysis, and spatial techniques, optimized for survey
administration. The analysis is meticulously conducted by evaluating a comprehensive array of parameters,
encompassing performance metrics, spatial elements, security, updates, and privacy assessments. Moreover, a
thorough examination of user feedback is presented, enriching the depth of our investigation. This mixed-method
approach ensures a robust and updated investigation of the system, facilitating a nuanced comprehension of the
process and the attainment of spatial consensus.
Due to the hydrologic non-stationarity and uncertainty related to the probability assignment of flood peaks under climate change, the use of flood statistics may no longer be applicable. Therefore, a sensitivity analysis (i.e., a scenario-neutral approach) is used to examine the impacts of climate change on flooding in the Ba River Basin. A Delphi method with a set of KAMET rules was used to obtain a representative and a threshold flood event. These inputs are used for hydraulic simulation using a MIKE FLOOD model package. Flood simulations were performed using parametrically varied rainfall and temperature conditions. In total, 22 conditions were explored and are in line with CMIP5 and CMIP6. The results obtained have several implications. Firstly, rainfall change is the primary factor affecting flood impact in the Ba River Basin. Secondly, the flood peak in the Ba River Basin is highly sensitive to an increase in rainfall by up to 10%. Thirdly, the flooded threshold is reached when rainfall increases beyond 20%. Fourthly, the flood extent and depth are expected to increase as rainfall increases. Further research could improve the study using satellite rainfall data, satellite digital elevation models, and stochastic weather generators.
The futures of humanity and planet Earth are at stake. This is reflected not only in the increasingly dire future imaginations of billions of people around the world but also in an ever-increasing body of future-related literature in the social sciences and humanities. However, despite growing sociological engagement with the future, an astonishing desideratum remains: the dissemination of future imaginations. Although many works imply that future imaginations disseminate, they rarely spell out how the cultural mechanisms of dissemination work. Therefore, in this article, I develop the notion of future-cultures to conceptualize how future imaginations disseminate throughout the social, drawing from cultural sociology and theories of social practices. I conceptualize the future-cultures framework in three steps: (1) how future-cultures generate future-cultural codes, which select and classify (il-)legitimate future imaginations, sites of futuring and futuring practices; (2) how future-cultural codes relate different (futuring) practices and discourses into broader practice-/discourse-complexes, which (3) organize transver-sally in fields of futuring and modes of futuring, thereby disseminating distinct future ima-ginations over space and time.
PWS Smart-Start is a behavioral caregiver training program developed specifically for caregivers of children with Prader-Willi syndrome (PWS) aged 3 to 14. The purpose of the current study was to evaluate the acceptability and preliminary efficacy of the program. Thirty-four caregivers of children with PWS received the PWS Smart-Start training using online live video-coaching across a 10-week period. The impact of the training on a variety of variables including caregiver and child behavior, caregiver stress and burnout, and family functioning were evaluated using a quasi-experimental pretest posttest design. Results indicated statistically significant decreases across child behavior challenges and parental stress and burnout following the training. Statistically significant improvements were also seen in parenting practices, the quality of the parent–child relationship, and family quality of life. Results of social validity measures indicated caregiver satisfaction with the program. These preliminary results indicate that the PWS Smart-Start program has the potential to not only support children with PWS through behavioral challenges but also reduce caregiver stress and burnout and improve a variety of aspects of family life. Future research is needed to replicate and further validate these results and to extend the exploration of other behavior analytic interventions for this population.
Simple Summary
Horse sports are popular worldwide, providing spectator enjoyment, benefiting human health, and contributing substantially to national economies. Training and management practices used to care for sporthorses are generally based on tradition rather than science; this combined with the high-risk nature of equestrian activities led to the public questioning if sporthorse health and welfare are being compromised. To understand better how sporthorses are being managed, experts, actively involved with national and international horse sports (dressage, showjumping, and eventing) were consulted across four rounds of a Delphi study. This approach allowed participants to interact to reach a point where everyone agreed on core areas (or domains) that they felt were essential to sporthorse management. Five areas were rated as essential: training management, competition management, young horse management, health status and veterinary management, and the horse–human relationship. Stable and environmental management, and welfare assessment were rated as important but not essential, as most experts felt that these areas were already managed well. Experts called for increased education and research to support riders, trainers, and federations. A welfare charter and evidence-based guidelines to inform management practices were advised to ensure sporthorses have a good life and to safeguard the future of equestrian sports.
Abstract
The public is increasingly questioning equestrianism’s social license to operate. While the focus historically centered on horseracing, increased scrutiny is now being placed on how dressage, showjumping, and eventing are addressing equine management and welfare concerns. Nominated equestrian federation and equestrian organization experts (n = 104) directly involved in international and/or national-level horse sports took part in a four-stage, iterative Delphi to obtain consensus on what factors should be considered essential to manage sporthorse health and welfare. Five core domains were agreed as essential: training management, competition management, young horse management, health status and veterinary management, and the horse–human relationship. Two further domains: stable and environmental management, and welfare assessment were rated as important but not essential, as most respondents felt that these areas were already managed well. Participants felt increased education and guidance combined with further policy development and regulation are needed to support stakeholders to optimize sporthorse management. An appetite to engage with research to generate evidence that promotes sporthorse welfare was evident. The development of a sporthorse welfare charter and evidence-based guidelines to inform the management and monitoring of sporthorses’ health and welfare are recommended to provide horses with a good life and to safeguard the future of equestrian sports.
Purpose: This study was conducted to develop a monitoring tool to manage unregistered mentally ill patients at community mental health centers (M-MUMI).Methods: The Delphi method was used in this study. The participants of this study were 27 psychiatric experts. In the first Delphi round, content analysis was conducted. In the second Delphi round, Kendall’s W rank and weight rank were used. The third Delphi round was analyzed based on the content validity ratio (CVR).Results: The finally developed M-MUMI consisted of 2 domains, 7 categories, and 22 items. Assessment of the present status consisted of 4 basic elements affecting daily life, 5 psychological statuses, 2 elements related to the utilization of social service, and 3 causes for the refusal of registration. The registration strategy of the individual cases consisted of 2 human resources to help registration, 2 additional social services for registration, and 4 individual needs to induce registration.Conclusion: It is important to manage unregistered mentally ill patients at the community mental health centers. We hope that the developed M-MUMI will help to monitor unregistered mentally ill patients how to maintain mental health in the community.
This is a BOOK on the Delphi Method published in the early 70's. It is very basic and the first two chapters are a necessary introduction.
The first chapter explains the method and the second gives the influences of different philosophies of truth on how they impact the design of a Delphi. Even different sciences have different types of concepts for what is truth in that scientific field. Many applications of Delphi require using experts in numerous sciences to understand the totality of problem being examined.
the online access to the delphi method book allows the user to download separate chapters or separate papers in a chapter.
The policy delphi is a specific design to address a policy issue that has many alternative resolutions to the solution of the policy issue.
It starts with a literature search to establish all the different published resolutions of the policy or recommend ones. The paper describes how the respondents to the Delphi should vote on the different resolutions for
desirability and feasible. Any respondent can mage positive or negative comments about the issue and anyone can vote on any of these comments for the true or false measure and the feasibility measure as well.
Participants may change (at any time) their votes at any time due to the contributions of the participants. Ideally this is done online and may be entered at any time for any participant and the system shows the user new items or new vote values they have not seen. The user may also review any part of the delphi they wish to see at that time. They may also introduce at any time new policy resolutions and new comments. The published paper has an example topic. It can be done as an mailed set of rounds also but it does make it harder to make a lot of dynamic changes to earlier rounds.
Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts? One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment. A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible). Delphi groups are substantially more accurate than individual experts and traditional groups and somewhat more accurate than statistical groups (which are made up of noninteracting individuals whose judgments are aggregated). Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.
Proposes a general theory (social decision scheme theory) for many kinds of group decision-making and illustrates some special case models with a variety of data from several experimental situations. While focusing upon the traditional issue of individual-group differences, the theory is aimed at accounting for the distribution of group decisions by using formal hypotheses about the effects of social interaction when the inputs to discussion are individual member preferences. The basic assumptions underlying the model are similar in several respects to proposals by F. Restle and J. H. Davis (see record
1963-06168-001) and I. D. Steiner (see record
1966-10588-001) in group problem-solving research. The model itself represents the general case of earlier theoretical notions by W. H. Smoke and R. B. Zajonc; Davis, Hoppe, and Hornseth (see record
1968-09967-001); and Zajonc, Wolosin, and Wolosin in group decision making. In addition, several nonintuitive consequences of group decision making, assuming some form of the model, are discussed. (41 ref.) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Since its origins, decision makers have broadly used the Delphi method as a collaborative technique for generating important events and scenarios about what may happen in the future. This is a complex process because of the different interrelations and the potential synergetic effects among the relevant events related to a decision. This fact, along with the uncertainty about the occurrence or non-occurrence of the events, makes the scenario generation task a challenging issue in Delphi processes. In the 1960's, Cross-Impact Analysis (CIA) appeared as a methodological tool for dealing with this complexity. CIA can be used for creating a working model out from a set of significant events. CIA has been combined with other methodological approaches in order to increase its functionality and improve its final outcome. In this paper, the authors propose a new step-by-step model for scenario-analysis based on a merger of Turoffs alternative approach to CIA and the technique called interpretive Structural Modeling (ISM). The authors' proposal adds tools for detecting critical events and for producing a graphical representation to the previous scenario-generation methods based on CIA. Moreover, it allows working with large sets of events without using large computational infrastructures. The authors present sufficient information and data so that anyone who wishes to may duplicate the implementation of the process. Additionally they make explicit a set of requirements for carrying out a Delphi process for a group to develop a set of significant events, collectively make the estimations of cross impacts, and to support a continuous planning process within an organization. They use two examples to discuss operational issues and practical implications of the model.
In an extreme event or major disaster, very often there are both alternative actions that might be considered and far more requests for actions than can be executed immediately. The relative desirability of each option for action could be a collaborative expression of a significant number of emergency managers and experts trying to manage the most desirable alternatives at any given time, in real time. Delphi characteristics can satisfy these needs given that anyone can vote or change their vote on any two options, and voting and scaling are used to promote a group understanding. Further utilized with Thurstone's Law of Comparative Judgment, a group decision or the range of acceptability a group is willing to consent to, can be calculated and utilized as a means of producing the best decision. A ubiquitous system for expeditious real-time decision making by large virtual teams in emergency response environments is described.
This paper examines the Delphi technique critically to determine whether it succeeds in alleviating the 'process loss' typical of interacting groups. After briefly reviewing the technique, we go on to consider problems with Delphi from two perspectives. First, we examine methodological and technical difficulties, and the problems these have brought about in experimental applications. We suggest that important differences exist between the typical laboratory Delphi and the original concept of Delphi. These differences, reflecting a lack of control of important group characteristics/factors (such as the relative level of panellist expertise), make comparisons between Delphi studies unrealistic, as are generalizations from laboratory studies to the ideal of Delphi. This conclusion diminishes the power of those former Delphi critiques which have largely dismissed the procedure because of the variability of laboratory study results (e.g. Sackman (45)). Second, having noted the limited usefulness of the majority of studies for answering questions on the effectiveness of Delphi, we go on to look at the technique from a theoretical/mechanical perspective. That is, by drawing upon ideas/findings from other areas of research (e.g. Hogarth (27)), we attempt to discern whether the structure of the Delphi procedure itself might reasonably be expected to function as intended. We conclude that inadequacies in the nature of feedback typically supplied in applications of Delphi tend to ensure that any small gains in the resolution of 'process loss' are offset by the removal of any opportunity for group 'process gain'. Some solutions to this dilemma are advocated, which are based on an analysis of the process of judgment change within groups and a consideration of factors which increase the validity of statistical/nominal groups over their constituent individual components.
We propose a pooling method to aggregate expert opinions. Intuitively, it works as if the experts were continuously updating their opinions in order to accommodate the expertise of others. Each updated opinion takes the form of a linear opinion pool, where the weight that an expert assigns to a peer's opinion is inversely related to the distance between their opinions. In other words, experts are assumed to prefer opinions that are close to their own opinions. We prove that such an updating process leads to consensus, i.e., the experts all converge towards the same opinion. Further, we show that if experts are rewarded using the quadratic scoring rule, then the above mentioned assumption follows naturally. We empirically demonstrate the efficacy of our method using real-world data.
Although there is debate about the appropriate place of guidelines in clinical practice, guidelines can be seen as one way of assisting clinicians in decision-making. Given the likely diversity of opinion that any group of people may display when considering a topic, methods are needed for organising subjective judgements. Three principal methods (Delphi, nominal group technique, consensus development conference) exist which share the common objective of synthesising judgements when a state of uncertainty exists.
To identify the factors that shape and influence the clinical guidelines that emerge from consensus development methods and to make recommendations about best practice in the use of such methods.
Five electronic databases were searched: Medline (1966-1996), PsychLIT (1974-1996), Social Science Citation Index (1990-1996), ABI Inform and Sociofile. From the searches and reference lists of articles a total of 177 empirical and review articles were selected for review.
The output from consensus development methods may be affected by: the way the task is set (choice of cues, recognition of contextual cues, the focus of the task, the comprehensiveness of the scenarios); the selection of participants (choice of individuals, degree of homogeneity of the group, their background, their number); the selection and presentation of scientific information (format, extent to which its quality and content is assessed); the way any interaction is structured (number of rating rounds, ensuring equitable participation, physical environment for meetings); and the method of synthesising individual judgements (definition of agreement, rules governing outliers, method of mathematical aggregation).
Although a considerable amount of research has been carried out, many aspects have not been investigated sufficiently. For the time being at least, advice on those aspects has, therefore, to be based on the user's own commonsense and the experience of those who have used or participated in these methods. Even in the long term, some aspects will not be amenable to scientific study. Meanwhile, adherence to best practice will enhance the validity, reliability and impact of the clinical guidelines produced.
This article describes some dilemmas and problems encountered in a Delphi study of general practitioners' (GPs) information requirements. The research involved a three-round Delphi administered to an expert panel of 50 GPs in one Welsh health authority area. The Delphi is generally perceived as a qualitative METHOD: However, we argue that the requirements of the technique make it difficult to sustain the kind of inductive analysis - faithfully reflecting respondents' perspectives - that is axiomatic to many of the theoretical approaches that underpin qualitative inquiry. We describe how our attempts to incorporate respondents' views in near-verbatim form in the first round were undermined by the need to classify and reduce statements for later rounds, and to impose judgments about what should count as consensus. The iterative 'consensus-building' process, that is so central to the Delphi, was difficult to take forward without active intervention by the research team, but this involved a re-ordering and reduction of the data, which moved the statements included in later rounds further and further from the verbatim responses on which they were based. Whilst the findings of the study were useful on one level, the final consensus statements took a general, virtually context-free form, which contained few references to background preoccupations (largely concerned with the recent NHS reforms) that were exercising GPs at this time. The method also coped badly with the polarized opinions that existed regarding the development of information systems for commissioning. We conclude that the Delphi is best used for large-scale research in areas where opinions are well established, where the problems and choices confronting the study group are well known, and where a major programme of organizational reform is not already underway.
Decision conferencing provides a structured group approach for assessing the risks associated with several options, with expert judgment providing some or all of the inputs and models based on decision theory used to combine the inputs into an overall assessment. A facilitated workshop brings together experts about a specific risk criterion, with an impartial facilitator taking the group through the process of discussing the meaning of each criterion, establishing measurement scales, and appraising options on the scales. At a subsequent decision conference attended by the key players, the facilitator guides participants through the process of making the final judgments about inputs and trade-offs between criteria that allow the decision theory model to bring together all the inputs to provide an overall ordering of the options. This approach allows the collective experience of specialists and key players to be brought to bear on issues of risk assessment, and creates committed alignment backed by the authority of the entire group.
Decision conferencing provides a structured group approach for assessing the risks associated with several options, with expert judgment providing some or all of the inputs and models based on decision theory used to combine the inputs into an overall assessment. A facilitated workshop brings together experts about a specific risk criterion, with an impartial facilitator taking the group through the process of discussing the meaning of each criterion, establishing measurement scales, and appraising options on the scales. At a subsequent decision conference attended by the key players, the facilitator guides participants through the process of making the final judgments about inputs and trade-offs between criteria that allow the decision theory model to bring together all the inputs to provide an overall ordering of the options. This approach allows the collective experience of specialists and key players to be brought to bear on issues of risk assessment, and creates committed alignment backed by the authority of the entire group.
Keywords:
decision conferences;
decision conferencing;
facilitated workshops;
group decision;
support rooms;
multicriteria decision analysis;
decision theory;
risk analysis;
radioactive waste disposal
This paper gives an account of an experiment in the use of the so-called DELPHI method, which was devised in order to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of questionnaires in depth interspersed with controlled opinion feedback.
Face-to-Face group discussion and Delphi-type groups are used to forecast interest rate for periods up to six months. Subjects were different groups of experts. The forecasting procedure of the Delphi-Group was supported by a data bank of possibly influencing variables. The use of te data bank was studied. Pros and Cons of both procedures are shown.
Review of J. Scott Armstrong's 1978 book on forecasting. Click on the DOI link above to read the review
Background
Although there is debate about the appropriate place of guidelines in clinical practice, guidelines can be seen as one way of assisting clinicians in decision-making. Given the likely diversity of opinion that any group of people may display when considering a topic, methods are needed for organising subjective judgements. Three principal methods (Delphi, nominal group technique, consensus development conference) exist which share the common objective of synthesising judgements when a state of uncertainty exists.
Objectives
To identify the factors that shape and influence the clinical guidelines that emerge from consensus development methods and to make recommendations about best practice in the use of such methods.
Methods
Five electronic databases were searched: Medline (1966-1996), PsychLIT (1974-1996), Social Science Citation Index (1990-1996), ABI Inform and Sociofile. From the searches and reference lists of articles a total of 177 empirical and review articles were selected for review.
Results
The output from consensus development methods may be affected by: the way the task is set (choice of cues, recognition of contextual cues, the focus of the task, the comprehensiveness of the scenarios); the selection of participants (choice of individuals, degree of homogeneity of the group, their background, their number); the selection and presentation of scientific information (format, extent to which its quality and content is assessed); the way any interaction is structured (number of rating rounds, ensuring equitable participation, physical environment for meetings); and the method of synthesising individual judgements (definition of agreement, rules governing outliers, method of mathematical aggregation).
Conclusions
Although a considerable amount of research has been carried out, many aspects have not been investigated sufficiently. For the time being at least, advice on those aspects has, therefore, to be based on the user's own commonsense and the experience of those who have used or participated in these methods. Even in the long term, some aspects will not be amenable to scientific study. Meanwhile, adherence to best practice will enhance the validity, reliability and impact of the clinical guidelines produced.
In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.
In this paper, we present a novel approach for generating scenarios in multi-stakeholder environments. In order to address one of the most imperative environmental and societal challenges related to mobility, we explore the future of electric drive vehicles (EDVs). Since many different stakeholders are involved in the socio-technological transition from internal combustion engines (ICEs) to EDVs, we present a dissent-based scenario development process, which uses the Delphi technique for data generation. In total, 140 experts from 15 stakeholder groups participated in this German-based survey and assessed controversial projections for 2030. Results reveal a considerably high degree of dissent. In order to cope with different viewpoints in the scenario development process, we utilize a novel 5-step dissent analysis for further insights into possible futures. Thus, we account for potential differences among stakeholder groups (step 1), the effect of a desirability bias (step 2), as well as the impact of outliers (step 3) and bipolarity (step 4) in the survey results. Finally, we identify different clusters of experts through latent class analysis (step 5). Based on the results of the dissent analysis, seven partially conflicting multi-stakeholder scenarios for the future of EDVs in 2030 are developed.
The significance of innovation for sustainable performance of companies, as well as entire economies, is indisputable. The proactive management of innovations has proven to be a central driver of sustainable long-term competitiveness. More recent research concentrates on the ability of corporate foresight to increase the innovation capacity of a firm. We contribute to this field by showing the development and implementation of a tailored scenario process for innovation management that has been implemented at a leading global manufacturer of high performance materials for their innovation field of future logistics. Our demonstration includes all phases from visualisation of contextual scenarios to market scenarios of target groups to new product concepts and finally concrete business models. In order to increase the innovation capacity in the scenario process we integrated a Delphi-expert survey, participatory workshops and an open network approach. The scope of our scenarios was global in nature as were the nationalities of the experts that took part in our innovation-focused scenario process. Represented countries included among others European countries, BRIC States and the USA.
The Delphi technique was largely developed to avoid the problems of freely interacting groups such as dominant individuals and pressure to conform to the majority view. Our review of the social psychological literature reveals some obstacles to Delphi achieving its full potential relative to other cheaper and easier methods of aggregating judgment. We identify residual normative and informational pressures towards consensus that potentially reduce process gain that might otherwise be achieved. For instance, panelist confidence may act as a signal of status rather than be a valid cue to expertise, whereas consensus appears to have a strong influence on the final outcome that can reduce its accuracy when there are valid minority opinions. We argue that process gain in Delphi must occur through those further from the "truth" changing their opinion more than those closer to the truth, with the general direction of opinion change being towards the truth. For such virtuous opinion change to occur we suggest the need to both facilitate opinion change and provide good cues as to where the truth lies. Research on Judge Advisor Systems shows that people usually do not change their opinion as much as they should, giving too much weight to their own opinion and too little to the views of others—this bias can be reduced by increasing involvement and motivation. In addition, we propose that the best way to provide good cues as to the direction of the truth is to elicit rich reasoning from panelists about the judgment or choice in question, then use this as feedback. We suggest practical ways of focusing and deepening panelists’ consideration and evaluation of such reasoning—such that all proffered opinions are well-evaluated. Additionally, we propose a model of opinion change in Delphi for use as a paradigm for future process-orientated research.
This work presents a new methodology based on three well-known qualitative techniques (Focus Group, Nominal Group Technique and Delphi method), with a view to harmonising their potentialities and reducing their limitations, through application in real contexts with experts who are professionals in their respective activities.The main contribution of this methodology is its joint consideration of the needs of the investigators and also of experts who act in professional contexts, in order to improve the effectiveness of preceding techniques in achieving the scientific and social objectives of the study.We have tested this methodological approach in three real cases, with experts holding different responsibilities in different companies and public organisations, and the results secured are highly satisfactory, due to both the quantity and quality of the proposals obtained, and the satisfaction exhibited by the experts taking part, with regard to the research methodology.
This report describes the results of an extensive set of experiments, conducted at RAND, concerned with evaluating the effectiveness of the Delphi procedures for formulating group judgements. The study represents a small beginning in the field of research that could be called ‘opinion technology’, and is of direct relevance for the use of experts as advisers in decision making, especially in areas of broad or long-range policy formulation.
This paper proposes a modified version of the Delphi method, starting from the assumption that in many situations the convergence of opinions can be considered in a spatial context. The Spatial Delphi we suggest is based, like the classical Delphi, on the judgments of experts, and it is useful in the consultations for decision and/or forecast purposes, provided that they concern matters of spatial location. The basis for the questionnaire is a map, on which each expert provides, as answer(s), one or more opinion-points, i.e., locations that, according to their opinion, are best for a specific purpose. We propose a method for narrowing the area of convergence along a multi-step procedure so that the final result of the consultation is a small portion of the initial territory with respect to the initial area considered. Besides a simulation study, we report also a prototype application with a panel of twelve experts.
Introducing new technology, business processes, and working methods requires input and agreement from numerous participants if it is to succeed. Often the potential for, and cost of, failure precludes innovation. The Modified Delphi Method is a consultative process that uses computer technology combined with a survey to obtain constructive participation in the input and consensus process that is often essential to the success of a planned change. The method adds the advantage that the experience and expertise of the participants can be used to validate the proposed process so that the risk of failure is significantly reduced. There are two elements to development of computer technology: the technology itself and the effective application of that technology to a specific problem area. This paper addresses the second of these aspects as it presents a process that takes advantage of available computer technology to collect and analyze opinions from a reasonably large and diverse group of participants, The paper uses actual experience in validation of a new contracting process as the vehicle for illustration; the Modified Delphi process itself is applicable to many other situations.
This article describes a field experiment which uses the Delphi Process to assign subjective probabilities to the set of possible demand levels for a company's product. The method is validated by the use of questions to which the "correct" answers are known.
This paper examines critically the Delphi technique to determine whether it succeeds in alleviating the “process loss” typical of interacting groups. After briefly reviewing the technique, we consider problems with Delphi from two perspectives. First, we examine methodological and technical difficulties and the problems these have brought about in experimental applications. We suggest that important differences exist between the typical laboratory Delphi and the original concept of Delphi. These differences, reflecting a lack of control of important group characteristics/factors (such as the relative level of panelist expertise), make comparisons between Delphi studies unrealistic, as are generalizations from laboratory studies to the ideal of Delphi. This conclusion diminishes the power of those former Delphi critiques that have largely dismissed the procedure because of the variability of laboratory study results. Second, having noted the limited usefulness of the majority of studies for answering questions on the effectiveness of Delphi, we look at the technique from a theoritical/ mechanical perspective. That is, by drawing upon ideas/findings from other areas of research, we attempt to discern whether the structure of the Delphi procedure itself might reasonably be expected to function as intended. We conclude that inadequacies in the nature of feedback typically supplied in applications of Delphi tend to ensure that any small gains in the resolution of “process loss” are offset by the removal of any opportunity for group “process gain”. Some solutions to this dilemma are advocated; they are based on an analysis of the process of judgment change within groups and a consideration of factors that increase the validity of statistical/ nominal groups over their constituent individual components.
This paper presents several main aspects of Bayesian reliability demonstration, together with a concise discussion of key contributions to this topic.
A number of approaches (e.g. Delphi) have been developed to structure information exchange in nominal groups to aid judgment under uncertainty. We address the rationale underlying such techniques and point to shortcomings in research on the validity of the techniques per se. We advocate a new direction of research, moving from the holistic appraisal and comparison of techniques, towards consideration of the processes responsible for inducing individual judgment change during such interventions, and the identification of key personal and situational factors that may predict such change.We describe one repeated-measures design in which groups of subjects are asked to forecast economic and political events using Delphi-like procedures that differ according to the presence/ absence and nature of feedback. Characteristics elicited from subjects include measures of ‘confidence’, ‘desirability of forecasts’, and ‘self-rated expertise’. We analyse how individual characteristics on the above dimensions are related to (a) accuracy and (b) willingness to change estimates, for each of the Delphi-like procedures. Results indicate an increase in accuracy over each of the three conditions, including a no-feedback control, although similar outcomes appear to derive from different processes. Measurement of panellists' ‘self-rated expertise’, ‘objective expertise’, ‘confidence’, and ‘desirability of outcomes’ show differential utility for a-priori panellist selection for structured groups. We discuss results in terms of the interaction of technique and panellist specifications. In general, our results support the “theory of errors” as an explanation of the effectiveness of the Delphi technique. Implications of our findings for the creation of judgment-aiding techniques, panellist selection, and future research are discussed.
Previous research findings suggest that group judgment is superior to individual judgment, although groups fail to reach their full potential because of problems associated with the interaction process. Thus, groups perform at a level generally better than the competence of their average members but rarely as well as their most proficient members. The current study explores two methods of group judgment making which have been developed to reduce the discrepancy between potential and actual group performance: social judgment analysis and the Delphi technique. These two methods are compared in a controlled experimental setting with regard to their potential both to significantly reduce group disagreement and to provide accurate judgments. The two methods were found to be equal in the quality of judgments produced. Social judgment analysis, however, was a significantly better method of reducing disagreement than the Delphi technique.
The potential of group (vs. individual) forecasting is analyzed from the perspective of the social psychology of groups. The social decision scheme theory (SDST) is summarized, and several simulations are presented to demonstrate the dependence of group aggregation accuracy upon factors such as group size, the accuracy and distribution of individual forecasts, and shared representations of the forecasting problem. Many advantages and disadvantages of group aggregation are identified and related to four generic methods of group aggregation (statistical aggregation, prediction markets, the Delphi method, and face-to-face discussion). A number of aspects of forecasting problems are identified which should govern whether or not group forecasting can be relied upon, and if so, what aggregation method should be used.
This article described three heuristics that are employed in making judgements under uncertainty: (i) representativeness, which is usually employed when people are asked to judge the probability that an object or event A belongs to class or process B; (ii) availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development; and (iii) adjustment from an anchor, which is usually employed in numerical prediction when a relevant value is available. These heuristics are highly economical and usually effective, but they lead to systematic and predictable errors. A better understanding of these heuristics and of the biases to which they lead could improve judgements and decisions in situations of uncertainty.
A standard approach to the combination of probabilistic opinions involves taking a weighted linear average of the individuals distributions. This paper reviews some of the possible interpretations that have been proposed for these weights in the literature on expert use. Several paradigms for selecting weights are also considered. Special attention is devoted to the Bayesian mechanism used for updating expert weights in the face of new information. An asymptotic result is proved which highlights the importance of choosing the initial weights carefully.
When a group of k individuals is required to make a joint decision, it occasionally happens that there is agreement on a utility function for the problem but that opinions differ on the probabilities of the relevant states of nature. When the latter are indexed by a parameter , to which probability density functions on some measure may be attributed, suppose the k opinions are given by probability density functions . Suppose that D is the set of available decisions d and that the utility of d, when the state of nature is , is . For a probability density function , write . The Group Minimax Rule of Savage [1] would have the group select that d minimising . As Savage remarks ([1], p. 175), this rule is undemocratic in that it depends only on the different distributions for represented in those put forward by the group and not on the number of members of the group supporting each different representative. An alternative rule for choosing d may be stated as follows: "Choose weights and ; construct the pooled density function choose the d, say , maximising ." This rule, which may be called the Opinion Pool, can be made democratic by setting . Where it is reasonable to suppose that there is an actual, operative probability distribution, represented by an `unknown' density function , it is clear that the group is then acting as if were known to be . If were known, it would be possible to calculate and , where is the d maximising and then to use these quantities to assess the effect of adopting the Opinion Pool for any given choice of . It is of general theoretical interest to examine the conditions under which \begin{equation*}\tag{1.1}u\lbrack d_{s\lambda} \mid p_a(\theta)\rbrack \geqq \min_{i = 1, \cdots, k} u\lbrack d_{si} \mid p_a(\theta)\rbrack.\end{equation*} Theorems 2.1 and 3.1 provide different sets of sufficient conditions for (1.1) to hold. Theorem 2.1 requires and places a restriction on (or, equivalently, on and ); Theorem 3.1 puts conditions on D and instead.
Statistics Reference Online This article is ©
- Wiley Statsref
Wiley StatsRef: Statistics Reference Online, © 2014-2016 John Wiley & Sons, Ltd.
This article is © 2016 John Wiley & Sons, Ltd.
Principles of Forecasting: A Handbook for Researchers and Practitioners
- G. Rowe
- G. Wright
The Delphi Method: Techniques and Applications
- K. Brockhoff
Allocating the weights in the linear opinion pool
- C Genest
- K Mcconwy
Genest, C. and McConwy, K. (1990) Allocating the weights in the linear opinion pool. J. Forecast., 9, 53-73.
- Wiley Statsref
Wiley StatsRef: Statistics Reference Online, © 2014-2016 John Wiley & Sons, Ltd.
This article is © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/9781118445112.stat07879