Article

Consensus measurement in Delphi studies Review and implications for future quality assurance

Authors:
  • Steinbeis-Hochschule Berlin
To read the full-text of this research, you can request a copy directly from the author.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... The Delphi method was employed as the primary method for developing a competency-based system because it allows researchers to explore critical phenomena on a given topic based on expert preferences. The Delphi method was used to evaluate and validate the competency analysis build consensus, and prioritize competencies (Heiko, 2012;Holtkamp et al., 2015). ...
... Participants were asked to read the ranking carefully and, if they had a different opinion, to reorder the ranking and provide more insight. We identified the criteria for consensus-building (Heiko, 2012). We calculated the value of Kendall's coefficient in each round (Kendall, 1977). ...
... The number of participants in the round was at least half the number of participants in the previous round (Kendall, 1977). Participants in the Delphi study were selected based on their knowledge and suitability for the Delphi study (Heiko, 2012;Linstone & Turoff, 1975). Comprehensive international representation was ensured through objective sampling based on their expertise. ...
Article
Full-text available
In this study, we looked at the competencies and changes in the competency spectrum required for global start-ups in the digital age. Specifically, we explored intergenerational collaboration as an intervention in which experienced business-people from senior adult groups support young entrepreneurs. We conducted a Delphi study with 20 experts from different disciplines, considering the study context. The results of this study shed light on understanding the necessary competencies of entrepreneurs for intergenerationally supported start-up innovation by providing 27 competencies categorized as follows: intergenerational safety facilitation, cultural awareness, virtues for growth, effectual creativity, technical expertise, responsive teamwork, values-based organization, and sustainable network development. In addition, the study results also reveal the competency priorities and the minimum requirements for each competency group based on the global innovation process and can be used to develop a readiness assessment for start-up entrepreneurs.
... The key features of this method are the anonymity of participants and the controlled feedback [12][13][14]. ...
... Definition of consensus was established before data analyses. It was determined that consensus, consequently to become a recommendation in this guideline, would be achieved if at least 80% of participants reached an agreement (score 7-9) or disagreement (score 1-3) [12][13][14]. A statement was retired if it had a mean vote below 3 or a "low" level of agreement. ...
... Statements whose rate came in the uncertainty score, (4 -6), were revised in view of the comments. The levels of agreement on each statement of recommendation were defined as "high" if after the second round of votes, all votes on a statement fell into the agreement bracket (7-9) [14,15]. ...
Article
Background We aimed to provide a consensus, evidence-based recommendations for the diagnosis, evaluation, and treat-to-target management of children living with FMF. This study was carried out to achieve an expert consensus on a treat-to-target management strategy for FMF using the Delphi technique. The preliminary scientific committee identified a total of 17 key clinical questions according to the Patient/Population, Intervention, Comparison, and Outcomes (PICO) approach. An evidence-based, systematic, literature review was conducted to compile evidence for the benefits and harms associated with JIA treatments. The core leadership team identified researchers and clinicians with expertise in FMF management. Delphi process was implemented (2 rounds) to reach a consensus on the management recommendations of FMF patients. Results Twenty-one expert panel participated in the 2 rounds with a response rate of 100%. A total of 12 recommendations, categorized into 2 sections (4 in the diagnosis section and 8 in management), were obtained. The agreement with the recommendations (rank 7–9) ranged from 85.7 to 100%. The consensus was reached (i.e., ≥75% of respondents strongly agreed or agreed) on all the clinical standards. Algorithms for management have been also developed. Conclusion A wide and representative panel of experts established a consensus regarding the management of FMF. The developed guidelines provide a comprehensive treat-to-target approach to the management of FMF for all healthcare professionals who are involved in its management.
... In the third round -the ranking -we did not strive for consensus, which constitutes a second deviation from Okoli and Pawlowski. Reaching consensus is often regarded as a necessary criterion for finalizing data collection in a Delphi study and this has been criticized before for its inappropriateness and artificiality of results (Dajani et al., 1979;von der Gracht, 2012). We asked our expert panel to rank the extracted adaptation strategies and user parameters for the monitoring in terms of maximal effect for the intervention (adaptation strategies) or clinical relevance (user parameters). ...
... The IQR consists of the middle 50% of the observations and therefore, an IQR of less than 1 means that more than 50% of all rankings or ratings fall within 1 point on the scale. The IQR is frequently used in Delphi studies and it is generally accepted as an objective and rigorous method for assessing consensus (von der Gracht, 2012). For the top five selection, we calculated a ranking score. ...
... Since we did not require a specific level of consensus as a stopping criterion and employed a fixed number of Delphi rounds instead, we limited the impact of consensus-seeking on the time effort required of the participants and the research team. The trade off between time and monetary investments and the extent to which consensus can be achieved in Delphis has been acknowledged (Hasson et al., 2000;von der Gracht, 2012). ...
Article
Full-text available
While much effort has been devoted to the development of mental e-health interventions, the tailoring of these applications to user characteristics and needs is a comparatively novel field of research. The premise of personalizing mental e-health interventions is that personalization increases user motivation and (thereby) mitigates intervention dropout and enhances clinical effectiveness. In this study, we selected user profile parameters for personalizing a mental e-health intervention for older adults who lost their spouse. We conducted a three-round Delphi study involving an international and interdisciplinary expert panel (N = 16) with two objectives. The first aim was to elicit adaptation strategies that can be used to dynamically readjust the intervention to the user's needs. The second aim was to identify a set of meaningful indicators for monitoring the user from within the grief intervention to escalate from self-help to blended care, whenever advisable. This Delphi study used as starting point an evaluated, text-based grief intervention composed of ten modules, including psychoeducation about grief and cognitive-behavioral exercises to support the user in adjusting their lives after bereavement. Every user follows this grief intervention in a linear fashion from beginning to end. The resulting conceptual adaptation model encompasses dynamic adjustments, as well as one-time adjustments performed at the initialization of the service. On the level of the application structure, the adaptations affect when which topic module is presented to the user. The adaptations further provide strategies for adjusting the text-based content of individual intervention modules dependent on user characteristics and for selecting appropriate reactions to user input. Eighteen monitoring parameters were elicited and grouped into four categories: clinical, behavioral/emotional, interactive, and external. Parameters that were perceived as most urgent to attend to for escalation were Suicidality, Self-destructive behavior, Client-initiated escalation, Unresponsiveness and (Complicated) Grief symptoms.
... The measurement of consensus is an important part of the statistical evaluation of Delphi results (Skulmoski et al., 2007;von der Gracht, 2012). Therefore, the median (Rowe et al., 1991), the IQR (Ecken et al., 2011;Schuckmann et al., 2012), and Kendall's W were chosen to evaluate the degree of consensus. ...
... df = 39; subjects = 40; raters = 26 p ≤0.0001), and it was 0.5557567 (ChiSq = 390.14; df = 39; subjects = 40; raters = 18; p ≤0.0001) in Round 3. Kendall's W is a value between 0 and 1, with a tendency towards 1 indicating stronger consensus (Schmidt, 1997;von der Gracht, 2012). The increase of Kendall's W between the two rounds suggests an increase in stability for the participants' evaluations of their projections. ...
... Kendall's W for Round 2 indicates a weak to moderate agreement. It shows a moderate to strong agreementand therefore significantly higher consensusfor Round 3 (Schmidt, 1997;von der Gracht, 2012). Even though overall stability increased between the rounds, experts did not agree on all projections. ...
Article
Robo advisors represent a digital financial advice solution challenging traditional wealth and asset management, investment advice, retirement planning, and tax-loss harvesting. Based on algorithms, big data analysis, machine learning, and other technologies, these services minimize the necessity for human intervention. Based on an international three-stage Delphi study, we provide a plausible forecast of the development of the robo advisor industry, with regards to market development, competition, drivers of growth, customer segments, challenges, services, technologies, and societal change. The results suggest that the financial advice market will experience a further increase in the number of robo advisor services available. Existing and traditional financial advice players will be forced to adjust to the changing environment of the market. Due to low fees and ease of use, robo advisors will be made available to a broad cross section of society, and will cause significant market losses for traditional investment advice companies. Ten years from now, the predominant investment class will remain Exchange Traded Funds (ETFs). Even though degrees of human intervention are expected to vary considering the complexity of advice, automation will increase in significance when it comes to the development of robo advisors.
... al., 2019;Kozak & Iefremova, 2014;Landeta, 2006;Skulmoski et. al., 2007;Rowe & Wright, 2001;von der Gracht, 2012): ...
... • A fourth and last characteristic is statistical aggregation, which implies that a number of statistics are used to determine the degree of consensus among the expert panel. To determine the opinion of the group, often reference is made to statistical indicators of central tendency and dispersion (Hsu & Sandford, 2007;von der Gracht, 2012) which are presented in a numerical or graphical way Schmidt, 1997). ...
... Apart from the percentage of agreement, also measures of central tendency can be used to determine consensus Hsu & Sandford, 2007;Rayens & Hahn, 2000;von der Gracht, 2012). Reference is often made to summary statistics like the mean and the median (Barrios et. ...
Technical Report
The report elaborates on the results of our Delphi study on insider threat mitigation. The goal of the study was to discover potential ‘red flags’ of insider threat incidents (i.e. factors that may point to imminent insider threat incidents) and good practices on insider threat mitigation. The study employed the Delphi technique to iteratively compare and contrast the opinions of insider threat experts. A multidisciplinary panel of 25 international experts in a field related to insider threats (corporate security, counterintelligence, insider threat training, …) completed three rounds of online questionnaires that contained questions on four insider threat mitigation phases, namely prevention, detection, pre-emption and the aftermath of an insider threat incident. Round 1 concerned open-ended, level-setting questions with panelists asked to share their expertise on both red flags and good practices. In round 2, experts were presented with a list of all important red flags/good practices shared by the panel in round 1 in the form of a structured questionnaire whereby they were asked to rate each individual red flag/good practice listed. The questionnaire of round 3, to conclude, provided the experts with a list of high-rated factors, after which they were asked whether they agreed or disagreed with the panel’s decision and to explain their reasoning in case of disagreement. The results of the study provide readers with useful insights on what experts consider to be red flags organizations should be vigilant of, as well as with mitigation measures to better secure organizations against insider threats.
... (5) The expert forecasts are combined to form the final forecasts. This technique has two significant advantages: participant anonymity and regulated feedback [13][14][15]. ...
... In view of the feedback, statements with an uncertainty score of (4-6) were changed. The levels of agreement on each statement of suggestion were considered as "high" if all votes on a statement fell into the agreement bracket following the second round of voting [15,20,21]. ...
Article
Background These updated guidelines aimed to provide appropriate and convenient guidelines for the treatment of various types of juvenile idiopathic arthritis (JIA). Using the Delphi technique, this study was conducted to reach expert consensus on a treat-to-target management strategy for JIA. According to the PICO (patient/population, intervention, comparison, and outcomes) approach, the preliminary scientific committee identified a total of 17 key clinical questions. To assemble evidence on the advantages and dangers associated with JIA treatments, an evidence-based, systematic literature review was conducted. Researchers and clinicians with experience in JIA management were identified by the core leadership team. To establish a consensus on the management suggestions for JIA patients, a Delphi approach (2 rounds) was used. Results An online survey was applied to the expert panel ( n = 27), and 26 of them completed both rounds. At the conclusion of round 2, a total of eighteen (18) recommendation items were gathered, which were divided into four sections to address the four key JIA categories. The percentage of those who agreed with the recommendations (ranks 7–9) ranged from 83.2 to 100% (average 86.8%). The phrasing of all 18 clinical standards identified by the scientific committee was agreed upon (i.e. 75% of respondents strongly agreed or agreed). Algorithms have been proposed for the management of JIA polyarthritis, oligoarthritis, and systemic JIA. Conclusion A wide and representative panel of experts initiated a consensus about the management of JIA. The created guidelines give a complete approach to the management of JIA for all healthcare professionals involved in its management, as well as a means of monitoring and evaluating these guidelines on a regular basis.
... The literature provided limited guidance on achieving to reach an agreement. The percentage of 51% was widely used and accepted as a good measurement to not go for further rounds to gain more consensus (Heiko, 2012;McKenna, 1994). However, the agreement can be high (70% of ratings are strongly agreed or 80% are agreed or more), medium (60% of ratings are strongly agreed or 70% are agreed or more), low (50% of ratings are strongly agreed or 60% are agreed or more), or none (less than 60% of ratings are agreed or strongly agreed) (Hasson et al., 2000;Keeney et al., 2006). ...
... However, the agreement can be high (70% of ratings are strongly agreed or 80% are agreed or more), medium (60% of ratings are strongly agreed or 70% are agreed or more), low (50% of ratings are strongly agreed or 60% are agreed or more), or none (less than 60% of ratings are agreed or strongly agreed) (Hasson et al., 2000;Keeney et al., 2006). Accordingly, the evaluation of agreement, and in turn, the selection of deficiencies of PMBOK ® Guide, and the adaptation proposals for the final adaptive PM-Model have been made using frequencies (high agreement, Medium agreement, low agreement, and non) supported by Mean ± 1.64 and Standard Deviation values (Heiko, 2012) in case of equality of level of agreement. ...
Article
This paper develops an adaptive Project Management Model (PM‐Model) for the International Development and Aid Projects (IDAPs) that International Non‐Governmental Organizations (INGOs) implement by integrating IDAPs' requirements grounded in Total Quality Management (TQM) and Core Humanitarian Standards (CHS). The three‐round Delphi process allows experts' opinions to identify the deficiencies of the PMBOK® Guide to propose an adaptive PM‐Model for the management of IDAPs. Seven experts suggest the required adaptations in Round‐1 and narrow down the transformations in Round‐2. Then, in Round‐3, nine practitioners validate the proposed new model of the final adaptive PM‐Model that entails eight adaptions to the PMBOK® Guide for the interest of IDAPs and vulnerable people reached by these projects. The result contributes to the PM theory by providing an adaptive PM‐Model for the international development industry. This study addresses the practical problems faced by the practitioners. Thus, it promotes people's inclusion and decision‐making, relevancy of provided assistance by IDAPs, accountability, coordination, local capacities and community leadership, monitoring and evaluation, and satisfying donors of IDAPs.
... The second survey ( Table 2) consisted of six items where no consensus was reached (18,19,21,23,29,30) and six new items suggested by the panel during Round 1 (31)(32)(33)(34)(35)(36). Items 18 and 21 reached consensus agreement as "very important". ...
... We extended these methods to the realm of a smart home system for people with heart failure. The Delphi process allowed for data to be collected anonymously, systematically and iteratively which allowed for reasoned expert feedback with less bias from more forthright participants (35). Whilst we recruited a multidisciplinary panel of experts from a range of clinical backgrounds, it was not representative of all clinicians involved in caring for people with heart failure. ...
Article
Full-text available
Background Smart home systems could enhance clinical and self-management of chronic heart failure by supporting health monitoring and remote support, but evidence to guide the design of smart home system functionalities is lacking. Objective To identify consensus-based recommendations for functions of a smart home system that could augment clinical and self-management for people living with chronic heart failure in the community. Methods Healthcare professionals caring for people living with chronic heart failure participated in a two-round modified Delphi survey and a consensus workshop. Thirty survey items spanning eight chronic health failure categories were derived from international guidelines for the management of heart failure. In survey Round 1, participants rated the importance of all items using a 9-point Liket scale and suggested new functions to support people with chronic heart failure in their homes using a smart home system. The Likert scale scores ranged from 0 (not important) to 9 (very important) and scores were categorized into three groups: 1–3 = not important, 4–6 = important, and 7–9 = very important. Consensus agreement was defined a priori as ≥70% of respondents rating a score of ≥7 and ≤ 15% rating a score ≤ 3. In survey Round 2, panel members re-rated items where consensus was not reached, and rated the new items proposed in earlier round. Panel members were invited to an online consensus workshop to discuss items that had not reached consensus after Round 2 and agree on a set of recommendations for a smart home system. Results In Round 1, 15 experts agreed 24/30 items were “very important”, and suggested six new items. In Round 2, experts agreed 2/6 original items and 6/6 new items were “very important”. During the consensus workshop, experts endorsed 2/4 remaining items. Finally, the expert panel recommended 34 items as “very important” for a smart home system including, healthy eating, body weight and fluid intake, physical activity and sedentary behavior, heart failure symptoms, tobacco cessation and alcohol reduction, medication adherence, physiological monitoring, interaction with healthcare professionals, and mental health among others. Conclusion A panel of healthcare professional experts recommended 34-item core functions in smart home systems designed to support people with chronic heart failure for self-management and clinical support. Results of this study will help researchers to co-design and protyping solutions with consumers and healthcare providers to achieve these core functions to improve self-management and clinical outcomes in people with chronic heart failure.
... 34 out of the 47 professionals from the 1st round (72.3% of the initial sample) participated in the second questionnaire. Both rounds had a good number of participants in agreement with last review articles about the Delphi technique (De Loë et al., 2016;Heiko, 2012;Sossa et al., 2019). We share the idea that when searching for opinions that represent the group, and while avoiding iterations, there is no need for the number of participants to be too small nor too large (Dalkey, 1969;De Loë, 1995;Garrod and Fyall, 2005;Moeller and Shafer, 1987). ...
... indicating short term -redefined as long term guidance. These intervals, although with some redefinitions, are in line with guidelines emanating from other studiesregarding the level of importance to be attributed to various items (Heiko, 2012;Perveen et al., 2017). ...
Article
Full-text available
The narrative of sustainable tourism transition in a context of adaptation to climate change is very relevant internationally. The availability and sharing of knowledge and information is a basic requirement for the successful planning of the tourism sector regarding this phenomenon. Planning adaptation in the urban tourism sector is widely regarded as a collectively-based process. However, collaborative planning is far from being the standard. This study reports the results of a Modified Delphi Approach (MDA) among experts about the future of urban tourism in a context of adaptation to climate change in Porto Metropolitan Area (Portugal), considering the outdoor thermal conditions perspective. Using an expert panel, the study gathered their opinions to analyze the degrees of responsibility of the main sectorial entities at different territorial levels, the conditions of action in the transformation agenda and the measures to be implemented in the adaptation and mitigation process-according to priority and time horizon. Two rounds were carried out to apply the methodology between January and April 2021. The first questionnaire had the participation of 47 professionals. 34 out of the 47 professionals of the 1st round participated in the second questionnaire. The evidence from different stakeholders demonstrates that there is an ambiguous process of understanding the problem, information needs, and a weak interaction between actors-resources-tasks. The effectiveness and efficiency of collaborative planning and outlined goals by 2050 for adaptation of urban tourism sector to climate change can be hampered. Experts consider the creation of structural (tangible) measures to be fundamental. Among other results, it was found that most participants consider that the intervention is dependent on the guidelines issued by the government and municipal councils when it comes to defining a proposal for adapting the urban tourism sector to climate change. Despite this, the options for more sustainable practices must be based on three axes: (i) solutions based on the energy sector in the hotel industry (e.g., energy certification, prioritization of the use of renewable energy); (ii) improvement and expansion of green infrastructure for tourist enjoyment [e.g., creation of green areas (small additional pockets), namely in the center of Porto; and pedestrianization of central areas of the city] and (iii) network participation through the collaboration of various stakeholders with relevance in tourism and urban planning.
... Taking into account the objective of this paper, authors participated in a Delphi session to select most relevant UN SDGs associated with the results of the bibliometric analysis. This method is used to encourage experts in achieving consensus on a certain topic (Heiko, 2012). The session has been organized online and authors of this paper had the possibility to select three UN SDGs. ...
Article
This paper provides an overview of how food production influences climate change and also illustrates the impact of climate change on food production. To perform such an overview, the (inter)link between different parts of the food supply chain continuum (agriculture production, livestock farming, food processing, food transport and storing, retail food, and disposal of food waste) and climate change has been investigated through a bibliometric analysis. Besides UN Sustainable Development Goal (SDG) 13, associated with climate change, other SDGs that are associated with this overview are goals #1, #2, #3, #6, #7, #12, and #15. Based on the evidence gathered, the paper provides some recommendations that may assist in efforts to reduce the climate-related impacts of food production.
... 25 52 Stability of response presents the consistency of responses within the study period and between respondent group stability, which is considered a necessary precondition for determining the level of agreement or if consensus was achieved. [53][54][55] Different measures of dispersion (eg., median, IQR) and statistical approaches (eg., descriptive, inferential) can be used 44 55 to measure stability, which can be calculated between rounds (traditional Delphi) or at the end of the study (RT-Delphi). 41 44 We will use the coefficient of quartile variation (CQV) as a descriptive measure of response stability. ...
... In this study, the content validity of the PsyCET-A was examined by asking experts in psychological assessment to rate the relevance and clarity of the items on the tool using the Delphi technique. The Delphi technique is a method of seeking information from experts in a particular field of interest and involves an anonymous survey process with multiple iterations that continues until consensus is reached (Hsu & Sandford, 2007;von der Gracht, 2012). This validation study focused on use of the PsyCET-A by neuropsychology supervisors given the emphasis on assessment skills in this area of specialty. ...
... These results may be a step in initiating large-scale international dialogue on the challenges involving diagnostics and management strategies. Future studies aiming for consensus could implement consensus methods such as the Delphi method, 43 which involves iteratively surveying relevant stakeholders on a specific topic until (predefined) consensus is reached. This could be relevant for defining the minimum diagnostic criteria for CFM, but also for continued dialogue on globally acceptable management strategies. ...
Article
Full-text available
To (1) appraise current international classification and clinical management strategies for craniofacial microsomia (CFM) and microtia, and (2) to assess agreement with the European Reference Network “European Guideline Craniofacial Microsomia” recommendations on screening and monitoring. This was a cross-sectional online survey study. The survey consisted of 44 questions on demographics, diagnostics and classification, obstructive sleep apnea, feeding difficulties, speech and language development, hearing, ocular abnormalities, visual development, orthodontic screening, genetic counselling, psychological wellbeing, and extracraniofacial anomalies. Respondents were participants of 3 international cleft and craniofacial conferences, members of the American Cleft Palate and Craniofacial Association and members of the International Society for Auricular Reconstruction. Respondents were requested to complete 1 questionnaire per multidisciplinary team. Fifty-seven responses were received from 30 countries (response rate ∼3%).The International Consortium for Health Outcomes Measurement diagnostic criteria were used by 86% of respondents, though 65% considered isolated microtia a mild form of CFM. The Orbit, Mandible, Ear, Facial Nerve and Soft Tissue classification system was used by 74% of respondents. Agreement with standardized screening and monitoring recommendations was between 61% and 97%. A majority of respondents agreed with screening for extracraniofacial anomalies (63%-68%) and with genetic counselling (81%). This survey did not reveal consistent agreement on the diagnostic criteria for CFM. Respondents mostly supported management recommendations, but frequently disagreed with the standardization of care. Future studies could focus on working towards international consensus on diagnostic criteria, and exploring internationally feasible management strategies.
... Once completed, the median and interquartile range (IQR) for the continuous variables were calculated as recommended by Jones and Hunter (49). An IQR of < 1 for a 5-point Likert scale is suggested as an acceptable indicator of consensus (50)(51)(52). In this study, statements that reached expert consensus (i.e., median 5 and IQR < 1) were deemed conclusive and were not reevaluated in subsequent rounds. ...
Article
Full-text available
Background Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively. Methods In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study. Results A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning. Conclusion Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.
... 25 52 Stability of response presents the consistency of responses within the study period and between respondent group stability, which is considered a necessary precondition for determining the level of agreement or if consensus was achieved. [53][54][55] Different measures of dispersion (eg., median, IQR) and statistical approaches (eg., descriptive, inferential) can be used 44 55 to measure stability, which can be calculated between rounds (traditional Delphi) or at the end of the study (RT-Delphi). 41 44 We will use the coefficient of quartile variation (CQV) as a descriptive measure of response stability. ...
Article
Full-text available
Introduction: An online interactive repository of available medication adherence technologies may facilitate their selection and adoption by different stakeholders. Developing a repository is among the main objectives of the European Network to Advance Best practices and technoLogy on medication adherencE (ENABLE) COST Action (CA19132). However, meeting the needs of diverse stakeholders requires careful consideration of the repository structure. Methods and analysis: A real-time online Delphi study by stakeholders from 39 countries with research, practice, policy, patient representation and technology development backgrounds will be conducted. Eleven ENABLE members from 9 European countries formed an interdisciplinary steering committee to develop the repository structure, prepare study protocol and perform it. Definitions of medication adherence technologies and their attributes were developed iteratively through literature review, discussions within the steering committee and ENABLE Action members, following ontology development recommendations. Three domains (product and provider information (D1), medication adherence descriptors (D2) and evaluation and implementation (D3)) branching in 13 attribute groups are proposed: product and provider information, target use scenarios, target health conditions, medication regimen, medication adherence management components, monitoring/measurement methods and targets, intervention modes of delivery, target behaviour determinants, behaviour change techniques, intervention providers, intervention settings, quality indicators and implementation indicators. Stakeholders will evaluate the proposed definition and attributes' relevance, clarity and completeness and have multiple opportunities to reconsider their evaluations based on aggregated feedback in real-time. Data collection will stop when the predetermined response rate will be achieved. We will quantify agreement and perform analyses of process indicators on the whole sample and per stakeholder group. Ethics and dissemination: Ethical approval for the COST ENABLE activities was granted by the Malaga Regional Research Ethics Committee. The Delphi protocol was considered compliant regarding data protection and security by the Data Protection Officer from University of Basel. Findings from the Delphi study will form the basis for the ENABLE repository structure and related activities.
... This study will use a survey to achieve consensus across an expert group of general practice and PHN staff. The Delphi technique has been selected due to its flexibility and anonymity provided to participants [21,22]. The survey will consist of three rounds to obtain opinions on a suite of indicators and measures previously developed by the research team to reach consensus on a core set of relevant and feasible high-quality performance indicators [11]. ...
Article
Full-text available
Background: High-quality general practice has been demonstrated to provide cost-effective, equitable health care and improve health outcomes. Yet there is currently not a set of agreed comprehensive indicators in Australia. We have developed 79 evidence-based indicators and their corresponding 129 measures of high-quality general practice. This study aims to achieve consensus on relevant and feasible indicators and measures for the Australian context. Methods: This Delphi consensus study, approved by WSU Human Research Ethics Committee, consists of three rounds of online survey with general practice experts including general practitioners, practice nurses and primary health network staff. The identified indicators and measures are grouped under an attribute framework aligned with the Quadruple Aim, and further grouped under structures, processes and outcomes according to the Donabedian framework. Participants will rate each indicator and measure for relevance and feasibility, and provide comments and recommendations of additional indicators or measures. In the last round, participants will also be asked their views on the implementation of a quality indicator tool. Each indicator and measure will require ≥70% agreement in both relevance and feasibility to achieve consensus. Aggregated ratings will be statistically analysed for response rates, level of agreement, medians, interquartile ranges and group rankings. Qualitative responses will be analysed thematically using a mixed inductive and deductive approach. Discussion: This protocol will add to the current knowledge of the translation of performance guidelines into quality practice across complex clinical settings and in a variety of different contexts in Australian general practice. The Delphi technique is appropriate to develop consensus between the diverse experts because of its ability to offer anonymity to other participants and minimise bias. Findings will contribute to the design of an assessment tool of high-quality general practice that would enable future primary health care reforms in Australia.
... A definição dos intervalos de previsibilidade [280] das ações baseou-se nas seguintes diretrizes: quando ≥ 65,0% dos especialistas e agentes regionais e locais consideraram ser necessário definir as medidas a curto prazo estas foram orientadas para a execução com a maior brevidade possível (curto prazo -até 2 anos); se menos de 65,0% dos especialistas e agentes regionais consideraram as medidas a curto prazo estas foram perspetivadas para execução a médio prazo ou nos casos com ≥ 25,0% indicando longo prazo e < 50,0% indicando curto prazo, foram redefinidas como orientação de longo prazo. Esses intervalos, embora com algumas readaptações neste estudo, estão de acordo com orientações emanadas de outros estudos com orientações quanto ao nível de importância a ser atribuído aos diversos itens (Heiko, 2012;Perveen et al., 2017). ...
Thesis
Full-text available
In recent years, those who create strategies and policies for urban tourist destinations have been increasingly concerned with the greater or lesser capacity to enjoy public space. Furthermore, the growth of urban areas on a global scale has caused significant changes in the (micro)climate, due to the increase in impermeable surfaces, the anthropogenic heat generated by human activities and the change in air circulation. Taking into account the increasing demands of tourists and residents and the need to improve cities in the face of climate change, the option is to design new measures and action solutions. However, the lack of quality of the input data or their (total) absence, as well as their low spatial resolution, are common. The inadequacy of structures for sharing information is also noted, which significantly limits planning and adaptation actions. This investigation aims to identify the main methods of analysis to monitor the current ability to enjoy tourism based on the integration of objective and subjective domains; and contribute to the definition of action plans which seek to mitigate and adapt the tourism sector to climate change, in the medium and long-term. To assess the validity of these assumptions, the Porto Metropolitan Area, in general, and the municipality of Porto, in particular, were used as case studies. In this investigation, different methods of information and units of analysis were combined, based on a meso approach and local scale for: (i) the identification of critical areas, in an office analysis based essentially on Big Data (i.e., Flickr photographs, AirBnB accommodation and MODIS and LANDSAT satellite imagery); (ii) the assessment of the comfort level for enjoyment in critical areas with high tourist potential through field data collection; and (iii) the identification of prioritization actions and measures to maintain tourism attractiveness in view of climate change, in the medium and long-term. This research highlights the need for more detailed information, the weak interaction between stakeholders and the limitation of resources. Thus, considering that Porto is a destination with a good climate for tourism, and committed to mitigating the effects of climate change, the proposed methodological triangulation allows to outline some measures with predictable action in the short, medium and long-term. Finally, this study aims to make some contributions at national and international level, with the likelihood of the methodological approach adopted to be replicated in other geographical areas, taking into account the particularities of each territory under analysis.
... A total of 34 of the 47 professionals in the first round participated in the second questionnaire (72.3% of the initial sample- Table 2). According to the latest review articles in the field, the number of participants in both questionnaires was quite acceptable [64][65][66]. The first questionnaire sought to identify the relevance of a package of measures related to climate change. ...
Article
Full-text available
When climate change became a global concern in the 1980s, mitigation was considered the best strategy to address all challenges. For a long time, it was thought possible to stabilize atmospheric concentrations of greenhouse gases (GHGs), which, according to many experts, brought on an unfit adaptation. There are international agreements designed to significantly reduce CO2 emissions and achieve carbon neutrality by 2050, but the policy measures taken so far are insufficient to achieve this goal. In addition, the crisis caused by the COVID-19 pandemic highlighted the relevance of placing this issue at the core of international policies and the need for bottom-up measures and options. The purpose of this paper is to explore how collaborative planning can contribute to adapting the urban tourism sector to climate change in the Porto Metropolitan Area (PMA), located in the northern region of mainland Portugal. In this investigation, we used mixed methods based on the following: (1) the discussion of urban tourism's adaptation planning to climate change with undergraduate students; (2) the application of a modified Delphi questionnaire survey, to 47 international researchers and technicians in the first round and 35 international researchers and technicians in the second round, about the predictability of the adaptation measures; and (3) a theoretical-practical workshop aimed to discuss the main action intentions and ways of adaptation in the short and medium term. All empirical data were collected during the year of 2021. This research highlights the need for more detailed information, the weak interaction between stakeholders and the limitation of resources. Our research identifies the main impacts and local vulnerabilities and determines priorities for adaptation and implementation of actions, aimed at mitigating the effects of climate change and maintaining tourism attractiveness in urban areas. In addition, this investigation allowed the definition of a research agenda, which seeks to guide the area of tourism climatology regarding the new challenges imposed by the COVID-19 pandemic.
... Delphi is a practical technique that assists scholars with advantages for forecasting and predicting challenges and dynamics associated with technology applications (Heiko, 2012;Fritschy and Spinler, 2019;Kattirtzi and Winskel, 2020). The valuableness and usefulness of the Delphi method are raised in its several unique features (Rowe et al., 1999;Rezaei et al., 2021c), which make it appropriate for this study. ...
Article
Purpose Despite the undoubted role of knowledge in the small family business (FB) in the restaurant and fast-food industry, there are some main challenges in the knowledge-sharing (KS) orientation for promoting their business. This study tries to recognise and explore the drivers influencing these enterprises' KS processes. Design/methodology/approach The authors applied combined exploratory and confirmatory analysis in two studies. In the first study, by the Delphi method, the 22 experts consisting of managers, FB owners and academic professors explore and forecast significant items and drivers. The second study is a confirmatory factor analysis (CFA) for validating the first study's indicators using the survey collected amongst 218 restaurants and fast foods employees. Findings The findings indicate 23 indicators in three main drivers of individual, organisational and technological, are critical for KS in small FBs in the investigated industry. Originality/value This research supports the understanding of knowledge management and the FB and contributes to recognising the influential factors for KS amongst small and medium-sized family businesses in the food-related industry. Also, by identifying and ranking the most significant factors, this research will help entrepreneurs facilitate FB entrepreneurship. Finally, the results provide practical implications for current and future KM and FB decision-makers.
... There are no defined criteria on setting cut-off consensus levels in Delphi studies [19]. Consensus levels are defined as a percentage higher than the average percentage of majority opinion [20], and many researchers have used different levels of agreement to achieve consensus. ...
Article
Full-text available
Background: Emergency general surgery (EGS) patients account for more than one-third of admissions to hospitals in the National Health Service (NHS) in England. The associated mortality of these patients has been quoted as approximately eight times higher than that of elective surgical admissions. This study used a modified Delphi approach to identify research priorities in EGS. The aim was to establish a research agenda using a formal consensus-based approach in an effort to identify questions relevant to EGS that could ultimately guide research to improve outcomes for this cohort. Methods: Three rounds were conducted using an electronic questionnaire and involved health care professionals, research personnel, patients and their relatives. In the first round, stakeholders were invited to submit clinical research questions that they felt were priorities for future research. In rounds two and three, participants were asked to score individual questions in order of priority using a 5-point Likert scale. Between rounds, an expert panel analysed results before forwarding questions to subsequent rounds. Results: Ninety-two EGS research questions were proposed in Phase 1. Following the first round of prioritisation, forty-seven questions progressed to the final phase. A final list of seventeen research questions were identified from the final round of prioritisation, categorised as condition-specific questions of high interest within general EGS, emergency colorectal surgery, non-technical and health services research. A broad range of research questions were identified including questions on peri-operative strategies, EGS outcomes in older patients, as well as non-technical and technical influences on EGS outcomes. Conclusions: Our study provides a consensus delivered framework that should determine the research agenda for future EGS projects. It may also assist setting priorities for research funding and multi-centre collaborative strategies within the academic clinical interest of EGS.
... The survey was distributed as a spreadsheet document via e-mail accompanied by a description of the Delphi exercise (Table 1). Anonymity amongst experts was carefully maintained throughout, as it is a key part of the Delphi methodology [23][24][25], contributing to (i) reduced risk of participant unwillingness to abandon publicly expressed opinions, (ii) protection from group pressure in cases of changing a previously expressed opinion, and (iii) elimination of the "halo effect" (i.e., participants favoring recommendations by dominant/prominent individuals) [26,29,30]. ...
Article
In a series of three companion papers published in this Journal, we identify and validate the available thermal stress indicators (TSIs). In this second paper of the series, we identified the criteria to consider when adopting a TSI to protect individuals who work in the heat, and we weighed their relative importance using a Delphi exercise with 20 experts. Two Delphi iterations were adequate to reach consensus within the expert panel (Cronbach’s α = 0.86) for a set of 17 criteria with varying weights that should be considered when adopting a TSI to protect individuals who work in the heat. These criteria considered physiological parameters such as core/skin/mean body temperature, heart rate, and hydration status, as well as practicality, cost effectiveness, and health guidance issues. The 17 criteria were distributed across three occupational health-andsafety pillars: (i) contribution to improving occupational health (55% of total importance), (ii) mitigation of worker physiological strain (35.5% of total importance), and (iii) cost-effectiveness (9.5% of total importance). Three criteria [(i) relationship of a TSI with core temperature, (ii) having categories indicating the level of heat stress experienced by workers, and (iii) using its heat stress categories to provide recommendations for occupational safety and health] were considered significantly more important when selecting a TSI for protecting individuals who work in the heat, accumulating 37.2 percentage points. These 17 criteria allow the validation and comparison of TSIs that presently exist as well as those that may be developed in the coming years.
... Participants were then encouraged to revise their earlier answers in light of the replies of other members. Questions were repeated until a clear majority (> 70% of agreement) was reached [9]. Each Panel member (with the exclusion of EHMA representatives) had the right to vote. ...
... The increase in severity and frequency of climate change-related extreme weather events, such as heatwaves, droughts and floods, can lead to multiple threats to physical and mental health, including loss of livelihoods, food insecurity, water scarcity and the spread of infectious diseases (Romanello et al., 2021). Climate change can also intensify conflicts due to resource shortages and exacerbation of existing tensions, which can drive displacement and force communities to migrate (UNHCR, 2021). In settings already experiencing humanitarian crises, communities are often facing multiple and overlapping risks to health and wellbeing, with additional climate shocks further eroding people's resilience and increasing humanitarian needs (OCHA, 2019). ...
Article
Priorities for Mental Health and Psychosocial Support Intervention Research in the Context of the Climate Crisis: A Modified Delphi Study
Article
The transportation industry is in the midst of a revolution with technologies, such as shared vehicles, drones, and autonomous vehicles, that are poised to reshape the way we move. Yet, the multitude of technologies present a difficulty in prioritizing which technologies should be invested in. While focusing on Klang Valley's transportation system, this research proposes the REsilience Brittleness and Emerging Technologies (REBET) framework , which aims to identify the transportation technologies with the highest potential of strengthening the system's resilience. We used a Dephi technique to identify the Sources of Brittleness (SBs) in the system and technologies with the highest relevance to the Malaysian setting. Using multiple linear regression, we then derived a relationship between the two aspects. The framework defines the relative resilience of the technologies according to their forecasted ability to eliminate system brittleness. The results ranked 23 technologies, with the topmost recommendations being ITS, Big Data, and Smart Buses. We highlight REBET's robustness as a global decision-making tool for infrastructure managers, researchers, and policymakers to identify ideal technologies for their transportation systems.
Article
Full-text available
Purpose: Platelet transfusions (PT) are commonly used as prophylaxis in patients with chronic liver disease (CLD) and severe thrombocytopenia (TCP) before invasive procedures, in order to reduce risk of bleeding. The aim of this cost analysis was to generate a comprehensive estimate of costs of platelet transfusions in Italy, focusing on patients with severe TCP due to CLD undergoing an elective procedure. Methods: The research was conducted in different phases: 1) assessment of a pre-specified framework for the identification of processes related to PT; 2) estimation of resource consumption through Delphi technique and collection of unit costs through literature; 3) development of a cost analysis to estimate the overall average costs per PT, focusing on a representative patient with CLD and severe TCP. Robustness of results was tested in a sensitivity analysis. Results: Despite the lack of some cost components estimation and uncertainty related to event probability, the analysis showed a total cost of 5297 € for each PT in patients with CLD and severe TCP. The total cost was largely driven by direct costs (4863 €) associated with platelet collection, transfusion, and management of refractoriness, which accounted for 92% of total. Conclusion: In an environment of limited resources, it is crucial for the healthcare service to have accurate and inclusive information on transfusion costs, incorporating not only the cost of blood products but also those related to collection and management. The analysis showed that platelet collection and administration costs add substantially to the cost of platelet products themselves. As expected, the highest cost was the transfusion process itself (44% of total), followed by refractoriness (43% of total). Since limited literature exists concerning these cost estimates, this analysis represents a step forward in understanding the economic burden of patients with CLD and severe TCP scheduled to undergo an invasive procedure.
Article
Methodological bias can directly affect the interpretation of research data. Studies reporting on excavated skeletons represent a valuable source of information in medicine, dentistry, archaeology and anthropology, and forensic sciences. However, these studies represent a specific setting with their own methodology, for which no quality assessment tool is available. To develop a critical appraisal tool to assess the methodological quality of studies reporting on archaeologically‐excavated human skeletons. An international Delphi study was conducted to support item generation and ensure content validity for a new tool. Experts from the following domains were consulted: dentistry, forensic sciences, archaeology and anthropology, general medicine, epidemiology, and statistics. Participants judged the relevance and comprehensiveness of items retrieved from the literature. Consensus was predefined as 75% agreement between experts, and achieved within two Delphi rounds. Results: 44 and 32 participants completed the first and second Delphi round respectively, achieving consensus on seventeen items. This research provides the first evidence‐based tool for the methodological assessment of studies reporting on archaeologically‐excavated skeletons. Clinicians and researchers can use this tool for critical appraisal of studies or when performing systematic reviews. Future research will focus on psychometric testing of the newly developed tool.
Article
Financial innovation and technological advances are growing at a pace unrivalled by any other period in history. However, as more stakeholders enter these markets, criminals are exploiting their inadvertent security deficiencies to launder illicit funds or finance terrorism. This three-round policy Delphi study involved consultations with 52 experts from different industries and countries to understand future risk-prone technological developments, possible prevention measures and relevant stakeholders. Results highlight a range of money laundering and terrorist financing risks being enabled by advances in distributed ledger technologies (predominantly through cryptocurrencies), new payment methods and financial technology (FinTech). These threats include privacy-enhanced cryptoassets, transaction laundering, e-currencies and digital-only financial services. Findings also suggest that detection-based countermeasures (currently the primary preventative approach) can be coupled with more diverse countermeasures to increase effectiveness. However, the unique circumstances and constraints specific to different stakeholders will affect the nature, utility, and extent to which they can implement certain countermeasures. As such, a ‘one-size-fits-all’ approach to prevention is undesirable. Drawing on expert insight from the study, we propose a framework and a 3-point standard of implementation to motivate cost-effective, user-friendly, and innovation-friendly measures to improve suspicious activity detection and futureproof technologies before their criminal exploitation becomes mainstream.
Article
The current life cycle assessment (LCA) methodology used in structural engineering does not consider hazards like fires within its scope. Various efforts have been made to identify research gaps that need to be filled in order to accomplish a Fire-LCA. This study presented research gaps from literature to experts to determine the research gap's impact, desirability, and probability of occurrence by the year 2030. The study provides insight into what researchers should prioritize to accomplish the proposed Fire-LCA methodology. In each round the experts were asked to evaluate and comment on the impact, desirability, and probability of occurrence by the year 2030 of each identified research gap. The results of the Delphi questionnaire identified high priority research gaps for the implementation of the proposed Fire-LCA methodology. The most prioritized research gap was statistical data that would describe the number and severity of fires, fire origin, and extent of fire spread in different types of building. The results of this study can guide researchers and funding agencies to prioritize research that will fill key gaps in knowledge that prevent the implementation of a Fire-LCA methodology for building structures.
Article
Background Persuasive design principles are increasingly employed in mHealth apps for motivating users and promoting healthy behaviours among individuals. However, how the persuasive design principles are perceived by the mHealth app users remains unclear. Aim To develop and validate the content validity of an instrument designed to measure the user’s perceptions of the persuasive design principles assimilated in a breastfeeding mHealth app. Methods A critical review of the literature guided the development of the research instrument. The developed instrument was refined and validated through a two-round modified Delphi process. Ten experts drawn from academic and clinical settings evaluated the items through two content validity indicators, relevance, and clarity and provided narrative feedback. The content validity of the instrument was determined by calculating the Content Validity Index (CVI). Content validity indicators at the scale-level (S-CVI) and item-level (I-CVI) were calculated. Results The results demonstrated high content validity index of individual items in the instrument. All items in the instrument reached an excellent I-CVI ≥ 0.78 for both relevance and clarity except one item. The overall content validity index of the instrument using the average approach was moderate to high (S-CVI/Ave was 0.89 and 0.92 for relevance and clarity, respectively). The developed research instrument is composed of 5 constructs representing 24 items: (1) primary task support, (2) dialogue support, (3) system credibility support, (4) social support and (5) perceived persuasiveness. Conclusions Using a modified Delphi approach, the development and validation of the research instrument demonstrated moderate to high content validity for measuring users’ perceptions of the persuasive design principles assimilated in a breastfeeding mHealth app. The evidence from this study supports that the research instrument is valid, relevant, and clear.
Article
Introduction New therapies, management approaches, and evidence regarding the management of gout have become available over the past years. This triggered the need for an updated recommendation for gout management. Through an up-to-date consensus evidence-based clinical practice guideline for the management of gout including recommendations for management of gout flares, optimum usage of urate lowering therapy for chronic gout, as well as patient education and lifestyle guidance. A wide systematic literature review was performed, and evidence-based recommendations were extrapolated, based on 16-key questions identified according to population, intervention, comparator, and outcomes (PICO) approach. These were evaluated by a panel consisted of 17 rheumatology experts via online surveys over a 2-round Delphi process. The purpose of this study is to offer an updated, consensus-evidence-based, and in the meantime patient-focused, expert recommendations for the treat-to-target approach of gout management. Results Results revealed that after round 2 ended, a total of 30-recommendation items, categorized into 10 domains, were obtained. Agreement with the recommendations (rank 7–9) ranged from 90 to 100%. Consensus was reached (i.e., ≥ 75% of respondents strongly agreed or agreed) on the wording, the grade of recommendation, and level of evidence of all the 30 clinical standards identified by the scientific committee. Conclusions This guideline provides updated evidence-based recommendations for the prevention and treatment of acute as well as chronic gout. This guideline provides an approach for physicians and patients making decisions on the management of gout. It will also facilitate improvement and uniformity of care.
Article
Full-text available
Aim: To define the competencies of nurse anaesthetists in the hospitals of Catalonia on the basis of their clinical practice through a consensus-building process. Design: We used the Delphi method to determine consensus among a group of 16 nurse anaesthetists. Methods: Between February and June 2020, we administered a questionnaire of 142 questions distributed among seven domains: expert, communicator, collaborator , manager, health advocate, scholar and professional. Two rounds were conducted. Results: In round 1, 18 competencies were discarded and nine had inconclusive results. Eighteen competencies were proposed by participants. The nine competencies with inconclusive results and the 18 newly proposed competencies were considered in round 2. In round 2, three of these 27 competencies tested were discarded, and consensus was reached on the other 24. Conclusion: Health education and the empowerment of the patient and family are fundamental pillars in the practice of nurse anaesthetists in Spain, especially in pre-anaesthetic assessment and pain care. These areas of practice can help define competencies in other countries where the profession of nurse anaesthetist is less developed. Impact: The lack of regulation of the competencies of nurse anaesthetists leads to great variability in training and practice. The results can help in their standardization in Spain and in other countries lacking regulation. Our approach can also help policy-makers and hospital administrators in health systems that are undergoing the process of regulation. The regulation of the competencies of nurse anaesthetists will allow them to contribute their expertise to the health-illness continuum, increasing safety and improving the quality of care. K E Y W O R D S advance practice nursing, competencies, nurse anaesthetists
Article
Background Because of the subjective character of symptoms, absence of a diagnostic test, modest response to treatments, and, at times, patient reports of important functional disability, fibromyalgia remains a challenge for the treating health care professionals in the standard clinical practice. The aim of this study was to develop an up-to-date consensus and evidence-based clinical practice guidelines for a treat-to-target management of fibromyalgia. Fifteen key clinical questions were identified by a scientific committee according to the Patient/Population, Intervention, Comparison, and Outcomes (PICO) approach. A literature review team performed a systematic review to summarize the evidence advocating the benefits and harms of available pharmacologic and nonpharmacologic therapies for fibromyalgia. Subsequently, recommendations were formulated. The level of evidence was determined for each section using the Oxford Centre for Evidence-based Medicine (CEBM) system. A 3-round Delphi process was conducted with 16 experts. All rounds were conducted online. A consensus was achieved on the direction and the strength of the recommendations. Results An online questionnaire was sent to an expert panel who participated in the three rounds (response rate 100%). At the end of round 3, a total of fifteen recommendation items, categorized into 10 sections to address the main fibromyalgia categories, were obtained. Agreement with the recommendations (ranks 7–9) ranged from 85 to 100%. Consensus was reached (i.e., ≥ 80% of respondents strongly agreed or agreed) on the wording of all the 15 clinical standards identified by the scientific committee. An algorithm for the management of fibromyalgia has been suggested. Conclusions These recommendations provide an updated consensus on both the non-pharmacological and the pharmacological treatments of fibromyalgia. The provided strategies to reach optimal treat-to-target outcomes in common clinical scenarios are based on a combination of evidence and expert opinions. Best treatment decisions should be tailored to each individual patient situation.
Article
The environmental performance of various aspects of animal origin food supply chains has been the focus of research in recent years, and has provided useful information. However, there were no studies that covered the entire egg supply chain from different perspectives. The aim of this study was to analyze the majority of environmental impacts in the table egg supply chain comprising of three subsystems: farms, retail outlets and households, with quantification of each individual subsystem and the entire supply chain. All data were gathered from 30 farms, 50 retail stores and 300 households in Serbia. In parallel, the perception and ranking of environmental impacts along the supply chain were also evaluated. Finally, the quality function deployment for the environment was used to determine the degree of correlation between the set of environmental requirements and the identified environmental impacts. Results revealed that the greatest environmental impacts come from the production of feed for laying hens and the use of natural resources, and they contribute the most to the pollution of each individual environmental indicator. Additionally, the results show the differences in the environmental impacts of each individual subsystem and identify opportunities to mitigate them through the optimization of animal feed, energy consumption and household food waste management. The overall perspective of the egg supply chain points to climate change effects as the most important. The differences in the perceptions of environmental impacts along the entire egg supply chain suggest the need for promotion of mitigation strategies to all stakeholders that would encourage them to achieve sustainable development goals.
Article
Rapid technological progress and digitalization have considerably changed the role of technology in sports in the past two decades. As the human limits of performance have been reached in many disciplines, reaching future limits will increasingly depend on technology. While this represents progress in how athletes train and compete, similar developments await sports managers in the way they lead sports organizations and sports consumers in the way they consume and engage with sports. Using the SportsTech Matrix (i.e., a framework to capture how different types of technologies provide solutions to different user groups in sports), we examine how technology will impact sports in the future. We present a Delphi-based prospective study with quantitative and qualitative assessments from 92 subject matter experts for six future projections and 35 non-Delphi prospective survey items. We find that, by 2030, technology will significantly impact all three user groups in sports: athletes, consumers, and managers. For athletes, experts anticipate technology to play a major role for sporting performance improvements. For consumers, the consumption of sports content will continue to change significantly. For management, new types of manager profiles in terms of backgrounds and skill sets would be desirable. We discuss two possible future scenarios: (1) a probable future and (2) a game changer. Our findings should provide relevant insights for decision-makers and other stakeholders in sports and raise promising directions for future research.
Article
Full-text available
Aim To define the competencies of nurse anaesthetists in the hospitals of Catalonia on the basis of their clinical practice through a consensus‐building process. Design We used the Delphi method to determine consensus among a group of 16 nurse anaesthetists. Methods Between February and June 2020, we administered a questionnaire of 142 questions distributed among seven domains: expert, communicator, collaborator, manager, health advocate, scholar and professional. Two rounds were conducted. Results In round 1, 18 competencies were discarded and nine had inconclusive results. Eighteen competencies were proposed by participants. The nine competencies with inconclusive results and the 18 newly proposed competencies were considered in round 2. In round 2, three of these 27 competencies tested were discarded, and consensus was reached on the other 24. Conclusion Health education and the empowerment of the patient and family are fundamental pillars in the practice of nurse anaesthetists in Spain, especially in pre‐anaesthetic assessment and pain care. These areas of practice can help define competencies in other countries where the profession of nurse anaesthetist is less developed. Impact The lack of regulation of the competencies of nurse anaesthetists leads to great variability in training and practice. The results can help in their standardization in Spain and in other countries lacking regulation. Our approach can also help policymakers and hospital administrators in health systems that are undergoing the process of regulation. The regulation of the competencies of nurse anaesthetists will allow them to contribute their expertise to the health‐illness continuum, increasing safety and improving the quality of care.
Article
Full-text available
Purpose: In the absence of literature allowing for an evidence-based approach to therapeutic patient education (TPE) in Oropharyngeal Dysphagia (OD), this study aims to reach a consensus of experts on the content of a competency framework of an educational program for OD patients and their informal caregivers (ICGs). Methods: We used the Delphi consensus-building method. Four categories of experts were recruited: 12 patients, 17 ICGs, 46 healthcare professionals (HCP) (experienced in OD, not necessarily certified in TPE), and 19 experts in TPE (trained individuals to set up and run TPE programs not necessarily HCPs). The content of the questionnaire of the first round (R) was established according to the result of a scoping review and the opinion of an expert committee. We carried out three rounds. In R1 and R2, we collected the opinions on the relevance (7-point Likert-type scale) and on comprehensiveness (YES/No question and asking participants to propose additional content). Participants were also invited to leave comments on each objective. In R3, we asked the participants to give their opinion about the relevance of the objectives again and asked them to rank the themes from highest to lowest priority. Results: Objectives were considered relevant for all participants if they reached consensus when the interquartile (IQR) ≤ 1, and if the median indicated agreement (Mdn ≥ 6) (6= appropriate, 7 = totally appropriate). Following three rounds, the final content of the educational program is composed of 23 educational objectives organized in 13 themes with an agreement about relevance amongst all participants (Mdn ≥ 6; IQR ≤ 1). The comprehensiveness criterion received also a consensus (IQR ≤ 1). The participants ranked the theme "normal swallowing vs difficulty swallowing" as the highest priority. Conclusion: This Delphi study resulted in a consensus, on the content of a competency framework of an educational program for OD patients and their ICGs. Further steps are needed to construct learning activities based on these objectives before testing their feasibility and efficacy.
Article
This study aims to investigate the supply chain challenges of public sector agriculture development projects in Bangladesh using the modified Delphi, Best Worst Method (BWM), and Interpretive Structural Modelling (ISM) methods. Based on these three widely acclaimed statistical techniques, the study identified, ranked, and identified interrelationships among the challenges. The study is unique not only in terms of research findings, but also in terms of methodology, as it is the first to use the three MCDM (Multicriteria Decision Making) tools to examine supply chain issues in public sector agriculture development projects in a developing country context. A literature review and two modified Delphi rounds with 15 industry experts’ opinions were applied to identify and validate a list of 11 key supply chain challenges. To determine the priority of the challenges, a panel of eight industry experts was consulted, and their responses were analysed using the BWM. Then, another group of 10 experts was consulted using ISM to investigate the contextual relationships among the challenges, resulting in a five-layered Interpretive Structural Model (ISM) and MICMAC (cross-impact matrix multiplication applied to classification) analysis of the challenges. According to relative importance (global weights), "improper procurement planning (0.213), "delay in project initiation (0.177), "demand forecasting error (0.146)", "lack of contract monitoring mechanism (0.097)", and "lack of competent staff (0.095)" are the top five ranked key challenges that have a significant impact on the project supply chain. Regarding contextual relationships, the ISM model and ISM-MICMAC analysis identified the "political influence" challenge as the most influential, and also independent of the other challenges. The findings are critical for project managers in managing challenges because understanding both relative importance and contextual relationships are required to address the challenges holistically. Additionally, these findings will benefit policymakers, academics, and future researchers.
Preprint
As children are considered as one of the most vulnerable groups of population, it is extremely 40 important to consider their rights, opinion and perception in city planning and design for their 41 wellbeing. Present research formulates a methodology in order to identify the measurable indicators 42 based on children’s rights, different Indian and global standards, and their perception of child-43 friendliness of a city as a stakeholder. The rights are associated with parameters of children’s 44 wellbeing as given in the standards, and children’s perception has been taken through focus group 45 discussion. Based on the rights and standards, and their perception; a set of measurable indicators 46 have been identified and ranked priority wise based on the perception and experience of the target 47 group. The research has two contributions – a checklist of measurable indicators of child-friendly 48 cities; and the priority ranking explaining the urgency of intervention to design child friendly city.
Article
Standard reading passages allow for the study of the integrated functions of speech and voice components in contextual, running speech, with target stimuli in a controlled environment. In both clinical practice and research, these texts provide rapid insight into the characteristics of the patient's speech, with fewer hesitations than in conversational speech and better predictability by the evaluator. Although a plethora of texts exist in different languages, they present various limitations. A specifically created standardised text in each language allowing for an ecological assessment of speech and voice functions, meeting most required criteria for standard speech and voice assessment and adapted to the target language's cultural and linguistic specificities, would therefore be an interesting option. However, no guidelines exist for the creation of such a reading passage. This article describes the international Delphi consensus study carried out to identify a minimal set of criteria to take into account when creating standard reading passages for an overall speech and voice assessment in adolescents and adults. This survey was conducted in three consecutive rounds; forty experts participated in the first round, with a total dropout of 17% from round 1 to round 3. It results in a minimal set of ten criteria which were selected by a majority of the experts and were rated as most important. This set contains five phoneme-level, two word-level, two sentence-level criteria and one global-level criterion. It can be used as a general guideline for the creation of standard reading passages in Indo-European Romance and Germanic languages such as English, French and German. The construction of a new reading passage in French following this guideline is briefly described.
Article
Background Macrophage activation syndrome (MAS) is a severe life-threatening hyperinflammatory state with uncontrolled activation and proliferation of macrophages and T-lymphocytes. MAS has variable causes and risk factors. Early diagnosis and optimum management could be lifesaving. Our aim was to develop a consensus, evidence-based recommendations for the diagnosis, evaluation, and treat-to-target management of pediatric MAS. This study was carried out to achieve an Egyptian expert consensus on a treat-to-target management strategy for MAS using the Delphi technique. The multistep process strategy was used in developing a consensus, evidence-based treatment guidelines for MAS, started by developing 7 key clinical questions by a scientific committee according to the Patient/Population, Intervention, Comparison, and Outcomes (PICO) approach. The core leadership team identified pediatric rheumatology clinicians and researchers throughout Egypt. To generate evidence for MAS management, an evidence-based, systematic literature review was done. To obtain a consensus, the Delphi procedure (3 rounds) was used. Results Twenty-three expert panel participated in the 3 rounds with a response rate of 100%. A total of 19 recommendations, categorized into 2 sections (11 in the diagnosis section and 8 in management), were obtained. The agreement with the recommendations (ranks 7–9) ranged from 86.9 to 95.7%. The consensus was reached (i.e., ≥75% of respondents strongly agreed or agreed) on all the clinical standards. Algorithms for management have been also developed. Conclusion This was an expert, consensus recommendation for the diagnosis and treat to target of MAS, based on the best available evidence and expert opinion. The guidelines fill a gap in the literature as it presents a T2T approach for MAS.
Article
Companies neither fully exploit the potential of Artificial Intelligence (AI), nor that of Machine Learning (ML), its most prominent method. This is true in particular of marketing, where its possible use extends beyond mere segmentation, personalization, and decision-making. We explore the drivers of and barriers to AI and ML in marketing by adopting a dual strategic and behavioral focus, which provides both an inward (AI and ML for marketers) and an outward (AI and ML for customers) perspective. From our mixed-method approach (a Delphi study, a survey, and two focus groups), we derive several research propositions that address the challenges facing marketing managers and organizations in three distinct domains: (1) Culture, Strategy, and Implementation; (2) Decision-Making and Ethics; (3) Customer Management. Our findings contribute to better understanding the human factor behind AI and ML, and aim to stimulate interdisciplinary inquiry across marketing, organizational behavior, psychology, and ethics.
Chapter
The simulated patient method is becoming an increasingly popular observational method to measure practice behavior in pharmacy practice and health services research. The simulated patient method involves sending a trained individual (simulated patient among other names), who is indistinguishable from a regular consumer, into a healthcare setting with a standardized scripted request. This method has come to be accepted as being well-suited for observing practice in the naturalistic setting and has also been used as an intervention when combined with feedback and coaching. This chapter presents an overview of the method, a brief history of its use, considerations for designing, implementing, and evaluating simulated patient studies, including ethical considerations, as well as methods of analysis and mixed-methods designs.
Article
Background We aimed to provide up-to-date, evidence-based and consensus-based recommendations for Treat-to-Target management of psoriatic arthritis (PsA) and associated clinical manifestations. In this recommendations, 14 key clinical questions were identified by scientific committee according to the Patient/Population, Intervention, Comparison, Outcomes and Timing (PICOT) approach. Literature Review team performed a systematic review to summarize evidence advocating the benefits and harms of available pharmacologic and non-pharmacologic therapies for psoriatic arthritis. Subsequently, recommendations were formulated. The level of evidence was determined for each section using the Oxford Centre for Evidence-based Medicine (CEBM) system. A 3-round Delphi process was conducted with 19 experts whom were drawn from different governorates and health centers across Egypt with diverse in their experiences, including private, governmental workplace, tertiary university hospitals, and insurance hospitals. All rounds were conducted online. A consensus was achieved on the direction and the strength of the recommendations. Results An online questionnaire was sent to an expert panel who participated in the three rounds (response rate 100%). At the end of round 3, a total of 51 recommendation items, categorized into 6 sections to address the main 6 psoriatic arthritis categories, were obtained. Agreement with the recommendations (rank 7–9) ranged from 89.5 to 100%. Consensus was reached (i.e., ≥ 75%of respondents strongly agreed or agreed) on the wording of all the 51 clinical standards identified by the scientific committee. Algorithms for the management of psoriatic arthritis have been suggested. Conclusion These recommendations provide an updated consensus on the pharmacological treatment of psoriatic arthritis and strategies to reach optimal treat-to-target outcomes in in common clinical scenarios, based on a combination of evidence and expert opinion. Best treatment decisions should be tailored to each individual patient situation.
Article
This study sought to coalesce sector knowledge on the use of systems approaches for sustainable water, sanitation, and hygiene (WASH) service delivery in low-income countries. To accomplish this objective, we remotely convened a panel of WASH sector experts within a multiround survey to identify, characterize, and prioritize the necessary attributes of WASH systems approaches. The first survey round asked experts to comment on the conceptual differences between traditional approaches and systems approaches to WASH. Emerging within responses from the first survey round were attributes of WASH systems (i.e., factors, actors) and attributes of WASH systems approaches (i.e., flexible, scalable). A three-round Delphi survey was then administered to reach consensus on these emerging attributes. By the final round of the Delphi, consensus was reached on every attribute of WASH systems, indicating alignment between the experts on the fundamental characteristics and implications of WASH systems. Consensus was also reached on the majority of attributes of WASH systems approaches, including the inherent interconnected and complex nature of WASH systems. However, consensus was not achieved on attributes related to mapping WASH system interconnections, convening stakeholders in collective action, and the appropriateness and feasibility of applying systems approaches at scale. This indicates a need for future research that explores practical and scalable tools and techniques to map and evaluate WASH system interactions, and ways to engage relevant actors in these approaches to collectively apply systems knowledge. © 2022 This work is made available under the terms of the Creative Commons Attribution 4.0 International license,.
Article
Full-text available
Humanitarian supply chains (HSC) have vital significance in mitigating different disruptive supply chain risks caused due to natural or man-made activities such as tsunami, earthquakes, flooding, warfare, or the recent COVID-19 pandemic. Each kind of disaster poses a unique set of challenges to the operationalization of HSC. This study attempts to determine the critical barriers to the operationalization of HSC in India during the COVID-19 pandemic. Initially, we determined and validated 10 critical barriers to HSC operationalization through a Delphi method. Further, we analyzed the barriers by computing the driving and dependence power of each barrier to determine the most critical ones. To do so, we coined a distinct form of interpretive structural modeling (ISM) by amalgamating it with the neutrosophic approach, i.e. Neutrosophic ISM. The findings indicate, "lack of Government subsidies and support, lack of skilled and experienced rescuers, and lack of technology usage" are the most critical barriers that influence the streamline operations of HSC during the COVID-19 outbreak, unlike other disruptions. This is the first-of-its-kind research work that has identified and analyzed the critical barriers to HSC operationalization during COVID-19 in the Indian context. The results and recommendations of the study can aid policymakers and HSC professionals in formulating suitable strategies for successful HSC operations. Supplementary information: The online version contains supplementary material available at 10.1007/s10479-022-04752-x.
Article
The Northern Sea Route (NSR) is an emerging alternative route to the Suez Canal; however, inconclusive research exists on its benefits and viability. This paper uses a two-stage Delphi approach to collect qualitative data from experts, on what actions Nordic countries can undertake to prepare themselves to enter NSR and capture potential benefits deriving from trading in global transport networks. By conceptually developing a framework based on First Movers Advantage (FMA) and Dynamic Capabilities Theory (DCT), this paper empirically identifies a path to develop capabilities that could allow stakeholders to reduce logistics costs through collaborations in NSR. Our paper has made a twofold contribution, first to literature by linking FMA/DCT in a single framework and elaborating it in a global transportation and logistics context. Second, a practical contribution by identifying how countries along the NSR, and specifically Nordic countries, can capture and maximise its benefits by using the right policy framework.
Article
Organizations strive to find new ways to manage the production of strategic spare parts with unanticipated demand and high delivery time. Additive Manufacturing (AM) is a revolutionary technology that effectively serves such needs by producing compatible spare parts in a shorter period without holding inventories. Although the promising benefits of spare parts production through AM, research is scarce in this domain. This study is developed to demonstrate the applicability of AM in spare part manufacturing. We have proposed a novel generic multi-criteria framework using Delphi, analytical hierarchy process (AHP), and segmentation approaches to identify the most compatible spares producible through AM. The applicability of the proposed framework is illustrated through a real-life case study. Delphi is used to determine and validate relevant criteria considered for spare part classification. This results in determining 11 criteria belonging to two group criteria, namely – Business Impact (BI) and Technical Compatibility (TC). AHP is used to compute the relative weights of criteria for prioritizing them. A normalization (objective criteria) and rating (subjective criteria) approach is used to evaluate the total score of each spare part for BI and TC. Spare parts are clustered into four quadrants (Type A, B, C, and D) using the segmentation approach according to the total scores of BI and TC. This generic framework benefits practitioners in identifying technically compatible spares for AM that improve business competitiveness.
Article
The Delphi approach is a central method in technological forecasting. It has been used in thousands of scholarly works and extensively outside academia, often in relatively narrowly scoped case studies. With the notable exception of Japan and a few other countries, broad Delphi studies run in the context of national-level policy planning are seldom seen. This study presents results on implementing the long-running Japanese Science and Technology Prediction Delphi survey developed by the National Institute of Science and Technology Policy (NISTEP) in Finland. This study reports the implementation of the 702 statement Delphi and on the reflections of the expert opinion study against quantitative data on Finnish research capabilities. The differences in expert opinion are discussed between Japanese and Finnish experts and the importance is highlighted that contextual factors play in responses received from the experts even on extremely global issues, such as the development of science and technology. It is proposed that quantitative data can serve as an important contextualization tool for the expert opinion results.
Chapter
The Delphi technique is a group judgment method which is typically used to reach agreement from a group of people with expertise in a particular area. It is an iterative process where panel members complete questionnaires over several rounds, often rating their agreement/disagreement against a statement, with changes made in later rounds based on the feedback received. It has been used widely in pharmacy-related studies relevant to education, research, and practice. This chapter provides a critical analysis of the various design choices which researchers may consider when planning a Delphi, namely, the panel of participants, the use of the Likert scale, the effect of feedback, what constitutes consensus, and the number of rounds. It also gives an overview of the development and origins of the Delphi, and discusses the advantages and disadvantages of the technique. Advantages include that the Delphi can be conducted with panel members in different geographical locations in their own time; however, the technique can therefore take longer to conduct and lacks face-to-face discussion. This chapter shows the importance of careful planning of the design choices to ensure the reliability and validity of the Delphi.
Article
Full-text available
Introduction The definition of sexual health evolves over time. As sexual health definitions and priorities are also context-dependent, there is a need to identify a diversity of sexual health challenges and priorities from a pan-European perspective. Methods We examined what a group of diverse experts in this area identifies as interdisciplinary, transnational priorities on sexual health using a Delphi method. In 2020, 93 participants from 29 countries took part in an online Delphi study. First, based on a three-round Delphi study, a hierarchy of priority topics was developed, comparing consensus rates across the items. Second, a qualitative content analysis of the participants’ responses to existing gaps and possible improvements in sexual health was administered. Results An inventory of priority items was created. The panelists identified 37 priority topics, divided into 10 overarching themes. Consensus was reached based on quantitative measurements regarding the importance of the suggested priority topics relevant to sexual health, resulting in 23 implemented items in the list of priorities. Qualitative data from the experts informed us about possible sexual health challenges and blind spots. Conclusion The study shows that the priorities chosen generally refer to (1) inclusion of sexual health into relevant medical health fields and education, (2) comprehensive sex education in schools, and (3) sexual violence. The importance of these three topics was, moreover, reflected in the qualitative data. Policy Implications By delineating a relatively consensual set of priorities for transnational sexual health research and advocacy, this study outlines a possible research agenda for sexual health in the pan-European region, potentially serving as the base and start of joint interdisciplinary practice.
Article
Full-text available
There is only one author Turoff and the end has the following references: Murray Turoff, "The Design of a Policy Delphi," Technological Forecasting and Social Change 2, No. 2 (1970). 2 Charles F. Schultze, "The Politics and Economics of Public Spending," Brookings Institution, Washington, DC., 1968. 3 Numerous references to Lindblom's writings on committee processes appear in the work cited in reference 2. 4 Jerry B. Schneider, "The Policy Delphi: A Regional Planning Application," Technological Fowashng and Social Change 3, No. 4 (1972). 5 Murray Turoff, "Delphi Conferencing," Technological Forecasting and Social Change, 3, No. 2 972). The original version in TFSC is crude text if you down load it. this is from the version in the book. There are many specific general designs in the various papers generated and i still hope to go back and summarize the most popular ones over many years. signed murray turoff. If anyone has a specific design that they think should be summarized as a unique design others could use feel free to send it to me. murray.turoff@gmail.com
Article
Full-text available
A Delphi study was performed to answer the question: Which global societal trends relate to future radicalization and subsequent terrorism in the Netherlands? An inventory of two hundred global societal trends and a literature study of radicalization together served as briefing of the participants in the Delphi study. The first two rounds were conducted in anonymous writing. The third round was a face to face focus group meeting. In the first and second round participants scored the trends, adjusted their scores relative to others and provided arguments, and in the focus group meeting, themes were discussed in which the participants remained to have strong opposite views. This approach emphasizes divergence in opinion (‘dissensus’), as opposed to convergence (‘consensus’). Consensus seeking to us seems to be the dominant application of Delphi, but we emphasize the value of seeking dissensus. We hypothesize that in those cases where the opposite views that were discussed kept their ground, this Delphi study may have found early warnings of future radicalization.The essence of this paper is that it is possible, with limited effort, to get a handle on the complex and poorly defined subject of global societal trends influencing future radicalization. Application of the Dissensus Delphi method provided a selection of early warnings that may be looked into with future research.
Article
Full-text available
uilding consensus is an essential com- ponent of any policy-making process. The hallmarks of the policy Delphi method are to bring together stakeholders with opposing views and to systematically attempt to facilitate consensus as well as to identify divergence of opinion (Strauss & Zeigler, 1975). As many health policy issues are complex, the policy Delphi method is an appropriate tool because it can address a multiplicity of issues and provide direction for policy changes (Critcher & Glad- stone, 1998). Unfortunately, this method has not been widely used or reported in the literature (Critcher & Gladstone, 1998; Linstone & Turoff, 1975). The purposes of this article are to describe the use of the policy Delphi method in building con- sensus for public policy and to propose a tech- nique for measuring the degree of consensus. The application of the method is illustrated by a case example from a study of state legislators' views on tobacco policy (Hahn, Toumey, Rayens, & McCoy, 1999). Because tobacco control policy development is highly contentious, particularly in tobacco- growing states, the policy Delphi method is well suited for building consensus on tobacco policy issues.
Article
Full-text available
In response to a request by the guest editors, we have set down our thoughts regarding the evolution of Delphi, beginning with our immersion in the subject in the late 1960s and concluding with some rumination about its future. Our focus is on the changing roles of Delphi. Most importantly, with the profound impact of the internet on organizational and community planning systems, it will foster a new age of participation through communication, coordination, and collaboration.
Article
Full-text available
A strategy to revitalize coastal shipping within Great Britain is investigated. An examination of the regulatory environment shows government, at both the EU and British levels, aware of the environmental benefits of coastal shipping compared to road freight transport, if only mildly supportive in tangible terms. The geography of Great Britain and the modern preference for just-in-time deliveries have severely restricted the ability of coastal shipping to compete effectively with road freight transport. The primary objective of the paper is to investigate whether coastal shipping could be integrated into a multimodal door-to-door supply chain, where it is currently hampered by high costs of transhipment, slow transport speeds, and the incongruity of load sizes between land and sea modes. Integration requires the co-operation of all organizations within the multi-modal supply chain. However, coastal shipping companies have been known to be individualistic and mistrusting of alliances. Therefore, a Delphi study is undertaken to investigate the standpoint of leading managers in such companies towards multimodal integration. The results of the Delphi study indicate that managers are in favour of multimodal developments, in particular cooperation between coastal shipping and road haulage. However, there was agreement that the business is highly competitive, and this may prevent the forms of collaboration required for multimodal systems. The secretive nature of the coastal shipping sector could impede joint marketing to promote a mode of transport that is often ‘invisible’ to shippers. There was little consensus on the relationship that should exist between coastal shipping and ports, a topic worthy of further investigation.
Article
Full-text available
The policy delphi is a specific design to address a policy issue that has many alternative resolutions to the solution of the policy issue. It starts with a literature search to establish all the different published resolutions of the policy or recommend ones. The paper describes how the respondents to the Delphi should vote on the different resolutions for desirability and feasible. Any respondent can mage positive or negative comments about the issue and anyone can vote on any of these comments for the true or false measure and the feasibility measure as well. Participants may change (at any time) their votes at any time due to the contributions of the participants. Ideally this is done online and may be entered at any time for any participant and the system shows the user new items or new vote values they have not seen. The user may also review any part of the delphi they wish to see at that time. They may also introduce at any time new policy resolutions and new comments. The published paper has an example topic. It can be done as an mailed set of rounds also but it does make it harder to make a lot of dynamic changes to earlier rounds.
Article
Full-text available
The article provides a n overview of the Delphi technique and its related adaptations, as well as practical hints on procedures gained from its use to forecast changes in the market analysis industry. The Lielphi technique has been used many times as a method of forecasting the future ofestab1ish:ed industries, but it has never been used to consider the future of a nascent industry. A case is presented which suggests that companies i n nascent industries have a greater need for long-range planning than those at other stages of a n industry's development, and a greater need for an industry, rather than a company, perspective. Problems such as time pressure on executives and the need for confidentiality are more acute when using judgmental forecasting techniques in new industries. The review of the Delphi process highlights some of the major considerations, s,uch as panel selection, questionnaire construction and administration, response analysis and number of rounds, as well as how Delphi can be modified. Since new industries rarely have established trade organizations to curry out such Delphi studies, the role could be played by institutes of management education.
Article
Full-text available
The emphasis in the Delphi literature to date has been on results rather than on methodology and evaluation of design features. The other articles in this chapter do address the latter aspects. Still, quite a number of issues remain unsolved, particularly those concerned with the details of the internal structure of the Delphi. For example
Article
Full-text available
Only a limited amount of research has been reported on factors influencing international location decisions for contemporary manufacturing operations. In this paper a comprehensive set of factors that may influence international location decisions is identified from an analysis of the existing literature. Results are presented from a Delphi study that used a worldwide panel of experts to investigate factors affecting international location decisions. Findings are reported on the motivations of firms in seeking to manufacture across national borders and the key steps that should be followed in making international location decisions. The top five major factors identified that may strongly influence international location decisions generally were: costs, infrastructure, labour characteristics, government and political factors and economic factors. Ten key sub-factors identified were: quality of labour force, existence of modes of transportation, quality and reliability of modes of transportation, availability of labour force, quality and reliability of utilities, wage rates, motivation of workers, telecommunication systems, record of government stability and industrial relations laws. Additional sub-factors of increasing importance include: protection of patents, availability of management resources and specific skills and system and integration costs. The factors identified have implications for management practice, for policy-making by governments and other agencies and for academic research in international operations.
Article
Scenarios can enable one to act on the future because they define a relevant system; they bring actors together to explore the possible futures of that system in a way that gives them an equal voice. They create fora where those actors can build new social knowledge, partly through challenges of culture and interpretation, and provide tools where individuals from different bodies of practice and knowledge can converse as equals. They are systems thinking without systems specialists. They enable the renegotiation of meaning and the reframing of understanding, and combine social exchange with knowledge and interpretation. It is all of this, together, which creates the opportunity, and the freedom, to act.
Article
This article reports on a survey of 37 educators regarding future directions in the education of students with disabilities. The survey used the Delphi technique. For the decade of the 1990s and after the year 2000, respondents’ predictions included the following: The movement toward increasing inclusion will occur; the belief will prevail that people with disabilities have a right to participate in inclusive environments; students with mild disabilities will be educated in general classrooms; teachers will increase their use of instructional approaches such as cooperative learning and instructional technology; and researchers will focus on matching instructional needs with learner characteristics.
Article
Graham Molitor's article provides a timely prompt for reflecting on the value of scenario prac-tices, especially given several data sources indicating their usage has increased significantly since 2001 (e.g. Ramirez, Selsky, & van der Heijden, 2008, p.9). Molitor is not alone in his struggle to clarify the effectiveness of scenario practices. Others, including myself, are endeavouring to address similar questions: how to judge effectiveness and what do we mean by 'effectiveness' when referring to such practices? As he implicitly suggests, his critique does not imply that we should throw the scenario 'baby out with the bathwater'. It is all too easy to agree with some of the criticisms of scenarios raised by Molitor. Three aspects are particularly relevant: The first is that futures work seems to be characterised by highly personalised practices. Such practices can be introduced by someone who thought it was "a good idea" but who failed to fully reflect on the complexity of the situation and bases their choice of techniques on sound theoretical principles. Secondly, as much of scenario work is secret – particularly in military and corporate sec-tors-and/or difficult to assess, it is very hard to engage in comparative research. Thirdly, common to other practitioner-led fields, scenario practices are blessed with a high degree of innovation and entrepreneurship and cursed by a lack of reliable accounts that render explicitly what has worked and what has not, why and for whom in different settings. In the limited space available, however, I would like to raise three areas that I feel are worthy of further reflection: 1. Scenarios Are Not Forecasts By implicating scenarios with "any technique that may advance forecasting capabilities", Molitor contributes to the already considerable methodological confusion that characterises the futures field, in general, and scenario practices in particular. In fact, scenarios – i.e. many futures -and forecasting – one future -have different ontological and epistemological underpinnings.
Article
Background and Purpose The do-not-resuscitate (DNR) order is a mechanism of withholding cardiopulmonary resuscitation (CPR). The lack of DNR guidelines specific for acute stroke may result in many stroke patients receiving unnecessary and futile resuscitation and ventilator-assisted breathing. Methods A prospective multicenter evaluation of disease-specific criteria for DNR orders in acute stroke was initiated using a modified Delphi process. The participants were the Canadian and Western New York Stroke Consortium members who are closely involved in caring for acute stroke patients and conducting clinical trials at the academic centers. Previously published provisional criteria were reviewed by the participants. Modifications were made to the criteria until statistically significant agreement (P<.05, z score, or 67% similar answers) was achieved. Results Disease-specific criteria for DNR orders in acute stroke were discussed by 26 physicians in three rounds of the opinion survey. An agreement was reached that a “no resuscitation” decision is appropriate when any two of the following three clinical criteria are present (the degree of agreement is given in parentheses): severe stroke (88%, P=.00007), life-threatening brain damage (73%, P<.01), and significant comorbidities (92%, P=.00003). The poor prognosis implied by these criteria should be discussed whenever possible among physician(s), the patient, and family members before the decision to withhold CPR is made. Eighty-one percent of the participants agreed that these disease-specific criteria are appropriate for clinical use (P=.0008). Conclusions Disease-specific criteria for DNR orders were developed to supplement general DNR policies for patients with hemispheric brain infarction and intracerebral hemorrhage during the first 2 weeks of stroke. A significant agreement was reached by a panel of physicians that patients with acute stroke should not be resuscitated if these disease-specific criteria are met.
Article
This paper gives an account of an experiment in the use of the so-called DELPHI method, which was devised in order to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of questionnaires in depth interspersed with controlled opinion feedback.
Article
The present study compared the quality and the acceptance of group decisions on an evaluative problem (NASA Lost on the Moon Exercise). Four decision-making formats were employed: interacting, consensus, the nominal group techinique (NGT), and the Delphi technique. No idiosyncratic modifications were made in any of the formats. The results indicated that the Delphi groups produced the highest quality decisions followed by those of consensus, interacting, and NGT groups. The decisions of the consensus groups had a higher level of acceptance than did those of the other three formats, which did not differ in terms of acceptance. Predictions based on past studies using unmodified decision-making techiques were consistently supported.
Article
This paper reports on a study to determine the opinion of expert practitioners of the most important risks in the development of e-commerce projects. The 32 respondents in the final round of the survey were mainly project managers from South African software houses. Various academics and users of e-commerce systems also contributed to the survey. The Delphi technique was used to gather the data and to rank the risks.Misunderstanding the users’ requirements emerged as the most significant risk, followed by the absence of declared business benefits. As with conventional systems, there is a risk of top management not getting totally committed to the project, very often giving verbal encouragement to the IT team but overlooking the impact on the business as a whole.Respondents place a high importance on the security issues surrounding e-commerce projects. Transactions are subjected to more threats, and developers have to incorporate procedures to ensure transaction integrity and confidentiality, and then convince potential customers of the system's security. Other related issues include transaction tracability and database security and integrity.Hype in the market suggested that there was a large risk of delivering systems too slowly as a result of “cumbersome” methodologies. The research did not find this to be the case.Different perspectives emerged from the viewpoints of developers, project managers, clients/users and academics.
Article
Comprehensive health planning must ultimately be based upon information about the people to be served. Their needs as potential patients or clients of the various service systems being planned must be identified and evaluated. Regional cooperation on meeting these health care needs is premised on some agreement amongst the population as to the existing problems and their relative priorities. In order to investigate how such a health needs assessment might be undertaken a study was conducted using a panel of citizens selected from a 10-country central Texas region. The results demonstrate the fesibility of employing the Delphi method for achieving consensus on some health care issues; the information obtained can become valuable input to the overall health planning process.
Article
This paper addresses the question of the methodology of testing for stability between successive Delphi rounds and the necessity of doing so prior to the analysis of the level or type of consensus. Following a discussion of a hierarchical set of a criteria for determining the termination of such studies, the paper reviews some of the more common treatments of the subject in the literature. It is finally proposed that a χ2 test be used to test the stability of responses between rounds. The method is demonstrated by a numerical example.
Article
Based on the Delphi technique, the project described in this article seeks to integrate the teaching of classical political science with the analysis of problems facing contemporary and future society. Experts on particular political philosophers of the past were asked to imagine how those men would have viewed and solved present political problems, and thus to use their knowledge to develop a new approach to policy analysis.
Article
a b s t r a c t The logistics services industry will be significantly affected by future developments throughout the world. Therefore, developing future scenarios is an important basis for long-term strategy development. Nevertheless, research exposes that there is a lack of awareness among logistics researchers and practitioners about future scenarios. In this paper, we apply scenario planning and present the findings of an extensive Delphi-based scenario study on the future of the logistics services industry in the year 2025. The major contribution of our research is the development of probable and unforeseen scenarios of the future which may provide a valuable basis for strategy development in the logistics services industry.
Article
In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.
Article
A novel and innovative real-time Delphi technique is introduced in order to address previously identified weaknesses of the conventional Delphi method, such as complicated facilitator tasks, lack of real-time presentation of results, and difficulties in tracking progress over time. We demonstrate how the real-time (computer-based) method increases the efficiency of the process, accommodates expert availability, and reduces drop-out-rates. Modifications in the Delphi procedure (e.g. change of iteration principle) not only increase efficiency but also change the nature and process of the survey technique itself. By identifying and analysing three individual effects (initial condition effect, feedback effect, and iteration effect) we examine whether the modifications in the survey process cause deviations to the survey results. Empirical data obtained from both conventional as well as real-time Delphi studies is analysed based on multiple statistical analyses. The research findings indicate that significant differences between the two Delphi survey formats do not exist and final survey results are not affected by changes in the survey procedure.
Article
The primary purpose of this study was to determine the characteristics of the turfgrass industry in the year 2020 in order to recommend curriculum content for agricultural education programs of the future. Specific objectives were to determine: the general characteristics of the turfgrass industry in the year 2020, a demographic profile of opinion leaders in the turfgrass industry, whether the Delphi technique could be used to achieve consensus among turfgrass experts, the workforce requirements of the turfgrass industry, and the educational requirements of those employed in the turfgrass industry in the year 2020. This was a national study using a three-round Delphi technique. The panel of experts was nominated by turfgrass professionals (teaching, research and extension) at all the land- grant institutions, the four-year and two-year colleges offering turfgrass, and the executive boards of related turfgrass associations. The 38 most frequently nominated experts were invited to participate in the study. A structured Delphi instrument consisting of nine categories was developed from the literature. This instrument was circulated to the panel of experts for comments on each category. Their responses were used in the development of a structured Delphi instrument consisting of 147 items on a Likert-type scale. The instrument was reviewed for content and face validity by an 20-member panel. Consensus was achieved on an item if at least 60% of the respondents were in agreement and the composite score fell in the "agree" or "disagree" range. The mean and standard deviations indicated that the central tendency was a movement toward consensus. The responses were found to be very stable from round 2 to round 3, as indicated by the Pearson product- moment correlation coefficient procedure and the Wilcoxon matched-pairs signed-ranks test. Thirty-two content items were identified as essential inclusion in agricultural education curriculum.
Article
Is Delphi just a namesake? Or does it make sense to connect the Delphi method to the ancient Delphic Oracle? To bring some elements of response to these questions, this article will present the Delphic Oracle in the first part and its actions with its prophecy seekers and the Oracle's staff, including the Pythia. This first part will show the importance of conforming to the procedure in the revelation of the Oracle and the latter's role in policy-making. The second part of the article will compare and contrast the oracular practice in Delphi with the Delphi method, and more generally with expert-based futures methods, from the characteristics of the knowledge revealed, on the one hand, and the role of the actors in this inquiring process on the other.
Article
This paper presents an adapted Delphi methodology that is, contrary to the classical Delphi design is not aiming to minimize expert estimation variance, but to maximize the range of expert opinions inputted sequentially into an online system. After discussing the traditional Delphi approach and its dissensus based derivatives, the author opens the case for a dissensus Delphi based explorative research tool with special consideration of the Delphi aim, the expert sample and the Delphi design. The proposed online Delphi process is then presented conceptually. Next, the proposed tool is demonstrated based on a prototype, exploring the barrier factors to the adoption of mobile data services. A discussion on the theoretical design and practical R&D experience of the dissensus based online Delphi approach concludes the paper.
Article
The Delphi method is a popular technique for forecasting and an aid in decision-making based on the opinions of experts, which has been in existence for over half a century. This work evaluates its methodology and reviews its validity in the present day, especially in the area of Social Sciences. Three recent applications in this field are also explained, professional in nature, which have some characteristics that are not frequent with respect to other Delphi studies published. The main aim of two of these studies was to provide input for economic or statistical quantitative models, using the judgement of expert groups, while the third study aimed to analyse a complex social reality by means of a Policy Delphi in order to obtain reliable information before taking a policy decision. These applications highlight how this technique may be adapted to different social realities and requirements, making a positive contribution to social progress, provided it is applied with the necessary methodological rigour and with a good knowledge of the social medium in which it is being applied. Finally, there is an explanation of a number of lessons learned from the theory and aforementioned experiences, which may contribute to the successful outcome of a Delphi exercise.
Article
A nationwide Delphi-based technology forecasting exercise was undertaken in India in electronics and information technology. First, scenario writings covering 10 subtechnological areas were done by a panel of experts based on certain guidelines provided by the Technology, Information, Forecasting and Assessment Council (TIFAC). These formed the basis of framing of more than 280 Delphi questions by a set of experts. Delphi inquiry required responses to be given in numerical as well as text form particularly to cover specific Indian scenario and needs. About 370 experts were asked to respond. The response rate was around 35%, which was considered statistically sufficient to go ahead with the analysis. A computer program analyzed the responses through the combination of interquartile range mean, median, and mode to find out the consensus among the respondents on a particular question. Text-based responses were analyzed to propose a roadmap for India. Technology forecasts, thus done on short, medium, and longterm, were then fine tuned in a one-day seminar of experts, planners, and administrators. These were then debated and short listed to arrive at a recommended plan of action for implementation. The modified Delphi methodology for technology forecasting proposed in this article is thus a combination of scenario writing. Delphi questionnaire, and response analysis using additional written inputs to develop a roadmap for India, fine tuning and short isting through a seminar for implementation. This methodology is considered unique for rapidly changing technology with all pervasive applications like electronics and information technology aiming at sustainable development.
Article
A critical phase of scenario making is the choosing of scenarios. In the worst case, a futures researcher creates scenarios according to his/her subjective views and cannot see the real quality of the study material. Oversimplification is a typical example of this kind of bias. In this study, an attempt towards a more data sensitive method was made using Finnish transport policy as an example. A disaggregative Delphi method as opposed to traditional consensual Delphi was applied. The article summarises eight Delphi pitfalls and gives an example how to avoid them. A two-rounded disaggregative Delphi was conducted, the panelists being representatives of interest groups in the traffic sector. Panelists were shown the past development of three correlating key variables in Finland in 1970–1996: GDP, road traffic volume and the carbon dioxide emissions from road traffic. The panelists were invited to give estimates of their organisation to the probable and the preferable futures of the key variables for 1997–2025. They were also asked to give qualitative and quantitative arguments of why and the policy instruments of how their image of the future would occur. The first round data were collected by a fairly open questionnaire and the second round data by a fairly structured interview. The responses of the quantitative three key variables were grouped in a disaggregative way by cluster analysis. The clusters were complemented with respective qualitative arguments in order to form wider scenarios. This offers a relevance to decision-making not afforded by a nonsystematic approach. Of course, there are some problems of cluster analysis used in this way: The interviews revealed that quantitatively similar future images produced by the panelists occasionally had different kind of qualitative background theory. Also, cluster analysis cannot ultimately decide the number of scenarios, being a choice of the researcher. Cluster analysis makes the choice well argued, however.
Article
The general question of whether Delphi users have learned from Delphi research of the past decade is addressed in this article. Within the wider context of Delphi expansion anddevelopment, specific attention is given to examining the nature and quality of the growing number of Delphi dissertations. Although there are some signs for encouragement, the findings from the sample of dissertationsexamined might also give cause for some concern. Findings raise continuing questions related to excellencein the execution of Delphi studies and also to the concentration of applications of the technique in certain fields. It appears that the future of Delphi could be brighter if users of the technique would be more careful in implementing “quality control.”
Article
The processes involved in an individual's response to the Delphi method are conditioned by a number of psychological affects. This paper examines aspects of two of these processes: the basis for changing opinions, and the factors related to continued participation in subsequent Delphi rounds. The attitudinal perspective is shown to be promising in explaining an individual panelist's response in the Delphi method. Cognitive dissonance apparently has a significant role to play in bolstering the assimilation of nonconforming respondents. This is, however, complicated by the respondents' perception of the credibility of the feedback received. Careful choice of panels may reduce dropouts, presumably by selecting those with a high degree of ego involvement. Nonetheless, elements of cognitive dissonance apparently lead certain panelists to abandon the Delphi process. It is evident that numerous parallels exist between psychological explanations of attitude change and the behavior of Delphi panelists.
Article
This article carries forward the work discussed in a recent article in this journal by Dajani, Sincoff, and Talley (Technol. Forecast. Soc. Change 13, 83, 1979) by investigating individual stability as opposed to group stability for successive Delphi rounds. It is demonstrated that individual stability implies group stability but that the converse does not hold. A χ2 test is proposed for testing the stability of individual responses between consecutive Delphi rounds. Also, an estimation procedure is presented for measuring the extent of individual stability.