To read the full-text of this research, you can request a copy directly from the authors.
The Delphi technique is a widely used and accepted method for gathering data from respondents within their domain of expertise. The technique is designed as a group communication process which aims to achieve a convergence of opinion on a specific real-world issue. The Delphi process has been used in various fields of study such as program planning, needs assessment, policy determination, and resource utilization to develop a full range of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by using a series of questionnaires delivered using multiple iterations to collect data from a panel of selected subjects. Subject selection, time frames for conducting and completing a study, the possibility of low response rates, and unintentionally guiding feedback from the respondent group are areas which should be considered when designing and implementing a Delphi study.
To read the full-text of this research, you can request a copy directly from the authors.
... Further, group discussions can be intimidating and inhibitive to an individual participant (McMillan et al., 2016). I was not able to offset the shortcomings associated with pooling opinions obtained from focus group interaction, i.e. participant anonymity, influences of dominant individuals and group pressure for conformity (Hsu & Sandford, 2007;McMillan et al., 2016). The panellists were drawn from the hospitality industry in the Netherlands, are associated with an educational institution (stage 1 of this study) and were expected to be unbiased in their opinion with a genuine aim of improving hospitality industry recruitment and retention. ...
... On the other hand, completely eradicating the aspect of personality is difficult in the Delphi technique too as the experts may use informal channels of communication which the researcher has no access to, such as in the case of Lugosi et al. (2012) where investigative research on internet streaming needed to gain permission to enter a blog. The Delphi technique ensures that each participant would have no pressure, either real or perceived, to conform due to social norms, customs, organisational culture, or standing within the profession (Hsu & Sandford, 2007). ...
... What constitutes an optimum number of participants in a Delphi study has not reached a consensus in the literature (Delbecq et al., 1975;in Hsu & Sandford, 2007;Sekayi & Kennedy, 2017). Some of the published studies seem to clearly agree that there are no existing criteria to identify the ideal sample size in a Delphi study (Hsu & Sandford, 2007;McMillan et al., 2016;Robertson et al., 2017). ...
The Delphi technique is a recognised research technique designed as a futures-oriented methodology that gathers data by engaging a panel of experts to make judgements on a specific real-world issue. The technique enables the researcher to interact with key stakeholders and decision makers in the field, who are well experienced and trained in the specialised area of knowledge related to the target issue of the research, enabling researchers to gain a deeper understanding of real-world opinions. The Delphi technique has been described as a qualitative, quantitative, and mixed methods approach as the collection of narrative group opinion, coupled with the strictly structured nature of the process and quantitively described results, poses a challenge to situate the approach in a specific methodological category and its usage. This methods tutorial discussion aims to illustrate the usage of the Delphi technique by providing a worked example to demonstrate the proposed method in a hospitality industry setting.
KEYWORDS: experts, future oriented, HR, hospitality industry, research approach
... Delphi studies use quasi-anonymity to reduce the negative effects related to other methods of group communication (e.g. focus groups), like for instance groupthink or dominant panel members (Dalkey & Helmer, 1963;Hsu & Sandford, 2007;Turoff, 2002). Quasi anonymity can either refer to anonymity of the panel or only to anonymity of the responses of the panel. ...
... al., 2021: 2), it is usually up to the researchers to decide the type of feedback (von der Gracht, 2012). The controlled feedback should however stick with essential information necessary to complete the next round of the study, eliminating irrelevant information or 'noise' (Hsu & Sandford, 2007). ...
... • A fourth and last characteristic is statistical aggregation, which implies that a number of statistics are used to determine the degree of consensus among the expert panel. To determine the opinion of the group, often reference is made to statistical indicators of central tendency and dispersion (Hsu & Sandford, 2007;von der Gracht, 2012) which are presented in a numerical or graphical way Schmidt, 1997). ...
The report elaborates on the results of our Delphi study on insider threat mitigation. The goal of the study was to discover potential ‘red flags’ of insider threat incidents (i.e. factors that may point to imminent insider threat incidents) and good practices on insider threat mitigation. The study employed the Delphi technique to iteratively compare and contrast the opinions of insider threat experts. A multidisciplinary panel of 25 international experts in a field related to insider threats (corporate security, counterintelligence, insider threat training, …) completed three rounds of online questionnaires that contained questions on four insider threat mitigation phases, namely prevention, detection, pre-emption and the aftermath of an insider threat incident. Round 1 concerned open-ended, level-setting questions with panelists asked to share their expertise on both red flags and good practices. In round 2, experts were presented with a list of all important red flags/good practices shared by the panel in round 1 in the form of a structured questionnaire whereby they were asked to rate each individual red flag/good practice listed. The questionnaire of round 3, to conclude, provided the experts with a list of high-rated factors, after which they were asked whether they agreed or disagreed with the panel’s decision and to explain their reasoning in case of disagreement. The results of the study provide readers with useful insights on what experts consider to be red flags organizations should be vigilant of, as well as with mitigation measures to better secure organizations against insider threats.
... A potential limitation is that there may be gatekeepers in the organizations who identified the experts, and there may thus be experts who are not included in the study. On the other hand, the ambition in the Delphi process of selecting a group of highly trained and competent experts within the specialized field of knowledge related to the target issue is considered as a strength of the study (Hsu & Sandford, 2007). All the eligible experts from the currently available cohort of active experts in the subgroups of instructors and educators were included. ...
... In order to strengthen the empirical perspective on the data, the subgroup of PPOs with direct experience of the practical field and who represented all the police regions in Sweden were the largest group of the experts. The size of the group of experts (n=43) correlates with the sizes referred to in the literature (Hsu & Sandford, 2007). However, consensus concerning the optimal number of experts in a Delphi study has not been achieved in the literature, with variations from approximately 10 to several hundred (Keeney, Hasson & McKenna, 2011;Powell, 2003). ...
... The 75% consensus cut-off point was predetermined and arbitrary. However, the percentage, which was considered to be reasonable by the authors, was supported by the lit-erature (Keeney et al., 2011;Hsu & Sandford, 2007) and relevant for the aim of the study. This predetermined consensus rate could benefit and strengthen the outcome of the study in the ambition of exploring areas of competence in emergency care among PPOs. ...
Police patrol officers (PPOs) face different types of encounters with acutely ill and injured people as first responders. The study aim was to explore and describe the emergency care competence needed for Swedish PPOs. The study had an exploratory design, and the data was collected using the Delphi technique from 43 experts in Sweden: police program educators (n=10), police authority instructors (n=11), and PPOs (n= 22). The results generated an understanding of emergency care competence that is relevant for PPOs. Basic assessment and basic life support meas-RESEAR CH PUBLICATION ures for traumatic injuries and cardiac arrest (e.g., airway management, cardiopulmonary resuscitation (CPR), and hemorrhage control) received the highest levels of agreement among the experts, whereas more advanced assessments (taking care of people suffering from mental/medical illness or with drug/alcohol abuse) received lower levels of agreement. These results provide support for educators at the police programs and for the police authority in preparing PPOs to perform relevant out-of-hospital emergency care.
... The Delphi technique was used to reach a convergence of opinions regarding the most valuable items to include in the questionnaire, based on the answers of an expert panel. 33,34 This phase involved providing the AAMQ-V1 to the panel for review. The panel review was organized in two rounds. ...
... The AAMQ-13 has a score range from 13 to 65 (Table 3). The final scores were divided into three categories of adherence: poor adherence (score= 13-29), moderate adherence (score= , and excellent adherence (score= 48-65). It is worth mentioning that we took into consideration that some patients with asthma do The overall agreement score not use inhalers, and consequently, we decided to reduce the 13 items AAMQ to 10 items for these patients (AAMQ-10). ...
... In this scenario, the 10-items AAMQ will have a total possible score of 50. To assist in the interpretation, scores were arbitrarily divided into three categories indicating poor adherence (score= 10-22), moderate adherence (score= , and excellent adherence (score= 37-50). ...
Background: Adherence to medication is the cornerstone to achieve the best treatment outcome. Pharmacists are healthcare professionals found in a
pivotal position to assess asthmatic patients’ adherence to medication. A brief, reliable, and valid measure of patients adherence to mediations is useful
to enable the pharmacists to deliver that vital service. Objective: To develop a reliable and valid adherence assessment tool for asthmatic patients.
Methods: The Adherence to Asthma Medication Questionnaire (AAMQ-13) was developed based on an extensive literature review, followed by applying
the Delphi technique, and then it was pilot-tested by 55 patients. The final AAMQ-13 was completed by 213 patients. Psychometric evaluation was
assessed including reliability, criterion validity, and construct validity. Results: The AAMQ-13 is a feasible 13-item questionnaire, as it can be completed
within an average of two minutes. It has high reliability (Cronbach’s alpha= 0.87). Criterion-concurrent validity was established by comparing the AAMQ-13
to the Test of the Adherence to Inhaler (TAI) and the pharmacy refill records. Criterion-convergent validity was established by comparing the AAMQ-13 to
the Asthma Control Test (ACT) questionnaire and the Positive Health Behaviors Scale (PHBS). Construct validity was established through AAMQ-13 factor
analysis which revealed two factors explaining 51.76% of the total variance. Conclusion: The AAMQ-13 is a reliable and valid questionnaire with several
desirable characteristics as it has high reliability, good criterion validity, and strong construct validity. The AAMQ-13 is a suitable questionnaire that can
identify non-adherent patients and reveal the reasons behind their non-adherence
... In this study, three rounds were planned to ensure the process did not become too repetitive 20 since three rounds are sufficient to collect the needed information to reach a consensus 21 and provide the participants the opportunity to reflect on their responses to statements in the previous rounds. 22 Between the rounds, we provided controlled feedback to the participants and informed them of the opinions of their anonymous peers. ...
... According to the literature, participants in a Delphi study are considered eligible to be invited to participate if they have related backgrounds and experiences concerning the target issue. 21 We considered the patients and their informal caregivers at an equal footing from the HCPs and the experts in TPE knowing their day-to-day challenges with OD. We reached out to four categories of participants: patients with OD and informal caregivers (known as family members, relatives, friends, and neighbors, with no training and providing care for free), 24 to have feedback about the content of an educational program relying on their personal experience with OD to help us tailor the content to their day-to-day needs; healthcare professionals (HCPs) with experience with OD to benefit from their clinical expertise with OD patients and expert in TPE to cover properly the fundamentals of a TPE. ...
... As recommended in the literature, we aimed to comprise at least 10 participants in each subcategory of experts at the end of the process. 21 The recruitment of participants was made based on the network of HCPs and personal recommendations (snowball system) and through social media like LinkedIn, Facebook, and WhatsApp. An invitation letter, including information about the study, information relating to the processing of personal data (General Data Protection Regulation-GDPR), and a consent form were sent via email and phone message to 185 potential participants in September 2020 in French, English, or Arabic depending on the anticipated preferred language. ...
In the absence of literature allowing for an evidence-based approach to therapeutic patient education (TPE) in Oropharyngeal Dysphagia (OD), this study aims to reach a consensus of experts on the content of a competency framework of an educational program for OD patients and their informal caregivers (ICGs).
We used the Delphi consensus-building method. Four categories of experts were recruited: 12 patients, 17 ICGs, 46 healthcare professionals (HCP) (experienced in OD, not necessarily certified in TPE), and 19 experts in TPE (trained individuals to set up and run TPE programs not necessarily HCPs). The content of the questionnaire of the first round (R) was established according to the result of a scoping review and the opinion of an expert committee. We carried out three rounds. In R1 and R2, we collected the opinions on the relevance (7-point Likert-type scale) and on comprehensiveness (YES/No question and asking participants to propose additional content). Participants were also invited to leave comments on each objective. In R3, we asked the participants to give their opinion about the relevance of the objectives again and asked them to rank the themes from highest to lowest priority.
Objectives were considered relevant for all participants if they reached consensus when the interquartile (IQR) ≤ 1, and if the median indicated agreement (Mdn ≥ 6) (6= appropriate, 7 = totally appropriate). Following three rounds, the final content of the educational program is composed of 23 educational objectives organized in 13 themes with an agreement about relevance amongst all participants (Mdn ≥ 6; IQR ≤ 1). The comprehensiveness criterion received also a consensus (IQR ≤ 1). The participants ranked the theme "normal swallowing vs difficulty swallowing" as the highest priority.
This Delphi study resulted in a consensus, on the content of a competency framework of an educational program for OD patients and their ICGs. Further steps are needed to construct learning activities based on these objectives before testing their feasibility and efficacy.
... The Delphi technique is used to gather information from individuals within their domain of expertise through a series of questionnaires and structured feedback cycles that provide individuals the opportunity to understand the group's perspective and adjust their views accordingly (Okoli & Pawlowski, 2004). This method is often used to identify or prioritize issues in a specific domain, ends in group consensus (Hsu & Sandford, 2007;Okoli & Pawlowski, 2004) and modified for use in health services research and policy (Coller et al., 2020;Degeling et al., 2019;Hasson & Keeney, 2011;Khodyakov et al., 2020;Powell et al., 2015;Roux et al., 2018). ...
... However, our response rate for Round 3 (53.6%) is within the range of other Delphi studies with three rounds (between 45 and 93%) (Gargon et al., 2019). This is still an adequate sample size for achieving consensus (Hsu & Sandford, 2007) given that the final sample of 23 was still representative of the initial sample of experts (e.g., 50% advocates, 11 out of 12 states represented). Additionally, of our initial list of 45 experts, 28 experts (62%) agreed to participate and we did not have a policymaker or advocate representing each of our twelve states. ...
To identify the state-level policies and policy domains that state policymakers and advocates perceive as most important for positively impacting the use of children’s mental health services (CMHS). We used a modified Delphi technique (i.e., two rounds of questionnaires and an interview) during Spring 2021 to elicit perceptions among state mental health agency officials and advocates (n = 28) from twelve states on state policies that impact the use of CMHS. Participants rated a list of pre-specified policies on a 7-point Likert scale (1 = not important, 7 = extremely important) in the following policy domains: insurance coverage and limits, mental health services, school and social. Participants added nine policies to the initial list of 24 policies. The “school” policy domain was perceived as the most important, while the “social” policy domain was perceived as the least important after the first questionnaire and the second most important policy domain after the second questionnaire. The individual policies perceived as most important were school-based mental health services, state mental health parity, and Medicaid reimbursement rates. Key stakeholders in CMHS should leverage this group of policies to understand the current policy landscape in their state and to identify gaps in policy domains and potential policy opportunities to create a more comprehensive system to address children’s mental health from a holistic, evidence-based policymaking perspective.
... In group decisions, a wider range of knowledge may be drawn upon, the interaction between group members can stimulate and challenge received ideas, and idiosyncrasies may be filtered out through the group prioritisation process [19,. The use of structured, systematic approaches to reach consensus is supported by the observation that, in an unstructured group meeting, there is the risk of a single individual dominating the discussion and decisions may be portrayed as unanimous when, in reality, there is dissent within the group . ...
... In group decisions, a wider range of knowledge may be drawn upon, the interaction between group members can stimulate and challenge received ideas, and idiosyncrasies may be filtered out through the group prioritisation process [19,. The use of structured, systematic approaches to reach consensus is supported by the observation that, in an unstructured group meeting, there is the risk of a single individual dominating the discussion and decisions may be portrayed as unanimous when, in reality, there is dissent within the group . Even within structured consensus meetings, depending on their roles, a few panel members can dominate the discussion . ...
Structured, systematic methods to formulate consensus recommendations, such as the Delphi process or nominal group technique, among others, provide the opportunity to harness the knowledge of experts to support clinical decision making in areas of uncertainty. They are widely used in biomedical research, in particular where disease characteristics or resource limitations mean that high-quality evidence generation is difficult. However, poor reporting of methods used to reach a consensus – for example, not clearly explaining the definition of consensus, or not stating how consensus group panellists were selected – can potentially undermine confidence in this type of research and hinder reproducibility. Our objective is therefore to systematically develop a reporting guideline to help the biomedical research and clinical practice community describe the methods or techniques used to reach consensus in a complete, transparent, and consistent manner.
The ACCORD (ACcurate COnsensus Reporting Document) project will take place in five stages and follow the EQUATOR Network guidance for the development of reporting guidelines. In Stage 1, a multidisciplinary Steering Committee has been established to lead and coordinate the guideline development process. In Stage 2, a systematic literature review will identify evidence on the quality of the reporting of consensus methodology, to obtain potential items for a reporting checklist. In Stage 3, Delphi methodology will be used to reach consensus regarding the checklist items, first among the Steering Committee, and then among a broader Delphi panel comprising participants with a range of expertise, including patient representatives. In Stage 4, the reporting guideline will be finalised in a consensus meeting, along with the production of an Explanation and Elaboration (E&E) document. In Stage 5, we plan to publish the reporting guideline and E&E document in open-access journals, supported by presentations at appropriate events. Dissemination of the reporting guideline, including a website linked to social media channels, is crucial for the document to be implemented in practice.
The ACCORD reporting guideline will provide a set of minimum items that should be reported about methods used to achieve consensus, including approaches ranging from simple unstructured opinion gatherings to highly structured processes.
... The Delphi study was used to assess the feasibility and seek consensus from a group of experts to validate the designed intervention. 12 A modified version of the traditional Delphi was used as depicted in Figure II. In the first round, the designed implementation framework was presented to the panel of experts to elicit consensus compared to the traditional first round that includes questions or expert opinions in order to gain insight into a specific issue. ...
... There are no criteria upon which to determine the nature of the "expert", the optimal panel size, or even selection criteria of the panellists in a Delphi study. 12 For this study, the experts -dental academics with experience in curriculum development, design, or as programme accreditorswere purposefully selected. These experts were invited from all four dental schools in South Africa. ...
The aim of this study was to test the feasibility of
an implementation framework for an undergraduate
competency framework by means of a Delphi study.
A modified Delphi research design consisting of two rounds
was conducted and aimed to seek consensus defined as
agreement of 70% or more from a group of experts. Twentytwo
(22) dental experts were purposefully selected from
the four dental schools in South Africa. An online survey
consisting mainly of closed-ended questions was used and
analysed quantitatively. In addition, the second part of each
question included a comment section to allow participants
to elaborate on their answer. Comments were considered
as adapted statements for the second round of the Delphi.
Thirteen (13) participants completed the first round, and 10
participants completed the second round of the Delphi study.
Following the first round of the Delphi, consensus (70%) was
reached for five of the seven statements. These statements
focus on the impact of internal and external stakeholders
on the dental schools, leadership support for faculty
development, teaching and learning changes, and curriculum
mapping. Statements that needed further consideration
in round 1 included impact of institutional framework and
diversity of academics. Consensus was reached for both the
diversity of academics statements in round 2.
The results suggest that the proposed framework to align
and implement the AfriMEDS core competency framework
was feasible. However, from the comments of the dental
experts there seems to be no common understanding of
implementation and alignment of the institutional frameworks
and the AfriMEDS core competency framework.
... This national study aimed to address this issue through the well-validated Delphi technique which aims to gather information from experts via an iterative process ( Hsu & Sandford, 2007 ). Initial participant responses to prompts are gathered and then in subsequent rounds, anonymized collated responses are distributed before seeking opinions again. ...
... Participants for this study were those who identified as "expert users" of simulation as defined in the literature ( Diamond et al., 2014 ;Hsu & Sandford, 2007 ). Thus they were all required to have had substantial experience of delivering simulation-based education or were key stakeholders in clinical skills training. ...
There is growing interest in utilizing structured extended periods of simulation as partial replacement for clinical placement (PRCP) due to workforce demands and COVID-19 impact. This project aimed to identify the extent to which stakeholders felt that simulated placements could reduce overall clinical placement time requirement for health care students.
Participants included a range of nurses and Allied Health Professionals including managers, clinicians and lecturers. A three-round Delphi study was conducted with a 75% consensus agreement target.
Overall, 71 participants engaged with at least one round of the study and identified benefits of PRCP: Safety, learning from mistakes, improved preparation, and reduced placement training burden. Challenges included poor realism, logistics, time, and access to facilities.
The consensus opinion indicated that between 11% and 30% of clinical training time could be replaced with simulated placement. Ongoing work will need to identify profession-specific consensus opinion and guidance on the use of simulation as PRCP.
Cite this article:
Bridge, P., et al. (Year, Month). Simulated placements as partial replacement of clinical training time: a Delphi consensus study. Clinical Simulation in Nursing, vol(no.), eXXX-eXXX. doi: 10.1016/j.ecsn.20XX.
... By applying participatory approaches in presence meetings, (e.g., workshops, sessions, panels), there may occur strategic, tactical, social, and psychological issues in the decision modeling process faced by individuals . Negative effects such as the dominance of stakeholders , strategic answers by stakeholders , and the groupthink phenomenon  have been observed. ...
... There are structured communication techniques, such as the Delphi method , which aim to reduce negative group effects by employing repeated questionnaires and aggregating facilitators for achieving consensus. Viera et al.  have proposed and evaluated a combination of the Delphi method and additional methods, such as decision conferencing to open the design of MCDA tools to a large number of participants. ...
Multi-criteria decision analysis (MCDA) is an established methodology to support the decision-making of multi-objective problems. For conducting an MCDA, in most cases, a set of objectives (SOO) is required, which consists of a hierarchical structure comprised of objectives, criteria, and indicators. The development of an SOO is usually based on moderated development processes requiring high organizational and cognitive effort from all stakeholders involved. This article proposes elementary interactions as a key paradigm of an algorithm-driven development process for an SOO that requires little moderation efforts. Elementary interactions are self-contained information requests that may be answered with little cognitive effort. The pairwise comparison of elements in the well-known analytical hierarchical process (AHP) is an example of an elementary interaction. Each elementary interaction in the development process presented contributes to the stepwise development of an SOO. Based on the hypothesis that an SOO may be developed exclusively using elementary interactions (EIs), a concept for a multi-user platform is proposed. Essential components of the platform are a Model Aggregator, an Elementary Interaction Stream Generator, a Participant Manager, and a Discussion Forum. While the latter component serves the professional exchange of the participants, the first three components are intended to be automatable by algorithms. The platform concept proposed has been evaluated partly in an explorative validation study demonstrating the general functionality of the algorithms outlined. In summary, the platform concept suggested demonstrates the potential to ease SOO development processes as the platform concept does not restrict the application domain; it is intended to work with little administration moderation efforts, and it supports the further development of an existing SOO in the event of changes in external conditions. The algorithm-driven development of SOOs proposed in this article may ease the development of MCDA applications and, thus, may have a positive effect on the spread of MCDA applications.
... The reason is also that a few review articles available to introduce to significance of Delphi use in LIS. Doubts made by researchers who use the Delphi strategy join that the contemplations created by experts are generally the best musings, that the examiner can lead this assessment without carrying precarious inclination into the iterative association, and that sufficient response rate can be cultivated for this kind of study (Hsu and Sandford, 2007). 1 There is some inspiration to acknowledge experts may not actually produce the best clever contemplations (Welty, 1972). 2 As found in industry and entertainment, as often as possible an outsider's perspective can truly convey the most supportive novel musings. In any case, perceiving potential untouchables that would be prepared for giving important analysis is fundamentally more irksome than recognizing the experts in a given region. ...
... Doubts made by researchers who use the Delphi strategy join that the contemplations created by experts are generally the best musings, that the examiner can lead this assessment without carrying precarious inclination into the iterative association, and that sufficient response rate can be cultivated for this kind of study (Hsu and Sandford, 2007). 1 There is some inspiration to acknowledge experts may not actually produce the best clever contemplations (Welty, 1972). 2 As found in industry and entertainment, as often as possible an outsider's perspective can truly convey the most supportive novel musings. In any case, perceiving potential untouchables that would be prepared for giving important analysis is fundamentally more irksome than recognizing the experts in a given region. ...
This study is made to find out the important factors of DSapce relevant to implementation in the academic libraries based on the experts’ opinion view. There is lack of use the MCDM techniques in the library field. In this study authors used the Delphi technique to validate the factors which are important for implementing the software. The different studies related to using MCDM are studied as review of literature. The findings of the study provide the priority of the factors through using the Delphi. The results will be helpful as theoretical and managerial perspective to take decision regarding the implementation of the software.
... During the final/third round of the Delphi procedure, panelists were asked to review their statement ratings compared to the group's mean and decide whether they would prefer to change or maintain their original ratings. If warranted, panelists could also provide reasoning for their support of specific statements especially if they were unfavorable amongst the group (i.e., rated <7 overall) (Hsu & Sandford, 2007;Hyslop-Margison & Armstrong, 2004;Ludwig, 1997). Statements maintaining a mean score >7 (Kroshus et al., 2018;Zumsteg et al., 2012) were considered to achieve consensus amongst the panel. ...
... Panelists also provided reasoning for their decision and/or additional edits/suggestions (Hsu & Sandford, 2007;Hyslop-Margison & Armstrong, 2004;Ludwig, 1997). Following the final round, statements maintaining a mean score > 7 (Kroshus et al., 2018;Zumsteg et al., 2012) were considered to achieve consensus. ...
Concussions affect millions of young adults annually, many of whom are enrolled in post-secondary education. To date, concussion research has yet to establish evidence-based protocols to assist university students in returning to the classroom (i.e., return to learn [RTL]) after concussion. Without clear guidelines, students are left to independently navigate the complex medical and university systems. In order to understand and address barriers to receiving and implementing post-concussion academic supports within higher education, this dissertation was conducted using an exploratory-sequential mixed methods approach comprised of three main aims: (1) to collate the experiences of academic stakeholders and previously concussed university students; (2) to develop the Post-Concussion Collegiate RTL Protocol and additional concussion management recommendations for higher education through a multidisciplinary Delphi procedure; and (3) to assess perceptions surrounding implementation of the novel RTL Protocol among clinicians and academic stakeholders across the Power 5 Conferences. The first investigation was conducted using semi-structured interviews among previously concussed undergraduate students (n=21) and university academic stakeholders (n=7). Using a Grounded Theory approach, several emerging themes arose among the two cohorts. Specifically, students recalled empathetic interactions with their instructors; however, academic stakeholders indicated they lacked sufficient information and/or resources on how to appropriately develop academic supports to facilitate the RTL process for students recovering from concussion. The second study sought to mediate the discrepancies in post-concussion RTL support within higher education by developing concussion management guidelines through a modified Delphi procedure. The Delphi panel was comprised of twenty-two multidisciplinary professionals (n=9 clinicians, n=8 researchers, n=5 university academic stakeholders). The iterative process resulted in consensus across four main categories including: recommendations for medical discharge documentation, procedures for developing and maintaining a multidisciplinary RTL approach within post-secondary institutions, processes for university students to obtain academic supports following concussion, as well as the development of the novel Post-Concussion Collegiate RTL Protocol. The third project explored perceptions of prospective utility of the Post-Concussion Collegiate RTL Protocol among clinicians and university faculty/staff within the Power 5 Conferences (Atlantic Coast Conference, Southeastern Conference, Big 10 Conference, Big 12 Conference, and the Pacific 12 Conference). Participants (N=49; n=25 clinicians, n=24 university faculty/staff) completed an electronic survey comprised of four validated implementation outcome measures including the Acceptability of Implementation Measure (AIM), Implementation Appropriateness Measure (IAM), Feasibility of Implementation Measure (FIM), and Organizational Readiness for Implementing Change (ORIC). Mann-Whitney U analyses yielded no significant between-group differences on the protocol’s acceptability, appropriateness, and feasibility in their setting. However, group differences within the change commitment and change efficacy subscales of the ORIC were noted, with university faculty/staff indicating lower organizational readiness for change in contrast to clinicians within the Power 5 Conferences. Collectively, these findings serve as a framework for future university policy to standardize the RTL process for students who require academic support after concussion. Additional research is warranted to investigate university infrastructure to develop implementation pathways to support the Post-Concussion Collegiate RTL Protocol across all institutions of higher learning.
... See Table 1 for eligibility criteria for each group. The current study aimed to recruit a sample of at least 15 participants per group to increase the likelihood a representative pool of opinions (Hsu and Sandford 2007). ...
Complex Regional Pain Syndrome (CRPS) is a rare but disabling pain condition. Accurate and timely education about CRPS is key to promote optimal clinical outcomes, but it is unclear what the content of that education should be. We aimed to determine the content that both people with CRPS and expert health care professionals (HCPs) reported as important..
An international three-round e-Delphi was conducted, recruiting adults diagnosed with CRPS and HCPs. In Round 1, participants were asked to list the most important information people with CRPS should know regarding the condition. Data were organised into concepts and allocated to themes. In Rounds 2 and 3, participants rated each concept on a 9-point Likert Scale, categorised as 'not important' (0-3), 'important' (4-6) and 'very important' (7-9). A concept attained consensus when >75% agreement was reached within a category.
Sixty-two participants (HCPs: n=7; CRPS: n=55), proposed 193 concepts in Round 1, resulting in 22 themes. Fifteen additional concepts were identified in Round 2, resulting in a total of 208 concepts. From that list, 48 concepts that emphasised understanding and evidence-based management of the disorder, the importance of self-management strategies, pacing and movement, reached joint consensus as 'very important'. One concept: "Advise that movement does not help" reached joint consensus as 'not important'.
Forty-eight concepts were jointly considered 'very important' for future CRPS-related educational content. Future research to better understand group differences and to canvas a broader HCP group is warranted.
... Eleven respondents completed two rounds. 71 Three options achieved consensus: Subsequently, the wider group of BASP and BSNR members (n = 64) assigned the highest approval ranking for transferring LAO stroke patients to the nearest neuroscience centre for thrombectomy based on the results of local CT/computed tomography angiography (CTA) (option 2). ...
Intravenous thrombolysis and intra-arterial thrombectomy are proven emergency treatments for acute ischaemic stroke, but they require rapid delivery to selected patients within specialist services. National audit data have shown that treatment provision is suboptimal.
The aims were to (1) determine the content, clinical effectiveness and day 90 cost-effectiveness of an enhanced paramedic assessment designed to facilitate thrombolysis delivery in hospital and (2) model thrombectomy service configuration options with optimal activity and cost-effectiveness informed by expert and public views.
A mixed-methods approach was employed between 2014 and 2019. Systematic reviews examined enhanced paramedic roles and thrombectomy effectiveness. Professional and service user groups developed a thrombolysis-focused Paramedic Acute Stroke Treatment Assessment, which was evaluated in a pragmatic multicentre cluster randomised controlled trial and parallel process evaluation. Clinicians, patients, carers and the public were surveyed regarding thrombectomy service configuration. A decision tree was constructed from published data to estimate thrombectomy eligibility of the UK stroke population. A matching discrete-event simulation predicted patient benefits and financial consequences from increasing the number of centres.
The paramedic assessment trial was hosted by three regional ambulance services (in north-east England, north-west England and Wales) serving 15 hospitals.
A total of 103 health-care representatives and 20 public representatives assisted in the development of the paramedic assessment. The trial enrolled 1214 stroke patients within 4 hours of symptom onset. Thrombectomy service provision was informed by a Delphi exercise with 64 stroke specialists and neuroradiologists, and surveys of 147 patients and 105 public respondents.
The paramedic assessment comprised additional pre-hospital information collection, structured hospital handover, practical assistance up to 15 minutes post handover, a pre-departure care checklist and clinician feedback.
Main outcome measures
The primary outcome was the proportion of patients receiving thrombolysis. Secondary outcomes included day 90 health (poor status was a modified Rankin Scale score of > 2). Economic outputs reported the number of cases treated and cost-effectiveness using quality-adjusted life-years and Great British pounds.
National registry data from the Sentinel Stroke National Audit Programme and the Scottish Stroke Care Audit were used.
Systematic searches of electronic bibliographies were used to identify relevant literature. Study inclusion and data extraction processes were described using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.
The paramedic assessment trial found a clinically important but statistically non-significant reduction in thrombolysis among intervention patients, compared with standard care patients [197/500 (39.4%) vs. 319/714 (44.7%), respectively] (adjusted odds ratio 0.81, 95% confidence interval 0.61 to 1.08; p = 0.15). The rate of poor health outcomes was not significantly different, but was lower in the intervention group than in the standard care group [313/489 (64.0%) vs. 461/690 (66.8%), respectively] (adjusted odds ratio 0.86, 95% confidence interval 0.60 to 1.2; p = 0.39). There was no difference in the quality-adjusted life-years gained between the groups (0.005, 95% confidence interval –0.004 to 0.015), but total costs were significantly lower for patients in the intervention group than for those in the standard care group (–£1086, 95% confidence interval –£2236 to –£13). It has been estimated that, in the UK, 10,140–11,530 patients per year (i.e. 12% of stroke admissions) are eligible for thrombectomy. Meta-analysis of published data confirmed that thrombectomy-treated patients were significantly more likely to be functionally independent than patients receiving standard care (odds ratio 2.39, 95% confidence interval 1.88 to 3.04; n = 1841). Expert consensus and most public survey respondents favoured selective secondary transfer for accessing thrombectomy at regional neuroscience centres. The discrete-event simulation model suggested that six new English centres might generate 190 quality-adjusted life-years (95% confidence interval –6 to 399 quality-adjusted life-years) and a saving of £1,864,000 per year (95% confidence interval –£1,204,000 to £5,017,000 saving per year). The total mean thrombectomy cost up to 72 hours was £12,440, mostly attributable to the consumables. There was no significant cost difference between direct admission and secondary transfer (mean difference –£368, 95% confidence interval –£1016 to £279; p = 0.26).
Evidence for paramedic assessment fidelity was limited and group allocation could not be masked. Thrombectomy surveys represented respondent views only. Simulation models assumed that populations were consistent with published meta-analyses, included limited parameters reflecting underlying data sets and did not consider the capital costs of setting up new services.
Paramedic assessment did not increase the proportion of patients receiving thrombolysis, but outcomes were consistent with improved cost-effectiveness at day 90, possibly reflecting better informed treatment decisions and/or adherence to clinical guidelines. However, the health difference was non-significant, small and short term. Approximately 12% of stroke patients are suitable for thrombectomy and widespread provision is likely to generate health and resource gains. Clinician and public views support secondary transfer to access treatment.
Further evaluation of emergency care pathways will determine whether or not enhanced paramedic assessment improves hospital guideline compliance. Validation of the simulation model post reconfiguration will improve precision and describe wider resource implications.
This trial is registered as ISRCTN12418919 and the systematic review protocols are registered as PROSPERO CRD42014010785 and PROSPERO CRD42015016649.
The project was funded by the National Institute for Health and Care Research (NIHR) Programme Grants for Applied Research programme and will be published in full in Programme Grants for Applied Research ; Vol. 10, No. 4. See the NIHR Journals Library website for further project information.
... In addition to the analytics obtained from the literature review, the Delphi technique was used to find lists and themes on which experts agreed on learning analytics indicators. Delphi is a widely accepted technique used to make predictions for the future, to reveal expert opinions, and to reach consensus (Hsu & Sandford, 2007), and is often resorted to to forming and making common minds and decisions, especially on complex issues (Linstone and Turoff, 1975). In this study, quantitative and qualitative research processes were used as part of the Delphi technique. ...
Öğrenme analitikleri eğitim ortamlarının ve paydaşlarının objektif ölçümlerle anlaşılabilmesine imkân sağlaması, bu ortamlarda yaşanan sorunlara çözüm sunması ve bu ortamların daha verimli hale getirilmesine yönelik yeni yaklaşımlar sunması nedeniyle üzerine dikkatleri çeken bir konu haline gelmiştir. Öğrenme analitiği göstergeleri ise, henüz öğrenme problemleri ile ilişkilendirilmemiş, ancak e-öğrenme ortamlarında kullanılabilecek türde analitiklerdir. Bir bakıma öğrenme analitiği adaylarıdır. Bu çalışmanın amacı gelişim aşamasında görülen öğrenme analitiklerinin temellerine değinmenin yanı sıra, araştırmacı ve uygulamacılara e-öğrenme ortamlarında kullanabilecekleri öğrenme analitiği göstergelerine ilişkin bir liste sunmaktır. Çalışmada, alanyazın taramasından elde edilen göstergelerin yanı sıra Delphi tekniği kullanılarak uzmanların fikir birliğine vardıkları göstergelere de ulaşılmıştır. Çalışma için geliştirilen çevrimiçi platform aracılığıyla yürütülen Delphi panelleri yaklaşık 1 yıl sürmüş, 22 uzman ile başlayan süreç 11 uzman ile 4 tur sonunda tamamlanmıştır. Çalışmada demografik, betimsel ve algoritmik üst başlıklarında toplam 41 maddeden oluşan öğrenme analitiği göstergesine ulaşılmıştır. Bu listenin gerek öğrenme analitiklerini kullanarak öğrenme ortamlarında nesnel kararlar almaya çalışan uygulayıcılara gerekse de öğrenme analitiği ile ilgili konularda çalışmayı düşünen araştırmacılara katkı sağlayacağı düşünülmektedir.
... In [10,11,12,13,14] are defined different types of criteria to choose who can be an expert. After analyzing, there is a conclusion that every research is unique and criteria also. ...
In this work one example of civil engineering research conducted with the Delphi method is shown. The Delphi method is one of the most popular scientific methods for research of problems with lack of known data. For this research, researcher must form the team of experts and analyze their answers on questions asked during the research with controlled feedback. In this article researcher wanted to prove hypothesis for making risk assessment model for planning and design processes of wastewater treatment plants. This research was conducted with Classical Delphi method is shown step by step.
... Previously published algorithms lack detail across the range of LSS symptoms and treatment options and/ or have been developed without consensus . The consensus process is underpinned by the assumption that the opinions of many outweigh those of the individual . In the absence of consistent level-1 evidence in the research literature, our modified Delphi approach leveraged the combined expertise and opinions of international spine researchers and clinicians, providing validity for the proposed algorithm as a framework to guide clinical care . ...
Lumbar spinal stenosis (LSS) is a common degenerative spinal condition in older adults associated with disability, diminished quality of life, and substantial healthcare costs. Individual symptoms and needs vary. With sparse and sometimes inconsistent evidence to guide clinical decision-making, variable clinical care may lead to unsatisfactory patient outcomes and inefficient use of healthcare resources.
A three-phase modified Delphi study comprising four consensus rounds was conducted on behalf of the International Taskforce for the Diagnosis and Management of LSS to develop a treatment algorithm based on multi-professional international expert consensus. Participants with expertise in the assessment and management of people with LSS were invited using an international distribution process used for two previous Delphi studies led by the Taskforce. Separate treatment pathways for patients with different symptom types and severity were developed and incorporated into a proposed treatment algorithm through consensus rounds 1 to 3. Agreement with the proposed algorithm was evaluated in the final consensus round.
The final algorithm combines stratified and stepped approaches. When indicated, immediate investigation and surgery is advocated. Otherwise, a stepped approach is suggested when self-directed care is unsatisfactory. This starts with tailored rehabilitation, then more complex multidisciplinary care, investigations and surgery options if needed. Treatment options in each step depend on clinical phenotype and symptom severity. Treatment response guides pathway entrance and exit points. Of 397 study participants, 86% rated their agreement ≥ 4 for the proposed algorithm on a 0–6 scale, of which 22% completely agreed. Only 7% disagreed. Over 70% of participants felt that the algorithm would be useful for clinicians in public healthcare (both primary care and specialist settings) and in private healthcare settings, and that a simplified version would help patients in shared decision-making.
International and multi-professional agreement was achieved for a proposed LSS treatment algorithm developed through expert consensus . The algorithm advocates different pathway options depending on clinical indications. It is not intended as a treatment protocol and will require evaluation against current care for clinical and cost-effectiveness. It may, however, serve as a clinical guide until evidence is sufficient to inform a fully stratified care model.
... We conducted a mixed-methods study based on a Delphi technique to facilitate consensus within a panel of experts. 13 The Delphi technique is based on structured, iterative, anonymous surveys where participants can rate and comment on statements. Between each iteration, researchers may report feedback to the panellists to facilitate the discussion. ...
To compare experts’ perceived usefulness of audit filters from Ghana, Cameroon, WHO and those locally developed; generate context-appropriate audit filters for trauma care in selected hospitals in urban India; and explore characteristics of audit filters that correlate to perceived usefulness.
A mixed-methods approach using a multicentre online Delphi technique.
Two large tertiary hospitals in urban India.
Filters were rated on a scale from 1 to 10 in terms of perceived usefulness, with the option to add new filters and comments. The filters were categorised into three groups depending on their origin: low and middle-income countries (LMIC), WHO and New (locally developed), and their scores compared. Significance was determined using Kruskal-Wallis test followed by Wilcoxon rank-sum test. We performed a content analysis of the comments.
26 predefined and 15 new filter suggestions were evaluated. The filters had high usefulness scores (mean overall score 9.01 of 10), with the LMIC filters having significantly higher scores compared with those from WHO and those newly added. Three themes were identified in the content analysis relating to medical relevance , feasibility and specificity .
Audit filters from other LMICs were deemed highly useful in the urban India context. This may indicate that the transferability of defined trauma audit filters between similar contexts is high and that these can provide a starting point when implemented as part of trauma quality improvement programmes in low-resource settings.
... The Delphi method involves a series of rounds to achieve consensus in which different activities will occur at each round. Care and attention are crucial to develop the initial broad question that is the Delphi's focus since if respondents fail to comprehend the question, there is a possibility for them to give unsuitable answers and/or become irritated (Hsu & Sandford, 2007). Therefore, in this research, Delphi Round 1 was conducted to brainstorm. ...
Purpose Globalisation and a significant reliance on technology for business management are on the rise in the twenty-first century. Organisations are growing increasingly exposed to both external and internal issues, making business more difficult. Drawing upon the system theory and self determination theory , this study examines the underlying theoretical mechanism between complex behaviour that emerges as a result of interactions among employees and the orginasations. Design/methodology/approach Employees of the Malaysian Digital Economic Corporation Sdn. Bhd (MDEC) registered .'s organisations were chosen as samples for this study .A total of 171 employees from these companies completed a survey. The data were analysed using PLS-SEM to discover employees collective behaviour capacities positively influenced the organisational capacity of MSC status companies. Findings The findings of the present study have validated the empirical model, showing a significant relationship in between the proposed constructs. Employees are an important commodity for a company, not only in leadership but also in survival. There is a positive set of behavioural capitals which allow the company to develop as a key agent of national economic advancement. The internal social structure influences resilience as a capacity. Thus the theoretical insight explain how organisations develop their internal capabilities as a capacity for resilience in the emerging market context of Malaysian MSCs Originality/value The study is first of its kind which has tried to investigate an exhaustive direct relationship model in the MSC sector. The study has postulated that it is not just the organizational resilience content but the perceptions of employees toward enhancing organizational resilience. The solution is based on an organisation's collective behavioral capabilities and internal organisational processes that connects the employees towards enhancing organizational resilience .
... We used modified Delphi methodology , including three sequential rounds of online questionnaires (referred to hereon as Survey 1, 2, and 3). We selected Delphi methodology because it is ideal to engage a large audience of geographically diverse, expert stakeholders , it is appropriate to establish consensus in health research , and it has been used previously to validate a variety of clinical support tools . ...
Oncology guidelines recommend participation in cancer rehabilitation or exercise services (CR/ES) to optimize survivorship. Yet, connecting the right survivor, with the right CR/ES, at the right time remains a challenge. The Exercise in Cancer Evaluation and Decision Support (EXCEEDS) algorithm was developed to enhance CR/ES clinical decision-making and facilitate access to CR/ES. We used Delphi methodology to evaluate usability, acceptability, and determine pragmatic implementation priorities.
Participants completed three online questionnaires including (1) simulated case vignettes, (2) 4-item acceptability questionnaire (0–5 pts), and (3) series of items to rank algorithm implementation priorities (potential users, platforms, strategies). To evaluate usability, we used Chi-squared test to compare frequency of accurate pre-exercise medical clearance and CR/ES triage recommendations for case vignettes when using EXCEEDS vs. without. We calculated mean acceptability and inter-rater agreement overall and in 4 domains. We used the Eisenhower Prioritization Method to evaluate implementation priorities.
Participants (N = 133) mostly represented the fields of rehabilitation (69%), oncology (25%), or exercise science (17%). When using EXCEEDS (vs. without), their recommendations were more likely to be guideline concordant for medical clearance (83.4% vs. 66.5%, X² = 26.61, p < .0001) and CR/ES triage (60.9% vs. 51.1%, X² = 73.79, p < .0001). Mean acceptability was M = 3.90 ± 0.47; inter-rater agreement was high for 3 of 4 domains. Implementation priorities include 1 potential user group, 2 platform types, and 9 implementation strategies.
This study demonstrates the EXCEEDS algorithm can be a pragmatic and acceptable clinical decision support tool for CR/ES recommendations. Future research is needed to evaluate algorithm usability and acceptability in real-world clinical pathways.
... The grounded theory approach was used in this study because it most appropriately addressed the need for an emergent theory construction based on data representing best practices from ITP managers lived experiences. Delphi design also involves close engagement between the researcher and the experts who are the participants; however, the aim would not be to construct a theory, but to arrive on a consensus regarding a difficult question in the field (Hsu & Sandford, 2007). Hence, the Delphi design was not adequate for this study. ...
Globally, most information technology projects (ITPs) are reported as unsuccessful. Poor project management practices have consistently been identified as the leading cause of ITP failures. However, ITP practitioners manage project processes in diverse ways without clear guiding principles in terms of what does or does not work in practice for success. Process management practices in projects were explored in this grounded theory qualitative study from a systems theory perspective. The purpose was to understand from project practitioners' experiences what guiding principles potentially influenced ITPs to success. These experiences were then analyzed to develop a theory describing how to best use process management in projects to achieve success. The main research question addressed in this study examined participants’ view of practices in successful ITPs that effectively led to success. The data were collected during in-depth interviews of 14 project participants using semistructured questions and were coded using the grounded theory continuous-comparison approach until theoretical saturation and themes were generated. The finding is an emergent theory, which indicates that practices in ITP process management consisting of continuous learning, regular engagement, constant surveillance, process orchestration, and timely response positively impacts a successful outcome. Leveraging this finding, process management principles are recommended to better understand ITP process management in practice. This study contributes to positive social change by providing a guide for practitioners’ use, potentially resulting in more successful educational and healthcare ITPs, especially in Africa.
... Its development process would have been more thorough if, in addition to the literature search, we had performed a needs assessment and prioritization for the different actors involved in order to optimize efficient collaboration. A more rigorous validation process using the Delphi Method (designed to reach a consensus among different actors taking into account their contexts, perspectives, and experiences in biomedical research and community-based primary health care) can mitigate this limitation in the future . However, it is more appropriate to address these weaknesses in an empirical study. ...
Translational research aims at reducing gaps between fundamental scientific discoveries and real-world applications. However, the trajectory of most scientific discoveries along the translation research continuum remains highly complex. Logic models are powerful tools that can help reduce this complexity. They are often used to lay out road maps and depict the relationship between activities and their intended effects. Few if any existing tools have been designed to guide the implementation and evaluation of collaborative models between community-based primary health care and biomedical research. To address this gap, we developed a logic model in two stages: 1) a literature review; and 2) the drafting and revision of the model by experts in the field. We describe its components, including objectives, inputs, activities, target groups, outputs, and results for a collaborative model involving fundamental biomedical research and primary health care practices. Our proposed logic model provides a road map that has the potential to reduce the complexity faced by translational research in chronic diseases by providing guidelines for decision-makers. Future work should attempt to validate the model before its broad-based implementation.
... The Delphi technique is based on statements from a panel of experts in the field to be developed. The panel expresses its views in a series of questionnaires until consensus is achieved . In particular, the technique has been used extensively for the development of education, but there only are a handful of reviews on the approach [38,40,42,43]. ...
After completing university education, biomedical laboratory scientists work in clinical laboratories, in biomedical research laboratories, in biotech, and in pharmaceutical companies. Laboratory diagnostics have undergone rapid development over the recent years, with the pace showing no signs of abatement. This rapid development challenges the competence of the staff and will most certainly influence the education of future staff. This study aimed to examine what was considered the necessary competencies needed to pursue a career as a biomedical laboratory scientist.
A modified Delphi technique was used, with the panel of experts expressing their views in a series of three questionnaire. Consensus was defined as the point which 75 % or more of the panel participants agreed that a particular competency was necessary.
The study highlights the perceived importance of mostly generic competencies that relate to quality, quality assurance, and accuracy, as well as different aspects of safety, respect, trustworthiness (towards patients/clients and colleagues), and communication skills. The results also stress the significance of self-awareness and professionality.
We identified important competencies for biomedical laboratory scientists. Together with complementary information from other sources, i.e., guidelines, laws, and scientific publications, the competencies identified can be used as learning outcomes in a competency-based education to provide students with all the competencies needed to work as professional biomedical laboratory scientists.
... In general, for items ranked on a five-point Likert scale, the consensus rule is 80%, 42 but there can be differences in how adjacent items are grouped or weighted toward consensus. 43 For instance, Naughton et al 44 quantified the Likert points from 1 (most important) to 5 (least important), and defined consensus for items scoring a median of 2.5 or less overall, when at least 80% of responders gave 1-3 points. More recent works proposed entropy-based consensus. ...
While there are guidelines for reporting on observational studies (eg, Strengthening the Reporting of Observational Studies in Epidemiology, Reporting of Studies Conducted Using Observational Routinely Collected Health Data Statement), estimation of causal effects from both observational data and randomised experiments (eg, A Guideline for Reporting Mediation Analyses of Randomised Trials and Observational Studies, Consolidated Standards of Reporting Trials, PATH) and on prediction modelling (eg, Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis), none is purposely made for deriving and validating models from observational data to predict counterfactuals for individuals on one or more possible interventions, on the basis of given (or inferred) causal structures. This paper describes methods and processes that will be used to develop a Reporting Guideline for Causal and Counterfactual Prediction Models (PRECOG).
Methods and analysis
PRECOG will be developed following published guidance from the Enhancing the Quality and Transparency of Health Research (EQUATOR) network and will comprise five stages. Stage 1 will be meetings of a working group every other week with rotating external advisors (active until stage 5). Stage 2 will comprise a systematic review of literature on counterfactual prediction modelling for biomedical sciences (registered in Prospective Register of Systematic Reviews). In stage 3, a computer-based, real-time Delphi survey will be performed to consolidate the PRECOG checklist, involving experts in causal inference, epidemiology, statistics, machine learning, informatics and protocols/standards. Stage 4 will involve the write-up of the PRECOG guideline based on the results from the prior stages. Stage 5 will seek the peer-reviewed publication of the guideline, the scoping/systematic review and dissemination.
Ethics and dissemination
The study will follow the principles of the Declaration of Helsinki. The study has been registered in EQUATOR and approved by the University of Florida’s Institutional Review Board (#202200495). Informed consent will be obtained from the working groups and the Delphi survey participants. The dissemination of PRECOG and its products will be done through journal publications, conferences, websites and social media.
... The second considered the ratings of the importance within the management of marinas, valuing the relevance using the following scale: insignificant (1), scarce (2), low (3), moderate (4), remarkable (5), high (6), or extreme (7). The significant statistics used in the Delphi method are measures of central tendency and the level of dispersion . Thus, descriptive statistics of the ratings considered were as follows: rating median (Md), standard deviation (SD), and quartile deviation (IQR). ...
Marinas are maritime features related to nautical tourism. The contemplation of pleasant surroundings acquires great importance in achieving this leisure character. The European Landscape Convention undertakes the necessity of integrating landscape into the planning policies. Thus, the marina’s management decision-making processes should reflect this awareness of the landscape. However, the landscape evaluation has not been appropriately considered despite its importance. This research attempts to introduce an initial framework to evaluate this influence, highlighting the strengths and weaknesses of the different subjects. For this purpose, the most significant elements of the marina management related to the landscape were rated, both from management and landscape perspectives. Two expert panels from Spain were used: 23 experts evaluated the above elements following the Delphi method, and 17 weighted the main management activities using DHP. Results show that there is a lack of concern for the landscape. Managers tend to consider physical conditions, whereas subjective conditions are relegated to the background. In this respect, this methodology provides the first stage for the landscape/management relationship, helping managers identify the main topics and prioritize related actions
... During the second phase, we validated and refined these principles through a two-stage Delphi technique (Hsu & Sandford, 2007). The Delphi method is used as a method of forecasting technological developments by pooling expert estimates (Rescher, 1998). ...
This paper reports on ongoing research aimed at characterizing a signature pedagogy (Shulman, 2005) of technology-enhanced task-based language teaching (TETBLT). To achieve this goal, we initially identified 15 pedagogical principles and practices distinctive of TETBLT. This initial set of principles and practices were motivated by second language acquisition theories (Doughty & Long, 2003), methodological approaches in foreign language teaching (Kramsch, 2014), and state-of-the-art publications on computer-assisted language learning (Chapelle & Sauro, 2017). During the first phase of the study, we consulted an initial group of 34 experts in the field, using the Delphi technique to achieve gradual consensus about the set of principles. After analyzing the first set of responses (N = 23) to the principles, which attained a degree of agreement averaging 71% and ranging from 48% to 96%, we refined the principles incorporating the feedback received and sent out a second questionnaire, which allowed us to reach a consensus about a set of eight robust pedagogical principles for TETBLT.
... With this technique, sequential questionnaire rounds are presented to an expert panel alternated with controlled feedback on the results collected from the prior rounds. The method allows participants to modify their opinion to achieve consensus regarding a specific topic or process (18,19). Anonymity between participants is guaranteed, which prevents bias resulting from opinions from dominant participants influencing group opinion (20,21). ...
Elastic compression stocking (ECS) therapy is commonly used in patients with deep venous thrombosis (DVT) and chronic venous disease (CVD). The provision of ECS therapy is complex, and studies indicate a lack of practical guidance and suboptimal collaboration among health care professionals. We aimed to reach consensus on critical issues of ECS therapy among the involved health care professionals and patients.
A three-round modified Delphi analysis was performed in the Netherlands in which 56 health care professionals (internists, dermatologists, general practitioners, emergency room nurses, home care nurses, medical stocking suppliers, and occupational therapists) and seven patients were invited. The 21 statements included in this analysis were based on information collected from a previously conducted Functional Resonance Analysis Method and Realist Evaluation. We used 7-point Likert scale questions and a 75% threshold for consensus.
Of the 63 persons invited for this study, 59 (94%) agreed to participate and responded in the first questionnaire round; of whom 52 were health care professionals and seven were patients (five DVT and two CVD). The overall response rate for the three questionnaire rounds was 91%. After completion of the rounds, full consensus was achieved on 19 out of 21 statements. No consensus was reached on the need for a follow-up appointment for CVD patients and who should be responsible to determine the ECS type (custom-made or standard).
We identified 19 consensus-driven recommendations on treatment decisions and collaboration in ECS therapy among an interdisciplinary panel of health care professionals and patients. These recommendations form a basis for consensus-driven optimization of ECS therapy and should ideally be incorporated in a general cross-domain protocol for ECS therapy in patients with DVT and CVD.
... An expert review was provided to check content and face validity, clarity, and conciseness, as well as relevance for teaching Physical Education to make sure the test represented all facets of what it is purported to measure . In order to address this, a panel of experts participated online via Delphi techniques [40,41]. German-speaking experts were asked to provide a judgment on the coherence of the translation (5-point Likert scale), a judgment on the extent to which the item was professionally relevant, to rate the potential for inclusive Physical Education (5-point Likert scale, only for the LIRSPE), and to provide general feedback and suggestions on every single item on the scale. ...
Overcoming participation barriers of students with disabilities in physical education is
of great importance and an internationally recognized goal. Research highlights that students with disabilities have mixed feelings about their inclusion experiences in physical education. Physical education teachers often do not feel prepared to appropriately support all students. In German-speaking countries in particular, there is a strong tradition of segregation, with varying interpretations of inclusion. In this light, an instrument to reliably assess the inclusive potential of physical education is needed, thereby providing data on the efficacy of teachers’ practices. Such an assessment scale
would be important to identify barriers to inclusive physical education while providing teachers with data that could potentially enhance the learning environment. The purpose of this study was to outline initial insights into the cross-cultural translation process of the Lieberman/Brian Inclusion Rating Scale for PE in German-speaking countries. The translation process followed suggestions for transcultural validation. Expert review was provided to check content and face validity. Major item challenges centered around paraeducators, gym management, and conceptual differences regarding physical education.
... e Delphi method is an iterative process for gathering expert opinions; these experts are known to the panel members, whose only comments are published . In this study, the panel members are health experts. ...
In this study, due to the importance of emergency centers and patient transport vehicles in epidemic conditions, the performance of emergency centers has been evaluated based on health protocols. e criteria were rst divided into preventive and operational sections by collecting opinions, health experts, standard criteria, and the Delphi method. Preventive criteria for evaluating emergency centers and operating criteria for assessing vehicles in these centers are considered. e weighting of the determined criteria was done by the triangular fuzzy aggregation method. According to the standard criteria, the emergency centers have been evaluated for a 30-day period. e results have been assessed as a qualitative and quantitative matrix using the PROMETHEE method. e results showed better performance of Center A (63%) due to proper performance and better compliance with protocols in both criteria (preventive and operational). e reason for the superiority of this center over Center B can be considered the better performance of this center in terms of prevention indicators and better performance of the center's vehicles (Ambulance A-1 and Ambulance A-2) in the performance index by observing the standards.
The Delphi technique is a group judgment method which is typically used to reach agreement from a group of people with expertise in a particular area. It is an iterative process where panel members complete questionnaires over several rounds, often rating their agreement/disagreement against a statement, with changes made in later rounds based on the feedback received. It has been used widely in pharmacy-related studies relevant to education, research, and practice. This chapter provides a critical analysis of the various design choices which researchers may consider when planning a Delphi, namely, the panel of participants, the use of the Likert scale, the effect of feedback, what constitutes consensus, and the number of rounds. It also gives an overview of the development and origins of the Delphi, and discusses the advantages and disadvantages of the technique. Advantages include that the Delphi can be conducted with panel members in different geographical locations in their own time; however, the technique can therefore take longer to conduct and lacks face-to-face discussion. This chapter shows the importance of careful planning of the design choices to ensure the reliability and validity of the Delphi.
The simulated patient method is becoming an increasingly popular observational method to measure practice behavior in pharmacy practice and health services research. The simulated patient method involves sending a trained individual (simulated patient among other names), who is indistinguishable from a regular consumer, into a healthcare setting with a standardized scripted request. This method has come to be accepted as being well-suited for observing practice in the naturalistic setting and has also been used as an intervention when combined with feedback and coaching. This chapter presents an overview of the method, a brief history of its use, considerations for designing, implementing, and evaluating simulated patient studies, including ethical considerations, as well as methods of analysis and mixed-methods designs.
Because of the subjective character of symptoms, absence of a diagnostic test, modest response to treatments, and, at times, patient reports of important functional disability, fibromyalgia remains a challenge for the treating health care professionals in the standard clinical practice. The aim of this study was to develop an up-to-date consensus and evidence-based clinical practice guidelines for a treat-to-target management of fibromyalgia. Fifteen key clinical questions were identified by a scientific committee according to the Patient/Population, Intervention, Comparison, and Outcomes (PICO) approach. A literature review team performed a systematic review to summarize the evidence advocating the benefits and harms of available pharmacologic and nonpharmacologic therapies for fibromyalgia. Subsequently, recommendations were formulated. The level of evidence was determined for each section using the Oxford Centre for Evidence-based Medicine (CEBM) system. A 3-round Delphi process was conducted with 16 experts. All rounds were conducted online. A consensus was achieved on the direction and the strength of the recommendations.
An online questionnaire was sent to an expert panel who participated in the three rounds (response rate 100%). At the end of round 3, a total of fifteen recommendation items, categorized into 10 sections to address the main fibromyalgia categories, were obtained. Agreement with the recommendations (ranks 7–9) ranged from 85 to 100%. Consensus was reached (i.e., ≥ 80% of respondents strongly agreed or agreed) on the wording of all the 15 clinical standards identified by the scientific committee. An algorithm for the management of fibromyalgia has been suggested.
These recommendations provide an updated consensus on both the non-pharmacological and the pharmacological treatments of fibromyalgia. The provided strategies to reach optimal treat-to-target outcomes in common clinical scenarios are based on a combination of evidence and expert opinions. Best treatment decisions should be tailored to each individual patient situation.
Cybersecurity requirements, such as data security, are often used as evidence for the Government's relationship with external service providers to process, store and transmit sensitive government data. However, cybersecurity researchers have not profoundly studied the practical application of government data security requirements (e.g. data confidentiality) in service level agreements (SLAs) in the context of an outsourced scenario. The relationships with external service providers are usually established through SLAs as trust-enhancing instruments. However, there is a concern that existing SLAs mainly focus on the system availability and performance aspects but overlook cybersecurity requirements (e.g. data security) in SLAs. Such an understanding is essential to develop government SLA data confidentiality requirements into the formulation of security-related SLAs. We seek to provide insights by developing and conducting a grounded adaptive Delphi method (GADM) with 35 government participants through group discussions and individual sessions. The work on the Indonesian Government's data confidentiality requirements was used as a case study. This paper provides insights into three understandings of the increasing considerations of the Government's data confidentiality requirements in SLA definitions. The three perceptions of security-related SLAs are the target of protection, the data confidentiality risks and the government SLA data confidentiality requirements. Our findings play important implications for a better understanding of how to incorporate data confidentiality requirements according to perceived threats for government data classification in security-SLAs. Based on these findings, we recommend that the Government and service providers improve existing security-related SLAs and future research lines.
Machine learning (ML) and artificial intelligence (AI) have had a profound impact on our lives. Domains like health and learning are naturally helped by human–AI interactions and decision making. In these areas, as ML algorithms prove their value in making important decisions, humans add their distinctive expertise and judgment on social and interpersonal issues that need to be considered in tandem with algorithmic inputs of information. Some questions naturally arise. What rules and regulations should be invoked on the employment of AI, and what protocols should be in place to evaluate available AI resources? What are the forms of effective communication and coordination with AI that best promote effective human–AI teamwork? In this review, we highlight factors that we believe are especially important in assembling and managing human–AI decision making in a group setting. Our simplified representation in Figure 1 is now augmented to include (1) the presence of AI agents in the group, and (2) the adoption of ML models to predict the behavior of a human+AI group. In this review, we confine our ML modeling efforts to the specific tasks of predicting the emergence of influence systems (the left big arrow) and the prediction of decisions under multiple risk/reward scenarios (the right big arrow). Note that ML techniques can be utilized broadly in other tasks in the group environment, such as deciding optimal group composition and the design of interventions.
Mortality rates among surgical patients in Africa are double those of surgical patients in high-income countries. Internationally, there is a call to improve access to and safety of surgical and perioperative care. Perioperative research needs to be coordinated across Africa to positively impact perioperative mortality.
The aim of this study was to determine the top 10 perioperative research priorities for perioperative nurses in Africa, using a research priority-setting process. A Delphi technique with 4 rounds was used to establish consensus on the top 10 perioperative research priorities. In the first round, respondents submitted research priorities. Similar research priorities were amalgamated into single priorities when possible. In round 2, respondents ranked the priorities using a scale from 1 to 10 (of which 1 is the first/highest priority, and 10 is the last/lowest priority). The top 20 (of 31) were determined after round 2. In round 3, respondents ranked their top 10 priorities. The final round was an online discussion to reach consensus on the top 10 perioperative research priorities.
A total of 17 perioperative nurses representing 12 African countries determined the top research priorities, which were: (1) strategies to translate and implement perioperative research into clinical practice in Africa, (2) creating a perioperative research culture and the tools, resources, and funding needed to conduct perioperative nursing research in Africa, (3) optimizing nurse-led postoperative pain management, (4) survey of operating theater and critical care resources, (5) perception of, and adherence to sterile field and aseptic techniques among surgeons in Africa (6) surgical staff burnout, (7) broad principles of infection control in surgical wards, (8) the role of interprofessional communication to promote clinical teamwork when caring for surgical patients, (9) effective implementation of the surgical safety checklist and measures of its impact, and (10) constituents of quality nursing care.
These research priorities provide the structure for an intermediate-term research agenda for perioperative research in Africa.
Several state-owned companies (SOEs) have successfully implemented process innovation. This paper examines the success factors that influence the successful implementation of process innovation in Indonesian state-owned companies. The present study used the three stages of an online questionnaire in the Delphi method to obtain consensus from experts. The consensus was measured based on the mean and standard deviation of the assessment answers provided by our respondents. Based on the development of process innovation implementation frameworks shown in previous studies, we obtained 28 factors that influence the successful implementation of process innovation. Our panelists believed that human resources and organizational factors, such as leadership, problem understanding, strategy, and culture, affected the success of process innovation in SOEs, even more than the research and development budget, support, and commitment factor of a given company. Meanwhile, cost efficiency was the main factor driving a company’s motivation to implement process innovation. The success factors in implementing process innovation are an essential consideration for the management of other SOEs eager to improve their company’s innovation performance, especially process innovation. Success factors were carried out comprehensively for all implementation indicators of process innovation, including inputs, process (idea generation, idea selection, and idea implementation), outputs and outcomes, diffusion, culture, strategies, and push and pull factors.
We investigated whether preference transfer occurred from 18 taught world music pieces to 18 untaught pieces from the same regions among preservice music teachers. Eighty-three preservice music teachers participated in an interactive, culturally diverse music curriculum over the course of a semester. Participants learned 18 African, Asian, and Latin American folk songs in small groups from culture bearers via video conferencing. They then team-taught the same songs to their peers. Before and after the curriculum intervention with taught pieces only, we tested participants’ preferences for the taught and untaught pieces. Wilcoxon signed-rank test results indicated that the participants’ preference scores increased from the pretest to posttest for both taught and untaught world music pieces in all three regions. This result suggests the existence of possible preference transfer from taught to untaught pieces. Through follow-up interviews and instructor’s reflective notes analyses, we identified four interconnected themes across cognitive, cultural, social, and pedagogical dimensions that explained reasons for preference transfer: applying principles from taught pieces, appreciating cultures with authenticity and legitimacy, learning safely via intra- and intergroup interactions, and experiencing reciprocal course structure. We discuss implications for the profession that bridge formal and informal endeavors of teaching and learning music from diverse cultures.
Companies neither fully exploit the potential of Artificial Intelligence (AI), nor that of Machine Learning (ML), its most prominent method. This is true in particular of marketing, where its possible use extends beyond mere segmentation, personalization, and decision-making. We explore the drivers of and barriers to AI and ML in marketing by adopting a dual strategic and behavioral focus, which provides both an inward (AI and ML for marketers) and an outward (AI and ML for customers) perspective. From our mixed-method approach (a Delphi study, a survey, and two focus groups), we derive several research propositions that address the challenges facing marketing managers and organizations in three distinct domains: (1) Culture, Strategy, and Implementation; (2) Decision-Making and Ethics; (3) Customer Management. Our findings contribute to better understanding the human factor behind AI and ML, and aim to stimulate interdisciplinary inquiry across marketing, organizational behavior, psychology, and ethics.
Background: Standardised neonatal referral forms (NRFs) facilitate effective communication between healthcare providers and ensure continuity of care between facilities, which are essential for patient safety. We sought to determine the essential data items, or core clinical information (CCI), that should be conveyed for neonatal inter-hospital transfer in resource-limited settings (Rounds 1 to 3) and to create an NRF suitable for our setting (Round 4).
Methods: We conducted an international, four-round, modified Delphi-consensus study. Round-1 was a literature and internet search to identify existing NRFs. In Round-2 and -3, participants were Rwandan clinicians and international paediatric healthcare practitioners who had worked in Rwanda in the five years before the study. These participants evaluated the draft items and proposed additional items to be included in an NRF. Round-4 focused on creating the NRF and used five focus groups of Rwandan general practitioners at district hospitals.
Results: We identified 16 pre-existing NRFs containing 125 individual items. Of these, 91 items met the pre-defined consensus criteria for inclusion in Round-2. Only 33 items were present in more than 50% of the 16 NRFs, confirming the need for this consensus study. In Round-2, participants proposed 12 new items, six of which met the pre-defined consensus criteria. In Round-3, participants scored items for importance, and 57 items met the final consensus criteria. In Round-4, 29 general practitioners took part in five focus groups; a total of 16 modifications were utilised to finalise the NRF.
Conclusions: We generated a novel, robust, NRF that may be readily employed in resource-limited settings to communicate the essential clinical information to accompany a neonate requiring inter-hospital transfer.
Progress in degenerative cervical myelopathy (DCM) is hindered by inconsistent measurement and reporting. This impedes data aggregation and outcome comparison across studies. This limitation can be reversed by developing a core measurement set (CMS) for DCM research. Previously, the AO Spine Research Objectives and Common Data Elements for DCM (AO Spine RECODE-DCM) defined ‘what’ should be measured in DCM: the next step of this initiative is to determine ‘how’ to measure these features. This protocol outlines the steps necessary for the development of a CMS for DCM research and audit.
Methods and analysis
The CMS will be developed in accordance with the guidance developed by the Core Outcome Measures in Effectiveness Trials and the Consensus-based Standards for the selection of health Measurement Instruments. The process involves five phases. In phase 1, the steering committee agreed on the constructs to be measured by sourcing consensus definitions from patients, professionals and the literature. In phases 2 and 3, systematic reviews were conducted to identify tools for each construct and aggregate their evidence. Constructs with and without tools were identified, and scoping reviews were conducted for constructs without tools. Evidence on measurement properties, as well as on timing of assessments, are currently being aggregated. These will be presented in phase 4: a consensus meeting where a multi-disciplinary panel of experts will select the instruments that will form the CMS. Following selection, guidance on the implementation of the CMS will be developed and disseminated (phase 5). A preliminary CMS review scheduled at 4 years from release.
Ethics and dissemination
Ethical approval was obtained from the University of Cambridge (HBREC2019.14). Dissemination strategies will include peer-reviewed scientific publications; conference presentations; podcasts; the identification of AO Spine RECODE-DCM ambassadors; and engagement with relevant journals, funders and the DCM community.
Professionalism is a critical attribute that occupational therapy students must establish throughout education, especially in the context of assistive technology (AT). This study aimed to construct a multidimensional perspectives checklist (MPC) from the stakeholders to reflect professionalism development in the AT service courses. Fifteen experts in AT clinical practice and educators (N = 15) consented to and completed three rounds of the Delphi survey. Data were qualitatively analyzed to form a preliminary checklist in the first round. In rounds two and three, participants rated their level of agreement with the MPC items. A descriptive analysis of consensus was performed. Thirty items were classified into five subscales (teachers, therapists, patients, peers, learners) and fit into a framework with three dimensions and seven categories. After the Delphi survey, the MPC achieved high consensus, convergence, and stability. Two additional categories of professionalism emerged in the study, namely aesthetic and personal characteristics and reflection. The MPC developed in this study reflects the perspectives of various stakeholders in occupational therapy practice, providing helpful information for students to prepare themselves. Therefore, the MPC could contribute to expanding and developing the scope of professionalism in occupational therapy, especially in AT service.
The decline of language and communication abilities is common among people living with dementia and impacts on many areas of everyday life, including active participation in social activities and decision‐making. Despite a growing body of supporting evidence for approaches that address language and communication decline in dementia, the concept of communication rehabilitation is largely neglected in this population. This paper reports on the content validation of a novel tool, the Communication Support Needs Assessment Tool for Dementia (CoSNAT‐D). The tool has been developed to assist in the initial identification of communication difficulties and related support needs of people living with dementia. Importantly, the CoSNAT‐D is the only available tool that takes a three‐way informed approach, considering the view of the person living with dementia, their carer and an administering healthcare professional. Content validity was established between September and December 2018 using a modified Delphi approach. An international expert panel rated 32 items of a face‐validated item pool regarding their importance and relevance through an iterative feedback process. Consensus was pre‐determined at 70% of agreement for both importance and relevance of an item. Data were analysed using descriptive statistics and qualitative content analysis of comments provided in each round. Twenty‐eight experts working in dementia, language and communication participated in the Delphi survey. Qualitative analysis resulted in the addition of five items, of which three reached the required consensus in Round 3. Consensus was established for 35/37 items in three rounds. The pilot version of the CoSNAT‐D demonstrates adequate content validity and face validity. The use of the CoSNAT‐D may assist a range of healthcare professionals in the decision‐making process about appropriate next management steps, and thereby improve the care path for people with dementia and language and communication impairment. The establishment of further psychometric properties is warranted.
Antidote stocking represents a major challenge to hospitals all over the world, including Kuwait. In order to assist hospitals to reduce costs and improve patient care, an essential antidote list can be used as an initial foundation for securing sufficient antidote availability at healthcare institutions. The aim of our study is to generate a nationally relevant essential antidote list for emergency care hospitals in Kuwait using the e-Delphi method by establishing consensus through a multidisciplinary expert group of healthcare providers. An electronic survey with 47 essential antidotes was developed. The e-Delphi method was used, with three rounds of voting, to determine expert consensus on an essential antidote list for hospitals in Kuwait. A purposive sample of healthcare professionals from governmental and private hospitals were selected for this study (n = 30). Consensus was gained if ≥75% of the expert panel agreed on the inclusion of the antidote, without any strong disagreements. Round 1 of the e-Delphi resulted in 41 antidotes reaching consensus and seven new antidotes suggested by the expert panel. Round 2 had two antidotes (out of seven newly suggested ones) reaching consensus. Round 3 was a confirmatory round, where the expert group agreed on their previous rounds’ opinions. This resulted in the development of an essential antidote list with 43 antidotes. The optimal approach for ensuring adequate availability of antidotes is continuous monitoring of local poisoning incidence and antidote requirements through collaborations between academic researchers and emergency care clinicians. The development of an essential antidote list, with expert consensus, is one of the initial steps in securing a foundation for appropriate provision of antidotes at all healthcare institutions. This is the first study that the authors are aware of that demonstrates that the e-Delphi technique can consolidate recommendations of experts in emergency medicine to provide a list of essential antidotes.
School-age children, adolescents, and young adults with autism spectrum disorder encounter many different types of providers in their pursuit of treatment for anxiety, behavior problems, and social difficulties. These providers may all be familiar with different types of intervention practices. However, research has not yet investigated patterns in expert providers' familiarity with different practices nor how these patterns are related to the characteristics of providers (years in practice, academic discipline, setting) and the youth (age and intellectual disability) they typically support. A panel of 53 expert transdisciplinary providers rated their familiarity with 55 intervention practices (derived from research and expert nominations) via an online Delphi poll. Advanced statistical methods were used to identify types of intervention practices with which providers were familiar, which included two approaches (cognitive and behavioral) and two strategies (engagement and accessibility). Providers who practiced outside a school setting or treated clients without intellectual disability were more familiar with cognitive approaches. Clinical psychologists, behavior analysts, and school-based providers were more familiar with behavioral approaches. Providers practicing outside school settings were also more familiar with engagement strategies, and providers with more years in practice were more familiar with accessibility strategies. These results may help families and researchers to better anticipate how services may vary depending on the types of autism spectrum disorder providers seen and work to reduce disparities in care that may result.
In recent years, artists who characterise themselves as freelancers with short-term contracts or running their own businesses are known as “arts entrepreneurs” rather than “artists”. In China, arts entrepreneurship is a new discipline while arts-specific entrepreneurial competencies have not yet been comprehensively identified. The purpose of this study was validating an entrepreneurial competence framework for tertiary arts education in China. A modified Delphi method was used, which involved a consensus approach through three surveys of an expert multidisciplinary panel with more than 10 years’ experience in the field. Results from 53 experts who completed all 3 rounds of survey indicated a wide range of entrepreneurial competences that could be taught at tertiary level to support artists to successfully navigate their current and future careers. Consensus centered 37 essential competencies of the 3 domains: general competence, artistic competence and entrepreneurial competence. The results of this study offer implications and directions for future research. The validated competencies can inform analyses of the extent to which these competencies are actually attained and pursued in arts entrepreneurship education programmes in China.
This handbook is designed as a resource for providing up-to-date and practical guidance on national health planning and strategizing for health. It establishes a set of best practices to support strategic plans for health and represents the wealth of experience accumulated by WHO on national health policies, strategies and plans (NHPSPs).
WHO has been one of the leading organizations to support countries in the development of NHPSPs. The focus on improving plans has grown in recent years, in recognition of the benefits of anchoring a strong national health sector in a written vision based on participation, analysis, and evidence.
Transparency Ecosystem for Research and Journals in Medicine (TERM) Working Group summarized the essential recommendations that should be considered to review and publish a high‐quality guideline. These recommendations from editors and reviewers included the ten components of essential requirements: systematic review of existing relevant guidelines, guideline registration, guideline protocol, stakeholders, conflicts of interest, clinical questions, systematic reviews, recommendation consensus, guideline reporting, and external review. TERM Working Group abbreviates them as PAGE (essential requirements for Publishing clinical prActice GuidelinEs), recommends guideline authors, editors, and peer reviewers use them for high‐quality guidelines.
Contrary to expectations, based on occurring changes in customer behaviours related to consuming and shopping triggered by the pandemic, the global container shipping market has continued to grow during COVID 19. Experts estimate that these increases will continue in the future due to changing consumption habits. However, container shipping companies (CSCs) may soon encounter many troubles and challenging situations. They have an extremely fragile structure and may be entirely unprotected when they encounter unexpected situations sourced from external factors. (i.e., due to grounding of a ship, complete blockage of the Suez channel for three weeks can be given as a clear example of that). Hence, selecting an appropriate container vessel type can help construct a healthier container shipping system less influenced by adverse conditions for decision-makers and practitioners. Besides, it can provide a more effective and productive maritime transportation environment for all stakeholders. However, selecting a proper container vessel type is a complicated decision-making problem since many conflicting criteria and complex ambiguities exist. The current paper proposes an extended version of the GRA technique with the help of type-2 neutrosophic fuzzy sets (T2NFN) for capturing and processing uncertainties better than the traditional MCDM frameworks. According to the obtained results, C6, container carrying capacity, is the most influential criterion and the type of post-Suezmax container vessel is the best option for the CSCs, as it provides advantages at a satisfactory level for almost all criteria than others. After the proposed model was applied, a comprehensive sensitivity analysis (SA) was performed to test the validity of the T2NFN GRA approach. The results of SA approve the applicability, effectivity, and robustness of the model.
Research guidelines for the Delphi survey technique
Consensus methods such as the Delphi survey technique are being employed to help enhance effective decision-making in health and social care. The Delphi survey is a group facilitation technique, which is an iterative multistage process, designed to transform opinion into group consensus. It is a flexible approach, that is used commonly within the health and social sciences, yet little guidance exists to help researchers undertake this method of data collection. This paper aims to provide an understanding of the preparation, action steps and difficulties that are inherent within the Delphi. Used systematically and rigorously, the Delphi can contribute significantly to broadening knowledge within the nursing profession. However, careful thought must be given before using the method; there are key issues surrounding problem identification, researcher skills and data presentation that must be addressed. The paper does not claim to be definitive; it purports to act as a guide for those researchers who wish to exploit the Delphi methodology.
The Examination of Anomalous Self-Experience ( EASE) is a symptom checklist for semi-structured, phenomenological exploration of experiential or subjective anomalies that may be considered as disorders of basic or 'minimal' self-awareness. The EASE is developed on the basis of self-descriptions obtained from patients suffering from schizophrenia spectrum disorders. The scale has a strong descriptive, diagnostic, and differential diagnostic relevance for disorders within the schizophrenia spectrum. This version contains interview-specific issues and psychopathological item descriptions ( Manual), a scoring sheet ( Appendix A), a reminder list of items for use during the interview ( Appendix B) and an EASE/BSABS ('Bonner Skala fur die Beurteilung von Basis-symptomen') item comparison list (Appendix C).
This paper describes a research design aimed at examining the risks and impacts of natural and anthropogenic catastrophic events on the future of the tourism industry. The focus is on the selection of a survey method, the Delphi technique, and the reasons behind selecting this qualitative forecasting tool. The Delphi technique relies primarily on the intuitive judgment of experts. This paper argues that with thorough pre-planning and rigorous management of the survey, the Delphi technique can provide futures information that other conventional extrapolative methods cannot reliably forecast.
This paper gives an account of an experiment in the use of the so-called DELPHI method, which was devised in order to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of questionnaires in depth interspersed with controlled opinion feedback.
Focusing on the general issues of reliability and validity, several specific problems evident in reported applications of the Delphi method to forecasting are discussed. Reliability threats, which are noted in a number of procedural aspects of the method, arise from ill-considered procedural variations and lack of standardization. While validity threats are also found in several procedural aspects of the tool, they arise principally from pressures for convergence of predictions. This feature of the method, along with certain structural characteristics, is found to undermine critically its forecasting ability. Having discussed in some detail the nature of these difficulties, the paper closes with consideration of the reasons for the continued use of Delphi in spite of its shortcomings and with comments on alternative approaches.
Recently, Keith Zoski and Stephen Jurs proposed a modified version of the scree test for analyzing Delphi survey data. Although the test has merit, three considerations or problems affecting its use are examined and examples of data related to those problems are given. Discussion is also provided regarding appropriate and inappropriate applications of the modified scree test to Delphi data.
This paper is concerned with describing the principal uses of the Delphi method in higher education. These areas of use are found to be (1) searching out goals and objectives, (2) campus and curriculum planning, and (3) development of evaluation criteria. Important to understanding the use of Delphi in a higher education setting are the methodological adaptations that have been employed. Delphi as used in technological forecasting requires modification when used in the generation and assessment of goals. Three principal problems in methodology are examined here. They are (1) selection of the panel, (2) the character of round one and (3) the consequences of feedback.
This report describes the results of an extensive set of experiments, conducted at RAND, concerned with evaluating the effectiveness of the Delphi procedures for formulating group judgements. The study represents a small beginning in the field of research that could be called ‘opinion technology’, and is of direct relevance for the use of experts as advisers in decision making, especially in areas of broad or long-range policy formulation.
Describes the Delphi Technique, an approach to elicit and refine the opinions of a group of people. Outlines the steps in the process and provides illustrations of the applications of the technique for schools and colleges in determining future research needs, assessing staff development needs, and identifying problems confronting administrators. (MB)
Leading professionals in the field discuss instructional design and systems, computer applications in education and training, research and evaluation in instructional technology, future prospects for instructional technology, and professional development. The book contains six parts. A brief outline of instructional development and educational technology is given in part 1. This is followed by consideration in part 2 of the current state of the art, trends and issues, and challenges in linking the cognitive sciences with the instructional design process. Instructional development, including needs assessment and task analysis procedures are covered in part 3, and part 4 reviews various applications of instructional technology, such as the Internet and distance education. Parts 4 and 5 present an overview of current research and cover certification and professional development in the field. New to this edition are seven chapters that address such current topics as educational systems development and instructional systems development, postmodernism and instructional technology, interactive technologies, the Internet and higher education, qualitative research, and instructional technology and attitude change. (Author/AEF)
This research study used the Delphi technique to determine the ideal content of a college-level outdoor leadership course for land-based outdoor pursuits in the Pacific Northwest. Topics were generated and value-rated by 61 Pacific Northwest outdoor leaders using a Likert-type scale in three separate questionnaires. Thirty-five topics were retained in the ideal content, of which the top 10 were: (1) risk management plans; (2) judgement; (3) wilderness ethics; (4) first aid; (5) analyzing risks; (6) minimum impact practices; (7) outdoor leadership objectives; (8) hazard analysis for hypothermia; (9) backcountry first aid; and (10) minimum impact philosophy. An outdoor leadership course outline constructed from research results included philosophy of outdoor leadership, risk management, decision making, outdoor leadership methods, teaching principles, small group dynamics in the outdoors, environmental awareness skills, pre-trip planning, hazards analysis, and map reading and interpretation. Tables list final rankings and mean scores of the 35 consensus topics and the 23 topics rated least important. An appendix provides a copy of the third and final questionnaire with ratings for 176 topics. (LFL)
This study determined how experts see the future of adult fitness by using the Delphi Technique as a measurement tool. Experts (N = 50), all Fellows of the American College of Sports Medicine, were selected on the basis of their reputations in adult fitness. Three questionnaires were submitted to the panel. In the first questionnaire each expert listed five or more events, trends, or developments they thought would take place and five or more events they would like to see take place in the next 25 years. From the events and trends listed on this questionnaire, 100 items were generated that formed the basis for the last two questionnaires. Responses to the items on the second questionnaire were evaluated on a 5-point Likert scale for probability and desirability. In the third questionnaire, group means and standard deviations were calculated and recorded for each item. Only those items which were significantly (p < .01) higher than 3.0, the average probability or desirability response on the Likert scale, were included in the analysis. The items were divided into seven categories: fitness in sport and recreation, fitness and wellness, fitness in industry, fitness in education, fitness in business, research in fitness, and fitness equipment and facilities. The highest single mean for any item for both probability (M = 4.31) and desirability (M = 4.83) was increased awareness of health and fitness among the elderly.
This is a review of the state-of-the-art of rational and primarily quantitative approaches to the utilization of expert opinion in the inexact sciences, among which the engineering sciences must be included. Of primary concern is the decision-maker's utilization of one or more experts, and the criteria and scoring rules used to evaluate their performance. Also reviewed are certain game-theoretic strategies used by experts to maximize their own goal satisfaction. With this review, it is hoped that chemical engineers, when acting as decision-makers, will be able to utilize expert opinion more effectively and will better understand the function of experts when it is necessary for them to adopt such roles themselves.
This paper presents the results of a two round Delphi survey conducted into expert opinion on the development of indicators to measure the movement of the tourism product at a company/resort level towards a position of greater or lesser sustainability. This research forms part of a wider project to develop indicators that consumers can use in the selection of their holidays and promote a more sustainable form of tourism. The results of this expert survey show considerable disagreement over “sustainability” and where the borders of the concept exist. In addition, the research identified contrasting views over the use of qualitative versus quantitative indicators and the role that consumer pressure can play. The use of the Delphi technique to address complex and uncertain issues is also explored.
This paper is concerned with a methodology for efficiently obtaining concensus from a panel of evaluators on questions which are shrouded in uncertainty and can not be measured or evaluated in the classical sense. The methodology and its historical development are described, it is placed in context vis-a-vis other methods of subjective scaling, and a critique is provided. An extensive and current annotated bibliography is presented.
Thesis (Ph. D.)--Ohio State University, 1994. Includes bibliographical references (leaves 163-170). Advisor: R. Kirby Barrick, Dept. of Agricultural Education. System requirements: World Wide Web browser and PDF viewer.
Altschuld, J. W. (2003). Delphi technique. Lecture, Applied
evaluation design. The Ohio State University.
Using the Delphi process to identify significant recreation research-based innovations
D H Anderson
I E Schneider
Anderson, D. H., & Schneider, I. E. (1993). Using the
Delphi process to identify significant recreation
research-based innovations. Journal of Park and
Recreation Administration, 11 (1), 25-36.
A comparative study of the effectiveness of decision making processes which utilize the Delphi and leaderless group methodologies. Unpublished doctoral dissertation
D C Douglas
Douglas, D. C. (1983). A comparative study of the effectiveness of
decision making processes which utilize the Delphi and leaderless
group methodologies. Unpublished doctoral dissertation,
The Ohio State University, Columbus.
Essential assessment criteria for physical education teacher education programs: A Delphi study
J M Jacobs
Jacobs, J. M. (1996). Essential assessment criteria for physical
education teacher education programs: A Delphi study.
Unpublished doctoral dissertation, West Virginia
Priorities within the health care system: A Delphi survey
C A Lindeman
Lindeman, C. A. (1981). Priorities within the health care system:
A Delphi survey. Kansas City, MO: American Nurses'
Rethinking the outlook of colleges whose roots have been in agriculture
J H Meyer
Meyer, J. H. (1992). Rethinking the outlook of colleges whose roots
have been in agriculture. Davis, CA: University of
Determining what could/should be: The Delphi technique and its application
L E Miller
Miller, L. E. (2006, October). Determining what could/should
be: The Delphi technique and its application. Paper presented
at the meeting of the 2006 annual meeting of the
Mid-Western Educational Research Association,
The Delphi method: Techniques and applications
Scheibe, M., Skutsch, M., & Schofer, J. (1975).
Experiments in Delphi methodology. In H. A.
Linstone, & M. Turoff (Eds.). The Delphi method:
Techniques and applications (pp. 262-287). Reading, MA:
Addison-Wesley Publishing Company.
Tourism marketing and management handbook
R E Taylor
L L Judd
Taylor, R. E., & Judd, L. L. (1989). Delphi method applied
to tourism. In S. Witt, & L. Moutinho, (Eds.). Tourism
marketing and management handbook. New York: Prentice
Gazing into the oracle: The Delphi method and its application to social policy and public health
S R Hiltz
Turoff, M., & Hiltz, S. R. (1996). Computer based Delphi
process. In M. Adler, & E. Ziglio (Eds.). Gazing into the
oracle: The Delphi method and its application to social policy and
public health (pp. 56-88). London, UK: Jessica Kingsley
Human resource development: The theory and practice of need assessment
F L Ulschak
Ulschak, F. L. (1983). Human resource development: The theory
and practice of need assessment. Reston, VA: Reston
Publishing Company, Inc.
Planning and conducting needs assessment: A practical guide
B R Witkin
J W Altschuld
Witkin, B. R., & Altschuld, J. W. (1995). Planning and
conducting needs assessment: A practical guide. Thousand
Oaks, CA: Sage Publications, Inc.