No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
RT Delphi: An efficient, “round-less” almost real time Delphi method
Abstract
The authors have recently developed a new approach to performing a Delphi study that does not involve the use of sequential “rounds” and as a result, greatly improves the efficiency of the process and shortens the time to perform such studies. This paper describes this process, RT Delphi, and illustrates its use in a decision-making application drawn from the Millennium Project of the American Council for the United Nations University. The illustrative application involves setting priorities among strategies for dealing with anticipated terrorist activities that might be initiated by a single deranged individual.
... • Real-Time Delphi. Within RT-GSCS, we put forth a noteworthy contribution to the scientific literature by integrating textual and spatial dimensions through the adoption of Real-Time Delphi (Gordon and Pease, 2006). Real-Time Delphi offers an expanded perspective on the context or phenomenon under study by facilitating the convergence of expert opinions on text-based or close-ended questions in real time (akin to classic questionnaires). ...
... It is particularly useful for decision-making and forecasting, and in situations where time is a critical factor or when there is coordination difficulty or a lack of sufficient expert engagement. Gordon and Pease (2006) introduced the method to overcome the time-consuming process of the classical Delphi method, which typically requires multiple rounds of questionnaires to reach consensus (or stability). The RTD aims to achieve a group consensus in a single round of questionnaire iteration In contrast to the traditional Delphi method, experts are not required to respond a fixed number of times at predetermined intervals, nor are they required to complete the questionnaire all at once. ...
... Furthermore, RTD can be conducted with a diverse group of participants from various locations, and the questionnaire can include attachments and reference materials to help respondents find supporting information online. Several studies have demonstrated the effectiveness of RTD in various fields, including technology forecasting, policy analysis, and disaster management, by adopting different web-based platforms (Gordon and Pease, 2006;Gnatatzy et al. 2011;Aengenheyster et al. 2017;Meyer et al. 2022). However, it is important to note that the RTD may be subject to some of the same strengths and limitations as the traditional Delphi method (see Tab. 3.2), including subjective choices in the response scales and high dropout rates. ...
The spatial decision-making process in urban planning, policy development, and territorial program implementation is fundamentally connected with the concept of location within a geographical context. Given the internal spatial complexity arising from numerous interconnected variables such as natural resources, infrastructure, environmental factors, cultural elements, and security challenges, accurately identifying their spatial positions is crucial. Assuming such
complexity of geographical spaces, the spatial decision-making process should adopt an
interdisciplinary approach. This approach should consider the opinions of differentsocial actors to
achieve a common consensus in the final decision.
This dissertation presents applied research aimed at developing a new Spatial Decision Support
System to enhance the decision-making process. This system facilitates the acquisition, analysis,
and visualisation of expert spatial input to determine optimal locations for goods, services, and
events in a territory. The proposed system is based on a spatial modification of the Delphi method,
a well-established and robust technique in the literature that aggregates expert judgments to
address specific problems, aiming to achieve consensus and stability. Considering the specific
technical framework, computational algorithms, and advanced spatial analysis, a web-based open
system called the “Real-Time Geo-Spatial Consensus System” (RT-GSCS) is developed. The
system enables the administration of a spatial questionnaire in real-time, allowing experts to
answer a series of questions by entering spatial information on a dynamic map. They can also
consider real-time spatial and textual analyses, along with convergence metrics. The final aim is
to achieve a spatial consensus, thereby facilitating the optimal location of goods, services, and
events within the territory.
To validate the proposed system, we present three major case studies in different geographical
areas. The first case study involves developing future spatial scenarios in the climate context for
ten coastal European cities. The second case study focuses on the transport sector, specifically
identifying logistics facilities such as parcel lockers and loading bays in the city of Dublin. The third
case study aims to identify suitable locations for active mobility pathways in Dublin. The primary
findings demonstrate the system’s effectiveness in real-time acquisition of spatial judgments from
multidisciplinary groups of experts, even those without prior experience with geo-technology tools.
This approach supports collective spatial analysis for making spatial decisions, particularly in the
realms of planning, organisation, and resource utilisation within a territory.
... ACCORD has been developed with the flexibility to apply to all methods or techniques used to achieve consensus, not only to the commonly used Delphi approach [1,3]. A non-comprehensive list of consensus methods used to measure, promote or gain consensus, and that can be reported using ACCORD is available in Table 1 [5][6][7][8][9][10][11][12][13][14]. Researchers may choose to adapt a standard method, or to combine methods to fit their specific research needs. ...
... Example 1. "There is a plethora of literature illuminating the experiences of members of the public and professional researchers, and the process of PI [Patient Involvement] and its potential impacts. [6][7][8][9] However, while studies report on use of PI in context of priority setting, 10 I2. State the aim of the consensus exercise, including its intended audience and geographical scope (national, regional, global). ...
... To help respondents to consider the full scope of the perioperative patient journey, we asked them to consider the preoperative, intraoperative and postoperative phases and also invited any further suggestions."[48] "A published systematic literature review (SLR)9 identified all the outcomes documented in studies since 1990 involving patients who had undergone surgery for CES [cauda equina syndrome]. The outcomes from the SLR were combined with the outcomes identified from the qualitative interviews to form those initially rated on within the Delphi Survey. ...
Background
When research evidence is limited, inconsistent, or absent, healthcare decisions and policies need to be based on consensus amongst interested stakeholders. In these processes, the knowledge, experience, and expertise of health professionals, researchers, policymakers, and the public are systematically collected and synthesised to reach agreed clinical recommendations and/or priorities. However, despite the influence of consensus exercises, the methods used to achieve agreement are often poorly reported. The ACCORD (ACcurate COnsensus Reporting Document) guideline was developed to help report any consensus methods used in biomedical research, regardless of the health field, techniques used, or application. This explanatory document facilitates the use of the ACCORD checklist.
Methods and findings
This paper was built collaboratively based on classic and contemporary literature on consensus methods and publications reporting their use. For each ACCORD checklist item, this explanation and elaboration document unpacks the pieces of information that should be reported and provides a rationale on why it is essential to describe them in detail. Furthermore, this document offers a glossary of terms used in consensus exercises to clarify the meaning of common terms used across consensus methods, to promote uniformity, and to support understanding for consumers who read consensus statements, position statements, or clinical practice guidelines. The items are followed by examples of reporting items from the ACCORD guideline, in text, tables and figures.
Conclusions
The ACCORD materials – including the reporting guideline and this explanation and elaboration document – can be used by anyone reporting a consensus exercise used in the context of health research. As a reporting guideline, ACCORD helps researchers to be transparent about the materials, resources (both human and financial), and procedures used in their investigations so readers can judge the trustworthiness and applicability of their results/recommendations.
... Real-time Delphi studies do not have defined rounds. Panelists receive individualized feedback showing how their response compares with all other previously provided responses and can alter their answers as often as they want (Gordon and Pease, 2006). Feedback on the panel's responses in the form of medians, interquartile range, and comments is continually updated to reflect the most recent responses (Aengenheyster et al., 2017). ...
... Nevertheless, the pervasiveness of email and a variety of relatively cheap and easy-to-use online survey and video conferencing platforms make online Delphi implementation a compelling option (Broder, Gibbs, and Yermilov, 2022). Moreover, certain types of Delphi studies (e.g., real-time Delphi) can only be conducted online (Gordon and Pease, 2006). Some online Delphi platforms, such as ExpertLens, can accommodate discussion between panelists while still maintaining anonymity and can automate data analysis and preparation of individualized panelist reports, which expedite data collection and reduce the costs of convening largescale national and international panels . ...
... diately flag these responses (Gordon and Pease, 2006). Therefore, panel conveners should decide whether and how to ask open-ended questions. ...
The Delphi method is an iterative, anonymous, structured, group-based communication process and elicitation technique designed to help policymakers make decisions under conditions of uncertainty and incomplete information. It is based on the premise that asking a hand-picked group of anonymous experts the same questions several times and sharing the other experts' answers will help objectively develop group consensus, which is used as a form of evidence. Although it was originally developed by RAND researchers as a forecasting methodology in military research, Delphi underwent many modifications and is now used by different disciplines, most notably by medicine, as a gold-standard approach for expert elicitation and stakeholder engagement. Researchers often rely on Delphi to estimate the probability of an event happening within a certain period of time, to forecast when an event is likely to occur, and to identify and prioritize key policy issues that need to be addressed.
Although the Delphi method is used by different disciplines and for different purposes, there is still a lack of methodological guidance for how to conduct rigorous Delphi studies. This manual addresses this critical gap by providing practical advice for when to use Delphi, what type of Delphi to choose, and what to consider when designing, implementing, and reporting Delphi panel results. The manual also includes the Delphi Critical Appraisal Tool (DCAT), which researchers can use both retrospectively to appraise the quality of an already conducted Delphi study and prospectively to design and implement a Delphi study.
... It is a tool for the analysis of policy issues and not an approach for decision-making [31]. According to Seker [53], other known Delphi variants are electronic Delphi [36], modified Delphi, ranking-type Delphi [52], real-time Delphi [14], disaggregative Policy Delphi [58] EFTE Delphi [29,42,53], mini Delphi, online Delphi, technological Delphi, and argumentation Delphi. ...
... Real-time Delphi [14] x Using digital Delphi e-tool: www.edelphi.org. ...
... Online Delphi [14] x Using environment www.edelphi.org. ...
This article presents a hybrid foresight study using the Delphi methodology. The study is part of a strategic research project of the Finnish state named “Foresight in Environmental Reference Laboratory Operations.” The main starting point for this study was the Policy Delphi method, which was supplemented with other Delphi variants and decision support methods. In this hybrid foresight application, Delphi methodology was combined with decision-making and strategic planning tools, namely (1) SWOT analysis, (2) Boston Consulting Group Matrix, (3) research infrastructure (RI) analysis, (4) platform analysis, and (5) integrative foresight workshop. This article draws strongly on the hybrid foresight methodology perspective of foresight tool orchestration. The authors argue that the Delphi methodology gains reliability and appeal with hybrid methodological orchestration. The function of reference laboratories includes services for the verification and maintenance of the quality and validity of environmental measurements, such as the maintenance of reference measurement equipment, expertise, and training, and the organization of national reference measurements as well as participation in international reference measurements, method development, and standardization. The study shows the importance of environmental reference laboratories addressing the following challenges by 2030: (1) new measurement targets, such as the diffusion of nanomaterials, microplastics, and drugs in both nature and the food chain and the environmental impact of recycled materials and indoor air issues; (2) automation of measurement activities and transition to technologically advanced measurement systems; (3) citizens’ opportunities to participate in the production of information about their environment through new technologies; (4) preparing for the incorporation and internationalization of laboratory operations; and (5) the concentration of measurement service business.
... We chose to use the RT-Delphi approach, which builds a consensus on the relevance, clarity and appropriateness on the proposed national de nitions and method assessing active HWF In Luxembourg while collecting stakeholder feedback. The RT-Delphi approach was designed to enhance the e ciency of the process, and reduce the time required for its execution, by continuous iteration and providing individual and aggregated feedback to participants in real-time (17,19). This RT-Delphi study included participants from ve different backgrounds: (i) research and education, (ii) HCP, (iii) patient representation (iv) public health and policy making professionals and (v) health insurance and social security. ...
... This paper represents the rst steps on the pathway to create evidence-based, de nitions and a method to assess the Luxembourgish HCW's headcount, level of activity and primary care. Choosing a RT-Delphi, allows engaging with stakeholder feedback in real-time and may result in a more direct reconciliation of opinions and ultimately consensus in a shorter time (17,19 Ethics approval and consent to participate Each participant will electronically provide informed consent on the study's entry page. a data protection assessment was carried out by the CGPD. ...
Background The effectiveness of healthcare delivery depends on the availability of adequately skilled healthcare workforce (HWF) to respond to the current and future needs of healthcare users. Obtaining accurate and timely HWF data is essential for health policy. This article reports on the conceptualization of definitions and a methodology to estimate the active HWF in Luxembourg and presents the protocol for a real-time online Delphi (RT-Delphi) study to consult stakeholders on the methodology proposed. Methods Definitions and methods for assessing an active HWF were established through a scoping review and discussions within the project’s steering committee. RT-Delphi study and online multistakeholder meeting are conducted to explore the level of agreement among stakeholders on the proposed definitions and methods determining the active HWF in Luxembourg and collect stakeholder feedback to improve the method. Stakeholders evaluate the proposed definition and methods. They can reconsider their evaluations, multiple times, based on aggregated feedback in real-time and during a final multistakeholder meeting planned for clarification and further debate on open topics, if applicable. We will quantify agreement on the whole sample and per stakeholder group and perform analyses of process to examine variation in responses. Open-text comments and workshop discussions will be analyzed qualitatively via content analysis, and results used to improve the methodology and its implementation in practice. Results The scoping review retained seven studies on defining and measuring active healthcare workforce (HWF), revealing a lack of standardized methods and definitions. Workshops recommended adopting the JQNMHC’s definition for Luxembourg due to its alignment with national data reporting needs. Three activity level measurement methods for self-employed HWF were retained. Additionally, a new definition for primary care in Luxembourg was proposed based on recent WHO guidelines, outlining essential functions such as accessibility, comprehensiveness, and coordination. Conclusion Involving stakeholders allows for a transparent and structured way to inform conceptualization of indicators to assess HWF activity and better prepare for their adoption and implementation on a national level.
... Experts' predictions were collected between 2009 and 2011 using a Real Time Delphi [15] online service and published in 2014 [6]. The European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) coordinates a network of national focal points (NFPs) set up in the 27 EU Member States, Norway and Türkiye. ...
... The Delphi method promotes a shared view of the issues under consideration and encourages consensus [58,59]. The role of Delphi as an expert 'discussion forum' has strengthened since the spread of real-time Delphi [15] utilizing webbased Delphi tools. In the Delphi study in 2009-2011, experts unanimously predicted that drug therapy would be integrated into primary care -furthermore, they unanimously stated that this was a desirable direction. ...
Background
The aim of the current study was to assess the accuracy of expert predictions, which were derived using a Delphi panel foresight study between 2009 and 2011, on a variety of drug-related topics in Finland in 2020.
Methods
The material used to evaluate the accuracy of the predictions consists of published reports on statistics, survey results, official register data, wastewater analyses and official documents. Whenever possible, we used multiple information sources to ascertain possible changes related to the predictions.
Results
Between 2009 and 2011, the majority – but not all – of the experts accurately predicted an increase in drug use. Indeed, more people experimented with or used drugs, and more drug residues were found in wastewater monitoring. The experts also correctly predicted an increase in population-level approval of drug use, but this development has been rather slow. Contrary to predictions, there was no marked increase in the use of new synthetic drugs. However, the misuse of buprenorphine increased during the 2010s. In the drug market, unit prices were surprisingly stable over the ten-year period. There were no changes in legislation related to the legal status of drugs, as was foreseen by the experts. However, enforcement moved in the direction foreseen by the experts: more lenient measures have been taken against users. Drug care system reforms favored a combination of mental health and addiction care units between 2009 and 2011, and 2020, as foreseen by the experts.
Conclusions
It seems to have been easier for the experts to foresee the continuation of existing trends, e.g., increasing use of drugs or widening approval of drugs, than to predict possible changes in the popularity of distinct groups of drugs such as new psychoactive substances (NPS). Even armed with the prediction that drug imports and wholesale would increasingly fall into the domain of organized crime, this undesirable development could not be stopped. Expert disagreement can also be seen as a valuable indication of uncertainty regarding the future. Foresight related to drug-related issues can produce relatively accurate and realistic views of the future at least up to ten years ahead.
... With the aim of creating a method that increases the speed and efficiency of opinion gathering where a quick response is required, a method called real-time Delphi has been developed (Gordon and Pease, 2006). Real-time Delphi requires only one round and each respondent can immediately see their previous answers as well as new averages, medians and distributions immediately. ...
... Respondents' comments and reasoning are also included. The experiment ends when a steady state is reached (Gordon and Pease, 2006). The whole process can take place during one session or participation can be asynchronous (Clayton, 1997). ...
... With the aim of creating a method that increases the speed and efficiency of opinion gathering where a quick response is required, a method called real-time Delphi has been developed (Gordon and Pease, 2006). Real-time Delphi requires only one round and each respondent can immediately see their previous answers as well as new averages, medians and distributions immediately. ...
... Respondents' comments and reasoning are also included. The experiment ends when a steady state is reached (Gordon and Pease, 2006). The whole process can take place during one session or participation can be asynchronous (Clayton, 1997). ...
... As in previous studies (Merma-Molina et al., 2017), the expert technique, which is the base for the Delphi Method, was used to determine the reliability of the rubric. The Delphi method was developed at the RAND Corporation in the late 1950s and 1960s as an effective means for collecting and synthesizing expert judgments (Gordon & Pease, 2006). The objective of the Delphi methodology (Dalkey & Helmer, 1963) is to obtain the most reliable consensus of opinion of a group of experts who analyse a problem or an instrument. ...
... However, one of its main drawbacks is the iterative nature of the feedback collected over several rounds, which can lengthen substantially the time it takes to complete a study. To improve the speed of the process, Gordon & Pease (2006) developed the real-time (RT) Delphi which reduces these multiple reiterations to one round in synchronous studies with a small number of experts. In this study, seven qualified experts (known to the author) were selected based on discipline, years of teaching and assessment experience using digital technologies in higher education (i.e., more than 5 years), and geographical context (i.e., that represented the geographical context of this study and beyond). ...
Over the past decade, self-assessment tools have garnered significant attention in the interest of measuring the skillset required by educators and students to function productively and ethically in digitally mediated environments, particularly in relation to education policy implementation. Since stated beliefs do not always align with actual practice, gaps have been shown to occur between self-reporting and performance in practice. Having an external assessor can counteract this imbalance; however, both perspectives should be taken into consideration as both are equally important. Against this background, this study develops and validates two rubric-based frameworks that supplement self perceived student and educator digital competence with classroom observation and task performance analysis. The DigComp and DigCompEdu self-assessment frameworks were used as a starting point to develop a student rubric and an educator rubric, respectively, underpinned by criteria validated in previous frameworks. The expert technique, which is the base for the Delphi Method, was used to validate each rubric, after which the rubrics were implemented at a Spanish university to test their reliability. The results indicated that the force of agreement across raters was consistent and both rubrics had a high degree of internal consistency, therefore both instruments are reliable.
... In 2006, Gordon & Pease introduced the Real-Time Delphi (RTD) as an alternative to multi-round Delphi surveys, aiming to improve the process's efficiency and shorten the required time. In this method, participants can view the group's responses for each question in real time and the total number of responses simultaneously [43]. RTD is round-less; however, in light of changing feedback, participants can be recommended to revisit and re-rate questions. ...
... The number of papers using RTD is increasing. The RTD has been used to develop consensus across a range of settings, including economics, [43,44] transportation [44], education [45,46], religion [47], professional development [48], and sustainability [49]. RTD has been used in healthcare studies that aim to develop a medication adherence technologies repository [50] and determine the negative impacts of substance use disorders among people with HIV in the United States [51]. ...
Neonatal sepsis is a serious public health problem; however, there is substantial heterogeneity in the outcomes measured and reported in research evaluating the effectiveness of the treatments. Therefore, we aim to develop a Core Outcome Set (COS) for studies evaluating the effectiveness of treatments for neonatal sepsis. Since a systematic review of key outcomes from randomised trials of therapeutic interventions in neonatal sepsis was published recently, we will complement this with a qualitative systematic review of the key outcomes of neonatal sepsis identified by parents, other family members, parent representatives, healthcare providers, policymakers, and researchers. We will interpret the outcomes of both studies using a previously established framework. Stakeholders across three different groups i.e., (1) researchers, (2) healthcare providers, and (3) patients’ parents/family members and parent representatives will rate the importance of the outcomes in an online Real-Time Delphi Survey. Afterwards, consensus meetings will be held to agree on the final COS through online discussions with key stakeholders. This COS is expected to minimize outcome heterogeneity in measurements and publications, improve comparability and synthesis, and decrease research waste.
... A combination of the Delphi method with a quality control technique to propose a systematic and flexible model for group decision making in the presence of multiple objectives also was proposed by Khorramshahgol [14]. An important modification was introduced to the Delphi method, such as the minimization or elimination of the sequential rounds in a way that improves drastically the method efficiency [15]. The minimization of the non-response of the Delphi participants was achieved by a series of proposed options [16]. ...
... Although there is no univocality, the consulted literature establishes that a large number of experts is not necessary. Studies such as Gordon and Pease (2006), Kauko and Palmroos (2014) and Liimatainen et al. (2014) agree that a group of between 8 and 25 experts is sufficient. Meanwhile, other authors point out that increasing the size of the expert consultation does not necessarily improve precision. ...
The objective of this article is to identify and prioritize technologies, innovations and new businesses related to the dairy agro-industrial chain that are expected to emerge by 2035. To do so, the two-round Delphi method was used and questionnaires were applied to 27 national and international experts. A technology tree was built with Python codes and libraries, consisting of 174 topics. Additionally, 39 variables were generated for scenarios in the Good Livestock Practices BPG; Research, Development and Innovation R&D&I; Sustainable Livestock and Agroindustry groups, as well as four hypotheses and a bet scenario, with the future objectives of sustainable specialization of forage production and mass production and standardization in collection centers. This can be achieved through projects on technologies and innovations prioritized in the Delphi method, including ultrasound, pulsed combustion drying, dairy-derived medicinal products, bioethanol produced from whey, artificial intelligence and selection assisted by molecular markers, electromembrane filtration technologies, whey protein concentrates, life cycle assessment, blockchain, neural networks and smart assays, among others. The opportunity that actors in the Science, Technology and Innovation system have in the chain for the development of programs, plans, public policies and open innovation challenges in the prioritized technologies is highlighted.
... The respondent profiles are summarized in Table 1. Gordon and Pease (2006) recommended piloting the primary Delphi survey before sending it out to the panel. Thus, this survey has been piloted with two experts. ...
Purpose-Construction projects in developing countries frequently face criticism for engaging in social and environmentally irresponsible practices. Therefore, the purpose behind this research stems from the critique, requiring to formulate a comprehensive corporate social responsibility (CSR) framework that integrates sustainable innovation into business processes through stakeholder engagement. Design/methodology/approach-In total, 23 experts from the Pakistan construction industry took part in the Delphi study. Criteria set in terms of diverse backgrounds, organization size, characteristics or experiences of experts. Findings-The finding reveals that the integration of CSR with sustainable innovation demands significant change in business processes, starting from planning, implementing and oversighting stages of project management and a demand for top management commitment to realize sustainable benefits that deliver value to all stakeholders. Originality/value-Construction projects in developing countries frequently face criticism for engaging in socially and environmentally irresponsible practices. Therefore, the purpose behind this research stems from the critique, requiring to formulate a comprehensive CSR framework that integrates sustainable innovation into business processes through stakeholder engagement. In total, 23 experts from the Pakistan construction industry took part in the Delphi study. Criteria set in terms of diverse backgrounds, organization size, characteristics, or experiences of experts. The finding reveals that the integration of CSR with sustainable innovation demand significant change in business processes, starting from the planning, implementing and oversighting stages of project management and a demand for top management commitment to realize sustainable benefits that deliver value to all stakeholders.
... A Real-Time Delphi review workshop (Gordon & Pease, 2006) was held during a face-to-face meeting organized on the pre-congress days of the 12th HOPE Congress in Milan, Italy, from May 8-12, 2023. This round discussed and provided feedback on the USRM, as organized and presented by the Delphi team. ...
Prolonged school absences among children and young people hospitalized due to medical or mental health conditions can significantly disrupt their social and academic development. This study addresses the critical process of reintegrating these learners into their school environments and develops a comprehensive, consensus-based model to facilitate successful school reentry. Utilizing the Delphi method, the research collected insights from 56 experts across 18 countries, representing education, healthcare, and academia, over three rounds of consultation between 2022 and 2024. The findings identify key temporal phases and actions crucial for planning and executing reentry strategies. The resulting models (one for physical health conditions and another for mental health conditions) offer structured guidance, emphasizing multidisciplinary collaboration, context-specific adaptations, and the active participation of the students in the process. The study underscores the need for tailored interventions that address the unique challenges faced by each learner, emphasizing the importance of integrating educational, healthcare, and social support systems with the local culture and values of children and their families to foster resilience and successful reintegration.
... The existing literature indicates that a large number of experts is not always necessary. For instance, studies such as [68][69][70][71][72][73][74] suggest that a participant pool ranging from 8 to 25 experts is often sufficient, emphasizing the importance of domain-specific expertise (in this case, expertise in the plantain sector value chain). While some researchers argue that increasing the size of the expert panel does not necessarily enhance precision, others propose that a range of 5 to 20 stakeholders is adequate [75,76]. ...
This study proposes a sustainable Innovation Management Model (MGI) for the plantain agroindustrial chain, applied to the Quindío region of Colombia. A mixed-methods approach was employed, incorporating various techniques, including a systematic literature review using the Scopus database, which analyzed 95 documents through the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework. The identification and prioritization of relevant variables, along with the validation of the proposed model, were also conducted through the use of three questionnaires, supplemented by relevance and congruence indices. The analysis identified 53 variables for the entry stage, 36 for the transformation stage, and 31 for the exit stage. The findings revealed significant gaps within the agroindustrial chain, particularly in areas such as leadership, strategic management, and resilience among key stakeholders. The MGI, as prioritized by stakeholders, provides a framework that integrates variables and information flows, facilitating the development of new open innovation challenges. These challenges engage not only seekers—entities that require innovation—but also solvers—organizations that offer solutions. The implementation of this model will require supportive public policies, projects, and web platforms, such as Innocentive/Wazoku, Ninesigma, Ruta N, Yet2.com, Itonics, and Ennomotive, which, with the aid of artificial intelligence, can connect innovation supply and demand through open challenges.
... One largely overlooked dimension of the real-time Delphi process (Gnatzy et al., 2011) is the value of the discussion element, which has the power to evoke insightful background information, nuance, evidence of heterogeneity of individual-and crop-led thinking. In addition to the resulting quantitative output, the process produces a rich idea-base and reasoning supporting the responses (Gordon and Pease, 2006). ...
... The RTD approach, however, collapses data collection into a 'roundless approach', enhancing efficiency and interactivity (T. Gordon & Pease, 2006), and mitigating the traditional Delphi's issues that are resource-intensive and the response burden can lead to participant attrition (Gnatzy et al., 2011;Hsu & Sandford, 2007). ...
In humanitarian settings with high levels of sexual violence (SV), care is often offered through fragmented silos, exacerbating the burden on the health workforce and survivors. We aimed to identify contextual and health systems barriers and enablers to providing integrated medical and mental health & psychosocial support (MHPSS) in the care for SV survivors in humanitarian settings. Using Valentijn’s framework, a qualitative, real-time Delphi study (RTD) approach was conducted with 17 experts representing seven geographical subregions. Challenges and enablers identified across the participants’ contexts were consistent. Contextual challenges included volatile contexts, collapsed health systems, and insufficient basic infrastructure. Professional-related challenges included lacking expertise among healthcare professionals (HCPs), high staff attrition rates, and compassion fatigue. Health systems-related challenges included poor referral and coordination mechanisms, lack of funding and resources, misaligned donor priorities and low prioritisation of SV comprehensive care. Effective networking, community engagement, capacity building, co-locating services, participatory management, promoting employees’ sense of ownership, establishing a digital information system, and a unified joint patient file were key identified enablers. Further research should be conducted to assess HCPs’ and SV survivors’ perceptions and experiences of how best to integrate MHPSS services, and understand the challenges and opportunities in delivering integrated services.
... Participants can contribute comments and revisit their responses and consensus multiple times throughout the study without the burden of engagement and analysis required in traditional roundbased Delphi studies. 18,19 Similar to other modified Delphi processes, the RT Delphi methods facilitates researchers to present participants with an initial survey based on a review and consideration of the best available evidence. The RT Delphi format has been validated against traditional approaches with researchers commenting on its practicality and efficiency within health care settings. ...
OBJECTIVES: We sought to develop recommendations to inform a framework for comprehensively assessing upper and lower limb pain in athletes including the key assessment items sports physiotherapists should consider.
DESIGN: Real-time Delphi
METHODS: We recruited sports physiotherapists who were currently working with athletes through the International Federation of Sports Physical Therapists and Irish Society of Chartered Physiotherapists. Participants voted on 86 pain assessment items chosen using best available evidence. The real-time Delphi method facilitated independent anonymous voting, commenting and immediate review of consensus. Participants indicated level of agreement for inclusion in an upper and lower limb athlete pain assessment framework on a 6-point Likert scale from strongly disagree to strongly agree, and how often they are/will be required in practice on a 5-point scale from never to always. Criteria for consensus agreement and inclusion were i) >70% sports physiotherapists voting agree/ strongly agree AND ii) median vote selected by physiotherapists was agree or strongly agree.
RESULTS: Forty-one sports physiotherapists (female n=20, male n=21), visited the survey an average of 5.3 times (±5), resulting in a completion rate of 98%. Sixty-four assessment items (neurophysiological n=20, biomechanical n=15, affective n=8, cognitive n=3, socioenvironmental n=10, general assessment aspects of assessment n=8) met the criteria for consensus. Frequency of use in practice was always for 28 items often for 32 items and sometimes for 4 items.
CONCLUSION: We have presented stakeholder-generated recommendations and priorities for assessing athletes’ pain.
... They can be conducted in a fully digital format, without face-to-face interviewing [78]. Such approaches allow experts to adjust and update their opinions in real time [79]. Given the ability of Delphi studies to provide expert consensus, enhance the accuracy of predictions through iterative feedback, and foster collaboration among diverse stakeholders, we decided to follow this approach for our study. ...
Background
In concert with other digital technologies, artificial intelligence (AI) is shaping the vision of smart hospitals. The transformation into smart hospitals, however, is all but trivial due to the lack of financial and human resources, digital skills, and supporting policies. Thus, the extent to which the vision of smart hospitals will eventually become reality is uncertain. In this context, our study provides a multidimensional conceptualization of the immediate future of smart hospitals to 2042.
Methods
This study employs an iterative mixed-methods approach, including expert workshops and a Delphi study. We conducted a real-time Delphi study to forecast the evolution of smart hospitals in 5-year steps from 2027 to 2042. A total of 39 experts in healthcare, artificial intelligence, and management participated.
Results
Our understanding of a technology-enabled smart hospital in this study includes four dimensions: artificial intelligence (AI), sustainability, ecosystems, and human-centeredness. Our findings underscore the critical need to address the shortage of hospital staff and general practitioners that models predict will peak by 2032. Additionally, our results show a significant shift to individualized medicine and home care. This shift indicates that smart hospitals are expected to leverage AI and digital technologies to tailor care to each patient. Furthermore, the roles and responsibilities of hospital staff will undergo significant changes. Healthcare personnel will have to adapt to new technologies that facilitate more efficient workflows and improve patient engagement in evolving healthcare environments. The results of our study suggest a shift in care to individualized medicine and home care, with corresponding changes in the roles and responsibilities of hospital staff who will employ new technologies.
Conclusions
The findings from our real-time Delphi study suggest that the vision of smart hospitals is gradually becoming reality over the next 20 years. Advancements in artificial intelligence should enhance operational efficiency and patient-centric care, while facilitating the integration of sustainability practices and fostering collaborative ecosystems. However, addressing challenges such as staff shortages, ethical considerations, and the need for robust digital skills will be essential. A deep pool of expert healthcare practitioners, clear ethical guidelines, and robust digital skills are essential to fully realize this vision and ensure that smart hospitals can meet the evolving needs of healthcare delivery.
... A praxe do método Delphi é a realização de mais de um ciclo de avaliação, mas Gordon e Pease (2006) destacam que o método não se limita a mais de uma rodada, a depender dos objetivos do estudo e do públicoalvo dos resultados. Marques e Freitas (2018) assinalam que não há na literatura científica padrões de corte para atestar a convergência de informações, já que cada pesquisa tem suas especificidades e o maior valor gerado pela aplicação do método Delphi é o conjunto de ideias e insights para as próximas etapas de um projeto (Gordon & Pease 2006). Assim, optou-se por realizar apenas uma rodada e sem a adoção de corte de convergência das respostas. ...
As microalgas e as cianobactérias são versáteis para diversos segmentos industriais, sendo insumos importantes para o fortalecimento da bioeconomia e para garantir a sustentabilidade de processos agroindustriais ao promover o uso racional do capital natural. A grande diversidade de mercados que podem ser contemplados pela exploração de microalgas e cianobactérias sinaliza a necessidade de investigar aspectos positivos e negativos do mercado brasileiro para impulsionar o mercado. O objetivo deste trabalho foi, pela aplicação do método Delphi de consulta a especialistas, identificar oportunidades científicas e tecnológicas para exploração de microalgas e cianobactérias no Brasil nos próximos dez anos. Segundo os especialistas consultados, os gêneros de maior destaque são Arthrospira sp., Chlorella sp., e Dunaliella sp. para os mercados farmacêutico/cosmético e de alimentação e suplementação animal. O Brasil tem grande potencial para se destacar no mercado de microalgas e cianobactérias, conforme já existe para outros microrganismos. No entanto, há empecilhos para explorar o potencial industrial das microalgas e cianobactérias para incluir o Brasil no mercado internacional a preços competitivos, dentre eles a baixa disponibilidade de recursos financeiros para investimentos em pesquisa, desenvolvimento e inovação e a escassez de profissionais para escalonamento de cultivo para oferta de biomassa. Palavras-chave: prospecção tecnológica; microalgas; cianobactérias. ABSTRACT Microalgae and cyanobacteria are versatile for various industrial segments and are important inputs for strengthening the bioeconomy and ensuring the sustainability of agroindustrial processes by promoting the rational use of natural capital. The wide range of markets that can be covered by the exploration of microalgae and cyanobacteria indicates the need to investigate positive and negative aspects of the Brazilian market in order to boost the market. The objective of this study was to identify scientific and technological opportunities for the exploration of microalgae and cyanobacteria in Brazil over the next ten years by applying the Delphi method of consulting experts. According to the experts consulted, the most prominent genera are Arthrospira sp., Chlorella sp., and Dunaliella sp. for the pharmaceutical/cosmetic and animal feed and supplement markets. Brazil has great potential to stand out in the microalgae and cyanobacteria market, as it already does for other microorganisms. However, there are obstacles to exploring the industrial potential of microalgae and cyanobacteria to include Brazil in the international market at competitive prices, including the low availability of financial resources for investments in research, development and innovation and the shortage of professionals to scale up cultivation to supply biomass.
... A praxe do método Delphi é a realização de mais de um ciclo de avaliação, mas Gordon e Pease (2006) destacam que o método não se limita a mais de uma rodada, a depender dos objetivos do estudo e do públicoalvo dos resultados. Marques e Freitas (2018) assinalam que não há na literatura científica padrões de corte para atestar a convergência de informações, já que cada pesquisa tem suas especificidades e o maior valor gerado pela aplicação do método Delphi é o conjunto de ideias e insights para as próximas etapas de um projeto (Gordon & Pease 2006). Assim, optou-se por realizar apenas uma rodada e sem a adoção de corte de convergência das respostas. ...
As microalgas e as cianobactérias são versáteis para diversos segmentos industriais, sendo insumos importantes para o fortalecimento da bioeconomia e para garantir a sustentabilidade de processos agroindustriais ao promover o uso racional do capital natural. A grande diversidade de mercados que podem ser contemplados pela exploração de microalgas e cianobactérias sinaliza a necessidade de investigar aspectos positivos e negativos do mercado brasileiro para impulsionar o mercado. O objetivo deste trabalho foi, pela aplicação do método Delphi de consulta a especialistas, identificar oportunidades científicas e tecnológicas para exploração de microalgas e cianobactérias no Brasil nos próximos dez anos. Segundo os especialistas consultados, os gêneros de maior destaque são Arthrospira sp., Chlorella sp., e Dunaliella sp. para os mercados farmacêutico/cosmético e de alimentação e suplementação animal. O Brasil tem grande potencial para se destacar no mercado de microalgas e cianobactérias, conforme já existe para outros microrganismos. No entanto, há empecilhos para explorar o potencial industrial das microalgas e cianobactérias para incluir o Brasil no mercado internacional a preços competitivos, dentre eles a baixa disponibilidade de recursos financeiros para investimentos em pesquisa, desenvolvimento e inovação e a escassez de profissionais para escalonamento de cultivo para oferta de biomassa.
... Una variación del Método Delphi es su versión en tiempo real, el cual permite aplicar solo una ronda del cuestionario, es decir las primeras cinco fases y la última, se caracteriza por presentar al participante de la encuesta lo siguiente (Gordon & Pease, 2005): Un espacio para que un encuestado proporcione su respuesta de cada ítem en una escala tipo Likert, El promedio de todas las respuestas del grupo hasta el momento, El número de respuestas hasta el momento, Una ventana que muestra las razones que sustentan las respuestas de otros encuestados, Se recoge de los encuestados las razones del sustento para sus propias respuestas. ...
En el contexto de cambios a nivel mundial, en la literatura se señala que la formación de nuevos negocios impulsa la economía mediante nuevos empleos e ingresos para los países, pues los países preocupados en mejorar las condiciones para el futuro en su política pública consideran el emprendimiento, la innovación y las condiciones para que se den estos fenómenos como importantes. En esta situación son de importancia las startups, integradas por equipos de emprendedores muy peculiares porque innovan dando origen a nuevas empresas de base tecnológica con recursos limitados. De acuerdo a la literatura las startups tienen interdependencia con una comunidad de emprendedores, mentores, incubadoras, aceleradoras, proveedores de servicios comunes, inversionistas ángeles, inversionistas de capital de riesgo, universidades, entidades públicas de apoyo, elementos que forman un ecosistema y este se vincula con otros ecosistemas. En investigaciones previas, primero un estudio de casos múltiple, se encontró que desde la perspectiva de los fundadores de startups, existe presencia de los diferentes elementos en el ecosistema de Lima y en una subsiguiente investigación, en base al análisis de redes inter-organizacionales, se encontró que los inversionistas son los actores más influyentes, además que las universidades y las instituciones públicas tienen una alta vinculación entre ellas. De este modo, este trabajo se enfoca en este ecosistema con el objetivo de comprender los desafíos al año 2030 mediante herramientas de la prospectiva, en especial aplicando el método Delphi en tiempo real, a una muestra de los fundadores de las startups. Inicialmente, se exponen los diferentes elementos de un ecosistema de emprendimiento tecnológico así como la teoría de la prospectiva. También se expone la metodología utilizada, así como el proceso de esta metodología, que permite identificar los probables escenarios al 2030 para el ecosistema de emprendimiento tecnológico en Lima. Finalmente, se presentan las conclusiones correspondientes al trabajo.
... Today, Delphi studies are widely used to derive and discuss opinions from groups of experts (Landeta, 2006). Methodologically, Delphi studies rely on evaluating clearly defined projections, which experts evaluate in various survey rounds that are increasingly conducted in real-time via dedicated online platforms (known as RT or Real-Time Delphi) (Gordon and Pease, 2006;von der Gracht and Darkow, 2010). Real-Time Delphi studies promise to contain common issues of round-based formats, like high dropout rates, low interaction, low engagement, and long study duration with high moderator effort . ...
Hydrogen might play an essential role in mitigating climate change. It can be applied across a set of both easy and hard-to-abate use-cases. But, most hydrogen-based technologies are not yet market-ready. To prevent wrong investment decisions, both corporates and policymakers need transparency on where the use of hydrogen is most likely and which technologies will be required. Due to their interdependence, all hydrogen-enabling technologies (e.g., fuel cells, electrolyzers, liquid hydrogen shipping) should be seen as a field of interrelated technologies rather than a disjunct set. Past studies viewed single hydrogen-based technologies as isolated or resorted to demand forecasting without detailing the required technologies. Thus, we ran an adapted Delphi-study to create foresight for entire technology fields and create future scenarios purely from Delphi-data. Additionally, we provide an up-to-date holistic scenario-driven view on the future development of the technology field of hydrogen in the year 2035, including its consequences. We ran two interconnected Delphi studies with 50 subject experts. Our results recommend a more targeted research and investment approach to bringing sustainable technologies for the right use-case to market.
... Specifically, in this paper, we refer to the method proposed by Di Zio et al. (2017), namely the Real-Time Spatial Delphi (RTSD). RTSD is a customized approach combining the Real-Time Delphi (Gordon and Pease, 2006) and Spatial Delphi (Di Zio and Pacinelli, 2011), specifically designed to aid in foresight and decision-making. It leverages Geographic Information Systems (GIS) and multiple spatial technologies to facilitate expert communication and collaboration within a virtual environment with the final aim to obtain a convergence of opinions on the territory. ...
In the Futures Studies context (FS), scenarios refer to speculative visions of potential
future states or situations that may arise based on a combination of existing trends,
emerging technologies, and variousfactors(Kosow and Gaßner, 2008). Scenarios provide
a framework for exploring and anticipating various possibilities, enabling individuals,
organizations, and policymakers to better prepare for the uncertainties of tomorrow.
Among the many methods used, the scenario method can be combined with different
approaches including the Delphi method, forming the so-called Delphi-based scenarios.
In the spatial context, one of the main implementations is the Real-Time Spatial Delphi
(Di Zio et al. 2017), a spatial version of the conventional Delphi method, integrating real-time feedback and visualization of group judgments. Its primary objective is to facilitate
consensus-building among experts regarding geographical locations in the decision-making and forecasting process. Nevertheless, while these assessments provide spatial
areas represented on digital maps, they do not manifest in the physical realm. With the
emergence of AI models, alternative implementations, such as leveraging Generative
Adversarial Networks (GANs), can enhance the visioning and planning process. In this
paper, we propose a hybrid approach combining Real-Time Spatial Delphi and GANs to
enhance the envisioning of spatial scenarios with the aim both to sensitize experts’
opinions with clear and realistic images of the future and plan suitable policies in the
present. In our approach, once a narrow area is denoted by the experts, we capture digital
images of these areas and employ GANs models to generate visual outputs of the
scenarios or suggested plans for specific points inside the consensus area. This innovative
approach aids experts and policymakers in visualizing the proposed policies more
effectively, enabling them to assess the impacts with greater accuracy. We apply the
method to the city of Dublin in the climate change context
... In this round-less method, participants can view the group response in real-time after responding and revisit and re-rate responses based on group feedback. 28 The Real-Time Delphi (RTD) approach has been successfully used in the medicine. 29,30 A randomized trial of the traditional approach and real-time approach for developing a core outcome set in NE showed no differences in results between the two methods. ...
Background
‘Neonatal encephalopathy’ (NE) describes a group of conditions in term infants presenting in the earliest days after birth with disturbed neurological function of cerebral origin. NE is aetiologically heterogenous; one cause is peripartum hypoxic ischaemia. Lack of uniformity in the terminology used to describe NE and its diagnostic criteria creates difficulty in the design and interpretation of research and complicates communication with families. The DEFINE study aims to use a modified Delphi approach to form a consensus definition for NE, and diagnostic criteria.
Methods
Directed by an international steering group, we will conduct a systematic review of the literature to assess the terminology used in trials of NE, and with their guidance perform an online Real-time Delphi survey to develop a consensus diagnosis and criteria for NE. A consensus meeting will be held to agree on the final terminology and criteria, and the outcome disseminated widely.
Discussion
A clear and consistent consensus-based definition of NE and criteria for its diagnosis, achieved by use of a modified Delphi technique, will enable more comparability of research results and improved communication among professionals and with families.
Impact
The terms Neonatal Encephalopathy and Hypoxic Ischaemic Encephalopathy tend to be used interchangeably in the literature to describe a term newborn with signs of encephalopathy at birth. This creates difficulty in communication with families and carers, and between medical professionals and researchers, as well as creating difficulty with performance of research.
The DEFINE project will use a Real-time Delphi approach to create a consensus definition for the term ‘Neonatal Encephalopathy’.
A definition formed by this consensus approach will be accepted and utilised by the neonatal community to improve research, outcomes, and parental experience.
... Anteil der Absolventen, die ein Startup gewählt haben, den Anteil der Studentinnen im Studienprogramm, den Anteil der Unternehmerinnen und Unternehmer, die mindestens ein Drittel des Eigenkapitals von privaten Investoren akquiriert haben, den Anteil der Startups, die zum Zeitpunkt des Rankings noch aktiv waren, das Ausmaß, in dem die erworbenen Fähigkeiten Delphi-Studien unter Einsatz einer Spezialsoftware -hier: Calibrum -stattdessen in einer längeren Runde(Gnatzy et al. 2011;Gordon/Pease 2006). Die mehrfache Bewertung dient dazu, die Streuung der gegebenen Antworten zu reduzieren bzw. ...
Entrepreneurship Education zielt darauf, Lernende zu Entrepreneur*Innen zu qualifizieren. Der weit überwiegende Teil der internationalen Entrepreneurship-Education-Forschung geht der Frage nach, wie erfolgreich – gemessen an unterschiedlichen Variablen, hauptsächlich der Gründungsabsicht – sie dabei ist. Dabei wird Entrepreneurship Education in der Regel jedoch unzutreffend als homogenes Gut betrachtet. Anstelle der dominierenden Ergebnis-Orientierung, nimmt der vorliegende Beitrag eine Binnenperspektive ein und geht der Forschungsfrage nach, wie Entrepreneurship auf tertiärer Ebene international gelehrt (Status quo) wird und wie die internationale tertiäre Entrepreneurship-Lehre optimiert werden kann (Verbesserungspotenziale) – im Hinblick auf zu identifizierende entrepreneurship-spezifische Lehr-/Lern-Ziele, Lehr-/Lern-Inhalte, Lehr-/Lern-Methoden und Methoden der Lernerfolgskontrolle. Zu diesem Zweck berichtet der Beitrag über fünf Teilstudien und führt diese zu einem holistischen Entrepreneurship-Curriculum zusammen. Der Status quo wird zum einen aus der Perspektive der Entrepreneurship-Education-Forschung durch eine bibliometrische Analyse und – spezifischer – aus der Perspektive des Teilgebiets der curricularen Entrepreneurship-Education-Forschung durch eine systematische Literaturanalyse, zum anderen aus der Perspektive der entrepreneurship-bezogenen Lehrpraxis mit Hilfe einer Curriculumanalyse ermittelt. Verbesserungspotenziale werden aus der Perspektive von Forschung und Lehre durch eine Delphi-Studie unter Entrepreneurship-Professor*Innen sowie aus der Perspektive der unternehmerischen Praxis durch problemzentrierte Interviews mit Unternehmer*Innen erhoben. Der Beitrag schließt mit theoretischen, lehrpraktischen und bildungspolitischen Implikationen sowie einem Forschungsausblick.
... A modified Delphi process based on an online round-less real-time format 30 was adopted to evaluate the telemedicine implementation constructs that were identified in phase 1. This study approach best suited the needs of the research, which calls for expert judgement where there is a lack of consensus, and a consolidation of geographically dispersed expert opinion. ...
Objectives
The call to scale up telemedicine services globally as part of the digital health transformation lacks an agreed-upon set of constructs to guide the implementation process. A lack of guidance hinders the development, consolidation, sustainability and optimisation of telemedicine services. The study aims to reach consensus among telemedicine experts on a set of implementation constructs to be developed into an evidence-based support tool.
Methods
A modified Delphi study was conducted to evaluate a set of evidence-informed telemedicine implementation constructs comprising cores, domains and items. The study evaluated the constructs consisting of five cores: Assessment of the Current Situation, Development of a Telemedicine Strategy, Development of Organisational Changes, Development of a Telemedicine Service, and Monitoring, Evaluation and Optimisation of Telemedicine Implementation; seven domains: Individual Readiness, Organisational Readiness, Clinical, Economic, Technological and Infrastructure, Regulation, and Monitoring, Evaluation and Optimisation; divided into 53 items. Global telemedicine specialists (n = 247) were invited to participate and evaluate 58 questions. Consensus was set at ≥70%.
Results
Forty-five experts completed the survey. Consensus was reached on 78% of the constructs evaluated. Regarding the core constructs, Monitoring, Evaluation and Optimisation of Telemedicine Implementation was determined to be the most important one, and Development of a Telemedicine Strategy the least. As for the domains, the Clinical one had the highest level of consensus, and the Economic one had the lowest.
Conclusions
This research advances the field of telemedicine, providing expert consensus on a set of implementation constructs. The findings also highlight considerable divergence in expert opinion on the constructs of reimbursement and incentive mechanisms, resistance to change, and telemedicine champions. The lack of agreement on these constructs warrants attention and may partly explain the barriers that telemedicine services continue to face in the implementation process.
... In the end, 50 valid questionnaires were analyzed. The survey was implemented as a real-time Delphi in spring 2020 using the software Calibrum; the participating experts could read and follow the opinions in the statements of other participants right from the start (Aengenheyster et al. 2017;Gnatzy et al. 2011;Gordon and Pease 2006). The statements for the Delphi survey were based on the results of the preceding expert interviews and the literature research and focused on the professions in the crafts sector, the skills needed for the future, challenges and opportunities, and measures required to ensure change. ...
The crafts sector, traditionally characterized by its adherence to heritage and small-scale familial enterprises, confronts an array of unprecedented challenges stemming from rapid technological advancement, globalization, climate change, and shifting demographics. This article presents a pioneering investigation into the profound and transformative effects of digitalization on the crafts sector, offering novel insights into the future trajectory of this industry. Through an explorative approach, we critically examine and interrogate the social imaginary underpinning the crafts sector, shedding light on the intricate interplay between tradition and innovation. Leveraging an inter- and transdisciplinary framework, our research brings together academics, experts, and practitioners from diverse regions in Italy and Austria to explore the nexus of digitalization and craft futures. We introduce three future scenarios for the crafts sector, each delineating varying degrees of digitalization and their potential implications. Employing a mixed-method approach encompassing expert interviews, a Delphi survey, focus groups, and scenario development, our study offers a comprehensive exploration of the multifaceted landscape of craft futures. Furthermore, through the visualization of these scenarios, we provide a tangible platform for envisioning and engaging with the myriad possibilities that lie ahead. By synthesizing innovative methodologies from future studies and social imaginaries with a specific focus on the crafts sector, our research offers a robust analytical framework for navigating the complex dynamics of digitalization and envisioning transformative futures. This paper not only illuminates the disruptive forces unleashed by digitalization but also provides strategic insights to inform future decision-making processes within the crafts sector and beyond.
... This is realized by the collection of topical aspects in the first stage and a rating of them in the second stage of the study. The second stage can either be separated in several iterative rounds or, like in our study, take place in a real-time design [63,64]. In both cases, the study participants rate the items, then see the interim results of the group rating, and can adjust their rating if they prefer. ...
Existing curricula for entrepreneurship education do not necessarily represent the best way of teaching. How could entrepreneurship curricula be improved? To answer this question, we aim to identify and rank desirable teaching objectives, teaching contents, teaching methods, and assessment methods for higher entrepreneurship education. To this end, we employ an international real-time Delphi study with an expert panel consisting of entrepreneurship education instructors and researchers. The study reveals 17 favorable objectives, 17 items of content, 25 teaching methods, and 15 assessment methods, which are ranked according to their desirability and the group consensus. We contribute to entrepreneurship curriculum research by adding a normative perspective.
... We used a Delphi approach to identify guideline items and obtain the expert panel's level of agreement on the items put forth [50,51]. Guideline items were recommended by the expert panel, identified from previously published literature reviews that highlighted insufficiencies in the reporting of stimulation parameters for experiments utilizing electrical SCS following SCI [31,32,34,[37][38][39]. ...
Objective. Electrical spinal cord stimulation (SCS) has emerged as a promising therapy for recovery of motor and autonomic dysfunctions following spinal cord injury (SCI). Despite the rise in studies using SCS for SCI complications, there are no standard guidelines for reporting SCS parameters in research publications, making it challenging to compare, interpret or reproduce reported effects across experimental studies. Approach. To develop guidelines for minimum reporting standards for SCS parameters in pre-clinical and clinical SCI research, we gathered an international panel of expert clinicians and scientists. Using a Delphi approach, we developed guideline items and surveyed the panel on their level of agreement for each item. Main results. There was strong agreement on 26 of the 29 items identified for establishing minimum reporting standards for SCS studies. The guidelines encompass three major SCS categories: hardware, configuration and current parameters, and the intervention. Significance. Standardized reporting of stimulation parameters will ensure that SCS studies can be easily analyzed, replicated, and interpreted by the scientific community, thereby expanding the SCS knowledge base and fostering transparency in reporting.
... The structured group communication process should create a convergence or a divergence of opinions, producing a more dynamic and accurate collection of data in comparison to traditional opinion-polling techniques. This method allows researchers to focus on the sharing process, reducing risks related to group dynamics that may emerge during in-person collaborative processes [38,39,40,41,42,37]. To sum up, the Delphi process can be divided in the following different steps: ...
Nowadays, there is growing attention on enhancing the quality of teaching, learning and assessment processes. As a recent EU Report underlines, the assessment and feedback area remains a problematic issue regarding educational professionals' training and adopting new practices. In fact, traditional summative assessment practices are predominantly used in European countries, against the recommendations of the Bologna Process guidelines that promote the implementation of alternative assessment practices that seem crucial in order to engage and provide lifelong learning skills for students, also with the use of technology. Looking at the literature, a series of sustainability problems arise when these requests meet real-world teaching, particularly when academic instructors face the assessment of extensive classes. With the fast advancement in Large Language Models (LLMs) and their increasing availability, affordability and capability, part of the solution to these problems might be at hand. In fact, LLMs can process large amounts of text, summarise and give feedback about it following predetermined criteria. The insights of that analysis can be used both for giving feedback to the student and helping the instructor assess the text. With the proper pedagogical and technological framework, LLMs can disengage instructors from some of the time-related sustainability issues and so from the only choice of the multiple-choice test and similar. For this reason, as a first step, we are proposing a starting point for such a framework to a panel of experts following the Delphi methodology and reporting the results.
Published at: https://ceur-ws.org/Vol-3605/16.pdf
... In recent years, different implementations of the Delphi method have been made, such as the possibility of having a valid and fast method to elicit the experts' judgments. In Real-Time Delphi (Gordon & Pease, 2006), participants have access to an online web-based questionnaire, where based on the answers of the panellists, an automatic calculation of the median and other statistical summaries takes place, so that the actors involved can access the questionnaire several times during the time window of the study. The rounds occur simultaneously and there are no more iterative rounds, which is why the two scholars call the method "round-less". ...
In the Futures Studies context, the Delphi-based scenario (DBS) is a valuable method for setting future-oriented strategies and actions by gathering expert opinions in multiple iterative rounds, in order to make better decisions in the present. However, one of the main challenges is to find a suitable representation of the scenarios both for evaluation and decision-making. The development of future scenarios requires high creativity and simple textual descriptions could not be enough to grab the attention of the experts and decision-makers, due to a probable lack of communication and/or technical language. Since the textual descriptions concern the future, the human mind may have difficulty imagining events that have not already happened. With the spread of AI models, different implementations can support the DBS process, such as the use of Generative Adversarial Networks (GANs) to generate specific images starting from the narratives of the scenarios. In this paper, we propose a new method for the development of Delphi-based Visual Scenarios (DBVS) adopting GANs, with the aim to support the experts and decision-makers in planning and communication, in order to have immediate images of what the future could be like. We apply this method with regard to the future of Italian families.
... The original Delphi methodology has been adapted for different conditions and purposes. Among the most noteworthy modifications to the initial design-for their early development and their subsequent and current success-are the Policy Delphi (De Loë et al., 2016;Turoff, 1970) and the Real-Time Delphi (Gnatzy et al., 2011;Gordon, 2009;Gordon & Pease, 2006;Turoff, 1972). The Policy ...
The Delphi method is a technique of social research that seeks to obtain a reliable group opinion from experts. It was first created for military purposes in the mid‐1950s. Since then, its use in the scientific field has continued to spread to different disciplines and aims. Despite this expansion, however, not set of indicators of the quality of Delphi studies has yet to be developed that might provide the reader—whether an expert in the technique or not—with some framework of reference whereby to gauge what credibility should be afforded to the results of the study. In this paper, following a thorough review of the literature on the criteria used to assess Delphi studies and the items of evaluation recommended for inclusion in Delphi reports, we determine what characteristics a quality evaluation indicator for this technique should have and propose a battery of indicators based on these characteristics, which should for preference be included in the final report of a Delphi study. The proposed indicators focus on three areas that are particularly relevant to the quality of Delphi research: the quality of the panel of participating experts, the way in which relevant information is obtained from the experts, and the quality of the interaction generated among the experts.
Argumentative Delphi-Befragungen gibt es schon seit den ersten Experimenten mit Delphi-Verfahren, die anfangs in zwei oder mehr Runden abliefen (Dalkey 1968; 1969a, b). Neu ist die Variante der dynamischen argumentativen Delphi-Methode (DAD). Sie wurde in Rumänien entwickelt und dort erstmals in nationalen Projekten ausprobiert. Das dynamische, argumentative Delphi ist ein Realtime Delphi, d.h. es verfügt über Echtzeit-Feedback-Funktion, und wurde im Projekt BOHEMIA auf europäischer Ebene erprobt. Der folgende Beitrag beschreibt die neue Methodik und seine erste umfassende Anwendung im Projekt BOHEMIA für die Europäische Kommission sowie die Herausforderungen, mit denen das Projektteam in der technischen und inhaltlichen Umsetzung konfrontiert wurde. Aus ihnen können wir Forschende in der Zukunftsforschung bzw. im Foresight Lehren für die weitere Anwendung der neuen Delphi-Variante ziehen. Die besonderen Vorteile des Verfahrens werden diskutiert, Nachteile sowie neue Anwendungsmöglichkeiten für argumentative Delphi- und einfache Zukunftsbefragungs-Ansätze werden zusammenfassend dargestellt.
Our research has focused on addressing the following research questions for the growth strategies of SMEs: (1) What are the stages of profitable growth, and what factors contribute to these growth stages? (2) Which critical factors must be addressed for an organization to progress to the next stages of growth? (3) What is the importance of management in identifying and addressing critical growth factors? We have utilized the Delphi method and emphasized the role of company managers who have experienced profitable growth process as experts. Based on our findings, we have developed a Six-Stage Model of Profitable Growth (SSMPG), which we explain in detail in the article. The SSMPG model is compared to the prevailing Death Valley and Startup growth company development models. The article identifies the most crucial factors for the profitable growth of SMEs at different stages of growth within the SSMPG model. This model emphasizes sales, profitability, the individual characteristics of the entrepreneur, and leadership, in contrast to the debt-driven growth models emphasized in the other approaches. Further research could explore developing a start-up business culture using the new phasing model in Europe and elsewhere. In the future, it is important to consider profitability at both the company level and within clusters and regions.
The present study is a field research with a descriptive-analytical approach and seeks to answer the basic question: What are the key components of increasing physical resilience in the 10th district of Tehran in the future according to the current situation? In this research, in order to codify a theoretical framework at first, the subject is described in detail using the documentary method, literature and theoretical foundations, and then by relying on the environmental scanning technique and the Delphi method, the experimental data is extracted. The statistical population includes 20 urban experts and specialists. Also, the sampling method used is a purposive sampling. In the first step of the research, based on the review of theoretical texts and documents about resilience and dysfunctional urban texture with emphasis on the physical dimension, 41 variables were extracted and then in the next step, experts' opinions regarding the outcomes of these variables on each other through distribution of 20 Questionnaires were collected and finally the answers were analyzed using structural interaction analysis method in MICMAC software. In this analysis, the extent of direct and indirect effects of variables on each other and on the future trend of physical resilience in the district 10 of Tehran against earthquake was determined and then the key explanatory factors of the research topic were extracted. In the other words, out of a total of 41 influential initial variables, 11 variables were chosen as the key factors affecting the future trend of physical resilience in the district 10 of Tehran in the face of earthquakes. The aftermath of this study represents that the relationships of key components are so effectual and impressionable.
Introduction
Neonatal death exerts long-lasting impact on parents’ mental health, finances and relationships, and the wider family. There is national and international momentum to evaluate interventions to support parents after the death of a baby. Core Outcome Sets (COSs) provide a minimum set of outcomes, agreed by stakeholders to be important, which should be evaluated in all studies to support evidence syntheses and identification of the most effective interventions. We aim to develop a COS for assessing interventions and care after neonatal death in high-income countries, to support future evidence syntheses and enable the identification of effective interventions and care for parents.
Methods and analysis
We will develop the COS in six phases. A parent involvement group and stakeholder steering committee have been established and have informed each planned phase: (1) systematic review of quantitative studies evaluating care and interventions provided after neonatal death, to describe interventions, outcomes and outcome measurement tools used to assess intervention effectiveness; (2) qualitative interviews with parents who have experienced neonatal death to identify outcomes important and relevant to them; (3) think-aloud interviews with stakeholders (bereaved parents, healthcare professionals and other stakeholders) to develop and refine an online survey; (4) real-time online international Delphi survey with bereaved parents, healthcare professionals and other stakeholders to shortlist outcomes for consideration in the COS; (5) adapted nominal group online consensus meetings with parents, healthcare professionals and other stakeholders to agree final COS and (6) identification of a preliminary set of measurement tools.
Ethics and dissemination
Ethical approval has been granted for all activities to be undertaken by the University of Bristol Health Sciences Faculty Research Ethics Committee (reference: 15121). We will disseminate the findings via peer-reviewed publications and relevant academic and professional conferences.
PROSPERO registration number
CRD42020151365.
Purpose
Traceability systems (TS) have enabled significant improvements in supply chain management. Despite these advancements, there is room for further enhancement in the acceptance and diffusion of TS among stakeholders in emergent agri-food supply chains. Conducting a strategic analysis of TS is crucial to reveal the associated challenges, opportunities, pros and cons. Doing so will foster the development of emergent agri-food supply chains and unlock their potential.
Design/methodology/approach
This study used a modified quantitative four-round Delphi methodology among experts, selected according to the quadruple helix model, covering end-to-end the supply chain. The research was carried out in Greece, as its agri-food sector epitomizes the type of emergent supply chain under investigation.
Findings
This study reveals the impact of TS in emergent agri-food industries, emphasizing the firm-focused positive effects like improving their supply chain performance. The research also highlights the benefits of TS adoption, suggesting that they outweigh any drawbacks based on the TS internal-external environment analysis. The multi-stakeholder participatory approach resulted in a blend of well-informed policy-making recommendations and managerial insights, paving the way for a broader and more efficient TS adoption in emergent agri-food supply chains.
Research limitations/implications
Impediments on sampling require caution when extending these findings to other contexts. Yet, it is crucial to conduct further research using a similar approach in a broader area, not just limited to Greece but also in other economies with similar conditions characterized by nascent supply chains.
Originality/value
The comprehensive and holistic approach used in this study, examining the extended supply chain context instead of just dyads, offers valuable insights that contribute to theory development in supply chain management. Such an output is momentous when researching emergent supply chains and complex issues like traceability, which require information sharing and involve interdependencies and hidden dynamics among stakeholders that are impossible to explore otherwise.
Background
People with dementia often experience violations of fundamental human rights and impeded access to healthcare. This study aims to investigate the views of experts regarding the use of the United Nations Convention on the Rights of People with Disabilities (CRPD) principles as quality standards for human rights–based care.
Methods
A single‐round Delphi e‐consultation with 15 dementia experts was designed to evaluate each CRPD principle and collect feedback on their views about the application of the CRPD principles in dementia care.
Results
The CRPD principles were fully endorsed as quality standards; however, several experts commented on the complexities of the use of CRPD principles in relation to information disclosure, capacity assessment, stakeholders' involvement in decision‐making, respecting needs and preferences, holistic approaches in care practice, and protection against abuse, neglect and discrimination.
Conclusions
These findings indicate the CRPD quality standards were fully applicable for people with dementia although some clarification around interpretation could assist in their use. Future research should elaborate on further points of support and guidance for dementia care providing examples of good practice from across the globe, and develop a concordant, human rights–based scheme for the implementation and evaluation of dementia services.
Implications for Practice
This study suggests that the CRPD can serve as a benchmark for human rights‐based practices in dementia services globally that could enhance nursing care practice. Nursing staff are encouraged to consider human rights in relation to the complexities associated with people with dementia, their caregivers, and other services involved in their care.
In contemporary higher education, aligning student competencies with sustainable development goals (SDGs) is crucial for enhancing their employability. This necessity extends to graphic design undergraduate programs, pivotal in nurturing designers capable of addressing global sustainability challenges. Using a mixed-methods approach, this study aims to outline essential competencies for graphic design undergraduates pertinent to the SDGs. The research commenced with a systematic review of fourteen seminal articles, leading to the initial identification of fifty-eight key competencies. Subsequently, a Delphi study involving 44 graphic design experts validated these competencies. This iterative process refined the list, excluding twenty-seven competencies and affirming thirty one as critical. These validated competencies were categorized into four groups: Systemic, Performance, Contextual, and Global. The study's findings provide valuable insights for graphic design educators and industry employers, illuminating competencies essential for addressing sustainability challenges. Furthermore, this research contributes to academic discourse by offering a structured framework for integrating SDG-related competencies into graphic design education. This facilitates the development of designers proficient not only in their craft but also in sustainability imperatives. Our aim is to enhance pedagogical strategies and inform curriculum development in graphic design education, aligning them with evolving professional demands and sustainable development objectives. Keywords: key competencies, SDG, design education, sustainability, Delphi method, systematic review, graphic design
Objectives: There is no current consensus on the key items sports physiotherapists should consider when completing a comprehensive biopsychosocial upper or lower limb pain assessment with athletes. We sought to develop recommendations to inform a framework for the assessment of upper and lower limb pain in athletes.
Design; Real-time Delphi
Methods: We recruited sports physiotherapists currently working with athletes through the International Society of Sports Physical Therapists and Irish Society of Chartered Physiotherapists. Participants voted on 86 pain assessment items chosen using best available evidence. The real-time Delphi method facilitated independent anonymous voting, commenting and immediate review of consensus. Participants indicated level of agreement for inclusion in an upper and lower limb athlete pain assessment framework on a 6-point Likert scale from Strongly Disagree to Strongly Agree, and how often they are/will be required in practice on a 5-point scale from Never to Always. Criteria for consensus agreement and inclusion were i) >70% sports physiotherapists voting agree/strongly agree AND ii) median vote selected by physiotherapists was Agree or Strongly Agree.
Results: 41 sports physiotherapists (female n=20, male n=21), visited the survey an average of 5.3 times, resulting in a completion rate of 98%. 64 assessment items (neurophysiological n=20, biomechanical n=15, affective n=8, cognitive n=3, socioenvironmental n=10, general assessment aspects of assessment n=8) met the criteria for consensus. Frequency of use in practice was Always for 28 items Often for 32 items and Sometimes for 4 items.
Conclusion: We have presented stakeholder-generated recommendations and priorities for athlete pain assessment.
Introduction: Neonatal death exerts long lasting impact on parents’ mental health, finances and relationships, and the wider family. There is national and international momentum to evaluate interventions to support parents after the death of a baby. Core Outcome Sets provide a minimum set of outcomes, agreed by stakeholders to be important, that should be evaluated in all studies to support evidence syntheses and identification of the most effective interventions.
Aim: We aim to develop a Core Outcome Set (COS) for assessing interventions and care after neonatal death, to support future evidence syntheses and enable identification of effective interventions and care for parents.
Methods and analysis: We will develop the COS in 6 phases. A parent involvement group and stakeholder steering committee have been established and have informed each planned phase: 1) Systematic review of quantitative studies evaluating care and interventions provided after neonatal death, to describe interventions, outcomes, and outcome measurement tools used to assess intervention effectiveness; 2) Qualitative interviews with parents who have experienced neonatal death to identify outcomes important and relevant to them; 3) Think-Aloud interviews with stakeholders (bereaved parents and professionals) to develop and refine an online survey; 4) Real-Time online international Delphi survey with bereaved parents and professionals to shortlist outcomes for consideration in the COS; 5) Adapted nominal group online consensus meetings with parents and professionals to agree final COS; 6) Identification of a preliminary set of measurement tools.
Ethics and dissemination: Ethical approval has been granted for all activities to be undertaken by the University of Bristol Health Sciences Faculty Research Ethics Committee (reference: 15121). We will disseminate the findings via peer-reviewed publications and relevant academic and professional conferences.
Registration: This COS has been registered on the COMET database.(1) The systematic review has been registered on the PROSPERO database (CRD42020151365)
This science for policy brief explores the critical role of Interoperability Assessments (IOPA) introduced by the Interoperable Europe Act, which aims to enhance the compatibility and efficiency of cross-border digital public services. By examining the expected benefits, necessary efforts, and potential challenges, this study employs a Wideband Delphi approach to gather expert insights into the intricacies of conducting IOPA. Key findings emphasize the importance of early integration, continuous investment in interoperability capabilities, and the need for cultural and organizational changes to achieve successful implementation. The study also highlights the long-term benefits of investing in interoperability assessments, including smoother implementation processes, higher-quality public services, and significant effort and time savings.
In Finland, agriculture is practiced in variable growing conditions that are further challenged by climate change and under external pressure caused by international competition, changing consumer preferences and demands, and the renewal of the European Union Common Agriculture Policy and Paris Agreement climate targets. Futures studies in agriculture have focused on scenario building with expert panels, though usually without farmers’ involvement. This study focuses on farmers’ views of shaping the future of Finnish agriculture. It builds on the disaggregative Delphi method and combines interviews and a representative survey of Finnish farmers. The study is based on a bottom-up process in which a farmer panel of 20 farms defined their views of the future in a semi-structured thematic interview. The views were turned into statements in a structured survey sent to the second panel, which comprised all Finnish farmers who had received agricultural subsidies in 2016, with 4401 respondents. The results were analyzed using quantitative factor analysis, which produced five future images for Finnish agriculture. The images were shown to the original farmer panel for reflection. They considered the probability of each future image and their own role in it. Technology solves as a future image was most favored by farmers, followed by Ecological and specializing small-scale production image. Business-as-usual was the least likely future images according to the interviewed farmers. This paper describes the process and discusses both methodological benefits and pitfalls, as well as farmers’ future views of the forthcoming decades.
Background and objective
Consensus methods are crucial in developing clinical guidelines. Different methods, such as the Delphi and nominal group techniques, are commonly used, but there is a lack of detailed instructions on how to implement them effectively. The survey aims to explore the opinions and attitudes of the chair, panel and working group on the critical elements of the consensus methods during guideline development.
Methods
We used a cross-sectional design to conduct this study and sent a structured questionnaire to stakeholders, including the chair, panel members, and working group participants, through the popular mobile phone application WeChat.We selected participants using a combination of purposive and snowball sampling. The questionnaire gathered information on demographics, experiences, opinions, and concerns regarding consensus methods and guideline development.
Results
The sample comprised 290 participants representing 31 provinces or municipalities. Among them, the most significant number of respondents (n = 107, 36.9%) were from Beijing. Most participants, specifically 211 (72.76%), held senior professional titles, while 186 (64.14%) adhered to ongoing guidelines. The Delphi method was the most commonly used consensus method (n = 132, 42.31%), but the respondents had only a preliminary understanding of it (n = 147, 47.12%). The consensus process also revealed the insufficiency of involving pharmacoeconomists, patients, and nurses.
Conclusions
Consensus methods have to be standardised and used consistently in the guideline development process. The findings of this study offer insights into diverse roles and more effective ways to apply the consensus process during guideline development.
The Suggested Upper Merged Ontology (SUMO) is an upper level
ontology that has been proposed as a starter document for The
Standard Upper Ontology Working Group, an IEEE-sanctioned working
group of collaborators from the fields of engineering, philosophy,
and information science. The SUMO provides definitions for
general-purpose terms and acts as a foundation for more specific
domain ontologies. In this paper we outline the strategy used to
create the current version of the SUMO, discuss some of the
challenges that we faced in constructing the ontology, and describe
in detail its most general concepts and the relations between them.
Abstract: We discuss the development of an environment for formal knowledge engineering. The Sigma system is an advance over previously developed systems in that it integrates a number of modern ontology development tools, which has motivated a number of research issues. Primary components include an ontology browsing and editing environment, a first order logic inference system and a natural language to logic translator. Although largely independent of any particular ontology, it supports a number of publicly available formal ontologies.
There are a number of differences in procedures when a large scale Delphi is undertaken. Some of the most important differences are discussed in the light of Delphi embracing a panel of 385 members responding to a study sponsored by the National Center for Higher Education Management Systems (NCHEMS) at the Western Interstate Commission for Higher Education (WICHE) in 1972. Such topics as panel fatigue, panel consister cy, the consensus phenomena, and early plurality are considered in the report.
The preparation of research and technology development (RTD) programmes calls for the analysis of prospective technological advances and the consideration of ''lessons learned'' from earlier programs. In this paper, we report experiences from a pilot project in which an internet-based survey and five participatory workshops were organised in conjunction with a Finnish RTD program in telecommunications. At these workshops, leading representatives from industry and the research community were invited to report their experiences from a concluding RTD program and to address future research needs, whereby they were assisted by a group support system (GSS). The workshop process was well received, which suggests that the preparation of RTD efforts can benefit from the deployment of GSS tools in support of face-to-face discussions. To our knowledge, this is the first account on GSS use in the foresight context.
Ontologies provide advantages of knowledge reusability, sharing, and greater robustness when used to build large knowledge-based applications. Unfortunately, translating between English statements and a specific ontology requires skill in knowledge engineering and an understanding of formal logic and the ontology itself. A knowledge engineer must be familiar with the concepts in the ontology, the fine distinctions between terms, and the specific way the ontology conceptualizes the world. We have developed a tool, CELT (controlled English to logic translation), to enable nonprogrammers to add knowledge expressed in terms of an ontology. CELT is an automatic translation tool to convert controlled English to KIF formulas using ontologies built with the suggested upper merged ontology (SUMO). Wordnet provides a base lexicon and a default preference for word senses. We do not expect CELT to obviate the need for knowledge engineers but to instead better leverage their time, as current machine translation tools assist professional human translators.
bComputer Based Delphi ProcessesQ an invited chapter Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health
- Murray Turoff
- Starr Roxanne Hiltz
Murray Turoff, Starr Roxanne Hiltz, bComputer Based Delphi ProcessesQ an invited chapter, in: Adler Michael, Ziglio Erio (Eds.), Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, Kingsley publishers, London, 1996.
The Delphi Method, Futures Research Methodology V2, CD ROM, The Millennium Project
- T Gordon
T. Gordon, The Delphi Method, Futures Research Methodology V2, CD ROM, The Millennium Project, American Council for the United Nations University, 2003.
Issues in large scale Delphi studies
- Huckfeldt