Conference PaperPDF Available

Ranking-type delphi studies in IS research: step-by-step guide and analytical extension


Abstract and Figures

Ranking-type Delphi is a frequently used method in IS research. However, besides several studies investigating a rigorous application of ranking-type Delphi as a research method, a comprehensive and precise step-by-step guide on how to conduct a rigorous ranking-type Delphi study in IS research is currently missing. In addition, a common critic of Delphi studies in general is that it is unclear if there is indeed authentic consensus of the panelists, or if panelists only agree because of other reasons (e.g. acquiescence bias or tiredness to disagreement after several rounds). This also applies to ranking-type Delphi studies. Therefore, this study aims to (1) Provide a rigorous step-by-step guide to conduct ranking-type Delphi studies through synthesizing results of existing research and (2) Offer an analytical extension to the ranking-type Delphi method by introducing Best/Worst Scaling, which originated in Marketing and Consumer Behavior research. A guiding example is introduced to increase comprehensibility of the proposals. Future research needs to validate the step-by-step guide in an empirical setting as well as test the suitability of Best/Worst Scaling within described research contexts.
Content may be subject to copyright.
Jörn Kobus
TU Dresden, Germany
Markus Westner
OTH Regensburg, Germany
Ranking-type Delphi is a frequently used method in IS research. However, besides several studies investigating a
rigorous application of ranking-type Delphi as a research method, a comprehensive and precise step-by-step guide on
how to conduct a rigorous ranking-type Delphi study in IS research is currently missing. In addition, a common critic of
Delphi studies in general is that it is unclear if there is indeed authentic consensus of the panelists, or if panelists only
agree because of other reasons (e.g. acquiescence bias or tiredness to disagreement after several rounds). This also applies
to ranking-type Delphi studies. Therefore, this study aims to (1) Provide a rigorous step-by-step guide to conduct
ranking-type Delphi studies through synthesizing results of existing research and (2) Offer an analytical extension to the
ranking-type Delphi method by introducing Best/Worst Scaling, which originated in Marketing and Consumer Behavior
research. A guiding example is introduced to increase comprehensibility of the proposals. Future research needs to
validate the step-by-step guide in an empirical setting as well as test the suitability of Best/Worst Scaling within
described research contexts.
Delphi, Best/Worst Scaling, MaxDiff, Maximum-Difference-Scaling
In Information System (IS) research the Delphi method has been used for almost three decades and enjoys
increasing popularity (Paré et al., 2013, p. 207). Research using Delphi covers a wide range of IS topics.
Recent publications include, e.g., complexity in IS programs (Piccinini et al., 2014), critical skills for
managing IT projects (Keil et al., 2013), or investigations of key issues in IS security management (Polónia
and Sá-Soares, 2013). In addition, the adaption and evolution of the Delphi method is of research interest,
e.g., in order to explore the application of Delphi as forecasting tool in IS research (Gallego and Bueno,
2014), assess rigor (Paré et al., 2013), or identify possibilities to build theory (Päivärinta et al., 2011).
While different types of Delphi studies exist (explained later in this paper), ranking-type Delphi can be
considered as the most relevant for IS research. It focusses on classical IS research topics, e.g., identification
and ranking of Critical Success Factors (CSF), identification of components of research frameworks, or
prioritization of selection criteria. In addition its use is indicated [i]n business to guide future management
action […]” (Paré et al., 2013, p. 208).
Thus, this paper contributes to IS research by: (1) Providing a rigorous step-by-step guide for conducting
ranking-type Delphi studies through synthesizing results of existing research; and (2) Offering an analytical
extension in order to increase the likelihood for authentic panelist consensus through introducing Best/Worst
Scaling. Best/Worst Scaling is a method originating from Marketing and Consumer Behavior research but is
relatively new to IS research.
Regarding (1), the paper synthesizes existing research regarding rigorous application of ranking-type
Delphi (Okoli and Pawlowski, 2004; Paré et al., 2013; Schmidt, 1997). Although these papers deal with
several main criticisms of ranking-type Delphi studies, e.g., regarding (a) The selection of appropriate experts
(Okoli and Pawlowski, 2004, p. 16); (b) The inappropriate use of statistics (Schmidt, 1997, pp. 764768); or
(c) The missing report of response and retention rates (Paré et al., 2013, p. 207), none of them provides
comprehensive and precise instructions by themselves. Therefore, we consolidate the main contributions of
these papers and suggest a step-by-step guide with increased density and a higher level of completeness of
information as a result.
Regarding (2), the paper proposes an analytical extension of the ranking-type Delphi method to decrease
a general problem of (ranking-type) Delphi studies: Response style biases (Paulhus, 1991, p. 17). An
example for response style biases is the pressure to conform with group ratings (Witkin and Altschuld, 1995,
p. 188). This could happen for example if participants agree to a consensus only because they are tired of
arguing and not because they got convinced.
The paper is structured as follows. Section 2 explains and defines relevant terms. Section 3 proposes a
step-by-step guide for ranking-type Delphi studies which consolidates existing research. Section 4 extends
this guide by adding Best/Worst Scaling to it. The final chapter concludes the paper and briefly discusses the
2.1 Delphi Method
The objective of the Delphi method is to achieve the most reliable consensus in a group of experts. This is
done by questioning individual experts during several rounds. In between rounds, feedback on the other
experts’ opinions is provided. Direct confrontation of the experts is avoided. (Dalkey and Helmer, 1963,
p. 458)
Originally, Delphi was used for forecasting. However, it continuously evolved during the last decades and
is used today in a variety of research types. Paré et al. (2013, p. 208) based on Okoli and Pawlowski (2004),
Schmidt (1997), and Rauch (1979) distinguish four types of Delphi studies: (1) Classical Delphi focusing
on facts to create a consensus; (2) Decision Delphi focusing on preparation and decision for future directions;
(3) Policy Delphi focusing on ideas to define and differentiate views; and (4) Ranking-type Delphi focusing
on identification and ranking of key factors, items, or other types of issues.
In the paper at hand we will focus especially on type (4): Ranking-type Delphi as (a) It is widely used
Paré et al. (2013, p. 209) found that 93% of investigated Delphi papers from 1984-2010 used this type and
(b) it fulfills the requirements of IS research best (see above).
2.2 Best/Worst Scaling
Best/Worst Scaling (also referred to as Maximum Difference Scaling or MaxDiff) is based upon random
utility theory (Louviere et al., 2013, pp. 293300). It is defined as “[…] a choice-based measurement
approach that reconciles the need for question parsimony with the advantage of choice tasks that force
individuals to make choices (as in real life) (ibid, p. 292). Best/Worst Scaling can be seen as a way to
overcome some major shortcomings of common rating approaches (e.g. ties among items, response style
bias, and standardization difficulties (Cohen and Orme, 2004, p. 32)).
Best/Worst Scaling builds on a body of items. A set consists of a number of items from the body. A
respondent gets presented a series of sets and is asked to choose one best item and one worst item in each set.
(Lee et al. 2008)
Compared to the Paired Comparison method, which also can overcome the above-mentioned
shortcomings, Best/Worst Scaling is more efficient (Cohen and Orme, 2004, p. 34) as more statistical
relevant information are provided by respondents in each comparison round. To ensure the validity of the
Best/Worst Scaling approach, a careful design is necessary to decide which items are shown in which sets.
This includes especially (1) Frequency balance, meaning that each item appears an equal number of times
during all sets; (2) Orthogonality, meaning that each item is paired with each other item an equal number of
times during all sets; (3) Connectivity, meaning that the sets are designed in a way that it is possible to infer
the relative order of preference for all items; and (4) Positional balance, meaning that each item appears an
equal number of times on the left and right side. (Sawtooth, 2013, p. 7)
Because of the previously mentioned benefits regarding preference mapping, Best/Worst Scaling is a
frequently used method in Marketing and Consumer Behavior research (e.g. Louviere et al., 2013, Cohen,
2009, Cohen and Orme, 2004). However, it seems not to be commonly used in IS research and hence
provides an interesting and original method which can contribute to the methodological development of the
research field. Applied to the given research context, Best/Worst Scaling is proposed as a ranking mechanism
to enrich the data analysis phase of a ranking-type Delphi study in a IS research setting as described later.
2.3 Guiding example
In order to make it easier to follow our proposal, we use a consecutive example of a ranking-type Delphi
study to illustrate each phase. The guiding example takes a strategic IS perspective and investigates the
identification and ranking of Critical Success Factors (CSFs) for the implementation of a process
improvement method in an IT organization. The example was inspired by own research, however was
simplified and complemented with fictional data where appropriate.
An overview on our proposed guideline to conduct rigorous ranking-type Delphi studies can be found in
Figure 1. Phase 1 is based on Okoli and Pawlowski (2004, pp. 2023), who themselves built on Delbecq et al.
(1975). Phase 2 to 4 is based on Paré et al. (2013, p. 210) and Schmidt (1997, pp. 768771). Especially
Schmidt (1997) states that ranking-type Delphi got very popular in IS research and introduced a consistent
approach for data collection (phase 2), analysis (phase 3), and presentation (phase 4).
Figure 1. Proposed guideline for ranking-type Delphi study based on (Okoli and Pawlowski, 2004; Paré et al., 2013;
Schmidt, 1997).
Figure 2 provides a detailed overview which elements of the ranking-type Delphi guideline are described
by which authors. We decided to build around these papers as they recently provided an overview on rigor in
IS ranking-type Delphi studies (Paré et al., 2013), are highly cited
(Okoli and Pawlowski, 2004), or are the
first that introduced a structured approach to ranking-type Delphi studies (Schmidt, 1997).
As of September 2015, citations in (1) Web of Science: 366 and (2) Google Scholar: 1238.
Figure 2. Sources for elements of ranking-type Delphi guideline.
3.1 Phase 1 Choosing right experts
The choice of the right experts for Delphi studies is described as “[…] perhaps the most important yet
most neglected aspect(Okoli and Pawlowski, 2004, p. 16). Since Delphi study results depend mainly on the
answers of chosen experts, it is necessary to define a thorough process for their appropriate selection.
Adapting Okoli and Pawlowski (2004, pp. 2023), we propose a five steps approach to choose appropriate
experts as initial phase of the study: (1.1) Identify expert categories; (1.2) Identify expert names; (1.3)
Nominate additional experts; (1.4) Rank experts; and (1.5) Invite experts.
Step (1.1) aims at developing selection criteria for experts, e.g., regarding disciplines or skills,
organizations or literature (academic or practitioner authors). Step (1.2) then identifies experts meeting those
selection criteria. A personal list of experts can serve as an initial starting point. Step (1.3) sends a brief
description of the Delphi study to the already identified experts and asks them to nominate further experts in
the field. Additionally, as much biographical information as possible about all (already identified and
nominated) experts’ demographics and profiles are documented. Step (1.4) then ranks the experts in priority
for invitation to the Delphi study based on their qualification. Step (1.5) invites the experts to the study in
descending order of the ranking. The subject, required procedures, and the type and extent of experts’
commitments are explained. This step is repeated, until an appropriate number of experts agreed on their
participation. Anonymity of expert participants has to be ensured at all times.
Guiding example: (1.1) A success factor study typically aims to incorporate the perspective of experts
with considerable experience on the topic of interest. In our case CSFs for the implementation of a process
improvement method in an IT-organization emerging expert categories could be (line) managers and
consultants with relevant expertise and experience. We decided for consultants.
(1.2) Since researchers’ personal networks can be a source to recruit experts (Paré et al., 2013, p. 210), we
use our network to contact a renowned global IT-consulting company. We set the threshold for consultants to
be able to participate to having supported at least seven implementations of the process improvement method.
Together with the main contact of the consulting firm, a list of 10 potential panelists is created.
(1.3) By reference of four members from the established list, seven additional possible panelists are
identified. Therefore, in total 17 possible panelists are identified.
(1.4) We rank the panelists in descending order by the number of implementations they have supported.
(1.5) While there seems no agreement on an optimal number of panelists for a ranking-type Delphi study,
the number of participating experts "[…] should not be too large (in order to facilitate consensus)" (Paré et
al., 2013, p. 208). We therefore decide to invite the 12 highest ranked experts. If one of these experts is
unable to take part, we invite the highest ranked of the remaining list instead until we achieve the number of
12 committed participants.
3.2 Phase 2 Data collection & Phase 3 Data analysis
In the Delphi method data is repetitively collected, analyzed, and reconciled with experts. Therefore, we
describe data collection (phase 2) and data analysis (phase 3) together as they cannot be separated
Before the iterative data collection (phase 2) can start, an initial instrument pre-test (i.e., instruction and
questionnaire) is conducted to ensure that all experts understand tasks and objectives (Paré et al., 2013,
p. 210). The following data collection phase itself consists of three steps (Schmidt, 1997, pp. 768771): (2.1)
Discover issues; (2.2) Determine most important issues; and (2.3) Rank issues.
An issue hereby could be for example an item or a factor. (2.1) To discover the most important issues,
first and foremost as many issues as possible have to be identified. Clear instructions are provided to experts
and there is no restriction on the number of answers experts can give. After the initial data collection, the
researchers consolidate and group familiar answers through content analysis (Mayring, 2000, pp. 45). The
consolidated results then need to be verified by the experts again to ensure the correct understanding of the
intended meaning and appropriateness of grouping.
(2.2) To not overwhelm the experts by the amount of issues they should rank in step (2.3), a further focus
on the most important issues might be necessary (as a rule of thumb the list should comprise approximately
20 issues or less (Schmidt, 1997, p. 769)). For this, the consolidated and validated list of issues is randomly
ordered, and sent together with clear selection instructions to the experts. The researchers then delete all
issues that were not selected. In case there are still too many issues left, step (2.2) can be repeated.
In step (2.3) the experts are asked to rank the issues in descending order, from most important to least
important. As the Delphi method is an iterative approach, step (2.3) is repeated until an appropriate trade-off
between level of consensus and feasibility (defined as indulgence of respondents and researcher’s resources
and additional time requirements) is reached. Within each new ranking round, respondents can revise their
ranking decision supported by a controlled feedback based on (3.1) Mean rank; (3.2) Kendall’s W a
coefficient of concordance (Kendall and Gibbons, 1990); (3.3) Top half rank (percentage of experts who
ranked respective item in their top half); and (3.4) Relevant comments/justifications by respondents.
Stopping criteria for the Delphi data collection are either a strong consensus or a clear indication that no
more differences in answers can be expected. Kendall’s W, assuming values between 0 and 1 can serve as a
quantitative measure for this purpose. Values around .1 indicate very weak agreement; values around .5
moderate agreement, and values around .9 very strong agreement (Schmidt, 1997, p. 767).
Guiding example: Before data collection starts, the instrument is pre-tested with two consultants whose
experience (five projects) is not sufficient to be included into the final participation list.
(2.1) In order to discover as many implementation success factors as possible, it seems necessary to
provide an efficient and convenient way for experts to take part in the Delphi study. For this, we offer the
panelists to either e-mail their input, conduct a call, or use a web-survey with personalized hyperlinks. In case
of ambiguity, the respective panelist is asked to clarify its input. There is no limit to the number of success
factors an expert can mention. In total 47 success factors are mentioned. Once the initial gathering is finished,
these 47 success are qualitatively investigated (Mayring, 2000, pp. 45) to check for duplicates and grouping
possibilities (e.g. ‘Leadership needs to role model the change’ and ‘Active leadership’ could possibly be
merged to one group of success factors named ‘Leadership involvement’). Further researchers review the
results independently to ensure consistency. A description for each category is created. After this, every
expert verifies if their mentioned success factors are correctly reflected in the grouping logic.
(2.2) After consolidation and grouping, 12 CSF remain. This means that there is no need to reduce the
number of success factors for the upcoming ranking round further.
(2.3) In the next round the panelists are asked to rank the list of 12 CSF. They start with the one they
believe is most important as first ranked, down to the one which they believe is least important as twelfth
ranked. In addition, the experts can justify their decision. After a ranking round, the experts are provided with
(3.1) Mean rank; (3.2) Kendall’s W; (3.3) Top-half rank and (3.4) Relevant comments/justifications.
We conduct two rounds of ranking. After the first round, the level of consensus is perceived as moderate
(W1 = 0.51). Therefore, we provide the experts with results (3.1-3.4) of the first round and conduct another
ranking round. It results in a strong consensus (W2 = 0.73). As for round 2 already several reminders were
necessary to keep the experts motivated and impatience was expressed by some experts we decide to not do a
third round as the results are deemed a satisfactory compromise between consensus and indulgence of
3.3 Phase 4 Data presentation
In the data presentation phase the final study results are presented. Regarding the choice of experts, this
includes (4.1.1) The response rate for the initial call for participation (as indication if experts consider the
exercise as relevant/important); (4.1.2) The number of participants for each Delphi round (as indication of
flagging interest and for replicable calculations); and (4.1.3) The documentations of profiles of participating
Regarding results, sub-results, and calculations, it is necessary to provide sufficient raw data to support
the accountability of statistics. At least the (4.2.1) Final whole rank; (4.2.2) Mean ranks for each round;
(4.2.3) Evolution of ranks of an item in each round; and (4.2.4) Kendall’s W for each round should be
reported. Additionally, the total number of issues generated in the first phase of data collection (2.1), and
transparency on consensus level of the pared list at the end of the second phase (2.2) need to be reported.
Guiding example: (4.1.1) The response rate for the initial participation call was around 83% (from the
first 12 experts asked, 10 took part. The two experts who have not taken part are replaced by two experts
from the remaining list). (4.1.2) In total, two ranking-type Delphi rounds were necessary. All 12 experts have
taken part in the two Delphi rounds. (4.1.3) Table 1 illustrates how the profiles of experts could be depicted
for this example.
Table 1. Template to illustrate expert profiles of Delphi panel.
expert ID
Main country
of involvement
(# of projects)
Expert 1 Partner Germany 11
Expert 2 Partner Sweden 10
Expert 3 Senior expert UK 20+
Expert 4 Senior expert Germany 30+
Expert 5 Partner Spain 12
Expert 6 Partner Norway 10
Expert 7 Partner Czech Republic 30+
Expert 8 Partner Germany 9
Expert 9 Partner France 10
Expert 10 Partner Sweden 20+
Expert 11 Partner UK 8
Expert 12 Partner Denmark 7
(4.2.1-4.2.4) The number of CSF generated in the first phase of data collection (step 2.1) is 47. After
consolidation and grouping (step 2.2), all 47 CSF can be assigned to 12 groups. This means there was full
consensus on the list shown in Table 2. The remaining study results can be obtained from Table 3. As the
order of the mean ranks (4.2.2) did not change in between the two rounds, we omit the information on
evolution in ranks (4.2.3).
Table 2. Remaining CSF after consolidation (step 2.2).
Expert 1
Expert 2
Expert 3
Expert 4
Expert 5
Expert 6
Expert 7
Expert 8
Expert 9
Expert 10
Expert 11
Expert 12
CSF 1 x x x x x x x x x x 10 83%
CSF 4 x x x x x x x x x x 10 83%
CSF 6 x x x x x x x x 867%
CSF 2 x x x x x x x 758%
CSF 9 x x x x x x x 758%
CSF 12 x x x x x x 650%
CSF 8 x x x x x x 650%
CSF 3 x x x x x x 650%
CSF 10 x x x x x 542%
CSF 11 x x x x x 542%
CSF 5 x x x x x 542%
CSF 7 x x x x 433%
Total 6 5 9 4 8 7 7 8 5 9 5 6
Table 3. Results of ranking (step 2.3) rounds 1 and 2 of the Delphi study.
Mean rank
(round 1)
Mean rank
(round 2)
Final rank
CSF 6 4,38 2,45 1
CSF 9 5,87 3,81 2
CSF 1 6,74 4,56 3
CSF 2 6,98 5,03 4
CSF 4 7,23 5,56 5
CSF 12 7,56 6,32 6
CSF 8 8,12 6,89 7
CSF 11 8,89 7,66 8
CSF 10 9,10 9,33 9
CSF 3 9,23 10,11 10
CSF 7 10,11 10,89 11
CSF 5 10,21 11,01 12
Kendall's W 0,51 0,73
Based on these results the discussion of the findings would take place. However, as this is highly content-
and less process-related, this is out of scope for the paper at hand.
IS ranking-type Delphi studies use several ranking mechanisms. These include direct ranking of items
(Kasi et al., 2008); ranking based on ratings on a predefined scale for example on a Likert scale (Liu et al.,
2010; Nakatsu and Iacovou, 2009); or ranking based on expert allocation of points from a predefined pool
(Nevo and Chan, 2007). However, all these mechanisms do have several well-known and documented
disadvantages related to response style biases. Paulhus (1991, p. 17) enumerates the three most prominent
response style biases as (1) Social desirability bias (tendency to lie or fake); (2) Acquiescence bias (tendency
to agree); and (3) Extreme response bias (tendency to use extreme ratings). A way to overcome these biases
in ranking-type Delphi studies is the introduction of Best/Worst scaling as a ranking mechanism (Lee et al.,
2008, p. 335). Since a subjective preference order can be calculated based on numerous smaller decisions, it
gets much harder for the panelist to predict/deliberately influence the final ranking list. In order to apply
Best/Worst Scaling to the introduced step-by-step guide, an extension of it is proposed as shown in Figure 3.
gure 3. Extended guideline for ranking-type Delphi study featuring Best/Worst Scaling for Data analysis.
Extended Phase 3 Data analysis: (3.1 new) In order to use Best/Worst Scaling as ranking
mechanism in ranking-type Delphi studies, proper (a) Design, (b) Execution, and (c) Analysis need to be
(a) Regarding design, the list of all remaining issues (result of phase 2.2) serves as body of items. In
addition, it needs to be decided on the number of sets (questions) experts get asked; the number of items per
set; and the appearance of items in sets considering frequency, orthogonality, connectivity and positional
balance (compare section 2.2).
(b) Regarding execution, it should be easy for the experts to take the questionnaire. While it would be
possible to use paper and pen or regular e-mail communication for this, a web-based version of the
questionnaire seems to be the most appropriate form for taking the survey.
(c) Regarding analysis, several possible options exist to transform the results of Best/Worst Scaling on
individual level to a ranking. The simplest option is to calculate ‘best minus worst’ (# of times when issue
was selected best - # of times when issue was selected worst). However, more sophisticated options include
for example the application of linear probability models, conditional logit models, or rank-ordered logit
models (Louviere et al., 2013, pp. 295296).
(3.2-3.5) The steps of Mean rank, ‘Top-half’ rank and Kendall’s W and comments by participants are
identically to the previously introduced step-by-step guide.
Guiding Example: (3.1 new) While it is possible to manually design and conduct Best/Worst Scaling,
we follow Louviere et al. (2013, p. 295) and use more sophisticated statistical software in order to prepare for
non-trivial analyses. We decided to use a web-based solution from Sawtooth Software (Sawtooth, 2015) to
design and analyze our proposed ranking. We did so, as provided technical papers (Sawtooth, 2013) offered
transparent and sufficient information on functionality and proficiency of the software.
(a) Design: As the body of items consists of 12 CSF, we decide to let the experts rank 12 sets (Sawtooth,
2013). In each set, four items are shown of which the experts select the best and the worst CSF. The
appropriate appearance (frequency, orthogonality, connectivity and positional balance) of CSFs in the sets
was ensured by the software. Figure 4 provides an example for a set.
Figure 4. Example of set in web-survey.
(b) Execution: We decided for a web-survey as this provided the participants freedom to take the survey
whenever and wherever they wanted. In addition, it turned out that a smartphone compatible version is
(c) Analysis: Using statistical software we calculated the results for each individual expert based on
conditional logit models, which can be used to investigate choice behavior (McFadden, 1974, pp. 105106).
Table 4 provides a sample result for an expert. The row ‘Value’ provides the zero-centered interval scores
derived by the conditional logit model. A higher value indicates more importance of the CSF. In our example
Table 4 is calculated for each expert and used to obtain respective ranking of CSF.
Table 4. Result for expert ranking derived using Best/Worst Scaling.
Rank 1 2 3 4 5 6 7 8 9 10 11 12
CSF 3 1 8 11 10 9 6 4 2 5 7 12
Value 49,0 23,4 23,4 3,0 0,1 - 2,1 - 2,7 - 3,6 - 4,0 - 5,2 -30,3 -51,0
(3.2-3.5) After having obtained the ranking of each individual expert with the help of a conditional logit
model, we can obtain mean rank, ‘Top-half’ rank and Kendall’s W and comments by participants as
described in above step-by-step guide.
The study at hand contributes to IS research twofold.
First, it introduces a rigorous step-by-step approach to conduct ranking-type Delphi studies, which are
used widely in IS research. In addition, a guiding example is introduced which provides possible
visualizations and helps IS scholars to think and act end-product oriented, while conducting a ranking-type
Delphi study.
Second, it introduces an analytical extension for ranking-type Delphi studies which can further increase
rigor. The level of consensus should neither be influenced by response style bias, nor by the researchers’
persistency (i.e., by asking experts again and again until some of them give in to achieve consensus) but only
by the real opinions of the experts. Through the introduction of Best/Worst Scaling as ranking mechanism the
problem of illusionary consensus of Delphi panelists can be reduced. In Best/Worst Scaling the experts are
forced to choose a CSF deliberately as most/least important compared to other CSFs. In this way a subjective
preference order can be calculated and it gets much harder for the panelist to predict/deliberately influence its
final ranking list. In addition, the guiding example introduces an easy-to-conduct, time efficient and cognitive
low demanding way for panelists to conduct a ranking based on Best/Worst Scaling.
While the study was inspired by a real world research example, its main limitation is that it has not yet
been applied in full to a real world research question. While we are confident that the described process
would yield expected results, future research still needs to prove that.
In addition, future research needs to investigate in detail if Best/Worst Scaling could even replace more of
phase 3 (Data analysis) as the only ranking mechanism. While Kendall’s W is a well-established and used
measure of consensus in ranking-type Delphi studies, future research should investigate what other measures
would be suitable with the additional data on preferences gained by described ranking mechanism.
Cohen, E. (2009) ‘Applying bestworst scaling to wine marketing’, International Journal of Wine Business Research,
vol. 21, no. 1, pp. 823 [Online]. DOI: 10.1108/17511060910948008.
Cohen, S. and Orme, B. (2004) ‘What's your preference? Asking survey respondents about their preferences creates new
scaling decisions’, Marketing Research, vol. 16, pp. 3237.
Dalkey, N. and Helmer, O. (1963) ‘An experimental application of the Delphi method to the use of experts’, Management
Science, vol. 9, no. 3, pp. 458467.
Delbecq, A. L., Van de Ven, A. H. and Gustafson, D. H. (1975) Group techniques for program planning: A guide to
nominal group and Delphi processes, Scott, Foresman Glenview, IL.
Gallego, D. and Bueno, S. (2014) ‘Exploring the application of the Delphi method as a forecasting tool in Information
Systems and Technologies research’, Technology Analysis & Strategic Management, vol. 26, no. 9, pp. 987999.
Kasi, V., Keil, M., Mathiassen, L. and Pedersen, K. (2008) ‘The post mortem paradox: a Delphi study of IT specialist
perceptions’, European Journal of Information Systems, vol. 17, no. 1, pp. 6278 [Online]. Available at http://
Keil, M., Lee, H. K. and Deng, T. (2013) ‘Understanding the most critical skills for managing IT projects: A Delphi
study of it project managers’, Information & Management, vol. 50, no. 7, pp. 398414.
Kendall, M. G. and Gibbons, J. D. (1990) Rank correlation methods, 5th edn, London, New York, NY, E. Arnold;
Oxford University Press.
Lee, J. A., Soutar, G. and Louviere, J. (2008) ‘The best-worst scaling spproach: An alternative to Schwartz's values
survey’, Journal of Personality Assessment, vol. 90, no. 4, pp. 335347.
Liu, S., Zhang, J., Keil, M. and Chen, T. (2010) ‘Comparing senior executive and project manager perceptions of IT
project risk: a Chinese Delphi study’, Information Systems Journal, vol. 20, no. 4, pp. 319355 [Online].
DOI: 10.1111/j.1365-2575.2009.00333.x.
Louviere, J., Lings, I., Islam, T., Gudergan, S. and Flynn, T. (2013) ‘An introduction to the application of (case 1) best–
worst scaling in marketing research’, International Journal of Research in Marketing, vol. 30, no. 3, pp. 292303.
Mayring, P. (2000) ‘Qualitative Content Analysis’, Forum: Qualitative Social Research, vol. 1, no. 2, pp. 110.
McFadden, D. (1974) ‘Conditional logit analysis of qualitative choice behavior’, in Zarembka, P. (ed) Frontiers in
econometrics, New York, NY: Academic Press, pp. 105142.
Nakatsu, R. T. and Iacovou, C. L. (2009) ‘A comparative study of important risk factors involved in offshore and
domestic outsourcing of software development projects: A two-panel Delphi study’, Information & Management,
vol. 46, no. 1, pp. 5768 [Online]. DOI: 10.1016/
Nevo, D. and Chan, Y. E. (2007) ‘A Delphi study of knowledge management systems: Scope and requirements’,
Information & Management, vol. 44, no. 6, pp. 583597 [Online]. DOI: 10.1016/
Okoli, C. and Pawlowski, S. D. (2004) ‘The Delphi method as a research tool: An example, design considerations and
applications’, Information & Management, vol. 42, no. 1, pp. 1529 [Online]. DOI: 10.1016/
Päivärinta, T., Pekkola, S. and Moe, C. E. (2011) ‘Complexity in IS programs: A Delphi study. ICIS 2011 proceedings.’.
Paré, G., Cameron, A.-F., Poba-Nzaou, P. and Templier, M. (2013) ‘A systematic assessment of rigor in information
systems ranking-type delphi studies’, Information & Management, vol. 50, no. 5, pp. 207217.
Paulhus, D. L. (1991) ‘Measurement and control of response bias’, in Robinson, J. P., Shaver, P. R. and Wrightsman, L.
S. (eds) Measures of personality and social psychological attitudes, New York: Academic, pp. 1759.
Piccinini, E., Gregory, R. and Muntermann, J. (2014) ‘Complexity in IS programs: A Delphi study. ECIS 2014
Polónia, F. and Sá-Soares, F. de (2013) ‘Key issues in Information Systems security management. ICIS 2013
Rauch, W. (1979) ‘The decision Delphi’, Technological Forecasting and Social Change, vol. 15, no. 3, pp. 159169
[Online]. DOI: 10.1016/0040-1625(79)90011-8.
Sawtooth (2013) The maxdiff system technical paper. Access date: 25.08.2015 [Online]. Available at https:// (Accessed 15 September 2015).
Sawtooth (2015) SSI web: maxdiff. Access date: 15.08.2015. [Online]. Available at
products/maxdiff-software/maxdiff (Accessed 15 September 2015).
Schmidt, R. C. (1997) ‘Managing delphi surveys using nonparametric statistical techniques’, Decision Sciences, vol. 28,
no. 3, pp. 763774 [Online]. DOI: 10.1111/j.1540-5915.1997.tb01330.x.
Witkin, B. R. and Altschuld, J. W. (1995) Planning and conducting needs assessments: A practical guide, Thousand
Oaks, CA, Sage Publications.
... The Delphi methodology used in the present study is adapted from the guidelines of Kobus and Westner (2016) for rankingtype Delphi studies. After identifying expert categories and selecting experts, three rounds of integrated data collection and analyses were performed. ...
... The aim of Round 1 was to discover the relevant issues through a form of brainstorming (Schmidt et al., 2001;Kobus and Westner, 2016). Therefore, experts were prompted with openended questions to elicit their opinions on the topic (Murry, 1995). ...
... The aim of Round 2 was to narrow down the most relevant items from Round 1 (Schmidt et al., 2001;Kobus and Westner, 2016). In order to determine which items the expert group agreed to be most relevant, a rating strategy was used in another Google Forms survey (Villiers et al., 2005;Hsu and Sandford, 2007). ...
Full-text available
Spotting is a typical isolated head coordination used by many dancers during rotation. However, with sporadic and inconclusive explanations as to why dancers spot, the critical characteristics and functionalities of spotting have yet to be identified. Therefore, a Delphi method survey was used as a novel methodology for providing greater insights into this under-examined motor behavior, bringing together experts from various disciplines to generate ideas and identify the crucial elements of spotting. Following the selection of experts, three rounds of data collection and analysis were conducted to narrow down relevant topics and evaluate consensus. To gather opinions in Round 1, experts were asked to respond freely to three prompts regarding the reasoning, characteristics, and uses of successfully spotting; responses were then grouped into predominant items. To rate agreement in Round 2, experts rated their agreement on the relevance of the grouped items from Round 1 on a 5-point Likert scale; items rated 4 or 5 by at least 70% of the experts were taken as those consensually relevant to the group. To rank importance in Round 3, Best-Worst Scaling was used to determine individual rankings of the relevant items from Round 2. In a series of comparisons, experts were prompted to select the most and least important items in multiple sub-groupings. Group mean ranking of items as well as ranking concordance and differentiation were analyzed to determine the most important items and the strength of consensus, respectively. Overall, consensus and differentiation in experts’ item rankings were low; however, novel insights were presented. As characteristics of successfully spotting, experts emphasized head isolation, timing, and gaze specificity alongside functional characteristics, substantiating spotting as purposeful action in rotation. Building on traditional notions of spotting for reduced dizziness and maintaining balance, successfully spotting was further deemed useful for multiple turns, orientation, and rhythm. The findings of this study thus provide informed guidelines for future analysis and training of this complex head coordination in rotations.
... The first step of the Delphi survey is to identify a panel of experts from diverse areas and closely related to the research question. The choice of the right experts for Delphi studies is described as the most important, yet most neglected step (Kobus and Westner, 2016). The expert should be willing to commit to participation in all rounds of the survey (Flostrand, Pitt and Bridson, 2020). ...
... A systematic way in which Delphi results can be presented is explained by Kobus et al. (Kobus and Westner, 2016). The response of the Delphi was analysed. ...
Locating a dry port depends on various criteria such as distance, modes of transport, cost associated, environmental, geographical, and social concerns. The paper's primary purpose is to identify the location-specific attributes impacting dry port locations, particularly in peninsular India, where seaports are very close to each other. The paper's objective has been achieved through a four-cycle Delphi survey and criticality through linear ranking and consistency through Kendall's 'W'. Initially, the criteria are identified through a systematic literature survey. They are then sieved within a focus group consisting of five experts with experience of more than twenty years in port operations. Final vetting of the criterion is done through a Delphi survey; the experts with a mutual interest in the subject but from different backgrounds are included. The final vetted list is determined. In the last two rounds of the survey, the rankings were determined, a consensus was reached, and the final rank was obtained. The results indicate that proximity and economic criteria are the most crucial in the chosen geography, which contradicts the developed regions, where the environmental criterion dominates. However, the environmental criteria have been ranked third. Even though the Delphi method is an age-old method used in many literatures in different contexts, it is not used in a dry port problem in the peninsular region. Consensus building is significant in strategic decisions, like dry port location selection. Since this study involves multiple stakeholders from diverse backgrounds and a subjective opinion was required, the Delphi method and linear ranking have been adopted.
... In the subsequent years, its focus extended to various other purposes [28]. The authors of [29] and many others distinguish four types of Delphi studies: (1) Classical Delphi studies with a focus on 'facts to create a consensus'; (2) Decision Delphi studies with a focus on 'preparation and decision for future directions'; (3) Policy Delphi studies with a focus on 'ideas to define and differentiate views'; and (4) Ranking-type studies with a focus on the 'identification and ranking of key factors, items, or other types of issues' [30]. Currently, the ranking-type Delphi is a frequently used research method [2]. ...
... On this basis, this methodology aims to synthesize the collective expertise of the participants (see [4] for further information). Regarding the implementation using a questionnaire, a web-based version of the questionnaire seems to be the most appropriate form for taking the survey [30]. For the feedback rounds, mean or median estimates of the panel should be provided [31]. ...
Full-text available
The circular economy (CE) is an essential societal topic of the 21st century. Although various scientific disciplines address it, many research gaps exist. The Delphi is a proven instrument for managerial decisions, which also gained importance in sustainability-oriented innovation research. Likewise, innovation processes using the input of crowds are emerging phenomena. Nevertheless, the Web of Science publication record includes a few articles applying crowdsourcing or the Delphi method to support CE-oriented management decisions only, and their further application has limitations. Addressing these gaps, this article presents an advanced concept integrating both methods to support the development of CE products and software responding to the worldwide need for more sustainable automotive products and CE solutions for tyres specifically. A combination of two-stage crowdsourcing and Delphi approaches was used, involving 509 participants from the EU member state Germany in total. This article provides, in particular, five contributions: First, it identified specific benefits of combining crowdsourcing and the Delphi method. Second, the attractiveness of a CE software system with product configuration tools is shown. Third, the interest in a quality label for sustainable tyres is unveiled. As the fourth contribution, the analyses show the importance of the CE software’s and labels’ consideration of social aspects in the tyre value chain and certain substances influencing appropriate recycling. Fifth, it represents consumers’ suggestions for products made of tyre recyclates and their interest in additional CE automotive products. The article finishes with recommendations for developing a tyre CE and applying the method combination in research and management.
... The categorization of the characteristics was a necessary step before the evaluation of their importance by a group of experts through a ranking-type Delphi survey (Kobus & Westner, 2016). The preparation of three initial lists (functional, technical and social characteristics) prevented experts from getting overwhelmed by the amount of issues they should rank during the Delphi method application. ...
... Apart from IS research where ranking-type Delphi is frequently used and covers a wide range of topics (Hanelt, 2020 (Lund, 2020) it was also identified that "although LIS Delphi studies are relatively few in number, they have a consistent record of being published in some of the most prestigious LIS journals." The methodology phases that were followed in the present survey are illustrated in Figure 1 and are slightly simplified in comparison to the ones presented by Kobus and Westner (2016), in order to be more compatible with the time constraints and effort required on behalf of the participants. ...
The purpose of this research is to identify the innovative features of the LSPs that differentiate them from the LMS, as well as to evaluate their importance, based on the opinions of the Greek information scientists. The method used is the Delphi 2-round questionnaire. The results show that the experts most value an all-in-one system that incorporates all modules within it, instead of different collaborating software. Interoperability between systems, the adoption of new metadata standards, the SaaS architecture and the multi-tenant model are also held in high regard. In contrast, the use of the mobile library applications feature has achieved low ranking.
... The principle purpose of the workshop was to develop the top ten strategic priorities that the MHPSS WG members believed should be their focus for the next twenty four months as part of the overall response to meeting the MHPSS needs of Rohingya registered and non-registered refugee camp populations. The strategic prioritisation process conducted in Cox's Bazar was based upon similar services conducted by the United Nations High Commissioner for Refugees (UNHCR) and the International Federation of Red Cross Red Crescent Societies Reference Centre for Psychosocial Support (IFRC PS Centre) in Turkey in 2016and Iraq in 2016(UNHCR internal communication and Harrison, 2017. The methodology on how to conduct a situational analysis, using the case example of the Cox's Bazar response, is the subject of another article in this special edition (see Harrison et al., 2019, p. X-X). ...
... Ten points represented the most prioritised area for an agency, with one point the least prioritised area. The ten topics that received the highest number of points were then prioritised (according to the overall number of points received) from highest to lowest, using an amended version of the Delphi (Kobus & Westner, 2016) ranking approach. The points ranged from 134 for the most prioritised area to less than 20 for the least prioritised areas. ...
In early January 2019, a participatory workshop took place in Cox’s Bazar, Bangladesh, with the mental health and psychosocial support working group (MHPSS WG) members. The principal purpose of the workshop was to develop the top ten strategic priorities that the MHPSS WG members wished to focus on over the next two years to improve the mental health and psychosocial wellbeing of the Rohingya registered and non-registered refugee camp populations. The strategic prioritisation areas also served as key advocacy messages that can be used in discussions with policy makers, government authorities and humanitarian donor agencies regarding the mental health and psychosocial needs of Rohingya refugee males, females, boys and girls residing in Cox’s Bazar, Bangladesh. The authors believe the strategic prioritisation process contributes to the functioning and purpose of a country-level MHPSS WG and therefore advocates for the approach in other humanitarian and refugee contexts.
... Consequently, neither the complete list of the CSFs as presented in this paper nor was consistent wording used during the prioritization within the interviews. Thus, we suggest that future research reevaluates the identified CSFs, e.g., with a survey study, further case studies (Esteves, 2004), or a Delphi study (Kobus & Westner, 2016). Fourth, management implications deduced from CSFs provide only suggestions to be interpreted within the contextual situation, rather than general application guidelines. ...
This paper identifies critical success factors of Business-managed IT based on case study results. Four groups of critical success factors emerge: (1) general approach to Business-managed IT/Business-managed IT strategy, (2) Business-managed IT project prerequisites/Business-managed IT team, (3) Business-managed IT project execution and outcome, and (4) information technology management for Business-managed IT. The results suggest that bilateral responsibility between the business unit and the IT organization is the most favorable governance option for Business-managed IT.
... Another interesting finding is that there is quite similarity between the statements of ranking between teachers and health [18]. ...
Full-text available
Background: School students play an important role in the prevention of TB at all levels of community. However, there is no consensus on essential preventive messages to be taught to the students. This study aims to develop consensus on essential information about preventive measures of TB. Method: It is qualitative research using three-round Delphi techniques among TB experts to develop agreement on essential preventive measures to prevent TB and verify their level of agreement. Result: The essential knowledge to prevent TB is information about causative agent, DOTS process and its importance, sign and symptoms, national strategic plans, and psychosocial counseling. Nearly 91% of the experts agree that information about causative agent of TB is most essential to be informed to the students. Conclusion: Delphi method is an effective tool to develop consensus. The results can be used for developing the IEC materials focusing school students and the communities. Introduction Tuberculosis (TB) remains a major public health problem. The incidence rate of TB is 152 per 100,000 populations in Nepal and it has not declined as expected [1]. It is fundamental to prevent new infections to shrink the burden of TB. Likewise, essential and adequate knowledge of TB, its services including stigma and discrimination helps to break the barriers for TB diagnosis and treatment services [2-4]. Community people including school students need to be aware of preventive measures at all levels of communities which could be essence to lower the burden of the disease [5]. Objectives The general objective of the study is to develop the consensus on basic and essential information on preventive measures of TB for high school students. Methods It is qualitative research that uses the Delphi technique to induce the opinions of experts on knowledge to prevent TB and develop the consensus with the most essential knowledge including level of agreement. The Delphi technique consist of three-round survey among the selected TB experts and agree in the essential knowledge to prevent TB (Figure 1). A heterogeneous group of twenty-two experts who have knowledge and experience in the field of a TB control program were selected for the surveys. Critical case sampling method is used because the characteristics of the case make them critical [6]. They are more likely to yield the most important information and have the greatest impact on the development of consensus on essential knowledge to prevent TB among high school students. The expert panel was selected for education and TB sector. The district TB Leprosy officer (DTLOs) of Kathmandu and Bhaktapur district and health workers of the Urban health clinic, school health teachers, and female community health volunteers (FCHVs) of the Kathmandu metropolitan city (KMC) ward 32, 10, 7 were selected purposively. Additionally, schoolteachers 2 from the public schools and 2 from the private schools located and ward 7 and 32 were selected purposively. The experts were selected based on the following characteristics that make the participants a critical case is (Table 1): Table 1: Characteristics of the selected panel. Experts Characteristics No. of Experts District TB/Leprosy Officer They have a clear understanding of national guidelines, plans, and policies related to tuberculosis. They also have the problems and issues in the prevention of tuberculosis at the national and district level 2 Health Workers Health workers from the DOTS clinic with TB patients every day. They are familiar and experienced with the problems and issues of TB patients, including, their knowledge on tuberculosis 10 TB Volunteers TB volunteers are critical because they are familiar and experienced with the health-seeking behavior, norms, and values of the community. They are also aware of the knowledge gap on tuberculosis in the community. 6 School Teachers School teachers know the educational level of the students as well as the depth of the knowledge they share about the prevention of tuberculosis in the school. 4 Total 22 Delphi Rounds 4.2.1. First Round: After the selection of experts, each participant was given a special code number. Then, the participants were explained about the research objectives, procedures, 3 rounds of Delphi methods, including their roles and time required. Thereafter, the research assistant distributed an open-ended questionnaire to enumerate seven major preventive measures. After the distribution of the questionnaire sheet, at least 15 days were provided to the experts to write the opinions. Then, the research assistant collected the filled questionnaire. The first round of data was analyzed using the Jiro Kawakita (KJ) method. To conduct KJ method analysis, the researcher prepares black pens, many pieces of card-sized paper, chart paper [7]. Each statement was written in pieces of paper. Then, stick in chart paper based on resemblance of meaning of the sentences. The investigator read the statements several times to identify similar and/or close opinions. Each label contains a sentence, which the researcher gets from the data. The KJ method was applied to consolidate similar or connected information to get summarized and categorized groups of items [8]. After that all the investigators reviewed the data, discussed within the team and prepared summarized 46 statements from 154 items. The summary of each group was written down on the business card-sized paper and placed on the top of each group. Lines and comments were written to explain the interrelationship among groups. After that sub-theme was prepared for the statements, based on the summaries of the sub-theme, major theme of the sentences was prepared. Second Round: In the second round, all the 46 items were sent to all the experts to rank from 1 to 7 based on the importance of each item for the prevention of TB. The first ranked items were given 7 score, 2 ranked was given 6 score. Similarly, the last ranked was given 1 score while for the non-ranked item, no score was given. The total points of each item were calculated. The items will be arranged in the descending order based on the importance. Third Round: In the third round, the new ranked items based on the total score were redistributed to the experts and asked them, do they agree or disagree with the ranking. If the experts do not agree with the ranking, they were requested to provide new ranking. The numerical value will be given from 1 to 7 based on the importance of each item. The data was entered and analyzed in SPSS version 20.0. And then, the mean was calculated to indicate the group opinion while the standard deviation will calculate to indicate the level of agreement. Each statement was analyzed quantitatively by the percentage of agreement ratings, importance rankings and the number of comments made for each of the statements, and qualitatively analyzed using the thematic analysis (Figure 2). Figure 2: Three round of Delphi techniques. The Delphi technique is based on gaining consensus among the selected experts. Though there is variation in opinions and interpretation among the experts and it is quite difficult to gain 100% agreement on all the statements. In fact, no standard methods or guidelines exist for determining appropriate consensus level [9]. The consensus level is 70% of participants scored the item and no consensus: item failed to meet either of the above criteria [10-12]. Ethical Consideration Within the Delphi, participants were allowed to participate voluntarily, and both written and informed consent was taken before the inclusion in the study. The quasi-anonymity was maintained as the participants know each other but their opinions and judgments were strictly anonymous. Along with that the confidentiality of the information, opinions, and judgments was strictly maintained. Furthermore, the ethical approval was taken from Nepal Health Research Council (NHRC), Kathmandu, Nepal (Reg.No. 530/2018).
... There is no clear agreement among methodologists regarding the optimal size of a Delphi expert panel. 15 A total of 20 flight instructors completed the ranking questionnaire. The literature suggests that in a homogeneous group, 10 to 18 experts might be enough. ...
INTRODUCTION: A flight is composed of many flight performance aspects. However, not all of these aspects are equally important for the success and safety of a flight. When investigating the influence of a stressor on flight performance, it is important to understand not only which flight performance aspects are important for the success and the safety of the flight, but also which of these aspects will most likely be affected by reduced alertness. METHOD: A total of 136 helicopter pilots of the Royal Netherlands Air Force (RNLAF) of all qualification levels were invited to participate in a three-round ranking Delphi study. RESULTS: A total of 41 (30%) helicopter pilots completed the first questionnaire round and 20 (77%) flight instructors completed the ranking round. The top ten skills elements comprised seven nontechnical skills (NTS), namely, awareness of the environment, decision making, workload management, stress management, planning and coordinating, general knowledge, and basic fitness; and three technical skills (TS), that is, advanced aircraft handling, flight maneuvers and procedures, and abnormal and emergency procedures. The top three ranked skill elements (awareness of environment, decision making, and workload management) were considered by the flight instructors to be highly influenced by reduced pilot alertness. CONCLUSION: NTS are considered more important and more affected by reduced pilot alertness during operational helicopter flight compared to TS.
Making efficient and effective decisions in the chaotic environment of humanitarian relief distribution (HRD) is challenging. Decision-makers need to concentrate on numerous decision factors categorized into decision objectives, variables, and constraints. Recent HRD literature focuses on optimizing procedures while neglecting the quantification of essential requirements (decision factors) for information systems to provide decision-making support. In this article, we address this gap by accumulating affecting decision factors from both literature and practice. We investigated the practical implications of these factors in HRD decision-making by measuring the preferences of a Delphi panel consisting of 23 humanitarian experts. The results from our study emphasize the importance of the decision factors in the proposed process model for HRD in a large-scale sudden onset. Our work provides researchers not only with a comprehensive set of practically feasible decision factors in HRD but also with an understanding of their influences and correlations.
The unemployment rate of graduate students in South Africa (SA) has increased more than 4% in the past 10 years. This increase can be ascribed to the current economic situation as well as the unemployability of these students. The focus was to determine the graduate attributes (skills and competencies that make students employable) required in entry-level recreation professionals in SA. A ranking-type Delphi study design was used, consisting of three iterations. Ten experts from the public, non-profit and private recreation sectors whose organizations employ entry-level recreation professionals were asked to rank the graduate attributes most required in students, and evaluate the relevance of graduate attributes for a SA context. Data were analzsed using descriptive statistics and inductive coding. “Passion for the profession”, “trainability and a willingness to learn” and “communication skills” were the top-ranked attributes. Clear differences were found in the expectations from SA graduates compared with United States graduates. “Creativity”, “ability to work with groups” and “conflict management” were added for the SA context. These results clarified what is expected of graduates entering recreation careers in SA, and it is therefore crucial that institutions preparing recreation students focus on these attributes in their programmes to enhance graduate employability.
Conference Paper
Full-text available
Information systems (IS) programs, i.e., temporary organizations consisting of a coherent group of interrelated projects that are jointly coordinated for achieving a common set of goals and objectives, are growing in importance for implementing strategic IT-driven organizational initiatives. Yet we hardly understand the specific challenges associated with them. In this paper, we draw connections between the nature of an IS program and the concept of complexity, arguing that a deeper understanding about the nature of complexity in IS programs is needed. Due to the lack of research in this area, we opted for an exploratory Delphi study. Our results point to three key themes: (1) the importance of organizational complexity in IS programs, (2) the role of organizational dynamic complexity, and (3) the challenges of coordinating and controlling IS programs. These findings offer a practically grounded foundation for exploring the specific challenges of managing IS programs.
The increasing dependence of organizations on information and the need to protect it from numerous threats justify the organizational activity of information systems security management. Managers responsible for safeguarding information systems assets are confronted with several challenges. From the practitioners' point of view, those challenges may be understood as the fundamental key issues they must deal with in the course of their professional activities. This research aims to identify and prioritize the key issues that information systems security managers face, or believe they will face, in the near future. The Delphi method combined with Q-sort technique was employed using an initial survey obtained from literature review followed by semi-structured interviews with respondents. A moderate consensus was found after three rounds with a high stability of results between rounds. A ranked list of 26 key issues is presented and discussed. Suggestions for future work are made. © (2013) by the AIS/ICIS Administrative Office. All rights reserved.
This paper gives an account of an experiment in the use of the so-called DELPHI method, which was devised in order to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of questionnaires in depth interspersed with controlled opinion feedback.
The Delphi method has been used widely in clarifying decision-making based on the opinions of experts in all research areas. This study focuses on Information Systems and Information Technology (IS/IT) research area. Its aim is to explore the use of the Delphi method in topics related to IS/IT as a forecasting tool. A sizeable amount of research has been examined about IS/IT using the Delphi method. This work proposes a guideline for applying the Delphi method in IS/IT studies based on a literature review of high impact IS/IT papers that apply Delphi with a view to answering forecasting questions. This guideline defines the most relevant aspects which should be considered when applying this method. Previously, the authors conducted a literature review of 2502 articles from different research topics to obtain evidence about the diffusion of the Delphi method in IS/IT compared to other areas.
The skill requirements for project managers in information technology (IT) projects have not been widely studied in the past, especially in terms of their relative importance. We addressed this gap in the literature by conducting a Delphi study with 19 IT project managers (PMs). Among the list of 48 skills identified, our panelists selected 19 skills as being the most critical for IT PMs and then ranked them based on their relative importance. Follow-up interviews were conducted with selected panelists to gain insights into the importance of the top-ranked IT PM skills. We compare our results with two previous studies of IT PM skills and discuss the implications for research and practice.
The ranking-type Delphi method is well suited as a means for consensus-building by using a series of questionnaires to collect data from a panel of geographically dispersed participants. This method allows a group of experts to systematically approach a particular task or problem. While information systems researchers have been using this method for almost three decades, no research to date has attempted to assess the extent to which Delphi studies have been rigorously conducted. Using the guidelines that have been prescribed by the leading Delphi methodologists, our descriptive review reveals many positive signs of rigor such as ensuring the anonymity of experts and providing clear and precise instructions to participants. Nevertheless, there are still several areas for improvement, such as reporting response and retention rates, instrument pretesting, and explicitly justifying modifications to the ranking-type Delphi method.
We review and discuss recent developments in best–worst scaling (BWS) that allow researchers to measure items or objects on measurement scales with known properties. We note that BWS has some distinct advantages compared with other measurement approaches, such as category rating scales or paired comparisons. We demonstrate how to use BWS to measure subjective quantities in two different empirical examples. One of these measures preferences for weekend getaways and requires comparing relatively few objects; a second measures academics' perceptions of the quality of academic marketing journals and requires comparing a significantly large set of objects. We conclude by discussing some limitations and future research opportunities related to BWS.