9)fuzzy-delphi-method public health

Full text

228 Med J Malaysia Vol 72 No 4 August 2017
SUMMARY
Background: The most crucial step in forming a set of
survey questionnaire is deciding the appropriate items in a
construct. Retaining irrelevant items and removing
important items will certainly mislead the direction of a
particular study. This article demonstrates Fuzzy Delphi
method as one of the scientific analysis technique to
consolidate consensus agreement within a panel of experts
pertaining to each item's appropriateness. This method
reduces the ambiguity, diversity, and discrepancy of the
opinions among the experts hence enhances the quality of
the selected items. The main purpose of this study was to
obtain experts' consensus on the suitability of the pre-
selected items on the questionnaire.
Methods: The panel consists of sixteen experts from the
Occupational and Environmental Health Unit of Ministry of
Health, Vector-borne Disease Control Unit of Ministry of
Health and Occupational and Safety Health Unit of both
public and private universities. A set of questionnaires
related to noise and chemical exposure were compiled
based on the literature search. There was a total of six
constructs with 60 items in which three constructs for
knowledge, attitude, and practice of noise exposure and
three constructs for knowledge, attitude, and practice of
chemical exposure. The validation process replicated
recent Fuzzy Delphi method that using a concept of
Triangular Fuzzy Numbers and Defuzzification process.
Results: A 100% response rate was obtained from all the
sixteen experts with an average Likert scoring of four to five.
Post FDM analysis, the first prerequisite was fulfilled with a
threshold value (d) ≤ 0.2, hence all the six constructs were
accepted. For the second prerequisite, three items (21%)
from noise-attitude construct and four items (40%) from
chemical-practice construct had expert consensus lesser
than 75%, which giving rise to about 12% from the total
items in the questionnaire. The third prerequisite was used
to rank the items within the constructs by calculating the
average fuzzy numbers. The seven items which did not fulfill
the second prerequisite similarly had lower ranks during the
analysis, therefore those items were discarded from the final
draft.
Conclusion: Post FDM analysis, the experts' consensus on
the suitability of the pre-selected items on the questionnaire
set were obtained, hence it is now ready for further
construct validation process.
KEY WORDS:
Fuzzy Delphi, survey questionnaire, validation, noise exposure,
chemical exposure
INTRODUCTION
The questionnaire is commonly used as a measurement tool
in Public Health research. Today, varieties of validated
questionnaires are easily accessible and retrievable from
various databases. However, the main challenge as Public
Health researcher is determining the items’ suitability of the
questionnaire to be used for the intended research scope.
Consulting the experts of the research scope is one of the
ways to solve the challenge.
Fuzzy Delphi method is the current trend in consulting those
experts. It is the modification method of former classic Delphi
method developed by two scientists, Olaf Holmer and
Norman Dalkey, which has been used widely to get the
expert opinions via surveys.1It has few disadvantages, such
as misinterpretation of experts’ opinions due to neglecting
the fuzziness, no dedicated rules to yield the desired outcome,
loss of experts' interest and data due to its time-consuming
process which will lead to repeated surveys and ultimately
make the study more expensive.2,3,4 In view of the importance
to solve the ambiguity of the experts, whom might have a
common understanding,3Fuzzy Delphi Method (FDM) was
introduced over three decades ago5which was again revised
by previous scholars.6, 7 It uses fuzzy set numbers or fuzzy set
theory whereby each set will have a value from 0 to 1. This
method reduces cost and time during evaluating each item in
a questionnaire. It reduces the survey rounds and increases
items recovery rate, allows the experts to express their
opinions without any ambiguity biases, which enhances the
completeness and consistency of opinion8and to get the
consensus from the experts without jeopardising their
original opinion and by giving their real reaction towards the
questions.9
As far as concern, there are no studies available pertaining to
the Pesticide Applicators (Foggers) of the Ministry of Health.
Their nature of work, which exposes them to both noise and
chemical hazards warrants a set of questionnaires from the
Pesticide applicators questionnaire content validation:
A fuzzy delphi method
Sujith Kumar Manakandan, MPH1, Rosnah Ismail, DrPH1, Mohd Ridhuan Mohd Jamil, PhD2, Priya Ragunath,
MPH3
1Occupational Health Unit, Department of Community Health, UKM Medical Centre, The National University of Malaysia,
Kuala Lumpur, Malaysia, 2Department of Mechanical Engineering, Politeknik Nilai, Negeri Sembilan, 3Occupational Health
Unit, Disease Control Division, Ministry of Health, Putrajaya, Malaysia
ORIGINAL ARTICLE
This article was accepted: 1 February 2017
Corresponding Author: Rosnah Ismail
Email: drrose@ppukm.ukm.edu.my
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 228

Pesticide applicators questionnaire content validation: A fuzzy delphi method
Med J Malaysia Vol 72 No 4 August 2017 229
experts from both occupational health and vector-borne
disease control unit of Ministry of Health. Therefore, we feel,
FDM is the most suitable method to be used to form a set of
questionnaire. In this article, Fuzzy Delphi Method was used
prior to constructing validation process of pesticide
applicators questionnaire pertaining to knowledge, attitude
and practice related to noise and chemical exposure in this
study. The main purpose of this study was to obtain experts’
consensus on the suitability of the pre-selected items on the
questionnaire.
MATERIALS AND METHODS
Pesticide applicator questionnaire
A number of items related to knowledge, attitude and
practice to noise and chemical exposure were compiled based
on the literature search of previous studies instruments10, 12,14
using keywords noise induce hearing loss, noise, pesticide
and, knowledge, attitude and practice (KAP).Three main
databases were explored i.e. PubMed, Ovid, and Google
Scholar. Some items were obtained directly from the authors
via email.11, 13 All those studies were done on sawmill workers,
vector control workers, industrial workers and the general
population. A total of three constructs were finalized for each
noise and chemical exposure. For noise exposure, the selected
items for knowledge, attitude, and practice were 10, 14 and 6
items, respectively. Meanwhile, for chemical exposure, there
were 10 items for each construct of knowledge, attitude, and
practice. Only 33% of the selected items were in the English
language. Due to limited resources, those items were
translated from English to Malay version using simplest,
traditional forward translation15 to best of the first author’s
ability. The items were constructed based on the nature of
work environment faced by the Pesticide Applicators and
considering the purpose and conceptual basis of the
questionnaire measurement. The items were developed in
terms of routine, simple terminologies without deviating
from the original theoretical meaning of the questions. For
example, "Chemical enters the body through breathing in"
and "I am confident that I can use PPE properly”10were
translated into “Racun memasuki tubuh badan melalui
pernafasan” and “Saya pasti saya boleh menggunakan alat
pelindung diri dengan betul”, respectively. Apart from that,
few items were adopted and modified to suit the study
population, whereby originally those items were in
behavioural questions, modified into a practical statement.
Example, “how often do you wash your hands before putting
on gloves” into “Saya mencuci tangan sebelum memakai sarung
tangan semasa mengendalikan racun serangga”. This
compilation of 60 items was later presented to a panel of
experts.
Panel of experts
A panel of experts is defined as a group of persons who are
skilful in the scope of a study area. They are selected based
on leading position in public health care system with a
significant practical knowledge in their field of practice.16
They also should represent his/her circle of professional
Occupational Health group as suggested by the previous
scholar.17 In this study, the inclusion criteria for the experts
were occupational health related specialization, familiarity
with the working zone, authority in the field, and the number
of years of experience. Each chosen experts was at least one
of the following; 1a public health physician that has
published an article related pesticide applicators, 2an
administrator who manages the pesticide applicators at
district/state/national level, 3had previously worked or
experienced in pesticide application related job,4minimum
five years of experience in the related field of noise and
pesticide exposure,5an academician or tutor in the
occupational health related field. A total of sixteen experts
were recruited as the panel of experts via non-probable,
purposive sampling method. The number was considered
optimum and complied with previous suggestions which
required 10 to 50 experts.18 Lesser amount of experts is
required, i.e. 10 to 15, if they are homogenous experts.2,4
The panel of experts was from various part of Malaysia. The
panel consisted of eight Public Health Physicians of
Occupational and Environmental Health Unit of Ministry of
Health, three academicians of Occupational Health from the
public and private universities, three Health Inspectors and
two Entomologists who are presently working in the Vector-
borne Disease Control unit of Ministry of Health. They were
contacted by the researcher via a phone call to brief the FDM
and get their verbal informed consent. A set of 60 items
questionnaire was distributed to each expert via email
between October and December 2016. They were instructed to
indicate their agreement level for each item using five-point
Likert scale i.e. 1= highly disagree to 5= highly agree. Upon
successful completion, each answer sheets were delivered to
primary researcher through emails.
Data Analysis
The analysis of the data was replicated from the latest Malay
version published material,8which discusses two important
concepts of FDM, namely Triangular Fuzzy Numbers and
Defuzzification process (refer Figure 1).
Triangular Fuzzy Numbers
Triangular Fuzzy Numbers (TFN) provided an opportunity for
each recorded response made by an expert in the form of
Likert scale scoring to be translated into fuzzy scoring (Refer
Table I). Each recorded response had three values to consider,
namely the average minimum value (n1), most reasonable
value (n2), and the maximum value (n3). The rationale of
TFN was to show the fuzziness or inexactness in the opinion
made by an expert. Every opinion had a certain amount of
ambiguity which can't be addressed by using a Likert scale
because it is a fixed score. Let us say an item “Racun
memasuki tubuh badan melalui pernafasan” was scored 5
(highly agree) by an expert. The score is converted into
minimum, most reasonable, and the maximum value of 0.6,
0.8 and 1.0 fuzzy scores, respectively. It indicated the expert
agreeable to the item is 60%, 80%, and 100%, respectively.
The fuzzy scores were averaged as indicated by m1, m2 and
m3 values for further Defuzzification process.
Defuzzification process
Defuzzification process (Amax) is a ranking process of each
item to identify the importance level of each item. This
ranking process was very helpful to determine whether to
keep or discard certain items based on the following formula:
Amax = 1/3 * (m1+ m2+ m3)
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 229

Original Article
230 Med J Malaysia Vol 72 No 4 August 2017
Determination of item acceptability
There were three prerequisites to be fulfilled to determine the
acceptability of the constructs and its respective items. The
prerequisites were (1) threshold value, d-construct ≤ 0.219, (2)
experts agreement on evaluated items ≥75%20 and (3)
ranking of the item. The threshold value,d-construct indicates
the selection of certain construct based on the consensus of
the experts for each construct. However, prior to that, a
threshold value (d) for each itemwas found, by calculating
the difference between average fuzzy number and each
expert fuzzy number (refer Figure 2& 3)using the formula
below:
Once the value was obtained, a threshold value (d-construct)
was calculated by using the formula below:
Threshold Value ∑ Average Threshold Value, (d) for each item
(d-Construct) Total Experts x Total Items in Constructs
Based on the value, the acceptability of the construct was
determined, whereby a construct was accepted if the
Threshold value (d-construct) ≤ 0.2. Expert agreement on
each evaluated item was also based on threshold value (d) for
each item, whereby (d) ≤ 0.2 are accepted. The frequency of
accepted values was presented as percentage as shown in
Figure 3. Items with expert agreement of less than 75% were
discarded. The rank of an item within a similar construct was
determined after Defuzzification process as mentioned earlier
(refer Figure 1). All respondents data were entered and
analysed using Microsoft excel version 2013. A complete
Table I: The difference between Likert scale scoring and Fuzzy scoring for a five-point scale
VariaLikert Scale Scoring Linguistic variable Fuzzy Scoring
5 Highly Agree 0.6, 0.8, 1.0
4 Agree 0.4, 0.6, 0.8
3 Moderately/Not Sure 0.2, 0.4, 0.6
2 Not Agree 0.0, 0.2, 0.4
1 Highly Not Agree 0.0, 0.0, 0.2
Table II: The summary of All Three Pre-requisites Post Fuzzy Delphi Analysis (Noise)
Construct/Items Average Likert Threshold Percentage of Average of Ranking Verdict
Score Value Experts’ Fuzzy
(d) ≤ 0.2 Consensus (%) Numbers
Noise-Knowledge 0.00 Acceptable
NK-1 5 75 0.738 2 Retained
NK-2 5 75 0.738 2 Retained
NK-3 5 75 0.738 2 Retained
NK-4 5 75 0.738 2 Retained
NK-5 5 81 0.750 1 Retained
NK-6 5 75 0.738 2 Retained
NK-7 5 75 0.738 2 Retained
NK-8 5 81 0.750 1 Retained
NK-9 5 81 0.750 1 Retained
NK-10 5 81 0.750 1 Retained
Noise-Attitude 0.01 Acceptable
NA-1 5 94 0.738 3 Retained
NA-2 5 94 0.700 5 Retained
NA-3 5 94 0.725 4 Retained
NA-4 5 81 0.738 3 Retained
NA-5 4 31* 0.588 6 Discarded
NA-6 5 81 0.763 1 Retained
NA-7 5 75 0.750 2 Retained
NA-8 5 81 0.763 1 Retained
NA-9 5 94 0.725 4 Retained
NA-10 5 94 0.725 4 Retained
NA-11 5 88 0.763 1 Retained
NA-12 4 25* 0.583 7 Discarded
NA-13 5 88 0.725 4 Retained
NA-14 4 38* 0.533 8 Discarded
Noise-Practice 0.01 Acceptable
NP-1 5 75 0.738 3 Retained
NP-2 5 81 0.750 2 Retained
NP-3 5 81 0.738 3 Retained
NP-4 5 81 0.738 3 Retained
NP-5 5 88 0.775 1 Retained
NP-6 5 88 0.725 4 Retained
* Item with Experts’ consensus ≤ 75% and lowest ranking within their construct
=
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 230

Pesticide applicators questionnaire content validation: A fuzzy delphi method
Med J Malaysia Vol 72 No 4 August 2017 231
Table III: The summary of All Three Pre-requisites Post Fuzzy Delphi Analysis (Chemical)
Construct/Items Average Threshold Value Percentage of Average of Ranking Verdict
Likert Score (d) ≤ 0.2 Experts’ Fuzzy
Consensus (%) Numbers
Chemical-Knowledge 0.00 Acceptable
CK-1 5 94 0.788 1 Retained
CK-2 5 88 0.696 6 Retained
CK-3 5 94 0.788 1 Retained
CK-4 5 88 0.750 3 Retained
CK-5 5 88 0.788 1 Retained
CK-6 5 81 0.733 4 Retained
CK-7 5 88 0.775 2 Retained
CK-8 5 88 0.788 1 Retained
CK-9 5 94 0.713 5 Retained
CK-10 5 94 0.775 2 Retained
Chemical-Attitude 0.01 Acceptable
CA-1 5 94 0.729 4 Retained
CA-2 5 88 0.742 3 Retained
CA-3 5 88 0.704 6 Retained
CA-4 5 88 0.717 5 Retained
CA-5 5 94 0.788 1 Retained
CA-6 5 94 0.717 5 Retained
CA-7 5 88 0.775 2 Retained
CA-8 4 81 0.692 7 Retained
CA-9 5 88 0.717 5 Retained
CA-10 5 94 0.729 4 Retained
Chemical-Practice 0.01 Acceptable
CP-1 4 31* 0.600 6 Discarded
CP-2 4 38* 0.575 8 Discarded
CP-3 4 94 0.725 3 Discarded
CP-4 4 75 0.750 2 Discarded
CP-5 5 94 0.692 4 Retained
CP-6 4 94 0.679 5 Retained
CP-7 4 94 0.725 3 Discarded
CP-8 4 88 0.775 1 Discarded
CP-9 5 13* 0.592 7 Retained
CP-10 5 13* 0.592 7 Retained
* Item with Experts’ consensus ≤ 75% and lowest ranking within their construct
summary of the study flow process has been illustrated in
Figure 4.
RESULTS
A 100% response rate was obtained from all the sixteen
experts. All the items within the six constructs had scored
average Likert scoring of four to five, which was in the scale
of agree to highly agree. These scores were converted into
fuzzy numbers. Post FDM analysis, the first prerequisite was
fulfilled whereby all the six constructs had threshold value (d)
≤ 0.2. For the second prerequisite, three items (21%) from
noise-attitude construct and four items (40%) from chemical-
practice construct had expert consensus lesser than 75%,
which giving rise to about 12% from the total items in the
questionnaire. The third prerequisite was used to rank the
items within the constructs by calculating the average fuzzy
numbers. The seven items which did not fulfill the second
prerequisite similarly had lower ranks during the analysis.
The whole findings were summarised in the Table II and
Table III.
Those seven items were discarded and the remaining which
fulfilled the pre-requisites was retained for the final draft for
content validation process. Apart from discarding items
based on these prerequisites, little modification of items in
terms of the structure, position and wordings were done based
on the comments by the experts. These were some minor
changes and it didn’t alter the objective and nature of the
items. As a final draft, a total of six constructs with 53 items
were finalised as the result of this Fuzzy Delphi analysis.
DISCUSSION
This article demonstrated the study objective which was the
content validation of pesticide applicators questionnaire by
obtaining the experts’ consensus on suitability of the pre-
selected items on the questionnaire and using FDM to
ultimately remove the unfit items. This study found that the
average Likert scale scoring by the experts for all the items
are from agreeable to highly agreeable range, which means
all 60 items can be accepted. However, post FDM analysis,
only 53 items were fulfilled all the pre-requisites. About 12%
of the items didn’t match the terms, hence those items were
regarded as failure to achieve consensus from the expert
panel and removed. This 12% is the fuzziness or uncertainty
among the expert panel which was not detected by the usual
Likert Scale scoring system. Every expert will have their own
uncertainty towards certain variable, which often regarded as
the “grey area”. The use of FDM is to deal with those “grey
area”, ensuring a qualified analysis outcome. Furthermore,
this method catered all the experts’ opinion, considering
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 231

Original Article
232 Med J Malaysia Vol 72 No 4 August 2017
Fig. 1: Triangular Fuzzy Number and Defuzzification Process.
Fig. 2: Method to obtain Threshold value (d) for each item.
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 232

Pesticide applicators questionnaire content validation: A fuzzy delphi method
Med J Malaysia Vol 72 No 4 August 2017 233
some expert are more experienced, some are more
knowledgeable, some with relevant skills and some has the
policy making authority in the field. This variety of opinions
is merged together to support each other’s deficiency to derive
at the desirable outcome. Moreover, the final draft of the
questions was arranged based on priority ranking derived by
the analysis. On the positive note, although the items were
picked from variety of literature which was very unusual
compared to the traditional practice of selecting a
questionnaire, the difference between the initial selection of
items and the level of experts’ opinion was very minimal
(12%). This could be possibly due to majority of the items are
originally from the local language and the remaining items
were hand-picked, translated and modified by the author
who is equally experienced and knowledgeable in the similar
field.
Generally, an indoor meetings or workshops will be
conducted to gather the experts under a roof in order to get
their consensus. This involves tedious process, starting from
the calling letter, arranging the venue, travelling expenses,
refreshment beverages and obviously plenty of time will be
spent. The main significant advantage of this study was, it
was conducted in a very short span of time, with zero costing
involved. It was also a hustle-free job for the experts as well.
The experts’ responses were gathered via emails and
messages at their convenience. This method will certainly
reduce the risk of bias by ensuring anonymity and welcoming
the opinion of atypical views among the experts and the
responses are totally independent without the fear of
judgemental by others which usually present in any routine
group discussions or meetings.21
Pertaining to this study, it introduces that FDM can be used
to get expert’s opinion and consensus in order to achieve a
decision. This method can be used as a pre-construct
validation tool to select the suitable items before subjecting it
to a construct validation process. Most importantly, this
method gives a proper quantitative approach to usual group
discussions or meetings which are in a qualitative manner.
This questionnaire can be considered as accepted by the
experts without any prejudice and it can be used for the
targeted population after confirmatory validation process.
However, there are some limitations with this method,
whereby, the researcher or a person who is conducting this
FDM should have some pre-existing background knowledge
regarding the subject, whereby he/she must be an expert too.
Moreover, FDM requires existing kinds of literature or matter,
to begin with, and this method is not suitable for developing
brand new items. On the other note, this study required
constant reminder to the experts to give their response. This
is mainly due to limited time factor and this might lead to the
emotional bias among the experts. In addition to that, the
selection of the expert was by purposive sampling method
based on their willingness and availability. A probable
sampling method among the experts and more time frame
would have been yielded a different result.
As a recommendation, FDM should be widely used in medical
related studies, to get expert’s opinion and consensus
especially in developing a protocol or guidelines related to
medical practices. Although limited, there are some studies
which use this method for medically related researches. It was
used in one of the studies to find consensus for Asthma
management guidelines.22 Another study in Mexico which
used this technique to determine the socio-ecological factors
that influence adherence to mammography screening.3
However, locally in Malaysia, this method is yet to be
introduced in the field of medicine. Furthermore, it is hoped
that this study can be beneficial as a guidance for any future
medical or health related research which intends to use FDM
for their studies.
Fig. 3: Construct and items acceptability based on experts' consensus.
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 233

Original Article
234 Med J Malaysia Vol 72 No 4 August 2017
CONCLUSION
Post FDM analysis, the experts’ consensus on suitability of the
pre-selected items on the questionnaire set were obtained,
hence it is now ready for further construct validation process.
ACKNOWLEDGEMENT
We would like to thank the Director General of Health
Malaysia for his permission to publish this article. This study
is part of doctorate research which is supported by the Dana
Fundamental PPUKM (Project code: FF-2016-291) and ethical
approval from the Medical Research and Ethics Committee
(MREC), Ministry of Health (NMRR-16-660-30666-IIR). The
research team would like to thank the sixteen experts for
their contribution to this study. Our sincere acknowledgment
to Associate Professor Dr. Retneswari Masilamani and
Associate Professor Dr. Razman Mohd Rus for their inputs in
forming the items. Last but not least, we express our gratitude
to the Department of Community Health, PPUKM and to
those who had extended their help in contributing to this
manuscript.
REFERENCES
1. Hsu Y-L, Lee C-H, Kreng VB. The application of Fuzzy Delphi Method and
Fuzzy AHP in lubricant regenerative technology selection. Expert Systems
with Applications. 2010;37(1):419-25.
2. Adler M, Ziglio E. Gazing into the oracle: The Delphi method and its
application to social policy and public health: Jessica Kingsley Publishers;
1996.
Fig. 4: Summary of content validation using Fuzzy Delphi Method.
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 234

Pesticide applicators questionnaire content validation: A fuzzy delphi method
Med J Malaysia Vol 72 No 4 August 2017 235
3. Sanchez-Lezama AP, Cavazos-Arroyo J, Albavera-Hernandez C. Applying
the Fuzzy Delphi Method for determining socio-ecological factors that
influence adherence to mammography screening in rural areas of Mexico.
Cadernos de saúde pública. 2014;30(2):245-58.
4. Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate
research. Journal of information technology education. 2007;6:1.
5. Murray TJ, Pipino LL, van Gigch JP. A pilot study of fuzzy set modification
of Delphi. Human Systems Management. 1985;5(1):76-80.
6. Alias N, Rosman F, Rahman MNA, Dewitt D. The potential of video game
in Malay language learning for foreign students in a public higher
education institution. Pro cedia -Soci al and Behavioral Sciences.
2015;176:1020-7.
7. Kaufman A, Gupta M. Introduction to Fuzzy Arithmetic: theory and
Application, van no strand Reinhold. New York. 1988.
8. Mohd Ridhuan Mohd Jamil SS, Zaharah Hussin, Nurulrabihah Mat Noh,
Ahmad Arifin Sapar. Pengenalan Asas Kaedah Fuzzy Delphi Dalam
Penyelidikan Rekabentuk Pembangunan. Bandar Baru Bangi: Minda
Intelek Agency; 2014.
9. Noh NM, Razak SHA, Alias N, Siraj S, Jamil MRM, Hussin Z. Usage of
Facebook: The future impact of curriculum implementation on students in
Malaysia. Procedia-Social and Behavioral Sciences. 2013;103:1261-70.
10. Geer LA, Curbow BA, Anna DH, Lees PS, Buckley TJ. Development of a
questionnaire to assess worker knowledge, attitudes and perceptions
underlying dermal exposure. Scandinavian journal of work, environment
& health. 2006:209-18.
11. Masilamani R, Rasib A, Darus A, Ting AS. Noise-induced hearing loss and
associated factors among vector control workers in a Malaysian State.
Asia-Pacific Journal of Public Health. 2014;26(6):642-50.
12. Nor Saleha IT, Noor Hassim I. A study on compliance to hearing
conservation programme among industries in Negeri Sembilan, Malaysia.
Industrial health. 2006;44(4):584-91.
13. Razman M, Naing L, Aziah D, Kamarul I. Validation of Noise Induced
Hearing Loss Questionnaire Among Malay Sawmill Workers in Kelantan
Malaysia. The International Medical Journal of Malaysia. 2010;9(2).
14. Reilly MJ, Rosenman KD, Kalinowski DJ. Occupational noise-induced
hearing loss surveillance in Michigan. Journal of occupational and
environmental medicine. 1998;40(8):667-74.
15. Harkness JA, Schoua-Glusberg A. Questionnaires in translation. DEU;
1998.
16. Baker J, Lovell K, Harris N. How expert are the experts? An exploration of
the concept of ‘expert’within Delphi panel techniques. Nurse researcher.
2006;14(1):59-70.
17. Fink A , Kosecoff J , Chassin M, Brook RH. Consens us methods:
characteristics and guidelines for use. American journal of public health.
1984;74(9):979-83.
18. Jones H, Twiss B. Forecasting technology for planning decisions: Springer;
1978.
19. Cheng C-H, Lin Y. Evaluating the best main battle tank using fuzzy
decision theory with linguistic criteria evaluation. European Journal of
Operational Research. 2002;142(1):174-86.
20. Chu H-C, Hwang G-J. A Delphi-based approach to developing expert
systems with the cooperation of multiple experts. Expert systems with
applications. 2008;34(4):2826-40.
21. Horner K, Islam M, Flygare L, Tsiklakis K, Whaites E. Basic principles for
use of dental cone beam computed tomography: consensus guidelines of
the European Academy o f Dental and Maxillofacial Radiology.
Dentomaxillofacial Radiology. 2014.
22. Paris MB. The asthma consensus in the light of the Delphi method.
Allergologia et immunopathologia. 2010;38(6):293-4.
5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 235