Content uploaded by David Dunaetz
Author content
All content in this area was uploaded by David Dunaetz on Oct 10, 2023
Content may be subject to copyright.
GREAT COMMISSION
RESEARCH JOURNAL
2023, Vol. 15(2) 5-24
Survey and Questionnaire Design
for Church-Based Research
David R. Dunaetz, Editor
Azusa Pacific University
Abstract
To better understand behaviors, beliefs, and attitudes in churches, survey
research can be very useful, especially to test a hypothesis that we believe
to be true. Before creating a survey or questionnaire, the phenomena
being examined need to be well understood and appropriate measures
chosen. The items included in the survey need to measure the concepts
desired and should be clear and unambiguous. The survey should be laid
out to motivate maximum participation and minimize biased responses.
This article provides many principles for how to accomplish these goals
and to ensure that the research undertaken is credible.
-------------------------------
Doing research in churches is very different from experimental research
done in controlled conditions. The researcher cannot randomly assign
participants to one of several conditions (e.g., those who are encouraged
to memorize Bible verses vs. those who are not so encouraged) and then
measure the consequences (e.g., the quality of the participants’ marriage
or their mental health). Rather, research in churches typically focuses on
real-life events, beliefs, feelings, and thoughts that have long-term
consequences. Since typically part of a church’s mission is to respond to
the needs of the congregants (Church Relevance, 2013), putting church
6 Great Commission Research Journal 15(2)
members in an experimental condition that may be detrimental to their
well-being would not be ethical (Lowman, 2006). In contrast to laboratory
research, survey research is much more appropriate for churches.
Survey research seeks to capture a picture of participants’ behaviors,
thoughts, feelings, and attitudes at a specific moment in time. If the
sample is large enough, the relationship between the variables measured
can be detected, including the strength and direction of the relationship
enabling hypotheses to be tested, such as “Christians who memorize
Scripture regularly have more satisfying marriages.” When we find
evidence that specific behaviors or beliefs are correlated, and we have good
reason to believe that one causes the other, then we can promote the
beliefs and behaviors which best correspond to what Christ has
commanded us, a critical aspect of fulfilling the Great Commission (Matt.
28:19-20).
What are Surveys and Questionnaires?
Surveys, typically used synonymously with questionnaires, are lists of
questions or items to which participants respond to provide a measure of
one or more of their characteristics that vary among them (Cameron &
Duce, 2013; Crano et al., 2015; Dunaetz, 2020; Morling, 2021). The word
“survey” emphasizes the idea that we need to ask these questions to a
broad range of people in the population that we are studying. The word
“questionnaire” emphasizes the many questions that need to be asked. If
we want to measure a phenomenon, we usually need to ask many
questions to capture the full range of thoughts, feelings, and behaviors that
characterize the phenomenon.
A survey differs from a census. A census (such as mandated by the U.S.
Constitution) seeks to obtain information from every person in the
population studied. In general, few or no inferences are needed to come to
a reasonably accurate conclusion concerning the variables measured. In
contrast, a survey only measures a sample of the population that interests
us. Statistical tests (for example, correlations and differences between
groups) are used to determine if we can make inferences (conclusions)
about the whole population based on the data collected from the sample.
An example of a census in a church would be a vote at the business
meeting. It is assumed that everyone who has an opinion expresses it, and
each vote counts. In contrast, survey research does not require everyone
concerned to participate. Once enough participants have completed the
survey, the results are compiled, and statistical inferences are made. For
example, a study may find that there is a positive correlation between how
engaging church volunteers find the ministries in which they are involved
Dunaetz 7
and the degree to which they participate in these ministries (Dunaetz &
Bocock, 2020). If this correlation is strong enough, we can infer that this
relationship is true for all church members (not just those who
participated in the survey). This inference should motivate church leaders
to find and develop engaging ministries that correspond to the interests
and gifts of their church members.
Why and When are Surveys Used?
Surveys can be used for both quantitative and qualitative research.
Quantitative research studies narrow questions and seeks to reduce
subjectivity by measuring the variables studied precisely. Qualitative
research examines broader, more over-arching questions but they tend to
be more subjective (Creswell & Poth, 2016; Dunaetz, 2023; Patton, 2014).
Surveys tend to be less useful than interviews for qualitative research.
Whereas humans are typically quite at ease discussing various topics with
a person who comes across as reasonably trustworthy, they are quite
resistant to writing sentences and paragraphs. It is generally much easier
to get detailed verbal information from interviews than from surveys (For
good examples, see Nehrbass, 2022, and Thigpen, 2023, in this issue).
When doing quantitative research, the goal is either to accurately
measure a specific variable (e.g., how satisfied congregants are with the
pastor’s preaching, using a 1 to 5 scale) or test a hypothesis that is
suspected to be true (Dunaetz, 2021). If the goal is to simply measure a
specific variable (known as descriptive statistics), the utility of the
information might be quite limited. Suppose a survey was used to measure
congregants’ satisfaction with their pastor’s preaching and the average
score was 4.0 out of 5 for the sample. We could also calculate the
confidence interval, the range of values which probably contains the value
of how much congregants like the pastor’s preaching if the entire
congregation had provided information. The confidence interval is
typically calculated at 95% certainty, which in this case might produce a
95% confidence interval of something like 3.8 - 4.2 out of 5. However, such
information in itself is not very useful. It says that on average, people like
the pastor’s preaching but not everyone likes it a lot, a statement that
would be true for most churches.
However, quantitative research becomes much more interesting when
hypotheses are tested. For example, the hypothesis “Congregants will
value a sermon series on apologetics more than a sermon series on
Leviticus.” To test this hypothesis, the same measure of satisfaction with
the pastor’s preaching can be used, both after the series on apologetics and
after the series on Leviticus. If the difference between average scores (e.g.,
8 Great Commission Research Journal 15(2)
apologetics is rated higher than Leviticus) is statistically significant (that
is, there is less than a 5% chance of getting this difference from the sample
measured if the congregation as a whole would rate the two series as
equal), then conclusions can be made about what types of sermons are
most valued by contemporary church members in churches like the one
studied. These statistical inferences provide more information than simple
descriptive statistics because they can be used for decision-making.
For surveys to be useful, they must be designed very carefully. Before
focusing on the specific items to be used, researchers need to be clear
about what should be measured.
What Should Be Measured in a Survey?
When doing church-based research, it is absolutely necessary for the
researcher to clearly define the purpose of the research. In general,
research is carried out to find at least a partial solution to a problem, often
called the research problem. To define the problem, not only must the
context be understood (e.g., a specific church, the technological context of
the 21st century, or contemporary evangelicalism in the U.S.), but the
phenomena surrounding the problem should also be understood to the
degree that it is possible. Generally, this is done through a literature review
(Cooper, 1988; Dunaetz, 2022b; Rosenthal & DiMatteo, 2001; Torraco,
2005) where past research relevant to the research problem is collected
and analyzed. The end result is a hypothesis concerning a possible
solution. The hypothesis is a specific statement involving the variables
associated with the research problem that, if true, would contribute to at
least a partial solution to the problem (Bordens & Abbott, 2011; Morling,
2021; Salkind, 2017).
Once the researcher believes that a hypothesis has the potential to be
true and that its confirmation would provide useful information upon
which one could act, the researcher can then list the constructs (variables)
that need to be measured in the survey (Dunaetz, 2022a). Each hypothesis
may involve two variables (e.g., “Time spent in Bible reading is positively
correlated with life satisfaction.”) or it may involve more variables (e.g.,
“Time spent in Bible reading is positively correlated with life satisfaction,
and the strength of this relationship depends on educational level.”). Apart
from demographic information that is collected to paint a picture of the
participants, only the constructs that are included in the hypotheses
should be included in the survey. It is tempting to want to add other
constructs out of curiosity (“I wonder how age of conversion relates to this
phenomenon?”). However, if a literature review has not been done to
understand the extra variable added, the likelihood of measuring the
Dunaetz 9
variable in a way that would provide useful information is not very high.
Moreover, after the data is collected, the researcher risks “hypothesizing
after the results are known” or HARKing, which artificially inflates the
likelihood of finding significant results and is considered an unethical
research practice (John et al., 2012; Kerr, 1998).
After the researcher determines what should be measured in the
survey, the best way to measure these variables must be determined. The
various ways of measuring a variable are called operationalizations. In
general, it is better to use psychometrically validated measures that have
already been demonstrated to measure the desired concept than to make
up a new version of the measure, which may or may not work well. These
operationalizations are found during the literature review, when all (or at
least as much as is possible) of the past research conducted on these
variables is studied. Even a concept as straightforward as church
attendance can be quite complex to measure and a number of approaches
have been developed to measure it (Marcum, 1999; Rossi & Scappini,
2014; Smith, 1998). When a variable is psychological or spiritual in nature,
there are likely to be many ways to measure it, none of which are perfect
(Hill & Hood, 1999). Researchers need to choose the measure that best
corresponds to their conception of the variable, explaining why their
chosen operationalization is the most appropriate for the study.
How Should the Questions Be Phrased?
In cases where new items need to be created, they must be clear enough to
provide the information sought (Dillman et al., 2014; Dunaetz, 2020;
Ekinci, 2015; Fowler, 2013). First, they need to be simple and
understandable. People who read the Great Commission Research
Journal or other academic journals are likely to be stronger readers than
the majority of people in the churches they attend or lead. What is simple
to the researcher is often not simple to a broader audience.
Second, the questions need to be unbiased. “When did you stop
beating your wife?” is an extreme example of a biased question because it
assumes that the participant used to beat his wife. “When did you become
a Christian?” similarly assumes that the person is a Christian, which might
not be true. “Do you think our forward-thinking and culturally sensitive
church should change its name to better reach our community?” is biased
because it is clear what the desired response is.
Third, the questions need to be unambiguous. “Do you use a computer
for your devotions?” is ambiguous because not everyone is going to
understand “devotions.” Moreover, it is not clear if using tablets and
phones counts. Rather, “Have you ever read the Bible on a digital device?”
10 Great Commission Research Journal 15(2)
or “Do you regularly read the Bible on a digital device?” are much less
ambiguous questions; however, it should be noted that these two
questions are measuring very different variables, again emphasizing the
need for clear, specific hypotheses.
What to avoid?
Sometimes it is easier to think in terms of what to avoid when designing
survey questions. Here is a list of nine things to avoid in surveys and
questionnaires (Dillman et al., 2014; Dunaetz, 2020; Ekinci, 2015;
Fowler, 2013).
1. Complexity. Indicate on a scale of 1 to 10 how much you agree
with the statement “My pastor has the personality necessary to lead a
team in the creation of innovative ideas that will transform the church
and our region.” This statement is too complex for most people. A ten-
point scale is difficult to use (e.g., what’s the difference between a 2 and a
3?). Most people have never thought about what type of personality is
needed to lead a team, much less know how to determine if a person has
it. Moreover, the item is not just asking about leading a team, but a very
specific type of team that most people have not thought about. Complexity
is increased by the length of the item, the number of clauses, and the
average number of words in a clause (Yan & Tourangeau, 2008). For a
complex question like this, people often mentally reduce it to “How much
do I like my pastor?” The more participants like their pastor, the more
likely they are to strongly agree with the long and complex statement.
2. Leading Questions. What words would you use to describe the
life-changing impact that our church has had on people? Similar to biased
questions, leading questions imply a certain answer or a certain type of
answer. In this case, the question implies a very positive response is
expected from the participant. This is due to the anchoring effect
(Furnham & Boo, 2011; Jacowitz & Kahneman, 1995). When we try to
measure or estimate something, we use the salient emotional concepts
most available to us as an anchor to help us decide, in this case, “life-
changing” and “impact.” In this case, we could get better information with
the use of more neutral terms, such as “What words would people around
you use to describe our church?” or “How has our church influenced what
you do?”
3. Ambiguous Categories. Please indicate your age: A. 20-30. B.
30-39, C. 55+. If the choices given to participants present an incomplete
range (How do 19-year-olds answer? How do 45-year-olds answer?) or if
the categories are not mutually exclusive (What response does a 30-year-
old choose?), participants often cannot provide meaningful responses.
Dunaetz 11
They tend to get frustrated with the survey and stop responding. “Please
indicate your age: A. Under 30, B. 30-39, C. 40 and above” would be a
better way to ask this question. However, even better would be a fill-in-
the-blank question, “How old are you? ________?” By getting a more
precise age, we would be better able to detect age-related trends.
4. Double-Barreled Questions. On a 1 to 5 scale, please indicate
how much you agree with the statement “I like my pastor because of his
preaching style.” Double-barreled items ask about two ideas at one time
(Menold, 2020). In this example, participants need to indicate how much
they like their pastor specifically because of his (or her) preaching style.
But what if the participant likes the pastor a lot but only tolerates his
preaching? Or what if the participant does not like the pastor, but finds
him to be a very entertaining preacher? Double-barreled items, which are
often formed with a main clause and a dependent clause or by connecting
two nouns with a conjunction like “or,” need to be broken up into two
separate items. “I like my pastor” and “I like my pastor’s preaching style”
will produce much more meaningful results. An especially important
double-barreled question to avoid is “Are you a born-again or evangelical
Christian?” The overlap in meaning between these words that used to be
near-synonyms is getting smaller and smaller, especially for non-white
people and people who are not politically conservative (Margolis, 2022).
5. Burdensome items. Please list ten reasons why you started
attending our church. Burdensome items are cognitively demanding
(Warriner, 1991). Humans, by nature, are cognitive misers; they do not
want to think hard about something unless they see clear benefits from it
(Stanovich, 2018). Asking participants to produce a list of something is
burdensome and will result in lowered participation rates; participants
who encounter a difficult item are more likely to give up on the survey.
In addition to the creation of lists, asking participants to rank order
items is burdensome and discouraging. In general, participants should not
be asked to rank more than four items in order of preference. Instead of
an item such as “Please rank the following 10 ministries in order of how
important they are to you,” Likert scale items should be provided for each
element being evaluated. Likert items are statements where participants are
asked to indicate to what degree they agree or disagree with the statements
(typically on a 5-point scale ranging from strongly disagree to strongly agree,
arranged like a multiple-choice quiz; Likert, 1932). Unlike ranked ordering
items, participants can easily respond to them. In this example, participants
would be given the prompt “Please indicate to what degree you agree with
the statement ‘This is an important ministry’” and then a list (typically in
a grid) of the ten ministries would be provided, one on each line.
12 Great Commission Research Journal 15(2)
6. Items that Generate Little Variation in Responses. To
maximize the power of statistics to detect significant results, answers
should be distributed fairly symmetrically (when there are only two
choices, such as true or false) or as a bell curve when multiple responses
are possible (such as Likert items). Asking church attenders how much
they agree with the statement “I believe Jesus said some things that are
valuable” would generate little variation in the responses; almost everyone
would answer that they strongly agree with that statement. The item used
needs to be more specific or more extreme to generate variation in
responses. The statement “I believe that Jesus’ teachings need to influence
every decision that I make” will perhaps capture the same idea (depending
on the researcher’s intention), but will be more normally distributed, that
is, shaped like a bell curve.
7. Inapplicable Items. What aspect of our church’s Sunday School
do your children enjoy the most? A. Interaction with the adults, B. The
Bible teaching, C. The snacks. Such an item would only be meaningful for
adults with children in the church’s Sunday School. If such a question is
given to people without children, they risk being flustered and
discontinuing the survey. The researcher can avoid such problems by
including a statement such as “Questions 13 through 17 should only be
answered by parents with children who attend the church’s Sunday
School.” Or even better would be to use the survey tool “skip logic” which
is available on many electronic survey platforms such as Google Forms or
SurveyMonkey. An initial question determines what question comes next,
such as “Do you have children who attend this church’s Sunday School?”
Those who respond that they do not have children will not be shown the
irrelevant questions. This makes the survey go smoother and contributes
to higher participation rates.
8. Forced Responses to Open-Ended or Controversial Items.
*Explain how you first started coming to our church? Most electronic
survey tools allow for the researcher to make some or all of the items
mandatory. These “forced choice” items are often indicated by an asterisk
(*). Forced choice items are good when the responses are not cognitively
demanding, as is the case for multiple-choice or Likert items. However,
open-ended questions are much more difficult to answer and should rarely
be mandatory. Requiring participants to type a phrase, a sentence, or a
paragraph will decrease the participation rates dramatically for people
who are not extremely motivated to complete the survey. Similarly, forced
response items that are controversial, or which can reveal the participant’s
identity are also to be avoided.
For example, demographic items should not be mandatory (e.g., age,
Dunaetz 13
sex, race, residence). There is a high percentage of people who do not wish
to respond to them. They may think that the information will be used
against them, that revealing such information would make them
identifiable, that the categories that the researcher has chosen for sex or race
are offensive, or that it is morally inappropriate to collect such information.
To reduce negative reactions to demographic questions, not only
should their completion be optional, but these questions should be at the
end of the survey. Placing them at the beginning can demotivate people to
complete the rest of the survey. The choices provided for sex and
race/ethnicity should correspond to the intended audience of the survey.
For a general audience (such as everyone who might ever visit the church),
it may be appropriate to offer three possible answers to “What is your sex?
A. Male, B. Female, C. Other/Prefer not to state.” It is best not to include
a fill-in-the-blank for the final choice unless this information is needed to
test the hypotheses.
An item on race can be even more controversial than an item on sex.
A list of expected races and ethnicities should be provided based on the
intended audience. In the U.S., this list would probably include at least
Asian, Black, Hispanic, and White. However, depending on the region or
church, other races or ethnicities should be included (but generally there
is no need to have separate questions for race and ethnicity). Moreover, a
fill-in-the-blank option “Other: _______” should be provided so that all
participants have the possibility of expressing their identity. This will also
offer participants who do not like race-related items to express their
discontent by responding with answers such as “human being” or “Jedi
knight.” Since many people have multiple racial and ethnic identities,
participants should be given the option to choose more than one response
and they should be encouraged to choose all that apply.
9. Items that Lead to Missing Data. How much do you appreciate
our church’s present worship style? A. Not at all, B. A little, C. A lot, D. No
opinion. By giving options such as “No opinion,” the researcher causes
much useful data to be lost. We do not know why people who respond this
way have no opinion. Perhaps they do not attend the worship service.
Perhaps they have many complex thoughts about the worship service.
Perhaps they are neutral about the worship style. But in any case, the “no
opinion” response provides no useful information, and the data is lost. To
avoid this problem, the use of Likert items allows the researcher to use all
the data. In this case, the item could be rephrased, “Please indicate to what
degree you agree with the following statement: I appreciate our church’s
present worship style. A. Strongly disagree, B. Disagree, C. Neither agree
nor disagree, D. Agree, E. Strongly agree.” All participants can respond to
14 Great Commission Research Journal 15(2)
this item and no data is lost.
How Should the Survey Be Laid Out?
A survey is much more than a list of questions. It needs to be structured so
that it is clear and provides motivation to participate. As Christians, we
should try to make our survey an act of love and respect so that it is a
positive experience for those participating. The following is a possible
outline that researchers can follow when designing a survey.
I. Title
II. Introduction
III. Informed consent
IV. Main Body of the Survey
V. Demographics
VI. Conclusion
Parts of the Survey
Title. The survey should have an interesting title that provides a
general idea of what it is about. However, the title must be general enough
to not give away any of the hypotheses. If a survey has a title such as
“Church Involvement and Mental Health,” the participant, especially if
they know the researcher or the organization sponsoring the research, may
be able to guess that the hypothesis is that greater church involvement
predicts better mental health. Although this is a legitimate research topic
(Nooney & Woodrum, 2002; Wright et al., 1993), participants should not
be aware of the hypothesis because it will bias their responses, especially
if they want to support the researcher. Cues that reveal the hypothesis are
called demand characteristics and act as an extraneous variable that
influence the results (Nichols & Maner, 2008). A better title for this survey
would be “Attitudes of Church Attenders.” It provides a general idea about
the topic of the survey, but not enough to reveal the hypothesis.
The Introduction. The purpose of the introduction is to motivate
readers to participate in the survey and to provide enough information so
that they know what they are getting into. It should begin with a warm
greeting that demonstrates that the researcher values their experiences,
thoughts, and feelings. It should describe the general purpose of the
survey, without giving away the hypotheses. It should provide information
about how long completing the survey will take. In general, a well-
structured survey with all multiple-choice items (e.g., Likert items) can be
completed quickly by most people, perhaps at the rate of 8-10 items/minute.
This means that many surveys can be completed by most people in less than
Dunaetz 15
5 minutes.
The introduction should also be clear concerning whether this is an
anonymous survey or not. If it is anonymous, completion rates may be
higher, especially if respondents are not accountable for participating, as
would be the case in most church-based research. However, when sending
reminders to complete the survey, the reminders will be sent to people who
have already completed it, which risks annoying them. A lack of
anonymity, however, may also influence responses so that they are more
socially acceptable (Fuller, 1974; Randall & Fernandes, 1991). In any case,
the introduction should make it clear if the data collected will be
anonymous, confidential (not shared beyond the group conducting the
research), or available to a broader audience.
The introduction should also motivate people to participate. Perhaps
the research will contribute to something that the participant values and
wants to support. For example, in research that would be appropriate for
publication in the Great Commission Research Journal, the introduction
could state “Your participation will help us to better understand how
churches and individuals can better contribute to the fulfillment of the
Great Commission.” The introduction can also increase motivation by
appealing to self-interest by including a statement such as “Participation
in this survey may help you to reflect on your church involvement and your
values in a new way and may give insight into your values and behaviors.”
Informed Consent. To ethically conduct research, the participants’
informed consent is often requested before the participant is allowed
access to the survey questions. The purpose of the informed consent,
originally developed in the world of medical testing to protect patients, is
to inform the participants of the potential risks involved in participation
(Faden & Beauchamp, 1986). The participant is given the right to not
complete the survey and the assurance that there will be no negative
consequences for their lack of participation. Although the right to not
participate may seem obvious for surveys distributed for research on
church-related phenomena, most universities, government institutions, and
publishers require researchers to only use data from participants who have
provided informed consent. Many institutions mandate that specific
informed consent texts be used. Examples of informed consent documents
may be easily found with an internet search in order to develop an informed
consent appropriate for the study when there is no institutional standard.
In paper-and-pencil surveys, the informed consent is usually a
separate document that must be signed by the participants before they
receive the survey which remains anonymous. However, most surveys are
administered electronically today. About 93% of Americans have internet
16 Great Commission Research Journal 15(2)
access (Pew Research Center, 2021). Those who do not have internet
access tend to be elderly and less educated, a population that is less likely
to participate in any survey. For participants to provide their informed
consent before participating in a survey, the informed consent text is often
shown after the introduction at the end of a page. The final line of the
informed consent may say something like “By clicking NEXT below, I am
providing my informed consent to participate in this survey.” Thus, only
participants who have provided their informed consent will have access to
the survey items found in the next section.
The Main Body of the Survey. This section presents the items that
will measure the main variables that are the focus of the study, such as
those used to test hypotheses. It is easiest for the participant if all the items
measuring one variable are presented together, followed by another
section for the next variable. For example, instead of mixing questions
about church participation and mental health together, it is less cognitively
demanding for the participants to answer all the questions about mental
health first, followed by all the questions on church participation.
There are various ways of formatting items in surveys. The following
examples are taken from (or are similar to) a survey-based study by
Neherbass et al. (2023) examining the relationship between missionaries’
education and their Great Commission behaviors (evangelizing, baptizing,
teaching, and teaching others to do the same). The study was done on
Google Forms, a web-based survey app that is available for free to most
people who have some sort of Google account.
Multiple Choice Items. The simplest type of item is the multiple-choice
item. A prompt is provided and afterwards, the participant chooses a
response (Figure 1). This is also called a radio button item because, like car
radios, when a button is chosen, it deselects the previously chosen button.
Dunaetz 17
Figure 1. A Multiple Choice Survey Item
Note that the asterisk (*) indicates that when this question was set up, it
was set as a mandatory response item; the participant is not able to go to
the next page or submit the survey without completing this item. This is a
good technique to reduce missing data that may cause a participant’s
responses to become unusable. Note also that the option “None” was
included so that all participants would be able to answer this. The option
“Other: ________” was provided so that participants could respond even
if they did not think their school fit into one of the provided categories.
Checkbox Items. Unlike multiple choice, checkboxes allow
participants to choose more than one response. This should be used when
the options are not mutually exclusive or more than one response would
be appropriate. Note that in survey software, checkbox items are often
indicated with a square (□), whereas multiple choice items are indicated
with an (○).
18 Great Commission Research Journal 15(2)
Figure 2. Checkbox
Linear Scale Items. Also known as scaling items, linear scale items
provide two endpoints of a scale, but the meanings of the intermediate
points are not specified. It is left up to the participant to determine their
meaning. This type of scale is usually easy for participants to complete (See
Figure 3). Some people tend to choose only the extreme responses
(Batchelor & Miao, 2016), but with a large number of participants, the
distribution will usually still be psychometrically valid.
Figure 3. Linear Scale
Multiple Choice Grid. The best choice for reducing the cognitive
demands of a survey in order to maximize completion rates is the multiple
choice grid. This format is especially convenient for Likert items
(Jamieson, 2004; Likert, 1932). When all the columns have the same
meanings, the participants can respond to the prompts very quickly
(Figure 4). They should be used whenever possible.
Dunaetz 19
Figure 4. A Multiple Choice Grid
Short Answer Items. When the participant is asked to type in a bit of
data (rather than to click to indicate their choice), short answer or fill-in-
the-blank items are used. These work best for very short answers that do
not require complete sentences such as “What is your age?” or “How long
(in years) have you attended this church? Please round your answer to the
nearest year. If less than 6 months, please enter ‘0’.”
Item Formats Less Frequently Used. Several other item formats are
available in survey software but should be used sparingly, if at all, because
they are prone to recording error or are burdensome. Dropdown menus
(such as to indicate in what country the participant lives) often frustrate
participants because they are time-consuming and often result in
accidentally choosing the wrong answer. Long answer items or paragraph
answer items provide one or more lines for the participants to provide
their responses. Composing complex phrases, complete sentences, or
paragraphs is cognitively taxing for many participants and these items are
often skipped, sometimes leading to quitting the survey at the point they
are encountered.
Demographics. At the end of the questions, demographic items should
be included. The purpose of these items is to know what type of people
participated in the survey. Often, no hypotheses are related to
demographic differences, but knowing who participated in a survey helps
readers of the research understand for whom it is most applicable.
Demographic variables measured usually include age, sex, and race, but
may also include education level, employment status, or church tenure
(how long they have attended the church). People often fear becoming
identifiable if too many questions are asked, so keeping them to a
20 Great Commission Research Journal 15(2)
minimum is appropriate.
Conclusion. The survey should end with thanking the participants for
their time and a reminder to click the final button (e.g., SUBMIT) so that
their responses are recorded. The researcher may want to offer the
participants the possibility of obtaining the results of the survey. To ensure
anonymity, the participants should be informed that they can email the
researcher directly at an email address that is included in the conclusion,
rather than responding to an item that asks for their address. Researchers
should make a list of such participants’ email addresses as they receive
them and then send the final report or a summary report to them when it
is ready.
Other Survey Design Issues
Three other survey design issues should be addressed: question order,
response order, and pretesting.
Question Order. As already mentioned, demographic questions
should appear at the end of the survey. To begin the survey, the first
questions should be interesting and easy to understand (e.g., no sentences
with negations) in order to help motivate and situate the participant.
When choosing which variables or constructs to measure first, the most
general ones should be first so that the order does not influence responses.
The question “How satisfied with life are you?” followed by “How satisfied
with your love life are you?” may have a very different response than if the
order were reversed. Starting off with specific questions makes that
specific topic salient which can prevent the participant from responding to
general questions from a more global perspective (Kalton & Schuman,
1982; Moore, 2002).
Response Order. For numeric responses (e.g., a scale from 1 to 5)
or responses to Likert items (e.g., Strongly Disagree to Strongly Agree),
lower or more negative responses should be presented first (at the top or
left) while the higher and the most positive responses should be at the end
(at the bottom or right). This creates a natural flow to the responses and
makes answering multiple items easier. Switching the order of the
responses within the survey must be avoided at all costs; some participants
are likely to not notice the change of order and will answer as if the order
had not been changed.
When multiple-choice responses do not have a natural order (such as
a numeric order), the order in which they are presented is likely to
influence the response (e.g., “What do you put on the bread first when
making a sandwich? A. Jelly B. Peanut Butter”). When people are not sure
of a response, they are more likely to choose the first response presented
Dunaetz 21
that seems reasonable, a phenomenon known as the primacy effect
(Krosnick & Alwin, 1987). To prevent this from biasing the responses, most
electronic survey applications have an option to randomize (or shuffle) the
order of the multiple-choice responses so that each response has an equal
likelihood of being the first on each participant’s survey.
Pretesting. Before being distributed to potential participants, the
survey should be proofread and evaluated by a trustworthy critic (often
one’s spouse). Because so many errors can occur in survey design and
because what is clear to us is often not clear to others, feedback should be
sought through pretesting to make the survey as clear and unambiguous
as possible.
In order to better understand how God works in people’s lives and how
we can more effectively carry out Christ’s Great Commission, survey
research can provide the data we need to test hypotheses and discover
what people do, believe, feel, and experience. Creating a good survey can
be difficult and time-consuming, but the information gained can help us
better love others, both those who attend church now and those who will
attend in the future.
References
Batchelor, J. H., & Miao, C. (2016). Extreme response style: A meta-analysis.
Journal of Organizational Psychology, 16(2), 51-62.
Bordens, K. S., & Abbott, B. B. (2011). Research design and methods: A process
approach (8th ed.). Mc Graw-Hill.
Cameron, H., & Duce, C. (2013). Researching practice in mission and ministry: A
companion. SCM Press.
Church Relevance. (2013). 50 examples of church mission statements.
https://www.churchrelevance.com/2013/03/28/50-examples-of-church-mission-
statements/
Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature
reviews. Knowledge in Society, 1(1), 104-126.
Crano, W. D., Brewer, M. B., & Lac, A. (2015). Principles and methods of social
research (3rd ed.). Routledge.
Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design:
Choosing among five approaches. Sage
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and
mixed-mode surveys: The tailored design method. John Wiley & Sons.
Dunaetz, D. R. (2020). Research methods and survey applications: Outlines and
activities from a Christian perspective (3rd ed.). Martel Press.
Dunaetz, D. R. (2021). The importance of hypotheses in church-based research.
Great Commission Research Journal, 13(1), 5-18.
Dunaetz, D. R. (2022a). Church-based research: Using theories, concepts, and
operationalizations. Great Commission Research Journal, 14(2), 5-17.
22 Great Commission Research Journal 15(2)
Dunaetz, D. R. (2022b). Writing literature reviews in church-based research. Great
Commission Research Journal, 14(1), 5-18.
Dunaetz, D. R. (2023). Metrics and measurements in church-based research.
Great Commission Research Journal, 15(1), 5-18.
Dunaetz, D. R., & Bocock, J. (2020). Ministry involvement of church staff and
volunteers: The role of organizational commitment and work
engagement. Theology of Leadership Journal, 3(1), 52-67.
Ekinci, Y. (2015). Designing research questionnaires for business and
management students. Sage.
Faden, R. R., & Beauchamp, T. L. (1986). A history and theory of informed
consent. Oxford University Press.
Fowler, F. J. (2013). Survey research methods. Sage
Fuller, C. (1974). Effect of anonymity on return rate and response bias in a mail
survey. Journal of Applied Psychology, 59(3), 292-296.
Furnham, A., & Boo, H. C. (2011). A literature review of the anchoring effect The
Journal of Socio-Economics, 40 (1), 35-42.
Hill, P. C., & Hood, R. W. (1999). Measures of religiosity. Religious Education Press.
Jacowitz, K. E., & Kahneman, D. (1995). Measures of anchoring in estimation
tasks. Personality and Social Psychology Bulletin, 21(11), 1161-1166.
Jamieson, S. (2004). Likert scales: How to (ab)use them? Medical Education,
38(12), 1217-1218.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of
questionable research practices with incentives for truth telling.
Psychological Science, 23(5), 524-532.
Kalton, G., & Schuman, H. (1982). The effect of the question on survey responses:
A review. Journal of the Royal Statistical Society Series A: Statistics in
Society, 145(1), 42-57.
Kerr, N. L. (1998). Harking: Hypothesizing after the results are known. Personality
and Social Psychology Review 2(3), 196-217.
Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of
response-order effects in survey measurement. Public Opinion
Quarterly, 51(2), 201-219.
Likert, R. (1932). A technique for the measurement of attitudes. Archives of
Psychology, 22, 5-55.
Lowman, R. L. (2006). The ethical practice of psychology in organizations.
American Psychological Association.
Marcum, J. P. (1999). Measuring church attendance: A further look. Review of
Religious Research, 41(1), 122-130.
Margolis, M. F. (2022). Born again but not evangelical? How the (double-barreled)
questions you ask affect the answers you get. Public Opinion Quarterly,
86(3), 621-642.
Menold, N. (2020). Double barreled questions: An analysis of the similarity of
elements and effects on measurement quality. Journal of Official
Statistics, 36(4), 855-886.
Dunaetz 23
Moore, D. W. (2002). Measuring new types of question-order effects: Additive and
subtractive. The Public Opinion Quarterly, 66(1), 80-91.
Morling, B. (2021). Research methods in psychology (4th ed.). Norton.
Nehrbass, K. (2022). Not just the sinner's prayer: People's experiences with
"stranger-evangelism". Great Commission Research Journal, 14(2), 45-64.
Nehrbass, K., Dunaetz, D. R., & Jow, J. (2023). Missionary education and effective
Great Commission participation: Does missionary education work?
Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating
participant demand characteristics. The Journal of General Psychology
135(2), 151-166.
Nooney, J., & Woodrum, E. (2002). Religious coping and church-based social
support as predictors of mental health outcomes: Testing a conceptual
model. Journal for the Scientific Study of Religion, 41(2), 359-368.
Patton, M. Q. (2014). Qualitative research & evaluation methods: Integrating
theory and practice (4th, Ed.). Sage Publications.
Pew Research Center. (2021). Internet/broadband fact sheet.
https://www.pewresearch.org/internet/fact-sheet/internet-broadband/
Randall, D. M., & Fernandes, M. F. (1991). The social desirability response bias in
ethics research. Journal of Business Ethics, 10, 805-817.
Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in
quantitative methods for literature reviews. Annual Review of
Psychology, 52(1), 59-82.
Rossi, M., & Scappini, E. (2014). Church attendance, problems of measurement,
and interpreting indicators: A study of religious practice in the United
States, 1975–2010. Journal for the Scientific Study of Religion, 53(2),
249-267.
Salkind, N. J. (2017). Exploring research (9th ed.). Pearson.
Smith, T. W. (1998). A review of church attendance measures. American
Sociological Review, 63(1), 131-136.
Stanovich, K. E. (2018). Miserliness in human cognition: The interaction of
detection, override and mindware. Thinking & Reasoning, 24(4), 423-444.
Thigpen, L. L. (2023). How a novel research framework resulted in fruitful
evangelism and discovery: Introducing ethnoscopy and spiritual
patronage. Great Commission Research Journal, 15(2), 25-43.
Torraco, R. J. (2005). Writing integrative literature reviews: Guidelines and
examples. Human Resource Development Review, 4(3), 356-367.
Warriner, G. K. (1991). Accuracy of self-reports to the burdensome question:
Survey response and nonresponse error trade-offs: Accuracy of self-
reports. Quality and Quantity, 25(3), 253-269.
Wright, L. S., Frost, C. J., & Wisecarver, S. J. (1993). Church attendance,
meaningfulness of religion, and depressive symptomatology among
adolescents. Journal of Youth and Adolescence, 22, 559-568.
Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of
age, experience and question complexity on web survey response times.
24 Great Commission Research Journal 15(2)
Applied Cognitive Psychology: The Official Journal of the Society for
Applied Research in Memory and Cognition, 22(1), 51-68.
