ArticlePDF Available

AM Last Page: Avoiding Four Visual-Design Pitfalls in Survey Development

DOI:10.1097/ACM.0b013e31826a06b2
Authors:
Anthony R Artino at George Washington University
Anthony R Artino
Hunter Gehlbach at University of California, Santa Barbara
Hunter Gehlbach
Download full-text PDF
Read full-text
Download citation
Content uploaded by Hunter Gehlbach
Author content
All content in this area was uploaded by Hunter Gehlbach on Feb 13, 2018
Content may be subject to copyright.
Academic Medicine, Vol. 87, No. 10 / October 2012
1452
AM Last Page
AM Last Page: Avoiding Four Visual-Design Pitfalls in Survey Development
Anthony R. Artino, Jr, PhD, associate professor, Preventive Medicine and Biometrics, Uniformed Services University of the Health
Sciences, and Hunter Gehlbach, PhD, associate professor, Harvard Graduate School of Education
A previous AM Last Page1 presented five common pitfalls of survey design as well as several solutions. This AM Last Page
presents four visual-design and layout pitfalls and offers solutions.
Pitfall: Explanation and Example Solution: Explanation and Example
Labeling only the end points of your response options
Labeling only the end points leaves the meaning of the
unlabeled options open to respondents’ interpretation.
Different respondents can interpret the unlabeled options
differently. This ambiguity increases measurement error.2
How interesting did you find this clinical reasoning course?
Verbally label each response option
Labeling each response option increases consistency in the
conceptual spacing between response options and increases the
likelihood that all respondents will interpret the response options
similarly. Additionally, the response options have comparable
visual weight, so the respondents’ eyes are not drawn to certain
options.
How interesting did you find this clinical reasoning course?
Labeling response options with both numbers and
verbal labels
Because of the additional information respondents must
process, providing both numbers and verbal labels extends
response time.3 The implied meaning of negative numbers can
be particularly confusing and may introduce additional error.
For example, in the item below, learning “a little bit” seems
incongruous with learning the amount of “–1.”
How much did you learn in today’s workshop?
Use only verbal labels
In general, use only verbal labels for each response option. Doing
so will reduce the cognitive effort required of your respondents
and will likely reduce measurement error.2
How much did you learn in today’s workshop?
Unequally spacing your response options
The visual spacing between options can attract respondents to
certain options over others, which in turn might cause them to
select these options more frequently.4 In addition, unbalanced
spacing of the response options can shift the visual midpoint of
the scale.
How much did you learn from your peers in this course?
Maintain equal spacing between response options
Maintaining equal spacing between response options will
reinforce the notion that, conceptually, there is equal space
or “distance” between each response option. As a result, the
answers will be less biased, thereby reducing measurement error.
How much did you learn from your peers in this course?
Placing nonsubstantive response options together with
substantive response options
Placing nonsubstantive response options such as “don’t
know,” “no opinion,” or “not applicable” together with
the substantive options can shift the visual and conceptual
midpoint of the response scales, thereby skewing the results.4
How satisfied are you with the quality of the library services?
Use additional space to visually separate nonsubstantive
response options
Using additional space to visually separate nonsubstantive
response options from the substantive options will align the
visual midpoint with the conceptual midpoint, thereby reducing
measurement error.4 This recommendation is an important
exception to the guidance above about maintaining equal
spacing between response options.
How satisfied are you with the quality of the library services?
not at all
interesting
extremely
interesting
not at all
interesting
slightly
interesting
moderately
interesting
quite
interesting
extremely
interesting
–2
almost
nothing
–1
a little
bit
0
some
1
quite
a bit
2
a tremendous
amount
almost
nothing
a little
bit
some quite
a bit
a tremendous
amount
almost nothing a little bit some quite a bit a tremendous amount
almost
nothing
a little
bit
some quite
a bit
a tremendous
amount
not at all
satisfied
slightly
satisfied
moderately
satisfied
quite
satisfied
extremely
satisfied
not
applicable
not at all
satisfied
slightly
satisfied
moderately
satisfied
quite
satisfied
extremely
satisfied
not
applicable
Disclaimer:
The views expressed in this article are those of the authors and do not necessarily reflect the official policy of the U.S. Department of Defense.
References:
1. Artino AR Jr, Gehlbach H, Durning SJ. AM Last Page: Avoiding five common pitfalls of survey design. Acad Med. 2011;86:1327.
2. Krosnick JA. Survey research. Annu Rev Psychol. 1999;50:537-567. http://communication.stanford.edu/faculty/krosnick/docs/annrevsurvey.pdf. Accessed May 30, 2012.
3. Christian LM, Parsons NL, Dillman DA. Designing scalar questions for web surveys. Sociol Meth Res. 2009;37:393-425.
4. Dillman DA, Smyth JD, Christian LM. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd ed. Hoboken, NJ: John Wiley & Sons; 2009.
Author contact: anthony.artino@usuhs.edu
... We chose to focus on Likert-type rating items because they are commonly used to assess constructs in HPE. 5,18 Selection of the five design tenets was accomplished through a review of the survey design literature, including summary articles in HPE, 19,20 empirical studies, 9,21,22 and survey design textbooks. 8,10 Descriptions and examples of the five survey design tenets, framed as design pitfalls, are provided in Table 1 and Supplemental Digital Appendix 1 (at http://links.lww.com/ACADMED/A499). ...
... Specifically, we calculated the percentage of survey instruments that were newly developed by the researchers, the percentage of those that were appropriated or adapted (in part or in full) from a previously published instrument, and the percentage of instruments Likert-type rating items that ask respondents to rate their agreement (or their agreement or disagreement) with a statement do not emphasize the construct being measured and are prone to acquiescence, which is the tendency to endorse any assertion made in an item, regardless of its content. 19,20 In addition, agreement response items may encourage respondents to think through their responses less thoroughly while completing the survey. 10 Avoid agreement response options and instead use construct-specific response options that emphasize the construct of interest being measured. ...
... Doing so reduces acquiescence and keeps respondents focused on the construct in question. 8,10,20 How confident are you that you can do well in this course? ...
“The Questions Shape the Answers”: Assessing the Quality of Published Survey Instruments in Health Professions Education Research
Article
Purpose: Surveys are widely used in health professions education (HPE) research, yet little is known about the quality of the instruments employed. Poorly designed survey tools containing unclear or poorly formatted items can be difficult for respondents to interpret and answer, yielding low-quality data. This study assessed the quality of published survey instruments in HPE. Method: In 2017, the authors performed an analysis of HPE research articles published in three high-impact journals in 2013. They included articles that employed at least one self-administered survey. They designed a coding rubric addressing five violations of established best practices for survey item design and used it to collect descriptive data on the validity and reliability evidence reported and to assess the quality of available survey items. Results: Thirty-six articles met inclusion criteria and included the instrument for coding, with one article using 2 surveys, yielding 37 unique surveys. Authors reported validity and reliability evidence for 13 (35.1%) and 8 (21.6%) surveys, respectively. Results of the item-quality assessment revealed that a substantial proportion of published survey instruments violated established best practices in the design and visual layout of Likert-type rating items. Overall, 35 (94.6%) of the 37 survey instruments analyzed contained at least one violation of best practices. Conclusions: The majority of articles failed to report validity and reliability evidence, and a substantial proportion of the survey instruments violated established best practices in survey design. The authors suggest areas of future inquiry and provide several improvement recommendations for HPE researchers, reviewers, and journal editors.
... The themes were described, and questions as well as response scales were developed and discussed continuously [37][38][39]. ...
Migrant physicians’ entrance and advancement in the Swedish medical labour market: a cross-sectional study
Article
Full-text available
Background: Worldwide, physicians are migrating to new countries and want to practise their profession. However, they may experience difficulties doing so. To optimise and accelerate their entrance into and advancement within the Swedish healthcare system, there is an urgent need to explore how they are currently doing so, as their competences should be put to use without any unnecessary delay. The aim of the study was to explore how migrant physicians with a medical degree from outside EU/EEA enter and advance within the medical labour market in Sweden and to identify perceived barriers and facilitating aspects in the process. The empirical findings are discussed in light of Bourdieu's concept symbolic capital as adapted in the Swedish medical field. Methods: A cross-sectional study with a self-administrated questionnaire was disseminated. A sample of 498 migrant physicians were identified. Descriptive statistical analysis and qualitative thematic analysis were used to analyse the data. Results: The response rate was 57% (n = 283). Respondents mainly found their first positions via spontaneous job applications, during internships, while participating in an educational intervention or via personal contacts. Perceived barriers to entering and advancing within the medical field in Sweden were mainly related to having a medical degree from and/or originating from another country, which could in turn represent discrimination and/or having one's competence undervalued as a result. Facilitating aspects included having or developing contacts in Swedish healthcare and gaining proficiency or fluency in the Swedish language. Conclusions: When MPs find their first positions, the contacts they have developed appear to play a role, and when advancing in their positions, the active development of a variety of contacts seems to be beneficial. MPs experience a variety of barriers to entering and advancing within the field that could be related to discrimination. Many MPs perceived having their competences undervalued due to their origin or to being educated abroad. Based on the respondents' experiences, our interpretation is that MPs as a group are hierarchically positioned lower in the Swedish medical field than physicians trained in the country. Facilitating aspects included educational interventions, having contacts and developing language skills. For optimal entry into the labour market, it is vitally important for MPs to learn the new language and obtain a job or internship in the field as soon as possible.
... Portions of this AMEE Guide were previously published in the Journal of Graduate Medical Education and Academic Medicine and are used with the express permission of the publishers (Gehlbach et al. 2010;Artino et al. 2011;Artino & Gehlbach 2012;Rickards et al. 2012;Magee et al. 2013;Willis & Artino 2013). ...
Developing questionnaires for educational research: AMEE Guide No. 87
Article
Full-text available
Abstract In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.
... Nonetheless, there is a plethora of item-writing guidance-evidence-based, best practices-that should be used to guide the item-writing process. 1,[4][5][6][7][8] Reviewing these best practices is beyond the scope of this editorial; however, we have provided a summary of several evidence-based recommendations in T A B L E 1 to assist readers. ...
Tracing the Steps of Survey Design: A Graduate Medical Education Research Example
Article
Full-text available
  • Mar 2013
Surveys are frequently used to collect data in graduate medical education (GME) settings.1 However, if a GME survey is not rigorously designed, the quality of the results is likely to be lower than desirable. In a recent editorial we introduced a framework for developing survey instruments.1 This systematic approach is intended to improve the quality of GME surveys and increase the likelihood of collecting survey data with evidence of reliability and validity. In this article we illustrate how researchers in medical education may operationalize this framework with examples from a survey we developed during the recent integration of 2 independent internal medicine (IM) residency programs.
Entrustment in physician-patient communication: a modified Delphi study using the EPA approach
Article
Full-text available
Background Competency based curricula across the globe stress on the importance of effective physician patient communication. A variety of courses have been used to train physicians for this purpose. However, few of them link competencies with practice resulting in confusions in implementation and assessment. This issue can be resolved by treating certain specific patient communication related tasks as acts of entrustment or entrustable professional activities (EPAs). In this study, we aimed to define a competency-based framework for assessing patient physician communication using the language of EPAs. Methods A modified Delphi study was conducted in three stages. The first stage was an extensive literature review to identify and elaborate communication related tasks which could be treated as EPAs. The second stage was content validation by medical education experts for clarity and representativeness. The third stage was three iterative rounds of modified Delphi with predefined consensus levels. The McNemar test was used to check response stability in the Delphi Rounds. Results Expert consensus resulted in development of 4 specific EPAs focused on physician-patient communication with their competencies and respective assessment strategies all aiming for level 5 of unsupervised practice. These include Providing information to the patient or their family about diagnosis or prognosis; Breaking Bad news to the patient or their family; Counseling a patient regarding their disease or illness; Resolving conflicts with patients or their families. Conclusions The EPAs for Physician-patient communication are a step toward an integrative, all-inclusive competency-based assessment framework for patient-centered care. They are meant to improve the quality of physician patient interaction by standardizing communication as a decision of entrustment. The EPAs can be linked to competency frameworks around the world and provide a useful assessment framework for effective training in patient communication. They can be integrated into any post graduate curriculum and can also serve as a self-assessment tool for postgraduate training programs across the globe to improve their patient communication curricula.
Survey and Other Textual Data
Chapter
  • Nov 2019
Surveys are commonly used in healthcare simulation research and evaluation. Many surveys have been created to fit specific simulation requirements or areas of interest. However, it is important to ensure that surveys have appropriate rigour for reliable representation and meaningful interpretation of data. This chapter covers concepts important to basic survey design for qualitative research in healthcare simulation education. However, surveys can be used simultaneously to also source quantitative data. The benefits and limitations of using surveys, as well as practical and ethical considerations in the collection and use of textual data are discussed. This chapter also presents the options of exploring alternate textual data sources, such as educational materials, course guides, reflective assessments, as well as data from online/social media platforms. More comprehensive guides on how to develop, administer and analyse survey and other textual data can be sourced from the resources and references included in this and other related chapters of the text.
Transitioning From Medical Educator to Scholarship in Medical Education
Article
  • Nov 2016
Clinician educators spend the majority of their time in clinical practice, educating trainees in all types of care settings. Many are involved in formal teaching, curriculum development, and learner assessment while holding educational leadership roles as well. Finding time to engage in scholarly work that can be presented and published is an academic expectation but also a test of efficiency. Just as clinical research originates from problems related to patients, so should educational research originate from issues related to educating the next generation of doctors. Accrediting bodies challenge medical educators to be innovative while faculty already make the best use of the limited time available. One obvious solution is to turn the already existing education work into scholarly work. With forethought, planning, explicit expectations, and use of the framework laid out in this paper, clinical educators should be able to turn their everyday work and education challenges into scholarly work.
Design Scalar Questions for Web Surveys
Article
This paper explores how the visual design of scalar questions influences responses in web surveys. We present the results of five experiments embedded in two web surveys of university students. We find that consistently presenting the positive end of the scale first did not impact responses but increases response times. Displaying the categories in multiple columns influence how respondents process the scale and increase response times. Separating the midpoint, "don't know'' option, or endpoints spatially does not impact responses when the visual and conceptual midpoint align. Removing the graphical layout of the scale influences responses when lower numbers indicate more positive categories and increases response time. Finally, response times are longer for polar point scales with numeric labels, but there are no differences in responses. Overall, our results suggest that the visual design of response scales impacts measurement, but that some manipulations produce larger and more significant differences than others.
AM Last Page: Avoiding Five Common Pitfalls of Survey Design
Article
  • Oct 2011
  • May 1999
  • ANNU REV PSYCHOL
  • 537-567
  • J A Krosnick
  • Survey
  • Research
Krosnick JA. Survey research. Annu Rev Psychol. 1999;50:537-567. http://communication.stanford.edu/faculty/krosnick/docs/annrevsurvey.pdf. Accessed May 30, 2012.
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
  • Jan 2009
  • D A Dillman
  • J D Smyth
  • L M Christian
Dillman DA, Smyth JD, Christian LM. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd ed. Hoboken, NJ: John Wiley & Sons; 2009. Author contact: anthony.artino@usuhs.edu