Applying an Expanded Set of Cognitive Design Principles to Formatting
the Kidney Early Evaluation Program (KEEP) Longitudinal Survey
José Luis Calderón, MD,1,2Erik Fleming, MPH,2Monica R. Gannon, BA,3Shu-Cheng Chen, MS,4
Joseph A. Vassalotti, MD,3,5and Keith C. Norris, MD2,6
Background: The National Kidney Foundation Kidney Early Evaluation Program (KEEP) is a free
community-based health-screening program targeting populations at greatest risk of chronic kidney
disease (CKD), those with high rates of diabetes and hypertension, and a high proportion of racial/ethnic
minorities. The KEEP Longitudinal Survey will adopt methods similar to those used in KEEP to gather
follow-up data to measure CKD-related heath status and gauge program effectiveness for repeated
KEEP participants with evidence of CKD stages 3 to 5. KEEP has defined objectives to enhance
follow-up survey response rates and target vulnerable populations who bear the greatest CKD
Methods: The KEEP Follow-up Form was assessed for adherence to 6 cognitive design principles
(simplicity, consistency, organization, natural order, clarity, and attractiveness) considered to summate
the techniques guiding good survey development and for the additional cognitive design principles of
readability and variation of readability across survey items.
Results: The KEEP Follow-up Form was found to include violations of each cognitive design principle
and readability principle, possibly contributing to item nonresponse and low follow-up rates in KEEP. It
was revised according to empirically substantiated formatting techniques guided by these principles and
found during qualitative assessment to be more user friendly, simpler, better organized, more attractive,
and easier to read. Subsequent development of the KEEP Longitudinal Survey form also was guided by
Conclusion: To ensure ease of use by populations with limited literacy skills, poor health literacy, and
limited survey literacy, survey researchers must apply cognitive design principles to survey develop-
ment to improve participation and response rates.
Am J Kidney Dis 51(S2):S83-S92. © 2008 by the National Kidney Foundation, Inc.
INDEX WORDS: Chronic kidney disease; cognitive design principles; readability; survey format; survey
usually is based on investigator preference and
experience. The formatting goal is to produce sur-
veys that are easy to comprehend, navigate, and
respond to regardless of whether self-administered
or administered orally. Thus, formatting choices
must take into consideration social characteristics
of the population to be studied, including educa-
tional attainment, literacy skills, disease burden,
and cognitive functioning.1On the receiving end,
respondents read or hear survey questions and
process the content in the context of memory and
experience to formulate responses. Therefore, re-
ealth-related surveys vary in formatting and
ease of use. Choice of formatting technique
sponding to surveys is a cognitive exercise, one
that is apt to be performed better by respondents
who comprehend the purpose of the survey and the
items, are healthy, and have higher educational
attainment and broader literacy skills.2This is par-
ticularly important for patients with chronic kidney
disease (CKD) or one or both of its 2 main risk
factors, diabetes and hypertension. These patients
are at risk of experiencing cognitive decline and
decreased literacy skills, making reading health
information and responding to surveys challeng-
Formatting techniques used in developing
health-related surveys are well documented and
From the1College of Pharmacy, Health Professions Divi-
sion, Nova Southeastern University, Fort Lauderdale, FL;
2Charles R. Drew University of Medicine and Science, Los
Angeles, CA;3National Kidney Foundation, New York, NY;
4Chronic Disease Research Group, Minneapolis Medical
Research Foundation, Minneapolis, MN;
Medicine, Division of Nephrology, Mount Sinai School of
Medicine, New York, NY; and6David Geffen School of Medi-
cine, University of California, Los Angeles (UCLA), Los
Received November 21, 2007. Accepted in revised form
January 14, 2008.
Address correspondence to José Luis Calderón, MD,
College of Pharmacy, Health Professions Division, Nova
Southeastern University, 3301 College Ave, Fort Lauder-
dale, FL 33314-7796. E-mail: email@example.com
© 2008 by the National Kidney Foundation, Inc.
American Journal of Kidney Diseases, Vol 51, No 4, Suppl 2 (April), 2008: pp S83-S92 S83
were empirically tested by social scientists. Six
cognitive design principles were proposed as
ment of user-friendly surveys: simplicity, consis-
tency, organization, natural order, clarity, and
attractiveness.5The Charles Drew University
Biomedical Research Center, Los Angeles, CA,
expanded on these by adding 2 components of
survey readability: readability of individual sur-
vey items and variation in the readability of
items across a survey.Assessing readability is an
important step in adapting and designing health-
related surveys for use with vulnerable popula-
tions who tend to have limited literacy skills.6
Readability refers to the semantic and syntactic
attributes of text. It determines the relative utility
of text for persons with varying degrees of read-
ing skill.7Readability of text can be estimated by
using one of many readability formulas based on
the number of syllables per word and number of
words per sentence to estimate the reading skill
level needed to decipher and comprehend the
were found to be the best predictors of text
readability. More polysyllabic words and longer
sentences are more difficult to read. Desired
readability for persons with limited literacy skills
is a score of fifth grade level or less, measured
using the Flesch-Kincaid Grade Level (F-K) for-
mula, or a score of 80 or higher measured using
the Flesch Reading Ease (FRE) formula.
The cognitive design principles of readability
of items and variation of readability across items
are of particular importance when developing
able populations are more likely to have limited
literacy skills and limited health literacy, particu-
larly the elderly and racial/ethnic minorities,
who also experience disparities in chronic dis-
ease prevalence.9-11They bear the largest burden
of chronic disease compared with the general
population. Therefore, surveys designed to mea-
sure behavior and health status over time must
take into account limitations in cognition inher-
ent to long-standing chronic disease and aging
that influence literacy skills. CKD is especially
pertinent in this regard because its 2 main risk
factors, diabetes and hypertension, are them-
selves chronic diseases that are pandemic and
contribute to overall cognitive decrease and de-
clining literacy skills.
a National Kidney Foundation program, is a free
community-based health screening program en-
rolling individuals 18 years and older with diabe-
tes, hypertension, or a family history of kidney
disease, diabetes, or hypertension. All program
participants are volunteers. Since the program
was launched in August 2000, more than 90,000
participants were screened by 47 National Kid-
ney Foundation affiliates in 49 states and the
is used to assess how participants experienced
the screening, what they learned about their
health, whether they followed up with a physi-
cian visit, and, if so, what health issues were
discussed. The KEEP Longitudinal Study will
adopt similar survey methods to gather fol-
low-up data to measure CKD-related health sta-
tus and gauge program effectiveness for repeated
KEEP participants with evidence of CKD stages
3 to 5.
ting health-related surveys has 3 objectives: (1)
diminish common navigation errors, (2) mini-
mize the administrative burden and cognitive
demands on respondents, and (3) increase the
ease of negotiating and responding to a sur-
ciples to developing health-related surveys was
shown to diminish item nonresponse, these prin-
ciples were used to assess the KEEP Follow-up
Form and begin development of the KEEP Lon-
gitudinal Survey (KEEP-LS). Applying cogni-
is an iterative and ongoing process. The present
and likely will be revised as we gain experience
in its use and data-collection capability in the
context of vulnerable populations.
KEEP targets communities at high risk of
CKD and its risk factors. It has detected greater
rates of CKD risk factors in targeted communi-
ties than in the general population, establishing
this approach as justified and productive.15-17
KEEP Longitudinal Study expands the number
of study communities and adds an educational
component for providers and program partici-
pants.18KEEP Longitudinal Study will identify
individuals from previous KEEP programs with
evidence of CKD stages 3 to 5 and enroll them
for long-term participation in a study designed to
Calderón et alS84
evaluate the effectiveness of educational pro-
grams in improving the process of care and
clinical outcomes. Educational programs will be
ers. Survey use is important to this study to
accrue baseline and follow-up data. The purpose
of this report is to describe the methods used to
assess the format design of the KEEP Follow-up
Form, measured by using 8 cognitive design
ment of the KEEP Follow-up Form; and qualita-
tively validate modifications made to the KEEP
Follow-up Form in developing the KEEP-LS.
We applied these steps to the original KEEP Follow-up
Form (version 1 [v1]) to develop the KEEP-LS: (1) assess-
ment of how well v1 adhered to cognitive design principles;
(2) assessment of v1 readability at the Charles R. Drew
Biomedical Research Center; (3) development of the KEEP
Follow-up Form v2 based on steps 1 and 2; (4) conducting
cognitive interviews to comparatively assess comprehen-
sion, perceived ease of use, and cultural appropriateness of
KEEP Follow-up Form v1 and v2; (5) KEEP Follow-up
Committee review and revision of v2 to develop KEEP
Follow-up Form v3 based on information gained from
reports of the Drew Biomedical Research Center; (6) inde-
pendent assessment of the KEEP Follow-up Form v3 for
adherence to the expanded set of cognitive design principles;
and (7) development of the KEEP-LS (v1 to v3) at the
These steps emphasize the iterative nature of methods
used to develop the present field version of the KEEP-LS.
Moreover, the original version of the KEEPFollow-up Form
(v1) underwent 2 independent reviews by the Drew Re-
search Centers in Minority Institutions and the Drew Center
ment of overall content, language, readability, and format.
Based on consensus between reviewers, revisions were
made to develop the KEEPFollow-up Form v2 and v3.
The field version of the KEEP-LS also was culturally
adapted into Spanish by using rigorous criteria. Discussion
of this aspect of the survey development is beyond the scope
of this report. However, Fig 1 shows methods for the
linguistic and cultural adaptation (including language) of
surveys, with a focus on their application to formatting, that
were developed at the Drew Research Centers in Minority
The KEEP Follow-up Form v1 has 14 numbered items
consisting of 18 closed-ended and 1 open-ended question.
We assessed KEEP Follow-up Form v1 according to the 7
cognitive design principles listed in Table 1. The F-K and
Charles Drew University protocol for the linguistic and cultural adaptation of surveys. RGL, reading grade
Cognitive Design Principles and KEEP SurveyS85
FRE readability formulas were used to estimate the readabil-
ity of items in KEEP Follow-up Form v1 and items devel-
oped for the KEEP-LS. The F-K formula rates text on a US
grade-school level such that the average eighth grader would
be able to read a document that scores 8.0. Scores generated
by the F-K formula highly correlated with scores from other
commonly used readability formulas.8The FRE formula
rates text on a 100-point scale; the higher the score, the
easier the document is to read. Both formulas generate
scores based on the average number of syllables per word
and number of words per sentence. Correspondence between
the scores for these 2 methods and the reading difficulty
rating for the scores are listed in Table 2.
Because the readability estimate for a passage is equiva-
lent to the average of the readability of its component
sentences, we used the F-K and FRE formulas to assess
the readability of single items, as well as the survey as a
whole. We selected these formulas because they are
available in Microsoft Word (Microsoft Corp, Redmond,
WA) and therefore are readily available to nearly all
investigators interested in assessing text and survey read-
ability. Moreover, use of software decreases the amount
of work required to produce readability estimates, elimi-
nates human error inherent in manual calculation, and
requires little training.19
Cognitive (intensive) interviews were conducted with 8
participants using KEEPFollow-up Form v3: 5 Hispanics (3
men, 2 women) aged 43 to 67 years and 3AfricanAmericans
(1 man, 2 women) aged 50 to 82 years. Four participants had
a high school education or equivalency, 2 had some college,
and 2 had college degrees (bothAfricanAmerican). None of
the women reported having diabetes, hypertension, or kid-
ney disease. One Hispanic man had uncontrolled hyperten-
sion despite medication, 1 had diabetes treated by diet, and 1
had diabetes and hypertension treated with insulin and
antihypertensive medication. All Hispanic men had moder-
ate to severe central obesity and admitted to being over-
weight. OneAfrican-American man had diabetes and kidney
disease.All Hispanic participants were fully bilingual.
A cognitive interview script was constructed and used by
2 ethnically matched interviewers (Table 3). Interviews
lasted 40 to 60 minutes. The first set of items in the script
queried perceptions about the original KEEP Follow-up
Form. The last items queried perceptions about the first
revision (v2) of the KEEP Follow-up Form. Participants
were asked to compare them for comprehension, ease of
reading, and preferred format. Comments, opinions, and
perception of the KEEPFollow-up Form (v2) were clustered
and reported as an item-by-item synopsis to the KEEP
For purposes of brevity, we report only results
of our assessment of adherence to cognitive
ability, and results of cognitive interviews for the
first page of the KEEP Follow-up Form v1 (Fig 2)
and KEEP-LS v3 (Fig 3). However, KEEP Fol-
low-up Form v3 and KEEP-LS v3 appear in their
entirety as online supplementary materials avail-
able at www.ajkd.org to allow readers to better
understand how the iterative nature of our meth-
ods resulted in the survey’s evolution and as a
tool for further study.
Overall, KEEP Follow-up Form v1 had defi-
ciencies in each of the 7 cognitive design prin-
ciple categories (Table 4).As part of the iterative
process in the development of the KEEP-LS,
adherence to these cognitive design principles
was assessed in each step of survey develop-
ment. For example, the KEEP Follow-up Com-
mittee revised v2 and developed v3 based on
Table 2. Reading Difficulty Rating of Flesch Reading
Ease Scores and Flesh-Kincaid Grade Level Scores
Table 1. Description of Cognitive Design Principles
Elimination of graphical complexities, such as grid lines and irrelevant information
Ensuring response tasks are consistent for similar types of questions
Adhering to proximity compatibility principle; the degree to which different displays of information are
relevant to common mental tasks should guide physical proximity of displays
Natural reading flow from left to right, top to bottom
Enhancing navigation and diminishing cognitive demand, such as eliminating matrices
User-friendly design to motivate completion, eliminate clutter, and highlight important points
Easy-to-read instructions, transition statements, and questions; elimination of variation in readability
from one item to the next; avoiding use of technical terms
4. Natural order
Calderón et alS86
information gained from the report of the Drew
Research Centers in Minority Institutions re-
search group. KEEP Follow-up Form v3 then
was independently assessed for adherence to
cognitive design principles. This showed that v3
formatting was still difficult to negotiate because
it used too many text boxes, making it visually
challenging and increasing cognitive demand. It
ing to cognitive demand when responding (see
supplementary materials). In addition, assess-
ment showed that readability of v3 item 13 was
reading grade level 12, measured by means of
the F-K method; this is considered difficult to
read. Thus, this iterative process represents a
quality control measure that ensures an end prod-
uct that will have the greatestutility for gathering
valid health-related information from popula-
tions with cognitive decrease and limited literacy
skills and for diminishing item and survey nonre-
The readability of many KEEP Follow-up
Form v1 items was at or less than the desired
item readability of 5 or less. However, the read-
ability of many items was considered difficult
marked variation. Figure 4 shows the variation in
readability of items on KEEP Follow-up Form
(v1) page 1 compared with KEEP-LS (v3) page
1. Items with difficult readability were simpli-
fied by using the for or nor but and yet so method
(which reduces long sentences to short simple
sentences) developed at the Drew Center for
Health Services Research.20
The cognitive interviews validated the ini-
tial assessment of KEEP Follow-up Form v1.
In summary, v1 did not offer skip patterns and
items were not ordered in a way that would
help respondents answer questions in logical
succession and avoid items that may not be
relevant to them. There was consensus agree-
ment across the 2 participant groups that hav-
ing to negotiate items not relevant to them “is a
waste of time,” “can be frustrating,” and may
cause participants to stop answering survey
questions. Other comments indicated that items
and response options were crowded, number-
ing patterns for questions and response options
were confusing, many questions were too long
and considered likely to be hard to read for the
average person, and the use of technical terms
was frustrating. There also was consensus
agreement that this version of the survey was
not easy to use. Table 5 lists additional com-
Table 3. KEEP Follow-up Form Cognitive Interview Script
1. What do you think this question is asking?
2. What does this question mean to you?
3. Are the words too technical?(a) Yes . . . . . How?
(a) Yes . . . . . How? (b) No 4. Should the wording be changed?
5. How would you ask this question?
6. Do you speak Spanish?
7. How would you ask this question in Spanish?
8. Please read the answer choices after this question. Are they understandable?(a) Yes
(b) No . . . . . How?
Please look at the form itself. Look at how the questions are written on the pages.
9. Is there too much information on the page?
10. Is the writing easy or hard to read?
11. Is there enough space between questions and answers?
12. If you had to change the form, what would you change?
Interviewer: Show the participant the KEEP Follow-up Form v1 and v2.
13. Please compare these 2 surveys. Which of the 2 forms looks easier to read? Why?
14. Please compare these 2 surveys. Which of the 2 forms is easier to read? Why?
Do you have anything else you’d like to say about the KEEP surveys?
(a) Yes . . . . How? (b) No
(a) Easy (b) Hard . . . . How?
Abbreviations: KEEP, Kidney Early Evaluation Program.
Cognitive Design Principles and KEEP SurveyS87
mentary and verbatim responses from cogni-
tive interview participants for the first 4 items
of KEEP Follow-up Form v1 that are represen-
tative of comments made for other items. When
asked to compare KEEP Follow-up Form v1
with the KEEP Follow-up Form v2, there was
consensus agreement among all participants
that the KEEP Follow-up Form v2 was easier
to use, better organized, simpler, easier to read,
and more attractive (Fig 3).
Applying expanded cognitive design prin-
ciples used at the Charles Drew University Bio-
medical Research Center resulted in the develop-
ment of KEEP Follow-up Form v2, which was
ment. Skip patterns were added with simple
graphics to help respondents navigate the survey.
However, skip patterns were kept to a minimum
page 1. The Flesch-Kincaid Grade Level (F-K) formula rates text on a US grade-school level such that the average eighth
grader would be able to read a document that scores 8.0. The Flesch Reading Ease (FRE) formula rates text on a 100-point
scale; the higher the score, the easier the document is to read.
Formatting and readability assessment of Kidney Early Evaluation Program (KEEP) Follow-up Form version 1,
Calderón et al S88
and used only twice in the KEEP Follow-up
Form v2 (Fig 3) because they have the poten-
tial to add to cognitive demand. The KEEP
Follow-up Form v2 was reviewed and modified
by the KEEP Follow-up Committee. The result-
ing version (v3) served as the basis for develop-
ing the KEEP-LS. The KEEP-LS has 21 closed-
the intent of the KEEP Follow-up Form to gain
information about participant perceptions of the
program and its impact on their health care–
seeking behavior. Importantly, it also preserved
the intent of the follow-up form to gain informa-
tion about the process of care as it relates to CKD
and CKD risk factor screening and treatment.
An expanded set of cognitive design prin-
ciples that includes 2 domains of readability is an
important contribution to the survey methods
1. Was the KEEP health screening helpful?
(a)__Very helpful (b) __A little helpful (c) __Not helpful
. How satisfied
were you with the KEEP Health Screening?
)__Very Satisfied (b) __A little Satisfied (c) __Not Satisfied
ree ing change how you thinkn3. Did the KEEP health sc
about your health?
(c) __Not at all
. Did you learn
(a)__Yes, a lot (b) __Yes, a little
about your health from the screening?
(b) __Yes, I learned a little (c) __I didn’t learn much (a)__Yes, I learned a lot
5. What did you learn about your health from the KEEP screening?
I lease C
(A)__No health problems
(B)__ No NEW health problems
(C)__ High Blood Pressure
(B)__I have Diabetes
(C)__I have High Cholesterol
(D)__I had a Urine Infection
(E)__I have Anemia
(F)__I have a calcium/phosphorus problem (parathyroid problem)
(G)__I have Kidney Problems
[Please CIRCLE which Kidney Problems]
(2). Protein in my urine
(3). Blood in my urine
(4). Chronic kidney disease
rned that I had: [Plea IRCLE all that apply]
(high sugar in the blood)
(high fats/lipids in the blood)
(kidney or bladder infection)
(low blood count)
6. Did you see a doctor since your KEEP Health Scree
[Go to next page]
( ). Kidne stones
Please Go to PART C
KEEP Follow-up Survey v2, Part A, page 1.
Cognitive Design Principles and KEEP SurveyS89
literature. Applying cognitive design principles
in formatting the KEEP-LS has the potential to
improve item response and diminish survey non-
response, which has been a challenge to the
program. Importantly, the iterative process used
in developing the KEEP-LS is a quality-control
measure that ensures that investigators adhere to
cognitive design principles in formatting health-
related surveys. For example, in response to the
violations of each of the 7 cognitive design
principles in the KEEP Follow-up Form, the
KEEP-LS was formatted into 4 main categories
(A, KEEPScreening; B, Doctor’sVisit; C, Medi-
cines and Care; and D, Follow-up) that likely
will promote user friendliness and diminish cog-
nitive demand (see supplementary materials). In
addition, the difficult-to-read item 13 on KEEP
Follow-up Form v3 was replaced with 3 short
ability of reading grade level 4 and serve to
introduce the last 4 items in the KEEP-LS. This
is an innovative approach that not only prepares
participants to respond to the last 4 items of the
survey, but also may contribute to enhancing
their CKD health literacy. (Compare Follow-up
Form v3 item 13 with sentences introducing
items 19 to 22 of KEEP-LS v3 in the supplemen-
tary materials online.)
Table 4. Assessment of Adherence to Cognitive Design Principles: Kidney Early Evaluation Program Follow-up
Form Version 1, Page 1
1. Simplicity Four items are numbered, but 8 questions are asked.
Use of ambiguous words: “feel” (item 1), “glad” (item 2).
Visually distracting, density of information.
Response task inconsistent.
Numbering and lettering inconsistent across items.
Format complicated by questions listed as lettered items after numbered items.
Concepts not grouped (item 1).
Response option on right for item 1, on left for items 2, 3.
Response option orientation inconsistent (1 vertical column, item 3; 2 vertical columns, item 4).
Lack of left-to-right orientation, item 4.
High cognitive demand to negotiate items on page because of lack of simplicity, consistency,
Questions and response options cluttered.
Numbering of items confusing in using lettered items within a numbered item.
Information density high.
Bolded response options items 1 and 2. but not 3 and 4.
Request for personal information first (social security number).
Readability is acceptable for 5/8 items, but in the “difficult” range for 3/8 items (2, 2b, 3).
One item (item 2) reads at the college graduate level (grade level 20).
Wide variation in readability across items.
Use of technical terms (glomerulonephritis).
4. Natural design
Page 1 Items
KEEP F/U Form
F-K Grade Level Score
ity of Kidney Early Evaluation Pro-
gram (KEEP) Follow-up Form
(KEEP-F/U) version 1 (page 1,
items 1a, 1b, 1c, 1d, 2, 2b, 3, and
4) and KEEP-Longitudinal Survey
(KEEP-LS) version 3 (page 1,
items 1 to 6). Abbreviation: F-K,
Flesch-Kincaid Grade Level for-
Variation in readabil-
Calderón et al S90
Surveys designed using these principles may
mitigate category fallacy and inaccurate re-
This is especially important because the KEEP
lations, such as racial/ethnic minorities and the
elderly, who are at greatest risk of CKD and may
have limited literacy skills and limited survey
literacy (limited experience in negotiating and
The purpose of this report was to convey how
best to format surveys for vulnerable populations
by using cognitive design principles. One limita-
ties of the KEEP-LS were not tested. However,
we currently are collecting data to test the instru-
ment’s construct validity and internal consis-
tency reliability. Of interest will be the tally of
missing responses from vulnerable populations.
To date, most KEEP participants were educated
and employed and tended to have health insur-
ance; however, follow-up response rates were
less than expected. We hypothesize that it will be
more effective at accruing robust data, measured
by diminished item and survey nonresponse.
Moreover, the Spanish version of the KEEP-LS
is undergoing further qualitative evaluation by
using focused group discussions with Spanish-
only speakers before it is field tested in Latino
By applying cognitive design principles to
formatting health-related surveys, researchers
may increase the likelihood that participants
from all walks of life with differing levels of
educational attainment, literacy skills, health
literacy, and survey literacy will be able to
more easily navigate the surveys. This is cru-
cial for enhancing our understanding of how
better to improve preventative care and pro-
mote compliance with care in populations at
high risk of CKD.
The authors thank Edward Constantini, MA, Shane Nyg-
aard, BA, and Nan Booth, MSW, MPH, of the Chronic
Disease Research Group for figure preparation, manuscript
preparation, and manuscript editing, respectively.
Support: The Kidney Early Evaluation Program is spon-
sored by the National Kidney Foundation Inc and supported
by Amgen, Abbott, Genzyme, Ortho Biotech Products LP,
and Novartis, with additional support provided by Siemens
Medical Solutions Diagnostics, Lifescan, Suplena, and
OceanSpray Cranberries. Additional support was provided
by National Institutes of Health grants RR019234 and
MD00148 (JC, KN); RR03026, RR011145, P30AG21684,
and RR014616 (JC, EF, KN); Agency for Healthcare Re-
search and Quality grant 1R24-HS014022-01A1 (JC), and
CMS grant 1H0CMS300041 (JC).
Financial Disclosure: Dr Vassalotti reports having re-
ceived grant support from the Centers for Disease Control
and Prevention. The authors have no conflicts of interest
with its subject matter.
Table 5. Summary of Cognitive Interview Comments for KEEP Follow-up Form Version 1, Items 1 to 4
Item No. Comments
1, 2 The meaning of “feel” and “glad” in the context of a question about health was unclear to some. One
participant suggested merging items 1a and 1b into one by using “satisfied” instead of “glad.” The term
“health care provider” was defined in a variety of ways: doctor, HMO, hospital. When asked for an
alternative, “doctor” was the preferred term. The alphanumeric designations for items were confusing. Item
2 was considered difficult to comprehend by several participants. “I had to read it more than once.” “Too
long and repetitive.” “Most people may not understand this.” Questions within questions were mentioned.
“Questions should have separate numbers.” “If I answered no to 2a, why do I have to answer 2b?”
Most participants considered the question too long and not clear. “Are you asking about KEEP or a doctor?”
“Should be asked differently.” “Make question clear.” “If I don’t have any health problems, why should I
answer all these questions?” Two participants astutely suggested that they couldn’t see the point of the
question because answering would not tell you if the answer is about KEEP or the provider. The words
triglycerides, urinary tract infections, phosphorous, or parathyroid could not be defined by all. “I have no
idea.” These terms are used in other items in the KEEP Follow-up Form. Calcium was recognized as a
common word: “It’s in milk,” “For the bones.”
Participants could not define estimated glomerular filtration rate or creatinine. They were considered too
technical. Common questions were “What’s creatinine?” and “Glomerular what?” or “I have no idea what
Abbreviations: KEEP, Kidney Early Evaluation Program; HMO, health maintenance organization.
Cognitive Design Principles and KEEP Survey S91
SUPPLEMENTARY DATA Download full-text
Item S1: KEEPFollow-up Form v3.
Item S2: KEEP-LS v3.
Note: The supplementary data accompanying this article
(doi:10.1053/j.ajkd.2008.01.008) is available at www.ajkd.org.
1. Sudman S, Bradburn N, Schwarz S: Thinking About
Answers: TheApplication of Cognitive Processes to Survey
Methodology. San Francisco, CA, Jossey-Bass, 1995
2. Aday L: Designing and Conducting Health Surveys.
San Francisco, CA, Jossey-Bass, 1996
3. Calderon JL, Zadshir A, Norris K: Structure and con-
tent of chronic kidney disease information on the World
Wide Web: Barriers to public understanding of a pandemic.
Nephrol News Issues 18:76, 78-79, 81-84, 2004
4. Calderon JL, Zadshir A, Norris K: A survey of kidney
disease and risk-factor information on the World Wide Web.
MedGenMed 6:3, 2004
5. Mullin PA, Lohr KN, Bresnahan BW, McNulty P:
Applying cognitive design principles to formatting HRQOL
instruments. Qual Life Res 9:13-27, 2000
6. Calderon JL, Morales LS, Liu H, Hays RD: Variation
in the readability of items within surveys. Am J Med Qual
7. Chall J: The beginning years, in Zakaluk B, Samuels S
(eds): Readability: Its Past, Present, and Future. Newark,
DE, Sage Publications, 1988, pp 2-13
8. Meade CD, Smith CF: Readability formulas: Cautions
and criteria. Patient Educ Counsel 17:153-158, 1991
9. Kirsh I, Jungeblut A, Jenkins L, Kolstad A: Adult
Literacy in America: A First Look at the Results of the
National Literacy Survey. Washington, DC, US Department
of Health, Education and Welfare, 2007
10. Williams MV, Parker RM, Baker DW, et al: Inad-
equate functional health literacy among patients at two
public hospitals. JAMA274:1677-1682, 1995
11. Murphy T, Davis T, Jackson R, Decker B, Long S:
Effects of literacy on health care of the aged: Implications
for health care professionals. Educ Gerontol 19:311-316,
12. Wright P, Barnard P:Asking multiple questions about
several items: The design of matrix structures on application
forms.Appl Ergon 9:7-14, 1978
13. Lohr KN,Aaronson NK,Alonso J, et al: Evaluating
quality-of-life and health status instruments: Develop-
ment of scientific review criteria. Clin Ther 18:979-992,
14. Dillman D, Sinclair M, Clark J: Effects of question-
naire length, respondent-friendly design, and a difficult
question on response for occupant-addressed Census mail
surveys. Public Opin Q 57:289-304, 1993
15. McGill JB, Brown WW, Chen SC, Collins AJ, Gan-
non MR: Kidney Early Evaluation Program (KEEP). Find-
ings from a community screening program. Diabetes Educ
16. Brown WW, Collins A, Chen SC, et al: Identification
of persons at high risk for kidney disease via targeted
screening: The NKF Kidney Early Evaluation Program.
Kidney Int Suppl 83:S50-S55, 2003
17. Brown WW, Davis J, Flack JM, Gannon MR: NKF’s
KEEP leads to [longitudinal study]. Nephrol News Issues
18. Ohmit SE, Flack JM, Peters RM, BrownWW, Grimm
R: Longitudinal study of the National Kidney Foundation’s
(NKF) Kidney Early Evaluation Program (KEEP). JAm Soc
Nephrol 14:S117-S121, 2003 (suppl 2)
19. Mailloux SL, Johnson ME, Fisher DG, Pettibone TJ:
How reliable is computerized assessment of readability?
Comput Nurs 13:221-225, 1995
20. Calderon JL, Smith S, Baker R: A simple method for
enhancing readability of patient information.Ann Behav Sci
Med Educ 14:20-24, 2007
Calderón et alS92