A Case Study of Engaging Hard-to-Reach Participants in the Research Process: Community Advisors on Research Design and Strategies (CARDS)®

Abstract

Lack of diversity among study participants in clinical research limits progress in eliminating health disparities. The engagement of lay stakeholders, such as patient or community advisory boards (CABs), has the potential to increase recruitment and retention of underrepresented groups by providing a structure for gathering feedback on research plans and materials from this target population. However, many CABs intentionally recruit prominent stakeholders who are connected to or comfortable with research and academia and thus may not accurately represent the perspectives of underrepresented groups who have been labeled hard-to-reach, including racial minorities and low-income or low-literacy populations. We developed a partnership between the University of Wisconsin-Madison School of Nursing and two community centers to deliberately engage hard-to-reach people in two lay advisory groups, the Community Advisors on Research Design and Strategies (CARDS)®. Community center staff recruited the CARDS from center programs, including parenting and childcare programs, women's support groups, food pantries, and senior meal programs. The CARDS model differs from other CABs in its participants, processes, and outcomes. Since 2010, the CARDS have met monthly with nurses and other researchers, helping them understand how research processes and the language, tone, appearance, and organization of research materials can discourage people from enrolling in clinical studies. We have successfully used the CARDS model to bring hard-to-reach populations into the research process and have sustained their participation. The model represents a promising strategy for increasing the diversity of participants in clinical research. © 2016 Wiley Periodicals, Inc.

Full text

A case study of engaging hard-to-reach participants in the
research process: Community Advisors on Research Design and
Strategies (CARDS)®
Betty L. Kaiser [Administrative Program Specialist],
University of Wisconsin-Madison School of Nursing, blkaiser@wisc.edu, 701 Highland Ave., Rm
5138, Madison, WI 53705, 608-263-5292 (office), 608-263-5323 (fax)
Gay R. Thomas [Senior Administrative Program Specialist], and
University of Wisconsin-Madison School of Nursing
Barbara J. Bowers [Professor]
University of Wisconsin-Madison School of Nursing
Abstract
Lack of diversity among study participants in clinical research limits progress in eliminating health
disparities. The engagement of lay stakeholders, such as patient or community advisory boards
(CABs), has the potential to increase recruitment and retention of underrepresented groups by
providing a structure for gathering feedback on research plans and materials from this target
population. However, many CABs intentionally recruit prominent stakeholders who are connected
to or comfortable with research and academia and thus may not accurately represent the
perspectives of underrepresented groups who have been labeled hard-to-reach, including racial
minorities and low-income or low-literacy populations. We developed a partnership between the
University of Wisconsin-Madison School of Nursing and two community centers to deliberately
engage hard-to-reach people in two lay advisory groups, the Community Advisors on Research
Design and Strategies (CARDS)®. Community center staff recruited the CARDS® from center
programs, including parenting and childcare programs, women’s support groups, food pantries,
and senior meal programs. The CARDS® model differs from other CABs in its participants,
processes, and outcomes. Since 2010, the CARDS® have met monthly with nurses and other
researchers, helping them understand how research processes and the language, tone, appearance,
and organization of research materials can discourage people from enrolling in clinical studies. We
have successfully used the CARDS® model to bring hard-to-reach populations into the research
process and have sustained their participation. The model represents a promising strategy for
increasing the diversity of participants in clinical research.
Keywords
subject recruitment; healthcare disparities; vulnerable populations; advisory committees;
stakeholder engagement
The authors declare no conflicts of interest.
HHS Public Access
Author manuscript
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Published in final edited form as:
Res Nurs Health
. 2017 February ; 40(1): 70–79. doi:10.1002/nur.21753.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

The elimination of health disparities is a national priority (National Institute on Minority
Health and Health Disparities, n.d.; U.S. Department of Health and Human Services, n.d.).
Clinical research can play an important role in eliminating health disparities by making
novel treatments available to underserved populations and identifying the effectiveness of
particular treatments for particular populations. However, progress toward eliminating health
disparities has been hindered by lack of participant diversity in clinical research studies
(Kitterman, Cheng, Dilts, & Orwoll, 2011; Schroen et al., 2010). Researchers often do not
reach recruitment goals for hard-to-reach participants such as racial and ethnic minorities
and people with low income. Reasons for low recruitment include mistrust of research,
perceived risks, and lack of culturally appropriate information about opportunities to
participate in research (Ford et al., 2008), as well as burdens such as time commitment and
lengthy questionnaires (Paskett et al., 2008).
A recommended strategy for overcoming barriers to recruitment is patient and community
stakeholder involvement in the design and conduct of clinical trials (Institute of Medicine,
2011). One popular form of stakeholder involvement is community advisory boards (CABs).
CABs serve several functions, depending on the project mission and requirements. They
provide input into research agendas; serve as gatekeepers for researcher entry into
communities; formally approve projects; and influence the design, conduct, and
implementation of studies (Fernandez-Pena et al., 2008; Silvestre, Quinn, & Rinaldo, 2010;
Strauss et al., 2001).
Recommendations for CAB membership often emphasize inclusion of influential
stakeholders and opinion leaders from the community (D'Alonzo, 2010). Typical CAB
members include service providers, community leaders, and representatives of local
agencies and organizations (Newman et al., 2011); in this paper, high-level stakeholders are
referred to as “prominent community representatives”. Prominent community
representatives bring valuable skills, perspectives, and resources to CABs due to their
experience with group and meeting processes, influence and connections within the
community, and decision-making power within their organizations. CABs also sometimes
include lay stakeholders who are not prominent community representatives, but can bring an
important viewpoint to a research project, such as residents of a particular neighborhood or
patients or caregivers with experience related to a particular health issue (James et al., 2011;
Pinto, Spector, Rahman, & Gastolomendo, 2013). Lay stakeholders can help researchers
improve the cultural sensitivity and appropriateness of recruitment materials and methods;
recommend study implementation strategies (Joosten et al., 2015); and design relevant,
meaningful interventions (GreenMills, Davison, Gordon, Kaigang, & Jurkowski, 2013). The
perspectives of lay stakeholders who are disconnected from academia and local power
structures may offer clinical researchers unique insights and perspectives on recruitment
barriers and strategies for improving recruitment and retention of hard-to-reach populations.
This paper describes the origins, participants, processes, and outcomes of a unique CAB
comprised solely of lay stakeholders that brings voices of rarely heard groups into the
research enterprise.
Kaiser et al. Page 2
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

Project Origins and Community-Academic Partnership
Our team at the University of Wisconsin-Madison (UW-Madison) School of Nursing
partnered with two local community centers to develop lay advisory boards of community
members who are not prominent community representatives. Throughout this paper, “we”
refers to the authors. The Lussier Community Education Center and Goodman Community
Center have served their neighborhoods for over 30 years, offering programs and services
that reflect the diversity of their communities. In 2010, we worked with staff at the
community centers to submit a proposal for a program grant to the National Institutes of
Health (NIH). We received a three-year grant to establish a sustainable infrastructure to link
community members with researchers to improve the quality of health sciences research.
The UW-Madison Health Sciences Institutional Review Board (IRB) designated our grant
activities as exempt from review. For one of our grant activities, we worked with the
community centers to develop and pilot two lay advisory boards, the Community Advisors
on Research Design and Strategies (CARDS)®. We established a CARDS® group at each
partnering community center. For the past five years, each group has met monthly with guest
researchers to provide feedback on the appeal, clarity, and accessibility of materials and
processes used in clinical research. Our NIH grant ended in 2013, and since then we have
sustained the CARDS® as a program within the Wisconsin Network for Research Support
(WINRS), a fee-for-service community and patient engagement center supported by the
School of Nursing (http://winrs.son.wisc.edu/). The UW-Madison Institute for Clinical and
Translational Research (ICTR), a Clinical and Translational Sciences Award site, provides
additional support for the program.
The CARDS® Program
The CARDS® are based in the community, so researchers often assume that CARDS® input
is relevant only for researchers interested in community-based participatory research
(CBPR) or community-engaged research (CER). In fact, since the program’s inception, the
CARDS® have worked extensively with researchers conducting clinic or hospital-based
studies. The CARDS® provide lay, patient-centered feedback on materials for any type of
human subjects research, regardless of the specific research topic, setting, methodology, or
study population. Researchers and their project staff typically attend one or two meetings for
each project that they bring to the CARDS®, although we place no limit on the number of
CARDS® meetings that a researcher may attend. The CARDS® program makes it feasible
for researchers who are not conducting CBPR or CER to get timely, meaningful lay input on
their research materials, without substantial investment of resources.
Program Participants
Members—For the CARDS®, we wanted not prominent community representatives, but
people who have been under-represented in healthcare research (Hasnain-Wynia & Beal,
2014). We intentionally sought to recruit people who are not connected with academia,
research, healthcare, or local power structures. Unlike recruitment for many CABs, we did
not establish specific inclusion criteria, such as residency in a certain neighborhood or
diagnosis of a particular health condition (Stewart et al., 2015). We asked community center
Kaiser et al. Page 3
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

staff (CARDS® liaisons) to recruit typical users of services at their centers. Liaisons shared
an informational flyer with community members who used center services such as parenting
and childcare programs, women’s support groups, job clinics, food pantries, and senior meal
programs. People who were interested followed up with the CARDS® liaison to complete a
membership application that asked applicants to briefly explain why they wanted to be part
of the CARDS®. Sometimes, people who worked or volunteered at the centers heard about
the program and completed applications. Currently, 15 CARDS® participate in the program.
Nine participants (60%) are women and 10 (67%) are African-American. Ages range from
early-20s to mid-70s, and educational attainment ranges from incomplete high school
preparation to post-high school coursework.
Many CABs have expansive roles for their members that may encompass defining research
agendas, providing entrée into a community (gatekeeping), engaging in the conduct of
research, and providing feedback on scientific papers (Israel et al., 2005). The CARDS®
sole function is to meet monthly with researchers to review, discuss, and provide feedback
on research plans and materials, including recruitment materials, data collection procedures
and instruments, web-based materials, and smartphone apps. They are never required to
complete work between meetings. For each 90-minute meeting, the CARDS® receive a cash
stipend of $35. Although it is convenient for institutions to pay board members with checks,
many people in hard-to-reach populations do not have checking accounts. As one of the
CARDS® explained, “$35 isn’t $35 if I have to pay a fee to cash the check.”
WINRS and community center staff—WINRS staff provides overall management of
the CARDS® program. The School of Nursing employs two WINRS staff members at a total
of 1.7 full-time equivalents. WINRS staff work on a variety of projects, but together devote
approximately 0.5 full-time equivalents to management of the CARDS®. Both employees
have advanced degrees (MA; PhD) and had previous experience with community
engagement. One employee has primary responsibility for the program, with the other
providing support. The School of Nursing and ICTR provide salary support for the full-time
employee, and revenues from WINRS support the salary of the part-time employee. In
addition, each community center provides a staff member who serves as the center’s
CARDS® liaison. Table 1 lists the respective responsibilities of WINRS, community center
staff, and the CARDS®. We pay the centers a monthly facility fee to compensate them for
liaison time, room rental, childcare during meetings, and transportation for members as
needed. Monthly facility fees range from $205 – $290, depending on whether childcare is
provided.
Guest researchers—Since the CARDS® program began in 2010, we have conducted
meetings with 40 research teams representing 21 disciplines, including Cardiology, Family
Medicine and Community Health, Kinesiology, Nursing, Pharmacy, Rheumatology, and
Surgery. We have also worked with researchers from other institutions, sometimes
conducting CARDS® meetings via teleconference. Nurses constitute the largest proportion
of researcher teams (58%,
n
= 23) who have used the CARDS® service, and our guests have
included nursing faculty, doctoral students, and nurses working in community settings.
Nurse researchers have brought a variety of materials and plans to the CARDS® for their
Kaiser et al. Page 4
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

review and feedback, including recruitment materials (brochures, flyers, letters, scripts);
study processes (recruitment plans, interview processes, website development); focus group
protocols; interview and survey questions; study information sheets; and consent forms.
We sometimes field inquiries from researchers who are interested in lay review of their
materials but are skeptical that lay advisors who are not members of their particular target
population can provide useful feedback. Materials used in human subjects research—
recruitment notices, information sheets, questionnaires and other instruments, consent forms
—share a common set of challenges for members of the public due to their technical
language, dense presentation of information, embedded assumptions, counter-intuitive
organization, and academic tone. As members of the general public with specific training on
how to give feedback to researchers, the CARDS® can offer a uniquely fresh perspective on
research materials, no matter the specific content of the documents. The CARDS® have
provided useful feedback on materials for a wide range of health studies, despite having
limited or no personal experience with many of the topics that researchers present. At one of
our most memorable meetings, no women in the CARDS® were able to attend, so five male
CARDS® provided feedback on a doctoral student’s survey about dysmenorrhea. We
introduced the survey by reminding the CARDS® that we all have experience with pain and
discomfort, and we asked them to think about the survey questions in the context of their
personal pain experiences. Based on CARDS® feedback, the student made substantial
revisions to her survey.
Program Processes
Member orientation—One of the key features of the CARDS® is our orientation
program. Many CABs provide minimal orientation or training for board members (Albert
Einstein College of Medicine, The Bronx Health Link, & Community-Campus Partnerships
for Health, 2012). In contrast, the focus of the CARDS® orientation program is hands-on
practice with research materials. We designed an interactive group orientation to help the
CARDS® develop and practice the skills needed to participate effectively at meetings. Our
orientation program emphasizes content that is directly relevant to the work that board
members will do and gives members experience reviewing and providing feedback on
recruitment flyers and letters, consent forms, and focus group questions. Completion of the
2.5-hour orientation is a requirement for membership in the CARDS® and provides
meaningful preparation for respectful, productive interactions with researchers.
Preparation of guest researchers—Each CARDS® group meets monthly at the
community centers with guest researchers. Several weeks before the meeting, WINRS staff
sends the scheduled researcher a timeline of steps to prepare for the meeting, along with a
link to a short web-based survey requesting key information about the researcher’s desired
outcomes from the meeting. WINRS staff then engages in a series of email messages and a
telephone call or face-to-face meeting with the researcher to plan and prepare for the
meeting.
During the planning meeting, WINRS staff and guest researchers review answers to the web-
based survey and discuss possible materials for the CARDS® meeting. Researchers often
Kaiser et al. Page 5
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

bring their staff members and students to planning meetings and CARDS® meetings. We
select the materials that can be thoroughly discussed during a 90-minute CARDS® meeting
and develop a detailed working agenda. Without this preparation, researchers may have
unrealistic expectations of how many documents can be reviewed during a meeting. During
the planning meeting, WINRS staff share advice on working with lay advisors. For example,
we remind researchers to use plain language, and we help them prepare brief,
straightforward explanations of their research interests. We also encourage researchers to
gracefully acknowledge all suggestions from the CARDS®. Realistically, researchers may
not want to or be able to follow certain suggestions, but thanking the CARDS® for all
suggestions helps them feel that their voices are heard and welcomed. Decisions about how
to use CARDS® feedback can be made by researchers after they attend meetings, when they
can decide whether to incorporate specific suggestions fully, partially, or not at all.
As part of meeting preparation, we discuss fees and payment arrangements with the guest
researcher. We bundle our costs for WINRS and community center staff time, CARDS®
stipends, and other costs for the CARDS® service into a fee of $1964 for a 90-minute
meeting, preparatory work with the guest, and post-meeting summary documents. As
required by the UW-Madison, our fee structure is based on a cost-recovery model; the fee
for the CARDS® service covers our costs for providing the service, but does not generate
profit. Researchers typically pay the fee with grant funds. For doctoral students or junior
faculty without funding, we offer a reduced fee. After we meet with guest researchers, we
send a brief meeting announcement to the CARDS® to provide information on the guest
researchers, including their focus areas, materials they will present at the meeting, and type
of help they would like from the CARDS®. The announcement also includes a personal
statement from the guest researcher about why she or he cares about the research topic. We
work closely with researchers to help them craft personal statements. The CARDS® have
told us that learning about a researcher’s personal motivations for doing research has helped
them overcome their stereotypes of researchers as cold academics who use research
participants to serve their own ends.
Meeting structure and practices—We follow a structured sequence of activities at each
meeting. We start each meeting with an opening question. Everyone at the meeting
participates in a round-robin sharing of names and brief responses to the opening questions.
Our opening questions give everyone at the meeting a chance to share something about
themselves. The activity helps to break stereotypes that researchers and lay community
members may hold about each other. Over time, the sharing that occurs with the opening
question helps group members establish personal connections and build a sense of
community. We often craft opening questions that are related to the research topic that we
will discuss at the meeting. For example, our opening question for a meeting with a
researcher studying nasal irrigation was “Please say your name, and tell us one favorite
home remedy for dealing with a stuffy nose or other sinus problems.”
After the opening question, guest researchers introduce themselves briefly, using plain
language to explain the goal of their research and why it is important. We use the remaining
75 minutes of the 90-minute meeting to discuss the researcher’s materials. The structure of
the discussion is standardized and uses 3 steps to elicit feedback from the CARDS®:
Kaiser et al. Page 6
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

1. The researcher describes how and where a prospective study participant
would encounter the research materials, e.g. mailed letter to a home
address; flyer posted in a primary care clinic; consent form presented in a
community setting; website that the participant will access from a home
computer.
2. We ask guest researchers to read their materials aloud, several lines at a
time, to facilitate full participation of everyone present, regardless of
literacy level.
3. WINRS staff facilitate a section-by-section review of the materials. After a
section is read, the CARDS® offer comments. We take detailed notes and
write CARDS® feedback on a flip chart, a practice that helps to affirm the
value of everyone’s contribution to the discussion. After finishing review
of one section, we move to the next section and repeat the process.
Post-meeting products and survey—During the first two years of CARDS® meetings,
we provided a summary report to researchers who attended meetings. The 1–3 page report
summarized the overall feedback from the CARDS® on the research materials reviewed at
the meeting and highlighted specific issues related to the content, language, organization,
and format of the materials. However, when researchers later shared their revised materials
with us, we saw that they often did not translate CARDS® feedback into concrete changes to
their materials, even though they rated the value of the meetings very highly. Consequently,
we now deliver revised materials to all guest researchers, to make it easy for them to
implement recommendations from the CARDS®. Researchers consider and balance
numerous factors when designing their study materials, and we encourage researchers to use
the CARDS®-revised materials in whatever way makes the most sense for them, based on
their experience and expertise. To complete our consultation, we send a post-meeting online
survey to all researchers who attend CARDS® meetings to ask them to evaluate their
experience.
Program Outcomes
CARDS® Insights and Recommendations
The value of the CARDS® lies in their ability to provide researchers with fresh insights and
feedback that academic or professional colleagues immersed in research may not be able to
offer. As the concept of health literacy has permeated health sciences research, researchers
have developed more awareness of how using technical jargon can undermine subject
recruitment. Use of readability tools such as SMOG (National Cancer Institute, 1989) or the
Flesch-Kincaid tool in word processing programs can be helpful for improving readability.
Researchers also sometimes rely on colleagues, graduate students, high-level stakeholders,
or IRBs to identify glaring problems in research materials. While these resources can be
helpful, they may not be sufficient for ensuring that materials are acceptable and inviting to
the general public. Over five years of monthly meetings, the CARDS® have highlighted
several key characteristics of research materials that may turn people away from
participating in research.
Kaiser et al. Page 7
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

Passive language—Plain language guidelines highlight the importance of using active
language instead of passive language (“staff on my research team will ask you several
questions” instead of “you will be asked several questions”). Passive sentences lack the
clarity of active sentences because they do not clearly identify the actor. The CARDS®
highlight other problems with passive language, describing it as confusing, impersonal, and
evasive. For them, passive language in a research document raises questions and reservations
about who is behind the research project, the true motives for the project, and what
researchers will do with participants and their information. Passive language pervades
consent forms, and it particularly troubles the CARDS® when it appears, as it does routinely,
in material related to privacy and confidentiality. Vague assurances that “your information
will be stored in a secure location” arouse their suspicions; they want to know exactly who
will take responsibility for protecting their information.
Specialized use of everyday language—The CARDS® have provided feedback of a
nuanced nature that highlights issues not measured by readability tools. The CARDS®
repeatedly point out things that are confusing, patronizing, or even offensive or frightening
to potential research participants. For example, even though researchers may scrupulously
avoid using technical jargon, they often use everyday language in a highly specialized
manner. The CARDS say that words such as “data”, “procedures”, “health outcomes”,
“technique”, and “investigate” sound ominous or threatening. Even the word “study”,
universally used by researchers in recruitment materials, can evoke images of being
subjected to dangerous tests or used like a guinea pig. The CARDS® strongly prefer
“project” as a non-threatening alternative. When referring to research subjects, the CARDS®
strongly prefer the phrase “project participants”, which sounds inviting, inclusive, and less
scary than the off-putting, objectifying “subjects”. Table 2 lists additional examples of how
the CARDS® interpret common research language.
When we meet with researchers to prepare for CARDS® meetings, we explain that the
CARDS® may occasionally suggest alternative language that an IRB will reject. To our
knowledge, these instances are infrequent and have not limited the value of the service.
Currently, we are collaborating with an IRB workgroup that is charged with improving
templates for informed consent. The working group has attended three CARDS® meetings
to learn what kinds of information about research are important to the non-academic
community and to help craft clearer language for consent documents. In the coming year, the
working group expects to pilot new templates that incorporate CARDS® feedback.
Perceived tone of documents—The CARDS® have noted several other key
considerations for making documents accessible and inviting. The overall tone of a
document can serve to engage or disengage potential participants. For example, although
best practices in health education often recommend repeating key messages in documents,
the CARDS® say that redundant language feels insulting, by implying that they are not
smart enough to understand something the first time that it appears in a document. In
addition, materials that sound non-judgmental may be more likely to engage potential
participants, especially for studies that address health behaviors. The CARDS® often use the
term “gentler” when recommending alternative language. For example, when researchers ask
Kaiser et al. Page 8
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

about tobacco use, the CARDS® note that the simple question “Do you want to quit
smoking?” is a complex, sensitive question and that asking a “yes” or “no” question may not
be appropriate. The CARDS® suggest “Are you thinking about cutting back or quitting?” as
a gentler alternative that raises the smoking issue without provoking defensiveness by
implying that the person is doing something wrong.
Requests for demographic information—Demographic questions are standard items
in many research instruments, and members of the public also frequently encounter them in
the context of consumer research or participation in various programs. Despite the ubiquity
of demographic questions, they may provoke negative reactions due to their highly personal
nature. The CARDS® have told us repeatedly that they expect researchers to use
demographic information to stereotype people and present them in a bad light. The
CARDS® have shared personal stories about skipping demographic questions, providing
false answers, or declining to complete an instrument altogether if it includes demographic
questions. They recommend that researchers ask only those questions that are critical for
answering the study questions. In addition, the CARDS® recommend that researchers
provide a short, straightforward introduction to demographic questions to explain why the
researcher needs the requested information. Explanations that simply invoke grant
requirements for demographic information may be insufficient and potentially offensive;
rather, a thoughtful explanation for asking demographic questions can explicitly
acknowledge that demographic questions are personal and offer a straightforward rationale
for asking them.
Lessons Learned on Engaging Hard-to-reach Groups
When we initiated the CARDS® program, we deliberately sought participants outside of the
academic environment and not situated in positions of power. Our community center liaisons
intentionally recruited people not typical of advisory boards. Many of them have had
challenging life experiences, including poverty, homelessness, long-term underemployment,
and chronic health problems. We have structured the CARDS® program to minimize barriers
to participation and to demonstrate our respect and appreciation for all members. Our use of
conscious practices that engender trust and help people feel valued have helped us to achieve
outstanding retention of our lay advisors. Six of the current members (40%) have
participated since 2010, and the remaining nine members have participated since 2012. Half
of the former members who dropped out of the groups from 2010 – 2012 (
n
= 6) left because
they moved out of the area, changed their work schedules, or experienced re-incarceration.
Attendance has been very consistent. Since 2013, when we standardized many of our
meeting processes, aggregated attendance for the program is 81% (486 instances of member
present for meeting/601 total opportunities to attend). Attendance for individual CARDS®
over this period has ranged from 58% of meetings attended to 97%.
Consistent staffing at both WINRS and the community centers has been a critical component
of developing trusting relationships with our members and retaining them in the group over
time. The same WINRS staff have facilitated every CARDS® meeting since 2010, and the
CARDS® liaisons have typically stayed in their positions for at least two years. Consistent
staffing has helped us to develop trusting, long-term relationships with our members, which
Kaiser et al. Page 9
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

in turn support honest dialogue and problem-solving with members when challenges arise.
On several occasions, we have met with individual CARDS® to address problems related to
inappropriate behavior toward a program participant or insufficient participation in meeting
discussions. The orientation that all CARDS® complete serves as a useful point of reference
for constructive discussions about performance and expectations.
During our first few years, we often wrote short individualized notes to send to the CARDS®
after meetings, to thank them for their contributions. We call these notes “affirmations”;
because they affirm something unique and positive that the person brings to the group. We
also hold annual celebrations with the CARDS®. Our celebrations have included meals,
holiday treats, and small tokens such as gift cards.
For some of our members, applying for jobs is a frequent and difficult task. We provide
tangible support with letters of reference and assistance with resume-writing. We write
letters of reference for members based on their work with the CARDS® and provide resume
templates that highlight the unique skills and tasks of the CARDS®. With the advent of web-
based systems for providing references, we have also completed on-line surveys to support
job applications for some of the CARDS®.
Program Evaluation by the CARDS®
In 2014, we completed a program evaluation with the CARDS®. We conducted separate
focus groups at each community center to explore members’ opinions and attitudes in four
areas: their orientation to the program; their work at CARDS® meetings; factors that
contribute to their retention in the program; and changes in how they view research and
researchers. The CARDS® valued the orientation and particularly emphasized the impact of
group training activities on communicating respectfully and giving effective feedback. They
also appreciated the orientation as a chance to experience what they would do at meetings,
and several members credited the orientation with giving them confidence that they could do
the job. The CARDS® described multiple characteristics of the monthly meetings that make
the work meaningful and enjoyable. These included learning about research topics that were
relevant to them and their community and having a chance to give advice directly to
researchers. The most prominent theme throughout the focus groups was the sense of
community that motivates the CARDS® to stay in the program. They used words such as
respect, good will, support, camaraderie, and fellowship to explain why they have stayed in
the program. They also described how our standard meeting practices such as the opening
question, use of the flip chart, meeting snacks, and non-hierarchical facilitation contribute to
a sense of community. Another major factor related to retention was the sense of service to
others. As one member said, “I like being able to help others. We may be helping folks who
will never even know we helped them.” Finally, the CARDS® discussed how their attitudes
toward research and researchers have shifted. They said that the opportunity to work
regularly with researchers has helped them respect researchers, rather than mistrust their
motives. Some CARDS® also indicated that they are more interested in participating in
research themselves: “Now I feel it’s important to participate in research. Researchers are
doing it for the community; it’s bigger than just one person.”
Kaiser et al. Page 10
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

Researchers’ Evaluation of CARDS® Service
Since 2012, all guests who attend CARDS® meetings receive an email one week after the
meeting inviting them to complete a brief web-based evaluation survey. To minimize the
response burden on researchers, the survey includes just a few key questions to assess
guests’ satisfaction with the overall CARDS® service, including pre-meeting preparation
and post-meeting reports. Guests at CARDS® meetings have rated the service very highly.
Ninety-seven percent (
n
= 70) of survey respondents have indicated that they felt sufficiently
prepared by WINRS to meet with the CARDS®. Guests typically brought one or two items
for review, and they most frequently brought data collection materials such as survey or
focus group questions (51% of researchers,
n
= 37) and recruitment materials (37%,
n
= 27).
A large majority of guests (90%,
n
= 65) planned to change their materials based on
CARDS® feedback. (Several respondents self-identified as project staff that did not have
authority to make changes to research materials.) Ninety-nine percent of survey respondents
(
n
= 71; 1 missing response) have indicated that they would recommend the service to
colleagues. Table 3 displays findings related to guests’ perceptions of the utility of the
CARDS® program.
When we initiated the program, we wondered whether regular contact with researchers
would acclimate the CARDS® to research language and over time diminish their value to
researchers. To preserve the lay perspective of the CARDS®, we have maintained a schedule
of meeting only once monthly at each center. Researchers’ evaluations of the CARDS®
service have been favorable across the lifespan of the program, suggesting that the CARDS®
have continued to offer valuable feedback. Additional findings related to the effectiveness of
the CARDS® are published elsewhere (Bowers, Krupp, & Jacobson, 2016).
Conclusion
Health sciences research should serve everyone. Everyone should have opportunities to
participate in research and benefit from research. When researchers have study participants
who reflect the rich diversity in our communities, our society has a real chance to develop
policies and practices to eliminate health disparities. But current research suggests that
equitable representation of diverse groups in research has yet to be achieved (Bonevski et al.,
2014).
Social justice is a central concern of the nursing profession, and nursing research literature
offers numerous examples of studies using approaches such as CBPR to engage
disenfranchised or hard-to-reach stakeholders in the research process (e.g. Perry & Hoffman,
2010; Stacciarini et al., 2011). In addition, many nurse researchers have designed or
implemented interventions with input from community health workers or promotores (e.g.
Nies, Troutman-Jordan, Branche, Moore-Harrison, & Hohensee, 2013; Whittemore,
Rosenberg, Gilmore, Withey, & Breault, 2014). The CARDS® represent a new, unique
model for engagement of people who are not prominent community representatives in the
design of research, and the model offers distinct advantages over other forms of public
engagement. Although lay stakeholder engagement can take diverse forms, sometimes it can
seem as if the options are all or nothing—either researchers conduct CBPR or they do
Kaiser et al. Page 11
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

nothing to seek community input. The CARDS® model makes reaching hard-to-reach
stakeholders feasible and convenient for researchers.
The CARDS® are a truly innovative stakeholder engagement model, one that brings hidden
voices into the research enterprise and makes these voices easily accessible to researchers.
With our partners at the community centers, we have built a program that provides a
valuable service to researchers and a meaningful experience for lay stakeholders. The
CARDS® represent an innovative model for nurse researchers to support sustainable,
meaningful engagement of hard-to-reach populations in research planning and activities;
increase the diversity of groups represented in research studies; and ultimately help to
reduce health disparities.
Acknowledgments
WINRS and the CARDS® are funded by the University of Wisconsin-Madison School of Nursing and the Clinical
and Translational Science Award (CTSA) program, through the NIH National Center for Advancing Translational
Sciences (NCATS), grant UL1TR000427. The content is solely the responsibility of the authors and does not
necessarily represent the official views of the NIH. The authors thank current and former CARDS® and staff at the
Lussier Community Education Center and Goodman Community Center for their contributions to the CARDS®
program.
References
Albert Einstein College of Medicine, The Bronx Health Link, & Community-Campus Partnerships for
Health. Community IRBs & research review boards: Shaping the future of community-engaged
research. 2012 Retrieved from http://depts.washington.edu/ccph/pdf_files/
Shaping_the_Future_of_CEnR.pdf.
Bonevski B, Randell M, Paul C, Chapman K, Twyman L, Bryant J, et al. Reaching the hard-to-reach:
A systematic review of strategies for improving health and medical research with socially
disadvantaged groups. BMC Med Res Methodol. 2014; 14:42. [PubMed: 24669751]
Bowers BJ, Krupp A, Jacobson N. Can lay community advisors improve recruitment of
underrepresented populations?. 2016 Manuscript submitted for publication.
D'Alonzo KT. Getting started in CBPR: Lessons in building community partnerships for new
researchers. Nursing Inquiry. 2010; 17(4):282–288. [PubMed: 21059145]
Fernandez-Pena JR, Moore L, Goldstein E, Decarlo P, Grinstead O, Hunt C, et al. Making sure
research is used: Community-generated recommendations for disseminating research. Progress in
Community Health Partnerships: Research, Education, and Action. 2008; 2(2):171–176.
Ford JG, Howerton MW, Lai GY, Gary TL, Bolen S, Gibbons MC, et al. Barriers to recruiting
underrepresented populations to cancer clinical trials: A systematic review. Cancer. 2008; 112(2):
228–242. [PubMed: 18008363]
GreenMills LL, Davison KK, Gordon KE, Kaigang L, Jurkowski JM. Evaluation of a childhood
obesity awareness campaign targeting head start families: Designed by parents for parents. Journal
of Health Care for the Poor & Underserved. 2013; 24(2, Supp):25–33. 29. doi: [PubMed: 23727962]
Hasnain-Wynia R, Beal AC. Role of the patient-centered outcomes research institute in addressing
disparities and engaging patients in clinical research. Clinical Therapeutics. 2014; 36(5):619–623.
[PubMed: 24852595]
Institute of Medicine. Public engagement and clinical trials: New models and disruptive technologies:
Workshop summary. Washington, DC: 2011.
Israel BA, Parker EA, Rowe Z, Salvatore A, Minkler M, Lopez J, et al. Community-based participatory
research: Lessons learned from the centers for children's environmental health and disease
prevention research. Environmental Health Perspectives. 2005; 113(10):1463–1471. [PubMed:
16203263]
Kaiser et al. Page 12
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

James S, Arniella G, Bickell NA, Walker W, Robinson V, Taylor B, et al. Community action boards:
An innovative model for effective community-academic research partnerships. Progress in
Community Health Partnerships: Research, Education, and Action. 2011; 5(4):399–404.
Joosten YA, Israel TL, Williams NA, Boone LR, Schlundt DG, Mouton CP, et al. Community
engagement studios: A structured approach to obtaining meaningful input from stakeholders to
inform research. Academic Medicine. 2015
Kitterman DR, Cheng SK, Dilts DM, Orwoll ES. The prevalence and economic impact of low-
enrolling clinical studies at an academic medical center. Academic Medicine. 2011; 86(11):1360–
1366. [PubMed: 21952064]
National Cancer Institute. Making health communication programs work. Bethesda, MD: National
Institutes of Health, US Department of Health and Human Services; 1989.
National Institute on Minority Health and Health Disparities. [Retrieved July 23, 2015] About
NIMHD: Vision and mission. (n.d.). from http://www.nimhd.nih.gov/about/visionMission.html
Newman SD, Andrews JO, Magwood GS, Jenkins C, Cox MJ, Williamson DC. Community advisory
boards in community-based participatory research: A synthesis of best processes. Preventing
Chronic Disease: Public Health Research, Practice, and Policy. 2011; 8(3):A70. doi:
Nies MA, Troutman-Jordan M, Branche D, Moore-Harrison T, Hohensee C. Physical activity
preferences for low-income sedentary urban African American older adults. Journal of
Gerontological Nursing. 2013; 39(6):20–29. 10.
Paskett ED, Reeves KW, McLaughlin JM, Katz ML, McAlearney AS, Ruffin MT, et al. Recruitment of
minority and underserved populations in the United States: The centers for population health and
health disparities experience. Contemporary Clinical Trials. 2008; 29(6):847–861. [PubMed:
18721901]
Perry C, Hoffman B. Assessing tribal youth physical activity and programming using a community-
based participatory research approach. Public Health Nursing. 2010; 27(2):104–114. 111.
[PubMed: 20433664]
Pinto RM, Spector A, Rahman R, Gastolomendo JD. Research advisory board members' contributions
and expectations in the USA. Health Promot Int. 2013
Schroen AT, Petroni GR, Wang H, Gray R, Wang XF, Cronin W, et al. Preliminary evaluation of
factors associated with premature trial closure and feasibility of accrual benchmarks in phase iii
oncology trials. Clinical Trials. 2010; 7(4):312–321. [PubMed: 20595245]
Silvestre AJ, Quinn SJ, Rinaldo CR. A twenty-two-year-old community advisory board: Health
research as an opportunity for social change. Journal of Community Practice. 2010; 18(1):58–75.
[PubMed: 20523763]
Stacciarini J-MR, Wiens B, Coady M, Schwait AB, Péérez A, Locke B, et al. CBPR: Building
partnerships with Latinos in a rural area for a wellness approach to mental health. Issues in Mental
Health Nursing. 2011; 32(8):486–492. 487. [PubMed: 21767250]
Stewart MK, Felix HC, Olson M, Cottoms N, Bachelder A, Smith J, et al. Community engagement in
health-related research: A case study of a community-linked research infrastructure, jefferson
county, arkansas, 2011–2013. Preventing Chronic Disease: Public Health Research, Practice, and
Policy. 2015; 12:E115.
Strauss RP, Sengupta S, Quinn SC, Goeppinger J, Spaulding C, Kegeles SM, et al. The role of
community advisory boards: Involving communities in the informed consent process. American
Journal of Public Health. 2001; 91(12):1938–1943. doi: [PubMed: 11726369]
U.S. Department of Health and Human Services. [Retrieved July 23, 2015] About Healthy People
2020. (n.d.). from http://www.healthypeople.gov/2020/about/default.aspx
Whittemore R, Rosenberg A, Gilmore L, Withey M, Breault A. Implementation of a diabetes
prevention program in public housing communities. Public Health Nursing. 2014; 31(4):317–326.
310. [PubMed: 24251677]
Kaiser et al. Page 13
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript

Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Kaiser et al. Page 14
Table 1
Responsibilities of Community-Academic Partners for CARDS® Program
Community Center Staff WINRS Staff CARDS®
Recruit people who use center
services to participate in CARDS®
program
Design, deliver orientation program
for new CARDS®Complete orientation to develop
skills for giving feedback to
researchers
Schedule community center facilities
for meetings, orientations Conduct outreach to identify guest
researchers for CARDS® meetings NA
Arrange child care, transportation for
monthly meetings; serve as contact
person for members to confirm
attendance
Meet with researchers to prepare
agenda, materials for meetings; send
agenda, meeting reminder to
CARDS®
RSVP for monthly meetings; read
meeting announcement with
description of research topic and
materials
Participate in CARDS® meetings Facilitate CARDS® meetings; take
notes; disburse member stipends Provide feedback to guest researchers
at meetings
Invoice UW-Madison School of
Nursing for facility fee Write summary reports, revised
materials for researchers; complete
post-meeting evaluation surveys with
guest researchers
NA
Problem-solve with members to
address barriers to their participation Work with liaisons, members to
support participation and sustain
program
As needed, meet privately with
program staff to address issues or
problems related to participation
Participate in program evaluation Design and lead program evaluation Participate in individual interviews
and group discussions to evaluate
program
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.

Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Kaiser et al. Page 15
Table 2
What Researchers Say, What CARDS® Hear, and What CARDS® Recommend
What Researchers Say What CARDS® Hear What CARDS® Recommend
My colleague is the PI on
this study. I work with a private
investigator who will poke into
your personal business.
I work with Xxx Xxxx. (S)he is
the lead researcher on this
project.
Data will be collected about
your lifestyle. We will violate your privacy and
make judgements about your
personal life.
My research team will ask you
some questions about a typical
day for you.
Participants will take part in
several procedures. Participants will have scary,
invasive medical acts performed
on them.
Participants will be involved in
several activities, including…
We will monitor your
progress during this study. We will track you with
something like an electronic
ankle bracelet during this study.
We will keep in touch with you
during the project.
This study will test an
experimental technique for
treating sinus infections.
You will be a guinea pig for
something dangerous, untried,
and invasive.
This project will test a new way
of treating sinus infections.
We would like to include
your name in our study
registry.
We want to put your name on a
registry, like a sex offender
registry!
We would like to add your name
to a list of people who might
like to be part of future projects.
Your input will contribute
to the design of an
intervention to support
couples in parenting their
babies.
You will help the research team
confront couples who are
having problems with parenting.
Your input will help us develop
a new program to support
parents as they care for their
babies.
This study aims to improve
how we provide care to
people with type 2 diabetes.
Being in this study will improve
your diabetes care. We’d like to hear about your
experiences to help us figure out
how to improve care for people
with type 2 diabetes.
We will consider your
individual home
environment and search for
assessable risks for falls.
We will judge you and your
home and snoop into your
personal belongings.
We will use a checklist to look
for risks that you can change to
prevent falls.
To manage your blood
pressure, be more active
and drink less alcohol.
We assume that you’re not
active and that you drink too
much.
To manage your blood pressure,
be active and limit your intake
of alcohol.
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.

Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Kaiser et al. Page 16
Table 3
Guests’ Perception of Utility of CARDS® Services, 2012 – 2016
Total
N = 72
n%
Which services of the CARDS® program were particularly useful?
Completing web-based survey to initiate planning 28 39
Meeting with program staff
a
44 61
Presenting ideas and materials to CARDS®56 78
Getting feedback from CARDS®70 97
Receiving summary of CARDS® feedback 65 90
Receiving revised study materials 60 83
Note
. Percentages do not sum to 100% because respondents could select multiple options.
a
New question added to survey in 2103.
Res Nurs Health
. Author manuscript; available in PMC 2018 February 01.