Using Evaluation to Shape and Direct Comprehensive Community Initiatives: Evaluation, Reflective Practice, and Interventions Dealing with Complexity

Abstract

The purpose of this article is to discuss how evaluation can be used to shape and direct Comprehensive Community Initiatives in an ongoing way. It does so by offering a case example of the Waterloo Region Immigrant Employment Network (WR-IEN). The article begins by reviewing how participatory action research can encourage reflective practice. After the WRIEN case example is presented, we consider this example in light of collaborative evaluation literature and five facilitators of reflective practice: (a) be location-based, (b) value experiential and practical knowledge, (c) provide ongoing feedback, (d) facilitate democratic dialogue, and (e) focus on a vision for the common good. The article ends by discussing contributions to the broader evaluation knowledge base, particularly developmental evaluation.

Full text

61
The Canadian Journal of Program Evaluation Vol. 26 No. 2 Pages 61–88
ISSN 0834-1516 Copyright © 2012 Canadian Evaluation Society
Corresponding author: Rich Janzen, Centre for Community Based Research, 73
King St. W, Kitchener, ON, Canada N2G 1A7; <rich@communitybasedresearch.ca>
USING EVALUATION TO SHAPE AND
DIRECT COMPREHENSIVE COMMUNITY
INITIATIVES: EVALUATION, REFLECTIVE
PRACTICE, AND INTERVENTIONS DEALING
WITH COMPLEXITY
Rich Janzen
Centre for Community Based Research and Wilfrid Laurier University
Daniela Seskar-Hencic
Regional Municipality of Waterloo
Yasir Dildar
Centre for Community Based Research, Kitchener, Ontario
Peter McFadden
Waterloo Region Immigrant Employment Network
Abstract: The purpose of this article is to discuss how evaluation can be
used to shape and direct Comprehensive Community Initia-
tives in an ongoing way. It does so by offering a case example
of the Waterloo Region Immigrant Employment Network (WR-
IEN). The article begins by reviewing how participatory action
research can encourage reective practice. After the WRIEN
case example is presented, we consider this example in light of
collaborative evaluation literature and ve facilitators of reec-
tive practice: (a) be location-based, (b) value experiential and
practical knowledge, (c) provide ongoing feedback, (d) facilitate
democratic dialogue, and (e) focus on a vision for the common
good. The article ends by discussing contributions to the broader
evaluation knowledge base, particularly developmental evalua-
tion.
Résumé: Le but de cet article est de discuter comment l’évaluation peut
servir à former et diriger d’une façon continue les initiatives
communautaires intégrées. On vous offre, par exemple, le cas
du réseau d’emplois des immigrants à la région de Waterloo
(WRIEN). L’article commence par examiner comment la recher-
che-action participative peut encourager la pratique réexive.
Après la présentation du cas d’exemple du WRIEN, on considère
cet exemple par rapport à la littérature sur l’évaluation collabo-
rative et cinq facilitateurs de la pratique réexive: (a) se baser
sur la localisation, (b) valoriser les connaissances expérientielles

62 The Canadian Journal of Program evaluaTion
62
et pratiques, (c) fournir la rétroaction continue, (d) faciliter le
dialogue démocratique, et (e) concentrer sur une vision de l’inté-
rêt commun. Cet article se termine en discutant les contributions
à la base plus large de connaissances en évaluation, en particu-
lier, l’évaluation développementale.
Since the 1990s, Comprehensive Community Initiatives
(CCIs) have emerged as a common means of addressing social con-
cerns within local communities. CCIs are guided by the principles of
comprehensiveness and community-building (Kubisch, Fulbright-
Anderson, & Connell, 1998). That is, CCIs are designed to address
social problems holistically, at multiple levels, and with a view to pro-
moting sustainable systems change (comprehensiveness), which is
best achieved through the broad-based, cross-sectoral, and equitable
engagement of citizens who act as the architects of their own solu-
tions (community-building) (Baker, 2003; Foster-Fishman, Nowell,
& Yang, 2007). The underlying assumption is that signicant com-
munity improvements will not occur unless the surrounding system
adjusts to accommodate the desired goals. As such, CCIs not only
work for changes for individuals and groups, but also seek to improve
the conditions in which they are more likely to thrive (Baum, 2001;
Natasi & Hitchcock, 2009). This implies that community stakehold-
ers (people who have a stake in the issue under consideration) are
challenged to create new and different ways of working together
(Foster-Fishman et al., 2007). It also means that CCIs are designed to
evolve over time so as to respond to ever-changing community needs
and priorities (Lafferty & Mahoney, 2003).
Given increasing demands for accountability on investment, there
are growing expectations that CCIs be evaluated. Evaluation is seen
to be important because CCIs (a) benet from the support of a wide
range of constituents who need to be kept informed of progress and
outcomes; (b) need to generate useful feedback to guide action within
a dynamic context; and (c) hold the opportunity to contribute to the
broader, longer-term “social learning” that can be transferred to
other community settings (Kubisch et al., 1998). It has been widely
recognized that it is difcult to evaluate CCIs (Foster-Fishman &
Long, 2009; Natasi & Hitchcock, 2009). With summative evaluations
the challenge lies in measuring and analyzing broad multisectoral
outcomes within a complex socio-political context where the inter-
vention is only one factor (Kubisch et al., 1998). However, and more
central to this article, formative evaluation of CCIs can also be chal-
lenging. Formative evaluations are conducted as the intervention
unfolds for the purpose of learning, shaping, and directing it in an

6363
la revue Canadienne d’évaluaTion de Programme
ongoing way (Shadish, Cook, & Leviton, 1991). Here the challenge
lies in the dynamic nature of CCIs, whose exible and developmental
nature in response to shifting local dynamics adds ambiguity and
uncertainty to evaluation design and implementation (Baum, 2001).
A program theory approach to evaluation is typically seen as a means
of beginning to address these challenges (Lafferty & Mahoney, 2003).
A program theory attempts to explain how an intervention is sup-
posed to work, rationally linking its resources and activities with its
short- and long-term intended outcomes (Chen, 2005). A program
theory approach can also be useful in sharpening program plan-
ning—isolating those efforts that lead to desired outcomes and those
that do not (Mackenzie, & Blamey, 2005). Nevertheless, a CCI’s the-
ory of change is not necessarily unied or stable, but rather evolves
over time in response to complex and uid environments (Baum,
2001; Lafferty & Mahoney, 2003). Evaluations of CCIs therefore
require mechanisms to enable ongoing reective practice—mecha-
nisms that enable stakeholders to collectively reect on what they
have done (practice) and what they have learned about what was
effective (theory), all for the sake of adapting and informing their
future practice and deepening their program theory (Natasi & Hitch-
cock, 2009). Such mechanisms of reective practice are particularly
needed within evaluations of innovative interventions (as CCIs often
are), where previous models of practice are lacking and where con-
texts are complex and uid (as is often the case with CCIs). Patton
(2011) recently coined the term “developmental evaluation” to dis-
tinguish evaluations of these complex systems-change interventions
from traditional formative evaluations that seek to improve xed-
models of practice in more stable environments.
Practical examples of evaluations of CCIs that use a theory of change
approach are beginning to emerge in the evaluation literature. How-
ever, these typically involve summative evaluations that report on
outcomes and related methodological challenges (e.g., Foster-Fish-
man & Long, 2009; Mackenzie & Blamey, 2005). There are limited
examples demonstrating how evaluation can be used to shape and
direct CCI practice in an ongoing way. The purpose of this article is to
speak to this gap by discussing the processes and mechanisms of how
evaluation can be an avenue to spur reective practice within CCIs.
It does so by offering a case example of an evaluation of one CCI (the
Waterloo Region Immigrant Employment Network—WRIEN) that
adopted a participatory action research approach. The article begins
by rst reviewing the broader literature of how a participatory ac-
tion research approach can facilitate reective practice. Following

64 The Canadian Journal of Program evaluaTion
64
the case example, we bring evaluation literature into the conversa-
tion and link WRIEN’s reective practice to this literature. In this
way, the article bridges the gap between two long-standing bodies
of literature (collaborative evaluative inquiry and participatory ac-
tion research) that often remain in parallel. We conclude by briey
discussing how this article contributes to the broader evaluation
knowledge base, including that of developmental evaluation. We note
how the case study illuminates that the formative-developmental
distinction may be less categorical than Patton (2011) presents, but
can rather be seen as a continuum in how loosely or tightly a par-
ticular program theory is held.
PARTICIPATORY ACTION RESEARCH AND REFLECTIVE PRACTICE
As one type within the broader typology of research, evaluation can
draw on the growing literature that discusses how research can
stimulate reective practice. Such approaches to research operate
under a variety of names, including action research (e.g., Stringer,
2007), community-based research (e.g., Travers et al., 2008), commu-
nity-based participatory research (e.g., Israel, Eng, Schulz, & Parker,
2005; Minkler & Wallertsein, 2008), or—the preferred term in this
article—participatory action research (e.g., Kemmis & McTaggart,
2005). Regardless of label, this approach fuses the “northern” utiliza-
tion-focused action research tradition initiated by Kurt Lewin (1948),
with the “southern” emancipatory participatory tradition initiated by
Paulo Friere (1970) (for fuller discussions of these historical roots,
see Cargo & Mercer, 2008; Wallerstein & Duran, 2003). Common to
these traditions is a commitment to interweave theory and practice
within a research process in a way that leads to knowledge-driven
collective action. Indeed, “learning as we go” is a key value of partici-
patory action research (Nelson, Ochocka, Griffen, & Lord, 1998)—a
value tting well with the evolving and responsive nature of CCIs.
Within participatory action research, theory and practice are inter-
connected through a process known as the reective action cycle.
The cycle generally includes some combination of planning, action,
and reection in successive spirals over time (Lewin, 1948; Stringer,
2007; Wallerstein & Duran, 2003). Within the process of research
(including evaluation), the reective action cycle could be conceived
as four nonlinear and repeated phases (laying the foundations, plan-
ning, information gathering/analysis, and acting on ndings) that
are ever-attuned and adaptive to an emerging context and ongoing
learning (see Figure 1). While we elaborate on these phases in more

6565
la revue Canadienne d’évaluaTion de Programme
detail elsewhere (Janzen, Hatzipantelis, Vinograd, & Kellerman,
2004; Taylor & Botschner, 1998), it is worth highlighting here that
the phases emphasize not only traditional technical elements of
research, but also foreground the relational aspects of collaborative
research. They do so because of a belief that a collaborative process
of inquiry is as important as the outcomes or ndings of the research
(Reason, 2006). Research produces not only a vision for future collec-
tive action, but also builds a sense of community and inspires people
to work together toward a common goal (Stringer, 2007).
Figure 1
The Four Phases of Participatory Action Research
Adapted from CCBR, 1998; 2004.
The broader research literature also provides insights into how reec-
tive practice can be facilitated through research. Below we outline ve
facilitators that we gleaned from the literature and that represent
our theoretical contribution to the participatory action research dis-
course. While we do not presume these facilitators to be exhaustive,
they do provide a means of connecting participatory action research
to the knowledge base of evaluation (discussed later in the article).

66 The Canadian Journal of Program evaluaTion
66
Be Location-Based
The rst facilitator of reective practice is to focus the scope of
research to a particular community setting. Being location-based
requires a deep ecological understanding of the local context (New-
brough, 1995). It also implies that design changes to respond to a
context, which is dynamic in nature (Reason, 2006). This is because
the emphasis is on solving local problems for a specic group of
people—local theory for local practice (Stringer, 2007). Reecting on
local practice is pre-eminent, as generalizing theory to benet other
settings is of secondary importance. Within the evaluation of CCIs,
being location-based implies a clear bounding of what constitutes
the intervention under study and what constitutes external context
(Foster-Fishman et al., 2007). It also implies that depictions of a pro-
gram’s theory of change, such as program logic models (McLaughlin
& Jordan, 2004), are not viewed statically but change and evolve in
response to the deepened understanding of what kind of intervention
is needed with a specic location.
Value Experiential and Practical Knowledge
Traditional research has privileged knowledge that is produced by
synthesizing data contributed by others, typically experts. Such a
view keeps theory and practice separate. But knowledge can also be
understood through the critical reection of personal and collective
experiences, whether these experiences are recent (Clare, 2006) or
historical (Fals Borda, 1987). In addition, knowledge can emerge as
people do something in a critically reective way. Heron and Reason
(1997) call these two other ways of knowing “experiential” and “prac-
tical” respectively. Valuing experiential and practical knowledge as-
sumes that people can create new understanding that is grounded
in their social involvements, which in turn creates a better informed
practice that is guided by newfound insights (Israel, Schulz, Parker, &
Becker, 1998). Evaluation of CCIs can value such knowledge by trian-
gulating the opinions of all stakeholders, encouraging them to critical-
ly reect on both their experience of, and their efforts within, the CCI.
Provide Ongoing Feedback
To ensure that the experiential and practical ways of ‘knowing’ de-
scribed above are a collective exercise, research should include cycles
of feedback and exchange among people (Heron & Reason, 1997).
Such co-operative inquiry implies that communities involved in re-

6767
la revue Canadienne d’évaluaTion de Programme
search own their knowledge (Fals Borda, 1987). Community members
are therefore in the best position to link what they collectively know
to what they collectively do. Evaluation of CCIs can incorporate regu-
lar feedback of ndings into the evaluation and intervention design,
using multiple formats and forums that speak most clearly to respec-
tive stakeholders (Chavez, Duran, Baker, Avila, & Wallerstein, 2008).
Facilitate Democratic Dialogue
Following the critical theorist Jurgen Habermas, Kemmis and McTag-
gart (2005) call for researchers to create “communicative spaces.”
Research participants are not seen as forced subjects of a research
process or passive recipients of its ndings. Rather, participants free-
ly interact within research and with the knowledge it produces in or-
der to reach inter-subjective agreement, mutual understanding, and
unforced consensus. They are engaged in dialogue because they are
encouraged to speak about what is important to them in ways that
promote liberty, fraternity, and equality (Newbrough, 1995). Such a
view implies that researchers are not only technicians, but also adopt
the role of facilitator (Conde-Frazier, 2006). Researchers become col-
laborators rather than controllers, and research quality rests on the
ability of researchers to stimulate, use, and interpret open discussion
(Heron & Reason, 1997). Such democratic dialogue provides the re-
ective space in which theory and practice can be brought together.
Evaluation of CCIs can nurture this reective space by incorporating
methods that encourage interaction (e.g., focus groups and commu-
nity forums), and by using feedback to facilitate further discussion
of what existing ndings mean for future action.
Focus on a Vision for the Common Good
A nal facilitator for reective practice is carrying out the research
collaboration in such a way that people gain the collective capacity
to imagine how the circumstances of their lives could be improved
(Kemmis & McTaggart, 2005). Indeed, the research process itself
could be conceived of as a negotiation among stakeholders about the
type of future they wish to co-create. If such a vision-focus is fostered
within a research process, participants will gain licence to reec-
tively link their collective knowledge with their desired collective
action. Within CCIs there are often diverse stakeholder perspectives
and interests, which are often in competition and at odds. The re-
search process can act as a forum in which diverse perspectives and
interests are aired and negotiated in a transparent and systematic
way (Ochocka, Moorlag, & Janzen, 2010).

68 The Canadian Journal of Program evaluaTion
68
THE WRIEN CASE EXAMPLE
Description of Waterloo Region Immigrant Employment Network
The Waterloo Region Immigrant Employment Network (WRIEN—
pronounced “Ryan”) was launched in 2006 in Waterloo Region, a
community of half a million in southern Ontario (see www.wrien.
com). WRIEN was created to act as a local strategy to help ensure
that the skills of immigrants would be more optimally used to the
benet of immigrants and their families, the region’s economy, and
the community as a whole. It was developed in response to the well-
documented under-utilization of immigrant skills across Canada (Al-
boim, 2009; Wayland, 2006), including considerably higher rates of
unemployment and under-employment, as well as lower earnings of
immigrants in Waterloo Region relative to Canadian-born—this de-
spite the higher education levels of immigrants (Miedema & Vande-
belt, 2006). WRIEN was a comprehensive community initiative that
provided a forum to explore, develop, and test innovative system
change responses to immigrant employment (comprehensive). It did
this through a broad-based and cross-sectoral engagement of com-
munity stakeholders who identied their own solutions and priorities
(community-building) (Janzen, Walton-Roberts, & Ochocka, in press).
WRIEN emerged as a result of extensive community consultation.
Led by the Centre for Community Based Research (CCBR), diverse
community stakeholders were brought together to identify local
needs, resources, and strategies in facilitating immigrants’ access
to employment. Stakeholder groups included immigrants, business,
local government, community organizations, education, and non-
government funders, many of which had not been previously engaged
on this issue. WRIEN was launched following an Immigrant Skills
Summit at which leaders from all stakeholder groups prioritized
and committed to a comprehensive set of community action strate-
gies (Janzen, Hatzipantelis, & Hogarth, 2005). WRIEN subsequently
functioned as a body that convened and facilitated the collective ac-
tion of these various local stakeholders in implementing the identi-
ed strategies. WRIEN was among the rst immigrant employment
comprehensive community initiatives to be developed in a mid-size
urban centre in Canada, if not beyond (McFadden & Janzen, 2007).
Winning awards from both nonprot and private sectors, WRIEN’s
innovation lay in leveraging diverse community resources toward the
common goal of immigrant employment, with few implementation
models to draw on. WRIEN was hosted by the Greater Kitchener-

6969
la revue Canadienne d’évaluaTion de Programme
Waterloo Chamber of Commerce and consisted of a multi-stakeholder
steering committee (privileging the immigrant and business perspec-
tives), four active work groups (investments, employment initiatives,
immigrant support, and qualication recognition and enhancement)
and was supported by four staff. WRIEN’s structure was intended to
be non-hierarchical with an emphasis on involving all community
stakeholders. Core WRIEN funding was provided by local govern-
ment and non-government funders. WRIEN was also successful in
securing additional project funding to support work group priorities.
It is important to emphasize that WRIEN was not a direct service
immigrant employment program. That is to say, WRIEN was not de-
signed to directly match individual immigrants with employers (the
region has numerous such programs). Instead, it was a broad-based
network designed to create the community conditions that would
make successful immigrant employment more likely (i.e., systems
change). It did this by facilitating diverse segments of the local com-
munity to work together in new ways. For example, WRIEN emerged
as a region-wide voice in advocating to senior levels of government
on an array of immigrant employment issues. WRIEN was also in-
strumental in launching and supporting a series of community initia-
tives: an immigrant mentorship program, an immigrant internship
program, an immigrant loan program (to support further education),
an immigrant web portal, and an employer connections program. In
all these examples WRIEN itself was not the lead organization but
was the convener and catalyst in securing the necessary partners
and resources to create and deliver “made-in-Waterloo” solutions
(Wayland, 2007).
Description of the WRIEN Evaluation
An ongoing process and outcome evaluation was built into WRIEN’s
initial three-year mandate. Evaluation was intended to determine
how and to what extent WRIEN’s multi-stakeholder and region-wide
efforts were impacting the lives of immigrants, the local economy,
and the broader community of Waterloo Region. The main research
questions related to the structure of WRIEN, the implementation of
its activities, outcomes observed to date, lessons learned, and recom-
mendations for the future. A research team from the CCBR led the
evaluation under the guidance of a cross-stakeholder steering group.
The evaluation used a mixed method design (Tashakkori & Teddlie,
2003) carried out in annual cycles over the three-year evaluation

70 The Canadian Journal of Program evaluaTion
70
period. Methods that were repeated each year and that considered
multiple stakeholder perspectives included key informant interviews
(n = 50); focus group interviews (n = 52 in seven groups); participant
observation of the WRIEN steering committee, work group meetings,
and WRIEN-sponsored community events (total meetings/events =
25; almost 40% of all meetings/events); brief meeting tracking forms
completed by participants at the end of each steering committee and
work group meeting (n = 263); and an internal and external docu-
ment review. In addition, an employers’ web survey (for private, pub-
lic, and nonprot sectors) was conducted in the rst and last years
of the study (n = 264).
The WRIEN evaluation adopted a participatory action research ap-
proach that we dened as a “research approach that involves active
participation of stakeholders, those whose lives are affected by the
issue being studied, in all phases of research for the purpose of
producing useful results to make positive changes” (Nelson et al.,
1998, p. 9). Three primary mechanisms were used to implement this
approach. First, the CCBR evaluation team comprised members
bringing content expertise (in immigrant employment and systems
change evaluation), experiential knowledge (recent immigrants who
had subsequently been unemployed or underemployed), and a com-
mitment to advancing immigrant employment as an issue of so-
cial justice. Second, a seven-member evaluation committee guided
each step of the evaluation process. Committee members gave input
into all evaluation tools, sampling criteria, participant recruitment,
analysis, interpretation, and the dissemination of ndings. Mem-
bers reected WRIEN’s diverse stakeholder groups and had varying
degrees of involvement within WRIEN. Third, the evaluation was
characterized by frequent feedback in multiple formats. In addition
to the annual evaluation reports, feedback was provided via evalua-
tion e-bulletins, summaries and presentations of individual methods,
ndings posted on the evaluation corner of the WRIEN website, and
evaluation updates at all WRIEN steering committee meetings. In
addition, a community forum was held each year at which time the
year’s annual evaluation ndings were reported.
The evaluation repeated the action reection cycle (Figure 1) three
times (i.e., one cycle for each of the three years). Following each year,
the evaluation committee reconsidered how to adapt the evaluation
process based on emergent learning. The research questions and
methods remained relatively constant from year to year with minor
changes. More pronounced changes were made in how the evaluation
ndings were shared from year to year, and how this subsequently

7171
la revue Canadienne d’évaluaTion de Programme
informed new practice within WRIEN (i.e., Phase 4 of the action
reection cycle). In addition, the program’s theory of change was
reviewed each year using an evolving logic model technique. The
baseline and Year 3 logic models are found in Figures 2 and 3 respec-
tively. These logic models served as reective tools to consider what
had been accomplished over the previous year, and looked forward
to the emergent theory of what should be done in the upcoming year.
Evaluation ndings highlighted the strength of WRIEN’s collabora-
tive approach, but also pointed out limitations to date (Dildar & Jan-
zen, 2009; Hogarth, Janzen, & Cushing, 2007; Janzen & Dildar, 2008).
WRIEN had increased community awareness about the issue of im-
migrant employment and was successful in leveraging additional -
nancial resources from senior levels of government. WRIEN was also
successful in creating an increased level of trust among its partners
that led to the formation of a series of new interventions dealing with
specic aspects of immigrant employment (e.g., internship, mentor-
ship, loan programs). However, further community engagement was
needed, particularly among employers and recent immigrants, as
was more attention on a rapid advocacy response and on marketing.
Another signicant constraint of WRIEN was its narrow focus on
skilled professionals to the exclusion of unskilled workers.
LINKING WRIEN’S REFLECTIVE PRACTICE TO THEORY
What did we learn about the role of evaluation in helping to shape
and direct the evolving WRIEN intervention? Below we use the ve
facilitators of reective practice that we previously identied as a
frame to discuss our learnings. We also link our learnings to the
evaluation literature, particularly that which has appropriated or
adapted collaborative modes of inquiry. This is done to demonstrate
how previous evaluation discourse has contained elements of partici-
patory action research.
Be Location-Based
The WRIEN initiative was situated in a community that had a long
history of addressing immigrant employment issues. The CCBR and
several service providers had provided nearly a decade of leadership
on various research and community development projects prior to
launching WRIEN (Centre for Community Based Research, 2010).
Those efforts included (a)research projects that analyzed the local
employment and support needs and capacities of immigrants, (b)en-

72 The Canadian Journal of Program evaluaTion
72
Figure 2
WRIEN Base Line Logic Model (2006)
Work Groups
(Vertical)
Sample
Priority
Activities
Main
Components
-Compile and
distribute
information on
immigrant
employment
issues locally
and nationally
Information
Clearing-
house
Steering
Committee
(Horizontal)
Activity
Areas
Communi-
cations
Advocacy
Qualification
Recognition/
Enhance-
ment
Immigrant
Support
Immigrant
Attraction
Investments
Network
Coordination
Community
Consultation
Employer
Initiatives
Direct
Outcomes
Overarching
Outcomes
Long Term
Visionary
Outcomes
-Develop
marketing
strategy for
entire network
-Conduct
region-wide
advocacy that
is informed by
work groups
and
community
consultations
-Determine
linkages and
encourage
flow of
information
across the
network
-Provide
forum for
community-
wide input into
WRIEN
priorities
-Ensure
financial
viability of
WRIEN
operations
-Attract $ from
senior levels
of gov’t
-Marketing
campaign
-Case studies
-Employer
resource
guides
-Networking
opportunities
-Advocate/
coordinate
bridging
programs
-Promote
access to
financial
resources
-Co-advocate
at all levels
-Provide
information to
immigrants
-Encourage
organizational
improvement
Brand/market
Waterloo to
immigrants
-Deploy
website
-Gather data
employment
resources
More
strategic
recruitment
and
retainment of
immigrant
skills
Better equipping
of immigrants in
overcoming
systemic
employment
barriers
Increased
resources
levered and
more efficient
allocations
More strategic
orientation and
support of
employers in
hiring immigrants
and successfully
integrating them
into the
workplace
Better
recognition and
enhancement
of the prior
learning and
credentials of
immigrants
Improved
collaboration,
partnerships and
synergizes
among
stakeholders
Improved region-
wide systemic
advocacy and
better lobbying of
senior
governments re:
policy changes
Increased
awareness of
immigrant
employment
issues and
improved access
to relevant info
More
immigrant-
friendly
workplaces
More inclusive hiring
practices/strategies
among employers and
reversed trend of
immigrant employment
barriers
Increased likelihood of
employers with vacancies
hiring immigrants/
immigrants are more visible
and successful within
recruitment and selection
processes
Improved sense
of belonging
within Waterloo
Region by
immigrants, and
immigrants more
likely to stay
More and more
effective
immigrant
supports, with
decreased
gaps/overlaps
Greater coherency
and effectiveness of
regional/provincial/
federal immigrant
employment policy
Increased # of
employers and
government/
community agencies
setting annual
program plans and
allotting funds
Increased
number of
immigrants and
employers
accessing
available
supports
Increased
marketability of
immigrant skills
Better
prioritization and
better
implementation
of a
comprehensive,
region-wide plan
Employers more
aware of
immigrant
employment
issues
Improved
attraction of
immigrant talent
relative to labour
market trends
Improved
immigrant well-
being
Strong Lives… Strong Economy… Strong Communities…
Increased
regional
economic
prosperity
Healthier, more
vibrant and more
inclusive
communities

7373
la revue Canadienne d’évaluaTion de Programme
Figure 3
WRIEN Year 3 Logic Model (2009)
Priority
Activities
Main
Components
Activity
Areas
Foundational
Change
Tangible
Community
Change
Long Term
Change
Work Groups
(Within issue)
-Compile and
distribute
immigrant
employment
library
-Regular email
updates
-Distribute
“Today’s
Workforce” e-
bulletin
-Web
development
Information Clearing-
house
Steering Committee
(Cross Network)
Communi-
cations and
Marketing
Advocacy
Qualification
Recognition/E
nhance-ment
Immigrant
Support
Investments
Community
Planning
Community
Consultation
and
Engagement
Employment
Initiatives
-Develop a
comprehensive
marketing /
communication
strategy for entire
network
-Conduct region-
wide advocacy
that is informed
by work groups
and community
consultations
-Develop a local
rapid response
advocacy
strategy
-Determine
linkages and
encourage flow of
information
across the
network
- Initiate and
support new
immigrant
employment
interventions
-Organize public
forum for input
and involvement
within WRIEN
priorities
- Strike an
immigrant
outreach task
force to involve
immigrants
-Ensure financial
viability of WRIEN
operations
-Attract $ from
senior levels of
gov’t
- Strengthen the
role of the group
to utilize non-
financial
resources
-Implement and
expand employer
engagement
strategy
-Provide
networking
opportunities for
employers and
immigrants
-Advocate for
credential
assessment
-Launch
Immigrant Loan
Program
-Advocate for
Federal points
system
- Advocate for
more bridging
programs
-Co-advocate at
all levels
-Provide
information to
immigrants
-Support
programs for
Immigrant
integration in the
community and
the workplace
Fewer systemic barriers faced by
immigrants in finding and
maintaining meaningful employment
Increased number of
immigrants and
employers accessing
available supports
Increased
marketability of
immigrant skills
Improved
immigrant well-
being
Strong Lives… Strong Economy…
Strong Communities…
Increased
regional
economic
prosperity
Healthier, more
vibrant and more
inclusive
communities
Increased resources
leveraged and more
efficient allocations
More responsive region-wide
advocacy to emerging local issues
and better lobbying of senior
governments re: policy changes
Increased # of employers and
government/
community agencies setting
annual program plans and
allotting funds
Increased awareness of immigrant employment
issues and improved access to relevant
information (including success stories)
Better prioritization, communication and
publicity of a comprehensive, region-wide plan
Increased community readiness for change: Pre-W RIEN activities caused people see the potential of collaboration and motivated people to
want to contribute to this unique community opportunity (energized vision).
Increased stakeholder (particularly employer
and immigrant) engagement
Improved environment for collaboration,
partnerships and synergizes among stakeholders
(shared ownership)
Inform and make requests of
each other
Decreased gaps and overlaps
in local immigrant support
programs (new collaborative
interventions)
Improved sense of belonging within
Waterloo Region by immigrants,
and immigrants more likely to stay
More immigrant-friendly workplaces
with successful employment
outcomes
Increased opportunities
for immigrants to
interact with business
community
Increased access to
educational/skill
upgrading for
immigrants
More inclusive hiring practices
among employers and
increased trend of immigrants
finding work in their field
Greater coherency and
effectiveness of
regional/provincial/federal
immigrant employment policy
More support for employers to hire
immigrants and successfully
integrate them into the workplace
(language & acculturation supports)
Better recognition and
enhancement of the prior
learning and credentials of
immigrants

74 The Canadian Journal of Program evaluaTion
74
gagement of diverse stakeholders in gathering ideas and sugges-
tions for action, (c)monitoring of relevant initiatives in Ontario and
beyond, and (d)extensive work on building a community of activists
around immigrant employment issues. These initiatives helped build
a foundation of local knowledge, a strong sense of local community
need, and an engaged local stakeholder base. But, perhaps most im-
portantly, these initiatives also laid the groundwork to identify the
priorities for local action that WRIEN could champion.
The WRIEN evaluation built on this local momentum. To begin, in-
volving the CCBR to head evaluation efforts ensured that the evalu-
ation process would be carried out with a deep understanding of local
context and history. Evaluations of system change initiatives such as
WRIEN require that a clear bounding of the system is established
where social problems, the multiple levels of the system, and the set-
tings, organizations, and actors are clearly understood and dened
(Foster-Fishman et al., 2007; Stake, 1978). In addition, the CCBR had
become a trusted entity in bringing new stakeholders together, and
could now leverage that trust toward cross-stakeholder participation
within the evaluation. A lack of trust has been identied as a leading
challenge in promoting successful community research partnerships
(Becker, Israel, & Allen, 2005), including between evaluators and
community members (Nelson, Janzen, Ochocka, & Trainor, 2010).
The WRIEN evaluation beneted from its past history of trust as
evaluators facilitated stakeholders to conceptualize and agree upon
their unique local theory of change. In this way the evaluation team
became an active member of the intervention team (Patton, 2011). In
the words of one WRIEN staff, “the evaluation was not leading, but
helping the leaders to lead.”
Value Experiential and Practical Knowledge
CCIs such as WRIEN are typically conceived by diverse stakehold-
ers and grounded in their lived experiences and perspectives. The
program theory, goals, and objectives are driven and negotiated by
stakeholders (Kubisch et al., 1998). This reected the efforts lead-
ing up to WRIEN and its subsequent implementation. The WRIEN
evaluation built on this tradition by inviting stakeholders to (a)rec-
reate objectives and actions in each of the stages of the intervention
development, (b)critically reect on the WRIEN’s contributions and
achievements from year to year, and (c)develop solutions for improve-
ments and options for meeting the overall goal of the intervention.
The WRIEN evaluation therefore mirrored Patton’s (2008) “theory of

7575
la revue Canadienne d’évaluaTion de Programme
action framework” by encouraging practitioners to examine, reect
on, and deal with the discrepancies between their “espoused theory”
(how practitioners explain what they are attempting to do) and their
“theory in use” (what their behaviour reveals about what actually
guides what they do).
Valuing the experiential and practical knowledge of evaluation par-
ticipants is a key component of reective practice (Nelson, Ochocka,
Janzen, Trainor, & Lauzon, 2004). Within WRIEN this was concretely
achieved through three means. First, many of the questions asked
within the evaluation encouraged participants to reect both on
their experience of being involved in WRIEN (e.g., “To what extent
was your opinion respected within this WRIEN meeting?”), as well
as what they practically accomplished through the involvement (e.g.,
“To what extent did this meeting move ahead the agenda of this work
group/steering committee?”). In addition, the design of the evalua-
tion emphasized triangulation of these opinions through multiple
methods (e.g., surveys, key informant interviews, observations, focus
group, meeting tracking) and from multiple stakeholder perspectives
(immigrants, business, government, service providers, education,
non-governmental funders). Finally, the process of analysis encour-
aged collective critical reection on emerging evaluation themes.
Data from the multiple mixed-methods were initially analyzed in
parallel by the research team. Preliminary evaluation ndings were
then veried and shaped through discussions among WRIEN partici-
pants who sat on the evaluation and steering committees.
Theory that is grounded in social experiences can lead to better in-
formed and more effective practice (Cargo & Mercer, 2008; Israel et
al., 1998). The WRIEN evaluation validated the experiences (whether
positive or negative) of the various stakeholders involved in WRIEN,
and leveraged that validation in moving participants from a place
of opinion and critique toward constructive new action. That is not
to say that valuing experiential and practical evidence was without
challenges. For example, expectations were raised among evaluation
participants that their opinions actually mattered in reshaping the
future of the intervention.
The stated openness to forgo a typical xed theory of change in
favour of “learning as we go” (Nelson et al., 1998) certainly intro-
duced the challenge of raised expectations that WRIEN would be
responsive to evaluation ndings. While the goals, objectives, and
actions of WRIEN were initially agreed upon, the evaluation was a
built-in mechanism to ensure that the numerous and diverse objec-

76 The Canadian Journal of Program evaluaTion
76
tives would be continuously negotiated over time, based on the ac-
knowledgement and documentation of the experiential knowledge. In
this way, the evaluation promoted an organizational learning system
(Cousins, Goh, Clark, & Lee, 2004). To illustrate, Year 3 evaluation
ndings obtained from multiple stakeholders’ experiences revealed a
need for much stronger promotion, communication, and marketing of
WRIEN as one of the main priorities for action even though this was
not a prominent part of the original set of activities. The challenge
was then to deliver on this new urgent set of actions.
Provide Ongoing Feedback
Knowledge utilization within evaluation is maximized when efforts
are made to share evaluation ndings in clear, useful, and respectful
language targeted at multiple stakeholder perspectives (Nelson et
al., 2010; Patton, 2008). A number of feedback strategies were used in
the WRIEN evaluation to maximize reective practice. The rst dealt
with the timeliness of feedback. Whether giving feedback of indi-
vidual evaluation methods or providing an annual overall summary,
an emphasis was placed on providing feedback as quickly as possible.
Closing the gap between data collection and analysis was important
in order for those involved in WRIEN (i.e., staff, work groups, steer-
ing committee) to make evidence-based decisions that were informed
by current information. “Real-time evaluation” was how one member
of the business community consequently dubbed the evaluation. The
timeliness of generating feedback was essential, particularly in mo-
ments when dissatisfaction was detected and when new solutions
were emerging. Such a situation occurred between Years 2 and 3,
when the evaluation results provided grounds for action that led to
signicant reshaping of its original ambitious goals and priorities.
A second strategy was to use multiple stages and forms of feedback.
Regular feedback of evaluation processes and preliminary ndings
was provided to evaluation committee members in e-news format
and during meetings. Summary bulletins for selected methods (e.g.,
employer web survey and meeting tracking forms) were also distrib-
uted and presented to both the evaluation and steering committees
for verication and to begin the discussion of implications for future
action. At year end, the nal report went through multiple stages
of feedback. It began with the evaluation committee verifying main
themes, providing interpretive comments about how best to frame
these themes, and identifying options for action (i.e., recommenda-
tions). A presentation to the WRIEN steering committee again served

7777
la revue Canadienne d’évaluaTion de Programme
to verify and frame themes, but, more importantly, to stimulate
decision-making about how evaluation recommendations could be
translated into concrete action steps. Finally, the year-end evalua-
tion ndings were presented to the broader community via a report
posted on the WRIEN website and at an open forum.
This leads to the third feedback strategy. Evaluation can be seen as a
servant in support of a broader social change movement, particularly
for those with limited power and opportunity (Mertens, 2009; Nelson
et al., 2004). In this light, the feedback loops within WRIEN’s evalu-
ation ndings could be viewed as an opportunity to constructively
advance an immigrant employment agenda. To this end, WRIEN
ndings were carefully positioned to provide constructive rather
than destructive criticism, pointing to the value of learning and evo-
lution rather than simply examining and judging what worked and
what did not. Such a tack enabled participants to provide not only
reections and comments but also options and scenarios for improv-
ing the advancement of immigrant employment.
In summary, a commitment to utilization was foregrounded with
researchers adopting feedback formats that tried to connect with di-
verse stakeholder audiences (Patton, 2008). Providing such feedback
served to engage stakeholders within the WRIEN intervention: (a)
it helped stakeholders continuously reect and stay on track with
their own goals, (b) it provided a venue for timely suggestions for
improvements, and (c) it served as a tool for synthesizing knowledge
and translating it into desired actions. In other words, evaluation
feedback was used to embed continuous reection into the “WRIEN
culture” (Cousins et al., 2004).
Facilitate Democratic Dialogue
In collaborative approaches to evaluations the evaluator may act
as a facilitator and co-agent of change (Janzen, 2011; Nelson et al.,
2010). The evaluator’s role is not only to enable the generation and
synthesizing of new knowledge, but also to lead participants into
a democratic dialogue that helps them to co-create knowledge and
use it as a catalyst for future action. The facilitation of democratic
dialogue resonates with the concept of “process use” that foregrounds
the learning that occurs during the evaluation process itself (Patton,
2008). Learning, including through evaluation, is viewed as a social
process (Amo & Cousins, 2007). Therefore, how an evaluation is con-
ducted (and not simply its ndings) may lead evaluation stakehold-

78 The Canadian Journal of Program evaluaTion
78
ers to acquire new knowledge and to learn new behaviours and skills
(Alkin & Taut, 2003).
Within CCIs, facilitating democratic dialogue is encouraged by an
openness to an evolving program theory that is ever responsive in
the emergent local setting (Lafferty & Mahoney, 2003). For example,
WRIEN’s rst-year evaluation revealed that despite considerable
preparatory work, WRIEN’s role as a systems change intervention
was still not understood across participants. Without tangible and
visible accomplishments in the rst year, some community members
began to question the intervention, seeing it as being too broad and
removed from their day-to-day realities. The steering committee dis-
cussions about these preliminary evaluation ndings, however, led to
the reframing of ndings by recognizing the importance of “founda-
tional” change (e.g., increased awareness of immigrant employment
issues and improved environment for synergistic partnerships) as
precursors to more tangible outcomes (e.g., new collaborative inter-
ventions, improved hiring practices). A bamboo analogy was used to
explain that considerable root development is needed (foundational
change) prior to bamboo shoots breaking through the earth (tangible
change). Rather than being simply frustrated with the slow pace of
tangible change, stakeholders came to articulate a local logic of how
observed foundational changes could lead to desired longer-term
impacts.
As the above example illustrates, democratic dialogue was stimu-
lated within the WRIEN evaluation primarily in two phases of the
reective action cycle: information gathering (Phase 2) and informa-
tion sharing (Phase 4). With regards to information gathering, a
mix of qualitative and quantitative as well as individual and group
methods provided variety in dialogue formats. Similarly, the multiple
feedback sessions in Phase 4 allowed multiple opportunities for ex-
change of opinion. In both cases a range of stakeholder perspectives
was included, but always emphasizing the voice of WRIEN’s two
central stakeholder groups (immigrants and employers). Following
the tradition of participatory evaluation, evaluation utilization was
therefore maximized by actively engaging these multiple stakehold-
ers throughout the research process, recognizing that their per-
spectives were critical given that knowledge is socially constructed
(Cousins & Earl, 1992).
Facilitating democratic dialogue was particularly important when
transformational changes were desired (such as in Year 2 of WRIEN’s
operation). At this time the stakeholders were clearly prepared to

7979
la revue Canadienne d’évaluaTion de Programme
look at whether the intervention was reaching its main goals and
were much clearer in their judgement of successes and challenges.
During this stage, the researchers took additional steps to clarify the
importance of carefully interpreting results and exploring proactive
solutions. This process ensured that the participants felt heard while
their issues were addressed in a constructive and mutually respect-
ful fashion. Moreover, those involved in the interpretation of ndings
and the generation of recommendations (i.e., evaluation and steering
committees) felt empowered to offer solutions that led to improve-
ments and desired changes (Fetterman & Wandersman, 2005).
Facilitating democratic dialogue was not without its challenges. To
begin, facilitating democratic facilitation can be resource-intensive.
WRIEN staff in particular acknowledged the strain that additional
information gathering placed on their busy workloads. Consistent
with the experience of staff within other collaborative system change
evaluations (e.g., Janzen, Nelson, Hausfather, & Ochocka, 2007),
WRIEN staff saw the direct benet of evaluation while also not-
ing resource challenges in supporting its activities. Second was the
challenge of engaging the full range of stakeholders in dialogue.
As with other efforts to engage employers on immigrant matters
(Public Policy Forum, 2004), the business sector in particular proved
most challenging to engage. Consequently, in the rst year of the
evaluation few employers were included in the key informant method
(contrary to the sampling frame). However, participation in dialogue
increased over time as the business sector became more involved in
WRIEN and its evaluation and offered its unique insights into im-
migrant hiring and retention.
Focus on a Vision for the Common Good
It is not uncommon for community coalitions to be composed of
partners with disparate, if overlapping, agendas, expectations, and
interests (Baum, 2001). Such was the case with WRIEN. For example,
partners came into the intervention with differing histories—some
having grappled with the issue of immigrant employment for many
years if not decades, while others being relatively new to the is-
sue. Some partners were ready for a systems change intervention
(recognizing the limitations of individual-level efforts in addressing
systemic barriers to immigrant employment), while others were en-
trenched in a mindset of the primacy of individual-level efforts and
outcomes. In addition, some partners were primarily motivated in
joining WRIEN as a means of promoting social justice: working to

80 The Canadian Journal of Program evaluaTion
80
give immigrants equitable opportunity to participate in the labour
market and consequently in community life. Others were primarily
attracted to an economic imperative agenda, recognizing that an
aging population meant that the local (as well as national) economy
was entering a time of labour skill shortages that immigrants poten-
tially could ll. Finally, partners held signicantly different social
norms for pursuing healthy partnerships. Even the seemingly trivial
expectation of what constituted a fruitful meeting exposed deep dif-
ferences among partners (i.e., the brief early morning meetings with
rational decision-making and a bias for action expected by the busi-
ness community, contrasted with the constant connection to lived
experience and ample discussion to understand the depth of issues
expected by immigrant service providers).
It was within this context that the evaluation was able to help focus
the disparate partnership on a common vision. The evaluation acted
as a forum in which these diverse perspectives and interests were
aired and negotiated in a transparent and systematic way. All stake-
holder perspectives were sought as evaluation data. All stakeholders
were also invited to re-examine and collectively recommit to their
collective vision within the various multi-stakeholder feedback and
discussion formats. Still, keeping the initial common vision alive
across many stakeholder groups was difcult.
What proved helpful in this regard was to continually focus stake-
holders, despite their different motivations, on what they considered
to be the “common good.” In evaluation language this translates
to reaching a shared understanding of desired outcomes. The slo-
gan that stakeholders had crafted for the original Immigrant Skills
Summit (“strong lives, strong economy, strong community”) was
subsequently reframed as long-term goals that remained constant
throughout the evaluation (see bottom of the logic models in Figures
2 and 3). Such a three-pronged “common good” satised both social
justice and economic imperative agendas, while framing them as
interdependent outcomes within WRIEN’s “ofcial” program theo-
ry. Consequently, these long-term outcomes served as a launching
point for the ongoing discussion of what intermediary outcomes were
expected and what priority activities were needed to reach them.
Stakeholders could therefore see how their specic efforts t into the
broader (and shared) system change goals.
Using evaluations within system change interventions as a platform
to bring clarity around theory of change is certainly not unique to

8181
la revue Canadienne d’évaluaTion de Programme
WRIEN (e.g., Foster-Fishman & Long, 2009; Janzen et al., 2007; Ku-
bisch et al., 1998). And using evaluation data as a means of facilitat-
ing partnership commitment and activities is also well documented
(e.g., Natasi & Hitchcock, 2009). The important point to make here
is that evaluation that focuses on a vision of the common good offers
participants an opportunity to reectively link their collective knowl-
edge with their desired collective action. In other words, evaluation
can build a new sense of community among previously separated
stakeholder interests—a community that inspires people to work
together toward a common goal (Stringer, 2007).
CONCLUSION
In this article we discussed how evaluation can help shape and direct
CCIs. This was done by offering the Waterloo Region Immigrant Em-
ployment Network (WRIEN) as a case example to illustrate reec-
tive practice—how theory and practice can be intertwined through
evaluation. By framing our reections around ve facilitators that
interconnect theory and practice, and by referring to literature on
collaborative evaluation, we have placed our discussion within a
broader body of knowledge. This body of knowledge represents two
parallel and complementary knowledge bases; on the one hand what
is known about reective practice through multidisciplinary writ-
ings on participatory action research, and on the other hand what is
known about reective practice through collaborative inquiry within
the trans-discipline of evaluation. Although obviously not exhaus-
tive, our intent was to bridge the gap that often exists between these
two knowledge bases.
The WRIEN case study outlined in this article contributes to the
emerging discourse distinguishing formative from developmental
evaluation. Proponents of developmental evaluation (e.g., Gamble,
2008; Patton, 2008, 2011) emphasize that developmental evaluation
develops a newly emerged intervention model, while formative evalu-
ation improves an existing intervention model. Intervention models
need to be developed (rather than tested and tweaked) when their
contexts are complex with low stakeholder agreement and certainty
about outcomes. We agree that this distinction is an important one,
and view the WRIEN example as being clearly developmental. The
emphasis on emerging program theory within WRIEN explicitly in-
vited ongoing reective practice and allowed evaluators to position
themselves as part of what Patton calls the “adaptive management
team.” Although WRIEN had developed a baseline program theory,

82 The Canadian Journal of Program evaluaTion
82
this theory was signicantly renegotiated over time. The WRIEN
experience therefore suggests that perhaps it is less a matter of
whether an intervention model does or does not pre-exist (can an
intervention ever be entirely theory-free?) than it is a matter of how
loosely or tightly a model is held. From this angle the formative-de-
velopmental distinction could be conceived of as a continuum rather
than as two discrete categories.
There is a second contribution to the discussion about developmen-
tal evaluation. In this article we reected on WRIEN’s evaluation
through ve facilitators of reective practice that were gleaned from
past literature. While in his book Patton (2011) introduces develop-
mental evaluation as a new approach to evaluation (as distinct from
formative and summative evaluation), the WRIEN case example
demonstrates that using research as a means of facilitating ongo-
ing reective practice within complex environments is neither new
nor novel. Although the term “developmental evaluation” was not
yet coined in the literature at the beginning of WRIEN’s evaluation,
the evaluation can be seen to stand on the shoulders of a deep par-
ticipatory action research tradition that spans more than 60 years.
It also draws on the growing interest among evaluators pursuing
collaborative forms of inquiry that adopt constructivist and other
postmodern paradigms of research. These paradigms of research
are comfortable with moving beyond the objectivity, linear cause-
and-effect assumptions, and controlled environments of positivistic
science, particularly when evaluating innovation (Janzen & Wiebe,
2010; Nelson et al., 2010). Patton’s developmental evaluation does
add helpful connections to complexity theory and the emerging body
of knowledge related to social innovation. It may yet prove to become
a consensus label useful in articulating and promoting a postmodern
approach to evaluation. Time will tell. Still, regardless of which label
we use, we believe that CCIs provide fertile ground to further explore
the link between evaluation, reective practice, and interventions
that deal with complexity.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the WRIEN evaluation and
steering committees for their input and support during the evalua-
tion process, and Drs. Geoffrey Nelson and Terry Mitchell for com-
ments on previous section drafts.

8383
la revue Canadienne d’évaluaTion de Programme
REFERENCES
Alboim, N. (2009). Adjusting the balance: Fixing Canada’s economic immi-
gration policies. Toronto, ON: Maytree Foundation.
Alkin, M. C., & Taut, S. M. (2003). Unbundling evaluation use. Studies in
Educational Evaluation, 29, 1–19.
Amo, C., & Cousins, J. B. (2007). Going through the process: An examina-
tion of the operationalization of process use in empirical research on
evaluation. New Directions for Evaluation, 116, 5–26.
Baker, T. E. (2003). Evaluating community collaborations: An overview. In
T. E. Baker (Ed.), Evaluating community collaborations (pp. 1–18).
New York, NY: Springer.
Baum, H. S. (2001). How should we evaluate community initiatives? Jour-
nal of the American Planning Association, 67(2), 147–158.
Becker, A. J., Israel, B. A., & Allen, A. J. (2005). Strategies and techniques
for effective group processes in CBPR partnerships. In B. A. Israel,
E. Eng, A. J. Schulz, & E. A. Parker (Eds.), Methods in community-
based participatory research for health (pp. 52–72). San Francisco,
CA: Jossey-Bass.
Cargo, M., & Mercer, S. (2008).The value and challenges of participatory re-
search: Strengthening its practice. Annual Review of Public Health,
29, 325–350.
Centre for Community Based Research. (2010). Mobilizing Waterloo Region
around immigrant employment. Retrieved from www. community
basedresearch.ca/Page/View/Immigrant_Employment.html
Chavez, V., Duran, B., Baker, Q. E., Avila, M. M., & Wallerstein, N. (2008).
The dance of race and privilege in community based participatory
research. In M. Minkler & N. Wallerstein (Eds.), Community-based
participatory research for health (pp. 91–106). San Francisco, CA:
Wiley.
Chen, H.-T. (2005). Practical program evaluation: Assessing and improv-
ing planning, implementation, and effectiveness. Newbury Park, CA:
Sage.
Clare, R. C. (2006). Putting faith into action: A model for the North Ameri-
can middle class. Religious Education, 101(3), 368–389.

84 The Canadian Journal of Program evaluaTion
84
Conde-Frazier, E. (2006). Participatory action research: Practical theology
for social justice. Religious Education, 101, 321–329.
Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation.
Educational Evaluation and Policy Analysis, 14(4), 397–418.
Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. L. (2004). Integrating evalua-
tive inquiry into the organizational culture: A review and synthesis
of the knowledge base. Canadian Journal of Program Evaluation,
19(2), 99–141.
Dildar, Y., & Janzen, R. (2009). Waterloo Region Immigrant Employment
Network (WRIEN) nal evaluation report. Kitchener, ON: Centre for
Community Based Research.
Fals Borda, O. (1987). The application of participatory action-research in
Latin America. International Sociology, 2(4), 329–347.
Fetterman, D. M., & Wandersman, A. (2005). Empowerment evaluation
principles in practice. New York, NY: Guilford.
Foster-Fishman, P. G., & Long, R. (2009). The challenges of place, capacity
and systems change: The story of Yes We Can. Foundation Review,
1(1), 69–84.
Foster-Fishman, P. G., Nowell, B., & Yang, H. (2007). Putting the system
back into system change: A framework for understanding and chang-
ing organizational and community systems. American Journal of
Community Psychology, 39(3), 197–215.
Freire, P. (1970). Pedagogy of the oppressed. New York, NY: Seabury Press.
Gamble, J. A. A. (2008). A developmental evaluation primer. Montreal, QC:
J.W. McConnell Family Foundation.
Heron, J., & Reason, P. (1997). A participatory inquiry paradigm. Qualita-
tive Inquiry, 3(3), 274–294.
Hogarth, K., Janzen, R., & Cushing, S. (2007). Year one evaluation report for
Waterloo Region Immigrant Employment Network (WRIEN). Kitch-
ener, ON: Centre for Community Based Research.
Israel, B., Eng, E., Schulz, A. J., & Parker, E. A. (2005). Introduction to
methods in community-based participatory research for health. In
B. A. Israel, E. Eng, A. J. Schulz, & E. A. Parker (Eds.), Methods in

8585
la revue Canadienne d’évaluaTion de Programme
community-based participatory research for health (pp. 3–26). San
Francisco, CA: Jossey-Bass.
Israel, B., Schulz, A., Parker, E., & Becker, A. (1998). Review of community-
based research: Assessing partnership approaches to improve public
health. Public Health, 19, 173–202.
Janzen, R. (2011). Reaching out to multicultural neighbours: Stories that
evaluate and encourage innovative church outreach. Unpublished
dissertation. Wilfrid Laurier University, Waterloo, ON.
Janzen, R., & Dildar, Y. (2008). Year two evaluation report for the Waterloo
Region Immigrant Employment Network (WRIEN). Kitchener, ON:
Centre for Community Based Research.
Janzen, R., Hatzipantelis, M., & Hogarth, K. (2005). Immigrant skills sum-
mit: Waterloo Region conference proceedings. Kitchener, ON: Centre
for Research and Education in Human Services.
Janzen, R., Hatzipantelis, M., Vinograd, J., & Kellerman, M. (2004). Good
practice and resource guide: Community needs assessments and serv-
ice evaluations in Military Family Resource Centres. Kitchener, ON:
Centre for Community Based Research (formerly Centre for Re-
search and Education in Human Services).
Janzen, R., Nelson, G., Hausfather, N., & Ochocka, J. (2007). Capturing
system level activities and impacts of consumer-run organizations:
Methods that inform future action. American Journal of Community
Psychology, 39(Special issue on systems change evaluation), 287–299.
Janzen, R., Walton-Roberts, M., & Ochocka, J. (in press). Immigrant integra-
tion and inclusion in Waterloo Region. In J. Biles & C. Andrew (Eds.),
Immigration, integration and inclusion in Ontario cities. Montreal,
QC: McGill-Queen’s University Press.
Janzen, R., & Wiebe, D. (2010). Putting God in the logic model: Developing
a national framework for the evaluation of faith-based organizations.
Canadian Journal of Program Evaluation, 25(1), 1–26.
Kemmis, S., & McTaggart, R. (2005). Participatory action research. In N. K.
Denzen & Y. S. Lincoln (Eds.), Handbook of qualitative research (3rd
ed.; pp. 559–603). Thousand Oaks, CA: Sage.
Kubisch, A. C., Fulbright-Anderson, K., & Connell, J. P. (1998). Evalu-
ating comprehensive community initiatives: A progress report. In

86 The Canadian Journal of Program evaluaTion
86
K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell (Eds.), New
approaches to evaluating community initiatives: Volume 2–Theory,
measurement, and analysis (pp. 1–14). Washington, DC: Aspen In-
stitute.
Lafferty, C. K., & Mahoney, C. A. (2003). A framework for evaluating com-
prehensive community initiatives. Health Promotion Practice, 4(31),
31–44.
Lewin, K. (1948). Action research and minority problems. In G. W. Lewin
(Ed.), Resolving social conicts (pp. 201–216). New York, NY: Harper
& Row.
Mackenzie, M., & Blamey, A. (2005). The practice and the theory: Lessons
from the application of a theory of change approach. International
Journal of Theory, Research and Practice, 11(2), 151–168.
McFadden, P., & Janzen, R. (2007). The importance of immigrants to Water-
loo Region’s Prosperity: A dynamic collaborative community process.
Our Diverse Cities, 4(Fall), 104–107. Retrieved from http://canada.
metropolis.net/publications/index_e.htm
McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. In J. S.
Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical
program evaluation (pp. 7–32). Thousand Oaks, CA: Sage.
Mertens, D. M. (2009). Transformative research and evaluation. New York,
NY: Guilford Press.
Miedema, J. M., & Vandebelt, D. (2006). Immigrants and growth: A look at
health and employment in Waterloo Region. Waterloo, ON: Region of
Waterloo Public Health.
Minkler, M., & Wallerstein, N. (2008). Introduction to community based
participatory research. In M. Minkler & N. Wallerstein (Eds.), Com-
munity-based participatory research for health (pp. 5–24). San Fran-
cisco, CA: Wiley.
Nastasi, B. K., & Hitchcock, J. (2009). Challenges of evaluating multilevel
interventions. American Journal of Community Psychology, 34(4),
360–376.
Nelson, G., Janzen, R., Ochocka, J., & Trainor, J. (2010). Participatory action
research and evaluation with mental health self-help groups and or-
ganizations: A theoretical framework. In L. Brown & S. Wituk (Eds.),

8787
la revue Canadienne d’évaluaTion de Programme
Mental health self-help: Consumer and family driven initiatives (pp.
39–58). New York, NY: Springer.
Nelson, G., Ochocka, J., Griffen, K., & Lord, J. (1998). “Nothing about me,
without me”: Participatory action research with self-help/mutual aid
organizations for psychiatric consumer/survivors. American Journal
of Community Psychology, 26, 881–912.
Nelson, G., Ochocka, J., Janzen, R., Trainor, J., & Lauzon, S. (2004). A
comprehensive evaluation framework for mental health consumer/
survivor organizations: Values, conceptualization, design, and action.
Canadian Journal of Program Evaluation, 19(3), 29–53.
Newbrough, J. R. (1995). Toward community: A third position. American
Journal of Community Psychology, 23(1), 9–37.
Ochocka, J., Moorlag, E., & Janzen, R. (2010). A framework for entry: PAR
values and engagement strategies in community research. Gateways:
International Journal of Community Research & Engagement, 3,
1–19.
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Beverly Hills,
CA: Sage.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity con-
cepts to enhance innovation and use. New York, NY: Guilford Press.
Public Policy Forum. (2004). Bring employers into the immigration debate.
Ottawa, ON: Author.
Reason, P. (2006). Choice and quality in action research practice. Journal of
Management Inquiry, 151(2), 187–203.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program
evaluation: Theories of practice. Newbury Park, CA: Sage.
Stake, R. (1978). The case study method of social inquiry. Educational Re-
searcher, 7(2), 5–8.
Stringer, E. T. (2007). Action research (3rd ed.). Thousand Oaks, CA: Sage.
Tashakkori, A., &. Teddlie, C. (2003). The past and future of mixed methods
research: From data triangulation to mixed model designs. In A.
Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social
& behavioral research (pp. 671–701). Thousand Oaks, CA: Sage.

88 The Canadian Journal of Program evaluaTion
88
Taylor, A., & Botschner, J. (1998). Evaluation handbook. Kitchener, ON:
Centre for Community Based Research (formerly Centre for Re-
search and Education in Human Services).
Travers, R., Wilson, M. G., Flicker, S., Guta, A., Bereket, T., McKay, C., …
Rourke, S. B. (2008). The greater involvement of people living with
AIDS principle: Theory versus practice in Ontario’s HIV/AIDS com-
munity-based research sector. AIDS Care, 20(6), 615–624.
Wallerstein, N., & Duran, B. (2003). The conceptual, historical, and practice
roots of community based participatory research and related partici-
patory traditions. In M. Minkler & N. Wallerstein (Eds.), Community-
based participatory research for health (pp. 25–46). San Francisco,
CA: Wiley.
Wayland, S. (2006). Unsettled: Legal and policy barriers for newcomers to
Canada. Ottawa, ON: Community Foundations of Canada and the
Law Commission of Canada.
Wayland, S. (2007). The Learning Exchange: Disseminating good ideas:
Skilled immigrants in the labour market. Report prepared for CIC
and the J.W. McConnell Foundation.
Rich Janzen is Research Director at the Centre for Community
Based Research (www.communitybasedresearch.ca). Rich recently
completed his Ph.D. (Community Psychology) at Wilfrid Laurier
University.
Daniela Seskar-Hencic is the former Manager of Planning, Evalu-
ation and Epidemiology at the Regional Municipality of Waterloo.
Presently Daniela is Associate Director, Institutional Analysis and
Planning at the University of Waterloo.
Yasir Dildar is Senior Researcher at the Centre for Community
Based Research. Yasir has M.A.s in sociology and development stud-
ies.
Peter McFadden is the former Executive Director of the Waterloo
Region Immigrant Employment Network (WRIEN).