Conference PaperPDF Available

Knowledge brokers in action. A game-based approach for strengthening evidence-based policies

Authors:

Abstract and Figures

Public policies need research results in order to effectively address the complex socioeconomic challenges (so-called: evidence-based policies). However there is a clear gap between producing scientific expertise and using it in public decision-making. This "know-do" gap is common in all policy areas. Knowledge brokering is a new and promising practice for tackling the challenge of evidence use. It means that selected civil servants play the role of intermediaries who steer the flow of knowledge between its producers (experts and researchers) and users (decision makers and public managers). Knowledge brokering requires a specific combination of skills that can be learnt effectively only by experience. However this is very challenging in the public sector. Experiential learning requires learning from own actions-often own mistakes, while public institutions tend to avoid risk and are naturally concerned with the costs of potential errors. Therefore, a special approach is required to teach civil servants. This article addresses the question of how to develop knowledge brokering skills for civil servants working in analytical units. It reports on the application of a simulation game to teach civil servants through experiential learning in a risk-free environment. Article (1) introduces the concept of knowledge brokering, (2) shows how it was translated into a game design and applied in the teaching process of civil servants and (3) reflects on further improvement. It concludes that serious game simulation is a promising tool for teaching knowledge brokering to public policy practitioners.
Content may be subject to copyright.
Paper for The International Simulation and Gaming Association (ISAGA)
The winner of Best Paper Award at 46th ISAGA Conference
Hybrid Simulation and Gaming in the Network Society
July 17th July 21st, 2015 | Kyoto, Japan
Knowledge brokers in action.
A game-based approach for strengthening evidence-
based policies
Karol Olejniczak1, Tomasz Kupiec2, Igor Widawski3
Abstract: Public policies need research results in order to effectively address
the complex socio-economic challenges (so-called: evidence-based policies).
However there is a clear gap between producing scientific expertise and using
it in public decision-making. This "know-do" gap is common in all policy
areas. Knowledge brokering is a new and promising practice for tackling the
challenge of evidence use. It means that selected civil servants play the role of
intermediaries who steer the flow of knowledge between its producers
(experts and researchers) and users (decision makers and public managers).
Knowledge brokering requires a specific combination of skills that can be
learnt effectively only by experience. However this is very challenging in the
public sector. Experiential learning requires learning from own actions -
often own mistakes, while public institutions tend to avoid risk and are
naturally concerned with the costs of potential errors. Therefore, a special
approach is required to teach civil servants.
This article addresses the question of how to develop knowledge brokering
skills for civil servants working in analytical units. It reports on the
application of a simulation game to teach civil servants through experiential
learning in a risk-free environment. Article (1) introduces the concept of
knowledge brokering, (2) shows how it was translated into a game design and
applied in the teaching process of civil servants and (3) reflects on further
improvement. It concludes that serious game simulation is a promising tool
for teaching knowledge brokering to public policy practitioners.
Keywords: knowledge broker, knowledge use, public management, teaching civil servants,
serious games
1 Olejniczak, K.: Centre for European Regional and Local Studies
(EUROREG) - University of Warsaw, Warsaw, Poland, k.olejniczak@uw.edu.pl
2 Kupiec, T.: Evaluation for Government Organizations - EGO s.c., Warsaw,
Poland, t.kupiec@evaluation.pl
3 Widawski, I.: Pracownia Gier Szkoleniowych PGS, Warsaw, Poland,
i.widawski@pracowniagier.com
2 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski Karol Olejniczak, Tomasz Kupiec, Igor Widawski
1 Introduction
1.1 The challenge of knowledge brokering
Decision-makers and public managers need research results in order to conduct
effective public interventions that serve citizens and improve socio-economic
development. The usefulness of evidence-based policies is confirmed by both
modern literature on public policies and the practice of public management
(Banks, 2009; Cartwright & Hardie, 2012; Nutley, Walter, & Davies, 2007;
Shillabeer, Buss, & Rousseau, 2011).
However, there is a clear gap between producing research studies (including
applied expertise such as evaluations) and using their results in decision-making
(Cartwright, 2013; Majone, 1992; Shulha, Cousins, 1997; Weiss, Bucuvalas,
1980).
Recent literature on evidence use in public policies points to "knowledge
brokering" as a promising strategy for tackling the "know-do" gap (Dobbins,
2009; Lomas, 2007; Waqa, 2013; Clark, 2005; Oliver, 2014). Knowledge broker-
ing requires a set of specific skills: (1) recognizing the knowledge needs of policy
actors, (2) acquiring credible studies, (3) reaching users with appropriate dissemi-
nation strategies, and (4) combining the results of different studies into an
evidence-based foundation for decisions.
Government agencies try to build the knowledge brokering skills of their
personnel. However, this is very challenging in the public sector. First, knowledge
brokers in the public sector operate on the brink of two rationalities, where the
rational, evidence-based approach collides with the logic of political negotiations
(Bots, Wagenaar, & Willemse, 2010; Sanderson, 2002). Second, effective learning
requires experimentation and learning from one’s own actions, often own
mistakes, while public institutions tend to avoid risk and are naturally concerned
with costs of potential errors (Barrados & Mayne, 2003; Hood, 2007). Therefore,
a special approach is required to teach these skills.
1.2 Article aim and contribution to the current practice of
teaching with games
This article addresses the problem of teaching civil servants knowledge bro-
kering skills, with the use of experiential learning in a risk free environment. The
article reports on the application of a specially designed simulation tabletop game
for teaching Polish civil servants. It is an example of gaming research in policy
area that uses serious games for policy implementation and organizational change
(compare: Caluwe, Geurts & Kleinlugtenbelt, 2012).
3 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
The article brings the following new contributions to current practice and
literature: In terms of the topic, the article introduces serious gaming as a new tool
for addressing an important public policy issue - effective research utilization in
decision-making. In terms of players, the article illustrates how the game was used
to teach a very conservative and demanding type of learners - public civil
servants. In terms of application, the article shows how the game can effectively
address the challenge of experiential teaching in an organizational environment
that has low tolerance of risk and experimentation.
The article has practical value for two groups of audiences. For professionals
who teach evaluation and for public sector officials it shows an innovative way of
approaching training. For experts in gaming and simulation it offers an illustration
of how the complex reality of public programs delivery can be turned into a game
design without loosing its connection with reality.
1.3 Method
The reported game application is grounded in sound, scientific evidence. The
content of the game is based on: (1) seven years of the Academy of Evaluation
post-graduate program for Polish senior civil servants, (2) a systematic review of
literature on evaluation use and knowledge brokering (over 900 research articles)
and (3) empirical research of evaluation units practices (a survey of Polish units,
interviews, consultations and focus groups with representatives of European and
American evaluation units) (Olejniczak, Kupiec & Raimondo, 2014; Olejniczak,
Raimondo & Kupiec, 2014).
The initial workshop and game design was developed during the ISAGA 2014
summer school. The game mechanics were tested during two game sessions with
10 representatives of Polish evaluation units. After each session both the content
and form of the game were modified. Further calibration of the workshop content
was performed during a session with MA students of regional development
studies.
The article is divided into three sections. In the next section the theoretical
framework of the simulation is discussed. The second section explains how the
theory was translated into a game design and applied in the teaching process of
civil servants. The final section presents an initial evaluation of the game’s
effectiveness and reflection on further improvements.
4 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
2 Theoretical framework
The content of the game focuses on the practice of knowledge brokering. It has
been grounded in extensive literature and empirical research on evidence use in
decision-making and knowledge brokering.
Knowledge Brokers (KB) are units in government that serve as an intermediary
between the worlds of science, politics, and public interest (Gutierrez, 2010;
Fischer, 2003). These are always persons or a group of people, not an automated
system or a database (McAneney et al., 2010).
The goal of knowledge brokers is to help decision makers in acquiring and
using credible knowledge for better planning and implementing of public
interventions. Thus, successful knowledge brokering leads to effective public
intervention and social betterment.
Figure 1 presents the logic of knowledge brokering activities as a factor that
improves the effectiveness of public interventions. The core narrative of this logic
is as follows: actors involved in running public policies have certain knowledge
needs at different stages of policy interventions. IF the knowledge broker executes
a set of actions that provide those policy actors with useful knowledge, THEN the
actors, by absorbing that knowledge, will deepen their understanding of public
intervention, AND THEN they will plan and implement intervention in a way that
better serves the public interest. As we can see, the job of knowledge brokers is
mainly about recognizing the knowledge needs of decision-makers, finding and
combining evidence and experiences from different sources, translating them into
the language of practice and introducing them to the world of practitioners (Lin,
2012; Mavoa et al. 2012; Willems et al., 2013).
There are four points that are crucial for the way knowledge brokers operate.
First, the focal point is public intervention or, to be more specific, knowledge
needs concerning a particular intervention and the problems and issues arising
from it. This means that brokers have to follow the policy implementation cycle.
Second, knowledge needs are always articulated by a particular actor politi-
cians, senior civil servants or managers. So, there is a clearly defined group of
knowledge users, in other words - clients of the brokers.
Third, success in brokering depends on the configuration of factors. The broker
has to match different elements with each other: (a) the type of knowledge needed
with the method of acquiring it (research design), (b) client types with knowledge
feeding methods and (c) timing of knowledge delivery.
Fourth, there is a certain degree of uncertainty between the brokers’ action and
their impact on a decision. Research evidence is only one of many factors in-
fluencing the decision-making process. Other factors are political rationality,
organizational dynamics, characteristics and reasoning processes of the knowledge
users. However the better the quality of brokers' activities and the stronger the
evidence-base they present, the higher the chances of positive influence.
5 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Fig. 1. The logic of knowledge brokering (Olejniczak, Raimondo & Kupiec, 2014)
Public
interventions are
better designed
and more likely
to successfully
serve citizens
Knowledge needs and
knowledge users
ACTIVITIES AND SERVICES
IF knowledge brokers perform certain activities they will
provide a high quality service to knowledge users...
Public interventions aim
to address certain socio-
economic issues
Those interventions are
designed and
implemented by different
public policy actors
Those actors have
certain KNOWLEDGE
NEEDS at different
stages of intervention
cycle
MECHANISM
…THEN they will trigger
desired behaviors of
knowledge users...
CHANGE
…AND THEN a
positive effect will
occur
INSTITUTIONAL DETERMINANTS
of decision-making process
PSYCHOLOGICAL DETERMINANTS
Human heuristics and biases
Acquiring
credible
knowledge
Building networks with producers and users
of knowledge
Accumulating knowledge over time
Defining
knowledge
needs
Feeding
knowledge
to users
Users
change
their
mental
models
Users
understand
message
Users use
knowledge
in practice
Knowledge needs of
users addressed at
the right moment
Credible knowledge
delivered
Knowledge
transferred to users
in an accessible way
6 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
3 Practical application
3.1 Game description
The overall aim of the game was to teach participants the key skills of
knowledge brokering required for playing the role of an intermediary who steers
the flow of knowledge between its producers and users. These skills are: (1)
understanding knowledge needs, (2) acquiring credible knowledge, (3) feeding
knowledge effectively to users, (4) building an evidence-based foundation for
public interventions and (5) managing an analytical unit.
The initial idea of the game was to allow civil servants see the simplified me-
chanisms of decision-making and reflect on them, while at the same time keeping
it concrete, not too abstract or out of their comfort zone. Therefore, the decision
was not to use a metaphor, but instead, to recreate in a game the key operational
rules and elements of the system that are familiar to civil servants. The challenge
was to transfer the key elements of the system into a game while at the same time
reducing the complexity of the real life operations of regional policies. Eventually,
the following narrative was developed:
Participants are divided into 6 groups. Each group manages an analytical unit
in a region. Their mission is to support decision-makers with expertise in imple-
menting four types of socio-economic interventions. These are: combating single-
mothers' unemployment, developing a health care network, revitalizing a down-
town area, and developing a public transportation system for a metropolitan area.
With each turn in the game knowledge needs appear for each intervention.
They can: relate to a descriptive or diagnostic issue of the problem tackled by the
intervention (know about the issue), explore the effects of the implemented or
planned solutions (know what works), inquire about the explanation for the
success or failures of the particular project (know why things work) or refer to
procedural, managerial issues (know how to implement).
Knowledge needs take the form of concrete questions that relate to issues
arising during different implementation stages of these projects. For example, in a
project on the public transportation system, during its implementation phase, the
following questions arise: (a) How do habitants of the metropolis use the new
network of transport connections (including the different transportation modes
available) provided by the public authorities? (b) What barriers do disabled
persons experience when using the newly introduced public transportation
system? (c) How to change prices for public transport tickets and charges for
parking in the downtown area to encourage citizens to switch from driving their
private vehicles to using public transportation?
7 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Over the course of the game players have to react to 19 different knowledge
needs, often appearing simultaneously in different public interventions. Players
have to: (1) contract out studies with an appropriate research design, (2) choose
key users of the study and (3) choose methods for feeding knowledge to users.
The spectrum of options available to players is presented in Table 1.
Table 1. Options available to players
Item
Available options
Research
designs
(1) Meta-analysis
(2) Experiments and quasi-
experiments
(3) Statistical study
(4) Simulation game
(5) Theory-driven evaluation
(6) Case study
(7) Participatory approach
(8) Descriptive study
Knowledge
users
(1) Politician
(2) Head of a department
(3) Project manager
Knowledge
feeding
methods
Forms of presentation
1. Policy brief
2. Recommendation table
3. Logic model
4. Video presentation or
iconographic
5. Argument map
6. Dashboard
Channels of dissemination
7. Small discussion meeting
8. Big meeting or conference
9. Contact through advisors
10. Personal contact with user
The choices of players are determined by the resources available to them: the
number of staff in their units and the time required to complete each task. Players
can be proactive and invest their resources in networking (to discover knowledge
needs in advance) or archive searching (to find already existing studies). Players
delegate staff members to these tasks. While networking or archive searching it is
8 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
impossible for that particular staff member to engage in any other activity during
the current round (e.g. report preparation).
After each turn, each group receives detailed feedback that includes three ele-
ments: (1) a percentage on how well the team matched research designs to
knowledge needs and feeding methods to users; the higher the match, the higher
are the chances that knowledge will be used by decision-makers, (2) information
on the final effect: if a policy actor made a decision based on delivered knowledge
or other premises (e.g. political rationale), (3) hints on good research designs,
types of users and feeding methods for future turns.
Groups of players compete with each other. Depending on how well they
match research designs, users and feeding methods they receive up to 100 points
per knowledge need. Teams accumulate points throughout the game and the
winning team is the one with the highest score. However, there is also another
way to assess playersperformance. Each result for an individual knowledge need
(ranging from 0 to 100) is a probability rate that determines what is the chance
that the report will be actually used by the decision maker. The algorithm checks,
based on this probability, if a particular report will be used by a decision maker
and then notes it in a different section of the team score. In effect, every team has
two types of score: the first based on accumulation of points throughout the game
and the second that informs players how many reports were actually used. The
second type of scoring involves a strong element of randomness and luck (a team
might succeed even if a report was worth only 20 points which gives it a 0.2
chance of being used), while the first one reflects how well players can prepare
reports. That is why facilitators put more emphasis on the first type of scoring, but
at the same time they also remind participants that there is always an element of
luck and randomness in decision makers use of reports for policy processes.
The learning goals of the workshop were grounded in research literature on
knowledge brokering (compare previous section). They were translated into the
list presented in Table 2.
Table 2.Knowledge brokers' skills translated into game learning goals
Key skills of
Knowledge Broker
What it means for players
(1) Understanding
knowledge needs
(1a) Players recognize different stages of an intervention
(1b) Players recognize involvement of different actors at
different intervention stages
(1c) Players recognize different types of knowledge needs
and the form of questions in which they are articulated
(2) Acquiring credi-
ble knowledge
(2a) Players match research questions with optimal
research designs
(2b) Players match research questions to types of
interventions
9 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
(3) Feeding
knowledge
effectively to users
(3a) Players match feeding methods to the types of users
(3b) Players recognize and combine two different types of
feeding methods - those related to communication forms
and those related to channels of dissemination
(4) Building
evidence-based
foundations for
public interventions
(4a) Players combine results of different studies to build a
coherent argument - a knowledge stream
(4b) Players understand that evidence is only a part of
decision-making; other considerations (e.g. politics) can
heavily influence the result of their mission
(5) Managing an
analytical unit
(5a) Players understand the whole sequence of KB unit
activities
(5b) Players manage the time and staff of their unit
(5c) Players know that a proactive approach pays off -
looking for knowledge needs in advance gives more time
for strategy development
3.2 Game’s structure and mechanics
The simulation is structured as a progression game. It relies on a tightly con-
trolled sequence of events that offers many predesigned challenges (Adams &
Dormans, 2012). Each team follows the same scenario, operating within a prede-
fined timeline and with access to a certain number of given resources.
Table 3.Sequence of the round
Step of the
round
Description
1. Timeline
update
Each team moves the time marker to indicate the current round.
Every round is one month.
2. Resource
recovery
Each team collects all the resources taken from them in the
previous round. Recovered resources include: networker,
archivist, components of a finished report: staff members,
research designs, knowledge users, knowledge feeding methods.
3. Event
In each round there is an event that influences the current state
of the evaluation unit. Some of the events are helpful to the
player (e.g. recruitment of new staff members) and some are
harmful (e.g. delays in report preparation, blocked resources
etc.). A number of events are formulated as an alternative:
players can do A or B. Each team is obliged to make a choice
between these two options and faces the consequences of the
chosen strategy. The order of events is strictly planned and
every group playing the game will encounter the same
10 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
challenges.
4. Delivery of
finished reports
Every finished report should be delivered with all the resources
placed on it to the facilitator. All the data from the report is then
transferred into the system and the score for the team is counted.
5. Distribution
of new
knowledge
needs
Each team receives a predefined number of new knowledge
needs designed for a current round. If a team has already
collected knowledge needs in advance (thanks to a networker),
it does not receive a knowledge need designed for the next
round. For example: in round 2 players receive 4 new
knowledge needs (1 for each intervention); in round 2 they send
a networker to collect knowledge needs in advance for
intervention A. Then at the beginning of the round 3 during the
phase “Distribution of new knowledge needs” they receive
knowledge needs for intervention B, C and D, but not A (they
already have it).
6. Action
Each team begins their work on new reports and can send a
networker or archivist.
The progression structure was chosen for three reasons. The first reason was
related to one of the learning goals (goal 5a). It was to introduce players to the full
process of knowledge brokering within analytical units. This is a well-defined
process (both in literature and in practice) and it is based on certain intermediary
phases and steps. The progression structure of the game allowed designers to
recreate the chronology of this process, so that the players can understand the
logic behind the procedures that they will encounter in real life situations.
The second reason was an approach to facilitation. Due to the complex nature
of the knowledge brokering process itself, it was important to grant the facilitator
tools for easy control over the game play. The tightly controlled sequence of
events and predefined progression of the game is helpful for having an overview
of the current situation and enhances the ability of the facilitator to identify
challenges and problems that players might face at a particular moment. This
knowledge is crucial in terms of ongoing observation, assistance and providing
feedback.
The third reason for the progression game structure was flow and learning. In
order to design an engaging learning process it is necessary to adjust the level of
challenges to the skills of participants and keep the right balance between the two
as the game progresses (Pavlas, 2010). Control over the exact order of incoming
‘knowledge needs’ and events are necessary in a game like KB to keep players in
a state of flow instead of anxiety or boredom. Even a minor disruption or the
wrong combination of resources, events and time given for a round might strongly
influence the stability of the learning process.
The game of knowledge brokering uses five types of mechanics. These are:
achievements, countdown, resource management, collaboration and unexpected
events. We briefly discuss them below.
11 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
(1) Achievements
The psychological drive of achievement is well known and often used in both
offline and online games (Felicia, 2011). A system of achievements motivates
players to perform a certain number of specific actions and provides automatic
feedback. In most cases, it is accompanied by some sort of progression mechanic
(e.g. progression bar), which helps players to notice how many actions of a certain
type should be performed to gain an achievement. In the KB game there are 19
achievements. These are 19 knowledge needs that need to be understood and
resolved by the player in a certain amount of time. Players are presented with up
to six steps to complete the report and deliver it to the decision maker. These steps
may be considered as a form of a progression bar that a player needs to complete
to gain an achievement. The player can decide whether to have a three, five or
even six step progress bar. Every additional step increases the probability that the
completed report will be used by a policy actor and so increases the quality of a
gained achievement. After delivering a completed report, players receive a
feedback form that informs them about the efficiency and the result of their work.
(2) Countdown
In many games a countdown mechanism is used to add more frenetic activity
(Penenberg, 2013). It forces players to accelerate their decision making process
and engages them on an emotional level. There are two ways in which a
countdown mechanic is used in the KB game. First, the main element of the board
is a calendar that sets deadlines for particular tasks. It is an axis of players’
activity that demands their attention and frames their experience. With every
round a special pawn is moved to indicate that the time is passing and there is not
much left to prepare new reports. Players need to constantly keep an eye on the
calendar and adapt to the current situation. The second countdown mechanism is
the set time limit for a round (from 7 to 20 minutes). This mechanism is still being
calibrated to the exact amount of time to create a “countdown effect” and generate
engagement.
(3) Resource management
The main resources in the game are the pawns that represent the staff of the
evaluation unit. Players can send a staff member to perform one of the listed
actions: (a) prepare report for a specific knowledge need, (b) provide additional
feeding methods to a report, (c) browse and collect materials from
archives/databases, (d) network with decision makers and acquire knowledge
needs in advance and (e) solve unexpected problems and deal with difficulties.
The choices that players make in terms of resource management determine their
final scores and to some extent influence the pace of the game (use of networker).
The goal of this mechanic is to enhance strategic thinking about real-life
constraints and to present various activities that might be performed within an
analytical unit.
12 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
(4) Collaboration
There are at least two types of in-game collaboration: the situation where
success in a game action is achieved more quickly when played collaboratively
and collaboration through discussion of game objectives (see: Washmi et al.,
2014). In the KB game both types of collaboration are included. At the beginning,
players receive five different pieces of information that describe various elements
of the game in detail (e.g. research designs, policy actors, feeding methods,
interventions and general rules). It is very challenging for one person to
comprehend all the delivered knowledge at once and perform all the necessary
actions within the given time limit. Well-organized teams split the responsibilities
between their members, so that each player specializes in a certain type of skills
(e.g. research design specialist) and collaborates with his or her colleagues. That
allows teams to complete tasks quickly and efficiently. At the same time players
need to have a general overview of the game’s objectives and together discuss
their overall strategy (like use of resources or dealing with unexpected situations).
This kind of collaboration also enables the players to learn from one another
instead of just from materials or the facilitator.
(5) Unexpected events
There are a number of unexpected events that take place between the rounds
and influence the gameplay. Some of them are helpful and some obstruct a
player’s efforts. A few events are presented in the form of a dilemma in which
players have to choose between alternatives. Each team has to estimate which
alternative will better fit their current strategy and will eventually pay the
predefined cost of their choice. As J. Schell (2014) put it: "Risk and randomness
are like spices. A game without any hint of them can be completely bland, but put
in too much and they overwhelm everything else." The KB game has a progression
structure with a predefined scenario that determines the specific order of incoming
‘knowledge needs’ and events. However, from a player’s point of view, the events
are unexpected and bring a sense of randomness that makes a game more
unpredictable and interesting.
4 Conclusions and future steps
The workshop session was conducted with 16 participants divided into 6 teams,
during 1 day’s training, from 10 am till 3 pm. The results of the workshop have
been evaluated based on the game results and discussion with participants. The
findings are summarized in Table 4.
13 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Table 4. Assessment of obtained learning goals
Key skills of Knowledge Broker
Discovered
by players
Missed by
players
(1) Understanding knowledge needs
(1a) Recognizing stages of the intervention
X
(1b) Recognizing actorsinvolvement
X
(1c) Translating needs into questions
X
(2) Acquiring credible knowledge
(2a) Matching questions with research designs
X
(2b) Matching designs to topics of intervention
X
(3) Feeding knowledge effectively to users
(3a) Matching feeding methods to users
X
(3b) Combining forms with channels
X
(4) Building evidence-based foundations
(4a) Building a coherent argument
X
(4b) Understanding limited influence
X
(5) Managing an evaluation unit
(5a) Scope and sequence of KB activities
X
(5b) Management of resources - time & staff
X
(5c) Using a proactive approach
X
Table 4 shows that the simulation successfully addressed most of the learning
goals. In the comments players also underlined that the game project (fit reality
well) was realistic, especially with regard to time pressure and the randomness of
political influence. Elements of the game mechanics such as comparisons between
teams and feedback after each round also worked well.
However, what emerged from the results is the fact that players clearly missed
the issue of knowledge credibility. For this brokering skill players were not able to
move beyond reactive behaviors and create mental models that would allow them
to grasp the systemic relation between research questions and research designs.
When asked about this issue, participants pointed at two aspects. First, the issue of
research design was relatively new to them. Although it is well established in
research practice, it is an emerging issue in the practice of government analytical
units. Second, teams felt they did not have enough time to properly analyze,
discuss and reflect fully on the feedback that arrived during the sessions.
These results lead the authors of the game to the conclusion that the game
design works well but that it should be integrated into a more coherent educational
experience. Therefore, three further improvements in the workshop design are re-
quired. First, players should be provided with a preparatory reader that includes
materials and examples of research design in the practice of public policy studies.
This would allow players to get familiar with this new and challenging concept.
14 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Second, teams should be given more time for their internal discussion after
getting the feedback in each round. This would allow them to proceed with more
group inquiry of system patterns and search for explanations.
Third, facilitators of the workshop should devote more attention in debriefing
sessions to both form and content. The workshop should include at least three
debriefing sessions, not only one at the end of the game. They could be designed
as mini-lectures with a questions and answers part (Q&A). They would be aimed
at group reflection on effective strategies of knowledge brokering. Participants,
guided by questions and comments posed by the facilitator (a) could discover the
relations and mechanisms underlying the dynamics of evidence use in public
decision-making, (b) would reflect on their own strategies implemented during
game (c) could develop new solutions to be tested further in the course of the
game.
This last discovery from the workshop is in line with recent literature that
underlines the importance of proper debriefing for the experiential learning and
reasoning of adult professionals (Crookall, 2010; Kato, 2010; Kriz, 2010). An idea
for a modified workshop agenda is presented in Table 5.
Table 5. Modified agenda of the workshop on knowledge brokering
10 minutes Introduction to the workshop aim
30 minutes Explanation of the rules of the game
20 minutes Training round 1 and clarification
30 minutes Rounds 2-3
10 minutes Break
10 minutes Presentation of partial results
30 minutes First debriefing session (mini-lecture on knowledge needs and
research designs, Q&A, group internal deliberation)
40 minutes Rounds 4-6
30 minutes Break
10 minutes Presentation of partial results
20 minutes Second debriefing session (mini-lecture on types of users and feeding
methods, Q&A, group internal deliberation)
40 minutes Rounds 7-10
15 minutes Break
15 minutes Presentation of final results and choice of the winners
30 minutes Final debriefing session & take away points for real-life practice
30 minutes Evaluation of the game the learning process itself
To conclude, the application of a serious game simulation proved to be a
promising tool for teaching knowledge brokering to public policy practitioners.
The game structure and mechanics worked well. The workshop structure (team
internal reflection after feedback, debriefing) requires further calibration to create
a fully integrated and experiential learning experience.
15 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Acknowledgements The simulation game is the result of the joint undertaking of two Polish
companies: Evaluation for Government Organizations (EGO s.c.) and Pracownia Gier
Szkoleniowych (PGS). The authors would like to express their gratitude to their colleagues
involved in the design and testing of the game: Łukasz Kozak (PGS), Jakub Wiśniewski
(PGS), Joanna Średnicka (PGS), Bartosz Ledzion (EGO s.c.) and Jagoda Gandziarowska-
Ziołecka (PGS).
The development of this game was a two-years process. The authors would like to thank all
those involved at different stages of the undertaking: Estelle Raimondo (George
Washington University) who coauthored the empirical research on knowledge brokering,
the group of participants from ISAGA Summer School Delft 2014 who contributed to the
initial design of the game Stephan van Ijperen, Wouter van der Horst, Marijn Knulst,
Stefania Munaretto, Nicole Plasschaert, and a number of Polish civil servants who
participated in the testing of the game and provided us with valuable feedback. Last but
not least the authors would like to thank two anonymous ISAGA reviewers for their
valuable comments on the content of this article.
References
Adams, E., & Dormans, J. (2012). Game Mechanics: Advanced Game Design. Berkeley, CA:
New Riders.
Banks, G. (2009, 4 of February 2009). Evidence-based policy-making: What is it? How do we
get it? Proceedings from ANZSOG/ANU Public Lecture Series, Canberra.
Barrados, M., & Mayne, J. (2003). Can Public Sector Organisations Learn? OECD Journal on
Budgeting, 3(3), 87-103.
Bots, P., Wagenaar, P., & Willemse, R. (2010). Assimilation of Public Policy Concepts Through
Role-Play: Distinguishing Rational Design and Political Negotiation. Simulation & Gaming,
41(5), 743-766.
Caluwe, L., Geurts, J., & Kleinlugtenbelt, W. J. (2012). Gaming Research in Policy and
Organization: An Assessment From the Netherlands. Simulation & Gaming, 43(5), 600-626.
Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn’t always travel.
Evidence & Policy, 9(1), 97-112.
Cartwright, N., & Hardie, J. (2012). Evidence-Based Policy: A Practical Guide to Doing It
Better. Oxford: Oxford University Press, USA.
Clark, G, Kelly, L (2005) New directions for knowledge transfer and knowledge brokerage in
Scotland: Office of Chief Researcher Knowledge Transfer Team briefing paper. Scottish
Executive Social Research. Scottish Executive Social Research.
Crookall, D. (2010). Serious Games, Debriefing, and Simulation/Gaming as a Discipline.
Simulation & Gaming, 41(6), 898-920.
Dobbins, M., Robeson, P., Ciliska, D., Hanna, S., Cameron, R., O’Mara, L., DeCorby, K.,
Mercer, S. (2009) A description of a knowledge broker role implemented as part of a
randomized controlled trial evaluating three knowledge translation strategies. Implementation
Science 2009, 4:23.
Felicia, P. (2011). Handbook of Research on Improving Learning and Motivation through
Educational Games: Multidisciplinary Approaches. Hershey, PA: IGI Global.
Fischer, F. (2003). Reframing Public Policy: Discursive Politics and Deliberative Practices.
New York: Oxford University Press.
Gutierrez, R. (2010). When Experts Do Politics: Introducing Water Policy Reform in Brazil,
Governance, 23(1), 59-88.
Hood, C. (2007). What happens when transparency meets blame-avoidance? Public Management
Review, 9(2), 191-210.
Kapp, K. M. (2012). The Gamification of Learning and Instruction: Game-based Methods and
16 Knowledge brokers in action. New approach for strengthening evidence-based policies
Karol Olejniczak, Tomasz Kupiec, Igor Widawski
Strategies for Training and Education. San Francisco: Pfeiffer.
Kato, F. (2010). How We Think and Talk About Facilitation. Simulation & Gaming, 41(5), 694-
704.
Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and
Development. Englewood Cliffs: Prentice Hall.
Kriz, W. C. (2010). A Systemic-Constructivist Approach to the Facilitation and Debriefing of
Simulations and Games. Simulation & Gaming, 41(5), 663-680.
Lin, Y.-H. (2012). Knowledge brokering for transference to the pilot’s safety behavior,
Management Decision, 50(7), 1326-1338.
Lomas, J. (2007) The in-between world of knowledge brokering. BMJ 2007, 334:129132.
Majone, G. (1992). Evidence, Argument, and Persuasion in the Policy Process. New Haven,
London: Yale University Press.
Mavoa, H., Waqa, G., Moodie, M., Kremer, P., McCabe, M., Snowdon, W. & Swinburn, B.
(2012) Knowledge exchange in the Pacific: The TROPIC (Translational Research into
Obesity Prevention Policies for Communities) project, BMC Public Health, 12(552), 1-9.
McAneney, H., McCann, J. F., Prior, L., Wilde, J. & Kee, F. (2010). Translating evidence into
practice: A shared priority in public health?, Social Science and Medicine, 70, 1492-1500.
Nutley, S. M., Walter, I., & Davies, H. T. O. (2007). Using Evidence: How research can inform
public services. Bristol: Policy Press.
Olejniczak, K., Kupiec, T., & Raimondo, E. (2014). Brokerzy wiedzy. Nowe spojrzenie na rolę
jednostek ewaluacyjnych. In A. Haber & K. Olejniczak (Eds.), (R)ewaluacja 2. Wiedza w
działaniu (pp. 67-112). Warszawa: Polska Agencja Rozwoju Przedsiębiorczości.
Olejniczak, K., Raimondo, E., & Kupiec, T. (2014). Evaluation Units as Knowledge Brokers:
testing and calibrating an innovative framework. Proceedings from European Evaluation
Society Biennial Conference, Dublin.
Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014). A systematic review of
barriers to and facilitators of the use of evidence by policymakers. BMC Health Services
Research 14:2. doi:10.1186/1472-6963-14-2.
Pavlas, D. (2010). A Model of Flow and Play in Game-based Learning: The Impact of Game
Characteristics, Player Traits, and Player States (Fall 2010). A dissertation submitted for
PhD in the Department of Psychology, University of Central Florida.
Penenberg, A. L. (2013). Play at Work: How Games Inspire Breakthrough Thinking. New York:
Penguin.
Sanderson, I. (2002). Evaluation, Policy Learning and Evidence-Based Policy Making. Public
Administration, 80(1), 1-22.
Schell, J. (2014). The Art of Game Design: A Book of Lenses, Second Edition. London, New
York: A K Peters/CRC Press.
Shillabeer, A., Buss, T. F., & Rousseau, D. M. (Eds.). (2011). Evidence-Based Public
Management: Practices, Issues, and Prospects. Armonk, New York, London: M. E. Sharpe.
Shulha, L. M. & Cousins, B. J. (1997) Evaluation Use: Theory, Research, and Practice Since
1986. Evaluation Practice, 18(3), 195-208.
Waqa, G., Mavoa, H., Snowdon, W., Moodie, M., Nadakuitavuki, R., Mc Cabe, M., Swinburn,
B. (2013) Participants’ perceptions of a knowledge-brokering strategy to facilitate evidence-
informed policy-making in Fiji, BMC Public Health 2013, 13:725
Washmi, R., Baines, M., Organ, S., Hopkins, G., Blanchefield, P., Busch, C. (2014)
"Mathematics "Problem Solving Through Collaboration: Game Design and Adventure" in: C.
Busch (Eds.), 8th European Conference on Games Based Learning: ECGBL2014, Berlin.
Weiss, C.H. & Bucuvalas, M.J. (1980) Social science research and decision-making. New York;
Guildford: Columbia University Press.
Willems, M., Post, M., van der Weijden, T. & Visser-Meily, A. (2013). Do knowledge brokers
facilitate implementation of the stroke guideline in clinical practice?, BMC Health Services
Research, 13(434), 1-17.
... The public servant of the future needs to be able to make strategic use of information, convince her colleagues of the indispensability of technology, data and other instruments for their work and the citizen wellbeing. This requires not only the skill to "sell" the new approach or work method, but empathy, sensitivity to different contexts and backgrounds, and the ability to push the agenda firmly without being overbearing (Olejniczak, Kupiec, & Widawski, 2016). ...
Chapter
Full-text available
Governments all over the world are facing enormous challenges – from austerity to changing demographics, climate change, and the digital deluge, among others (Hutt, 2016; Van der Wal, 2017). Developing countries rarely feature in conversations on the challenges that the future poses for public service, despite the fact that they are especially vulnerable to “wicked” policy problems (Rittel and Webber, 1973) due to capability constraints in the form of subpar economic performance, scarce resources, weak or compromised institutions, and political turbulence (Pattanaik, 2017). These both limit and define the type of skills that public servants in less developed contexts need, for the governments to thrive in the current realities and deliver quality public services to the citizens. The three main pressures of changing citizenry, complexity, and technology require public servants in developing countries to foster a number of skills and roles: collaborator, empathizer, resource manager, political navigator, tech literate, and knowledge broker. This chapter discusses some of the ways in which governments in developing countries can facilitate the attainment and improvement of these skills in public servants. The key message for developing countries is to tailor human resource management practices – including recruitment, pay, training, and job design – to the specific circumstances and pre-existing conditions in which they operate, rather than import ready-made solutions, generated outside of the local context.
... The simulated roleplay is followed by a discussion designed to expose trainees to the conflicts of everyday life, and to develop their critical and creative thinking. Olejniczak et al. (2016) stress that simulation workshops expose trainees to alternative strategies and enable them to experience the different possible outcomes and develop deeper insights regarding key aspects and dilemmas in the real world, while strengthening their social awareness. ...
Article
Full-text available
This study reveals the ethical dilemmas encountered by social workers who mentor workers with intellectual and developmental disabilities that work in the free labour market through supported employment frameworks. The aim of the study is to examine the social workers' ethical dilemmas, while extracting rules of conduct and ethical codes that are unique to supported employment frameworks, through team simulation training. The study included forty-eight social workers who attended nine sessions in the course of one year. The findings, which were analysed using a qualitative methodology, revealed that the dilemmas, rules of ethical conduct and the derived ethical codes ranged from flexible solutions to setting firm boundaries. It emerged that the ethical dilemmas were associated with interactions that take place on all layers of the socio-ecological model. The contribution of the study is in developing an ecological approach to coping with the multidimensional dilemmas that arise in the context of supported employment. The findings will help develop a systemic approach amongst social workers towards coping with the ethical challenges involved in this type of employment.
... The mentoring process through team simulations is based on role-play, which enables participants to experience together authentic events from the field occurring in the workplace in an artificial supporting environment (Benckendorff et al., 2015). Additional studies emphasize the fact that mentoring through simulations exposes the learners to alternative action strategies that enable participants to experience varied possible results and to create deep insights regarding central aspects and ethical dilemmas in the real world while understanding the causes of these dilemmas and reinforcement of social awareness (Olejniczak et al., 2016). ...
Article
Full-text available
This study explored different mentoring strategies: simulations and case studies regarding ethical dilemmas involved with employment of intellectual developmental disability workers in supported employment using questionnaires of psychological empowerment, self-efficacy, and attitudes toward employing intellectual developmental disability workers in the free market. Participants were 83 social workers and 60 employers and were divided into simulations, case studies, and control groups. All participants in the simulations and case studies groups attended nine mentoring sessions throughout one year, where they completed questionnaires at the beginning, the end, and six months after mentoring. Findings The level of employers’ psychological empowerment and their positive attitudes toward employment of individuals with intellectual developmental disability in their organizations were higher than among the social workers, but the social workers’ self-efficacy level was higher. In addition, we found that mentoring using simulations had more influence than the case studies. The study findings showed that the more dynamic the mentoring, the greater the changes in psychological empowerment perceptions, self-efficacy, and attitudes toward employing intellectual developmental disability workers, both during the study and subsequently. Applications There is considerable importance in developing in-service training and active mentoring for all those involved in the challenges of supported employment. Simulations with participation of actors among social workers and employers, regarding ethical challenges in supported employment are the recommended mentoring methods, as compared to the case studies tested in the current study. Therefore, the study findings encourage the development of advanced mentoring processes among social workers and employers based on simulations, for coping with ethical challenges in supported employment.
Article
Full-text available
The claim that evidence-based policy produces better outcomes has gained increasing support over the last three decades. Knowledge brokering is seen as a way to achieve improve policy making and governments worldwide are investing significant resources in knowledge brokering initiatives. It is therefore important to understand the range of these activities and to investigate whether and how they facilitate evidence-based policy. This article critically reviews the extant literature on knowledge brokering. It identifies six important limitations: the existence of multiple definitions of knowledge brokering; a lack of theory based empirical analysis; a neglect of knowledge brokering organisations; insufficient research on knowledge brokering in social policy; limited analysis of impact and effectiveness; and a lack of attention to the role played by politics. The paper proposes an agenda for future research that bridges disciplinary boundaries in order to address these gaps and contribute new insights into the politics of evidence use.
Conference Paper
Full-text available
The Cohesion Policy of the European Union has been one of the most intensively evaluated policies. Evaluation units perform a key role in this process. Despite the massive production of evaluation reports, the utilization of knowledge remains limited. The current evaluation literature does not explain well the reality of evaluation use and role of evaluation units in complex programming and institutional settings. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review on knowledge brokering and empirical research of evaluation units in Poland with complementary evidence from a range of institutional context (US federal government and international organizations). The proposed framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of "reflexive social learning" that steer streams of knowledge to decisionmakers.
Article
Full-text available
When is a well-established study result that a given policy/programme/treatment produced a given outcome in a particular study setting ('there') evidence that that policy/programme/treatment will produce that outcome in a new setting ('here')? This paper insists that 'there' and 'here' be firmly distinguished and offers in answer that we must have evidence that two further facts obtain: (a) that the policy can play the same causal role widely (widely enough to cover both here and there) and (b) that a complete set of the support factors necessary for the policy to operate here are present in some individuals here.
Conference Paper
Full-text available
Games designed for learning need to be enjoyable games with the learning content encoded in the game play mechanism. They also need to be designed in a user centric way. In this paper we describe the design of a game to teach maths to UK primary school children (age 7 9). Our work follows on from an earlier requirements gathering study where we observed collaborative game play in a commercial game to determine key game characteristics that promote collaboration. This paper reports on the subsequent design steps and how children have been included in the design process to ensure that the user interface is age appropriate and engaging. The pedagogy and the game mechanic for the game design come from the learning outcomes and approaches used in the classroom. However, on their own these can easily lead to a design that the players find boring. By integrating challenge which adapts with player achievement the idea of “flow” can be maintained and thus player engagement sustained. This paper outlines our approach to the game design showing why choices were made about the game levels and how they are designed to achieve engagement and maintain flow while at the same time providing a game mechanic that enshrines the learning outcomes. User input to the design process has been maintained through the use of pilot testing in local primary schools. This testing has shown the game to be effective in the main aims of the design – being engaging and requiring the players to interact with the learning outcome and requiring the players to collaborate. Further longer term testing will be undertaken to evaluate the effect of the game in enhancing player learning of the game objectives.
Article
As part of a larger effort by members of the American Evaluation Association (AEA) Topical Interest Group on Evaluation Use (TIGEU), we undertook an extensive review and synthesis of literature in evaluation use published since 1986. We observe several recent developments in theory, research and practice arising from this literature. These developments include: the rise of considerations of context as critical to understanding and explaining use; identification of process use as a significant consequence of evaluation activity; expansion of conceptions of use from the individual to the organization level; and diversification of the role of the evaluator to facilitator, planner and educator/trainer. In addition, understanding misutlilization has emerged as a significant focus for theory and to a limited extent, research. The article concludes with a summary of contemporary issues, particularly with regard to their implications for evaluation practice.
Article
Evaluation units, located within public institutions, are important actors responsible for the production and dissemination of evaluative knowledge in complex programming and institutional settings. The current evaluation literature does not adequately explain their role in fostering better evaluation use. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review and empirical research on evaluation units in Poland within the context of the European Union Cohesion Policy, with complementary evidence from the US federal government and international organizations. In the proposed framework, evaluation units are to perform six types of brokering activities: identifying knowledge users’ needs, acquiring credible knowledge, feeding it to users, building networks between producers and users, accumulating knowledge over time and promoting an evidence-based culture. This framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of reflexive social learning that steer streams of knowledge to decision makers.
Book
Over the last twenty or so years, it has become standard to require policy makers to base their recommendations on evidence. That is now uncontroversial to the point of triviality—of course, policy should be based on the facts. But are the methods that policy makers rely on to gather and analyze evidence the right ones? In Evidence-Based Policy, Nancy Cartwright, an eminent scholar, and Jeremy Hardie, who has had a long and successful career in both business and the economy, explain that the dominant methods which are in use now—broadly speaking, methods that imitate standard practices in medicine like randomized control trials—do not work. They fail, Cartwright and Hardie contend, because they do not enhance our ability to predict if policies will be effective. The prevailing methods fall short not just because social science, which operates within the domain of real-world politics and deals with people, differs so much from the natural science milieu of the lab. Rather, there are principled reasons why the advice for crafting and implementing policy now on offer will lead to bad results. Current guides in use tend to rank scientific methods according to the degree of trustworthiness of the evidence they produce. That is valuable in certain respects, but such approaches offer little advice about how to think about putting such evidence to use. Evidence-Based Policy focuses on showing policymakers how to effectively use evidence. It also explains what types of information are most necessary for making reliable policy, and offers lessons on how to organize that information.
Article
Challenges the common assumption that policy analysts engage in a purely objective technical assessment of policy alternatives. This book argues that what analysts really do is produce policy arguments that are based on value judgements and are used by policymakers in the course of public debate.
Book
In recent years a set of new postempiricist approaches to public policy, drawing on discursive analysis and participatory deliberative practices, have come to challenge the dominant technocratic, empiricist models in policy analysis. In this book, Frank Fischer brings together this work for the first time and critically examines its implications for the field of public policy studies. He describes the theoretical, methodological and political dimensions of this emerging approach to policy research. The book includes a discussion of the social construction of policy problems, the role of interpretation and narrative analysis in policy inquiry, the dialectics of policy argumentation, and the uses of participatory policy analysis. After an introductory chapter, ten further chapters are arranged in four parts: Part I, Public Policy and the Discursive Construction of Reality (two chapters), introduces the re-emergence of interest in ideas and discourse. It then turns to the postempiricist or constructionist view of social reality, presenting public policy as a discursive construct that turns on multiple interpretations. Part II, Public Policy as Discursive Politics (two chapters), examines more specifically the nature of discursive politics and discourse theory and illustrates through a particular disciplinary debate the theoretical, methodological, and political implications of such a conceptual reframing of policy inquiry. Part III, Discursive Policy Inquiry: Resituating Empirical Analysis (four chapters), offers a postempiricist methodology for policy inquiry based on the logic of practical discourse, and explores specific methodological perspectives pertinent to such an orientation, in particular the role of interpretation in policy analysis, narrative policy analysis, and the dialectics of policy argumentation. Part IV, Deliberative Governance (two chapters), discusses the participatory implications of such a method and the role of the policy analyst as facilitator of citizen deliberation .