Content uploaded by Samantha Stone-Jovicich
Author content
All content in this area was uploaded by Samantha Stone-Jovicich on Feb 01, 2016
Content may be subject to copyright.
In this Pracce Note I share our experience using an evaluaon and monitoring approach called ‘rubrics’ to
assess a complex and dynamic project’s progress towards achieving its objecves. Rubrics are a method for
aggregang qualitave performance data for reporng and learning purposes. In M&E toolkits and reports,
rubrics looks very appealing. It appears capable of meeng accountability needs (i.e. collang evidence that
agreed-upon acvies, milestones, and outcomes have been achieved) whilst also contribung to enhanced under-
standing of what worked, what was less successful, and why. Rubrics also seems to be able to communicate all
of this in the form of comprehensive, yet succinct tables. Our experience using the rubrics method, however,
showed that it is far more dicult to apply in pracce. Nonetheless, its value-add for supporng challenging
projects – where goal-posts are oen shiing and unforeseen opportunies and challenges connuously emerging
– is also understated. In this Pracce Note, I share the process of shaping this method into something that seems to
be a right t for the project (at the me of producing this note the project is ongoing and insights sll emerging).
Project context
The experiences shared in this Pracce Note come
from the Food Systems Innovaon (FSI) iniave. FSI
is a partnership between the Department of Foreign
Aairs and Trade (DFAT), the Commonwealth Science
and Industrial Research Organisaon (CSIRO), and
the Australian Centre for Internaonal Agricultural
Research (ACIAR). Launched in 2012, FSI’s aim
is to bring together Australian and internaonal
partners and experse to help improve the impact of
Australian-supported aid investments in agriculture
and food systems in the Indo-Pacic region.
FSI is ambious and complex. It brings together three
Australian government agencies into a new mode of
collaboraon (i.e., partnership versus donor-client
relaonship) whilst also working closely with
partners overseas and with global networks of food
systems experts and other domain specialists. It aims
to balance the planning and management of a
detailed, collecvely agreed-upon set of acvies
and associated outputs with exibility to take on new
opportunies and partner needs as they emerge.
These, along with other aspects that make FSI a
challenging project, have made it quite dicult for the
FSI team to idenfy and develop eecve monitoring,
evaluaon and learning (MEL) approaches and tools.
When we came across the rubrics approach, it seemed
to ck many of the boxes we needed: the ability to
monitor acvies and outputs delivered (primarily
quantave informaon) and assess progress towards
outcomes (largely qualitave evidence). An addional
value-add was that it looked promising as a plaorm
for fostering reecon and learning among the FSI
team and partners. We have yet to test the laer but
in this Pracce Note we share our experiences-to-date
using rubrics as a monitoring and evaluaon tool.
So what are rubrics?
Rubrics – oen called ‘evaluave rubrics’ – is a
qualitative assessment tool. Rubrics involves
articulang and clarifying ‘the things that maer’ in a
project or iniave, which can encompass aspects
related to the performance, quality, usefulness, and
eecveness of project acvies, services or products.
These are in essence the things that are considered
by those involved in the project as important to pay
aenon to. These are evaluated using a qualitave
rang system (e.g. excellent, good, adequate, poor).
Rubrics typically look like a table or a matrix.
There are many readily-available rubrics guidelines
and reports online. Some of the resources we used
and found most useful are listed on the last page of
this Pracce Note.
PRACTICE NOTES SERIES Managing for impact
To rubrics or not to rubrics?
An experience using rubrics for monitoring, evaluating and learning in a
complex project
Samantha Stone-Jovicich, CSIRO August 2015
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
2 │
Our experience with rubrics in FSI
We were rst introduced to rubrics by an evaluaon
praconer who was helping us reect on why the
approaches and tools we had developed were simply
not working for us (an indicator-based framework and
a home-grown narrave approach – see Pracce Note
‘Journey to Fit for Purpose’ for more details). Rubrics
was proposed as a middle-ground between the
rigid, quantave, accountability-oriented indicator
M&E system, on the one hand and the uid, qualitave-
heavy, process-focused, reecon-oriented Learning
Trajectory approach, on the other hand. Below we
describe our journey with rubrics.
Sussing rubrics out
A small team of us (evaluaon praconer included)
spent half a day brainstorming whether the rubrics
approach was worth pursuing (see Figure 1 below).
This was a crical decision as we had already invested
a signicant amount of me in the development of
previous MEL tools only to nd them failing. At this
stage we were not only low in energy but also
quite cognisant, and nervous, that FSI did not have
a funconing, running MEL system to support the
management and implementaon teams.
So how did we go about making the decision? In our
discussions, three quesons dominated:
▪Would rubrics allow us to collate, in a non-
cumbersome and rapid way, the evidence needed
to demonstrate that we were delivering on what
we promised? – we thought yes
▪Equally crical, would it enable us to report on
our outcomes which, up to that point, we were
struggling to do? – rubrics’ focus on outcomes
conrmed this for us
▪And, would the rubrics outputs (tables) be an
eecve way to communicate our achievements (as
well as challenges and deviaons from plans) to
FSI’s Steering Commiee? – again we thought yes
The rough road to a rst dra of rubrics
Idenfying the evaluaon criteria. We began by
looking at various guidelines available online on how
to develop rubrics. The rst thing these told us was that
we had to revisit FSI’s programme logic, parcularly
the outcome statements which are at the heart
of rubrics’ evaluaon process. For each outcome,
rubrics requires that you clearly dene the key
qualies or changes that are deemed crical for your
project’s progression towards achieving its goals.
FIGURE 1 Brainstorming the potential value of using rubrics is a critical step. For us, this entailed scrawling on a blackboard a rough
illustration of what it would look like and how it would support different groups within FSI’s governance structure and revisiting the
project’s programme logic using coloured cards.
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
│ 3
This sounds easy, but it’s not. We found pinpoinng
which specic aspects of outcomes we wanted our
evaluaon to focus on an incredibly challenging task.
It became quickly apparent to us that these would
dier depending on the ‘eye of the beholder’ (i.e. each
member of the FSI team and the Steering Commiee
would most likely dene these dierently). This is not
a new insight – the guidelines on rubrics emphasize
the importance of doing this using parcipatory,
consensus-building processes. Our challenge was that
we simply did not have the me to do this; we had
to pursue the less opmal (but, in projects, typically
more common) path of having to rely on a small set
of team members – those responsible for FSI’s MEL
system – to dene these. While far from ideal we did
have more than a year’s worth of working closely as a
team and liaising frequently with Steering Commiee
members. We did the best we could to ensure that
the reworked outcome statements reected (to the
best of our knowledge) the aggregate perspecves
of FSI members and partners. Aer many, many
iteraons we came up with a series of carefully worded
outcome statements (re-labelled ‘implementaon
and output performance statements’ and ‘primary
outcome’; see Figure 2 below). These formed the
basis for the next step – the evaluaon criteria used
in the rubrics tables.
We then created rubrics tables for each of the
outcome statements (an example for ‘learning events’
is provided in Figure 3 on the next page). We thought
the hardest part was behind us. Not so. We found
that developing the rangs that would be used to
assess the evaluaon criteria was equally dicult.
FIGURE 2 The FSI iniave’s outcome statements were rewrien in such a way that the specic qualies and changes being targeted become
explicit (words/phrases underlined). These subsequently formed the evaluation criteria listed in the rubrics table (see further below).
ACTIVITIES AND
OUTPUTS
PRIMARY
OUTCOME
IMPLEMENTATION AND OUTPUT PERFORMANCE
STATEMENTS
Relevant, timely, appropriately-designed learning events that
involve Australian and/or in-country and partners and their
networks, and are perceived as wor thwhile
Learning events
Knowledge products are relevant, practice-based and
practice-oriented, and aligned with current and emerging
(ag, food and nutrition) international development thinking,
practices and needs within Australia and overseas;
collaboratively-created, reader-friendly and
audience-appropriate, and produced and delivered in a timely
manner
Knowledge
products
An expanded range of relevant experts and development
practitioners who actively contribute to FSI Australian and
in-country partners’ and their networks’ international
development discussions, designs, and practices in ways that
are perceived as collaborative, salient, credible and useful
Expertise and
practice networks
FSI creates opportunities for in-country par tners to participate
in reec tion and learning in food systems innovation
In-country
engagement
Activities and a web-based platform that bring together
analyses, opinions, lessons, experiences and capacit y building
resources derived from FSI and its Australian and regional
partners; are easily accessed and easy to navigate; and
audience-appropriate and reader-friendly; and updated on a
regular and in a timely fashion
Outreach and
external visibility
FSI key partners (DFAT, ACIAR, CSIRO) work together
collaboratively, respecting the agreed-upon partnership
principles, to collectively learn and manage FSI in a responsive
and agile manner
FSI governance
Enhanced
knowledge-exchange;
learning; and
networking among
FSI partners and other
stakeholders in
Australia and
overseas, thereby
strengthening
capacity to progress
food systems
innovation
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
4 │
Dening the rangs categories. Guidelines on rubrics
provide very clear denions of each of the rang
categories (i.e., ‘poor’, ‘adequate’, ‘good’, ‘excellent’,
‘insucient evidence’). These are purposely generic
which meant that we had to reect on ‘so what do
these really mean for us in the context of FSI?’ We
came up with contextualised descripons of the
categories (see Figure 4 below). These were never
formally included in any of the rubrics reports; rather
they were used informally to help us pin-down what
we thought we needed to do next: provide a descripon
for each of the rang categories, for every evaluaon
criteria, across all outcome statements (an example
of what this meant for us is provided below). This is
where we made a costly mistake.
We spent weeks wring up the descripons for rangs
categories. Soon the rubrics became bigger than Ben Hur,
taking up pages and pages of tables lled with minuscule
wring (an example is provided in Figure 5). For us, the
proverbial ‘straw that broke the camel’s back’ came
with the feedback from team members: no-one could
agree on what constuted a ‘poor’, ‘adequate’, ‘good’,
and ‘excellent’ learning event in terms of its relevance,
for example. This was the case for the majority of the
other evaluaon criteria across dierent outcome
statements. We decided to do something we should
have done at the very start: consult an evaluaon prac-
oner who had extensive experience with rubrics. The
rst thing she told us (contrary to some of the online
guidelines we had consulted!) was that we did not need
to dene the rangs categories. “Let the people who
are aending the learning events (or involved in other
FSI acvies) tell you why they cked ‘poor’, ‘adequate’,
‘good’ or ‘excellent’” she said. She added that, like us,
she had learned this lesson the hard way but that
years of trialling dierent approaches to rubrics had
demonstrated that the simpler method worked best.
EVALUATION CRITERIA RATINGS EVIDENCE
Poor Adequate Good Excellent Insu cient
evidence
Relevant
Timely
Appropriately designed
Involve Australian and in-country
partners and networks
Worthwhile
Enhance knowledge-exchange
Enhance learning
Enhance networking
FIGURE 3 Rubrics for FSI’s Learning Events
Implementaon and output performance statement: Learning events are relevant, mely, appropriately designed; involve Australian
and/or in-country partners and their networks; are perceived as worthwhile; and are eecve in progressing FSI’s primary outcome
(enhanced knowledge-exchange; learning; and networking)
FIGURE 4 The rangs categories used in rubrics, generic descripons, and our FSI-contextualised descripons
CATEGORIES GENERIC DESCRIPTIONS TR ANSLATION
(a.k.a. what this really meant for us within the context of FSI)
Poor ▪Performance is weak. Does not meet
minimum expectations/requirements.
▪What we don’t want.
Adequate
▪Performance is reasonable. Still major
gaps or weaknesses.
▪Still not where we want to be; supporting status quo and
unlikely to contribute to new ways of thinking and practice
that will improve ODA impacts overseas.
Good
▪Performance is generally strong. Might
have a few slight weaknesses but
nothing serious and are being managed
eectively.
▪Exposing partners including in-country partners and their
networks to new ideas and practices which hopefully we
see being picked up in their policies, programs, project s.
Excellent
▪Performance is clearly very strong or
exemplary. Any gaps or weaknesses are not
signicant and are managed eectively.
▪The ideal; all conditions needed for maximising space for
innovation (individual to organisational learning, etc. to
systemic changes) are being enhanced.
Insucient
evidence
▪Evidence unavailable or of insucient
quality to determine performance.
▪Flags areas we need to be collecting more evidence on
and/or simply can’t because of time-lags or other factors.
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
│ 5
Towards a more manageable and
praccal rubrics (where we are at now)
With all of the rangs category descripons deleted,
the rubrics tables became so much more user-friendly
and useful. We were nally able to use them.
Gathering the evidence. We drew up a rubrics
table for each learning event we held – i.e., for each
workshop, training events, and reecon session
(we also produced rubrics tables for the other
types of acvies and products FSI was engaging
in/producing). Rather than show the rubrics tables
to parcipants of the events, we put together a very
simple 1-page ‘survey’ that we invited parcipants
to ll out at the end of each event. Other sources
of evidence that enabled us to complete the rubrics
tables included: reecons from FSI team members
who had led or parcipated in the event; unsolicited
e-mails from parcipants about the event; and
follow-on acvies and ‘wins’ that we could trace back
to having been triggered or inuenced by the event.
Reporng the rubrics. The report that emerged from
the rubrics tables was comprised of three dierent
layers of informaon (see Figures 6 and 7 on next
page): (1) a series of individual tables for each
acvity and output (e.g. each learning event held),
(2) a summary table that aggregates the data from
the individual tables (e.g. one table that summarises
all of the learning events held), and (3) a one page
rubrics table with supporng narrave that disls
the massive amount of informaon collated in the
individual and aggregate tables.
FIGURE 5 An example of our (erroneous) aempt to describe each rangs categories for each evaluaon criteria
EVA LUATION
CRITERIA
RATINGS
Poor Adequate Good Excellent Insucient
evidence
Majority of
learning events
Majority of
learning events
Majority of
learning events
Majority of
learning events
RELEVANT
▪Are on topics or
issues that are
irrelevant or
misaligned with key
partners’ and
learning event
participants’ interests
or needs, and with
global/regional/
country-specic
development priorities
and strategies/
programs/projects.
▪Are relevant
for Australia
(Canberra-based)
partners and not for
in-country partners.
▪Meet partners’
and participants’
mandated or
technical needs
(e.g. training in a
particular
M&E system).
▪Are not focused
only on technical
issues but also on
current or emerging
areas of
development thinking
and prac tice that are
aligned with global/
regional/
country-specic
development priorities
and strategies
(e.g. Aid4Trade,
private-public
partnerships).
▪Are on current and
emerging areas of
development thinking and
practice that are aligned
with global/regional/
country-specic
development priorities
and strategies, and
▪Make an explicit link to
the specic development
priorities and strategies/
programs/projects/practices
of participants’
organisations and
networks.
▪Not
enough
evidence
available to
assess this
objective.
TIMELY
▪Are out of sync (not
timely) with partners’
and participants’
information and skills
needs for
enhancing their
capacity to design
and delivery eective
strategies/programs/
projects.
▪Are ‘hit-and-miss’,
i.e. some events
are oered in a
timely manner,
whereby partners
and participants
can make use of
the information;
whereas others
are out of sync.
▪For the most part
are oered at the
right time, i.e. partners/
participants are able
to incorporate at
least some of the
information and
learnings gained
from the event to
inform their strategies/
programs/projects/
practices.
▪Are oered at the right/
appropriate time, i.e. most
of the information and
learnings can be used to
inform partners’ and
participants’ strategies/
programs/projects/
practices.
▪Include some
development thinking/
practices that are emerging
or are on the cutting-edge
(i.e. are not mainstreamed
likely to be) that oers
partners and participants
a ‘head start’/’competitive
edge’ in global development
arena.
▪Not
enough
evidence
available to
assess this
objective.
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
6 │
We found geng from the summary rubrics tables
(#2 in Figure 6) to the ‘summary report’ (#3) a
challenge. The praconer and online guidelines
we consulted recommended that a consultave,
parcipatory process be used (e.g., team members
and partners spend a day together looking at all
the individual and summary rubrics tables and
collecvely write up the summary report). Again, for
the same reasons previously menoned (i.e., me
constraints, logiscal challenges in geng everyone
together) we had to come up with a dierent process.
In our case it was having the project leader look at
the rubrics tables across all FSI acvies, outputs,
and outcomes and summarising them into a single
overarching table that was accompanied by a succinct
narrative statement; and it had to fit in 1 page
(see Figure 7 on the next page).
FIGURE 6 FSI Rubrics approach: An example for Learning Events (fabricated data)
Implementaon and output performance statement: Learning events are relevant, mely, appropriately designed; involve Australian
and/or in-country partners and their networks; are perceived as worthwhile; and are eecve in progressing FSI’s primary outcome
(enhanced knowledge-exchange; learning; and networking)
2. SUMMARY RUBRICS OF ALL EVENTS
LEARNING
EVENTS
EVALUATION CRITERIA &
RATINGS
Relevant
Timely
Appropriately
designed
Involve
partners
Worthwhile
Enhance
knowledge
Enhance
learning
Enhance
networking
Workshop 1
Seminar
Lunch seminar
Reection event
Workshop 2
Training event 1
Training event 2
Presentation
OVERALL
1. INDIVIDUAL RUBRICS FOR EACH EVENT
EVA LUATION
CRITERIA RATINGS EVIDENCE
Relevant Feedback from part icipants who lled pos t-event evaluation form (N=13):
▪Aligned w ith my current work requirements or needs (85%)
▪Knowledg e I gained can be used to improve my work (77%)
▪Structured in a way t hat supp orted my learning style (100%)
▪Benets of attending the seminar outweighed th e time away from the oce (85%)
▪Will share t he infor mation I learnt at the seminar with collea gues (77%)
▪I met people who have the potent ial to be va luable in my work (100%)
Selec tion of additional comments from participants:
▪“I learn some n ew things in this presentaon. I will use some of the things learned to my work.”
▪“ The inform aon presented was very useful for me as a praconer and research er. The exampl e
gave me a clea r idea on how important it is to conside r these issu es.”
Other evidence of success:
▪Since the training, part icipant X and Y have re-designed their program to incorporate
concepts and practices from the event
▪Participant Z wrote the following, unsolici ted e-mail: “I learned a lot from the event and was
wonderi ng if FSI will be oering a follow-on training course.”
Timely
Appropriately designed
Involve Australian and in -country
partners and networks
Worthwhile
Enhance knowledge-exchange
Enhance learning
Enhance networking
Sources of evidence:
▪Participant s’ responses to evaluat ion feedback form
▪Reections from FSI team member leading event
▪Feedback from collaborators and FSI steer ing committee
OVERALL RATING – GOOD
A series of training events on private-public partnerships
and Theories of Change have been delivered and well
received. There is a growing demand from in-country
programs for similar events and 3 are being planned in
the nex t quarter.
3. SUMMARY REPORT
Excellent
Good
Adequate
Insucient information
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
│ 7
This one-page summary report is what we
communicated to our Steering Commiee. All of
the other more detailed tables were included but
(and this is important – we discuss why in the next
secon), they were led away as appendices.
So what do we like about rubrics?
1. ITS MULTIPLE-FUNCTIONALITY
The main purpose of the rubrics is to enable
evaluaon of progress towards outcomes. It does this
really well, but it also is quite eecve as a monitoring
tool for assessing delivery of acvies and outputs. For
example, the summary tables (table #2 in Figure 6)
provide an easy way to list all of the acvies and
outputs that have been achieved. It thus works well for
accountability purposes, which is a desired, demanded
and important aspect of M&E systems. Furthermore,
the process of rang the evaluaon criteria essenally
captures qualitave informaon. Supplemenng
the ‘ck-the-box’ (or, in our case, ‘colour the box’)
rangs categories with wrien and verbal feedback
in the ‘evidence’ column provides another layer
of informaon: what was it about those parcular
qualies or changes that were important? And why
(or why not)? For us, these complimentary narrave-
based sources of evidence enable the M&E system
to have performance management capability, which
opens the possibility of exploring the less tangible
aspects and dynamics (e.g. knowledge brokering,
trust building, co-ownership) that were facilitang
(and, reversely, hindering) FSI to progress towards
its goals.
2. NOT INDICATORS, BUT NOT TOO FAR OFF
Using indicators for measuring success is standard
pracce in many organisaons and projects. So much
so that it is arguably imperave that whatever MEL
system you come up with has to have some capacity
to ‘hold conversaons’ with exisng assessment
structures formally being used by the organisaons
or groups you are working with. What we liked
about the rubrics was that the evaluaon criteria
are essenally indicators but the process of rang
these brings in addional informaon on the ‘how’,
‘why’, and ‘so what’? Even the most die-hard
advocates of quantave indicators acknowledge
the importance of being able to capture this more
qualitave, process-level informaon.
FIGURE 7 Overarching summary report based on rubrics tables (fabricated data)
ACTIVITIES/
OUTPUTS RATING COMMENTARY, EVIDENCE AND PROSPECTS
Learning events Good A series of training events on private-public partnerships and Theories of
Change have been delivered and well received. It was a slow start but there is
a growing demand from in-countr y programs for similar events and 3 are being
planned in the next quar ter.
Expertise and
practice
networks
Adequate External expertise has been used in the areas of M&E, partnership development
and impact investing. Engaging more readily accessible regional expertise has
been a challenge but tasks have been dened to strengthen these ties, with an
immediate focus linking to the inclusive agribusiness innovation round table.
In-country
engagement
Excellent The project has been building strong alliances and collaborations with par tner
projects in East Africa, South Asia, Indonesia and Timor Leste. This is
demonstrated by positive feedback and increasing demand for further
activities from in-country partners. Two major events in South Asia are
being planned that will further strengthen these relationships and enhance
knowledge-exchange, learning and capacity-building.
Outreach and
external
visibility
Poor The interact ive web based platform developed for the project is not being
used. In the coming quarter, the site will be substantially redesigned to reect
its greater out reach orientation and encourage a wider set of stakeholders to
engage with and contribute mater ial.
FSI Governance Good FSI has been eectively responding to numerous and frequent requests from
FSI key partners and their overseas partners to collaborate. Partnership
principles, as ar ticulated in the Partnership Agreement, are being used to
guide FSI governance processes . There is further work to be done to foster
joint learning and ownership by all partners .
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
8 │
3. YOUR PICK OF PROCESSES TO CONSTRUCT
RUBRICS
The rubrics approach is not wed to any singular
approach to developing them. One can choose to
take a boom-up, parcipatory route or a more
top-down one, or somewhere in the middle. This
makes it quite a exible approach that can be
developed in a context-appropriate way, that is
sensive to me- and funding-constraints, and other
issues (for example, conicts among project/program
parcipants which may make it undesirable or unre-
alisc to pursue a parcipatory approach to building
rubrics). In our case, me-constraints and urgency
dictated a less parcipatory approach to developing
the rubrics.
4. INDUCTIVE, EMERGENT RATINGS PROCESS
The process of dening the outcome statements
involves, to various degrees, bringing together the
dierent perspecves and interests of those involved
in a project or iniave. But perhaps one of the
greatest (and understated) value-adds of the rubrics
is not that it just integrates diverse viewpoints but
that it can do so in an inductive, emergent way
(i.e. as one gets more and more feedback, one starts
to get a beer picture of what ‘excellent’, ‘good’,
and ‘poor’ means through the eyes of parcipants).
This is parcularly true with regards to the rangs.
Rather than needing to dene ‘poor’ upfront (which
then throws you in the conundrum we faced), it
becomes dened over me as one accumulates
feedback from a range of people and perspecves
(e.g. dierent team members, external partners,
parcipants in project sponsored acvies, etc.).
This is not insignicant. Let me share an example.
Had we used the predened rangs categories we
had rst come up with and used those to assess the
learning events, a good poron of them would have
more or less bombed on the selected evaluaon
criteria. That is because our (FSI team’s) assumpons
about what comprised a “successful” learning event
was heavily informed by our broader thinking
underpinning the project: that to achieve innovaon
in food systems thinking and pracces, our learning
events had to go beyond the standard-of-the-mill
stu, i.e. training courses, seminars (in other words,
transfer-of-knowledge stu). We have since found,
however, that parcipants who aended conven-
onal learning events overall tended to rate them more
posively then we would have. More importantly, the
wrien comments that accompanied their rangs
reveal a more nuanced, and complex, picture of the
role of these types of learning events in contribung
towards food systems innovaon.
Having said this, it is not always easy to get parcipants
or users to provide comments and thus help give
meaning to the criteria evaluaon. For example,
we have not been able to set up a mechanism for
capturing feedback – in a succinct, regular, and
non-cumbersome manner – from readers of our
reports, Pracce Notes, and the dozens of other
knowledge products we have produced. While
this does not mean that we cannot use rubrics for
assessing knowledge products, it does mean that
our assessment is limited because it is being done
through the eyes of project team members only.
5. SUCCINCT YET RICH – BRINGS SIMPLICITY
TO THE COMPLEXITY OF PROJECTS WITHOUT
BEING SIMPLISTIC
Finally, we liked the nested tables and what they provide
in terms of informaon. They enable moving from the
niy griy to broad level assessments. The one-page
summary report that draws upon the detailed tables
are perfect for the quick read on short ights between
Brisbane and Canberra, or during commung-to-work-
me in Jakarta. For me-poor people with mulple
compeng demands (as most members serving project
management and steering commiees are) this is
essenal. The more detailed tables can be included as
appendices, allowing those interested in more details
to have access to the evidence base.
What have we found challenging?
1. THOSE *%! RATING CATEGORIES
We have already said enough on this issue. We
highlight it again as our blunder of developing detailed
descripons for the rangs categories is apparently a
very common mistake. And a very me consuming one.
2. THE WHOLE IS MORE THAN SIMPLY THE
SUM OF ITS PARTS
Rubrics, because it forces you to break down specic
aributes and changes you want to assess, it focuses
your aenon on dierent bits and pieces of a project.
Don’t get me wrong – this is very valuable (no one can
argue that it is not important to know that a workshop
you invested lots of money in and lost considerable
sleep over, was great at helping people build networks
and enhance knowledge). However, while rubrics are
great for assessing individual or parcular acvies and
outputs in isolaon they are less eecve in capturing
the cumulave outcomes or impacts of mulple dimen-
sions of a project, as it does not capture how they
interact and what emerges as a result. Also the more
components you have the more daunng becomes the
task of pulling the data together into tables.
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
│ 9
3. IT IS A LONG, TIME-CONSUMING PROCESS
It took us 10 months to get from inially assessing
whether the rubrics approach was worth pursuing to
developing a report. There are many reasons why it
took so long. We were novices in this approach and
this meant a steep learning curve and me-consuming
mistakes along the way. We also aimed to cover
over a year’s worth of FSI acvies and outputs; it
was not a maer of developing a handful of rubrics
tables but, rather, several dozens. Moreover, as
highlighted in this Pracce Note, rubrics is not just
a table but rather involves a whole series of steps
and processes. These began with renement of
the outcomes statements, to the development of
dierent tools, to gathering and collang feedback
and other sources of evidence. Each of these bits
and pieces that lead to the actual rubrics table
takes considerable forethought and me. And
nally (and perhaps most signicant) was the fact
that developing rubrics was only one of many, many
other project acvies that the team had to juggle at
the same me. It was thus by necessity a piecemeal
process that had to t and be woven in and amongst
other project acvies, the majority of which were
of greater priority. That said, now that we have the
basic structure of the rubrics table and report along
with the methods for collang supporng evidence,
we ancipate that using the rubrics approach for
assessing future acvies and outputs can be done
relavely quickly.
In summary: if we had to do it all over again, what would we have
done dierently?
RUBRIC STEPS
AND
PROCESSES
WHAT WE DID
WHAT WE WOULD HAVE
DONE DIFFERENTLY
(IN AN IDEAL WORLD)
WHY
Rening the project
outcome statements
and dening the
evaluaon criteria
Two team members
worked together to
reword FSI’s outcome
statements as dened in
the programme logic, and
in the process dene the
evaluation criteria.
A face-to-face participatory
process whereby the whole FSI
team, Steering Committee
members and other key
stakeholders jointly rework
the outcome statements and
dene the evaluation criteria.
Other practitioners’ experience using
rubrics have found that having everyone
jointly articulate and agree upon, in an
explicit and transparent manner, the
evaluation criteria (i.e. what matters,
what is important to focus on) increases
interest in and ownership of the rubrics
process and ultimately, the assessments
made.
Developing the
rangs
We rst dened, for each
evaluation criterion, what
each of the ratings
category were
(i.e. detailed descriptions
of what would constitute
an ‘excellent’, ‘good’,
‘adequate’, and ‘poor’
learning event along each
of the criteria identied).
Leave the ratings undened
(what we ended up doing,
once we realised our mistake
in attempting to ll them out);
allow them to become dened
over time; feedback from
participants/users accumulated.
Dening the ratings is incredibly
time-consuming and produces lengthy,
unreadable (and ultimately unusable)
rubrics tables.
It is impossible to develop a set of ratings
descriptions that will be agreed-upon by
everyone (unless several days are spent,
face-to-face, to jointly develop them).
They are rigid: having a set of
predetermined ratings descriptions does
not allow for changes in how ‘excellent’,
for example, may be perceived and
understood by participants/users over the
course of a project.
Allowing participants/users to assign their
own meaning provides a mechanism for
dierent perspectives of what mat ters
and why to emerge (arguably leading to
more accurate and valid assessments).
Other practitioners’ experiences using
undened ratings has shown that it works
just as well, if not better.
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
10 │
RUBRIC STEPS
AND
PROCESSES
WHAT WE DID
WHAT WE WOULD HAVE
DONE DIFFERENTLY
(IN AN IDEAL WORLD)
WHY
Gathering evidence The evidence for the
majority of FSI’s activities
and products is FSI team
members’ personal
assessments and judgement
calls. The exception is the
Learning Events for which
we developed a concise
survey to gather feedback
from participants.
Develop other
non-cumbersome, systematic
methods for incorporating a
broader set of opinions and
perspectives for the other
FSI activities.
Currently, the FSI assessments provided
through the rubrics are very
internally-focused. Dierent viewpoints
from a broader range of people are required
to get more diverse lines of
evidence. This is needed for achieving a
more transparent, shared, and valid
assessment of how well a project is tracking.
Communicang
the rubrics
First drafts of the
rubrics-based report
included in the main body
of the report all three levels
of rubrics information
(individual tables, summary
tables, and overarching
summary report).
We subsequently reduced
the report to one-page,
ling all the detailed
tables as appendices.
Focus on developing the
one-pager and co-develop it
with the Steering Commit tee.
The look and feel of the report depends
on your audience. In our case, our main
audience has been the Steering Committee.
Previous exper iences with MEL
reporting made it clear to us that whatever
we produced had to be ‘short and sweet’.
The one-pager we produced (which is still
in the works) was designed to meet the
needs, and t in with the busy schedules
of the Steering Committee.
So, to rubrics or not to rubrics?
While we are sll in experimenng with rubrics, we can
share some emerging insights and words of wisdom.
1. GIVE YOURSELF PLENTY OF TIME TO PUT
TOGETHER THE RUBRICS
As we learned the hard way, pung together the
rubrics is not quick nor easy. It is a mul-step process,
whose steps and me-frames vary depending on
how well-developed the project programme logic is,
the degree of consensus among project members
and other key stakeholders about the parcularies
of the outcomes most crically in need of being
tracked, and compeng demands from other project
acvies and priories.
2. TAKE AN EMERGENT APPROACH TO
DEFINING THE RATINGS USED TO ASSESS
THE EVALUATION CRITERIA
Let those providing feedback – be it event parcipants
or team members – dene what ‘poor’, ‘adequate’,
‘good’, and ‘excellent’ is. As a crical mass of assessments
come back, look for paerns or common themes
among people’s responses.
3. EXPERIMENT WITH IT UNTIL IT WORKS
FOR YOU
The generic approach to developing rubrics outlined
in most guidelines may or may not work for your
project. You need to allow some me to experiment
with dierent versions.
4. DON’T GO AT IT BLINDLY – BUILD ON THE
EXPERIENCES OF OTHERS USING RUBRICS
There are some fantasc resources out there,
including people with extensive experience applying
rubrics in dierent formats and varying contexts, and
modifying them along the way. It is worth invesng
in connecng with others who have actually applied
rubrics and building on their lessons learned.
5. BE CAUTIOUS ABOUT TURNING RUBRICS
INTO A JACK-OF-ALL-TRADES MEL TOOL
The rubrics can seem very aracve to use as THE
monitoring and evaluaon tool. In all likelihood, it
needs to be complemented with other approaches.
For example, almost fortuitously, we found that a
monthly e-mail lisng detailing all of the acvies and
outputs that had been completed and were underway
(with a few bullet point details) is, by far a, more eecve
monitoring tool. We have yet to trial the rubrics’ value
for learning-purposes, so the jury is sll out on this
one. But we think it has potenal if it is complemented
with other tools that have been developed explicitly
for triggering reecon and learning. ▪
Rubrics for monitoring, evaluang and learning in a complex project – Samantha Stone-Jovicich, CSIRO
│ 11
Some useful resources
King, J., McKegg, K., Oakden, J. and Wehipeihana,
N. (2013) Rubrics: A Method for Surfacing Values
and Improving the Credibility of Evaluaon. Journal
of Muldisciplinary Evaluaon 9 (21): 11. hp://
journals.sfu.ca/jmde/index.php/jmde_1/article/
view/374
Oakden, J. and Weenink, M. (2015) What’s on the
rubric horizon: Taking stock of current pracce and
thinking about what is next. Presented at ANZEA
Conference, Auckland, 6 – 8 July 2015. hp://prag-
matica.nz/wp-content/uploads/pragmatica-nz/
sites/326/150718-Oakden-Weenink-ANZEA-vxx.pdf
Oakden, J. (2013a) Evaluave rubrics – passing fad or an
important contribuon to evaluaon pracce? Presented at
ANZEA Conference, Auckland, 6 – 8 July 2015. hp://
www.anzea.org.nz/wp-content/uploads/2013/10/
Judy-Oakden-2013-Evaluave-rubrics-fad-or-here-
to-stay.pdf
Oakden, J. (2013b) Evaluaon rubrics: how to ensure
transparent and clear assessment that respects diverse
lines of evidence. BeerEvaluaon, Melbourne, Victoria.
http://betterevaluation.org/sites/default/files/
Evaluaon%20rubrics.pdf
About the author
Dr Samantha Stone-Jovicich is an anthropologist with the
Commonwealth Scienc and Industrial Research Organisaon
(CSIRO). Her research focuses on understanding why innovave
and creave approaches (such as social learning approaches) that
have emerged to address global social-environmental challenges
are so dicult to implement eecvely. Her primary interest is
strengthening science’s contribuon to on-the-ground impacts
by rethinking sciensts’ roles, pracces, and the communicaon
of scienc knowledge. Prior to joining CSIRO in 2006, she was
an applied researcher working with small scale farmers and loggers
in Brazil and other Lan American countries.
This series of Pracce Notes facilitate shared learning
and innovaon to improve pracce amongst research and
development praconers. The series is a deliverable of
The Food Systems Innovaon (FSI) Iniave, a partnership
between the Australian Government Department of
Foreign Aairs and Trade (DFAT), the Commonwealth
Scienc and Industrial Research Organisaon (CSIRO),
the Australian Centre for Internaonal Agricultural
Research (ACIAR) and the Australian Internaonal Food
Security Research Centre (AIFSRC).
LIMESTONE AVENUE, CAMPBELL ACT 2601
PO BOX 225, DICKSON ACT 2606 AUSTRALIA
TEL. +61 2 6276 6621 • ABN 41 687 119 230