Content uploaded by Folkert Haanstra
Author content
All content in this area was uploaded by Folkert Haanstra on Nov 30, 2019
Content may be subject to copyright.
Self-Assessment in Art
Education through a Visual
Rubric
Talita Groenendijk, Andrea K
arp
ati and
Folkert Haanstra
Abstract
A tool for self assessment in secondary art education was developed and tested.The tool
includes rubrics for assessing production and reception activities in art education and consists of
visual and text rubrics.The criteria in the rubrics are based on the Common European
Framework of Reference for Visual Literacy which was developed by The European Network of
Visual Literacy (ENViL). The way teachers and students use the rubrics, whether they consider
them helpful and to what extent students’self-assessments are in line with teacher assessments
was studied. It was concluded that teachers work with the rubrics intensively and both students
and teachers appreciate its visual form. However, it was found that the agreement between
teachers and students about the students’scores was moderate and needed to improve. The
results show that it is untrue that students, or boys in particular, overestimate their own
performance in art education.The current study contributes to the development of feasible and
valid assessment criteria and instruments in secondary art education.
Keywords
art education, rubrics, self-assessment, secondary education, visual literacy, reflection
Introduction
Self-assessment in education can be defined as ‘a method in which students relate
their own work to educational goals’(Sluijsmans et al. 2013, 49). Self-assessment is
important for learning as it is related to self-regulation (Nicol & MacFarlane-Dick
2006). Self-regulation enables students to direct their own learning. This is an impor-
tant skill to acquire at school, because it is required for lifelong learning thereafter,
when external feedback is not always available (Boud & Soler 2016). Nicol &
MacFarlane-Dick (2006) suggest that self-assessment is always present in engaged
task execution, but teachers should develop this capacity also by offering structured
opportunities. But self-assessment as a method involves problematic issues.
DOI: 10.1111/jade.12233 iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Ross (2006) suggests that even though a substantial number of teachers use
self-assessment, they have doubts about the value and accuracy of this method.
For instance, teachers think that good students underestimate their work
whereas underachievers often overestimate. Ross reviewed about 30 empirical
studies on self-assessment with students ranging from 4th grade to university
level and from a wide range of countries (Australia, Canada, England, Finland,
Portugal, Taiwan). One of his conclusions is that self-assessment produces consis-
tent results across items, tasks and short time periods. He concludes that with
regard to concurrent validity (agreement between students’self-assessments and
teacher assessments), results are mixed. Boud & Falchikov (1989) provided a
quantitative review of 48 studies of student self-assessment in higher educaton.
They conclude that over- and underestimation depends on the type of students
and the cirumstances. Pallier (2003) conducted two studies on self-assessment in
Australia. Pallier found different results for different domains. Overestimation
would occur more often in the cognitive domain, whereas in the visual-perceptual
domain, participants tend to underestimate their performance and men are gen-
erally are more confident than women. In their meta-analyis, Boud & Falchikov
(1989) suggest that results on gender remain inconclusive. The disciplinary area
involved may be an important factor for the type of self-assessment male and
female respondents formulate.
Ross (2006) argues that students hold other interpretations of assessment cri-
teria than their teachers. Therefore, training students to assess their work may
increase agreement between self-assessment and teacher scores. He lists four
ways to train students to improve their self-assessments: (1) involve students in
the definition of criteria, (2) instruct them how to apply them, (3) provide feedback
after self- assessment, (4) help students planning their activities based on assess-
ment results.
In order to provide feedback, the outcomes of self-assessment of a student
can be compared to the outcomes of teacher or peer-assessment of that student.
‘Feedback from teachers is a source against which students can evaluate pro-
gress, and check out their own internal constructions of goals, criteria and stan-
dards’(Nicol & MacFarlane-Dick 2006, 208). Peer-assessment is often connected
to self-assessment, but is used less frequently (Sluijsmans et al. 2013). Peer feed-
back may stimulate better understanding of assessment criteria as students
express the criteria in their own words while communicating with peers (Sadler
1998).
Summative assessment is employed by teachers to reveal the effects of an
educational process (Brookhart 2001). Formative assessment, on the other hand,
refers to assessment which aims at improving student performance (Brookhart
2001). Sluijsmans et al. (2013) describe self-assessment as one of the methods of
formative assessment.
Self-assessment in art education
Self-assessment in art education has been propagated in the past by different
authors (e.g. Beattie 1997; Hulks 2003; Cunliffe 2007), emphasising that artists
engage in self-assessment continuously (Soep 2004) to regulate and stimulate their
artistic process and learning. Lindstr€
om (2006) considers the capacity for self-
assessment important in the creative process. According to Sch€
onau (2013), stu-
dents should be given more freedom and responsibility with regard to assignments
and assessment criteria. The student should be the first to judge the quality of the
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
2
work, using the goals and criteria set at the start. ‘From here the assessment will
become a joint enterprise of the student, the teacher, and when possibe and rele-
vant, other students or experts’(Sch€
onau 2013, 159).
Tools for self-assessment
Students may work with portfolios in which self-evaluation plays an important role
(Lindstr€
om 2006; Pereira de Ecßa 2005). Students have to select works to be
included into the portfolio and / or describe and evaluate their creative process.
This, however, is a complex and labourious procedure.
Andrade (2007) suggests rubrics for facilitating self-assessment. A rubric is an
assessment tool which consists of a criteria list and descriptions of different per-
formance levels for each criterion (Andrade 2000). The description should be rich
to offer students insight about teachers’expectations. Providing rubrics prior to
assessement makes the process transparent. Rubrics can be either general or task
specific and holistic or analytic in nature (Groenendijk et al. 2016). Task-specific
rubrics are easier to score as indicators are closely related to the task. General
rubrics, however, enable student to monitor their progress over different tasks.
Analytic rubrics are detailed and transparent. Holistic rubrics, however, may result
in more valid evaluations, because student performance is not easily reflected in
strict definitions.
Disadvantages of the rubric in general is that its construction is time con-
suming and it is textual: unattractive for students whose communication has
become predominantly visual (Maarleveld & Kortland 2013). In a visual rubric
system developed by the art teachers Maarleveld & Kortland (2013), pictures
represent the criteria and students can colour these to indicate levels (see Fig-
ure 1). The images are intended to make the assessment criteria more visible.
This type of visualisation provides symbols that facilitate and also enrich the
interpretation of concepts. Art students with a well-developed visual literacy are
motivated to explore the meaning of an evaluation criterion through connecting
associations evoked by the image and the word. Results from a first field test
revealed that art teachers and students in the Netherlands were enthusiastic
about these visual rubrics (Maarleveld & Kortland 2013). But more research is
needed to determine the cognitive and motivational value of this kind of instru-
ment for art education.
Development of a visual rubric system based on ENViL
competencies
The current study describes a self-assessment system of visual rubrics, based on
the system of skills, abilities, attitudes and values described in the Common Euro-
pean Framework of Reference for Visual Literacy. The European Network of Visual
Literacy (ENViL) aims to describe the competencies that ‘European citizens must
have in the field of fine arts and applied arts (architecture and design) as well as
the everyday visual culture when they participate in society and culture as respon-
sible citizens’(http://www.envil.eu). The Common European Framework of Refer-
ence for Visual Literacy (CEFR_VL) is the outcome of an international collaborative
effort to compare curricula and describe visual competencies relevant for art and
design education worldwide.
Its main dimensions are production (creating and using images) and reception
(responding to images). A broad overarching competency dimension is metacogni-
tion, as this addresses reflection on productive and receptive activities. Underlying
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
3
these main dimensions are 16 sub-competencies which are related to production,
reception, or both. Examples of sub-competencies are: Create, Describe, Draft, and
Empathise. The sub-dimensions are not mutually exclusive as some overlap or hold
hierarchical relations to one another. Therefore the model is a fluid and dynamic
representation of the main competencies of visual literacy.
Based on this model, we attempted to create a self-assessment instrument for
secondary art education. We asked one of the authors of the original Dutch instru-
ment (Maarleveld & Kortland 2013) to ‘translate’the criteria based on CEFR_VL
competences into images and find visual solutions for them. We strongly believe
that students, especially in upper secondary education, need to understand sub-
ject-specific requirements and goals to assess their own learning in any school dis-
cipline. As the ENViL model is a quite complex and extensive document (Wagner &
Sch€
onau 2016), we intended to translate the central ideas in a concise and under-
standable assessment instrument for classroom use. This rubric offers several
assessment criteria useful for a wide range of tasks, but not all criteria need to be
relevant in one single task.
Research hypothesis
The visual rubric (and the accompanying text rubric) based on the ENViL model is
a feasible instrument for self-assessment of production and reception activities in
secondary art education.
We formulated a series of research questions to prove or disprove this claim:
Figure 1
Criterion ‘Experimenting’from the original visual rubrics by Maarleveld & Kortland (2013)
which was also included in the current instrument (see appendices 1–4)
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
4
1. How do art teachers and students in secondary education apply the visual rub-
rics (and accompanying text rubrics) in self-assessment?
2. Do teachers and students consider the visual rubrics (and accompanying text
rubrics) as a feasible tool for self-assessment that is facilitating the learning pro-
cess?
3. To what extent do teachers and students agree with regard to the level of the
students’competences? (Comparison of mean self-assessment scores).
4. Are there any differences in self-assessment related to gender and competence
level?
Method
Instrument construction
The visual rubrics based on the ENViL model were developed in two iterations,
with two rounds of data collection. The first round (2015) raised several questions
with regard to the criteria and rubrics. The instrument was therefore revised, tak-
ing into account feedback from educational practice. Assessment criteria (round 2,
2016) are presented in Table 1 and the entire instrument is attached in Appen-
dices 1–4.
Implementation
The assessment instrument was implemented in four countries: Austria, the
Netherlands, Hungary and Germany (Austria participated only in the first round of
data collection). As our aim was to develop an assessment instrument fitting curric-
ula in most European countries, teachers were instructed to work with their regu-
lar assignments and use the production or the reception assessment rubrics.
Teachers participated with grades of their choice. They were asked to explain and
discuss the criteria with their students before the start of a lesson series, have the
students complete a visual self-assessment form after the lesson series and pro-
vide teacher assessments for each individual student by completing the same visual
assessment sheets.
The teachers decided about the way they would introduce the rubrics, which
criteria to use, frequency of use, use of peer assessment, and activities after self-
assessment (for example, feedback dialogue). Teachers were supposed to evaluate
the use of the rubrics with the students and use them in a formative way. In some
cases, summative assessments by the teacher were informed by self-assessments,
but these were never directly used.
Participants
In Table 2 we present an overview of the participating countries, teachers and stu-
dents. The teachers were contacted through the professional networks in the dif-
ferent countries and they were asked to contribute voluntarily. The teachers
predominantly work in urban areas and the schools are populated by a socio-eco-
nomic mix of students. Although the assessment instrument was intended for use
in secondary education, the teachers also used the instrument in primary and ter-
tiary education. For this study, only data from secondary education are included.
In both testing cycles the production rubric was selected by teachers for
implementation more often (37 times), than the reception rubric (10 times).
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
5
Data collection and analysis
Participating teachers and 23 Dutch students (round 1) and 25 Dutch students
(round 2) were interviewed about the implementation of the assessment and their
evaluation of the instrument. The teachers from Germany, Austria and Hungary
answered interview questions in writing.
In round 1, students used the visual rubric, in round 2, both teachers and stu-
dents filled in assessment forms.
In this article, we summarise results of both rounds of data collection. The
quantitative results (research question 3 and 4 on production) are based only on
the second round of data collection as the instrument changed from round 1 to
round 2. For reception, rounds 1 and 2 were taken together as too little data were
collected in the round 2 and the reception instrument hardly changed from round
1 to round 2.
Results
1. How do art teachers and students in secondary education apply the visual
rubric (and accompanying text rubric) in self-assessment?
Implementation
The instrument was implemented in several art education related disciplines, such
as drawing, art, art history, art&design and visual culture. It was used in relation to
a variety of tasks, ranging from relatively structured to very open tasks and from
traditional drawing and painting tasks to film projects and design tasks. All teachers
presented the criteria in a discussion with students.
TABLE 1 Assessment criteria production and reception
Production Criteria
Designing Researching
Experimenting
Realising Using materials and techniques
Using visual elements
Using Saying something with images
Presenting
Reception Criteria
Describing Materials and techniques
Visual elements and representation
Function, style and genre
Interpreting Analysing
Using various perspectives
Evaluating Being curious and open
Having an opinion and supporting it
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
6
TABLE 2 Overview of participating countries, schools, teachers and students
Countries #Schools #Teachers #Classes Grades
†
#Students
Production instrument
tested
Reception instrument
tested
Data collection round 1
(February-June, 2015)
The Netherlands 5 6 8 2 7,8,9,10,12 236
Hungary 3 5 5 2 7,9,10,11,12 133
Austria 3 3 5 5 8,11 and adults 82
Germany 1 1 1 - 10/11 15
Total 12 15 19 classes 9 classes 466
Data collection round 2
(February-June, 2016)
The Netherlands 4 4 7 not tested 7,8,10 126
Hungary 4 6 9 1 9,10,11 157
Germany 1 1 2 not tested 9,11 35
Total 9 11 18 classes 2 classes 318
†
grade 7 =age 12–13, grade 8 =age 13–14, grade 9 =age 14–15, grade 10 =age 15–16, grade 11 =age 16–17, grade 12 =age 17–18
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
7
Not all the criteria in the instrument were used. Sometimes it was the teacher
who selected task-relevant criteria. In some cases, students decided together with
their teachers and in one case the students themselves chose which criteria to use
and which to exclude. The criterion Presenting (included in the production rubric)
was often excluded from the assessment because there was no time or room for
presentations and exhibitions in the piloting schools.
Although teachers found them useful, half-way assessments (performed in the
middle of the creative / reflective process) and peer assessments were rarely
employed, due to time constraints. One teacher asked students to add an explana-
tion to assessment decisions –a practice that helps realise if and how students
understand the criteria.
When using the self-assessment forms, students reported reading the criteria
and looking at the images, but hardly ever reading the text rubrics. Their reason:
teachers had explained them already. Seventh grade students thought the text rub-
rics contained too many complex words or were long. A few students said they
used the text rubrics when something was unclear to them. Apparently, most stu-
dents found the visual rubrics enough for performing assessment.
Reflective dialogues based on self-assessment
Teachers who engaged in reflective dialogues with students after the self-assess-
ments encouraged students to talk first. A typical procedure:
1. The student explains the self-assessment scores criterion after criterion (follow-
ing the order of the assessment sheet).
2. The teacher shares her opinion on the student’s performance.
3. They discuss what can be improved and what is satisfactory.
Most teachers discussed performance on all criteria. One teacher remarked
that she had an idea about which criteria were important for which student, so
she considered focusing on them only, but discussing all criteria, without presuppo-
sitions, seemed to be fairer.
Another teacher said the essence of the process was raising issues, having stu-
dents talk about artistic processes and reflect on their work. Sometimes the tea-
cher and the student reread the text rubrics together during the conversation. A
third teacher said she preferred to fill in the assessment sheet together with the
student. Sometimes she added a special focus. An example: Visual elements, focus-
ing on composition:
I use it as a conversation tool, [...] I always have to explain visual elements
although they should know what they mean, but I turn their focus on one issue,
for example, composition. With the book cover task; [I ask them] why did you
choose that composition? Why did you choose those colours? (Teacher R, grade
10, the Netherlands)
The goal of the dialogue, according to the teachers, is to provide insight in the
grading system, explain teacher judgement, check whether there is agreement
between teacher and student or discuss what can be done to improve and grow,
and to become owner of their learning processes.
Some teachers used to have feedback conversations with students before they
started working with the rubrics. These teachers said that the rubrics make
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
8
evaluative conversations more structured and transparent for students. It stimu-
lates talk about what can be done to improve, to get higher scores and positive
feedback. Teachers who had not intitiated such conversations before, discovered
that students made more conscious creative decisions than expected.
2. Do teachers and students consider the visual rubrics (and accompanying
text rubrics) as a feasible tool for self-assessment that is facilitating the learning
process?
Advantages of the instrument
The teachers mentioned several advantages of the instrument: it is a way to stimu-
late conversation, it makes assessment more objective, it visualises the structure
of the curriculum and it can be a tool to teach subject-specific concepts to the stu-
dents. One said:
I notice I enter into conversation more easily with students who I normally tend
to forget a little, because it forces you to just look at certain issues. I will keep on
doing that. I see my students better because the process is elaborated in front of
you. (Teacher G, grade 7, the Netherlands)
The reception instrument was used less often than the production instrument.
Art reception is not taught in a very structured way in the Netherlands in the
lower grades, and the instrument was expected to structure the curriculum
through clarifying requirements.
Some students said that due to the self-assessment instrument which was
introduced before starting assignments, they worked differently. They also consid-
ered clearly formulated assessment criteria important:
Student: You get an overview; you have to collect sources of inspiration, you have
to experiment a little, those kinds of things. Otherwise it was just drawing; do the
task and hand it in. Now I know what to look at if I want a good grade [...]
(Student, Grade 10, the Netherlands)
Most students considered self-assessment a useful exercise as it gave more
information than a grade alone and provided issues to talk about, to make the tea-
cher understand them better:
Interviewer:Do you want your teacher to keep using it?
Student:Yes, I think it is handy [...] after she had explained it, I understood it
and I think it is a good way. It is not a big thing, you do not have to
read an entire book. It is clear and provides much information, you can
talk a lot about it [...] And she [the teacher] understood me better,
because I could explain what I thought about it. [...] Before it was ‘I
think this and that...’but now you have a strategy. (student grade 10,
the Netherlands)
Finally, some students said that the teacher cannot realise all aspects of stu-
dent work. For example, Experimenting may result in some observable output, but
the student may not show this during assessment or simply lose the sketches.
Therefore, when documentating a creative process, self-assessment is very useful
as the student knows her work phases best.
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
9
Students’understanding of the instrument
The teachers noticed that some words or concepts were difficult for the students.
In grade 7, for example, the phrase ‘visual elements’had not been explained before.
Another problem was applying broad concepts to specific tasks. Although students
said they understood most criteria, they seemed to have different interpretations
than those in the text rubrics. Some of these interpretations are more problematic
than others. Many students interpreted the criteria in a purely quantitative way,
such as ‘the more materials and techniques used, the better’or ‘the more experi-
mentation, the better’. A few students even interpreted scales literally: for exam-
ple, the 4-point scale was thought to represent the number of materials/
techniques used.
Most students said that images illustrating criteria on the visual rubric sheet
were clear and understandable. It was because of the images that they could
understand the criteria. However, according to teachers, some images were taken
literally at first. For example, Using materials and techniques caused problems as
students thought they had to use a photo camera or a phone as it was repre-
sented on the image (See Figure 2). The image for Researching was questioned by
one student: why does the person at level 4 fall into a hole? (Level 4 is about
being heavily involved in the research process).
3. To what extent do teachers and students agree with regard to the level
of the students’competences?
Agreement between students and teachers
Teachers generally have doubts about self-assessment scores (Ross 2006) and
suppose that students overestimate their work. Therefore, we studied the relation
between students’self-assessments and teachers’scores. In Table 3 we present
the mean self-assessments and teacher assessment scores for the production
instrument. The mean self-assessment scores of the students hardly differ from
the mean scores given by the teachers. Only the criterion’Using visual elements’
produced significant differences between teachers’and students’scores (paired
samples t-test: t(252)=2.379, p=.018*). Teachers’scores are higher than students’
self-assessments on this criterion. This is in contrast to the general assumption
about students’biases about their own work.
Figure 2
Assessment criteria ‘Using materials and techniques’and ‘Researching’
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
10
TABLE 3 Student and teacher mean scores production (SD), Pearson correlation and kappa
Criteria N Self-
assessment
Teacher
assessment
Paired samples t test
(difference student-
teacher scores)
Correlation r:
teacher score –
self-assessment
Weighted Kappa
(teacher-student)
%‘Accurate’
self-assessments
MSDMSD
Designing Researching 253 2.88 .66 2.91 .75 t(252)=.707, p=.480 .557** .461 59%
Experimenting 245 2.83 .89 2.79 .77 t(244)=.799, p=.425 .587** .465 57%
Realising Using materials
and techniques
253 2.84 .79 2.86 .82 t(252)=.381, p=.704 .575** .446 55%
Using visual
elements
253 2.76 .81 2.90 .76 t(252)=2.379, p=.018* .355** .265 47%
Using Saying something
with images
171 2.93 .83 2.99 .78 t(170)=1.035, p=.302 .618** .458 56%
Presenting 93 3.00 .79 3.04 .81 t(92)=.514, p=.608 .492** .369 51%
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
11
To explore the relation between students’and teachers’scores further, Pear-
son correlations, weigthed kappa and percentages of ‘accuracy’were calculated.
Correlations show whether students and teachers agree in the relative position
of a student’s score in relation to the group. The baselines of teachers and stu-
dents may differ, but the estimated relative position of a product within the
group of products can be the same for self assessments and teachers’scores.
Percentage accuracy provides a percentage of the number of ‘accurate’self-
assessments: instances in which students’and teacher’s scores coincide. We think
‘accuracy’is normative as the teacher does not hold the absolute truth. This per-
centage, however, does not take chance into account, as a 4-point scale provides
ample opportunity to guess right. Cohen’s kappa takes chance into account.
Weighted kappa is used for ordinal scales in which a difference of one scale
point between raters is taken less serious than a difference of two scale points.
The correlations between students’and teachers’scores are generally low to
moderate. Only about 25 to 36 per cent of the variance between the scores is
explained by the students and the teacher assessing the same object. The lowest
correlation and weighted kappa exists for ‘Using visual elements’. This was exactly
the criterion the students reported as the most difficult to interpret. About half of
the students assessed their work ‘accurately’. Weighted kappa’s are fair
(.20<kappa<.40) to moderate (.40<kappa<.60) (Altman 1991).
In Table 4 we present the assessment scores for the reception rubric. As few
teachers used the reception rubrics, we obtained less data and therefore we must
be cautious interpreting them. Generally, teachers scored slightly (but not signifi-
cantly) higher than students. Teacher scores are significantly lower than those of
students for ‘Using various perspectives’only. This means that students overesti-
mate their performance on this criterion.
The highest scores are given to ‘Being curious and open’, the lowest to ‘Describ-
ing function style and genre’(students) and ‘Using various perspectives’(teachers).
The strongest correlation between teachers’and students’scores are observable for
‘Being curious and open’(r=.755). Weighted kappa is good for this criterion. The low-
est correlation is shown for ‘Analysing’; here, weigthed kappa is poor, too. It seems
that students had problems to assess their work on this criterion. For most criteria,
however, about two-thirds of the students made accurate assessments.
We conclude that, on average, students do not overestimate their work, but
agreement between students’and teachers’scores is not yet optimal. For all crite-
ria (production and reception), the difference between teachers is significant, indi-
cating different methodologies. As we did not use a random sample of teachers
per country, we cannot draw conclusions about the difference between the coun-
tries in this respect, but we may assume that teachers as jurors of student work
show a variety of professional backgrounds and educational cultures. This juror
bias can, however, be modified through training, as proven by the Dutch-Hungarian
project-based final examination experiment (K
arp
ati et al. 1998).
4. Are there any differences in self-assessment related to gender and
competence level?
Differences between boys and girls
Several studies suggest that self-assessments differ by gender, age and compe-
tence level. Men are thought to be more confident than women, older people are
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
12
TABLE 4 Student and teacher mean scores reception (SD) Pearson correlation and kappa
Criteria N Self-
assessment
Teacher
assessment
Paired samples t test
(difference student-
teacher scores)
Correlation r:
teacher score –
self-assessment
Weighted Kappa
(teacher-student)
% Accurate
self-assessments
MSDMSD
Describing Materials and techniques 17 2.53 .72 2.53 .62 t(16)=.000, p=1.000 .591* .333 65%
Visual elements and
representation
37 2.76 .60 2.81 .85 t(36)=.388, p=.701 .347* .207 46%
Function, style and genre 38 2.39 .72 2.45 .76 t(37)=.404, p=.689 .410* .312 53%
Interpreting Analysing 29 2.76 .79 3.03 .57 t(28)=1.684, p=.103 .180 .153 48%
Using various perspectives 38 2.55 .86 2.16 .68 t(37)=3.224, p=.003* .541** .414 61%
Evaluating Being curious and open 37 3.11 .77 3.27 .80 t(36)=1.782, p=.083 .755** .608 68%
Having an opinion and
supporting it
38 2.68 .96 2.95 .90 t(37)=1.959, p=.058 .606** .531 63%
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
13
thought to assess more accurately (Pallier 2003) and more competent students
are thought to assess more accurately (Ross 2006).
In the case of the production rubrics, for most criteria (except for Experiment-
ing and Presenting), girls score their work significantly higher than boys (.000<
p<.002), see Table 5. Also, the teachers give girls significantly higher scores than
boys (.000<p<.031 for all of the criteria). The mean absolute value of the differ-
ence between teacher’s and students’scores differed significantly between boys
and girls for Researching (see last column in the Table). The mean difference was
significantly larger for boys than for girls. This means that girls assessed their work
more accurately than boys for this criterion only. As the mean self-assessment of
boys and teachers differs only .01 point on this criterion, we conclude that there
are boys who underestimate and who overestimate their work. This finding
negates the general assumption that boys overestimate their work.
In the case of reception (Table 6), the results are preliminary due to the lack
of an appropriate amount of data. Based on existing information, we conclude
that the difference between boys and girls is significant for the criterion Using
various perspectives (t(36)=3.116, p=.004). Here, girls give higher scores than
boys. For the other criteria, however, there are no differences between boys and
girls. Teachers’scores did not differ for boys and girls for any of the criteria
either. The absolute value of the difference between teacher’s and students’
scores differed significantly between boys and girls for ‘Being curious and open’
(see last column in the Table). The mean difference was significantly larger for
boys than for girls. This means that girls assessed their work more accurately
than boys for this criterion. As the mean self-assessment of boys is lower than
the mean teachers’score, we conclude that boys underestimated their attitude.
Or else, teachers overestimated boys’attitudes, as students may be regarded as
the experts of their own attitude. For the other criteria, teachers’scores were
equal for boys and girls.
We also tested whether more competent students assess their work more ‘ac-
curately’. Students with an average teacher score of 3 or more (on the 4-point
scale) were considered more competent students. These students were compared
to the other students with regard to the ‘accuracy’of the self-assessment. We
found that competent students were indeed more accurate for the criteria
‘Researching’,‘Using visual elements’and ‘Saying something with images’(t(251)
=3.044, p=.003; t(251)=1.288, p=.048; t(169)=2.544, p=.012). In the case of recep-
tion, this pattern was not found for any of the criteria.
The current data do not allow analysis of ‘accuracy’for different age groups as
most teachers only participated with one class only, therefore possible effects of
age interfere with teacher effects.
Conclusion and discussion
First, we have shown that rubrics (especially visual formats) are useful in sec-
ondary level art education. Teachers from four countries actively and successfiully
employed the rubrics, explaining and discussing the criteria thoroughly with the
students before they started working on assignments. The students competently
used the visual rubrics, but hardly employed the accompanying text rubrics. All
teachers acknowledged the relevance of discussing results of self-assessments, but
some of them lacked time for this activity.
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
14
TABLE 5 Production: girls’and boys’means and SD
Criteria Self-assessment
girls
Teacher
assessments girls
Self-assessment
boys
Teacher
assessments
boys
Independent
samples t-test Difference
with teacher (girls- boys)
N mean SD mean SD N mean SD mean SD
Designing Researching 130 3.00 .57 3.07 .63 113 2.74 .74 2.73 .84 t(241)=3.343, p=.001**
Experimenting 125 2.93 .88 2.93 .74 110 2.74 .84 2.67 .76 t(233)=1.157, p=.248
Realising Using materials and
techniques
130 3.02 .76 3.12 .75 113 2.62 .80 2.58 .82 t(241)=.866, p=.387
Using visual elements 130 2.90 .79 3.12 .72 113 2.58 .80 2.65 .77 t(241)=.656, p=.512
Using Saying something with
images
93 3.13 .76 3.22 .66 68 2.68 .85 2.66 .78 t(159)=1.148, p=.253
Presenting 56 3.09 .75 3.18 .77 30 2.80 .85 2.80 .76 t(84)=.411, p=.682
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
15
TABLE 6 Reception: girls’and boys’means and SD
Criteria Self-assessment girls Teacher
assessments girls
Self-assessment
boys
Teacher
assessments
boys
Independent samples
t-test Difference with
teacher (girls- boys)
N mean SD mean SD N mean SD mean SD
Describing Materials and techniques 10 2.50 .53 2.60 .52 7 2.57 .98 2.43 .79 t(15)=.517, p=.612
Visual elements and representation 21 2.71 .64 2.90 .77 16 2.81 .54 2.59 1.00 t(35)=.266, p=.792
Function, style and genre 21 2.38 .74 2.43 .68 17 2.41 .71 2.47 .87 t(36)=1.113, p=.273
Interpreting Analysing 16 2.81 .91 2.94 .57 13 2.69 .63 3.15 .56 t(27)=.037, p=.970
Using various perspectives 21 2.91 .83 2.33 .58 17 2.12 .70 1.94 .75 t(36)=.707, p=.484
Evaluating Being curious and open 20 3.15 .75 3.30 .73 17 3.06 .83 3.24 .90 t(35)=2.612, p=.013*
Having an opinion and supporting it 21 2.71 1.15 3.00 1.10 17 2.65 .70 2.88 .60 t(36)=.023, p=.982
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
16
Secondly, we addressed the question whether teachers and students experi-
ence the visual rubrics (and accompanying text rubrics) as a feasible tool for self-
assessment. Teachers thought it helped them to structure the curriculum and their
feedback conversations. Students said it helped them understand learning objec-
tives and assessment criteria. All responses about user experiences received from
the teacher and student community were positive. Some criteria, however, were
ambiguous. Andrade (2007) argues that students have to be engaged in the con-
struction of rubrics to understand which features of their work are evaluated and
how. In our study, some reflective dialogues took place, but not all teachers man-
aged to organise them.
Third, we investigated the extent of agreement between teachers and students
on the level of students’competences. In contrast to earlier findings, students did
not overestimate their own work, their self-assessment was fair. In fact, we found
that mean teacher and student scores hardly differed. Correlations and weighted
kappa’s show, however, that the agreement between teachers and students is mod-
erate for most criteria. We obtained somewhat different results for the individual
criteria. The teachers and students agreed that students are very ‘Curious and
open’. The teachers thought students are not so good at ‘Using various perspec-
tives’. Agreement between teachers and students was good for ‘Being curious and
open’and Having an opinion and supporting it. Students seemed to have good
understanding of these criteria. There was hardly any agreement for ‘Analysing’
and ‘Using visual elements’. It seems that students did not understand these crite-
ria. These results provide good indications for improving the instrument, which is
possible and desirable.
Fourth, we wanted to reveal differences in self-assessment related to gender
and competence level. We found that boys do not overestimate more than girls
do. The current instrument seems to stimulate fair self-assessments for boys and
girls, against the commonly held assumption about boys’bias for their perfor-
mance.
The study lacked a control group which can be considered a serious limitation.
It would have been interesting to compare the self-assessments on the same crite-
ria of students using visual rubrics and students using text rubrics only. This is a
recommendation for future studies.
We agree with Cunliffe (2007, 102) who states that students’formative self
assessment and teacher feedback in art and design education ‘should be diagnostic
and aimed at pinpointing how learning can be improved’. He adds that this process
needs to be be carried out more rather than less frequently.
We think that our research shows that visual rubrics have the potential to be used
as assessment instruments to serve this purpose, because they are flexible, appealing
and can be used quickly. But improving self-regulated learning involves much more
than having students use a rubric. The role of the teacher is essential in guiding the
self-assessment process and stimulating students to engage with the rubric actively in
order to avoid misunderstandings. These misunderstandings are also shown in our
research and are partly due to ambiguity in the visualisations of the competencies.
Constructing valid and reliable visual rubrics is an ongoing process. We hope this arti-
cle will stimulate others to experiment with self-assessment through visual rubrics and
publish research on the value and accuracy of this method.
Talita Groenendijk studied Art education and Social and cultural anthropology. In 2012
she finished her PhD dissertation about observational learning in art education at the
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
17
Research Institute of Child Development and Education, University of Amsterdam. After
that she worked on several research projects as a postdoctoral researcher. She has pub-
lished on observational learning and creativity in art education, media literacy and art educa-
tion, altermodern art education, disciplinary writing, and assessment in art education.
Currently she works as a lecturer and researcher at the Amsterdam University of the Arts.
Contact address: Amsterdam University of the Arts, Master of Education in Arts, Overhoek-
splein 2, 1031 KS Amsterdam, the Netherlands. Email: talita.groenendijk@ahk.nl
Andrea K
arp
ati is Professor of Education and UNESCO Chairholder at E€
otv€
os Lor
and
University (ELTE, since 2002), Hungary. She is Head of the Visual Culture Research Group
of the Hungarian Academy of Science. She has served as Vice President of InSEA and was
on the Executive Committee of EARLI for two terms each. She is a member of the European
Network of Visual Literacy (ENViL) and editorial boards of four international research jour-
nals. She is a recipient of the Edwin Ziegfeld Award for Distinguished International Leader-
ship in Art Education and several Hungarian awards. Her publications include 18 books, 60
book chapters and more than 150 research papers in five languages. Contact address: ELTE
TTK TUDKOM, H-1117 P
azm
any s
et
any 1/A, 1117, Budapest, Hungary. Email: an-
drea.karpati@ttk.elte.hu
Folkert Haanstra studied psychology and Fine Art. He held the special chair for Cultural
Education and Cultural Participation at the University Utrecht (2000–2015). He was Profes-
sor of Arts Education at the Amsterdam University of the Arts (2001–2016). He has co-au-
thored A review of assessment instruments in arts education in S. Schonmann [Ed.]
International Yearbook for Research in Arts Education 3/2015 –The Wisdom of the Many –Key
Issues in Arts Education (M€
unster: Waxmann, 2015) and Interestingness and pleasingness of
drawings from different age and expertise groups, Empirical Studies of the Arts, Vol. 31, No. 2
(2013). Contact address: Amsterdam University of the Arts, Overhoeksplein 2, 1031 KS
Amsterdam, the Netherlands. Email: folkert.haanstra@ahk.nl
References
Altman, D. G. (1991) Practical Statistics for
Medical Research. London: Chapman and Hall.
Andrade, H. (2000) Using rubrics to
promote thinking and learning, Educational
Leadership, Vol. 57, No. 5, pp. 13–18.
Andrade, H. (2007) Self-assessment
through rubrics, Informative Assessment, Vol.
65, No. 4, pp. 60–63.
Beattie, D. K. (1997) Assessment in Art
Education. Worcester, MA: Davis
Publications.
Boud, D. & Falchikov, N. (1989)
Quantitative studies of student self-
assessment in higher education: a critical
analysis of findings, Higher Education, 18(5):
529–549.
Boud, D. & Soler, R. (2016) Sustainable
assessment revisited, Assessment and
Evaluation in Higher Education, Vol. 41, No.
3, pp. 400 –413.
Brookhart, S. M. ( 2001) Successful
students’formative and summative
uses of assessment information,
Assessment in Education: Principles,
Policy and Practice, Vol. 8, No. 2, pp.
153 –69.
Cunliffe, L. (2007) Towards a more
complex description of the role of
assessment as a practice for nurturing
strategic intelligence in art education, in
T. Rayment [Ed.] The Problem of
Asessment in Art & Design. Bristol:
Intellect, pp. 89–106.
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
18
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Groenendijk, T., Damen, M., Haanstra, F.
& Van Boxtel, C. (2016).
Beoordelingsinstrumenten in de
kunstvakken -een review. [Assessment
instruments in the arts subjects: a review].
Pedagogische Studi€
en, 93(2), 62–82.
Hulks, D. (2003) Measuring artistic
performance: the assessment debate and
art education, in N. Addison & L. Burgess
[Eds] Issues in Art and Design Teaching.
London: RoutledgeFalmer, pp. 134–42.
K
arp
ati, A., Zempl
eni, A., Verhelst, N. V.,
Velduijzen, N. H. & Sch€
onau, D. W. (1998)
Expert agreement in judging art projects –
a myth or reality? Studies in Educational
Evaluation, Vol. 24, No. 4, pp. 385–404.
Lindstr€
om, L. (2006) Creativity: What is it?
Can you assess it? Can it be taught?
Journal of Art & Design Education, Vol. 25,
No. 1, pp. 53–66.
Maarleveld, O. & Kortland, H.( 2013)
Beeldend reflecteren: een nieuwe
beoordelingsmethode voor praktisch
beeldend werk in het voortgezet onderwijs.
[Visual reflection: a new assessment
method for studio art in secondary
education] (online). Available at: http://
www.ahk.nl/fileadmin/download/ahk/Lectora
ten/Kunst-_en_cultuureducatie/beeldend_ref
lecteren_-_o_maarleveld_en_h_kortland.pdf
(accessed 3 May 2018)
Nicol, D. J. & MacFarlane-Dick, D. (2006)
Formative assessment and self-regulated
learning: a model and seven principles
of good feedback practice, Studies in
Higher Education, Vol. 31, No. 2, pp.
199–218.
Pallier, G. (2003) Gender differences in self-
assessment of accuracy on cognitive tasks,
Sex Roles,Vol. 48, No. 5/6, pp. 265–76.
Pereira de Ecßa, M. T. T. (2005) Using
portfolios for external assessment: an
experiment in Portugal, International Journal
of Art & Design Education, Vol. 24, No. 2, pp.
209–18.
Ross, J. (2006) The reliability, validity and
utility of self-assessment, Practical
Assessment, Research & Evaluation, Vol. 11,
No. 10, pp. 1–13.
Sadler, R. D. (1998) Formative
assessment and the design of instructional
systems, Instructional Science, Vol. 18, pp.
119–44.
Sch€
onau, D. (2013) Developmental self-
assessment in art education, in A. Karpati &
E. Gaul [Eds] From Child Art to Visual
Language of Youth. Bristol: Intellect, pp.
143–74.
Sluijsmans, D., Joosten-ten Brinke, D. &
Van der Vleuten, C.( 2013) Toetsen met
leerwaarde. Een reviewstudie naar de
effectieve kenmerken van formatief toetsen
[Learning from assessment: a review of
effective features of formative assessment].
Maastricht: NWO-PROO.
Soep, E. (2004) Assessment and visual arts
education, in E. W. Eisner & M. D. Day
[Eds] Handbook of Research and Policy in Art
Education. Mahwah, NJ: Lawrence Erlbaum,
pp. 579–83.
Wagner, E. & Sch€
onau, D. [Eds] ( 2016)
Common European Framework for Visual
Literacy - Prototype.M
€
unster: Waxmann.
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
19
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
Appendix 1
20
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
(a)
(b)
Appendix 2
21
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Appendix 3
Talita Groenendijk, Andrea K
arp
ati and Folkert Haanstra
22
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
Appendix 4
23
iJADE (2019)
©2019 The Authors. iJADE ©2019 NSEAD/John Wiley & Sons Ltd
