ChapterPDF Available

Participative evaluation with children in educational maker projects Experiences from a pilot action

Authors:

Abstract

The European project DOIT follows the maker pedagogy approach to develop an early entrepreneurial education program for children between the age of six and sixteen. In this program, children work collaboratively on solutions for challenges that they know from their personal environments with the aim to aspire to the entrepreneurial spirit and social innovation at the same time. The DOIT program involves 1,000 children in two pilot phases at 11 pilot sites in ten different European countries. A comprising evaluation design has been setup that comprises not only conventional evaluation tools such as questionnaires and tests but also participative instruments. Being aware of that conventional evaluation tools are somewhat in contrast with the innovative setting of a maker project where children are the agents for change, we tried to incorporate participative evaluation instruments where possible. The latter ones have an empowering function as children are not only the 'object' of research but they themselves take over the role of co-researchers. The paper describes some of these instruments and shares some first-hand experiences with the tools and preliminary results, as the project will still be ongoing at the time.
Participative evaluation with children in educational maker
projects
Experiences from a pilot action
Elisabeth Unterfrauner
Technology & Knowledge
Centre for Social Innovation
Vienna Austria
unterfrauner@zsi.at
Christian Voigt
Technology & Knowledge
Centre for Social Innovation
Vienna Austria
voigt@zsi.at
Margit Hofer
Technology & Knowledge
Centre for Social Innovation
Vienna Austria
hofer@zsi.at
ABSTRACT
The European project DOIT follows the maker pedagogy
approach to develop an early entrepreneurial education
program for children between the age of six and sixteen. In
this program, children work collaboratively on solutions for
challenges that they know from their personal environments
with the aim to aspire to the entrepreneurial spirit and social
innovation at the same time. The DOIT program involves
1,000 children in two pilot phases at 11 pilot sites in ten
different European countries. A comprising evaluation
design has been set-up that comprises not only
conventional evaluation tools such as questionnaires and
tests but also participative instruments. Being aware of that
conventional evaluation tools are somewhat in contrast with
the innovative setting of a maker project where children are
the agents for change, we tried to incorporate participative
evaluation instruments where possible. The latter ones have
an empowering function as children are not only the ‘object’
of research but they themselves take over the role of co-
researchers. The paper describes some of these
instruments and shares some first-hand experiences with
the tools and preliminary results, as the project will still be
ongoing at the time.
CCS CONCEPTS
• Evaluation • Active learning • Collaborative learning
KEYWORDS
Participative evaluation, early entrepreneurial education,
maker education.
ACM Reference format:
Elisabeth Unterfrauner, Christian Voigt, Margit Hofer. 2019.
Participative evaluation with children in educational maker
projects: Experiences from a pilot action. In Proceedings of
The 9
th
International Conference on C&T –Transforming
Communities. ACM, New York, NY, USA, 4 pages.
https://doi.org/10.1145/3328320.3328372
1 Introduction
The Maker pedagogy builds on constructive learning
approaches, learning by doing principles, learning in
collaborative and interdisciplinary teams and learning
through trial and error where mistakes are acknowledged as
learning opportunities [3,7]. Acknowledging that all these
principles perfectly match the conditions of entrepreneurial
education, the DOIT project has incorporated the maker
pedagogical approach.
First, the paper describes the DOIT project and the
evaluation design. Secondly, the focus lies on the
participative evaluation approach and different participative
evaluation instruments. Finally, first hand-experiences with
these instruments will be shared.
2 DOIT project
The European project DOIT (Entrepreneurial skills for young
social innovators in an open digital world) is a 3-year
ongoing project with 13 partner organizations from 11
different countries. Its aim is to develop and test an early
entrepreneurial education program for children between the
age of six and sixteen involving 1,000 children in two pilot
phases in ten different European countries.
The developed DOIT program consists of seven
consecutive elements with a course length of 15 hours. It
starts with a motivation phase where students envision the
scope of their possibilities for tackling challenges of their
personal environments. The second step is about Co-
design: Students are asked to collect and select potential
ideas for innovations. In the Co-creation phase students
gather in teams and develop first prototypes based on their
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work
owned by others than ACM must be honore d.
Abstracting with credit is permitted. To copy otherwise, or republish,
to post
on servers or to redistribute to lists, req uires
prior specific
permission
and/or a fee. Request permissions from
permissions@acm.org.
C&T, June 3--7, 2019, Vienna, Austria
© 2019 Association for Computing Machinery.
ACM ISBN
978-1-4503-7162-9/19/06…$15.00
https://doi.org/10.1145/3328320.3328372
C&T Conference 19, June, 2019 Vienna, Austria E. Unterfrauner et al.
ideas. In the iteration phase, the prototype is continuously
improved and in the next step, in the reflection phase,
feedback on the prototypes is exchanged between the
teams. In the last two steps, scaling and reaching out, the
robustness of the prototype is tested with a bigger group of
users outside of the educational setting and finally students
share their projects with a wider public.
3 Participative evaluation with children
The comprehensive evaluation design is based on a mixed
method approach with a theoretical backbone on Lackeus’
model of entrepreneurship education [4], which
distinguishes between entrepreneurial attitudes,
entrepreneurial skills and entrepreneurial knowledge. A pre-
post-test with quantitative measures as well as qualitative
data from interviews with facilitators, students and workshop
protocols will reveal the effectiveness of the program. The
quantitative measures comprise a questionnaire on self-
efficacy and entrepreneurial attitude that was specifically
developed for the purpose of the study as well as a
standardized creativity test [8].
Criteria for selection of the instruments were that they would
work for the big age span, from six to 16 year-olds, and that
they would be easily transferable and translatable from one
language to the other.
However, the maker pedagogy approach is somewhat in
contrast with these rather conventional evaluation tools as
they might create a feeling of being judged and assessed
and thus might resemble a traditional school setting.
Although these rather traditional instruments cannot
completely be omitted as they have their scientific value and
are key to answer some of the core evaluation research
questions, we tried to incorporate participative evaluation
methods were possible. Participative elements allow
children to have an active role in the research process, no
longer being just the ‘object’ of the research but becoming
themselves co-researchers with eventually some insight in
the research process [2,5,9]. This can have a very
empowering function and might even more contribute to
STEM (Science Technology Engineering Mathematics)
education as they are in this way even more directly
involved in the research project. Like in maker projects, also
in participative evaluation the path how to get to specific
results is not strictly pre-defined but depends on the
individual or group, which very much influences and
regulates the process of the data collection.
For this purpose, the DOIT project has installed a children’s
advisory board in order to hear their voices at different
stages of the project. Following a co-design approach, we
have tried out different instruments for participative
evaluation.
4 Participative methods
In the following, we briefly describe these instruments.
The first instrument here, Children interviewing children, is
about the how, i.e. to find out more about how they have
learnt and how they have developed their ideas and turned
them into prototypes. The second one, self-defined learning
goals assessment, is about the what – what have they
learnt in the action? And have the participants achieved
what they wanted to achieve? Finally, the third instrument,
the video commercial, is about the why – why did they
develop the prototype and for whom?
4.1 Children interviewing children
Children were asked to imagine that they were co-
researchers (or reporters for a local newspaper) who
wanted to find out more about the project, what was going
on, what students had learnt, what they had built etc. In the
briefing to the children who volunteered to act as
researchers (reporters), the facilitator of the DOIT action
described the importance and the scope of the interview
and emphasized how important their assistance was and
gave some hints how to do the interview. It was a very open
setting in the sense that children could decide how to
arrange the interviews, whether they preferred to have
another child acting as co-researcher by their side or
whether they would work on their own, or whether the
wanted the facilitator of the workshop assisting during the
interviews. Equipped with a camera and a voice recorder (or
mobile phone) and a rough interview guideline, they were
asked to do the interview and also to take some pictures of
the developed prototype. The interview guideline did not
have to be strictly followed but rather set the frame for the
interview and served as inspiration for potentially different
questions the child wanted to ask. Besides the empowering
function for the child interviewer, the aim of the ‘children
interviewing children’ method was really to hear their voices.
We expected that answers might eventually also be
different if they spoke to a person of their own age whom
they knew.
4.2 Self-defined learning goals assessment
In this exercise we worked with Lego bricks to create an
atmosphere that was more in the maker spirit than a
questionnaire for instance (but it could be replaced with any
maker material).
Children were asked in the beginning of the DOIT action
what they wanted to learn over the course of the action in
the sense of co-creation and learning [1] - instead of us
telling them what they would learn. This could be either an
individual goal (see figure 1) or a group goal (see figure 2).
Participative evaluation with children in educational maker
projects
C&T 2019
,
June 3–7, 2019, Vienna, Austria
Figure 1: Lego exercise – individual result
Figure 2: Lego exercise – group result
The DOIT goals could comprise technical skills, for
instance, learning how to program a Calliope, realizing the
own idea in a prototype, but also emotional components
such as having fun, etc. The goals were noted down and
visualized on a flip-chart. In the last workshop, the children
were asked again whether they thought they had
accomplished what they had wanted to learn. Now the
participants received Lego bricks to visualize their answers
to the self-defined learning goals. For instance, one brick
per goal meant ‘learnt a little’, and four bricks ‘learnt a lot’.
The bricks of all students built together showed then the
group result (see figure 2).
The aim of this evaluation instrument is not only to assess
the accomplishment of goals but in line with the participative
approach to let them define their learning goals, instead of
the facilitators telling them [6].
4.3 Video commercial
Groups of children who had worked together in a group and
had jointly developed a prototype were equipped with a
video camera or a mobile phone. The facilitator instructed
them to shoot a video in the style of a TV commercial where
they present their prototype with some basic information to
share, like what is the name of the prototype, what can it do,
which problem does it solve and for whom is it made. A
simplified business canvas helped them to better prepare
for the presentation and the video shoot. They also received
a simple version of a script template where they could note
down the sequence of their shooting, e.g. the setting and
location, who would speak, when and what. Besides these
little hints, the children were free to design the video the
way they wanted and they decided on their own, how to
actually shoot the video. In the still image from a video
below (figure 3), children show their prototype of an alarm
system for flooding of creeks. The real story behind was
that some of them spent their holidays on a mountain
pasture surrounded by little creeks. It happened quite often
that the creek would flood the pasture and this was
particularly a problem if this happened at night. Their alarm
system would alert them during the night well before the
flooding, so they would still have time to prepare against it.
Figure 3: Still image from video
5 Preliminary results
As in the moment of writing we are in the midst of the first
pilot phase, we do not have the complete picture yet but still
we do have some preliminary insights and first-hand
experiences with these participative instruments that are
worth sharing.
C&T Conference 19, June, 2019 Vienna, Austria E. Unterfrauner et al.
Children interviewing children: Although we recommended
that for this method interviewing children should have a
minimum age of 10 years, also younger children
volunteered to act as co-researchers. The analysis of the
interview protocols shows indeed quite an age difference in
the depth of the interviews. The younger ones strictly
followed the interview guideline and did not add questions
on the spot. Also they seemed to be happy with any answer
given. For instance, when they asked another child what
they had done during the DOIT action and that child would
answer simply with ‘I don’t know’, then they would not follow
up on the question but move to the next one in the guideline
instead. The elder ones, above the age of ten, however,
digged deeper and also added some questions
occasionally. All in all, the interviews lasted less than
expected, sometimes as short as two minutes, up to about
ten minutes. The interviewing children seemed to like the
role and called it a fun experience (which we could tell also
from the recording where there was some giggling in
between). The method in so far worked well as we would
get sometimes very sincere answers, which potentially
might have been different if an adult were there to ask the
question instead (e.g. when asking how they liked the
workshop and what they did not like). However, a good
briefing before the interview turned out to be key in order to
produce meaningful results.
Lego bricks exercise: The exercise worked very well, also
for the younger children as they could easily relate to the
quantitative measure displayed with Lego bricks. Thus, it
was easy for the children to grasp what is a lot and what is
a little and what is the difference between. Especially with
younger children, the definition of learning goals worked
better as group than individual goals as defining goals
seemed rather challenging for the younger age group.
Interestingly, the exercise resulted in spontaneous
assemblies of group results. For instance, children would
work together in the groups they were prototyping with and
show a combined Lego bricks visualization of their answers.
Commercial video: The commercial video shoot worked
surprisingly well. All children managed to shoot a video that
contained the necessary information, some in a very
creative way. For instance, in one video they would do role-
playing or in another one they would find a funny way how
to change from one speaker to the next. All were tech-savvy
enough to finalize the video. Although it was technically low-
key (we did not apply video cutting), it was surprising how
few shoots they would need before they would have their
final video. The results were quite impressive as the
facilitators found. They acknowledged that the videos also
showed an emotional facet, i.e. how proud they were of
what they had done and produced during the DOIT
workshop.
6 Conclusions
Participative methods deserve their spot in scientific
research projects complementing the more conventional
evaluation instruments and methods. They are specifically
well suited to evaluate some of the evaluation questions in
educational maker projects, as they seem to reflect the
maker pedagogical approach better than the more
traditional instruments, which might create some tensions
as participants might be reminded of an assessment
situation. The paper has explored three different
instruments to answer different research questions to get
potentially different results from children interviewing other
children instead of adults performing this task, to assess the
learning goals and to understand why and what for they
have developed their prototypes. The three exercises have
worked in so far that most resulted in meaningful evaluation
data. However, of these three the most demanding one was
certainly the interviews conducted by children. As the first-
hand experiences show particularly younger children need
support by a facilitator. Sufficient time allocated for briefing
the children acting as co-researchers is key in order to
result in interviews with a certain level of details that can
actually be used for further analysis.
ACKNOWLEDGMENTS
DOIT has received funding from the European Union’s
Horizon 2020 research and innovation program under grant
agreement No 770063.
REFERENCES
[1] K. B. Akhilesh. 2017. Co-Creation and Learning: Concepts and Cases.
Springer.
[2] Caroline Bradbury-Jones and Julie Taylor. 2 015. Engaging with
children as co-researchers: challenges, counter-challenges and
solutions. International Journal of Social Research Methodology 18, 2
(2015), 161–173.
[3] Gwen Snyder Kaltman. 2010. Hands-on learning. Childhood Education
87, 2 (2010), S7–S7.
[4] Martin Lack éus. 2015. Entrepreneurship in education: W hat, why,
when, how. Background Paper (2015).
[5] Laura Lundy, Lesley McEv oy, and Bronagh Byrne. 2011. Working with
young children as c o-researchers: An approach informed by the United
Nations Convention on the Rights of the Child. Early education &
development 22, 5 (2011), 714–736.
[6] T. Shen, Dezhi Wu, V. Archhpiliya, M. Bierbert, and Roxanne Hiltz.
2004. Participatory learning approach: A research agenda. Information
Systems Department (2004).
[7] Elisabeth Unterfrauner, Christian Voigt, and Sandra Schön. 2018.
Towards a model of early entrepreneurial education: appreciation,
facilitation and evaluation. In Methodologies and Intelligent Systems for
Technology Enhanced Learning, 8th International Conference, 139–
146.
[8] Klaus K. Urban and Hans G. Jellen. 1995. Test zum Schöpferischen
Denken-Zeichnerisch (TSD-Z). Swets.
[9] Fionnuala Waldron. 2006. CHILDREN AS CO-RESEARCHERS.
Research Methods in Social Studies Education: Contemporary Issues
and Perspectives (2006), 85.
... While democratizing innovation is the traditional focus of maker movement, there is also an entrepreneurial vision that positions making as an enabler of business innovation and a key to entrepreneurship [22,38]. However, studies that combine entrepreneurship education with digital fabrication and making are scarce [35,36], even if both entrepreneurship education and digital fabrication and making have been widely studied separately. Studies on digital fabrication and making tend also to focus on success stories rather than on scrutinizing the challenges involved [25]. ...
... The combination of digital fabrication and making with entrepreneurship education has been acknowledged as having potential to produce the 'digital innovators of future' [16] and found to increase university students' entrepreneurial self-efficacy and entrepreneurial intentions [24]. Attempts to try the combination out in education have still been rarely reported [12,15,35,36,38] and very little is known on how to do it in practice. Hollauf et al. [15] propose it is important to motivate children (allow mistakes, solving problems that are easily relatable, start with easy tasks), have an open process (asking open design questions, scaffolding children's work but encouraging them to do as much as possible themselves, making technology exploration as easy as possible), and open outcomes (iterative process, focus on learning from the process itself, aiming for positive outcomes). ...
Article
Full-text available
Participatory approaches have become de rigueur in social research involving children. A growing trend is research by children where researchers engage (or employ) children as co-researchers or primary researchers. In this paper, we critique the ethical, methodological and practical issues associated with this participatory approach. The discussion is framed around six challenges: (1) Children lack research competence; (2) A comprehensive training programme is required; (3) Insider/outsider perspectives are difficult to balance; (4) Remuneration is complex; (5) Power differentials need to be overcome; and (6) Children need to be protected. For each challenge we propose a counter-challenge. Additionally, we offer pragmatic solutions to the issues raised, so that the paper holds practical utility to social researchers who utilise this type of participatory approach. Overall we argue that despite the approach’s inherent challenges, children as researchers are a powerful conduit for other children’s voices.
Article
Full-text available
Research Findings: Under the United Nations Convention on the Rights of the Child (UNCRC), children have the right to express their views on all matters affecting them and to have those views given due weight. This right applies in the context of research; however, examples of young children being engaged as co-researchers remain rare. Practice or Policy: This article examines the implications of adopting an explicit UNCRC-informed approach to engaging children as co-researchers. It draws on a research project that sought to ascertain young children's views on after-school programs and that involved a university-based research team working along with 2 groups of co-researchers; each composed of 4 children aged 4 to 5. The article discusses the contribution made by children to the development of the research questions and choice of methods and their involvement in the interpretation of the data and dissemination of the findings. It suggests that, although there are limits to what young children can and will want to do in the context of adult-led research studies, an explicit UNCRC-informed approach requires the adoption of supportive strategies that can assist children to engage in a meaningful way, with consequent benefits for the research findings and outputs.
Book
This book introduces readers to co-creation --- a complex, value-based, context-driven and collaborative effort to develop new paradigms, products and services to satisfy human wants. Co-creation is built not only around the perceptions of challenges, cause-and-effect relationships and constraints, but also around available alternatives for dealing with or overcoming those challenges. Co-creation is not about transferring or outsourcing activities, and neither is it about the customization of products and services. This book explains the emergence of the co-creation approach. It describes various models of value creation, as well as different stages and the contract process involved in co-creation. It also explores different types of learning and learning techniques, and how co-creation impacts the learning process. The book allows practitioners and policymakers to understand the processes involved in implementing co-creation in any organization, while also presenting case studies to show how to apply the relevant concepts in their day-to-day activities.
Participatory learning approach: A research agenda
  • T Shen Dezhi Wu
  • V Archhpiliya
  • M Bierbert
  • Roxanne Hiltz
T. Shen, Dezhi Wu, V. Archhpiliya, M. Bierbert, and Roxanne Hiltz. 2004. Participatory learning approach: A research agenda. Information Systems Department (2004).
Towards a model of early entrepreneurial education: appreciation, facilitation and evaluation
  • Elisabeth Unterfrauner
  • Christian Voigt
  • Sandra Schön
Elisabeth Unterfrauner, Christian Voigt, and Sandra Schön. 2018. Towards a model of early entrepreneurial education: appreciation, facilitation and evaluation. In Methodologies and Intelligent Systems for Technology Enhanced Learning, 8th International Conference, 139-146.
CHILDREN AS CO-RESEARCHERS. Research Methods in Social Studies Education: Contemporary Issues and Perspectives (2006) 85. Fionnuala Waldron. 2006. CHILDREN AS CO-RESEARCHERS
  • Fionnuala Waldron
Fionnuala Waldron. 2006. CHILDREN AS CO-RESEARCHERS. Research Methods in Social Studies Education: Contemporary Issues and Perspectives (2006), 85.
Entrepreneurship in education: What why when how
  • Martin Lackéus