Content uploaded by Peter Sommerauer
Author content
All content in this area was uploaded by Peter Sommerauer on Feb 07, 2020
Content may be subject to copyright.
From ESCO to LAAR
(white paper)
A reflection on the use of Augmented Reality and Learning Analytics.
This project LAAR (20 17-1-LI01-KA202-000087) has been funded with support from the European Commission.
This publication [communication] reflects the views only of the author, and the Commission cannot be
held responsible for any use which may be made of the information contained therein.
Colofon
Erasmus+ KA2 Strategic Partnership Project 2017-1-LI01-KA202-000087
Project team: Angelika Erlacher, Ulrich Habicher, Will Guest, Leonard Maxim, Oliver Müller, Tommy
Neumann, Niels Østman, Sacha Ritter, Joshua Secretan, Peter Sommerauer, Ralf Stroetmann , Chris
Van Goethem, Fridolin Wild
Text: Chris Van Goethem, Peter Sommerauer
Lay-out: Chris Van Goethem
ISBN 978-3-03912-006-2
Publishing date: 30-09-2019
Publisher: Laurentius Verlag
i-smARt Trust reg.
Feldkircher Str. 50
9494 Schaan
Principality of Liechtenstein
E-Mail : verlag@laurentius.li
Web : www.laurentius.li Tel.: +423/370 24 01
Copyright
This document is published under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
International Public licence
CC BY-NC-SA
Index
Introduction ....................................................................................................................................... 7
Project description ............................................................................................................................. 8
The writers ......................................................................................................................................... 9
Part 1................................................................................................................................................ 10
1 Introduction part 1 .................................................................................................................... 11
1.1 Target groups .................................................................................................................... 12
1.2 Learners of today .............................................................................................................. 13
1.3 We are teaching “history” ................................................................................................. 13
1.4 4 scenarios ........................................................................................................................ 14
1.5 ETTE as a test case ............................................................................................................. 14
2 The concept of a competence ................................................................................................... 15
2.1 Occupations vs. competence ............................................................................................. 15
2.2 Confusion about competence descriptions ........................................................................ 16
2.3 Competence vs knowledge ................................................................................................ 16
2.4 The conflict with traditional school evaluation................................................................... 16
2.5 The conflict with a traditional “course model” ................................................................... 17
2.6 How and why does this feed into LAAR? ............................................................................ 18
3 ESCO ......................................................................................................................................... 19
3.1 ESCO structure .................................................................................................................. 19
3.2 The ESCO theatre Technical competences ......................................................................... 20
3.3 Where it went wrong, a critical note .................................................................................. 22
3.4 Limitations of ESCO ........................................................................................................... 23
3.5 ESCO and “General education” .......................................................................................... 24
3.6 ESCO as a backbone........................................................................................................... 24
3.7 ESCO in comparison with other transparency systems for VET ........................................... 24
3.8 How and why does this feed into LAAR? ............................................................................ 25
4 A sectoral layer ......................................................................................................................... 27
4.1 Structure and information of a sectoral layer competence ................................................ 28
4.1.1 Skills, knowledge and attitude ................................................................................... 29
4.1.2 Credit systems, ECVET and ECTS ................................................................................ 30
4.1.3 Levels and EQF ........................................................................................................... 30
4.1.4 Example of a sectoral layer competence description .................................................. 31
4.2 Structure and information of a sectoral layer occupation or profile ................................... 32
4.2.1 The logical order of competences .............................................................................. 32
4.2.2 Example of a sectoral layer profile ............................................................................. 33
4.2.3 Occupational vs. educational profiles ......................................................................... 35
4.2.4 work focused profiles ................................................................................................ 35
4.3 How and why does this feed into LAAR? ............................................................................ 36
5 Tools and applications based on a sectoral layer ....................................................................... 37
5.1 Developing occupational and educational profiles ............................................................. 37
5.2 Developing curricula and courses ...................................................................................... 37
5.3 Training needs analysis ...................................................................................................... 37
5.4 Follow up of apprentices ................................................................................................... 37
5.5 A structured portfolio ........................................................................................................ 38
5.6 Personal Development plan ............................................................................................... 38
5.7 Combining tools ................................................................................................................ 38
5.8 How and why does this feed into LAAR? ............................................................................ 39
6 The learning wallet.................................................................................................................... 41
6.1 The learning wallet and the Sectoral Layer of ESCO ........................................................... 42
6.2 Lifelong Learning and the Learning Wallet ......................................................................... 42
6.3 Future use ......................................................................................................................... 43
6.4 Badges and virtual certificates ........................................................................................... 43
6.5 How and why does this feed into LAAR? ............................................................................ 44
7 Teaching and training ................................................................................................................ 45
7.1 Different types of learning content .................................................................................... 45
7.2 Different pathways to competence ................................................................................... 45
7.3 Complexity of a real-life working environment for training ................................................ 46
7.4 The importance of repetition ............................................................................................. 46
7.5 How and why does this feed into LAAR? ............................................................................ 47
8 Measurement, evaluation and assessment in vocational education .......................................... 49
8.1 Developing evaluation measurement ................................................................................ 50
8.2 Developing assessment ..................................................................................................... 51
8.3 How and why does this feed into LAAR? ............................................................................ 52
9 Generic types of applications for digital learning environments ................................................ 53
9.1 Digital paper books ............................................................................................................ 53
9.2 AV-Media based content ................................................................................................... 53
9.3 Interaction tools ................................................................................................................ 55
9.3.1 Simulation tools ......................................................................................................... 55
9.4 Simulating real equipment ................................................................................................ 57
9.5 Augmented Reality and Virtual Reality (VR) applications .................................................... 59
9.5.1 Augmented Reality (AR) applications ......................................................................... 60
9.5.2 Virtial Reality (VR) applications .................................................................................. 60
9.5.3 Learning analytics (LA) ............................................................................................... 60
9.6 Learning Management Systems (LMS) ............................................................................... 60
9.6.1 Micro learning ........................................................................................................... 61
9.7 Virtual Learning Environments (VLE) .................................................................................. 61
9.8 How and why does this feed into LAAR? ............................................................................ 62
Part 2................................................................................................................................................ 63
Introduction ..................................................................................................................................... 65
10 Preparation and implementation in LAAR .............................................................................. 67
10.1 The choice of training scenarios......................................................................................... 67
10.2 The theoretical framework ................................................................................................ 68
10.3 The design framework ....................................................................................................... 69
10.4 The implementation .......................................................................................................... 70
10.5 Learning the safety signs ................................................................................................... 71
10.5.1 Analysing the conventional learning method ............................................................. 71
10.5.2 The i-Phone/i-Pad app “Explore-App” and “Who Am I?” ............................................ 71
10.6 Two ways to train the egg.................................................................................................. 75
10.6.1 Analysing the procedure ............................................................................................ 75
10.6.2 Analysing the conventional learning method ............................................................. 76
10.6.3 The phone app “Follow Me” ...................................................................................... 77
10.6.4 The Hololens app ....................................................................................................... 81
10.6.5 Virtual learning environment ..................................................................................... 84
11 Learning Analytics (LA) .......................................................................................................... 87
11.1 What is happening in the background? .............................................................................. 87
11.1.1 The xAPI statements .................................................................................................. 88
11.1.2 The developed xAPI statements ................................................................................. 89
11.1.3 The Learning Record Store (LRS) ................................................................................ 90
11.1.4 The competence analyser .......................................................................................... 91
11.1.5 Measuring performance for Learning analytics .......................................................... 91
11.2 The dashboard .................................................................................................................. 91
12 Testing one two .................................................................................................................... 95
12.1 Prototype testing............................................................................................................... 95
12.2 User experiences ............................................................................................................... 96
13 Strengths and limitations of AR - a SWOT Analysis ................................................................. 99
13.1 AR strengths for workplace trainings ................................................................................. 99
13.2 AR weaknesses for workplace trainings ............................................................................. 99
13.3 AR workplace training Opportunities ................................................................................. 99
13.4 AR Workplace Training threats ........................................................................................ 100
14 Conclusion .......................................................................................................................... 101
7
Introduction
The document in front of you is the reflection of the Erasmus+ strategic partnership project LAAR, or
Learning Analytics and Augmented Reality. With this text, we aim at the layman reader that is
interested in education and training, without being an expert in the fields of education or training,
neither as the field of digital learning platforms and applications.
In the first part of this text, we take you along our path to discover the existing knowledge, systems
and visions that inspired us during the development of the project. In the second part you find the
developments the project team added to these inspirations, the way we used them and the concrete
results of the project.
We hope you will enjoy it and get inspired by it.
8
Project description
Augmented Reality (AR) has the potential to support effective learning in informal and non-formal
learning environments and not only in school or higher education but also in professional education.
Especially in the building/construction industry or in the Event Industry, trainings for safety and
security are needed to protect workers at their workplace. One central issue hereby is, which
elements must be taken in consideration (i.e. teaching and learning elements, design elements, etc.)
when developing effective learning applications. Moreover, how can such applications support
responsible persons at the workplace so that they can ensure, that each employee has been trained
in safety & security and has understood the key elements of the training and has the ability to
behave according the instructions.
This is where Learning Analytics (LA) comes into place. LA provides various tools and concepts to
support learning, especially for people who are involved in learning processes (trainer, trainee,
evaluator). The project aims to create a framework on the basis of LA insights, for implementing
elements in AR learning applications which supports learners and trainers in their teaching and
learning processes towards a higher efficiency and efficacy. At this point, elements from LA like
functions and integration of analytical elements shall be implemented to receive feedback about
handling and ease of use from the apps, as well as measures for learning performance. In various
iterations, a series of pilot applications shall be developed and tested within specific training areas,
like facility management, building/construction, and in the Event Industry (i.e. Hilversum test as
developed by the ERASMUS+ project “ETTE” for traditional training). From the results and insights,
key questions should be answered like how can AR and LA be used in professional training
applications to create more effective learning experiences for learners on their individual path and
pace. Additionally, receiving insights about data required for "big-data analysis methods" and how
can they be processed in a small work environment, or which methods can be developed to derive
knowledge from established, large learning environments. A concrete, desirable result would be to
provide recommendations about how the transparency of learning services in the understanding of
learning outcomes can be depicted with the aid of LA and AR in order to dismiss the responsibility for
employers not only to offer the possibility to participate in safety briefings (consumption) but also to
ensure that employees have understood and will behave adequately.
The consortium of leading European scientist coming from the areas LA and AR aims to establish a
strategical partnership with VET partners, in which the conceptualisation of the topics LA and AR
should be used to apply in the field of professional education, especially in informal learning
environments and directly at the workplaces. Since one result should cover also a contribution to the
research community, the project aims to develop new opportunities at workplaces and also for LA
and AR to provide an added value and give answer to current questions from professional education.
The consortium consists of the partner organisations i-smARt (Coordination in LI), IT University
Copenhagen (DK), Oxford Brookes University (GB), STEPP vzw (BE), VPLT (DE) and aims to establish a
strategic partnership in which the indicated topics and aspects should be elaborated. Within a first
development of a generic framework, current research results and their leading questions are
included, also to develop the framework as a software application. In a second step, a first functional
prototype of AR learning application will be developed for analysis and evaluation purposes, which
then will be - according the design science approach - applied in various iterations within field
experiments at workplaces. Finally, the goal of the project is to support a full session training and
assessment (i.e. Hilversum test, project ETTE) on the basis of AR learning applications and the use of
LA tools and measures.
9
The writers
Chris Van Goethem
Chris Van Goethem started working as a stage manager in the
eighties and toured around Europe with several Belgian
companies. In the nineties he started to work as a consultant and
educator in technical theatre.
He consulted the creation of the Flemish professional profiles
and standards for theatre technicians, and the start-up of several
education programs in initial, higher, and adult education. Today,
he consults the Social fund for the performing arts in educational
matters.
Chris Van Goethem chaired the ESCO sectoral reference group for arts, entertainment and recreation
and he is an active member of the TALQ research project. He is involved in several other projects
using ESCO competences such as the ETTE project, which is developing a European safety passport,
TeBeVat, which is researching validation and portfolio techniques, and LAAR, which is developing
learning analytics and augmented reality applications.
He is responsible for the Expertise Centre for Technical Theatre of RITCS School of Arts, Erasmus
University College Brussels, and is involved in research on education, competence management, and
history of technical theatre. ( www.podiumtechnieken.be/onderzoek/english )
He is member of the OISTAT Education Commission (Organisation international des scénographes,
techniciens et architectes de théâtre) and the board of STEPP (the Belgian Technicians organisation).
Peter Sommerauer
Peter Sommerauer is a successful entrepreneur and founder of 7+
small size organizations with up to 25 employees. He grew up with
the early development of IT and studied information science right
after his first career as mechanical engineer. Later, he obtained a
Bachelor’s degree in Business Informatics, a Master’s degree in IT
and Business Process Management, and a PhD in Business
Information Systems.
Peter initiated, lead and participated in outstanding and highly
esteemed European projects for the Event technology, e.g. ECVAET,
ETonTour, step2mice, LAAR. He has more than 11 years of
experience in European project management. He conducts his
research activities at the interface between vocational training and higher education and with a focus
on media-based learning. With his technology-oriented focus, Peter is investigating digitization for
brain-friendly learning.
10
Part 1
11
1
Introduction part 1
The LAAR project develops learning applications for the event and the performing arts sector. The
choice for this specific sector is not arbitrary. The event and performing arts sector is a very flexible,
international and fast evaluating sector. Therefore, it incorporates almost all tumbling stones
occurring in other sectors. Solutions that work in this sector will easily be transferred to other
sectors.
In this first part we introduce step by step the concepts that are behind the LAAR project. These
concepts are mainly derived from the EU vision on competence management and a series of projects
that have been developing these concepts into practical solutions. They support the vison of the
project and underbuilt the application development from an educational / competence management
point of view. The text includes the analysis and reflections made during the development of the
project.
The ETTE project, a project that developed a basic European Safety Certificate, with supporting
learning material, is used as a test case for the development of the LAAR applications. It shares a
common vision and uses the different elements described below.
(https://www.podiumtechnieken.be/en/health-safety-and-sustainability/ette )
• The first chapter describes the specific target group and reflects on the learning context of
contemporary learners. This chapter also introduces the scenario’s used to clarify the
concepts.
• In chapter two we describe the concept of competence in general, define the relation with
knowledge and occupations, tackle common misunderstandings and conflicts with traditional
learning systems.
• In chapter three we explain the structure and concepts of The European Skills/competence,
Qualifications and Occupations (ESCO) and how the technical theatre competences are
developed. We reflect on the possibilities and limitations, their relation with education and
other transparency systems for VET.
• Chapter four explains the idea of a sectoral layer, on top of ESCO, as developed in the TALQ
project. It details the information about the content of competences needed to create
learning systems.
• Chapter five gives an overview of tools and applications that are developed based on the
sectoral layer concept and can be (re)used in the LAAR applications.
• Chapter six describes the concept of a learning wallet or learning locker, the xAPI
(Experience API) and it’s use to exchange information between different applications and to
integrate different learning methods into one coherent structure.
• Chapter seven reflects on how teaching and training is organised for the specific target
group in a complex real-life working environment.
• Chapter eight reflects on how measurement, evaluation, and assessment in vocational
education can be organised.
• Chapter nine gives an overview of existing applications for digital learning environments
and how they can be used.
The second part of this work will elaborate on the development of the applications and the practical
outcomes of the LAAR project.
12
1.1
Target groups
The target groups of the LAAR project are related with the event and performing arts sector, a sector
interrelated with other industries and occupations like carpenter, electricians, scaffolders, light-
/audio/media managers, … . The target group is also very diverse if we look at their background. This
implies a large variety of learning contexts that apply in parallel of the traditional learning pathways.
Even though every learner and his/her situation are unique, we can group them in typical learning
situations:
Often formal training in a European educational (school) context attracts mostly young learners that
are still in their “school career”. The education is not only focused on sector specific skills, but also
includes general education.
A specific form of formal training for young learners is Dual Learning. The combination of theoretical
education in a school context and practical training in a work placement is very suitable for the
theatre-technical and event field. It combines the “safe space” of a school with the complex reality of
the sector that is hard to simulate.
Professionals that want to “climb the ladder” are a second target group. The organisation of crews,
the tradition in the sector and the stepped relation between occupations makes this an obvious path.
A worker starts as a stage hand, becomes a technician after some years, grows on towards a
specialism and becomes crew chief. The difficulty with this target group is that it is hard to get them
together. They have a lot of (waste/waiting) time while doing their job, but it is difficult to plan off
days for training because they depend largely on the production planning.
A related group to the one before is professionals in need of continuous education. The rapid change
in technology and production methods provokes the need for a permanent update of competence.
Even though the principles of live entertainment have stayed the same for centuries, the technical
means that are used change at high speed. The planning issues are similar to the group above.
Both groups above consist largely of freelancers. This is not only the result of an economic reality,
but also a choice. High level professionals need to be challenged to keep their level, so they look for
productions that challenge their competence. Moreover, high level professionals are not only asked
for their skills, but also for their artistic taste. To be able to cooperate with artist they need to
incarnated with the style of the artist and “feel” the underlaying artistic vision. This often means they
“follow” artists to different productions. Most freelances work for several artists at the same time,
making planning even more difficult. Freelancers in most countries are not very good protected and
every day they don’t work is a day they are not payed. As they don’t have a permanent employer, a
day of training is a day not worked.
Another specific group are adult learners that want a career switch. Often these are artists that want
to get into the technical field. This can be because their career ended (for example dancers) or
because they can’t make a living out of their art. The advantage is they already understand the
production process and the artistic environment. On the other hand, there are also people from
other sectors that enter laterally in the field of technical theatre, often because they decide to make
their dream come true or because they want to leave a purely competitive, commercial environment.
A last group are practitioners that need extra training because of mobility. This can be professionals
moving to another country, refugees, … They need to adapt their skills to a new, local context and
need to learn a new professional vocabulary. Often their general language skills are (still) limited and
they need to adapt to the local culture. Especially in a sector where the artistic language requires
13
understanding of nuances in wording, describing feelings and emotions, this is a limitation. On the
other hand, most artistic contexts are multilingual and multicultural.
For many individual learners, a combination of these definitions will apply. This asks for flexible,
individualised solutions that adapt to the personal needs of the learner.
1.2
Learners of today
The learners of today haven’t changed so much since the introduction of the internet, but their
environment has. They are constantly exposed to large amounts of information, they live in a more
complex, permanently changing world. This makes they have to develop a strategy to deal with this
overload of incoming stimuli in an efficient way. One of these strategies is “scratching the surface”,
getting a basic understanding of the incoming knowledge by reading the headers or short
descriptions. This strategy helps in daily life, but is not sufficient to build skills and competence on.
They miss a framework to put all this information in and relate it.
In the event and performing arts sector, a more than average number of learners have been
diagnosed earlier with all kind of learning difficulties like ADHD, elements of autism spectrum,
dyslexia, … These difficulties have often limited their “school career”, but they seem to adapt well in
a creative sector, where the properties of these difficulties seem to become valuable gains.
Based on our experience, most learners attracted to the creative sector are active learners. They
don’t like to listen but they do like to do. They are hands on and like to learn on a “need to know
basis”, starting from the practice and then looking for the theory behind it. Often this coincides with
a “renaissance man” type of interest, a combination of specialisation with a broad, wide angled view
on the totality of the result and the elements that influence them. This attitude is needed for creative
occupations that combine technical, social and artistical skills in a complex environment.
The learners of today have to adapt to a high-quality standard that is different from the daily life
standard. Where in daily life quality is “what is acceptable to the majority”, in arts and entertainment
the standard is “the best possible”. We are supposed to deliver the best show, not an acceptable
show.
Most learners (in contradiction with a lot of pure artists) are instinctively proficient in digital media
and don’t like to be “beaten” by a computer, which is a strong motivation.
1.3
We are teaching “history”
The event and performing arts sector is based on century old principles and traditions, but
encounters on the other hand an extreme fast technological evolution. New technology is brought
out so fast that the debugging is “outsourced” to the users. This makes that whatever equipment or
technology we teach, it is outdated by the moment students leave school. It becomes impossible to
keep curricula and training with the latest, state of the art, insights and technology. We are almost
per definition teaching “history”.
This insight has an important influence on what we teach and train. It does no longer make sense to
train learners for a specific device, technology or application, we need to train them to acquaint
themselves with new equipment and master new technologies by themselves.
14
1.4
4 scenarios
To make the use of the different concepts, systems and methods more visible, we use four scenarios
to follow the needs and labour status of four different people. These four people represent typical
situations occurring in the performing arts and event sector.
§ Nina Tiptoe, 38, a dancer at the end of her dance career, wanting to get into technical
theatre or stage management. She wants to prepare for the transition during her last year of
work as a dancer. She is originally Spanish, but lived and Worked in Germany, Austria,
Belgium and Sweden for years.
§ Tony Solo, 30, a freelancer in the peak of his career. He works 7/7 and tours all over Europe
with different companies. In low season he works for events and trade shows.
§ Ben Young, 21, is fascinated by the technical side of performance and the music scene. He
didn’t fit into the traditional education system, but his head was always into inventing and
experimenting with new equipment. At the moment he is apprentice in a rental company. In
his free time you can find him in the youth club.
§ Mary Standard, 17, is at school to become a multiskilled technician. She has always been
good in science and technology. But she has no experience in the real world of theatre or
rock and Roll. In her free time, you can find her in the sports club, where she likes to climb.
1.5
ETTE as a test case
As mentioned above, the content developed by the ETTE project is used by the LAAR project as a test
case for the concepts, tools and applications. Roughly we can say the project developed learning
content and assessment methods based on a set of ESCO competences that were detailed in a
sectoral layer. We will introduce all these elements step by step when describing the concepts
behind them.
15
2
The concept of a competence
The LAAR project developed applications for learning in a vocational context. To describe and
measure what is learned we need a common language that is related to the larger context of work
and education in Europe. We use competences as descriptor for the learning outcomes. To define
the exact meaning of the concept of competence, we looked into definitions that are adapted to
vocational training, and are general accepted and used in the EU.
Cedefop defines competence as “The ability to apply learning outcomes adequately in a defined
context (education, work, personal or professional development).” Or “The ability to use knowledge,
skills and personal, social and/or methodological abilities, in work or study situations and in
professional and personal development.“. Additionally, it comments that “competence is not limited
to cognitive elements (involving the use of theory, concepts or tacit knowledge); it also encompasses
functional aspects (including technical skills) as well as interpersonal attributes (e.g. social or
organisational skills) and ethical values.”
ESCO (European Skills/Competences, qualifications and Occupations) and EQF (European
Qualification Framework) apply the same definition: "competence means the proven ability to use
knowledge, skills and personal, social and/or methodological abilities, in work or study situations and
in professional and personal development." ESCO clarifies further that “They are described in terms of
responsibility and autonomy.”
In relation to the difference with the concept of skills ESCO states: “While sometimes used as
synonyms, the terms skill and competence can be distinguished according to their scope. The term
skill refers typically to the use of methods or instruments in a particular setting and in relation to
defined tasks. The term competence is broader and refers typically to the ability of a person - facing
new situations and unforeseen challenges - to use and apply knowledge and skills in an independent
and self-directed way.”
The differences between the definitions are small and can be reduced to the different point of view
of the makers. Where Cedefop focusses on learning, ESCO has a broader view and can also be used in
Occupational Standards, HR, etc. This is reflected in the fact that ESCO states that the competence
has to be proven.
The above and the fact that that ESCO has an existing database of competences that can be used in
the event and performing arts sector makes that we chose to use the ESCO definition.
In the side-line, we want to stress the importance of readability of the competence descriptions.
They should not only be understood by educational or labour market experts, but also by workers
that can use them to describe their own practice. This is especially important and useful for lower
EQF level occupations. On a higher level, the descriptions will be, per definition, more abstract. But
one can expect also from workers on this level to deal with this abstractness.
2.1
Occupations vs. competence
In an international context, common descriptions of occupations in the performing arts and event
sector are not very workable. A stage manager is not a "toneelmeester" or " régisseur général ". A
stage manager in an opera house needs other skills than his colleague in a theatre. The titles and the
content vary depending on the organization of work, the type and size of organization, type of
productions and the composition of the crew. Of course, all necessary tasks for a stage production
are performed by someone. However, they are not always performed by the same function or in the
same combination of functions.
16
This ascertainment makes that the focus for a workable system that can describe what is needed for
a job or what a professional should know or learn, is more on the level of competence, rather than
on the level of occupation. An additional advantage of this focus is that it honours the unique skills
that are available in each person and that can be exploited in a creative/flexible organisation.
2.2
Confusion about competence descriptions
Describing competences is not abstract science. There are different views on how to describe an
individual’s competence or how to describe the needed competence for a job or activity. The
descriptions occur in different sizes, using different language and with different amounts of detail.
These different views often derive from different uses. Job applications, HR, training, teaching,
assessment, careers coaching, all use different shapes of descriptions. The reason for this is often
that they have to match existing formats, systems or methods. These formats, systems or methods
will often set technical standards and limitations on how competences should be written and on the
amount of competences that are allowed. Sometimes, the descriptions are just a “translation” of
existing material like school programs or job descriptions into a format fit for the requested use.
All this makes that competences are not very exchangeable or transferable between different
services.
2.3
Competence vs knowledge
In daily life we often say things like “you have to know French”, “you have to know your history” or
“you have to know how to repair your car”. Even if we all understand what it means, these sentences
do not make sense when you analyse them. Knowing French without being able to speak it or to
write it, is not very useful. Knowing history without being able to relate facts to each other or to the
present does not help us. And knowing how to repair my car without being able to do this only leads
to large frustration.
Knowledge is like a closed book, on its own it is not really helpful if you can’t apply it. This does not
mean knowledge should be limited to what is directly applicable in the labour market. Building a
framework of knowledge to understand the context you are working on is a good and valid reason to
gather knowledge. For example: understanding the world history or the history of the arts is crucial
for people working in an artistic environment.
In relation to the term “competence” we could say the knowledge is “underpinning”, it supports (and
is an essential part of) the competence. In other words, if you are competent you have gathered the
required knowledge.
2.4
The conflict with traditional school evaluation
To master a competence means that you are able to perform the described activity completely. You
are able to do what is written down as “needed to fulfil a task”. The result is yes or no, you are able
or you are not able. In traditional school systems we grade students. We decide how much of the
content we offer them is understood, how much competence they have gathered. We make a
gradation between the performance of different students in relation to the targets set. And the
students “pass” if they reach 50% of these targets.
In real life we don’t expect 50%, we expect that you are able to perform in full. A fireman that is able
to extinguish a fire for 70% fails in real life, but in most education systems he/she would be
congratulated for excellent performance.
17
To compensate this, (most) education systems set their targets higher than what would be expected.
The problem is that we don’t know what the student is able to and what part is missing. A second
issue is that if we compare “school” competences with “sector” competences we get a false image of
the reality.
2.5
The conflict with a traditional “course model”
Many education programs are structured in courses or modules. These courses are still structured
based on knowledge that is developed step by step. The result of this is that the relation with the
competences that are supported by this knowledge is lost. The knowledge is presented in an abstract
way, as if there is no relation with the final target of the training program.
In more technical programs, the knowledge is often taught independent from the practical
application, the program component where competence is developed. For a student, competence
and knowledge almost seem to be two elements that exist independent from each other. This way of
working has several consequences:
- The student has no “framework”, no context to put the knowledge in. The knowledge stays
abstract without concretisation in practice.
- Teachers do not see the relation with competence needs, but teach an independent
“knowledge unit”, which sometimes shifts towards teachers’ “fetishes” or elements that may
be interesting, but not useful in practice.
- The knowledge education is not based on a concrete need, which is often an obstacle for
students that are practice oriented and learn better when they understand the need and
have a context.
- Knowledge is often repeated in different courses (In theatre technical education almost
every course starts with the basic laws of electricity and physics). Moreover, the knowledge
is presented in different, sometimes conflicting, ways, leading to confusion with students.
Repetition and different presentation can be useful, but it should be a choice based on the
needs of the students, not on unintended coincidence.
- Competences often overlap, as they are secondary to the knowledge.
An education program based on a set of competences with underpinning knowledge would provide a
more clear structure. Teachers would be able to relate the knowledge to the competences they
support. In this way the knowledge becomes more concrete. Reorganising the underpinning
knowledge in logical units would improve the efficiency of the teaching process, as clear choices can
be made on where this knowledge is taught and where it is useful to have repetition.
§ Nina Tiptoe, the dancer, has gathered a lot of different competences during her dance
career. She has worked on a stage for years, is able to make an artistic interpretation, reads
music, developed a strong feeling for timing, worked with technical and artistic staff, speaks
different languages, … All these competences will help her in the transition to a new job.
§ Tony Solo, the freelancer, has a wide set of very specialised competences. He is able to work
in different work environments and labour traditions. He uses the competences he acquired
on tour also when he’s working in events or trade shows. His technical skills need permanent
updates because he needs to stay in the frontline of the technical evolution
§ Ben Young, uses the competences he acquired experimenting with the technical equipment
of the youth club and his inventive skills to shape his path through apprenticeship. Of course,
the apprenticeship also provides him with new skills. He discovered the need for more
theoretical background and subscribed in an online course about electronics.
18
§ Mary Standards background in science and technology is a good ground to build the skillset
for a multiskilled technician. Her climbing skills become useful now in a professional context.
She visits concerts and theatre performances to get more artistic background.
2.6
How and why does this feed into LAAR?
The LAAR project uses competences as a starting point to develop applications and measuring /
assessment systems. This guarantees a wide understanding and a connection with other European
systems and visions.
The basic idea behind the ETTE Safety Passport was that the focus of safety training and certification
needs to be on competence, rather than on knowledge. To work safe means “to be able to”, more
than “to know how to” do things. Even if knowledge and understanding is essential to work safe, it is
not enough. At the end we want practitioners to act and behave safe. This is best expressed in
competence.
19
3
ESCO
In order to exchange between (unemployment centres of) countries, the European commission
developed a “translating tool” based on a common competence language.
On their website ESCO describes itself as:
ESCO (European Skills, Competences, Qualifications and Occupations) is the European
multilingual classification of Skills, Competences, Qualifications and Occupations. It is a
European Commission project, run by Directorate General Employment, Social Affairs and
Inclusion (DG EMPL).
ESCO works as a dictionary, describing, identifying and classifying professional occupations,
skills, and qualifications relevant for the EU labour market and education and training.
ESCO provides descriptions of 2942 occupations and 13.485 skills linked to these occupations,
translated into 27 languages (all official EU languages plus Icelandic, Norwegian and Arabic).
Over time, it will also display the qualifications awarded in the education and training
systems from Member States, as well as qualifications issued by private awarding bodies.
The aim of ESCO is to support job mobility across Europe and therefore a more integrated and
efficient labour market, by offering a “common language” on occupations and skills that can
be used by different stakeholders on employment and education and training topics.
(https://ec.europa.eu/esco/portal/howtouse/21da6a9a-02d1-4533-8057-dea0a824a17a )
3.1
ESCO structure
The structure contains 3 pillars representing skills/competences, occupations and qualifications. The
3 pillars are related to each other.
(ESCO structure)
The system makes it possible to compare skills/competences, occupations and qualifications by
creating a “go between” that can be used by all countries to relate their national systems to.
20
A simple example shows the gain this system creates: the comparison between all countries of a
single qualification.
Before ESCO
With ESCO
• 28 countries make comparisons with 27
countries
• 756 comparisons to check 1 occupation
• 28 countries use 1 common (ESCO)
qualification
• 28 comparisons to check 1 occupation
(Overview or comparisons with and without ESCO)
ESCO is an open source database and can be used to develop different competence related tools and
applications.
§ Nina Tiptoe has worked in several different countries in Europe, she used ESCO to describe
all her work experiences in her Europass CV.
§ Tony Solo looks for jobs in different countries. The local unemployment office sends him
offers they receive on a regular base. They receive these offers form their colleagues through
the EURES-ESCO system.
§ The qualification that Ben Young and Mary Standard receive is listed in the ESCO
Qualifications database which makes it easier to get it recognised in other countries.
3.2
The ESCO theatre Technical competences
The team that wrote the theatre technical competences for ESCO based their work on an earlier
project named TTT-LPT. In the documentation of the project we find the core elements for describing
the competences:
To make skills useful in a European context, it is essential that the descriptions are language
independent, independent of culture and independent of technology. This because they
should be transferable to new local contexts. This puts a significant responsibility to the
reader / user. He / she must interpret the content and put it in the local context.
To translate existing job descriptions into a transparent, readable and comparable system,
the competences have to be small enough. They should be smaller than or equal to the
21
existing descriptions. Otherwise, an existing profile can never be described accurately. Within
this project we like to make the comparison with the "lowest common denominator" to
describe this.
The units should also be written in sufficient detail so that a user in a different context
understands what the precise scope is. The description must indicate not only the content of
the competence, but also the level at which it is governed.
Where possible we also sought the greatest possible universality. This creates a "reusability"
of competences. The competences can be transferred from one profile to another. Because
the competences are written as "lowest common denominator" the set designer and lighting
designer will both have the competence "design" in their profile. The context of technical skills
on light or set decide what kind of designer is meant.
Because a large number of competences are created by this method, it is necessary to place
them in a matrix or cluster to improve searching. Within this project we have chosen for a
combination of both. The competences are classified in fields and subfields, but they are also
placed on the timeline of a production.
Finally, we found it important that the descriptions were usable for different purposes. The
same competency units must be useful for making individual profiles, portfolios, job profiles,
training needs analysis, making of training programs, monitoring of learning progress, etc.
We are convinced that these conditions contribute to more transparency between the various
countries and organizations. Mobility is strongly facilitated in this way.
(Competence units for technical theatre, Mia Erlin, Tony Bond, Anders Larson, Chris Van
Goethem, Umberto Bellodi 2013, TTT-LPT Leonardo project)
This way of writing in smallest common denominators facilitates the writing of the competences in a
way that they are maximum reusable. The competences used in the occupations can be divided in
• level competences, these are competences that reoccur in different occupations on the same
(EQF) level) for example design competences that describe the designing aspect of a
designer, independent of what is designed.
• Field competences, competences that are related to a specific field and that reoccur on
different levels. For example, using a light board is done by a technician, an operator and a
designer.
• Unique competences, related to a specific competence, unique to a specific occupation.
(Different types of competences in a theatre technical profile)
Moreover, this way of writing and organising the competences reflects the sectoral reality where a
lot of occupations have “stepped profiles”, they are built on top of each other. For example, the
Lighting technician
Lighting
competences
Technicians
competences
Unique competences
22
lighting operator will have the competences of the lighting technician, combined with higher level
skills.
When the occupations are organised in a grid, the reusability of the different types of skills becomes
visible.
(Combining different types of competences in different levels and fields)
3.3
Where it went wrong, a critical note
The first version of ESCO was written by groups of experts that worked independent from each other
with only minimal guidelines. This made that the competences are not written in a consequent way
or size and that often multiple occurrences of the same or similar competence occurred.
Some experts from other sectors developed a different approach, which included more knowledge
elements. Often these knowledge elements were only a translation/adoption of existing
competence. For example, the knowledge element “health and safety in the workplace“ described as
“The body of rules, procedures and regulations related to safety, health and welfare of people in
their workplace.“ can be seen as a translation of or underpinning the competence “work with respect
for own safety”.
The system limited the possibilities to develop a clear content. For example, it was impossible to
create a logical order in the competences (they are displayed alphabetical). The division between
optional and essential skills was arbitrary, because a numeric relation between both types was
demanded. For example, “work with respect for own safety” is an optional skill for a high rigger, one
of the most dangerous occupations in the sector.
Based on the work of the expert groups, the results were adapted and translated by language experts
that had no expertise in the field and didn’t consult the experts. A sector specific language is hard to
translate and needs accurate wording to grab the exact span of the content.
The content of the ESCO taxonomy is not crosschecked with other European systems like the
Regulated professions database.
Moreover, in the process leading to the first public version, decisions were made that didn’t help the
clarity and transparency of the system. The scope notes and other additional information was lost
(left out). Some examples:
Lighting technician
Sound technician
Lighting operator
Sound operator
Level
Field
Unique
Sound designer
Lighting designer
23
The relation between the competences was made by non-experts, this results for example in
“prevent technical problems with lighting equipment” being a narrower skill of “prevent
technical problems with stage equipment” (mechanical lifting devices).
In the process of the language check “Change-over scenic elements during performance”
became “modify scenic elements during performance” (change-over means changing the
sets on stage by moving them on or off).
Alternative labels were given, without looking at the exact content or the description. For
example “use personal protection equipment” has an alternative label “oversee personal
protection equipment” (which is clearly a different level).
In some cases competences from other sectors were reused, that had a different meaning.
“working on heights” became for example “enforce safety procedures when working at
heights” (which is a different level and another content, moreover it reflects a different
vision on safety).
Sometimes competences were linked to other sector’s knowledge, which leads for example
to a knowledge unit “Know and understand the structures located besides railway tracks
known as signal boxes, interlocking towers, signal posts, and signal cabins, from which
signals, points, and other equipment are controlled.“ connected to “distribute control
signals” which is about entertainment lighting control.
The use of ISCO as a logical structure, instead of the structure that the sector developed,
leads to confusion and difficulties in finding matching similar occupations. For example, the
designers, all working in a theatre context, are found on different levels and in different
fields.
- Set designer – interior designers and decorators (3432)
- Lighting designer – other artistic and cultural professionals (3435)
- Costume designer – industrial designer – product and garment designers (2163)
- Video designer – graphic and media designer (2166)
- Sound designer – audio-visual technicians – broadcasting and audio-visual technicians
(3521)
All these elements have caused a certain reluctance by professionals to build further based on the
system. Some other projects decided to use the non-published last version before the changes. The
next version will be published in 2021 with input from experts of the field, so there is good hope that
the above issues will be solved by then.
3.4
Limitations of ESCO
Due to its original purpose, ESCO has limitations in terms for a direct usable system for education,
training, assessment or qualification. The system lacks detail to ensure required trust between
different partners or countries in the description of qualifications or learning content.
For example, the competence “Use personal protection equipment” is used for the “armed forces
officer” as for the “stage manager”. One can imagine that, even if the concept is applicable to both
occupations, the concrete learning content and the span of the competence are reasonably different.
The occupation descriptions are too open to be useful in practice.
24
3.5
ESCO and “General education”
ESCO focusses strongly on occupational competences. The occupations are written to be useful for
the labour market, but do not include “general education”. The European commission defined a set
of key competences for lifelong learning (last updated in March 2019) but these are not included in
the occupations.
Qualifications often include a set of general education competences, history, language,
understanding of society, physics, math, biology, geography, … are an important part of our
education. We do not only train people for a job, we also prepare them to active members of society.
Identity building, functioning in society, entrepreneurship, democratic values, cultural awareness, …
are key to this. These competences are essential for living together, being a human being, but not
specific for an occupational context.
This discrepancy between occupational profiles and qualification including “general education” can
cause a conflict in how we measure level and content of a qualification. If credits are time related
and level is dependent on level of thinking, not taking in account general education would lower level
of qualifications.
But there is more, every employer expects a certain amount of general education of his/her workers.
These competences are needed for building relationships with clients, to work in a team, to
understand context, etc. Leaving them out lowers artificially the level of an occupational profile.
3.6
ESCO as a backbone
On the other hand, ESCO comes as close as possible to a common language that spans all European
countries. It gives, for a well willing reader, a good idea of what the competences mean within a
specific context.
This common language can be used as a backbone, connecting more detailed sector specific
descriptions. The first ideas about a backbone were developed in the OPTiV (PWO) project (see
Building blocks for unique people, Chris Van Goethem, Expertise Centre for Technical Theatre RITS,
Erasmus University College Brussels) and its successor “Backbone”. The common descriptions of
competences are “enriched” with extra information about skills, underpinning knowledge, attitudes,
weight and level. The information is organised in a way that it can be used for occupational and
educational profiles, structured portfolio’s, self-evaluation, measuring training demand, etc.
3.7
ESCO in comparison with other transparency systems for VET
In the past several methods and taxonomies have been developed to ensure transparency between
different countries and different uses. We focus here on developments that are focussed on
vocational training and occupations.
The DISCO taxonomy provided a common language that could be used for CV’s and job applications.
These statements were more open and less defined than the ESCO competences and were focussed
on the specific needs in HR application. (http://disco-tools.eu) The taxonomy can be used in
combination the different Europass tools. (https://europass.cedefop.europa.eu/ )
EURES, the European Job Mobility Portal used a HR based taxonomy, but is connected to ESCO now.
(https://ec.europa.eu/eures/ )
O*Net (https://www.onetonline.org ) is a US based system that provides cross occupation
connections, but use very generalist sets of technological skills, general skills, knowledge abilities and
25
work activities. On the other hand it is more concrete describing tasks, tools used and detailed work
activities. This is an interesting starting point, but ignores the competence-based view.
The VQTS (Vocational Qualification Transfer System) (https://vocationalqualification.net ) project
developed a methodology to describe occupational profiles. The matrix concept they developed is
based on Competence Areas wherein the competences are described as a fluidum of two and six
successive steps of the competence development process, rather than defined level of competence.
This is an interesting way to describe learning paths towards qualifications, but the concept also
limits a more universal use of the competences.
• The competences are based on specific occupation, and strongly context related which limits
(automatic) transferability
• The competences include multiple measuring criteria that are not necessarily related to each
other, which makes it difficult to assess them with Yes of No, there is a lot of interpretation
needed from the assessor.
The successive ECVAET (http://www.ecvaet.eu ) projects have developed a competence matrix for
Event technology worker (EQF4) and the Meister (master craftsperson, EQF6). The project also
developed, based on the matrixes, a training program for technicians moving between the different
partner countries of the project. These training programs adapt the existing competence to the
different working context. Finally, the project developed a safety matrix, related to the both
matrixes, and describing in detail the competence area’s and the related objects of protection,
including the skill level expected.
3.8
How and why does this feed into LAAR?
The LAAR project embraces ESCO as a concretisation of the competence concept. By using ESCO as a
starting point to develop applications and measuring / assessment systems, we ensure a wide
understanding and a connection with other European systems and visions.
The ETTE competences are written in ESCO competences, that occur in almost every technical
theatre profile. The competences are:
• Working with respect for your own safety.
• Contributing to a safe and sustainable working environment.
• Working ergonomically.
• Use personal protective equipment.
• Fire prevention in a performance environment.
• Safe working at height.
• Work safely with mobile electrical systems under supervision.
• Work safely with tools.
• Safe working with chemicals.
• Fit up and rig stage equipment.
The LAAR project used a selection of these competences to develop test cases.
26
27
4
A sectoral layer
A competence model that not only can be used for exchange of employment opportunities, but also
for education and training clearly needs more (sector specific) detail than can be offered by the ESCO
taxonomy. This can be solved using the principles for detailed descriptions developed in the TTT-LPT
(https://www.podiumtechnieken.be/wp-content/uploads/2018/05/Inleiding-TTT-NL-Tot-v8_0.pdf )
and OPTiV project. (Building blocks for unique people, Competence profiles for theatre technicians,
Chris Van Goethem, Erasmus University College)
If we also want to use the same model for mutual recognition of certification, competence, and
qualification we need more than just descriptions of competences. This mutual recognition is
essential to allow mobility in a context where regional or national governments want to keep
authority over education and training. The TALQ project (Final Report, Umberto Bellodi, Chris Van
Goethem, 2017, KA3 TALQ - Transparency in Arts Levels and Qualifications) identified 3 requirements
for a workable model.
Transparency
To be able to accept each other’s certifications and qualifications, we need to understand each other,
we need a common agreed language, we need to grab the exact meaning and depth of each
competence. For this purpose, ESCO as an international agreed taxonomy, is the preferred system to
build on. The competences can be described in detail on a sectoral level, but keep their (one to one)
connection with the ESCO taxonomy. Additional to this, EQF, ECVET and ECTS can be used as a
reference to define the level and volume of the competences.
We also need a common sectoral profile, an agreed description of what an occupation or activity,
covered by a certificate or qualification, exactly covers. These sectoral profiles can be developed
based on the ESCO occupations, but making a more strict definition of the required competence set
and organising the competences in a logic readable way.
Trust
The value of a qualification or certification is about trust. We need be sure that the holder is able to
perform what is mentioned on the document. This can only be guaranteed by an objective,
independent and standardised measurement that meets the highest quality standards. A sectoral
layer should include the needed information needed to execute the measurement. Ideally this
information is given on the level of the competences in order to reuse them in different profiles.
Freedom of education and organisation
Recognition of common definitions of competence and occupations should not interfere in the way
education is organised. Every country, region, school, … should be free to organise their education as
they want. This is the best guarantee for an education that is embedded in society and adapted to
local needs, culture and traditions. Moreover, the primary stakeholders of education and training are
local so it makes sense the decision making and quality control is organised on that level.
In the same logic of stakeholder ship, every learner as the owner of a learning process, should be free
to choose the learning path of his/her choice. This can be formal school education, apprenticeship,
self-learning, or any form or combination that suits the learner.
To allow enough flexibility for education organisers and learners, the TALQ project proposed a
variable occupational profile with 70% fixed competences and 30% variable competences (out of a
28
predefined set). This 70/30 relation guarantees a common understanding of the content of an
occupation and leaves enough possibilities to adapt to local occupational differences.
On the other hand, the stakeholders of certification and qualification are per definition international
in a wold that promotes mobility. The value of a certificate or qualification can only be measured by
the value given by the (international) users of it. Therefore, the definition of the competences,
generic occupation descriptions, the used assessment methodology and the quality control needs to
be on an international level.
The measures to guarantee transparency and trust described above should therefore be agreed on
an international level in cooperation with international stakeholders. To ensure a high-quality
standard of the assessment, the ISO/IEC 17024 standard could be used as a guideline.
This results in a model that relates ESCO with the components of the sectoral layer.
(ESCO with the sectoral layer)
4.1
Structure and information of a sectoral layer competence
The minimum requirements of a sectoral layer competence to comply with the demands of the 3
principals stated above are the following:
• ESCO reference, referring to the ESCO system. This makes it possible to reference them back
to national systems. It also provides the opportunity to get more detail about the ESCO
competence like the reusability level, in which occupations it is used, alternative titles, etc.
• Competence title retrieved from the ESCO nomenclature.
• Description retrieved from the ESCO nomenclature. The description gives a short definition
to give the user better understanding of the competence.
• Context is a more free, sector adapted explication of where and in what context the
competence is used. (This field has been removed of the ESCO nomenclature, but is useful in
a sectoral context.)
• Scope note is used to draw borders where one can assume the title or definition is not clear
enough. It can include or exclude specific complexity levels or situations. (This field has been
removed of the ESCO nomenclature, but is useful in a sectoral context.) Examples: “Includes
safety shoes, hearing protection, gloves, hard hats, fall protection, etc.”, “Includes protection
29
against occupational diseases”, “Excludes climbing equipment”, “Excludes the actual fire
intervention and evacuation organisation.”
• Skills, knowledge and attitude, a set of descriptors that define in detail the competence.
(see further).
• Assessment strategy/methods describes in short what assessment methods are acceptable
for this competence and what elements should be measured in an assessment setting. This
will help coaches and trainers to prepare for a final assessment.
• A level and a volume description indicating the relation to EQF/SQF and to a credit system.
(see further)
Next to this essential information, extra information can be added to support the teaching and
training process. Some examples:
• Rubrics for self-evaluation and evaluation
• Links to further reading, References, National notes and Legislation
• Teaching material, tools, video’s, apps, etc. that can be used
• Training good practices
4.1.1
Skills, knowledge and attitude
It is sometimes hard to make a clear distinction between skills, knowledge and attitude. All can be
rewritten in one of the other forms. On top of this, sometimes they are all referred to in the
educational terminology as learning outcomes or performance criteria, but in other cases these
terms refer only to one of them. In a simplistic model, we could state from the teacher/trainers’
point of view:
• Skills is what we train
• Knowledge is what we teach
• Attitude is what we stimulate by mentoring/coaching
Or we could state from the learners’ point of view:
• Skills is what the learner is able to
• Knowledge is what the learner needs to know/understand to be able to
• Attitude is how the learner should behave to be able to
If we go more in depth, we can define them as follows:
Skill
The skills describe concrete and objective, observable, measurable actions that you would expect
from a worker that masters a competence. Where possible, the skills are written in observable
sentences. (In some countries, the term "performance criteria" is used for the skills.) Often, they are
written with “the learner is able to” or “the practitioner” in front of the sentence, but leaving such
obvious elements out improves readability. To get a better understanding of the level of mastering,
“bloom like” descriptors are used.
Examples:
• Works according safety training and instructions
• Protects oneself against hazards
• Signals risks to responsible
30
In exceptional cases, knowledge is described as a skill. This is the case when it is necessary to
measure the underpinning knowledge because it doesn’t show in the skills.
Examples:
• Understands the risks in a performance environment and the mechanisms behind them.
• Understands own position in the safety chain and acts accordingly.
Knowledge
The knowledge refers to logical blocks of underpinning knowledge needed to perform the
competence. The knowledge is limited to what is really necessary to master the skills and the whole
of the competence. The blocks are divided and sized in a way that they can easily be reused for other
competences. Often, they are written with “knowledge of” in front of the sentence, but leaving such
obvious elements out improves readability. The knowledge blocks can be described in detail in a
separate format.
Example: Accident theory
Attitude
The attitude section lists attitudes that will support the mastering of the competence. General
attitudes that can be expected from every professional are not included.
Example:
- Awareness of long-term impact on personal health
4.1.2
Credit systems, ECVET and ECTS
Credit systems measure the “value” of (part of) a qualification of profile. This value is related to the
effort it takes for an average learner to reach the qualification. Roughly we can distinguish two major
systems:
• ECTS (European Credit Transfer System) is used in Higher Education. It is time based,
a credit refers to the average time use for learning in a standard trajectory. A credit
is 1/60th of an average year.
• ECVET (European Credit system for Vocational Education and Training) is used in in
vocational education. The system is indirectly time based. It is recommended that
the ECVET points refer to (1/60 of) a year of learning in a standard learning path.
In training, apprenticeship or recognition of prior learning this time-based approach does not really
work, but even if time is not really related, it is a good and understandable reference to the effort
that is supposed to be done to acquire the qualification and so of the “value”.
Both systems are too large to measure the “value” of a single competence. But it would be possible
to use a partial of a credit to (for example a centi-credit) to express this “value”.
4.1.3
Levels and EQF
To define the level of a qualification, EQF (European Qualification Framework) is used. In the
performing arts, sometime also SQF (Sectoral Qualification Framework) is used, which is an
adaptation with descriptors that are more accurate for the arts, but do not change anything to the
general concept or level of EQF.
31
The EQF is meant for measuring a full qualification. In theory it does not make sense to define a level
for a single competence. On the other hand, defining a level helps to understand the span of a
competence. In this case it works rather as an indicator than an absolute figure. In the TALQ project
the choice is made to use the lowest level of occurrence in a qualification as the EQF reference of a
competence.
4.1.4
Example of a sectoral layer competence description
This example is the first of the ETTE competences, as developed by the ETTE project. The knowledge
elements refer to detailed descriptions of the knowledge required for the competence.
Work with respect for own safety
Description
Show awareness of risks related to activities in the performing arts and act
accordingly to ensure the own safety.
Context
Understand the risks and prevention measures for your own personal health and
safety on stage and apply the basic safety rules in your own practice, according to
training and instruction.
Scope Note
• Includes protection against occupational diseases
• Includes the theoretical background of risk prevention
Skill
• Understand the risks in a performance environment and the mechanisms behind
them
• Understand your own position in the safety chain and act accordingly
• Work according to safety training and instructions
• Protect oneself against hazards
• Signal risks to a responsible
List of Knowledge
• Accident theory
• Five steps to reduce risk
• Your rights and obligations
Attitude
• Safety awareness
• Awareness of your own behaviour
Level and volume
• occurs from EQF level 3 upwards
• 0.3 ECTS
Assessment
32
Measuring sentences
• I do not understand risks on stage or procedures.
• I don't feel completely competent to deal with risks on stage.
• I understand the risks and prevention measures and I apply the basic safety rules.
• I point my colleagues to riskful situations and promote the basic safetyrules.
Assessment strategy
• Can be assessed by observation in combination with other professional skills in a
real life environment
ESCO reference
• work with respect for own safety
• https://ec.europa.eu/esco/portal/skill?uri=http%3A%2F%2Fdata.europa.eu%2Fe
sco%2Fskill%2Fb73035a9-bf06-412c-9796-
db579f85995c&conceptLanguage=en&full=true
4.2
Structure and information of a sectoral layer occupation or profile
The minimum requirements of a sectoral layer occupation to comply with the demands of the 3
principals stated above are the following:
• ESCO reference, referring to the ESCO system. This makes it possible to reference them back
to national systems. It also provides the opportunity to get more detail about the ESCO
occupation like the alternative titles (labels), ISCO codes and hierarchy, regulation, etc.
• Occupation title retrieved from the ESCO nomenclature.
• Description retrieved from the ESCO nomenclature. The description gives a short definition
to give the user better understanding of the competence.
• List of obligatory competences, these link to the sectoral layer competences. They are
organised and clustered in a logical order. The whole set represents 70% of the occupation.
• List of optional competences, these link to the sectoral layer competences. They are
organised and clustered in a logical order. The set is used to choose 30% of the occupation’s
competences that are variable.
• A level and a volume description indicating the relation to EQF/SQF and to a credit system.
Profiles can be used to describe (parts of) occupations that used in different countries, but that are
not included in the ESCO database. This can be for example Health and Safety certificates or licences
to use specific equipment.
4.2.1
The logical order of competences
Even if the competences are written to be used independent, it helps the reader to use a specific
order in a profile. In the TTT-LPT project a logical order, combined with a numbering system was
developed.
The system was based on 3 elements: the field, an additional subfield and the place in the process.
Using this numbering system results in competences automatically clustered by field, with specialised
subfields at the end. The order of the competences is the order “of appearance”, in other words,
competences that occur first in a process will also be earlier in the list. This helps the reader to
understand the relation between the competences and so to understand better the whole
occupation. (unloading a truck appears before setting up the stuff unloaded).
33
4.2.2
Example of a sectoral layer profile
The example below is the occupational profile for an assessor in the event and performing arts field,
developed by ETTEC for the ETTE assessment process. In this case the profile has no level or volume
assigned to it because the profile is not used in a educational context.
Assessor Occupational Standard
Description:
An assessor measures the competences of a candidate against a given qualification profile,
according to a pre-defined standard or procedure and judges objectively if the candidate
reaches the performance criteria of the competences.
Scope note:
Apart from the specific skills needed to assess candidates, the assessor also needs a profound
understanding of
• The field, and the occupations
• The qualifications and standards for the targeted occupations
• The different standards, assessment methods and procedures of the targeted
occupations
Assessors need to have substantial demonstrable experience in the targeted occupations
they are assessing.
This is not further developed in this profile, as it depends on the type of field, education and
assessment.
Competences
Assessment
• administer exams (Set the dates and policies for the exam session and make sure all
accommodations and materials are provided)
• prepare examinations for vocational courses (Prepare examinations that test both
theoretical and practical understanding of the content and the procedures imparted
during a course or teaching programme. Develop exams that assess the most
important insights that trainees should have gained from participating in the course.)
• manage tests (Develop, administer and evaluate a specific set of tests relevant to your
activities or to the people who have to complete the tests)
• conduct examination processes for apprentices (Provide examinations to trainees and
students on their practical knowledge. Follow procedures to assure transparency and
accuracy in the assessment exercise.)
• instruct on safety measures (Provide instruction on the possible causes of accidents or
sources of danger and explain the protective measures that should be taken to
guarantee health and safety.)
34
• assess students (Evaluate the students' (academic) progress, achievements, course
knowledge and skills through assignments, tests, and examinations. Diagnose their
needs and track their progress, strengths, and weaknesses. Formulate a summative
statement of the goals the student achieved.)
• assist clients with special needs (Aid clients with special needs following relevant
guidelines and special standards. Recognise their needs and accurately respond to
them if needed.)
Communication and feedback
• show social competences (Ability to interact effectively with other people.)
• listen actively (Give attention to what other people say, patiently understand points
being made, asking questions as appropriate, and not interrupting at inappropriate
times; able to listen carefully the needs of customers, clients, passengers, service users
or others, and provide solutions accordingly.)
• use different communication channels (Use various types of communication channels
such as verbal, handwritten, digital and telephonic communication with the aim of
constructing and sharing information and ideas.)
• use questioning techniques (Formulate questions appropriate to the purpose, such as
eliciting accurate information or supporting the learning process.)
• interview people (Interview people in a range of different circumstances)
• assess others (Assess, estimate and understand the feelings or temperament of others;
showing empathy)
• give constructive feedback (Provide founded feedback through both criticism and
praise in a respectful, clear, and consistent manner. Highlight achievements as well as
mistakes and set up methods of formative assessment to evaluate work.)
Quality and ethics
• follow ethical code of conduct (Carry out workplace activities according to accepted
principles of right and wrong, including fairness, transparency and impartiality in work
practices and conduct towards other people.)
• apply quality standards (Follow procedures which prevent errors in creation and
delivery of a product, a solution or a service to customers.)
Professional development
• manage personal professional development (Take responsibility for lifelong learning
and continuous professional development. Engage in learning to support and update
professional competence. Identify priority areas for professional development based
on reflection about own practice and through contact with peers and stakeholders.)
• reflect on practice (Routinely evaluate own practice, critically evaluating and
monitoring the practice methods and outcomes in consistent, coherent and
appropriate ways, being aware of relevant methodologies and utilising feedback from
managers, supervisors, other professionals, and patients/clients, in order to adapt the
practice accordingly.)
• monitor developments in field of expertise (Keep up with new research, regulations,
and other significant changes, labour market related or otherwise, occurring within the
field of specialisation.)
• develop professional network (Reach out to and meet up with people in a professional
context. Find common ground and use your contacts for mutual benefit. Keep track of
35
the people in your personal professional network and stay up to date on their
activities.)
Administration
• maintain professional administration (File and organise professional administration
documents comprehensively, keep customer records, fill in forms or log books and
prepare documents about company-related matter.)
• keep personal administration (File and organise personal administration documents
comprehensively.)
Knowledge
• assessment processes (Various evaluation techniques, theories, and tools applicable in
the assessment of students, participants in a programme, and employees. Different
assessment strategies such as initial, formative, summative and self- assessment are
used for varying purposes.)
4.2.3
Occupational vs. educational profiles
Most countries make a distinction between occupational and educational profiles. Where
occupational profiles only state the competences needed to perform a profession, the educational
profiles include also more general competences that define the level of the education program.
These general competences are transferable to other profiles of the same level. One can for example
expect a certain level of abstract thinking from a bachelor level.
In technical theatre profiles the difference is minimal, as most “occupational” competences inherent
include a general level of thinking.
For other fields there are two options, the educational profile could be a level higher or the
educational competences are included in the 30% optional competences.
4.2.4
work focused profiles
In some cases, a specific task does not need an educational or occupational profile as a whole. But it
needs a form of certification based on a common description. Examples are certificates for fork lift
drivers or to work in a specific environment.
The ETTE basic safety competences are a other example of such a profile. It only contains 10
competences that are essential for safe working on stage and ensures mutual recognition between
countries. The result is that every employer can be sure a worker can work safe on his stage.
§ Nina Tiptoe, the dancer uses the sectoral layer to understand better what the exact meaning
of the ESCO competences is in het (future) work context. This helps her to clarify which of
her dance competences are transferable and which need some adaptation.
§ Tony Solo, the freelancer, uses the sectoral layer to understand better what exactly is
expected in different countries. He also uses the sectoral layer to show to employers in
different countries what his exact skills are and where/if they differ from the local ones.
§ Ben Young, the apprentice, uses the sectoral layer in his evaluation talks with his mentor. The
descriptions help both to define what is gained and what need some work. The ETTE
competences allow him to work is high risk areas.
§ Mary Standard, school program is based on the sectoral layer, in this way it is easy to discuss
international exchange, because everyone has a good understanding of the exact content
and variations of her program.
36
4.3
How and why does this feed into LAAR?
The sectoral layer helps within the LAAR project to grasp the exact meaning of a competence and
what needs to be trained or measured. The detailed description ensures that the competence is
understood in all countries in the same way and that different developers use the same level and
context to create training and measuring methods.
The description of the ESCO competences in ETTE follow the structure of the sectoral layer.
Therefore, developers can get access to the extensive database that provides enough details about
the context and the content and to develop applications. The assessment guidelines strengthen the
understanding of the exact level that is expected.
37
5
Tools and applications based on a sectoral layer
Once a sectoral layer is developed with all the extra information and it is agreed on by the sectoral
stakeholders, it can be used as a common language bridging different types of applications. These
applications can be used for training, assessment, certification and qualification, but also for job
application and matching, HR and Life Long Learning.
Because all applications can use the same language it becomes easier to transfer results from one
application to another, from one organisation to another, and from one country/language to
another. This improved transparency supports the learner/worker during his professional path.
Below we describe some examples of good practices, applications and tools that can be integrated in
the concept of a sectoral layer.
5.1
Developing occupational and educational profiles
Aside from the profiles developed in ESCO, extra occupational or educational profiles can be
developed to serve specific local needs. Because these local profiles reuse competence descriptions
from the Sectoral Layer, they are easily connectable to the existing profiles. Moreover, the presence
of agreed assessment methods and detailed information about the content will facilitate the
acceptance of the profiles in an international context.
5.2
Developing curricula and courses
Using a Sectoral Layer for the development of curricula and course increases the efficiency of the
course setup. There are several elements that improve the efficiency:
• Often competences share the same knowledge by using the Sectoral Layer (on the level of
skills and knowledge), such overlaps are detected. Based on this, clear decisions can be made
on who teaches the knowledge and if it is needed to double the knowledge in several
courses.
• Teachers know from each other what knowledge is taught.
• Teachers know what skills are served by the knowledge and can focus on relevant issues
related to these skills.
• Knowledge elements can be clustered and planned in relation with the practical training.
• Teaching and training material can be reused easily (in case of stepped profiles even over
different levels).
5.3
Training needs analysis
The Sectoral Layer can be used as a starting point to make an inventory or the needed vs. the
available competences in an organisation. This inventory can be used to develop training for staff.
The smallest common denominator structure makes that short, very focussed training can be
provided. The training needs analysis can be combined with a permanent competence follow us
system for the employees.
5.4
Follow up of apprentices
The Flemish Cue-One-go (https://www.podiumtechnieken.be/en/competentions/cue-one-go/ )
project uses a set of competences, similar to ESCO, to follow up apprentices. In a first phase, the
company uses the competence set to describe what they can offer, and the apprentice describes
what competences he/she already har acquired. Based on the combination of this information, a
learning path is set out. From this learning path, a selection of competences is made for the first
period of learning.
38
At the end of the first learning period, apprentice and learning coach both evaluate the work done
with a set of rubrics, connected to the competence set. In the following evaluation talk, they can
focus on points that they interpret differently, which improves the efficiency of the evaluation. At the
end of the evaluation new goals are set for the next learning period.
As snapshots are kept of the evaluations, the learner and the coach have a good view on progress
made.
5.5
A structured portfolio
A traditional portfolio is a collection of evidence pieces of previous work that can be used to
accompany a CV or job application. In reality it is hard for a reader to relate these documents to a
specific job or occupational profile.
The TeBeVat (https://tebevat.eu/ ) project built further on the portfolio developed by TTT-LPT. This
type of portfolio is structured based on the competences of the Sectoral Layer. Each evidence piece
is linked to one or more competences and the relation is commented. The user is now able to
document his/her career in a structured way, connected to the sectoral Layer and ESCO.
The portfolio also implements other tools, developed by the EU.
- It gathers information that can be used by the European Curriculum vitae
- It implements the European Language Passport (a self-assessment tool for language skills and
qualifications.)
- It implements the European Digital Competence Grid
- The Europass Mobility, the Certificate Supplement and the Diploma Supplement can be
translated into ESCO competences and used as evidence pieces.
- Use of other EU tools
The fact that all evidence is related to competence also means one does not have to redo all the
work in case of a recognition of prior learning procedure. It is enough to verify if all the competences
are covered by convincing evidence.
Further, the portfolio includes some sector specific elements that are not common in other sectors:
- A job list, specific for freelance workers, to collect information of their work
- A technology list, specifying what equipment one is able to operate.
The portfolio does not integrate tools like DISCO ( http://disco-tools.eu ) or the regulated professions
5.6
Personal Development plan
Based on the Cue One Go methodology, a Personal Development Plan can be set up in which the
professional can set goals, described as a set of competences and keep overview on his/her progress
in learning by gathering evidence for these personal goals. Based on rubrics the professional can
easily follow up his/her progress.
5.7
Combining tools
One of the main advantages of this way of structuring tools is that existing content and parts of tools
can be reused. The fact that the same descriptions are reused in all labour related applications also
improves the acceptance and habituation for the staff.
39
§ Nina Tiptoe, the dancer has made a structured portfolio of her previous career. She
compared the competences she acquired with different occupational profiles and developed
based on the differences a Personal Development plan.
§ Tony Solo, the freelancer uses the portfolio to keep track of his work and related
competences. He uses this portfolio to present himself to different employers. These
employers get the basic information in their own language due the link to ESCO.
§ Ben Young, the apprentice has used his prior learning, recorded in his portfolio, at the start
of his apprenticeship. In the portfolio, he also notes his voluntary work at the youth club. He
uses the follow up tool to discuss his progress with his mentor. Based in this talks they set
new targets every month.
§ Mary Standard, the student studies a program developed based on the sectoral layer. At the
end of her studies, she will transfer all the acquired competences in het portfolio. She
already has done a first work experience and could add some competences already, together
with her climbing skills that will be useful for rigging.
5.8
How and why does this feed into LAAR?
The LAAR project uses the same methodology to develop the applications. More specifically, all
applications will provide information in a format that fits into the portfolio.
The size of the ETTE competences is based on the “smallest common denominator” concept.
Therefore, they can easily be integrated in different curricula. Based on the ETTE profile, (traditional)
course material is developed for learners and teachers. This material is modular, so it can be used in
different training strategies. The integrated rubrics can be used for training needs analysis and follow
up of apprentices. The competences can be used as a backbone for a structured portfolio and a
personal development plan.
40
41
6
The learning wallet
The different tools above use the same taxonomy, derived from ESCO and the Sectoral Layer, but
work still independent of each other. The information about the achievements of the learner are
kept in different systems and formats. A learner has no common place to safe/store all his/her
achievements. This becomes an issue when a student leaves school, an apprentice changes
organisation or even more when a Lifelong Learner learns in different contexts. The learner loses
information about his learning that could be reused in other contexts.
A similar issue occurs using different digitalised tools to support learning. They all use and keep
information about the learning activity, retrieved from the tools, but there is no way to combine or
exchange information between the tools.
Moreover, the ISO/IEC 17024 quality standard for bodies operating certification of persons (and
other similar standards), states that the information about the leaning and certification is owned by
the learner. He/she is entitled to decide who has access, what the information is used for, etc.
Where the Sectoral Layer provides us with static information and the different tools and applications
provide us with information about what a learner has achieved, we need a centralised place, owned
by the learner, to store this information and connect it with the taxonomy of the Sectoral Layer. A
Learning wallet, a Learning Record Store or a Learning Locker is such a place. It is defined as “a type
of data repository designed to store learning activity statements”
(http://v1docs.learninglocker.net/welcome/) .
To be able to exchange information between the different tools and applications and the learning
wallet, we need a standardised language that is understood by all entities in the system. Such a
language is e.g., specified by the Experience API (or xAPI).
Experts define the Experience API as: “a new specification for learning technology that makes it
possible to collect data about the wide range of experiences a person has (online and offline). This
API captures data in a consistent format about a person or group’s activities from many technologies.
Very different systems are able to securely communicate by capturing and sharing this stream of
activities using xAPI’s simple vocabulary.” (https://xapi.com/overview/ )
The information model looks like this:
(Overview of the information model)
42
The principle of the xAPI language is described as follows:
The xAPI provides a normed set of services for storing and retrieving statements about learner
activity. These packets are logged in the form <Actor, Verb, Object>. An example is “John found
the exit”. Further fields can be added to these statements, like ‘result’, ‘context’, ‘authority’ and
‘attachments’.
(Learning Analytics in Augmented Reality, Blueprint for an AR / xAPI Framework)
The place where the information about the learning activities is stored (or retrieved from) is the
learning wallet, or more technically, the Learning Record Store. This can be defined as follows:
The Learning Record Store (LRS) acts as the streaming database for xAPI statements. In our case,
we deployed the Open Source project LearningLocker [12].
(Learning Analytics in Augmented Reality, Blueprint for an AR / xAPI Framework)
6.1
The learning wallet and the Sectoral Layer of ESCO
Every learning activity has a link to a (part of) a competence. After all, the learning activity is set up to
develop this competence. This relation should be reflected in the vocabulary of the learning wallet.
As every tool and application already has a relation with the competences described in the Sectoral
layer, the most logical way to include the competence in the learning statements is to refer to the
Sectoral Layer. This makes it possible for every tool and application to retrieve information
originating from other tools from the Learning Wallet.
Most tools and applications will only cover a part of a competence, mostly one or more skills. If the
tools or applications are also measuring performance, the “observations” need to be described. In
this way, we have exact information that can be reused for further analysis.
When the “observations” are done by human interaction, it is preferable that they are described in
Objective Observable Statements. In other words, actions that can be observed in an objective way.
For example, “uses gloves” is an objective observable action, there is no interpretation needed.
While “uses Personal Protection Equipment” is less or not observable, the observer needs to
interpret the situation and decide by him/herself what Personal Protection Equipment is needed in
this situation. The reason to avoid interpretation or decision-making is that it influences the bias of
the observer. Ideally the interpretation is only made at the end when all activities are finished. Such a
tool or application should deliver information about Skills and eventual also Objective Observable
Statements to the learning wallet.
6.2
Lifelong Learning and the Learning Wallet
Once we have a central repository in place, it can be used by different actors, and for different uses.
The repository is independent from the application that provides or retrieves the information. It is
also independent from the training provider or the type of learning chosen by the learner. The
information is owned by the learner, be it that some restriction concerning assessment and
certification have to build in (see further in future use).
In the first place the Learning Wallet is used to gather information about the learner. This
information can be sent by applications that generate statements, but also applications that translate
more conventional information form about off-line activities, training, etc. into objective statements.
Secondly, the information in the Learning Wallet can also be used to monitor the performance of the
learner. Learner and trainer who know about the actual state of the students’ achievements, they
43
can evaluate the students’ performance based on objective recorded information. Since the
information is connected to the Sectoral Layer of ESCO, it can be presented in a logical format that is
understandable by the end user.
Finally, the information can be retrieved by applications to steer learning. Based on previous
achievements, the application can decide to repeat a learning experience, give another experience
for same purpose or skill, present more background or theory, or raise the level. This can be done
within a single exercise or on the level of a sequence of exercises.
Trainers or supervisors can follow up the performance of a group of learners. In this way, the learning
process can be evaluated and adjusted additionally.
6.3
Future use
In the frame of lifelong learning more functionalities could be added. The Wallet also could hold
traditional evidence sent by the learner in the form of a portfolio. Or it could include self-assessment.
All these elements can be linked to the sectoral layer.
The Learning Wallet now contains all the information needed to assess prior learning. This
assessment is based on interpretation by a human being (ideally at least two) and can’t be replaced
by automated response. The learner can give access to an assessor to verify the information, assess
the level and “sign of” competences. Here a conflict raises with the ownership of the information of
the learner. The learner still owns the information, but can’t be allowed to change it for reasons of
integrity of the assessment. Once the competences are assessed, we have to be sure the
underpinning information isn’t changed.
A learner (or professional) can also use this information to build and send out a CV to an employer.
Therefore, a learner can select what information is send to an individual employer and restrict access
in time. In principle, the information could also be used in a first step of selecting appropriate
candidates in a selection procedure.
6.4
Badges and virtual certificates
The assessment of the competences of a learner could lead to certification. Based on the idea of a
more competence (and less qualification) focused labour marker this could take the shape of badges,
connected to the sectoral layer. Ideally it would be connected to an accepted standard like Open
Badges. (https://openbadges.org/ )
For Certificates and qualifications, the security and integrity demands are even more stringent, thus a
block-chain based system could be connected (e.g., https://www.bcdiploma.com/ ).
§ Nina Tiptoe, the dancer recorded all her prior learning in the Learning Wallet. Based on this,
she discovered the missing links that needed an update. She trained and learned by different
providers, that all recorded the information in the Wallet. At the end of the year, she will
look for an assessor that will verify all information and she hopes to get a qualification for
stage management.
§ Tony Solo, the freelancer trains mostly on line and through applications, as he is always on
the road. His training sessions are recorded in the wallet. Depending on where he is, he also
follows product training with different providers that are recorded. He uses the badges,
portfolio and CV to prove his capabilities in different countries.
§ Ben Young, the apprentice, gathers badges bit by bit, in his workplace, but also in the youth
club. He feels good that he can show what he is already acquired.
44
§ Mary Standard’s school uses the learning wallet to record all her activities and achievements.
This makes that she, but also all her trainers have a good overview of her strengths and
weaknesses. The learning system sends her on a regular base extra exercises on het phone to
keep her level up and to work on things that need improvement.
6.5
How and why does this feed into LAAR?
The LAAR project uses the ETTE competence descriptions as identifiers for different xAPI statements
in the learning wallet. A series of application specific xAPI statements were developed for the LAAR
project. By doing so, the learning statements can be mapped and, if the competence is proved,
reused in other certificates. This makes it possible, for the long run, to create an open badges system
or even a virtual certificate.
45
7
Teaching and training
The target group of young initial learners, professional lifelong learners and transition learners as
described in chapter 1 asks for a specific approach. Their hands-on, active learning style asks for a
learning path starting from real live practice and adding theory and knowledge on a “need to know”
basis. They need a framework to relate and connect the acquired knowledge and skills that is
adapted to the unique “renaissance” approach of learning that is typical for the sector.
Their disappointment in and sometimes aversion for traditional education makes that their intrinsic
motivation needs to be addressed to make them perform towards optimum results. Important
elements for this are the relation to the actual needs and the immediate usability of the things
learned.
Without going in depth into learning theory, there are some points that are essential to develop
teaching and training for this target group:
7.1
Different types of learning content
Even if the target group prefers to start from practice, there is a need for a different approach to
support different types of learning contents. It is not evident to train each content type starting in
practice, but it is important to link each type to the practice. We can roughly define five groups of
content:
• Underpinning factual knowledge, needed to recognise tools, equipment, safety signs, etc.
This can partly be trained, but will also develop further by presence of the learner in a real
live environment.
• Theoretical background, needed to understand and apply electrical principles, make
mechanical calculations, etc. This may need a more traditional teaching approach, but the
link to the sectorial reality has to be guarded.
• Procedural problem-solving strategies, to be able to find errors in systems, but also to
comply with the demands of artists for example. These can be trained and developed by
confronting the learner with different types of situations and problems.
• Practical procedural knowledge (skills), to be able to perform routine tasks in a changing
environment. This type of knowledge can be trained in a step by step approach towards full
mastering of the skill.
• Artistic/creative understanding, to be able to work in an artistic environment. This includes
the “language” of the artists, a background in the arts, a critical sense and developing a
personal “taste”. This type of knowledge is not objectively measurable and is less important
on lower levels as communication about artistic results is limited.
7.2
Different pathways to competence
The modern technical theatre field is - educationally spoken - a very young field. Formal learning
paths only exist since the 90’s and are not even developed in all European countries. Traditionally,
the path to a competence was defined by “trial and error“, informal peer learning and manuals,
instruction sheets and documentation. Later, this concept was transformed in a more formal way of
apprenticeship, supported by (some) formal education and certification. The development of formal
training, based on traditional education is a rather recent phenomenon.
In most countries, a mix of traditional learning and a strong practical component, often in the form of
apprenticeship, is used to deliver competent practitioners. In fact, this kind of blended learning
“avant la lettre”. It uses methods out of the best of both worlds to train starting practitioners. This
46
training is later on followed by continuous training, partly in the form of product training, delivered
by the manufacturers.
7.3
Complexity of a real-life working environment for training
The real live working environment is a complex combination of factors that surround the actual
activity to be trained. Specifically, in the technical theatre sector, external influences to the task or
competence that has to be trained are significant. This often results in the need of a higher level of
competence than would be expected for a similar task in other sectors.
The activities to be trained are supporting to the arts and making art is, per definition, making new
things. This means procedures, standards, safety regulations, … need to be evaluated, questioned
and adapted where needed to the demands of the artistic result. So, the activity does not occur in a
standardised environment, nor will the activity itself be standardised completely.
The environment (the stage) where the training has to take place is permanently changing. In
contradiction to other work places, where every activity has a designated place that is adapted to the
needs of this activity, the stage as a working post is permanently changing. One moment it is dark,
another moment hanging loads are above the learner, the next moment sound levels are high, …
Secondly the work is a cooperative action, every worker depends on the result of the other. On top
of this, some of the activities to be trained only occur once in the production cycle. Where, for
example, in a car workshop the same activity is performed every day it is not unusual that a specific
task on stage is performed only once every three months on a stage, with several consequences:
• A learner needs to be conscious of the environment that influences his/her work.
• The safety situation is constantly changing, this requires a permanent attention.
• There is little room for failure, as this would compromise the whole organization.
• The training possibilities depend on strict planning and deadlines.
• The priority is often put on the direct deadlines that have to be met and less to the long-term
goals like training.
In this type of situation, the unexperienced learner is subject to and forms a risk at the same time.
Therefore, a training program should start within a “safe environment” where one is allowed to fail
and gets the time to build a certain routine. After this, the learning path can introduce more
complexity until the learner is ready to enter the complex environment.
7.4
The importance of repetition
To be competent, it is not enough to be able to perform a specific set of tasks, one needs to be able
to do this within acceptable time limits and in different contexts. The activity has to become part of
unconscious behaviour, which means a certain level of automation of actions.
Often the path to mastering a competence is based on building complexity step by step. In a first set
the learner will perform the different steps several times under supervision and in a controlled and
safe environment. In a next step, he/she will repeat the whole of the activity. This is done till a
certain routine has been developed. In further steps, the complexity is augmented by adding external
factors or additional activities. The “safe zone” is build off towards the level of a real live activity.
Finally, when the competence is mastered, it becomes part of a larger activity and
performed/repeated continuously. It becomes unconscious behaviour. Repetition is essential to this
process.
47
§ Nina Tiptoe, the dancer, has a lot of general underpinning knowledge and is used to work in
the complex environment. Her focus will be on more technical knowledge, procedural and
problem-solving skills. She agreed with the crew to do some informal apprenticeship while
she is still working.
§ Tony Solo, the freelancer, often uses the “waste time” when waiting for sound checks, to
train new skills and lightboards. He already has a deep understanding of the underpinning
knowledge, but needs repetition to get these skills to the level he can use them in a
performance.
§ Ben Young, the apprentice is trained by his mentor. He will first make him train with
equipment when the stage is free and outside of a production process. When questions
occur, the mentor will lead him to the underpinning knowledge. Once he reaches a certain
level of mastering a competence, this will be deployed in rehearsals and later in
performances. He is now used to look up the underpinning knowledge, a skill he uses also in
the Youth Club.
§ Mary Standard’s school program starts from a theoretical background. Once the knowledge
is processed, they do simple practical exercises in a studio environment. Bit by bit the
complexity is increased until they are ready to run a performance.
7.5
How and why does this feed into LAAR?
The LAAR project takes this needs and specificities in account while developing applications. The
developed applications focus on the first steps of the learning process, creating a safe environment
where the learner can experiment without external influence, but in a real-life environment, and
supporting collaboration. The larger concept of the learning analytics combined with the ESCO
competence framework and the sectoral layer are used for the whole of the learning process.
The ETTE learning content is structured based on competence. The underpinning knowledge
supports one or more competences to maximise efficiency. The competence elements focus on the
“how to” aspect, the procedural knowledge and problem-solving strategies. The underpinning
knowledge delivers factual knowledge and a theoretical background. This supports the LAAR
approach.
48
49
8
Measurement, evaluation and assessment in vocational education
A lot of the applications above depend largely on measurement for evaluation and assessment. We
have to make a clear distinction between these different types of measurement, their use and the
consequences of this differences.
Measurement for evaluation is process measurement, in other words we measure an intermediate
state of competence of a learner in order to be able to steer, support, evaluate, … the learner and
the learning process he/she passes through. This measurement is relative to the expected level, and
can even go beyond the expected level. A learner that performs well will surpass the expected level.
This is a good thing, because otherwise we would train for average performance and would fail to
appreciate the possibilities of a learner. Often the grading is also used to reflect progress. A student
that started with a low level will be scored higher than one that already had a relatively high level.
This motivates and rewards learning efforts.
The “client” of this type of measurement is the learner. It helps him/her to understand how they
have performed. This also means the quality of the measurement has to relate to the learners (and
secondary to the process owner’s) expectations.
A typical example of process measurement is grading in a traditional school system. A student is
graded on a scale from 0 to 10, where a grading of 5 means that the student has enough knowledge
or competences to fulfil the expectations.
Measurement for assessment is result measurement. In other words, we measure if a learner
complies to a set standard. This can be a curriculum or an occupational standard. The measurement
is absolute to the expected level. The assessor decides if the learner has the skills yes or no. This
demands a higher level of objectivity.
The “client” of this type of measurement is the society as a whole. Society, in the form of employers
or users of a service, has to be sure the standard is reached. The result of the assessment also
influences the result of other individuals with the same certificate or qualification. The public
perception of the qualification is reflected by the level of the lowest performer that passed.
A typical example of this type of measurement is the test for a certificate or a drivers’ licence. The
assessor will state if the candidate is capable to perform the tasks described in the certificate or is
able to drive a car.
There are some issues when we mix these different types of measurement. Within our traditional
education, the occupational standard is often used as a base for the development of a curriculum.
Progress, performance, but also result is measured based on grading. The result of the school career
is a qualification referring to the standard. The society, the “outside world” expects the graduated
student to be able to perform all competences which are stated in the qualification. But in reality, a
student that reaches 50% of the goals mentioned in the qualification will pass.
A secondary issue is that the assessment is done often by individuals that have a working or authority
relation with the assessed person. This lowers the objectivity level of the measurement. A teacher for
example that grades a student will take in account the learning process and will at the same time
assess his personal achievement in bringing the student to the required level. These elements
influence the bias of the teacher as assessor.
50
8.1
Developing evaluation measurement
Evaluation measurement can take different forms and shapes. In a traditional, knowledge-based
education, this will often take the form of a paper test or a written assignment. For our target group,
this type of testing often has some disadvantages. One of the most important ones is the fact that
one needs written expression skills to be able to perform the paper tests. The lack of these skills will
bias what we really want to measure. On the other hand, a student that is very good in expressing
him/her in writing will probably have an advantage, especially in a group of low skilled writers.
When we want to evaluate concrete skills, the best way seems to be to make a learner perform the
skill and to observe is the learner is capable. To support objectivity in an observation, we can
translate the skill into concrete and objective observable criteria or actions. In this way we can be
sure that the same things are measured for each learner.
An example of these observable criteria or actions for the skill “Inspect the technical performance
equipment visually for damage“ of the competence “Fit up and rig performance equipment“ in an
exercise where the learner has to connect a truss are:
• Checked for damage on outside
• Checked damage in holes
• Checked pivot for damage
• Checked if pivot fits
• Checked if split pen closes properly
These objective criteria can also be used as part of an xAPI statement. In this case the observer can
mark if the actions are observed and this is sent to the xAPI, in combination with information about
the competence and the skill.
Concrete situations need to be created that incorporate the expected level of complexity. This
complexity can be raised bit by bit during the training process.
For other competences, the result of the actions can be verified. We do not look at the process, but
only to what was expected as result. For example, a learner is asked to build a lighting rig, and we will
verify if the rig is built according to the set standards. These results can be incorporated also in xAPI
statements, but we can’t be sure if the process of building has been done properly.
Evaluation can also be done with a system of rubrics. Rubrics are statements that describe the level a
learner has reached towards a specific competence. They can be used by the trainer, but also by the
learner in the form of a self-assessment. Often rubrics are generalised statements. The problem with
this generalisation is that it almost always includes an appreciation, which is conceived negative by
the learner. An example of generalised statements is:
• I have never done this
• I am still learning how to do this
• I am able to do this
• I am more than able to do this.
In the ETTE project the rubrics are based on a four-level scale in which the third level is what is
expected. Learners reaching the fourth level are appreciated in this way to perform better than
expected. The lowest level is the starting level, this is what you expect from a person without
training. The second level reflects a learner that is on his/her way, but didn’t reach the expected level
yet.
51
The ETTE rubrics are written concrete and specific for a competence. Describing the concrete
situation seems to ensure a “more honest” answer from both learner and trainer. An example of the
ETTE rubrics is:
• I do not understand risks on stage or procedures.
• I don't feel completely competent to deal with risks on stage.
• I understand the risks and prevention measures and I apply the basic safety rules.
• I point my colleagues to risk-full situations and promote the basic safety rules
The highest level reflects here “more than expected”. The fact that one is not only working on ones’
own safety, but also tries to ameliorate the safety of ones’ colleagues is a benefit, but not needed for
this competence.
The final result of a process evaluation is always focussed on the next step. We steer the training
process based on where the learner is in the process, on what he/she has achieved. The fact that we
give insight to the learner about goals reached or not reached is supporting and motivating the
learner.
8.2
Developing assessment
When developing an assessment situation, we need to ensure that all elements required for an
independent and objective procedure are met. These elements can be derived from the ISO/IEC
17024 standard.
The type of assessment will depend on what we are measuring.
• An observation in a simulated environment is appropriate for concrete skills. A real-life
situation is simulated, including all elements that reflect the complexity. Of course, not all
elements of a skill or competence can be measured in all their occurrences. During the
development of the assessment, one needs to pick specific occurrences that reflect the
whole of the competence. The observation is objectivised by using observable actions.
• An observation in a real-life environment is more difficult to objectivise. The environment
complexity depends on actions of other practitioners and other elements that are not under
control. But in some cases, where influences like audience behaviour are important, this can
be used.
• Role play can be used for skills where human interaction and communication is essential. We
have to be sure that the assessed individual understands the concept of a role play.
• Post-box assignment is an assignment that is also in real live done by the individual alone,
without any influence from the outside. One can think of developing a planning or a light
plot. The assessed person gets all the information needed and at the end the result is
checked against clear criteria.
• In the case of recognition of prior learning, based on portfolio the assessor will not observe
the actions, but will assesses the evidence that a candidate has brought together about his
prior learning. The evidence will be matched with the competences to be proved. The
difficulty here is that one has to ensure that the evidence can be traced back to the
candidates’ performance.
• Traditional tests or multiple-choice test can be used if the underpinning knowledge cannot
be successfully assessed based on the testing of the skills. But even in this type of testing, a
focus on the competence rather than a pure reproduction of the acquired knowledge is
recommended.
52
In all the different types of assessment we need to try to make the assessment as objective as
possible. This can be done in the development of the assessment itself, using exact defined criteria,
limiting bias, not taking in account skills that are not part of the assessment, … But it can also be
done in the organisation of the assessment by smoothing the influence of an assessor or by limiting
the possibilities for a “lucky shot” for example.
The ETTE assessment procedure for example foresees two assessors observing independently from
different positions (two times two eyes principle). Every skill is at least measured twice in different
situations to avoid “lucky shots”.
One of the specific problems in the event and performing arts sector is that some, more artistic, skills
can’t be objectivised fully. In this case the principle of intersubjectivity is used. A group of, by
definition subjective, assessors will discuss and try to reach a consensus.
§ Nina Tiptoe, the dancer, asked an assessor to look into her portfolio. Based on the evidence
of her prior career a set of competences was assessed. In her lifelong learning process het
mentor evaluates her achievements on a regular base to steer her learning.
§ Tony Solo, the freelancer, gathers evidence about his career. When he gets product training,
the achievements are assessed and added to his portfolio.
§ Ben Young, the apprentice, is evaluated on a permanent base. At the end of his learning
period, he will take an assessment by observation in a simulated environment.
§ Mary Standard’s school combines traditional tests with elements of different assessment
methods.
8.3
How and why does this feed into LAAR?
The LAAR project incorporated the principles for good process and result measurement. The ETTE
assessment procedure is used as a good practice.
53
9
Generic types of applications for digital learning environments
In this chapter we give, without being complete, a short overview of existing applications that are
used in a digital learning environment. And we introduce specific learning environments itself.
Central in the choice of the applications and the way they are used is the motivation based on the
type of learner. In other words, we need to find the application which fits the purpose based on
learner type, situation, …
9.1
Digital paper books
Often, digital learning platforms are filled with “digital paper books”. In fact, this is the concept of a
traditional book in a digital version, without extra functionality that motivates or challenges the
learner. It is just an easy way to distribute information, the learner still has to find a way in the whole
of the book but the advantage is the learner has access at any moment in any place.
This “digital paper book” is often the first step in the transfer from analogue to digital. In a next step,
the texts often become more website style text. The text is split in smaller parts, fitting a single
screen and are reinforced with links, visual content, etc.
9.2
AV-Media based content
The use of (audio-visual) media strengthens the understanding of the content, especially when both
visual and auditive stimuli are used. We use “multiple inputs to the brain” which leads to a longer
retention of knowledge and a deeper understanding.
The range of audio-visual tools varies from recorded lectures, over instruction video’s, towards
animations and purpose made documentaries.
(Screenshot Youtube Rigging a pull through for access from the ground, Access Techniques Ltd )
54
(Screenshot, Youtube, Basic Event Rigging Techniques and Equipment, SXS Event Production Services)
(screenshot www.napofilm.net )
55
9.3
Interaction tools
The above content delivering tools can be reinforced with secondary tools that add interactivity. The
content is still passive, but at the end of the “consumption” of the information, the learner is
challenged or tested. Question banks with different types of multiple choice, matching, … and
questions giving automated feedback help the learner to check if the content has been understood.
(Canvas questionnaire)
Other interactivity tools include discussion fora that can be used in groups of students or
professionals, peer evaluation tools, etc.
9.3.1
Simulation tools
Simulation tools are built to give better insight in physical principles or phenomenon. The advantage
is that they need an active involvement of the learner that can give one or more inputs and gets a
simulated visual result.
These simulation tools are built for one specific principle. Below some examples for electricity,
lighting (colour gel filtering) and stage mechanics.
56
(Pictures from www.physicsclassroom.com )
(Pictures from www.physicsclassroom.com )
57
(Pictures from www.physicsclassroom.com )
(Picture from http://www.paul-pelletier.com )
9.4
Simulating real equipment
Simulation tools for equipment or for results are often developed for preparation or visualisation of
use, rather than for training. Depending on the type of product, they can be developed as a service to
the customer (product training) or as a specific product.
58
As the simulation tools are not developed for basic training as such, they often need some extra
preparation before they become useful. Specific content needs to be developed using these tools.
For example, a prepared light set to learn how to make cues, or a multi-track recording to learn how
to mix the sound.
(Pictures from www.lgo.be )
(Pictures from www.lgo.be )
59
(Picture from Resone (www.ohl.to/about-audio/audio-softwares/resone)
9.5
Augmented Reality and Virtual Reality (VR) applications
Augmented Reality refers to technologies that dynamically blend virtual/digital information into the
real environment by the use of devices like smartphones, tablets, goggles, etc. What makes AR so
special is that it is a variation of VR that creates digitally enriched experiences in the real world with
which users can interact in real time. VR is defined as a real and a simulated environment in which a
perceiver experiences telepresence and as a fully computer-generated virtual environment. VR’s
associated hardware provide the user with the illusion of physical presence within those
environments. The difference between AR and VR is that VR completely immerses a user inside a
synthetic environment such that the user cannot see the real world around him, while AR allows the
user to see the real world, with virtual objects superimposed on or composited within the real world.
Such information can be overlaid (e.g., on recognized objects) using smartphones, tablets or AR
goggles as user interfaces between the real and the virtual world. AR also allows a full three-
dimensional (3D) view of virtual objects and enables users to interact with them in the real
environment. In that way, AR supplements reality, rather than replacing it (Azuma, 1997).
(A Taxonomy of Mixed Reality Visual Displays; Milgram & Kishino, 1994)
Both AR and VR applications bring in an interactive, enhanced environment that offers both, learners
and teachers to receive feedback and status reports about the own learning / teaching track and
therefore enables to put quality aspects into learning and teaching processes, emphasizing self-
evaluation on individual learning / teaching tracks. Hence, teaching and training will become much
more transparent, this enables access on collected data and information for identifying individual key
performance indicators for these processes.
60
Moreover, these types of applications introduce opportunities for game-based learning
(gamification) as a motivational aspect of learning. On the other hand, more traditional applications
for digital learning can be integrated in the AR and VR applications.
9.5.1
Augmented Reality (AR) applications
Augmented reality (AR) applications create an interactive experience of a real-world environment
where the objects that reside in the real world are enhanced by computer-generated perceptual
information, sometimes across multiple sensory modalities, including visual, auditory, haptic,
somatosensory and olfactory. In other words, the user/learner is still connected with the real world,
but gets extra information superimposed on the real world and related to what he/she is looking at.
In such an augmented reality one can introduce interactivity. A user/learner watching an object can
be questioned or be asked to perform a task. The application on its turn can respond to the actions
taken by the user. This creates opportunities to integrate motivational aspects and competition.
Object recognition can be based on tags placed on the object or it can be based on machine learned
recognition.
9.5.2
Virtual Reality (VR) applications
Virtual Reality (VR) applications create a virtual world, independent from the real world. The objects
in the virtual world can also be annotated with extra information, questions or tasks to perform
ensuring interactivity.
The VR applications can be screen based, and sensor or tracking controlled. In a sensor-controlled
environment, the learner uses controllers to move around, zoom, point, act, … In a tracked
environment, the movements of the user are tracked and analysed.
9.5.3
Learning analytics (LA)
Learning Analytics (LA) supports the measurement, analysis and reporting of data about learners and
their contexts for purposes of understanding and optimizing learning and the environments in which
it occurs. LA focus on learning and teaching activities in digital supported learning environments such
as Learning Management Systems (LMS) and increasingly will be used in connection with AR- and VR-
applications. Learners, teachers and trainers, and organisational entities can draw conclusions based
on feedback provided by LA, thus are able to e.g. reflect on a learner’s performance, the results from
learning groups and aspects for training programs.LA often implements aspects from big data
analytics.
9.6
Learning Management Systems (LMS)
A learning management system (LMS) is a software application for the administration,
documentation, tracking, reporting, and delivery of educational courses, training programs, or
learning and development programs. The LMS concept emerged directly from e-Learning.
In that sense, the LMS acts in the first place as a container for the different types of
content/applications above. It also makes interaction with and automated response to the students
possible. And it helps the learner and the teacher to keep control over the process and the logical
order of learning events/activities. This becomes more important if learners have individualised,
customised learning paths, like in lifelong learning pathways.
LA is often implemented on a LMS, in that way supports the steering of learning, providing a flexible
pathway, adapted to the specific learner at the specific point of the learning path.
61
Depending on the type of learner and his/her situation, learning can be steered “Pulled” or “Pushed”.
In the Pulled steering version, the learner is responsible for the progress, for actively look for the
content, but the LMS can propose priorities based on past activities. In the Pushed steering mode,
the LMS will actively bring content to the learner in a predefined way, but based also on past
activities.
9.6.1
Micro learning
Micro learning is a typical, but specific type of Pushed learning. The content is divided in small parts,
that can be learned in ten- or fifteen-minutes sessions. The LMS pushes these parts on a regular basis
to the learner. The learner can use “waste time”, time in between other activities, to learn the parts.
The advantage is that these smaller parts can be learned with a high focus.
9.7
Virtual Learning Environments (VLE)
A VLE can be an integrated distance learning platform that typically incorporates course materials,
homework, tests, assessments, and other tools that are external to the classroom experience. VLEs
may also include social media tools that allow learners and instructors to interact via chat or online
discussion boards. VLEs generally use Web 2.0 tools, and also include a content management system.
VLEs are often integrated with an enterprise’s LMS/LCMS, or ERP systems. Virtual learning
environments may provide for both synchronous and asynchronous learning programs. Some VLEs
now incorporate social gaming environments as an extension of the platform.
With the rise of AR and VR, VLE incorporates AR and VR content or were directly build upon these
technologies. In that way, VLEs contain one or multiple applications in a virtual context that relates to
them. The learner moves from one application to another in the same way he/she moves in a game
environment. Interaction with other learners or trainers is possible in a similar way.
(Screenshot from Learning through falling: Second Life in UK academia. April 2009 keynote
presentation at the JISC RSC Northern conference on virtual worlds in education.)
62
- Nina Tiptoe, the dancer uses her free time between warming up and performance to read
short texts and watch instruction videos. When running to keep in condition, she listens to
recorded lectures.
- Tony Solo, the freelancer uses his free time during rehearsals to read short texts, send by a
micro learning system. After the show, he likes to wonder around in the virtual learning
environment in his hotel room. He chooses instruction videos based on the issues he
encounters in the daily work.
- Ben Young, the apprentice, gets text, simulations and instruction videos from his mentor,
based on the needs of his training path. He also trains at home on different simulators to
master new equipment.
- Mary Standard, the student, has access to a traditional LMS, where all her courses are
gathered in order of het school program. She uses the questionnaires as feedback of her
learning activity. In her free time, she trains het ears with a sound recognition software.
9.8
How and why does this feed into LAAR?
The LAAR project developed prototypes of Augmented Reality applications and a virtual learning
environment based on the above concepts. Aspects from LA are integrated and systems like a LRS
(e.g. to record a learner’s experience), a LMS (e.g. to analyse and prepare a learner’s experience
data) and a dashboard (e.g. to show a trainee’s performance) are used in combination. The ETTE
content was used as a test case.
63
Part 2
64
65
Introduction
Where the previous chapters in part 1 describe the underlaying background and research that is used
in LAAR, part 2 focuses on the developments of the project.
Chapter 10 describes the developed design framework, the choice of the scenarios, and the different
applications developed during the project.
Chapter 11 goes deeper into the developed learning analytics application, the learning locker and the
educational dashboard used.
Chapter 12 describes the prototype testing and the user experiences
Chapter 13 concludes with a critical analysis of the strengths and limitations of the use or AR in
workplace training and looks at the future.
66
67
10
Preparation and implementation in LAAR
The LAAR project aimed to create new learning tools for work-based training and learning in form of
Augmented Reality (AR) applications. Within the project, different apps are developed to train
learners by interaction with AR technology in a real live environment. In addition to this, tools for
analysing and tracking learning progress (LA, learning analytics) were developed. The developed tools
were tested with different target groups.
10.1
The choice of training scenarios
A scenario is the description of a block of learning content that can be trained in a practice
environment. The size of the scenario is chosen so that it includes a set of actions that need to be
trained as a whole and that have no or insufficient meaning on their own. On the upper side of the
size the scenario is chosen so that it can’t be split without losing meaning.
To select possible scenarios that could be useful for the LAAR project, we developed a series of
criteria. These criteria ensure that the scenarios are within the scope of the project and provide the
opportunity to verify the developed solutions for the chosen target group.
We defined the criteria as follows:
- The competences to train with the scenario are part of the ESCO taxonomy.
- The scenario offers the possibility to incorporate the principles of the sectoral layer and the
assessment requirements.
- The scenario’s competences are part of the ETTE content
- There is existing conventional training material and methods to compare with.
- The competences incorporate different learning types/levels of complexity
- The scenarios are sector specific, but at the same time they are transferable or at least
understandable for other fields.
- The focus of the scenario is on hands on training
- The scenarios are simple, the focus is on the implementation of AR and LA, not on the
complexity of the content.
- Of course, the scenarios need to provide the opportunity to use AR and LA
- It must be possible to use them on different types of devices (Phone, Hololens, …)
- The scenarios have to be usable in a real live context
- They must provide a possibility for objective observation assessment (no arbitrary
assessment)
Towards the testing environment and the target group, we added some extra requirements:
- The scenario must be easy to implement without the need of a full theatre infrastructure
- It must be usable for learners without any background in the field
- It must be usable for learners in different types of learning paths
- The language skills needed are limited
- The scenario is safe for learners in a testing environment
Based on this requirements, together with experts in the field we listed a series of possible scenarios:
- Inspection of a ladder
- Use of a ladder
- Recognition of safety signs
- Focussing a profile spot
- Recognising safety signs
68
- Venue check
- Storage of chemicals
- Use of chemicals
- Recognising chemical signs
- Choosing PPE (personal protection equipment)
- Connecting trusses
- Connecting spotlights
- Find electrical errors
- Replace light bulb
- Fire reaction
- Find the … (based on safety signs)
- Go get … (recognition of tools
- Find errors in rig
- Make a risk assessment
After the evaluation of a first initial description of these scenarios, we selected two scenarios that fit
all criteria to test and represent two types of learning:
- Training of factual knowledge, by recognising safety signs (from ETTE) in a real live
environment
- Training of procedural competence, by connecting trusses with conical connectors in a
simulated environment
10.2
The theoretical framework
The theoretical framework consists of a technical and methodical framework for Learning Analytics in
Augmented Reality based learning applications. The technical and methodical framework supports
the project’s progress for prototype development in AR, including current technology (devices),
developer tools, design principles, research results useful and other research currently running,
including LA and AR in professional education and training. This contains furthermore a framework
for tracking learner behaviour (xAPI).
The innovation elements are thereby the generic framework in a conceptual form, as well as
functional prototypes in software (e.g., node editor for training processes, training recorder for
recording training sequences, generic tools for the creation of AR/VR environments), for developing
further AR based training applications. Both, framework and prototypes are focussing on learner
behaviour and are designed to support competence and qualification profiles, developed in previous
projects like ECVAET, ETTE, TeBeVAT, following the ESCO guidelines. A further innovation is, that the
conceptual approach for the training applications follows a competence-based structure, in
comparison to a course-based structure in traditional learning environments. Finally, the framework
incorporates elements for assessment to have both, training and assessment integrated in one
application.
Expected impact is first, that the generic framework will fulfill the requirements for developing AR
based training applications and will be used in further developments of such tools, not just for the
present LAAR project but also for future projects in practice.
69
(LAAR conceptual framework)
ELEMENTS OF THE FRAMEWORK:
SYSTEMS:
- ETTE, ESCO; Competence descriptions
- LRS Learning Record Store
- LMS Learning Management System
REQUIREMENTS:
- Competence portfolio; Source: ETTE, ESCO
- Competence requirements catalogue;
Source: Competence Portfolio
- User / Technology affordances catalogue;
Source: Scientific Reports
ACTION:
a learning / training activity supported by AR
applications
PERFORMANCE:
ACTION observed,
investigating Competence in ACTION
SUB-FRAMEWORKS:
- Design principles/elements framework
Source: Learning Theories/Research, Actions
- Performance measures framework
Source: Competence requirements + Actions
- Learning records framework
Source: Competence in Action
- Learning Analytics framework
Source: LA Theory/Research, LRS
- LA information framework
Source: LA framework, Requirements cat.
TWO MODES:
Observation Mode: during learning activity
Assessment Mode: in connection with learning
activity (accompanying, subsequently)
10.3
The design framework
The design framework is the inner core of the conceptual framework and connects theoretical and
empirical foundations with the form and function of concrete AR applications (apps). Therefore, a
systematic literature review was performed to identify and group the learning theories that have
been used in empirical studies on AR for teaching and learning. In a second step, the main system
features of the apps in each theory group were identified, grouped to design elements, and arranged
in a logical and hierarchical structure (i.e., the layers in the final framework). As a result, design
70
elements were identified that can be traced back to both abstract learning theories and concrete
system features.
(The design framework)
The first layer addresses the preparation of the learning content used in an AR application (app) and
contains elements derived from Mayer’s CTML. The second layer introduces mobile aspects of the
learning design to break the boundaries of a single location and to enable, for example, the
integration of location awareness. Since motivation and engagement are fundamental components
of a learning process and AR has the potential to support these aspects of learning, the design
framework incorporates elements in the third layer that include motivational aspects of AR learning
design. Finally, a primary added value of implementing AR learning is that learning can be an isolated
individual task or a collaborative activity. Therefore, the framework includes in the fourth layer the
design of a single learning activity and multiple activities composed in a learning sequence. This layer
also contains elements derived from situated learning theory (i.e. learning at specific places) and
collaborative learning.
Future trends for AR in teaching and learning focus on measuring and evaluating learning in
personalized student learning experiences, which requires collecting information about the learner’s
behaviour, activities, and results and exchanging this information with external systems. Both the
internal and external process management and the communication between learning activities and
with learning management systems can then be supported by the implementation of an input-output
hub (IO-hub).
10.4
The implementation
We developed several collaborative AR-based prototypes for smartphones, tablets and the Microsoft
Hololens and filled it with content from an existing competence requirement catalogue, developed in
collaboration with training experts from industry.
We collaborated with domain experts from the event technology industry in all stages of the
development and evaluation process. The app followed a predefined storyline and considered the
training requirements derived from a catalogue covering competence requirements, which should be
71
implemented in the training session. Furthermore, the design of the app was based on design
elements for AR learning app design suggested by Sommerauer and Müller (2018).
The following figure shows the conceptual setup. For logging the user data, we implemented so-
called experience-statements recorded on a learning record store (LRS), applying the xAPI. For this
purpose, we used LearningLocker® and shaped the xAPI statements to log a user’s data to fit the
requirements of the performance measurement catalogue.
(Conceptual setup for app development)
Aspects derived from the competence requirements catalogue determine the applied design
elements for the app design and the learning records implemented in the app and define the
investigated performance metrics, e.g. task performance. The app in practice sends user data to the
LRS to provide it for further processing, i.e. for learning analytics and to prepare a feedback for users
(e.g. on a dashboard).
10.5
Learning the safety signs
For the training of factual knowledge, we chose a scenario for recognising safety signs (from ETTE) in
a real live environment. Understanding the meaning of safety signs, and learning where they can
appear in a real live environment is part of the factual knowledge needed for safe behaviour.
Safety signs are explicit knowledge, one must be able to recognise them and explain the meaning in
the specific context.
10.5.1
Analysing the conventional learning method
Traditionally, the learner would be thought the different symbols or read them from a book. The
learner would memorise them and get a kind of test or examination at the end.
The limitations in this methodology are that it is focussed on short term remembering (learning for
the exam). There is limited consolidation (permanent storage in the memory) of the knowledge.
Moreover, the knowledge is disconnected from the real live environment.
To improve this, repetition is needed in different situations in a real live environment. This creates
references to the normal use and context. Ideally the repetition includes also motivating elements, to
avoid that the activity becomes boring.
10.5.2
The i-Phone/i-Pad app “Explore-App” and “Who Am I?”
The LAAR project developed an application to learn about objects in a real environment, e.g. the
different safety signs. The app was developed in multiple iterations (first version: Explore-App, final
72
version: Who Am I – app), is multilingual and includes three different modes: a training mode, an
exploration mode and a quiz mode.
(The features of the “Who Am I” app)
In training mode, the user can browse to a catalogue of safety signs to learn. Alternatively, the signs
can be searched based on keywords. For each sign, a detailed description is available based on the
ISO descriptions and the developed learning outcomes. This mode is similar to the conventional
learning method.
(The “Who Am I” app in learn mode using database search and keyword search)
The exploration and quiz mode incorporate AR and machine learning technologies to recognize
objects in the environment. The app combines machine learning techniques for image recognition
with machine translation to identify objects that are in the focus of the mobile phone camera in real
time and superimposes information like the object’s name onto the object in various languages. This
helps learners to visually recognize objects in their natural environment and learn about their names
73
and descriptions. The fact that the application does not need markers to identify objects makes it
usable in every environment. The learner focuses on a particular safety sign using the device’s
camera (e.g., smartphone, tablet, any head-mounted device).
In the exploration mode the app shows the most likely label for the sign and superposes extra
information about it when clicking on the image.
(The “Who Am I” app in explore mode)
The quiz mode, which is implemented to support learning at any time and at any place, presents a
selection of labels for an identified safety sign, and the user chooses the one that is correct. The app
gives feedback for correct and incorrect answers.
(The “Who Am I” app in quiz mode)
74
(The “Who Am I” app in explore mode and quiz mode)
The “Explore App” / “Who Am I” app follows the idea of implementing object recognition and its
application in both school and professional education (VET). The applied design principles follow
Billinghurst et al. (2015) (i.e., real physical objects/virtual elements displayed and linked with an
interaction metaphor) and the design framework, as presented earlier.
The “Explore-App” also supports the task of learning names related to physical objects used in an
every-day environment (e.g., at home, at the office) or in other professional domains. The examples
below show the use in an office/home environment and a professional florist environment.
(The “explore app” in explore mode, quiz mode and selection of language and theme)
(The “explore app” in a field test using the flowers theme)
75
10.6
Two ways to train the egg
For training of procedural knowledge (competence), we chose a
scenario for connecting trusses. Trusses are the modular
aluminium structures used at concerts and events. They are used
to hang sets, light, sound and video equipment. There are
different ways to connect trusses. In the chosen scenario we use
conical connectors (eggs) that fit into the truss ends.
This is an activity that is often performed on a low level. It is a
very basic action, taking only a couple of minutes for an
experienced practitioner. But on the other hand, the procedure
includes a set of activities, checks and choices that have to be
performed in the right order. Additional difficulty is that there
are more than one “right orders”.
10.6.1
Analysing the procedure
In a first step, we analysed the procedure, listing every action
and choice to be made, including loopbacks if for example a part
does not have the required quality or a part doesn’t fit perfectly.
In the procedure typical common elements reoccur:
- Choosing parts or equipment
- Checking parts
o If no loop back (replace)
o If yes, continue
- Action putting parts in the right place
- Check mounting is done properly
o If no loop back (redo)
o If yes, continue
All these elements were put in a “flow chart like“ map that
supported the development of the app. The map also indicated
the complexity of choices and decisions to be made even on this
low level. This becomes even more significant if we realise that
one wrong choice can end up in a lethal accident.
(Flow chart connecting a truss)
76
10.6.2
Analysing the conventional learning method
Next, we analysed the conventional methods used in different types of training situations. As this is a
basic competence, it is trained as well on the floor, through self-instruction, by peer learning and in a
traditional school context.
10.6.2.1 Trainer method
In a trainer or “Master - Apprentice” approach, the trainer shows step by step how the action is to be
executed. The trainer will point out all the tricky elements, the choices to make and the checks to be
done. The learner copies each step separately. After this is done, the trainer would repeat whole
sequence at once for the learner to repeat. In a next step the learner will perform independently, but
the trainer will keep overview and double the checks to be sure no risks occur. Bit by bit the learner
gains independency until the trainer decides checks are no longer needed. This type of training is
often based on experience.
10.6.2.2 Instruction sheets
Several organisations (like USITT) and
manufacturers have developed instruction
sheets that show all details to be considered
when performing the truss connections. These
are typical examples of self-training material.
The learner bases his training on the
instructions, but there is no check or feedback
on his learning.
10.6.2.3 Peer method (for practicing)
Once the learner has acquired basic
understanding of the competence, peer
learning/checking can be used. The learner
performs the actions and checks and a peer
verifies the actions. To improve the quality of
this peer verification, the peer uses a checklist.
The checklist is developed so that it only needs objective observation and no judgement is involved.
An example of such a checklist is the one below:
- Wears the proper PPE (safety shoes, gloves, hard hat)
- Handles trusses with care (they are fragile)
- Lifts and carries ergonomically
- Puts trusses on blocks
- Checks trusses visually for damage
- Chooses the right connectors and pivots
- Checks the connector for damage
- Checks the connector and the receiver for dirt
- Slides connector in, in the right direction
- Checks spigot for damage
- Fits the spigot in the connection
- Secures the spigots with R springs
- Checks the R springs are fixed
- Lines the trusses up
- Checks the second receiver for dirt
(Instruction sheet trusses, ESTA – USITT)
77
- Checks the truss connection fits
- Fits the spigots on the second part
- Wiggles the truss to fit the spigots
- Chooses the right type of hammer (if needed)
- Uses a minimum of force
- Secures the spigots with R springs
- Checks the R springs are fixed
All the statements are written in a way that they can be observed, without interpretation. At the end
of the activity, the learner gets feedback based on the statements observed. The statements can be
traced back to larger competences, of which they are part. By doing so, the learner can prove the
mastering of these competences in different situations.
10.6.2.4 School method
In a traditional school environment, the learner will learn first the theory, the background, the
calculations, … behind the activity. Then the procedure is explained and the learner has the
opportunity to try this out in protected environment. The problem is often that there is limited time
to practice. In the best case, practical work experience is added to create more training time.
10.6.2.5 Disadvantages of the existing methods
Most methods are very time consuming for the teacher/trainer. Often there is a lack of visual
material to give insight (you can’t see the inside of the connection). While training, there are no time
limits, while in reality there are. On top of this, the trainer will unconsciously correct learner or
influence the situation. Therefore, the learner cannot learn from mistakes. There is no tracking of the
quality.
An improved method should include visual information, no correction during the training (learning
from mistakes). It should take advantages of the existing methods in account like the fact that a
learner verifying the result learns from his/her peers mistakes.
10.6.3
The phone app “Follow Me”
The phone app for learning the conn