Content uploaded by Ulf- Daniel Ehlers
Author content
All content in this area was uploaded by Ulf- Daniel Ehlers on Aug 24, 2016
Content may be subject to copyright.
Quality Management
of Peer Production -
Quality Management
Handbook
Ari-Matti Auvinen, Ulf-Daniel Ehlers, Duisburg-
Essen, January 2009
Reference: Ehlers, U.-D. (2009): Quality Management of Peer Production
(QMPP) - Quality Management Handbook
Correspondence address: Prof. Dr. phil. habil. Ulf-Daniel Ehlers
Duale Hochschule Baden-Württemberg | Baden-Württemberg Cooperative State University
Friedrichstraße 14, 70174 Stuttgart
Deutschland | Germany
mail: ehlers@dhbw.de | ulf.ehlers@googlemail.com
skype: ulf.ehlers
web: www.dhbw.de | www.ulf-ehlers.net
Quality
Management of
Peer Production
Quality Management Handbook
The project Management of Peer Production of eLearning (QMPP) has been
funded with support of the Lifelong Learning Programme of the European Commission.
This publication reflects the views only of the authors, and the Commission cannot be
held responsible for any use which may be made of the information contained therein.
Handbook version 0.99
Supported by
Handbook for Quality Management of Peer Production
2
Table of Contents
I FOREWORD - ABOUT THE QMPP PROJECT 4
II PURPOSE AND TARGET AUDIENCES OF THE QMPP HANDBOOK 7
III DEFINITIONS IN QUALITY ASSURANCE 9
1 INTRODUCTION 11
2 PEERS, PEER GROUPS, PEER PRODUCTION, AND QUALITY OF PEER
PRODUCTION 13
2.1 WHAT IS A PEER? 13
2.2 WHAT IS A PEER GROUP OR PEER COMMUNITY? 14
2.3 WHAT IS PEER PRODUCTION? 15
3 COMPARING EXISTING PEER PRODUCTION QUALITY MEASURES 17
4 APPROACHES AND METHODOLOGIES FOR PEER PRODUCTION 30
4.1 QUALITY DEVELOPMENT OF PEER PRODUCTION 31
4.2 CONDITIONS FOR QUALITY OF PEER PRODUCTION 33
4.3 CONCEPTS AND METHODS OF QUALITY DEVELOPMENT FOR ELEARNING 2.0 39
4.3.1 SELF-EVALUATION 39
4.3.2 QUALITY ASSESSMENT WITH E-PORTFOLIOS 41
4.3.3 SOCIAL RECOMMENDATION AND COMMUNITY PARTICIPATION 44
4.3.4 EVALUATION PROCESSES AIMED AT A TARGET GROUP 48
5 TECHNICAL TOOLS AND TECHNOLOGIES FOR PEER PRODUCTION 50
5.1 TECHNICAL TOOLS AND TECHNOLOGIES IN THE WEB 2.0 ERA 50
5.2 TECHNICAL TOOLS FOR PEER PRODUCTION 51
5.3 NEWER WEB 2.0 SERVICES 55
Handbook for Quality Management of Peer Production
3
6 ENABLING PEER PRODUCTION 58
6.1 ENABLING POLICIES AND PROCEDURES 59
6.2 ENABLING PROCESSES 60
6.3 ENABLING TOOLS 61
7 THE QMPP QUALITY APPROACH 63
7.1 BASIS FOR THE QMPP QUALITY APPROACH 63
7.2 QMPP QUALITYSCAPE 64
7.3 QMPP EXPERIENCES AND CASE STUDIES 67
8 BUILD YOUR OWN QUALITY DEVELOPMENT APPROACH 73
8.1 ASSESSING YOUR CONCURRENT PRACTICES 73
8.2 DEVELOPING AND DEFINING YOUR QUALITY APPROACH 76
8.3 DOCUMENTING YOUR QUALITY PLAN FOR PEER PRODUCTION 78
9 RECOMMENDED READINGS 81
9.1 RECOMMENDED LITERATURE 81
9.2 WEBLIOGRAPHY 83
Handbook for Quality Management of Peer Production
4
I Foreword - About the QMPP project
QMPP (Quality Management of Peer Production of eLearning) is a joint
European project funded by the European Commission under the Leonardo da
Vinci programme. The overall aim of the project was to accelerate the creation of
peer-produced eLearning content by providing a methodology and a process to
manage its quality. This overall aim also supports the empowerment of
eLearners in vocational education and training in their development from passive
receivers of eLearning content towards active producers of content of their
specific knowledge areas.
Peer production of eLearning content is a growing trend, which will play an
elementary role in creating, validating, enriching, editing, and updating of digital
learning content. Thus it is important to pay growing attention to the management
and support of peer production in eLearning, which will also enable new learning
material production and updating work methods.
The specific aim of the project was to develop a solid approach and methodology
on how to organise and support the quality management process of peer-
produced eLearning content this Handbook is guiding the users to design,
create and implement their own quality management processes to support the
peer production of eLearning in their organizations.1
The QMPP project itself did not take a position on the eLearning tools (such as
eLearning platforms or Learning Management Systems) used, but aimed to
develop and implement a systematic process for the quality management of
peer-produced eLearning content.
A critical part of the project were the real life pilots. These piloting experiences
were an important element in the authoring of the Handbook. Additionally, other
participatory working methods were used during the QMPP project such as
expert panels in various European countries in planning and validating the work.
As a European project, the QMPP had an active and versatile European
partnership. The partnership consisted of the following partners:
HCI Productions Oy (Finland) contractor of the project
Scienter (Italy) coordinator of the project
European Foundation of Quality in eLearning (EFQUEL) (Belgium)
Finnish eLearning Centre (Finland)
IAVANTE Foundation (Spain)
Institute for Innovation in Learning (FIM NewLearning), Friedrich-Alexander-
University Erlangen-Nuremberg (Germany)
Scienter Espana (Spain)
University of Macerata (Italy).
1
general term to describe educational institutions, companies and various
other entities
Handbook for Quality Management of Peer Production
5
Peer production is as we will explore in this Handbook a wide phenomenon.
We have limited the scope of this project in order to cover in more detail the
selected area of peer production.
We wanted to concentrate on the organizational level / institutional level and
provide practical tools primarily for the quality management of process peer
production of eLearning in vocational education institutions and vocational
training institutions. However, we feel also that most if not all of our work can
also be applied by other educational institutions and also within the field of
corporate training.
We made also the choice to emphasize in peer production the importance of the
peer group in the various phases of the learning provision. According to our
reading of the literature, the peer produced (or if you like: the user-created
content) is mainly describing the digital artefacts produced by various individuals,
as in our approach the communicative element is essential and it takes often
place by the strong involvement of the peer group. Thus according to the
different terms and their use we can summarize our emphasis according to the
following picture (see picture 1).
Picture 1: Focus of the QMPP work
Handbook for Quality Management of Peer Production
6
This Handbook is a result of joint work within the QMPP project. However, the
Handbook was mainly authored and edited by two main authors, namely Mr. Ari-
Matti Auvinen (Senior Partner, HCI Productions Oy) and Dr. Ulf-Daniel Ehlers
(Vice President, EFQUEL).
In addition to the co-authors, several active members of the QMPP project
consortium have contributed to the development of the Handbook in various
roles. The authors would like to thank all project partners, and in particular the
following persons:
Ms. Kaisa Honkonen-Ratinen and Ms. Kristiina Jokelainen (HCI
Productions Oy)
Mr. Tim McQuaid, Mr. Jose Pinzon and Mr. David Riley (IAVANTE)
Mr. Thomas Fischer, Mr. Thomas Kretschmer and Dr. Walter Kugemann
(FIM Neues Lernen)
Ms. Michela Moretti (Scienter Italy)
Ms. Laura Fedeli (University of Macerata).
Handbook for Quality Management of Peer Production
7
II Purpose and target audiences of the
QMPP Handbook
The objective of this QMPP Handbook is to serve as a practical guide in the
creation and implementation of quality measures for peer production in
eLearning. The QMPP Handbook is written with a very pragmatic approach, and
thus the key intention is to provide guidance and assistance for the practical daily
work.
The QMPP Handbook is authored as a practical guide on quality management of
peer production in eLearning, and it links the theoretical concepts and latest
research outcomes with practical decisions and actions to be undertaken.
Although firmly based on concurrent research and literature, the QMPP
Handbook serves also users not familiar with quality management concepts. We
have included to the Handbook also a wide selection of recommended readings,
which cover thoroughly the field of peer production in eLearning.
The structure of the Handbook is based on the combination of theoretical and
experimental knowledge of peer production in eLearning and practical application
of this knowledge to individual organizations. The aim is to create wit a step-by-
step approach the own quality development approach for the users. The
structure of the Handbook is visualized below.
Handbook for Quality Management of Peer Production
8
T
users, but they also build the necessary basis for the building of the own quality
development approach for the users.
The target audiences of the QMPP Handbook include all trainers and educators
working with peer-produced eLearning content. Many of the case studies are
linked with vocational education and training, and thus the QMPP Handbook can
be of special interest to educators and trainers working with vocational education
and training as well as corporate training. However, we expect also that
educators from primary education to higher education find the Handbook useful
in their work.
In addition to the Handbook, the QMPP project will also more content through its
website. On the website you can find also additional electronic resources you
find the QMPP website at
http://www.qmpp.net
Handbook for Quality Management of Peer Production
9
III Definitions in Quality Assurance
The following definitions are commonly accepted and should be a useful point of
reference throughout the Handbook.
QUALITY ASSURANCE
The means by which an institution can guarantee with confidence and certainty,
that the standards and quality of its educational provision are being maintained
and enhanced.
QUALITY CONTROL
Quality control refers to the verification procedures (both formal and informal)
used by institutions in order to monitor quality and standards to a satisfactory
standard and as intended.
QUALITY ENHANCEMENT
Quality Enhancement is the process of positively changing activities in order to
provide for a continuous improvement in the quality of institutional provision.
QUALITY ASSESSMENT
Quality Assessment is the process of external evaluation undertaken by an
external body of the quality of educational provisions in institutions, in particular
the quality of the student experience.
QUALITY AUDIT
Quality Audit is the process of examining institutional procedures for assuring
quality and standards and whether the arrangements are implemented effectively
and achieve stated objectives. The underlying purpose of Continuation Audit is
"to establish the extent to which institutions are discharging effectively their
responsibilities for the standards of awards granted in their name and for the
quality of education provided to enable students to attain standards."
STANDARDS
Standards describe levels of attainment against which performance may be
measured. Attainment of a standard usually implies a measure of fitness for a
defined purpose.
QUALITY CULTURE
Quality Culture is the creation of a high level of internal institutional quality
assessment mechanisms and the ongoing implementation of the results. Quality
Culture can be seen as the ability of the institution, program etc to develop
quality assurance implicitly in the day to day work of the institution and marks a
move away from periodic assessment to ingrained quality assurance.
ACCREDITATION
Accreditation is the result of a review of an education program or institution
following
recognition that a program or institution fulfils certain standards.
Handbook for Quality Management of Peer Production
10
This page is left intentionally blank
Handbook for Quality Management of Peer Production
11
1 Introduction
Peer production can be defined to include the digital content created, edited,
contemporary examples in the Internet of peer produced digital content include
e.g. YouTube2, Facebook3, blogs, flickr4, slashdot.org5 etc.
6
rson who is equal to
7
as follows:
8
peer group is a group of approximately the same age, social status, and
interests; generally, people are relatively equal in terms of power when they
9.
In the context of involving peers to the educational process, some use also the
-to-Peer-to-peer is often linked in the technical sense
-to-
several computers with another as equals. Some authors claim also that the
concept of peer-to-peer networks is increasingly evolving to an expanded usage
as the relational dynamic active in distributed networks - not just computer to
computer, but human to human. Thus e.g. Yochai Benkler claims that associated
with peer production are the concepts of peer governance and peer property in
the digital world.10
Quality development for education and eLearning, which means evaluating
learning contents and processes, certifying and accrediting programs and
institutions, is becoming more and more important. Quality management means
defining comprehensive organizational processes within an educational
institution and agree on indicators for their goodness. Quality assurance
examines whether a promised level of quality can actually be kept. Quality
control is supposed to detect mistakes and prevent them. What, however,
happens in learning scenarios in which eLearning 2.0 is involved? In cases in
which learning material is not fixed beforehand, learning processes are highly
diverse and not unified and learners find their own way of learning? And what
about those education processes which happen outside of the programs and
2 see http://www.youtube.com
3 see http://www.facebook.com
4 see http://www.flickr.com
5 see http://slashdot.org
6 see http://www.thefreedictionary.com/peer+ (read 16 July 2008)
7
Random House 1989
8 see http://www.thefreedictionary.com/peer+group (read 16 July 2008)
9 see http://en.wikipedia.org/wiki/Peer_group (read 16 July 2008)
10 see Benkler 2006
Handbook for Quality Management of Peer Production
12
formal educational institutions? Who determines the quality of such learning
scenarios, what can then be assessed at all and which methods can be used to
improve quality?
In this handbook we will outline resource, methods, actors and processes to
support the quality development and evaluation in peer-production scenarios.
Action point
You have probably used some digital content provided by Web 2.0 applications –
such as YouTube, Wikipedia or Flickr. Reflect in the following your understanding
HOW quality assurance works in these “social software” applications (reason ,
based on your practical experience, how their quality assurance works) and
WHAT kind of a quality management approach they promote (reason, based on
your practical experience, what type of quality management approach they
promote)?
Handbook for Quality Management of Peer Production
13
2 Peers, peer groups, peer production, and
quality of peer production
This chapter of the handbook assists the users with the key definitions
of peers, peer groups, peer production and quality of peer production.
The basis for this chapter has been developed by the expert panels of
the QMPP project held in different European countries.11
2.1 What is a peer?
What are the key characteristics of a peer? We described previously that a peer
a person of equal social standing, rank, age, etcposes
this might be a too generic definition. We should deepen this understanding of a
peer in our context of peer production and define some characteristics of a peer
and his/her attitudes. Thus our expert panels defined
as follows:
As a part of a community a peer is or should be open-minded, perceptive and
aware of his competencies and responsibilities in terms of effort, attention and
reaction rate. In this context peers are expected to possess strong relational and
auto-critical skills, exhibit good relationship management and a deep subject
matter competence in a specific sector of knowledge.
The characteristics of a peer can be visualized as follows (see picture 2):
Willingness to share, re-edit and create
Clear and explicit objective to support each other
Key characteristics/attitudes
of a peer
Open-mindedness
Effort, attention and reaction rate
Awareness of competencies and responsibilites
Relational and auto-critical skills
Good relationship management
Competence in a specific sector
Picture 2: Key characteristics/attitudes of a peer
11 see in more detail: Fischer, T. and Kretschmer, T: Benchmarking Peer
Production Mechanisms, Processes & Practices. Deliverable of WP3 of the
QMPP project, 2008.
Handbook for Quality Management of Peer Production
14
2.2 What is a peer group or peer community?
A peer group is the group of peers relating to a single person or group. First of all
it is agreed on the fact that due to the multitude and diversity of peer groups and
communities it is virtually impossible to determine concrete characteristics which
hold true for all sorts of peer groups. According to this circumstance the following
general characteristics of a peer group were specified in order to determine an
abstract definition.
Peer group members are willing to share materials, re-edit existing ones and
create knowledge and they have a clear and explicit objective to support each
other in order to grow together. Authority within the group is very seldom based
on a hierarchy of roles. Rather than that, in most communities a level of
members develops naturally and in accordance with the degree of participation,
quality of contributions and the confidence acquired from experience and
competencies.
Furthermore openness and recognizing and exploiting ways of shared
communication and operational modalities are considered to be essential for a
peer group to prosper. This becomes even more evident in face of the
understanding that sharing should not be limited to the exchange of learning
products but also explicitly address the sharing of experiences concerning
learning processes, paths and projects.
The characteristics of a peer group or a peer community can be visualized as
follows (see picture 3):
Definition/Meaning of a
“Peer Group/Community“
No authority by hierarchy of roles
Presupposes suitable attitude of peers*
Level of members develops naturally
Sharing experiences of learning processes, paths and projects
Sharing of needs, interests, objectives and learning products
Provides ways of shared communication and operational modalities
Picture 3: Definition and meaning of a peer group or peer community
Handbook for Quality Management of Peer Production
15
2.3 What is peer production?
We start with assessing with the given definition of peer production “The digital
content for learning created, edited, enriched by peers, in other words by people
on the same hierarchical level” The definition can be regarded to be adequate for
limitation to digital content is considered to be inappropriate and too restrictive.
nal modules and/or
whole courses, or at least to the path pro
the misleading idea that it only refers to individuals of the same organisation. And
thus to conclude: it is perceived to be important to explicitly highlight the fact that
collaboration very often occurs cross-organisational and in the corporate sense
can even reach across different companies or suppliers.
In addition another field which has been given importance in the expert
statements is learning by peer-interaction. These learning approaches can be
number of developments, trends and points of view, which require change from
teaching to learning. According to the discussions in the expert panels, the new
point of view essentially connects eLearning with five characteristics:
1. Learning takes places always and everywhere (ubiquitous) and therefore in
many different contexts, not only in the classroom.
2. Learners take on the role of organizers.
3. Learning is a life-long process, has many episodes and is not (only) linked
to educational institutions.
4. Learning takes place in communities of learning (so called communities of
practice, see e.g. Wenger 1999): Learners participate in formal, as well as
informal communities.
5. Learning is informal and non-formal and takes place at home, at the work
place and during leisure time and is no longer centred on teachers or
institutions.
ELearning 2.0 means using social software and learning services, which can be
com
context as technology alone does never determine its use.
According to the expert panels the most significant characteristics of peer
production are recognizing the value and necessity of complex processes within
peer production, such as creation, sharing and editing and the readiness to be
open and receptive towards inputs from different individuals, fields and
directions. Besides that it is agreed on the fact that sharing the same fields of
interest and objectives is a prerequisite.
The characteristics of peer production can be visualized as follows (see picture
4):
Handbook for Quality Management of Peer Production
16
Definition/Meaning of a Peer Production
Presupposes suitable attitude of peers*
Attendance to and preparedness for peer-interaction
Presupposes suitable constitution of the group**
Understanding of learning as an life-long and ubiquitous quest
Readiness to colaborate beyond the groups scope;
Final validation often occurs outside the group
Appreciation of formal and informal approaches
Learning as a process organised by learners;
"Learning-team-centered" - approach
Adequate utilisation of tools and techniques
Picture 4: Definition and meaning of peer production
In conclusion the understanding of peer production builds on the key
characteristics/attitudes of a peer and the definition and preconditions of peer
groups and communities.
Action point
You have now been acquainted with the views of the European expert panels on
peers, peer groups and peer production. Reflect on the following – based on your
own experience – these definitions. How would you supplement these
definitions? Are there some characteristics in your organization that these
definitions are not covering? Who is your direct peer? Who are forming your peer
group?
Handbook for Quality Management of Peer Production
17
3 Comparing Existing Peer Production
Quality Measures
This chapter of the handbook guides users through the different
contexts in which peer-production can take place, ranging from
educational learning scenarios to production scenarios for content.
“Download a lecture off the seminar web page as a pod cast in the
morning, take part in an online session of an international studying group
for the purpose of preparing for an exam in the afternoon and log into the
virtual world of Second Life to take part in a tutorial relating to the
morning’s lecture – the daily routine of studying looks like this or similar
more and more often. In companies, online trainings are no longer visions
of a distant future, but reality for more and more employees” (Ehlers
2008).12
Teaching and learning is changing. If peer production is an inspiring method in
the joint knowledge creation and knowledge refinement, it also has a lot of
potential in the area of education and training. It is about the use of online tools
such as blogs, wikis or pod casts for learning and teaching, and for joint
knowledge creation. Learners can create their own contents and exchange
information in networks like the video platform YouTube
(http://www.youtube.com eLearning
ed", "immersive learning",
"connected learning", "game-based learning", "workflow (informal) learning",
"mobile learning". On top of that, he sees a development from standardized
13
But what is really meant by that? What constitutes the new, innovative element
that is described by Web 2.0 and eLearning 2.0? Peer-Production? And above
all: Will this development have consequences for quality assurance,
management and development in eLearning? And if so: Do we need new
methods and concepts to improve and assure the quality of eLearning 2.0.
12 the introduction to this chapter is based on Ehlers 2008
13 see in more detail Downes 2007
Handbook for Quality Management of Peer Production
18
In the following section we present several case studies of various practices
where knowledge is peer-
mode to
Case I: The Finnish Tax Academy14
The Finnish Tax Academy is in charge of the learning activities within the Finnish Tax
Administration. The challenges in the work of the employees within the Tax
Administration include, among others, the following: i) quickly changing regulatory
environment (and thus permanent need for updates and maintenance), ii) expertise in
taxation issues really is within the Tax Administration operational staff (and thus
restricted potential to utilize external expertise).
The Finnish Tax Academy has been an active user of eLearning since the year 2000 in
2007 approx. 1300 employees studied at least on one eLearning course (more than 20 %
of the workforce). The key of the eLearning approach is that all the eLearning courses
are produced by the Tax Administration staff and experts; within the staff of approx. 6000
persons, there are more than 100 trained eLearning content providers and 110 eLearning
tutors.
Authoring eLearning courses provides an important potential to enrichen the work of
many experts by adding the learning provision task to their work. This is enabled by a
clear workload assignment and thus eLearning content creation is an elementary part of
n use 32 person-days per course unit (20
learner hours) of allocated workload. It is also important that instead of compensating
separately for the learning material provision, it is seen (in the spirit of a learning
organization) as an organic part of everyday work.
The peer producers of eLearning content have been specially trained by an intensive
training program, which has been organized by an external university entity. It consists of
eight face-to-face training days as well as of own web-based courses and own web-
based platform for communication, discussion and reflection.
The process of eLearning course content authoring has been jointly defined to include
the following steps:
- first edition
- peer reviews
- revised edition
- piloting with a limited audience (including structured feedback collection)
- final version
Operational and pedagogic support is provided both by the own Tax Academy team as
well as by external actors (universities, polytechnics, consultants).
As a result of this work, the Tax Administration regards eLearning as an elementary part
of their learning, the Tax Academy produces or reviews more than 20 eLearning courses
annually (in all they have more than 60 courses on offer) and they also provide their
eLearning courses to all Tax Administration employees with a free access also for self-
studying.
14 this case study is authored by Ari-Matti Auvinen (HCI Productions Oy)
based on the first hand personal experiences in working with the Finnish Tax
Academy, see also the Annual Report of the Finnish Tax Administration at
http://www.vero.fi/nc/doc/download.asp?id=6169;1863506 (read on 18 July
2008)
Handbook for Quality Management of Peer Production
19
Case study II: Eureka at Xerox15
Eureka project was started within the Xerox company in 1994. The challenge for Xerox
was to create a working mechanism to collect and distribute technical and maintenance
information for the front-line people. This was based to the notion, that within the Xerox
in other words not in well-structured databases or documents.
The principal idea in the Eureka project was to support the field engineers and
were the following:
- the maintenance manuals were outdated quickly and the printed versions were
soon out-of-date and difficult to maintain in an accurate condition
- the maintenance manuals did not include many innovative front-line work methods
or pract
The experiences Xerox collected of its front-line co-workers showed that, in addition to
the official technical documentation and in many cases instead of the official technical
documentation the front-line people used their own notepads, selected only certain
important pages of the manuals and wrote their own procedural notes. A part of the
of the self-made and self-summarized technical documents.
Within the Eureka project a simple-to-use database was constructed and every front-line
person could contribute to that database and save his/her own experiences. The basis
structure for the input was the following:
- product/component/part
- experienced problem
- proposed solution
- chain of messages (tree metaphor of messages e.g. a new message, answer,
comment etc.)
- author
- validator.
The database could be accessed also through a search engine, which was either a drop-
down menu or free text search. The tools developed within the Eureka project included
also a peer review tool (close to what e.g. Amazon uses today in peer rating of their
books), by which the users could assess the usability and validity of each input. It was
also important for the company, that the Eureka documents were routed to the R&D
entities, which could thus receive direct, real, and timely feedback from the front-line.
Eureka provided its users also with small guides (hint books) covering the following
subjects:
- authoring what to cover, how to describe the problem and solution etc.
- validation how to validate, what aspects to consider, what references to use etc.
- sharing how to share my knowledge, what are the key benefits of sharing etc.
- utilization how to use, what are the limitations of using Eureka etc.
15 sources for this case study included, among others, Ahmed, P.K. Kok,
L.K. Loh, A.Y.E.: Learning through knowledge management. Butterworth
Heinemann 2002 and Brown J.S. - Duguid P.: Balancing act: How to capture
knowledge without killing it. Harvard Business Review 78 (3) (2000), 73-80
Handbook for Quality Management of Peer Production
20
Although the reward system of producing content to Eureka was one critical aspect to
consider, in some countries the front-line co-workers spoke against monetary rewards of
information to be saved into Eureka, as they claimed it might corrupt the quality of the
content, would it only be provided triggered by monetary rewards.
As a summary one can state that by using the Eureka Xerox increased its confidence to
their own front-line organization, their problem-solving skills and their continuous
innovativeness. In addition, Xerox claims that according to its own calculations, it has
saved 100 million USD (by the year 2000) in various stages of its operations by using the
Eureka approach.
Case study III: Bank of innovative practices at the IAVANTE foundation16
Bank of Innovative Practices (BPI) has been developed by IAVANTE foundation for the
Ministry of Health in the Andalucian Regional Government. It is a platform with the aim of
fostering and improving innovation and diffusion in the Andalucian health sector, by
means of identification of innovative actions which will be carried out in itself and its
diffusion to the rest of the system, so that they can act from pilot or exemplary
experiences to the rest of the System. Each innovative practice or innovative idea is
developed by a peer professional group from the system.
BPI allows the registration as participant, which offers the following alternatives that can
ease peer production:
- to inform and to be informed about the innovative practices which are being
developed in the Andalucian Public Health System
- to create collaborative working spaces for the development of innovative practices
between professionals of different organizations (belonging to the Ministry of
Health for the Andalucian Regional Government)
- to communicate with other professionals or working parties with similar problems
- e practices which are being
developed, as well as to contribute with ideas and suggestions.
The BPI provides its participants with a set of collaborative tools that allow for the peer
production of content. Project teams have at their disposal a common space where they
can share documents, post links, comments and calendar events. In addition to that,
there are other site-wide tools available such as a tagging system for all content and a
voting system. Comments by users are allowed for every content item in the site. Users
can choose the configuration of their public profile.
The registration in innovative practices which have recognition as such after the
evaluation by BPI, entitles members of the team responsible for them to get a certificate.
Moreover,
repetition of those initiatives that are more innovative.
For each innovative practice (a project) or innovative idea (a proposal), there is a set of
parameters analyzed by t
aspects such as efficiency, effectiveness, level of innovation, gender equality, and
adequacy with wider health policies. For the ideas or proposals, innovation potential,
feasibility and potential impact are assessed.
16 this case study has been authored by Jose Pinzon (IAVANTE)
Handbook for Quality Management of Peer Production
21
Case study IV: KONE Elevators and peer-produced technical training17
-
in various organizations among others, the Alcan aluminum factory in Kurra Kurra
(Australia) documented their approach in a sound and condensed manner. In a pilot
project in KONE Elevators (the large Finnish elevator company) Australia, new models to
create front-line maintenance information by the front-line people themselves were
piloted.
The important basic notion was that in developing maintenance instructions and manuals
for the front-line, the quality of the content can be improved by using as an author an
experienced supervisor than a documentation expert or a trainer. This was simply due to
the fact that the experienced supervisor knew and could govern the variety of existing
front-line documentation, and thus avoid double or triple amount of work in creating the
documents (thus e.g. in an maintenance documentation the general process of a
maintenance operation was only described once, if it was different from the ordinary
maintenance operation; if it was similar to the ordinary maintenance operation, there was
just a link created to the conventional procedure). Another key element was that this
method also condensed the front-line documentation, as it was concentrating on the
-in-
The key component of this model was thus the utilization of the supervisors in both
content creation as well as content validation. They were supported by a number of
critical tools (such as resource guides). The experience in the project showed, however,
that the supervisors could more accurately than the documentation unit of the HQ identify
the key essential guide lines and shortcuts and thus they could also avoid unnecessary
work phases and authoring of unnecessary (or overlapping) documents.
The results showed also that the front-line initiated technical documentation was by far
cheaper in its global costs (authoring, translation, printing, distribution) than centrally
created documentation was. However, the basis of the meaningful work load in the front-
line and thus the cost savings is the appropriate division-of-labor: the various planning
documentation (e.g. CAD pictures etc.) must be done centrally, but the front-line experts
are far more efficient in summarizing the created knowledge to field-friendly
documentation and can effectively link the new documents to the existing mass of
documents.
However, the most important result in summarizing the content to the most critical one
and to have people from the actual front-line organization to develop these materials, is
the economy of labor in daily work situations. As often noted, engineers seek to learn
what is needed to do their job; there is no time to learn things completely. Thus the real
issue in workplaces is to cope effectively with real-life situations.
17 this case study is authored by Ari-Matti Auvinen (HCI Productions Oy)
based on the first hand personal experiences in working with KONE Elevators
Handbook for Quality Management of Peer Production
22
Case study V: Wikipedia18
According to their own presentation on their web site (see in more detail
http://www.wikipedia.org -based, free content
encyclopedia project. The name Wikipedia is a portmanteau (combination of words and
their meanings) of the words wiki (a type of collaborative Web site) and encyclopedia.
Wikipedia's articles provide links to guide the user to related pages with additional
information.
Wikipedia is written collaboratively by volunteers from all around the world. Since its
creation in 2001, Wikipedia has grown rapidly into one of the largest reference Web sites,
attracting at least 684 million visitors yearly by 2008. There are more than 75,000 active
contributors working on more than 10,000,000 articles in more than 250 languages. In
July 2008, there are 2,460,544 articles in English; every day hundreds of thousands of
visitors from around the world make tens of thousands of edits and create thousands of
new articles to enhance the knowledge held by the Wikipedia encyclopedia.
Visitors do not need specialized qualifications to contribute, since their primary role is to
write articles that cover existing knowledge; this means that people of all ages and
cultural and social backgrounds can write Wikipedia articles. Most of the articles can be
edited by anyone with access to the Internet, simply by clicking the edit this page link.
Anyone is welcome to add information, cross-references or citations, as long as they do
so within Wikipedia's editing policies and to an appropriate standard. Substandard or
disputed information is subject to removal. Users need not worry about accidentally
damaging Wikipedia when adding or improving information, as other editors are always
around to advise or correct obvious errors, and Wikipedia's software is carefully designed
Wikipedia is a new social innovation, and by using peer production there has been
created a new form of encyclopedia to be used in the Web. However, as the number of
articles and contributors has grown, also Wikipedia has had to alter its operative
mechanisms, including the creating norms on authoring and editing the articles,
developing the internal quality criteria and quality processes and modifications in its user
administration. In the various articles, Wikipedia also has warnings of the potential
incompleteness of articles, missing references etc.
The key issue in Wikipedia has been the eagerness of various people to share their
knowledge and information with oth
been strong and the huge majority of authors of Wikipedia are experts in their own area.
Wikipedia´s strength has also been the opportunity to comment and edit the content, and
thus also to add comments from different perspectives.
In admiring the success of Wikipedia, it should also be stressed that solutions, such as
Wikipedia, require also strong centralized resources these are e.g. data structure,
content structure, structure of quoting, search tools and search structure, various tools to
author and edit content etc. Fluent dispersed content creation requires a strong
centralized structure, which can enable various dispersed operations.
18 this case study has been authored by Ari-Matti Auvinen (HCI Productions
Oy) and is based on the Wikipedia site (http://www.wikipedia.org read on
17 July 2008), on Tapscott, D. Williams, A.: Wikinomics. Portfolio, USA
2006, and on personal experiences
Handbook for Quality Management of Peer Production
23
Case study VI: Sourceforge19
SourceForge.net is the world's largest Open Source software development web site,
hosting more than 100,000 projects and over 1,000,000 registered users with a
centralized resource for managing projects, issues, communications, and code.
SourceForge.net has the largest repository of Open Source code and applications
available on the Internet, and hosts more Open Source development products than any
other site or network worldwide. The essence of the Open Source development model is
the rapid creation of solutions within an open, collaborative environment.
Members of the Sourceforge community are responsible for developing the software that
provided by Sourceforge that they use in their work.
A project at Sourceforge may be started by a developer or a group with a software
project at one of several stages of development
- an idea for a Software project
- a proposal with a specification and development plan
- a functioning or partially functioning software application
- a fully functioning software title
In most cases the initiating developer wants to attract other developers to help with the
project and has chosen Sourceforge as the best place to do it. The vast majority of
projects are started by inexperienced developers, attract little or no interest from others
and after a brief flurry of activity, languish in the archives until the initiator finally loses
interest. At the other end of the scale a very few are extremely successful, with many
participants and thousands of downloads of the projects products by the public.
that is calculated from the overall popularity of the project amongst its participants and by
the public.
Sourceforge consists of a large number of projects. Anybody can start a project and
begin using the tools provided. There is no charge for the service; it is funded by
advertising, sponsorship and user donations. When a project is started it is immediately
provided with a complete set of tools to begin work. Each project on Sourceforge has the
same set of tools.
During the development of a software project, a great deal of existing knowledge is
utilized and new knowledge created. All software is built upon a foundation of existing
software so a comprehensive understanding of the connecting modules and system
within which it operates is vital for a project to succeed.
Types of knowledge utilized or created in a software project:
- technical documentation of related or connected software
- books & articles describing techniques or processes
- archived discussions from previous related projects
- source code
- help files & FAQs
19 this case study has been authored by Michela Moretti (Scienter Italy) based
on the materials by Marcus Clements (Brighton Art, United Kingom) within
the framework of Praxis project
Handbook for Quality Management of Peer Production
24
Everyone understands the value of archiving information. We only need to be asked the
make it easily found by the next person with that question. Thus elementary in
Sourceforge are the archived discussions (nearly all important decisions made by
developers collaborating on software projects will be made in some kind of archived
discussion, be it archived email, a forum or a news group) and documentation (as a
project continues the consensus of opinion and accumulated knowledge that is
represented in the archives is compiled into a more formal structure in the project
documentation). However, also cross-referencing and linking are essential elements in
the successful projects.
The success of Sourceforge shows clearly both the demand in the open-source area for
such a joint workspace, which enables communication among peers in often very
complex issues. However, also the provision of effective tools for all various actors within
the Sourceforge is one critical element of success.
Case study VII: Wiki for Health Care Professionals (~ ‘Pflege-Wiki’)20
-
English) Wiki project for Health Care Professionals and especially for nurses. The Wiki
(German version21) started in August 2004 as a private initiative and is run since July
the services via member fees and donations. The English version22 was added in May
2006; in March 2008, collaboration with the Italian NursePedia23 was established.
At the moment, the German version contains more than 4.400 articles free for use and
distribution in the sections:
health care in practice (with subchapters like geriatric care, sick-nursing, paediatric
nursing, etc.)
terminology pathology
vocational education and training
management & pedagogy for the health care sector
academic studies
professional associations; press releases
links, forums, chat
care facilities, ambulant care services
book reviews, events, podcast.
- - like typical Wikis - offers the possibility to read, write and modify
articles about subjects relevant to the health care sector. Each contribution is reviewed
by the other users of the service, so that each article is produced collaboratively.
-
history of an article / a concept visible. In rare cases it is used to re-establish an article
20 this case study has been authored by Thomas Fischer and Thomas
Kretschmer (ILI)
21 see http://www.pflegewiki.de
22 see http://en.nursingwiki.org
23 see http://www.nursepedia.net
Handbook for Quality Management of Peer Production
25
-
to add photos, to link the Wiki to personal and official websites and to distribute the
-
under the GNU General Public License and can be used and copied without any
restrictions24.
Each article as well as each registered user has a discussion forum, where authors and
users can get in contact or clarify certain aspects of an article. Registration is not
necessary, but offers certain advantages (e.g. each article is signed with the nickname of
the author; registered users have a personal page for self-presentation; registered user
observed pages).
-nterviews
with experts concerning actual problems and discussions).
24 for more details please see
http://en.wikipedia.org/wiki/GNU_General_Public_License
Handbook for Quality Management of Peer Production
26
Action point
You have now been acquainted with seven different case studies of peer
produced content. In the discussion of quality, two different aspects have often
been addressed. The first one is functional quality, which in educational context
addresses the issues of meeting the objectives and the rightful content. The
second aspect is technical quality, which in educational contexts addresses the
issues of technical fluency (e.g. appropriate presentation, right spelling, adequate
linking etc.).
Based on the presented case studies, you should now define the mechanisms
the various presented cases utilized in their urge for functional quality and
technical quality.
Functional quality measures
Handbook for Quality Management of Peer Production
27
Technical quality measures
Handbook for Quality Management of Peer Production
28
Based on the discussion in this section as well as the cases presented, we can
present a simple classification of the various critical dimensions of peer
production in eLearning and related fields.
The two critical dimensions are
setting of objectives who is setting the objectives for the peer-produced
content: is it controlled and unified for all potential peer producers (e.g. a
an article of my personal interest)
structure what structure is given: is the structure firm (e.g. composed of
certain elements, tools and requirements) or the structure loose and not
pre-defined (e.g. peers can use tools and structure of their own choice).
This is summarized in picture 5.
Picture 5: Contexts of Peer Production
Handbook for Quality Management of Peer Production
29
Action point
In the following you should address two issues: Firstly you should define to which
quadrant of the picture 5 do the various cases belong to. Secondly, based on
your understanding and experience, discuss what are the challenges of quality of
peer-produced eLearning in the different modalities (descibed by the different
quadrants).
Handbook for Quality Management of Peer Production
30
4 Approaches and methodologies for peer
production
In this section we will give an overview of existing quality development tools and
approaches for the field of peer-production. After a short introduction we give
some background knowledge on quality for peer-production, and work out
changes which occur due to the shift from expert models of quality to more
learner and peer-oriented models of quality development.
The quality of eLearning has been discussed and researched in many European
projects as well as in many international contexts. Ehlers and Pawlowski
describe, in the discussion on quality of eLearning, one can distinguish between
three different aspects in the discussion, namely
- different interpretations of quality
- different stakeholders with different perspectives on quality
- different forms of quality (input-quality, process-quality, output-quality).
This discussion has also lead to different interpretations of quality and
numerous definitions from various fields are available.25 As in particular the wide
discussion on quality in eLearning in general is well captured by the recent
comprehensive book edited by Ehlers and Pawlowski26, in this context it is not
necessary to repeat this discussion, but rather concentrate on the key issues
which are altering in the peer production of eLearning.
In the peer production of eLearning the essential feature is that the learners are
also acting as creators of the content in the new learning settings the
sep
learners are no longer purely consumers but they actively participate in the
learning process and thus influence it. As the borders between user and author
are blurring, so do the roles of student and teacher.
This fundamental feature is also imposing a different view on quality, as quality is
often to be defined and assessed by the same group of actors as the actual
creation of the learning content. However, the quality approach to peer
production can be more than just an emphasis on self-evaluation and its
practices.
Many quality approaches also in eLearning rely on the conventional quality cycle.
This quality cycle has included since the writings of W. Edwards Deming in the
1950s - the steps of PDCA (Plan, Do, Check, Act). This approach has been
modified during the last decades in many different ways, and also applied into
25 Ehlers, U-D. Pawlowski, J.: Quality in European eLearning: An
introduction. In Ehlers, U-D. Pawlowski, J. (eds.): Handbook on Quality and
Standardisation in ELearning. Springer 2006.
26 see Ehlers, U-D. Pawlowski, J. (eds.): Handbook on Quality and
Standardisation in ELearning. Springer 2006.
Handbook for Quality Management of Peer Production
31
the area of vocational training. Wirth has presented that in eLearning the
essential steps could be:
- plan
- do
- check
- compare.27
As a hypothesis we can claim, that in the development of a quality approach to
issues in other words: which processes implemented are assisting the quality
of peer produced eLearning content. In addition, in regarding the quality
processes, the peer production of the learning materials in their quality
assessment can also be linked with benchmarking or even more precisely, on
4.1 Quality development of peer production28
describes a number of developments, trends and points of view, which require
change from teaching to learning. The new point of view essentially connects
eLearning with five characteristics:
Learning takes place always and everywhere (ubiquitous) and therefore
in many different contexts, not only in the classroom.
Learners take on the role of organizers.
Learning is a life-long process, has many episodes and is not (only)
linked to educational institutions.
Learning takes place in communities of learning (so called communities of
practice. Learners participate in formal, as well as informal communities.
Learning is informal and non-formal, takes place at home, at the work
place and during leisure time and is no longer centered on teachers or
institutions.
In the new environment, learners are highly self-directed, as learning does not
life in a number of different episodes, in learning communities and social
networks, using social software and individually compiled contents. Securing and
developing quality in such learning scenarios thus has to focus mainly on the
individual learning processes and the shown achievements (performance). The
or the co called input factors. Quality assessment does not take place by using
classical methods of expert- and standard based quality management, quality
assurance or control, but by making use of more participative methods and
27 see Wirth 2006
28 this chapter is based largely on Ehlers 2009 (Web 2.0 E-Learning 2.0
Quality 2.0? Quality for new Learning Cultures)
Handbook for Quality Management of Peer Production
32
responsive designs. The aim of the process is to reach an individualized
assessment, which relates to the learning process.
Initially it seems paradox to talk about the quality of eLearning 2.0, as quality is
often linked with checking by externally imposed standards. However, quality can
also be understood in a development-oriented way, which means the enabling of
learners to develop themselves in their own learning processes and
consequently reach better results as far as quality is concerned. In this view,
methods of self-evaluation, reflection and peer-evaluation are seen as more
important. This kind of quality methodology does not have anything to do with
normative, universally valid, but aims at improving the quality of the learning
process.
In eLearning 2.0 learning approach, the learner has an important role as active
constructor of learning materials (co-creator), personal learning environments
and initiator of his or her own learning processes. Interestingly, this is a
characteristic, which is often felt to be a barrier for integrating eLearning 2.0 into
formal educational processes. This is because the competition of learners and
teachers and/or other institutional actors during quality assessment seems to be
insurmountable and only resolvable through a loss of power for the institution.
Action point
One of the key basic requirements for appropriate peer production are the
motivated, self-directed learners. They also play an important role in the quality
approach in peer production. In the following, discuss how in your organization
you can strengthen and enhance the self-directness of the learners and discuss
also the role of the learners in the quality approach.
Handbook for Quality Management of Peer Production
33
Action point
Another key basic requirements for appropriate peer production are motivated,
self-directed teachers and learning material developers, who could join their
efforts. In addition, in many organizations also SMEs (subject matter experts)
can be sharing their valuable knowledge by the methods of peer production. In
the following, discuss how in your organization you can strengthen and enhance
the motivation and self-directness of teachers, learning material developers and
subject matter experts and discuss also their role in the quality approach.
4.2 Conditions for quality of peer production29
As a point of departure, eLearning 2.0 does not require a new mode of thinking
or method of quality development, such as a new and completely altered
philosophy of quality
basic conditions and contexts need to be taken into account. Doing justice to
these different contexts, different questions need to be posed when dealing with
quality development, different objects evaluated, different criteria of quality
applied and specific methods of quality assurance, enhancement and
development used. In short: the role of quality development is changing.
While in traditional learning scenarios it mostly means the checking and
controlling of quality, in eLearning 2.0 it is becoming more the role of an enabler
of learning progress. Learning methods and quality development are moving
closer together. Methods such as feedback, reflection and recommendation
mechanisms are becoming more important. Typical basic conditions, which need
to be taken into account in quality development for eLearning 2.0 scenarios, are
explained in the following:
29 this chapter is based on Ehlers 2009 (Web 2.0 E-Learning 2.0 Quality
2.0? Quality for new Learning Cultures)
Handbook for Quality Management of Peer Production
34
From reception to participation: the metaphor used for learning is changing.
In eLearning 2.0, quality cannot be tied to the evaluation of a pre-determined
learning environment or learning contents produced by an expert. Not the
reception but the active participation is most important, that means the question
in how far a learning scenario stimulates the creation of individual personal
learning environments, the compilation of individual learning contents and
sharing them with others.
Action point
As the metaphor for learning is changing from reception to participation, discuss
in the following what does that imply for the quality approach in peer production
of eLearning and what measures must be planned for and implemented to take
advantage in this metaphor change.
From inspection to reflection: quality development for eLearning 2.0 shifts
the focus from conformity to a reflection of the learning process. Learners are
supported in reflecting, recognizing and putting into effect their own learning
progress, educational strategies, needs, etc. and in the course of their actions
critically reflect the contribution of educational media. The aim is to achieve a
personally ideal configuration of educational media and strategies, which is
continuously developed through autonomous reflection.
Handbook for Quality Management of Peer Production
35
Action point
Another important shift is the quality development alteration towards learning
from the reflection of the learning process. Discuss in the following what
measures must be planned for and implemented to take advantage in this
change.
From product orientation through process orientation to
performance and competence orientation: the material that is used for
learning and the processes of its supplier are not the focus of quality
their individually developed learning products, steps in development and
similar aspects (for example in e-portfolios), which shape their way to
decision-making and responsibility.
Action point
The focus of quality development is shifting from “product orientation” towards
“performance and competence orientation”. Discuss in the following what
consequences this change has in the quality approach of your organization as
well as in the quality approach of peer production in eLearning.
From planning education for the leaner to planning education by
the learner: quality of learning scenarios is often attempted to be achieved
through careful analysis of the need for education, a comprehensive
conception phase, feedback as far as the design of learning material and
Handbook for Quality Management of Peer Production
36
development processes are concerned and the evaluation of learning
processes. In eLearning 2.0, many of these processes shift from the supplier
of a program to the learner. Quality concepts must therefore support the
learners in their ability to develop quality through reflection, enable learner-
oriented forms of evaluation and offer the necessary tools for quality
development to the learners in their PLEs.
Action point
As stated previously, in eLearning 2.0 many key processes shift from the
supplier of a program to the learner. Also the quality approach must therefore
support the learners in their ability to develop quality through reflection.
Discuss in the following what consequences this change has in the quality
approach of your organization as well as in the quality approach of peer
production in eLearning.
From receiver to developer of learning materials: quality assessment
in eLearning 2.0 scenarios does not follow the logic of a marketing
effectiveness research to find out how the materials and characteristics of
media optimally affect the learning process. It is not about learning process
taking part in a unified learning scenario. Rather, the focus lies on processes
of development, flexible usage and the validation of social communication
processes with other learners.
Handbook for Quality Management of Peer Production
37
Action point
In eLearning 2.0 the emphasis is on learners as key developers of learning
materials. This sets also new requirements for the quality approach of
organizations. Discuss in the following what consequences this change has in
the quality approach of your organization as well as in the quality approach of
peer production in eLearning.
From the “learning island” LMS to the internet as a learning
environment: eLearning 2.0 approach understand Learning Management
Systems (LMS) as a mere starting point, as a signpost for their own search
and use of material from the internet, their development and linking to other
tools which can be flexibly arranged to become personal learning portals.
Quality assessment then does not focus on materials from the LMS anymore
but rather on the learning products and perhaps on the learning processes
documented in an e-portfolio.
Action point
In eLearning 2.0 the emphasis is also moving from internal, proprietary learning
management systems and “learning material reservoirs” towards the Internet as
a learning environment. Discuss in the following what consequences this change
has in the quality approach of your organization as well as in the quality
approach of peer production in eLearning.
Handbook for Quality Management of Peer Production
38
From tests to performance: learning progress and achievements
become visible not only in tests but rather in the learning process
documented in portfolios (for example in wikis or web logs), learning products
and social interactions.
Action point
In eLearning 2.0, learning progress and achievements become more visible and
the assessment of learning shifts from tests to wider assessment of performance.
Discuss in the following what consequences this change has in the quality
approach of your organization as well as in the quality approach of peer
production in eLearning.
Handbook for Quality Management of Peer Production
39
4.3 Concepts and methods of quality
development for eLearning 2.030
Quality assessment of eLearning 2.0 focuses on the learning process. There is
no use of external standards and inter-individual comparisons (such as tests or
assessments). Rather, methods of self-evaluation, intra-individual development
processes are employed for this purpose, which are not made via tests but via
reflection and evaluation of learning products and e-portfolios. Even though
eLearning 2.0 is a new development as a trend, substantial experiences have
already been made with the learning models of autonomous learning and
learning in communities, which are the basis for it, as well as with methods for
quality assessment of learning processes.
Teachers can use these methods in order to evaluate the learning progress
together with students and to enable individual planning. Teachers take on the
role of mentor who gives feedback and helps with reflecting the learning
experiences or evaluates e-portfolio postings.
Table 1: Methods of quality development for eLearning 2.0 (Ehlers 2009)
Methods of quality
development
Quality assessment by
Self-evaluation
Learners with the help of/ feedback by
teachers
Assessment of e-portfolios
Teachers
Social recommendation
Peers, learning communities
Evaluations aimed at target
group
Teachers
In the following section, important aspects of methods for quality assessment,
which are listed in table 1, are presented.
4.3.1 Self-evaluation
One important approach, which contains enormous potential for quality
assessment of learning processes in eLearning 2.0 scenarios, is the concept of
self-evaluation. The aim of it is not a complete (summative) assessment of
learning achievement, but rather an improvement of learning abilities.
30 this chapter is based largely on Ehlers 2009 (Web 2.0 E-Learning 2.0
Quality 2.0? Quality for new Learning Cultures)
Handbook for Quality Management of Peer Production
40
“Self-evaluation is defined as students judging the quality of their work,
based on evidence and explicit criteria, for the purpose of doing better work
in the future. When we teach students how to assess their own progress,
and when they do so against known and challenging quality standards, we
find that there is a lot to gain. Self-evaluation is a potentially powerful
technique because of its impact on student performance through enhanced
self-efficacy and increased intrinsic motivation. Evidence about the positive
effect of self-evaluation on student performance is particularly convincing
for difficult tasks31 32, especially in academically oriented schools33 and
among high need pupils34.“
In scientific literature, positive effects of self-evaluating processes on the learning
achievements can be found (Maehr/Stallings 1972, Arter et al. 1994, Hughes et
al. 1985). When undertaking these processes, students can gain insights into the
profile of their own strengths and weaknesses. Rolheiser & Ross state that, if
students evaluate their own achievements positively, they aim for more
challenging objectives, engage in their own learning process more and mobilize
more personal resources35. A self-evaluating process follows the following four
steps:
Step 1: Learners are involved in the definition of the criteria that are used for
assessment. This happens in the form of negotiation. It has been shown that
neither pre-determined criteria nor criteria solely developed by students are as
effective as criteria that are developed together. Surveys show that criteria which
are development in cooperation with learners, enhance agreement and
motivation of the learners. Learners are also simultaneously coached in
developing of their own goals and make experiences when choosing the level of
difficulty. Furthermore, an attitude of advice develops between teacher and
learner, which can be of great significance in eLearning 2.0 learning processes.
Step 2: In this step, learners apply the criteria they have chosen to their own
learning processes. As they do so, it can be important to provide them with
examples for what such assessments can look like.
Step 3: In a third step, learners receive feedback on their self-evaluation. The
aim of this st
teachers by using this feedback process. A triangulation of their own
assessment, that of the teachers and that of the peers is taken into account.
Step 4: In step four, the students are asked to develop plans for developing
their own competences on the basis of their self-evaluation. They discuss
strategies with the teachers in order to reach these goals.
31 see Maehr and Stallings 1972
32 see Arter et al. 1994
33 see Hughes et al. 1985
34 see Henry 1994
35 see Rollheiser and Ross 2001
Handbook for Quality Management of Peer Production
41
Action point
Self-assessment can be a powerful tool in the quality management of peer
production. Regarding your own organization, assess in the following the benefits
and disadvantages of self-assessment and discuss what supportive measures
your organization should take in enhancing self-assessment by the learners.
4.3.2 Quality assessment with e-portfolios
E-portfolios web-based portfolios integrate different media and services.
Students collect those learning products in their e-portfolio, which are made in
the course of a class or even during the whole course of their studies. Students
can use electronic portfolio to show competences and reflect their learning
processes. Learning results, connected with remarks by tutors, teachers and
peers, feedbacks and personal reflections are collected.
E-portfolios lend -portfolios an
assessment of or for
2006). E-portfolios can used when making the final assessment (summative) or
for continuous improvements (formative). As can be seen in table 6, purpose,
design and contents of portfolios are clearly different when used for summative
assessment of the learning achievement or for formative assessment in order to
support the learners.
Handbook for Quality Management of Peer Production
42
Table 2: Purposes of an e-portfolio for assessment (based on Hornung-
Prähäuser et al. 2007)
Portfolio for summative
assessment
Portfolio for formative
assessment
Purpose of e-portfolio is prescribed
The purposes of the portfolio are
negotiated with the learner
It is fixed which learning products
have to be part of the e-portfolio so
assessment if possible
Artefacts have been chosen by the
learner to tell the history of his or
her learning process
Portfolios are usually fabricated at
the end of a school term, semester
or program and there is a deadline
for handing them in
The portfolios are constantly
updated over the course of a school
term, semester or program with
flexible timing
The portfolios and/ or artefacts are
generally graded bases on a matrix
and quantitative data for an
external audience
The portfolios and artefacts are
evaluated together with the learner
and are used to give feedback, so
that the learner can improve his or
her learning process
The portfolio is normally structure
by specified results, aims or
standards
The organization of the portfolio has
been determined by the learner or
has been negotiated together with
the mentor/ advisor/ teacher.
Sometimes the portfolios are used
to make important decisions
The portfolios are hardly ever used
to make important decisions
Summative: what has been learned
up to now) (past present)
Formative: Which needs for learning
will exist in future? (Present
future)
Extrinsic motivation is necessary
Intrinsic motivation mobilizes the
learner
Audience: external, little possibility
for choice
Audience: learners, family, friends
Working with the portfolio has a double function. On the one hand, it is an
innovative instrument for teaching and learning; on the other hand it serves as an
alternative instrument for assessment. Learning scenarios supported by e-
portfolios emphasize the learning process and enable a deeper understanding of
learning processes in all participants.
Concerning quality assessment, the portfolio is understood as a way from
achievement diagnosis, which is exclusively defined externally and test-oriented,
to a more strongly self-directed achievement presentation by the learners. E-
portfolios are aimed at competences. The idea is not to emphasize the mistakes
the learner has made but what they are capable of doing. Advocates of portfolios
often stress the natural function of a portfolio for bridging purposes, i.e. the link it
Handbook for Quality Management of Peer Production
43
creates between teaching, learning and evaluating36. Thus, an e-portfolio is a
method of evaluating achievements, which offers a combination of external and
self-evaluation. Table 2 presents an overview of qualities for assessment
oriented at e-portfolios in comparison to online examinations. In this process, e-
portfolios can be used for evaluation/ assessment of subject-related abilities as
well as self-competence. If e-portfolios are used as an instrument for assessing
learners, the following aspects have to be taken into consideration:
The new way of learning, presenting and refection requires mentoring and
E-portfolios are an instrument of development rather than for checking
A qualitative assessment supports the learner-oriented, customized
approach to prove achievements.
The high level of subjectivity when it comes to the evaluation decreases
when there are several evaluators (see also peer-review).
It needs to be clarified beforehand in which way data will be exchanged
and published.
Action point
E-portfolios integrate different media and services, and thus provide a
comprehensive view on the competences of the learners. However, in addition to
self-assessment, e-portfolios provide a unique view to the capabilities of the
learners. Regarding your own organization, assess in the following the benefits
and disadvantages of e-portfolios and discuss what supportive measures your
organization should take in enhancing the use of e-portfolios.
36 see Häcker 2005
Handbook for Quality Management of Peer Production
44
4.3.3 Social Recommendation and Community
Participation
In eLearning 2.0 learning scenarios, communication, feedback and the exchange
within learning communities is essential. With the help of social software tools
collaborations can be conducted and information exchanged, as well as
evaluated mutually. Three methods are of special significance and first
experiences have been made:
- Social recommendation mechanisms
- Peer-review method
- Peer-assist method
Social recommendation mechanisms are defined as those methods that serve
material in contrast to
methods focused on experts37. According to this method, the members of a
learning community evaluate materials available online. This happens for
instance in databases for learning material which contain an assessment of
saved learning materials on the basis of their usefulness and quality, or in a
less structured form through learners creating link lists of materials, classes
and resources available online which they deem especially valuable and
qualitative.
course of which each learning material is assessed by learners. On the other
hand it is also possible to give learners recommendations á la Amazon on
which learning material is thought to be especially useful, so called social
recommendations. Eric Duval, a Belgian professor, suggests a concept he terms
c
for learning recommendations. Of course, this does not guarantee that one finds
the right text, but it increases the probability to find useful contents.
37 see Duval 2006
Handbook for Quality Management of Peer Production
45
Action point
Consider your own organization and its eLearning. In which areas could you use
social recommendation mechanisms? What are the benefits and disadvantages
of social recommendation mechanisms?
Peer review is a concept that has been introduced a number of times, especially
in the academic sector. It deals with assessing quality by peers that is
colleagues or other learners giving each other feedback. In the scientific sector,
the texts discussed are often scientific proposals or publications. In the area of
learning, especially in eLearning 2.0 scenarios, the peer-review can be used to
attain feedback and quality assurance for results, learning progress and aims,
which is given from other learners or members of the learning community. A
simple application of the peer-review method for the purpose of quality
enhancement in eLearning 2.0 scenarios is to invite different learning
communities or members of different learning communities, to present them with
the learning intentions, progress and the problems as well as solutions worked
on and to ask them to do a review.
Peer reflection is a process aimed at creating situations for reflecting, in which
the peers are asked to encourage the reflection of learning processes by means
of their own experiences. One community could, for instance, share with another
one how it structure their projects, why they used the material they used and so
on.
One possibility to check on the quality of learning processes is learning from
One model that has recently been gaining more importance is the peer assist
model38. It is a structured reflection in the context of a social network, which is
carried out via social software. This method is clearly distinct from peer review
(see table 3). It primary aim is to simulate learning processes. By employing the
38 The models of peer assist, peer learning and bench learning are so to speak
the logic continuation of peer review processes.
Handbook for Quality Management of Peer Production
46
method for eLearning 2.0 scenarios, social assets are used for further developing
learning process. Structured reflection of a learning process is possibly by
broaching the issue of the learning processes, the results and documented
outcomes in the peer assist process.
Table 3: Differences between peer review and peer assist processes (based
on Commonknowledge 2007)
Peer review
Peer assist
aim: evaluation
aim: learning, improving knowledge
Evaluative
collaborative
The task is to criticize a paper
The task is to learn with and through a
team.
Reviewers are chosen by others.
The members themselves choose the
assistants.
Often, there is an attempt to reach
constructive and in every case
processes for solving problems.
Some actors are always reviewers.
That who assist today can call for a peer
assist process tomorrow changing
Report is mostly made available for
the management.
The process is aimed only at those who
called for it.
The peer assist process is a structured process that can be employed in
eLearning 2.0 scenarios by using social software. It deals with linking and
strengthening a
strategies for problem solving and learning approaches, reflecting and improving.
Table 4 shows how peer assist can be used in eLearning 2.0 scenarios.
Handbook for Quality Management of Peer Production
47
Table 4: Online peer assist processes (Ehlers 2009)
Phase
WEB (2.0) tools
Preparation
introductions to (six) peer
assistants
E-mail
A peer assist moderator
needs to be found and
invited.
E-mail
A peer assist wiki or blog
needs to be made.
WIKI, blog, protopage, etc.
Execution
Round 1: presentation of
problem (10 min.)
Notes on peer assist wiki/ blog/ application
sharing, collaboration platform
Examples: A concept developed by learners
for the purpose of problem solving is
presented in form of notes, a concept for a
(final) paper etc. is briefly presented, a
problem is presented
Round 2: peer assistants
can ask topical questions
(30 min.)
Online chat, collaboration platform
Round 3: peer assistants
make suggestions for
solutions and give
evaluations (45 min.)
Suggestions are entered into a forum for
discussion, everyone reads the other
Round 4: moderator
invites all participants to
give a finishing suggestion
(30 min.)
Final round as a post in a forum for
discussion.
Round 5: peer assistee
decides on how to continue
and informs the group (10
Min.)
The peer assistee informs participants in an
online chat which suggestion he o she has
chosen.
Handbook for Quality Management of Peer Production
48
Action point
Consider your own organization and its eLearning. In which areas could you use
peer reviews and peer assist methods? What are the benefits and disadvantages
of peer reviews and peer assist methods?
4.3.4 Evaluation processes aimed at a target group
Today, evaluation is often used for assessing learning processes and results. A
large number of contributions in scientific and praxis-related literature with
processes that have turned out to be successful can nowadays be drawn upon.
In the educational sector, the current practice for evaluation envisages a group
evaluating a learning/teaching situation with the help of an evaluating instrument
(e.g. a questionnaire). This is problematic in eLearning 2.0 scenarios as the
learning progressions and PLEs are potentially different even in one and the
same class. That is why, as regards evaluation, it makes sense to resort to a
practice of evaluation strongly aimed at the target group.
This can happen by admitting a target-group-specific profile of evaluation
instruments. One way of doing so is for example asking the learners to not only
answer questions from an evaluation questionnaire but simultaneously inquiring
into how important respectively relevant they deem this evaluation item for a
learning process. Is the item irrelevant to a learning process than it is evaluated
to be of small importance and is consequently also not taken into account as
estimation of dimensions that are unimportant to the learning progress are thus
avoided. Another advantage results from learners conducting not only an
assessment but at the same time a reflection on what was of significance to their
personal learning progress. The questionnaire, which should be used for such an
evaluation, should cover all relevant areas in an adequate manner.
A method like this, which is aimed at a certain target group, is conceptually close
to experiences that have been made in the area of responsive evaluation.
According to this method, participants do not only assess given objects but are
included in the definition of objects to be evaluated. In eLearning 2.0 scenarios,
a different questionnaire by assessing issues differently. The results of such an
Handbook for Quality Management of Peer Production
49
evaluation procedure cannot be processed and handled in the same way as
homogenous entity. In contrast, target-group-specific suggestions and solutions
for the occurring results the evaluation yields will have to be found.
Handbook for Quality Management of Peer Production
50
5 Technical tools and technologies for peer
production
In this chapter we summarize the technical tools and the technology which are
used in peer production. It is also necessary to define some key terms which are
closely related into the peer production concept. Although during the recent
years learning and information provision through Internet and its different
services have become one of the most popular themes in educational
development, it is good to remember both the relatively short history of the
“Internet age” as well as the utilization of some of the key concepts already
mentioned earlier in different contexts.
5.1 Technical tools and technologies in the Web 2.0 era
The key to the potential to use Internet and its services in vocational education
and training stems from the fact that the metaphor for using Internet has changed
active participation and information creation by the users (which may call the
39 and
40 the peer production has been recognized, not only as a
challenge the traditional education, but also creating a new valuable approach
into education - and particularly into the eLearning development.
41.
is not any new remarkable invention which has been the starting point for new
innovations in web-based tool development.
denominator for all intensive development in web-based communication. Thus it
is rather an evidence for achievement of a next stage in networking and
ncept which
sums up different development trends of web-based services, applications
standards, technologies and most of all how these all are used in social
communication and networking.
In education and training development and in implementing eLearning we also
can see the
and information flow tools are used for educational and learning purposes.
However, the (open) social networks and social media are critical for peer
production and its integration to traditional learning approaches. We understand
39
early years stems already from the beginning of the 1990s see e.g. Barrett
1994
40 on early stages of social networks see e.g. Harasim 1995
41 see in more detail O´Reilly, T.: What is Web 2.0? - Design Patterns and
Business Models for the Next Generation of Software
- http://www.oreilly.de/artikel/web20.html (read 24 July 2008)
Handbook for Quality Management of Peer Production
51
participation, information production and receiving, handling and providing it for
others to use. As discussed in previous chapters, the social media have been
essential in the success of user-
cessful peer production.
One important feature in the recent development is also that the creative power
of the users has been nurtured by the provision of different media for peer
production. Thus the users are not limited to text only they can also produce
easily and cheaply their content by other media, such as photos, images, audio
clips, video clips etc. Another important feature is that in addition to the
conventional computer networks, also novel networks such as mobile networks
can be widely used.
5.2 Technical tools for peer production
In the following we describe some contemporary approaches for peer production
these might often be called in the quick
new tools, approaches and services emerge
almost daily
also their commercial dimension, the important feature in the development of
seen as services, applications or distribution platforms, which provide platforms
for peer production. With the open source approach also the business models
and earning models alter however, open source does not mean free or totally
non-commercial.42
Blogs have been around since the mid 1990s when they were mainly presented
as personal diaries on various Internet home pages (the name stems from the
combination of web + log book). Today blogs have become among other things
- efficient tools to express opinions, to distribute information and also to serve as
commonly read personal diaries. Modern blogs can also easily mix different
media text, graphics, pictures, video clips, audio clips etc.
platforms by forming so called blogospheres where information can spread
extremely fast. The linking of different blogs has become a very important tool for
peer production particularly in cases where students themselves are encouraged
to become peer producers and at the same time be active on commenting blogs
of fellow classmates43 44. Blogs are often tagged with pointing out keywords or
they are listed according the titles which are then visible in different blogosphere
or blog account hosting sites also typical for blogging is regular updating which
ation to be invisible and history of the blogging is thus
difficult to follow.
42 see e.g. Goldman Gabriel 2005
43 on educational uses of blogs see e.g. Williams 2004
44 see also Richardson 2006
Handbook for Quality Management of Peer Production
52
Typical for blogging tools is that most of them are free for the users and thus the
threshold to start active blogging is very low.45
Wikis and other text based collaboration formats are describing a webpage
or set of web pages that can be easily edited by anyone who is allowed access
the most well-known of the wikis is Wikipedia46. The fundamental idea of wikis is
to provide information voluntarily, decentralized and openly. The information can
be added, corrected or totally new topics can be created without changing the
whole structure of the site. Information providers are also reviewers of the
information. Wikis provide also a way to common structured knowledge creation
and distribution and thus they can serve as an effective tool for learning.47
The quality of information has been staying rather solid in the various wikis -
despite of minor vandalism appearing time to time in wiki-based web sites. Today
is easy to create own wiki environment for various topics. There are so called
evelopment of wikis
also in universities and other educational institutions. In wikis the history of
information or individual page is always visible and that helps to keep provided
information valuable.48
Tagging and social bookmarking allow users to save their bookmarks online,
tag them and share them with others. In practice it is possible to install from web
(free of charge) programs which help you to store your bookmarks online, tag
them and share them with your colleagues and students. Use of tagging has
become common way to look for information by using keywords. By using the
tagging the user is able to select information wanted. The key to the applications
of social bookmarking is metadata (information on information), which enables
the sharing of this information. The probably best known social bookmarking
application is del.icio.us, in which users can tag each of their bookmarks with a
number of freely chosen keywords.49
Different tools used for tagging and social bookmarking help users to search and
identify information with keywords and topics and to save it all in own directory
with favorites/bookmarks online. In practice they are personal websites where
user can store archive their favorite web pages. The importance of tagging and
social bookmarking is in sharing the important links and information sources with
other users. This enables - instead of replicating the information content on
certain web sites - the sharing of the links and ensure the validity of the
information.50
45 see e.g. http://c4lpt.co.uk/Directory/Tools/blogging.html (read 24 July
2008) of the various blogging tools available
46 see http://www.wikipedia.org
47 see e.g. Parker Chao 2007
48 see e.g. http://c4lpt.co.uk/Directory/Tools/wiki.html (read 24 July 2008)
of the various wiki tools available
49 see http://del.icio.us
50 see e.g. http://c4lpt.co.uk/Directory/Tools/bookmarking.html (read 24 July
2008) of the various social bookmarking and tagging tools available
Handbook for Quality Management of Peer Production
53
Media sharing means the services which enable everyone interested to share
multimedia in the public. Most common example of this service is YouTube for
videos51. There are plenty of different sharing services for videos, photos and
podcasting.
Podcasting as a term were initiated with launching of iPod - a portable digital
audio player by Apple. Contemporary, the term is referring any software and
hardware combination that permits automatic downloading of audio files for
been the audio signal compression technology in particular, the MP3 standard.
In education the podcasting is a powerful approach as the students are familiar
with the underlying technology application. By combining the audio blogging with
rviews etc., it is possible to provide
convenient methods in delivering the educational content. Also the students can
easily add their own content using audio blogging and distribute those audio
blogs by podcasting.
Podcasting is becoming increasingly popular in education. Podcasts enable
students and teachers to share information with anyone at any time. An absent
student can download the podcast of the recorded lesson. It can be a tool for
teachers or administrators to communicate curriculum, assignments and other
information with parents and the community. Remarkably, a number of leading
US universities and colleges provide their selected lectures through the iTunes
University site hosted by Apple.52
51 see http://www.youtube.com
52 see http://www.apple.com/education/itunesu_mobilelearning/itunesu.html
(read 24 July 2008)
Handbook for Quality Management of Peer Production
54
Action point
Today there are available a number of good and tested web-based tools for peer
production. Important technical features according to the quality of peer
production are e.g. easy access to the tools, their technical reliability, and their
feasible economic costs. In many developed cases of peer production of
eLearning, several compatible technologies and tools have been used. In the
following assess the strengths and weaknesses of different tools for peer
production and also consider these tools according to the quality of peer
production.
Blogs
Wikis
Social tagging and bookmarking
Media sharing
Podcasting
Handbook for Quality Management of Peer Production
55
5.3 Newer Web 2.0 services
There has been quick development in a number of new services, which will also
change the environment of peer production in eLearning. This following summary
table is based on the publication by JISC Technology and Standards Watch from
February 2007 and it is authored by Paul Anderson.53 We have added the last
three areas from other sources to the table (see table 5) by Anderson.
Social networking
Professional and social networking sites that
facilitate meeting people, finding like minds,
sharing content uses ideas from harnessing
the power of the crowd, networks effect and
individual production/user generated content.
Aggregating
services
Gather information from diverse sources
across the Web and publish in one place.
Includes news and RSS feed aggregators and
tools that create a single webpages with all
your feeds and emails in one place uses idea
from individual production/user generated
content.
Collect and aggregate user data, user
uses ideas from the architecture of
participation, data on epic scale and power of
the crowd
Data “mash-ups”
Web services that pull together data from
different sources to create a new service (i.e.
aggregation and recommendation). Uses, for
example, ideas from data on epic scale and
openness of data-
Tracking and
filtering content
Services keep track of, filter, analyze and allow
search of the growing amount of Web 2.0
content from blogs, multimedia sharing
services etc. Uses ideas from e.g. data on epic
scale.
Collaboration
Collaborative reference works (like Wikipedia)
that are built using wiki-like software tools.
Uses ideas from harnessing the power of the
crowd.
Collaborative, Web-based project and work
group productivity tools. Uses architecture of
participation.
53 Anderson, P.: What is Web 2.0? - Ideas, technologies and implications for
education. JISC Technology and Standards Watch, Feb. 2007 at
http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf (read 24
July 2008)
Handbook for Quality Management of Peer Production
56
Replicable office-
style software in
the browser
Web-based desktop application/document
tools. Replicate desktop application. Based on
technological development.
Sources ideas or
work from the
crowd
Seek ideas, solutions to problems or get tasks
completed by outsourcing to users of the Web.
Uses the idea of power of the crowd.
Virtual worlds54
Online virtual environment where content is
created in the context of online game-like 3D
digital environment to which users subscribe.
Social tagging or
folksonomy55
Ad hoc classification scheme (tags) that web
users invent as they surf to categorize the
data they find online.
Peer production
news
Web sites combining social bookmarking,
blogging and syndication with a form of non-
hierarchical, democratic editorial control. News
stories and websites are submitted by users,
and they promoted to the front page through a
user-based ranking system.
Table 5: Description of new Web 2.0 technologies (Anderson)
The richness of newer services on the Web provides also novel opportunities to
utilize them to promote quality in peer production of eLearning. Social networking
can work as an important tool to identify interested people outside of the own
organization for peers to conduct peer reviews as well as to work in
receiving/
contents.
Aggregating services are providing easy-to-use services in collecting essential
content across various web sites. It easens benchmarking of existing learning
i.e.
finding out what other entities have published on the Internet. The data mash-ups
serve the same function, although with a different technical realisation.
The new collaboration tools and services provide fresh opportunities to work
across organizations. The various wikis are increasing, but also other
collaboration and groupware tools are gaining ground quickly. Thus practical
day-to-day work both within organizations as well as across organizations is easy
to organize. For the quality management of peer production this means
practically that the back-and-forth sending of files and difficult change tracking
can be replaced by the use of common collaborative tools and real time
maintenance of various documents.
It refers to common business applications online that are accessed with an
Internet browser, while the software and data are stored on the servers. The
loud computing customers generally do not own the physical
54 see e.g. Guest 2007
55 see e.g. Pettenenati Cigognini 2007
Handbook for Quality Management of Peer Production
57
infrastructure rather they are renting the usage from a third-party provider
(such as Google, Microsoft or Yahoo). The users consume resources as a
service and pay only for resources that they use; however, individual users are
not often directly paying for the services, but accept e.g. advertisements in the
services used. Thus also the replicable office-style software will be used with an
Internet browser, although the files are saved on a server administrated by the
service provider. Importantly for peer production, the sharing of files and content
becomes easier and more fluent, if the users want to share their conetnt with
their colleagues.
Also virtual worlds (such as SecondLife) are creating novel opportunities for peer
production of eLearning and its quality assurance. While the content is displayed
in virtual worlds easily accessed by the users, also the feedback from the users
can be organized fluently.
Social tagging is basically sharing your tags with other users. Social tagging can
be linked with the mechanisms of social recommendation, but in social tagging
the users can also share their sources of knowledge and information with other
users. The peer production news are news created by peers, and thus they also
can provide an important element in keeping the various users on track of the
latest developments.
In all, many of the newer Web 2.0 services are still young (and some still in their
infancy in late 2009), but many of them provide easy-to-use tools to the quality
work, especially in areas of fluent document sharing and routing as well as
collaboration both within the organization as well as across organizations.
Action point
The amount of new tools for peer production are increasing with a fast speed.
Above we have listed some emerging technologies. In the following, explore your
experiences (if you have gained any) of these new tools and discuss how they
could be assisting the task of quality management of peer production of
eLearning in your organization.
Handbook for Quality Management of Peer Production
58
6 Enabling peer production
In this chapter we discuss the important enablers of peer production of
eLearning. As we discuss in detail later in this chapter, using peer production can
be seen and should be seen as a strategic choice in organizing learning. Thus it
must also be supported accordingly within the organization.
Peer production of eLearning requires various enabling and supporting
structures. Peer production does not just happen, it can be and it must be
planned and supported. We should remember that the quality approach of peer
produced contents in various environments is supported by a number of
compatible features.
Consider Wikipedia (see also the case study provided in the case study section)
Wikipedia enables easy creation of new entries and easy editing of the existing
content. However, we should also recall that the Wikipedia is based on a well-
defined database (or in fact many databases) with appropriate features, such as
user account administration, content routing (including navigation), content
indexing, search engines etc. Wikipedia provides also a web-based text editor,
which provides also spell checking and other word processing features. It is
technically possible to attach annexes, create links (most links are created
automatically), create tags for contents etc. And finally: there is a clearly defined
structure how to present an article and a clear route from the creation of an
article to the publishing and updating of an article.56
Or consider the Finnish Tax Academy (see their case in the case study section).
Their management ensures that the peers producing eLearning content can use
their working time 20 person hours per one hour of learning materials. Also in
their annual job definitions is included the task of producing learning content for
sharing it justifies these peer producers to use their working hours for content
creation. Importantly, the Finnish Tax Administration does not separately
compensate peer production of learning materials, but wants to see it as an
important part of everyday work of the subject matter experts.
Or consider Xerox and their Eureka application (see their case in the case study
section). They provided a clear toolset with a clear procedure to empower the
maintenance technicians in the field organization to produce their practical
content. They systematized the process of content creation by the provision of a
clear structure and a technical interface more importantly, they also provided a
clear route of the peer-produced content from creation to updating and enabled
the participation of the field maintenance technicians to this process by ratings
etc. However, Xerox was providing an appropriate database application and
ensuring the access to application (both for the content creation as well as the
utilization of the content) for its field organization.
56 of Wikipedia see in more detail Lih 2009
Handbook for Quality Management of Peer Production
59
These cases show that quality of peer production can and must be planned. The
approaches might vary according to the level of freedom in the structure and in
the setting of objectives. In the following, we have defined the enablers of quality
in peer production of eLearning to consist of the following aspects:
- enabling policies and procedures
- enabling processes
- enabling tools.
6.1 Enabling policies and procedures
Although the use of peer production in eLearning might be an appropriate
pragmatic way to develop eLearning in an organization, it should also be
supported on a strategic level by the management of the organization. Naturally,
we can see in organizations different levels of managerial support: the
management measures can allow a working form or they can actively support
and promote a working form.
The organizational support to peer production of eLearning can take place by
several different methods. For instance, these can include the following:
- active management support in promoting peer production as a Web 2.0
approach
- use of working time and other resources of the organization for peer
production (this aspect is mainly covering the teachers and supporters),
including clear guidelines for the compensation policies and practices
- pedagogical support for peer production, including clear procedures for
peer reviews and other joint working methods
- rewarding peer production are peer produced outcomes rewarded in a
similar way than conventionally produced outcomes (e.g. are learners
accredited for peer production of eLearning, are teachers rewarded for
peer production of eLearning)
- access to various (digital) resources , including also assistance in the
questions of Intellectual Property Rights (IPR).
Handbook for Quality Management of Peer Production
60
Action point
In the following, assess you own organization’s policies and procedures for peer
production and also list recommendations to promote policies and procedures to
promote peer production of eLearning.
Existing policies and procedures
for peer production
Recommendations for policies and
procedures
6.2 Enabling processes
We discussed above the enabling policies and procedures, which build an
organizational basis to enhance the use of peer production in eLearning.
However, these policies and procedures must be also supported by various day-
to-day working processes.
The practical support processes by the organization can be many fold. Examples
of various enabling supportive processes can include, among others, the
following:
-
learning experiences and good practices
- training the key actors in the peer production approach within the
organization
- facilitation of active communication and experience sharing between the
key actors
- support for intraorganizational and interorganizational work in the area of
peer production
- practical support to peer production by the provision of appropriate tools.
Handbook for Quality Management of Peer Production
61
Action point
In the following, assess you own organization’s processes for peer production
and also list recommendations to promote policies and procedures to promote
peer production of eLearning.
Existing enabling processes for
peer production
Recommendations for improved
enabling processes
6.3 Enabling tools
The number of various tools for peer production has been quickly growing
many of us use tools, such as wikis, blogs, collaborative working spaces, file
sharing etc. However, for an organization it is important to ensure that it
promotes tools to which all have access and which can be used easily by as
wide an audience as pos
Thus the organizational challenge is not whether you are using the newest and
most advanced tools, but rather that the tools you are using can serve in the best
possible way your actual goals.
Unfortunately, often also the problems with the compatibility of the tools create
unnecessary thresholds for many to participate in peer production. This might
also mean that the technical support staff and the IT support staff are reluctant in
supporting the peer production modality, as they fear that their work will increase
with the technical support staff. In particular in large companies and
governmental agencies there might be also clear restrictions to use many
applications of not, this must be taken into
consideration in the design of your activities.
In discussing the enabling tools, the following aspects, among others, should be
considered:
- access to the tools used: do all potential users have an easy access
- technical features of the tools: are the tools easy-to-use
Handbook for Quality Management of Peer Production
62
- financial impact of the tools: are the tools provided free-of-charge or are
there economical limitations for use
- data security and Intellectual Property Rights: are the tools provided
ensuring data security and are their IPR policies clear and acceptable
- required user support: is user support required and how is it organized
- longevity of the tools: do we expect that the tools are available in the
foreseeable future?
Action point
In the following, assess you own organization’s tools for peer production and also
list recommendations to additional tools to be used or tools to be replaced to
promote peer production of eLearning.
Existing tools for peer production
Recommendations for improved
tools or additional tools
Handbook for Quality Management of Peer Production
63
7 The QMPP quality approach
In this chapter we describe the QMPP quality model which has been evolving
and developing based on our field work both with the four different practical pilot
projects as well as with our discussions with the experts in different expert
panels. We also introduce four case studies undertaken during the QMPP project
and discuss their quality factors.
7.1 Basis for the QMPP quality approach
The quality management challenge in eLearning content produced by peer
production can, however, undermine the merits of this approach and method.
The quality work methodology in peer production is at its best dispersed and
fragmented. Often it has also been claimed, that the very nature of peer
production is its free flow and thus any formal mechanism (including the quality
approach) would be drastically against the creativity factor. At the moment there
are already a number of useful tools and approaches used (such as tools for
peer reviews, tools for creating own wikis, dictionaries etc.) to ensure and
improve the quality of peer produced eLearning content.
The importance of peer production of eLearning content will grow especially in
the sector of vocational education and training as well as in professional
continuing education. Many organizations face challenges of shortening life-cycle
of learning content as well as operational challenges in providing required
learning content with short lead times and lower costs. However, it is also
understood that the learning content produced by peers (based on professional
learning content by external e-learning experts.
Peer production has great potential in the area of vocational education and
training. The future workforce in Europe in many professions has not only to
access and handle great amounts of information and knowledge, but even more
importantly to produce various elements of information by themselves as an
integral part of their work. Peer production is not only a novel method to produce
eLearning content, but it is also an approach to empower a wide variety of
professionals to the learning content production. Thus it has also an important
democratic element in bringing the work-related learning content production to
the actual level of users, tutors and learning supporters.
Handbook for Quality Management of Peer Production
64
7.2 QMPP Qualityscape
The fundamental finding in our work on quality of peer production is that quality is
created as an interplay between peer production of digital content and peer
validation processes. Naturally peers have different roles at different times they
can participate in the quality process as creators, but their role is essential also
as validators (see picture 6).
Picture 6: The QMPP Qualityscape
In peer creation there are different modes of work, which actualize on different
times. The creation work can include authoring the content (such as making a
new article or video), editing of the digital content (such as proof-reading),
enriching the digital content (such as adding new information or new media) or
updating the content (such as updating the existing content with contemporary
data).
However, the key aspect in the quality management of peer production is the
active and thorough peer validation work. The peer validation work can include
benchmarking (comparing with other sources), peer reviews (systematically
validating your content with other peers), peer reflection (reflecting the content
with other peers) and peer learning (joint learning and mutual development
through continuous assessment).
In the following table (see table 6) we have summarized some typical activities
during in the peer creation peer validation work.
Handbook for Quality Management of Peer Production
65
Peer creation (including peer
authoring)
creating digital learning content by
authoring, editing, enriching and
updating using various media
Authoring
(shared) authoring of texts and other
digital resources; creating images,
audio materials, video materials;
creating content for wikis etc.
Editing
(shared) editing of digital content (from
proof-reading to translation), creating
alternative navigational routes, creating
collages etc.
Enriching
creating additional digital content,
publishing individual works and team
works, sharing or learning (b)logs,
adding library links, social bookmarking
etc.
Updating
monitoring existing content, updating
existing content, adding specific area
content etc
Peer validation
validating digital content with subject
matter experts, validating content with
peers, rating the validity and usability
of the content etc.
Benchmarking
identifying of good cases and practices
for comparative purposes, identifying
of additional digital resources,
identifying areas of lacking content etc.
Peer reviews
providing feedback by peers of
learning goals, progress and aims
within a learning community
Peer reflection
encouraging the reflection of learning
processes by means of own
experiences and sharing the reflections
within the learning community or
between different learning communities
Peer learning
joint learning also by the exchange of
learning experiences and learning
outcomes, such as e-portfolios
Table 6: Peer creation and peer validation
Quality management of peer production of eLearning must also focus on
providing enabling processes and enabling tools for those situations in which
peer-created content shall be peer-validated. The enabling policies for peer
creation, peer validation and quality management must thus support the entire
process of Peer Creation and Peer Validation, by providing enabling tools and
suggesting enabling processes.
Handbook for Quality Management of Peer Production
66
Action point
You have now been acquainted with many aspects of quality management of
peer production of eLearning and we have now presented you the QMPP
Qualityscape model. According to your own organization, assess the QMPP
Qualityscape – does it answer your situation? What should be added to the
QMPP Qualityscape approach for it to serve your needs?
Handbook for Quality Management of Peer Production
67
7.3 QMPP experiences and case studies
During the QMPP project we ran four different pilot projects. These pilot projects
had all their specific nature as well as their methodology to work with the quality
methodology. In the following we describe these pilots and their experiences.
Case study A: Manolo’s Business Trip - IAVANTE57
is part of a peer produced, online, multi-level, English language
course. It is a project of the IAVANTE English Training Program, which serves 200
students from all over the Andalusian Public Health System in Spain (APHS). This blog-
based project was devised as a transversal strategy to adapt learning content to the
specific language needs of the professionals who work in public health care as well as to
address two additional factors. The first is the growing reliance on Web 2.0 structures in
the workplace of health professionals and therefore attraction to their use for training
needs, and the second is to provide a continually stimulating environment in class cycles
with no fixed ending points.
Manolo is the virtual protagonist of a series of episodes published on an open Wordpress
blog using interactive Flash presentations. His character, represented by an avatar, is a
flexible mixture between professional and personal (doctor, trainer, researcher, bon
vivant, musician, etc) to facilitate adaptation to and incorporation of all the necessary
language scenarios commonly required by a gamut of health professional positions:
clinicians, health managers, medical researchers, administrative staff, IT personnel etc. A
team of eight language-training consultants and teachers collaboratively created the
initial episodes. These were tested in the classroom environment for didactic
effectiveness and also as a means of introducing the concept in context to the students.
The following and current chapters are based on student peer process created avatar
characters forming a community of friends, family, co-workers, and other tangent
characters interacting with Manolo.
Each APHS student group (in average eight students per group) creates a single avatar
Manolo and provide texts that are spoken by the avatar through recorded voice
technology. Each group creates a dialogue between their character and Manolo.
Avatar profiles and dialogues are published in a separate category on the blog dedicated
to student content. Student content is proofread by student group consensus moderated
by the teacher and is posted as is, without additional editing or translation. Episode
content is proofread and edited by a teacher peer process and accompanied by a
translation.
Groups vote for their favourite avatar from among all the groups, excluding their own.
The winning interaction is then prepared by the teaching team as the next episode in the
the blog decide what Manolo will do next.
57 this case study has been authored by Tim McQuaid and David Riley
(IAVANTE)
Handbook for Quality Management of Peer Production
68
Students define the direction of class content through group consensus twice a year.
They are responsible for contributing to class didactic material which is later housed in
the Moodle platform. Periodically, students are asked to reassess the class direction and
modify it to fit their evolving needs. Manolo's Business Trip episodes are adjusted to
reflect these changes. In addition to determining class and blog content, each student
creates an individual language goal specific to their personal language learning needs.
Classmates receive and provide group support in their efforts to fulfil these goals.
Basic, Intermediate and
Advanced and consists of a narrative based story followed by a question and answer
section, language and grammar tips, an area providing translation practice, quiz
possibilities and glossary. Language learning sections provide possibilities for audio, still
images, texts as well as video options. Material from each episode is used in conjunction
with face-to-face classes and participation in a Moodle platform. All texts and audio
materials are downloadable.
the teachers administering the course. Teachers communicate through emails in a
narratives and following exercises is created through group consensus based on
solicitudes from all teachers. Final validation of material is produced through a peer
review followed by administrator approval. Teachers also have their own avatars, which
are available for interaction with student created avatars in classroom creative scenarios.
A pilot group of eight advanced level students and three teachers provided feedback on
usiness Trip. Pilot group students and teachers completed questionnaires on
the principles for quality. Further peer validation processes are being defined.
Case study B: Hygiene passport - how to use the existing expertise in a cost-
effective way58
Vocational education has a strong place in Finnish educational system. It gives both
practical training and also basic academic skills, so after the vocational school one can
still continue his/her studies for instance in higher education institutions. Vocational
education is also attractive: more than half of the youngsters in the age of 16 choose the
vocational education as their educational alternative. In addition to the education of the
youngsters, vocational schools in Finland have strong professional training. Almost every
vocational school in Finland has also an adult vocational training centre or institute.
The QMPP pilot took place in Jyväskylä Institute of Adult Education. Its main function is
to provide initial and continuing vocational training and education for adults. It provides
instruction to 6000 youth and to 12 000 adult students every year. It has also quite a long
tradition in blended learning and using eLearning as a part of its learning provision. The
use so far has been very curriculum-driven and teacher/trainer orientated, so the
variation of the materials and courses have been very wide. With this QMPP pilot project
a new content creation method, peer production based on expert/teacher collaboration,
was introduced within the organization.
The most used method is teacher/expert content creation is a model, in which a group of
execution plan is then divided with the content creation group, and every member of the
58 this case study has been authored by Petri Lounaskorpi (Finnish eLearning
Centre)
Handbook for Quality Management of Peer Production
69
group will create one part of the agreed content. As one teacher/expert creates one
concrete part of the curriculum, he/she gets the other parts buy other content creators in
exchange. However, this model requires much codification and negotiation between the
different teachers and experts.
The alternative method is to gather the experts as a group and then concentrate on one
specific topic. The key agreement between the creation team is made about the learning
process, the content and the evaluation of the learning aims. The team has to reach a
mutual understanding of these aspects. The actual content creation process is then just
amalgamating the materials of the various content creators together and organizing them
into a right order. Hardest in this process is to expose the essential: the core information
what the student need to achieve the learning goals. In the QMPP pilot the latter method
used.
Hygienic passport has been quickly increasing and it requires more and more workload
for the expert teachers. The need of the Hygienic passport is included in the legislation
both on national as well as on European level. Practically, this means that every person
w - the
Hygienic passport. It is required from farmers, truck drivers, shop clerks, chefs, nurses
etc. already working in their profession.
So far the courses have been held traditionally face-to-face teaching and after the course
the learners have taken a test. The test is nationally coordinated with a questions
database maintained by the Ministry of Social Affairs and Health. Only licensed
teachers/trainers can correct the test and give out the Hygienic passport. In Jyväskylä
there were four licensed teachers and they were drowning to their work.
The content creation process was divided in five phases:
The functional quality of the training is linked with the actual topic of the training. The
course content must fulfil the legislative and regulative requirements of the Hygienic
passport. These objectives had to be achieved and there was also a clear testing
mechanism available by the national exams. On the other hand, the content creation
group of experts also automatically steers the quality with its work. If the content, learning
need to compromise and jointly agree how to maintain the high quality of the learning
provision.
The feedback and the update phase after the piloting phase provides also an important
step in the quality work. The development process must be seen as an ongoing process,
which assists in the updating and renewing of the course.
The course production in a vocational educational institution is challenging. eLearning
course creation is seldom seen as an investment. New ways of working are often
interpreted as additional costs. The Q
were that the person power required for good peer production was severely
underestimated by the management of the institution. During the pilot, the experts spent
Handbook for Quality Management of Peer Production
70
104 real working hours and the support staff (in assisting with the LMS etc.) another 112
working hours for the Hygienic passport course. This showed that good peer production
must be well resourced and its nature as a method in content creation requires joint work
by many different actors. However, the institute has now a permanent eLearning
resource to widen its learning provision. Thus the key learning point of the QMPP pilot in
Finland was that good and thorough peer production by experts must be provided with
time resources.
The experiences of the training were excellent. All the students had access to the course
either from their workplace or from their homes. The course structure was logical and all
the students of the first cohort studied thoroughly the course and all passed the
required test.
Case Study C: Critical care advanced training course for nurses - peer production
of eLearning content combined with advanced patient simulation at IAVANTE59
In the piloting of the QMPP approach,, the Granada Medical Simulation Centre of the
IAVANTE Foundation has incorporated a Critical Care course using advanced simulation
techniques combined with peer production of eLearning content in the training of expert
professionals.
In Critical care course which the students expert nurses - share their knowledge and
experiences, but also develop the competences in practice. The piloting group consisted
of seven nurses. They were expert professionals on Critical care, although working in
different fields (organ donation, emergency etc). Furthermore, the support for the pilot
consisted of a training coordinator and three facilitators for methodological aspects,
technical skills and non-technical skills. The main outspoken objective of the course was
to improve knowledge and skills in Critical care, and the specific objectives were set by
the peer group itself.
There were two phases of the course:
- eLearning phase: the students (peers) produced all theoretical content and the content
of the training sessions through peer production using an eLearning platform (Moodle).
The content was divided into four main modules in Moodle, and the peers uploaded and
discussed their deliverables (in the form of work sheets) divided for knowledge skills,
technical skills, non-technical skills and clinical scenarios.
- Face-to-face learning phase: the face-to-face learning phase took place over a two-day
se
simulations. The peer group adopted their acquired knowledge into practice in advanced
simulation training (robotic and actor-based simulation). At the end of the course, the
peer group entered the eLearning platform to share reflections about their own
experiences.
The course was oriented to results and performance: in all 24 documents were uploaded
into e-learning platform by peers. Documents covering knowledge skills and technical
skills were produced by the peers as well as seven clinical cases were created by peers
these were implemented in clinical scenario simulations at the end of the training
process. Moreover, the learners shared a high level of satisfaction, which was
documented by questionnaires after the training.
Various eLearning 2.0 principles were implemented along the course. Self-assessment of
own needs was very important in the first moments of the training. Reflection of the
59 this case study has been authored by Jose Pinzon (IAVANTE)
Handbook for Quality Management of Peer Production
71
learning as well as of the work practices was of key importance, and some aspects of the
training methodology were defined by peer consensus.
As a conclusion of this QMPP pilot we can state that:
- Participants were free to create content. They agreed methodology and specific
objectives at the beginning.
- Learning process promoted refection and learners as pro-active producers.
- The facilitator had a key role because of the novelty and features of peer
production processes.
- All peers have produced content and have exchanged knowledge from their own
fields.
- Created content is reusable for future courses as a modular content.
- Peer production enabled the sharing of tacit knowledge and put it into practice in a
simulation scenario.
Case study D: Master Course in Educational Design at University of Macerata60
The Department of Education of University of Macerata (Italy) run a QMPP pilot within
the Master course was organized in collaboration
with the following Italian institutions: University of Molise and the corporate IFOR (Istituto
di Formazione Orientamento e Ricerca) located in Matera (Italy). This course was
designed as a one year course (60 ECTS points) and it lasted from October 2008 to April
2009
The objective of the course was to provide in-service and future teachers with the proper
competencies to be able to
- design learning paths which focus on authentic tasks, balanced assessment and
the use of new technologies
- match the design phase with the educational research process
- build curricula for the different disciplines.
The Master course offered three different learning paths:
- Educational research
- Disciplinary curricula (Italian, Maths, English, History, Science)
- Instructional technology.
The participants were expected to be in-service school teachers of any level (from
primary to high school level) and graduates whose degree allows them to start the school
career.
The participants of the course were required to allocate a total amount of 1500 hours to
the course and its activities this consisted of:
- 12 hours of face-to-face instruction (at Faculty of Education University of
Macerata connected via videoconference either to IFOR or Faculty of Social Science
University of Molise)
- 375 hours of eb 2.0 applications)
- 300 hours of project work in the project work stage (to be agreed with the scientific
committee; in-service teachers could ask to do their stage in the school they are working
in)
- 813 hours of individual study work.
During the course different resources were be available and delivered via the LMS, such
as
60 this case study has been authored by Laura Fedeli (University of Macerata)
Handbook for Quality Management of Peer Production
72
- web resources (links to websites, weblogs, wikis, articles etc.)
- materials prepared by both university professors and experts in the educational
area coming from the school context (articles, case-studies, presentations)
- materials created by the participants during their activities; these materials serve
as a resource to be accessed by all teachers of the current ongoing course as an input to
foster reflection on personal habitus related to the design process as well as a resource
to be used in future master courses
The method used in the Master course was a practice-theory-practice approach following
the inputs theorized by Marguerite Altet, Philippe Perrenoud, François Tochon, Léopold
Paquay and Shulman. The key activities done partially in collaboration with the peer
group - could be divided as follows:
- individual reflection process (through an e-portfolio)
- shared reflection and collaborative tasks in small group activities (chats, discussion
in forum, collaborative production of reports)
- final project (personal presentation using a selection of tools).
Action point
You have now been acquainted four different case studies, in which peer
production has been implemented in practice. Regarding your own organization,
reflect in the following what were the key quality mechanisms in the cases. You
should also discuss what were the key findings in the presented cases, which
have an impact in your organization when implemented peer production in
eLearning.
Handbook for Quality Management of Peer Production
73
8 Build your own quality development
approach
In this chapter we guide you in building a suitable quality approach to
your organization to support the quality management of peer
production of eLearning. As you have worked with the previous
chapters, you have already made a number of definitions and set up a
number of guidelines.
8.1 Assessing your concurrent practices
The first recommendable step is to assess your concurrent practices. The
mapping of the existing situation provides you with an appropriate understanding
of your practices, but it can also provide you ideas to improve your work in the
quality area.
Action point
Use the following list to assess your concurrent practices. It has been formulated
in order to give indication how peer production practice can become high quality
peer production. Please go through the list and assess your own practice. You
are encouraged to write into the right column if and how you have implemented
the specific principle in your own practice.
Principles for design
Principles
How is this principle
implemented in your own
practice?
1. The peer production environment should
be designed in a non restrictive way and
giving suggestions rather than restrictive
laws.
2. Democratic design includes shared
ownership of processes; focus on peer
interaction, quality is then defined as the
groups quality consensus.
3. Pedagogical Design of learning
environments should focus on openness
and not limit the peer group up front, it
Handbook for Quality Management of Peer Production
74
should be participatory and strive for peer-
agreement on objectives, methodologies
(at the beginning).
4. The design process should focus on
consistency of each members level of
input and positive interdependences
between peers as well as complimentarily
of the role of peers
5. It is important that a code of practice for
democratic design is existing which
underpins the changed objectives of
quality evaluation in peer-learning
6. The educational design should be
relevant to the professional context; the
contents should be fresh and updated,
should focus on a supportive context and
relate to a real identified need.
7. It needs a clear Purpose and ideological
support in term of culture for working
together, culture of continuous learning,
culture of acceptance of errors,
continuous improvement
Principles for the process/ educational process
Principles
How is this principle implemented
in your own practice?
1. The educational process needs to focus
on both quality and motivation (internal /
external)
2. It should allow low external
determination and high internal reflection
3. Reflection is of key importance for
learning 2.0 processes.
4. Learning process should be open,
however Users should be accountable
and registered in order to be able to
validate the contributors
Handbook for Quality Management of Peer Production
75
5. The learning process should promote
learners as active producers rather than
passive consumer, should follow the
path from reception to participation and
focus on interaction
Principles for technology
Principles
How is this principle implemented
in your own practice?
1. Functional quality is important:
coherence of used tools, Technical
support, User friendliness, testing and
validating, Accessibility, Tutorials are
available, traceability of production
2. High contribution possibilities should be
sidelined by highly accessible media
Principles for organisation
Principles
How is this principle implemented
in your own practice?
Peer Learning has to be economic, concepts
reusable and an appropriate documentation
should be available at all the stages
Principles for outcomes
Principles
How is this principle implemented
in your own practice?
1. Peer learning should be oriented to
results.
2. The learning should be based on
established validation processes
through the evaluation by peers, review
committees for content and a peer
review needs to be in place.
Handbook for Quality Management of Peer Production
76
8.2 Developing and defining your quality approach
Although we would like to see quality management of peer production
implemented quickly and widely throughout our organization, it is needed to
understand that different modalities might require different aspects of emphasis.
Consider the case studies presented in the Handbook they describe
organizationally very different situations with different actors and different
ambitions. Based on the cases presented, we can present a simple classification
of the various critical dimensions of peer production in eLearning and related
fields.
The two critical dimensions are (see picture 7):
setting of objectives who is setting the objectives for the peer-produced
content: is it controlled and unified for all potential peer producers (e.g. a
article of my personal interest)
structure what structure is given: is the structure firm (e.g. composed of
certain elements, tools and requirements) or the structure loose and not pre-
defined (e.g. peers can use tools and structure of their own choice).
Picture 7: Setting of objectives vs. structure
Handbook for Quality Management of Peer Production
77
Action point
In developing your quality approach, consider at least the following aspects:
Elements of the quality approach
Your decisions
Define your target audience and
learners (also assess their abilities and
readiness for peer production)
Define your subject area (also assess
whether it lends itself easily to peer
production)
Define the setting of objectives (is it
controlled or loose) in your eLearning
work
Define the structure (is the structure
firm or loose) in your eLearning work
Define the tools and technologies to be
used and available (are you providing
the tools or are the learners free to any
tools)
Define the resources required (what
resources are required this should
include manpower as well as other
resources)
Define support required (technical
support, pedagogical support etc.)
Other aspects, what
Other aspects, what
Handbook for Quality Management of Peer Production
78
Other aspects, what
8.3 Documenting your quality plan for peer production
After assessing your concurrent practices, developing and defining your quality
approach, you should now document clearly your quality plan for peer production
of eLearning in your organization.
We have described many elements for a quality approach by both presenting the
QMPP Qualityscape as well as describing several practices based on real life
case studies in chapters 3 and 7. Based on this background information, you
should now start to define your quality measures for peer production of
eLearning.
We propose that for your quality approach, you utilize the QMPP Qualityscape
approach and the elementary areas in the Qualityscape. Thus you should pay
attention to the following aspects:
- what measures are required to ensure quality of peer creation activities
- what measures are required to ensure quality of peer validation activities
- what measures are required to ensure the interplay of peer creation and
peer validation
- what measures are required to ensure the use of enabling tools and
processes
- what measures are required to ensure the support of enabling policies.
Action point
Following the QMPP Qualityscape (and based on your own experiences as well
as the case studies we have presented) define the following aspects:
Quality measures
Your decisions
What measures are you taking in
ensuring appropriate peer creation
activities?
Handbook for Quality Management of Peer Production
79
What measures are you taking in
ensuring appropriate peer validation
activities?
What measures are you taking in
ensuring the interplay of peer creation
and peer validation?
What measures are you taking in
ensuring appropriate enabling tools
and processes?
What measures are you taking in
ensuring appropriate enabling policies?
However, as we know, quality does not it needs to be planned
and implemented by people working within the organization. Thus after defining
the key elements of the quality plan, you should also define the roles and tasks
for each of the persons involved in the quality work.
Key persons
Roles and tasks
Handbook for Quality Management of Peer Production
80
And finally, as discussed already in the early stages of the QMPP Handbook, the
approach in quality can be summarized with the PDCA acronym: Plan Do
Check Act. This suggests that we see quality development as a continuous
activity. You have now defined the basic elements for your quality approach and
quality measures for peer production of eLearning. You should also now define
the actions for continuous improvement.
PDCA phases
Actions
Plan
Do
Check
Act (improve)
Handbook for Quality Management of Peer Production
81
9 Recommended readings
We have collected to the following list of recommended readings such literature,
which we have used in the text, but also such books, articles and web resources,
which might interest you in getting a deeper understanding of the issues of peer
production and quality management.
9.1 Recommended literature
Ahmed, P.K. Kok, L.K. Loh, A.Y.E. (2002), Learning through knowledge
management. Butterworth Heinemann.
Anderson, C. (2006), The Long Tail: Why the Future of Business Is Selling
Less of More. Hyperion.
Anderson, P. (2007), What is Web 2.0? - Ideas, technologies and
implications for education. JISC Technology and Standards Watch, Feb. 2007
at http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf
Barrett, E. (ed.) (1994), Sociomedia Multimedia, Hypermedia and the
Social Construction of Knowledge. The MIT Press.
Benkler, Y. (2006), The Wealth of Networks. Yale University Press.
Brown J.S. - Duguid P. (2000), Balancing act: How to capture knowledge
without killing it. Harvard Business Review 78,73-80.
Cross, J: Informal Learning (2006), Rediscovering the Natural Pathways
That Inspire Innovation and Performance. Pfeiffer.
Duval, E.: LearnRank (2006), Towards a real quality measure for Learning.
In Ehlers, U.-D. Pawlowski, J.M.: Handbook of Quality and Standardisation in
E-Learning. Springer.
Ehlers, U.-D. Pawlowski, J. (eds.) (2006), Handbook on Quality and
Standardisation in E-Learning. Springer.
Ehlers, U.-D. Pawlowski, J. (2006), Quality in European e-learning: An
introduction. In Ehlers, U.-D. Pawlowski, J. (eds.): Handbook on Quality and
Standardisation in E-Learning. Springer.
Ehlers, U.-D. (2008), Web 2.0 eLearning 2.0 Quality 2.0 Perspectives
on a change in learning culture and quality concepts. In Hohenstain, A.
Wilbers, K. (eds.): Handbuch E-Learning. Köln 2008.
Ehlers, U.-D. (2009), Web 2.0 E-Learning 2.0 Quality 2.0? Quality for
new learning cultures. Quality Assurance in Education 17, 296-314.
Ghosh, R.A. (2005), CODE Collaborative Ownership and the Digital
Economy. The MIT Press.
Goldman, R. Gabriel, R.P. (2008), Innovation Happens Elsewhere
Open Source as Business Strategy. Elsevier.
Guest, T. (2007), Second Lives A journey through virtual worlds. Arrow
Books.
Harasim. L. et al. (1995), Learning Networks. The MIT Press..
Hietanen, H. Oksanen, V. Välimäki, M. (2007), Community Created
Content. Turre Legal.
Keen, A. (2007), The Cult of the Amateur: How Today's Internet Is Killing
Our Culture and Assaulting Our Economy. Nicholas Brealey Publishing.
Handbook for Quality Management of Peer Production
82
Kotzinos, D. et al. (2005), Online Curriculum on the Semantic Web: The
CSD-UoC Portal for Peer-to-Peer E-learning. Proceeding of WWW 2005, May
10-14, 2005, Chiba, Japan.
Leadbeater, C (2008), We-think: The Power of Mass Creativity. Profile
Books.
Lih, A. (2009), The Wikipedia Revolution: How a Bunch of Nobodies
Created the World's Greatest Encyclopedia. Hyperion.
Maehr, M. Stallings, R. (1972), Freedom from external evaluation. Child
Development 43, 177-185.
McGuinness, D.L. et al. (2006), Investigations into Trust for Collaborative
Information Repositories: A Wikipedia Case Study. WWW2006 Workshop on the
Moore, M.G. (2003), Network Systems: The Emerging Organizational
Paradigm. The American Journal of Distance Education 17 , 15.
Nonaka, I. Takeuchi, H. (1995), The Knowledge-Creating Company.
Oxford University Press.
OECD - Working Party on Information Technology (2007) Participative
Web: User-created content. DSTI/ICCP/IE(2006)7/FINAL. OECD.
O´Reilly, T. (2007), What is Web 2.0? - Design Patterns and Business
Models for the Next Generation of Software -
http://www.oreilly.de/artikel/web20.html
Parker, K.R. Chao, J.T. (2007), Wiki as a Teaching Tool. Interdisciplinary
Journal of Knowledge and Learning Objects 3, 57 72.
Pettenenati, M.C. Cigognini, M.E. (2007), Social Networking Theories
and Tools to Support Connectivist Learning Activities. International Journal of
Web-based Learning and Teaching Technologies 2, 42 60.
Prahalad, C.K. Krishnan, M.S. (2008), The New Age of Innovation
Driving Cocreated Value through Global Networks. McGraw-Hill.
Prahalad, C.K. Ramaswamy, V. (2004), The Future of Competition Co-
Creating Unique Value with Customers. Harvard Business School Press.
Rask, M. (2007), The Richness and Reach of Wikinomics: Is the Free Web-
Based Encyclopedia Wikipedia Only for the Rich Countries? . Proceedings of the
Joint Conference of The International Society of Marketing Development and the
Macromarketing Society, June 2-5, 2007 (available at SSRN:
http://ssrn.com/abstract=996158).
Reardon, R.F. (2004), Informal learning after organizational change. The
Journal of Workplace Learning 17, 385 395.
Richardson, W. (2006), Blogs, Wikis, Podcasts, and Other Powerful Web
Tools for Classrooms. Corwin Press.
Rolheiser, C. Ross, J.A. (2001), Student self-evaluation: What research
says and what practice shows. In Small, R.D. Thomas, A. (eds.): Plain Talk
About Kids. Covington.
Senge, P.M. (1990), The Fifth Discipline The Art & Practive of The
Learning Orgnization. Century Business.
Surowiecki J. (2005), The Wisdom of Crowds: Why the Many Are Smarter
Than the Few. Abacus.
Tapsoctt, D. Ticoll, D. Lowy, A. (2000), Digital Capital Harnessing the
Power of Business Webs. Nicholas Bearley Publishing.
Tapscott, D. Williams, A.D. (2006), Wikinomics. Portfolio, USA.
Wenger, E. (1999), Communities of practice: learning, meaning, and
identity. Cambridge University Press.
Handbook for Quality Management of Peer Production
83
Williams, J.B. (2004), Exploring the use of blogs as learning spaces in the
higher education sector. Australasian Journal of Educational Technology 20,
232-247.
Wirth, M.A. (2006), An analysis of international quality management
approaches in e-learning: Different paths, similar pursuits. In Ehlers, U.-D.
Pawlowski, J. (eds.): Handbook on Quality and Standardisation in E-Learning.
Springer.
9.2 Webliography
http://www.youtube.com
http://www.facebook.com
http://www.flickr.com
http://slashdot.org
http://www.thefreedictionary.com
http://en.wikipedia.org/wiki
