Defining Human-Machine Micro-Task Workflows for
Nuno Luz1,2, Marta Poblet3, Nuno Silva1, Paulo Novais2
1 GECAD (Knowledge Engineering and Decision Support Group), Polytechnic of Porto
2 CCTC (Computer Science and Technology Center), University of Minho
3 Graduate School of Business and Law, RMIT University
Abstract. This paper presents a novel task-oriented approach to crowdsource
the drafting of a constitution. By considering micro-tasking as a particular form
of crowdsourcing, it defines a workflow-based approach based on Onto2Flow,
an ontology that models the basic concepts and roles to represent workflow-
definitions. The approach is then applied to a prototype platform for
constitution-making where human workers are requested to contribute to a set
of tasks. The paper concludes by discussing previous approaches to
participatory constitution-making and identifying areas for future work.
Keywords: Micro-tasking, micro-tasks, workflows, ontologies, political
crowdsourcing, legal crowdsourcing, constitution-making, participation.
Constitution-making can be broadly defined as a set of activities intended to produce
a constitution, the highest law of a state. To the UN, constitution-making “covers both
the process of drafting and substance of a new constitution, or reforms of an existing
constitution” . Klein and Sajo have also defined it as a “decision-making process
carried out by political actors, responsible for selecting, enforcing, implementing, and
evaluating societal choices” . Given that constitution-making may only happen
once in a generation, it is often seen as a unique moment shaping both the present and
the future of a country. As Elster has put it, “if there is one task for which ‘wisdom’
would seem highly desirable, it is that of writing a constitution” .
This is a pre-press version of a paper with the same title in the proceedings volume of 15th International
Conference, GDN 2015, Warsaw, Poland (June 22-26, 2015) in Lecture Notes in Business Information
Processing 218: 333-344. © Springer International Publishing Switzerland. Personal use of this material is
permitted. However, permission to reprint/republish this material for advertising or promotional purposes
or for creating new collective works for resale or redistribution to servers or lists, or to reuse any
copyrighted component of this work in other works must be obtained from Springer International
This paper reviews a few examples of how the wisdom of the crowd has been
tapped in recent constitution-making processes across the world and proposes a new
approach to write a constitution based on micro-tasking, a particular form of
Section 2 provides definitions of crowdsourcing and micro-tasking and additional
background knowledge on recent examples of constitutional crowdsourcing. Section 3
briefly reviews ontology-based micro-tasking workflows and presents Onto2Flow, an
ontology designed to retrieve structured and semantically enriched data from micro-
tasks. Section 4 applies this framework to a prototype platform that enables the micro-
tasking of a constitutional text. Section 5 discusses both the potential and limitations
of this approach. The conclusion, finally, suggests future work in this area.
The word crowdsourcing was coined by Jeff Howe and Mark Robinson in 2006 to
represent “the act of taking a job traditionally performed by a designated agent
(usually an employee) and outsourcing it to an undefined, generally large group of
people in the form of an open call” . This broad conceptualization has been
followed by a myriad of definitions of crowdsourcing drawn from different but
connected approaches: collective intelligence (CI), human computation, social
intelligence, and social computing. It also has been noted that "while human
computation (HC) is a term that is mostly used by the scientiﬁc community,
crowdsourcing (CS) is a term highly employed in the business world . Despite the
variety of perspectives, all approaches highlight three key elements in crowdsourcing:
crowds, tasks, and mediating technologies.
Micro-task crowdsourcing, in particular, is a special kind of human computation
where relatively complex tasks are divided into smaller and independent micro-tasks
. These micro-tasks are then modelled and published through a computational
platform (e.g. Mechanical Turk and CrowdFlower), which distributes them through a
crowd of workers.
Micro-tasks are often employed for solving large-scale problems that are often too
complex for computers to solve on their own . These problems usually require a
degree of creativity (or just common sense), plus some background knowledge [7, 8].
In our view, the drafting of a constitution: (i) can be represented as a large-scale
problem that can be divided into smaller tasks; (ii) these micro-tasks can be
completed by a crowd of heterogeneous citizens with different degrees of legal
expertise (from none to expert).
2.1 Crowdsourced constitution-making
In the political and legal domains, crowdsourcing methods and tools have been used
as a means to collect input from citizens on a variety of areas, such as legal drafting,
legal reform, legal education, policy-making and human rights advocacy [9–12].
Crowdsourced constitution-making, in particular, was famously displayed in Iceland
in 2011 with the use of social media to collect peoples’ views and opinions on the
constitutional draft . Similar initiatives were taking place almost simultaneously
in Kenya (2010), Ghana (2010-2011), Somalia (2011), Egypt (2012), and Libya
(2012), among other countries . Likewise, Morocco announced a constitutional
reform in early 2011 and, shortly after, a citizen-based initiative launched reforme.ma,
a dedicated crowdsourcing platform fully integrated with Facebook and Twitter where
citizens could like or dislike the proposed articles and comment on them .
In the effort to make constitution making as participatory as possible, these
initiatives have all taped on social media (and, in some cases, e-mail and text
messages) to elicit comments from the public. In all cases, and regardless of the final
number of participants, thousands of comments were posted and eventually collected.
The analysis of how these contributions were classified a posteriori and their eventual
impact on the final drafts would require a case-by-case approach. Yet, it seems clear
that in all mentioned examples the public was invited to comment, answer questions,
vote, or “like”, but not to “write” the constitution itself. To date, crowdsourced
constitution-making has heavily relied on online deliberation, but the impact of such
deliberative processes on the final outcome is yet to be fully assessed. While
deliberative processes are core to constitution-making, we aim at a complementary
approach where the constitutional draft is also the product of coordinated micro-
tasking via the participation of a large number of participants.
2.2 Ontologies in description logics
Our approach adds a new layer to constitution-making by considering a micro-task
workflow-based approach to the drafting and refinement of the document. Drafting
and refinement workflows are modelled using ontologies, which allow a formal,
explicit and shared conceptual representation while maintaining machine
interpretability. Ontologies are formal because they are supported by unambiguous
formal logics; explicit since they make domain assumptions explicit for reasoning and
understanding; and shared for its ability to provide consensus.
Ontologies “represent the best answer to the demand for intelligent systems that
operate closer to the human conceptual level” . Thus they are an appropriate
representation mechanism for environments where both human and machine agents
must interpret the data and perform a particular set of actions. Furthermore, the
inherent extensibility of ontologies allows the growing set of domain ontologies in the
Semantic Web to be re-used in the representation of workflows.
3 Ontology-based Micro-Task Workflows
Micro-tasks (or simply “tasks” from now on) can be seen as atomic operations that
produce a specific set of data. These atomic operations occur within a specific domain
of operation involving certain domain knowledge. Given a task, its domain of
operation is defined by its input and output specifications.
Onto2Flow is an approach to the representation, instantiation and execution of
workflows that represents workflows of tasks as extensions of other domain
ontologies. These extensions are called workflow-definition ontologies. Workflow-
definition ontologies assemble two different data dimensions: (a) the static domain
dimension (corresponding to the domain ontology) and (b) the dynamic task and
workflow dimension (corresponding to the Onto2Flow ontology). In this perspective,
task-definitions (or task representations) are extensions of the domain ontology,
which add an operational dynamic dimension. Fig. 1 illustrates these two dimensions
and their assemblage.
Fig. 1. Static and dynamic dimension in a workflow-definition ontology.
3.1 The Onto2Flow process
Onto2Flow assumes that domain ontologies represent the structure and semantics of
the data presented to (and retrieved from) workers. Accordingly, the approach
considers two steps (outlined in Fig. 2): (i) task-definition and workflow-definition
(the ontology of the workflow), and (ii) the instantiation and execution of the
workflow on a particular input dataset.
Fig. 2. Overview of the Onto2Flow approach.
At the stage of workflow-definition (1), the requester must clearly define the activities
involved in the workflow through a semantic model of the input and output data and
create a workflow-definition. For workflow-definitions containing task-definitions
that human workers have to solve, crowd (user) interface templates must be supplied
along with the workflow-definition ontology. The interface templates present the task
data to the worker and retrieve the submitted response.
At stage 2 (instantiation and execution of the workflow-definition), workflow-
definition ontologies can be instantiated multiple times and executed by any workflow
engine that is able to interpret the Onto2Flow ontology and apply the ground rules
established by the proposed method. Furthermore, Onto2Flow-based workflow
engines may dispatch the execution of the tasks to external micro-task execution
communities such as Mechanical Turk and CrowdFlower, or provide their own task
resolution interfaces that may interact with external social networks.
3.2 The Onto2Flow ontology: concepts and roles
The Onto2Flow ontology defines the basic concepts and roles required to represent
workflow-definitions (see Fig. 3). It captures concepts and lessons learnt from
workflow-definition languages and approaches such as the XPDL (XML Process
Definition Language) and BPMN (Business Process Modelling Notation) .
Furthermore, it incorporates concepts that support the crowdsourcing, distribution,
and delivery of tasks.
Fig. 3. Overview of the Onto2Flow ontology.
The concept Job represents a workflow execution environment created by a
Requester, which may contain more than one Workflow.
Activities are the interconnected components that form a workflow. There are three
main types of activities: the Workflow, the Task and the Event. Among these,
Deliverables, which include Task and Event, represent a group of activities that
require worker or external interaction through some kind of Interface.
Two main types of actors are considered by the Onto2Flow ontology: the
Requester (the one requesting the execution of a workflow) and the Worker (the one
solving the tasks of the workflow), which are either Human or Machine.
Each Actor may belong to several ActorGroup. Actor groups allow requesters to
associate and filter groups of actors for participation in particular tasks. An
ActorGroup may include a wide set of attributes, including social network analysis
clustering measurements (e.g. clusterability), improving the control of the requester
over the selection of workers. Inclusively, each ActorGroupMembership may feature
a wide set of actor specific attributes and measurements (e.g. centrality and prestige).
Workflows are graphs of activities linked through transitions, which establish a
process that delivers a specific result dataset given an input dataset.
The flow of activities in a workflow is established through Transitions. There are
six types of transitions, depending on the set of (i) incoming activities
(BasicTransition, MergeTransition or SynchronizationTransition), (ii) outgoing
activities (ParallelTransition or DisjunctTransition), (iii) whether there is one or
more conditions to be fulfilled in order to continue its execution
An Event is an external occurrence that either triggers the continuation of a running
workflow (RunningEvent) or triggers the execution of a new workflow
A Task is a set of assignments and operations on top of input data, which must be
performed by workers. The representation of a task involves multiple concepts and
roles in the Onto2Flow ontology. These concepts are:
The Assignment concept, representing the actual operationalization of the task;
The Unit concepts, which represent the input unit of work given to the worker;
The UnitContext concepts, which represent relevant contextual input data that
must be presented along with the unit (and possibly related to it);
The Response concepts, which represent the top-level response or output given
by the worker;
The ResponseContext concepts, which represent additional output given by the
worker, usually related to the response.
Each work unit (represented by the Unit concept) is assigned to a worker through an
Assignment. The same unit may be assigned to different workers, resulting in different
solutions to the same problem.
The execution of a workflow requires interaction with external actors and services
during the execution of Event and Task activities. While an Event is typically listened
for, and arrives through an EventInterface, a Task must be delivered to and retrieved
from workers through a TaskInterface. Thus, interfaces represent logical and/or
physical components through which the interaction with workers (machine or human)
is performed (e.g. a Web service interface, a graphical user Web interface).
The ability to represent different types of interfaces enables the specification of
distinct interfaces, commonly used on user-centric environments :
Simple, where a single medium or modality is used. For instance, tasks can be
delivered to workers through a visual interface, a sound interface, or simply
through a web interface (the common case for crowdsourcing applications);
Multi-modal, i.e. capable of merging and coordinating multiple mediums and
modalities as a single interface.
Accordingly, and of particular interest in the crowdsourcing scenario, different types
of user interface implementations, such as a game interface or a mobile interface, can
be used to distribute tasks through human workers.
4 Catalan Constitution-Making Scenario
The Catalan constitution-making scenario is a prototype of a micro-tasking platform
to crowdsource the elaboration of a constitutional text. This scenario uses the
Constitute project ontology as the static domain dimension. The Constitute project is a
database of constitutional texts to search and compare constitutions across the world
. On top of the Constitute project ontology, a workflow-definition following the
Onto2Flow method was built. The resulting workflow-definition, as shown in Fig. 4,
aims to take the ontology-based representation of a proposed Catalan constitution and
crowdsource its elaboration, stemming from a basic initial text .
The process contemplates the following tasks, all performed by human workers:
T1 - evaluates sections of the current constitution document and is performed
by any worker;
T2 - revises and updates sections of the current constitution document marked
in the previous task and is performed by expert workers;
T3 - selects the best version of a section from the set of proposed sections in
the previous task and is performed by any worker.
Fig. 4. Overview of the Catalan constitution-making workflow-definition.
The Constitute project ontology represents the constitution document through
sections. A partial illustration of the Constitute project ontology is presented in Fig. 5.
An additional set of concepts was added to the static domain dimension in order to
represent the opinion and the assessment of the constitution sections.
Fig. 5. Partial Constitute project ontology and additional assessment concepts.
4.1 The Workflow-Definition
The constitution-making workflow-definition was built using both a construction
framework prototype implementation and the Protégé ontology editor. The Protégé
ontology editor was used to establish some common axioms that are not yet featured
by the construction framework, such as the union of input and output concepts, and
inverse roles. A detailed illustration of the workflow-definition is presented in Fig. 6.
Notice how each task-definition contains a complete representation of all the concepts
and relationships involved. Also, this representation is directly mapped to the
Constitute project ontology.
Fig. 6. Task-definitions in the constitution-making workflow-definition ( represents a
dependency relationship, which can be reduced to a subsumption).
In T1, the amount of assignments per unit will correspond to the amount of
evaluations given to each section. Thus, T1 must have an amount of assignments per
unit greater or equal to X, where X is the amount of evaluations that request an update
of the section. This amount (X) is used in T2 to assess which sections must be revised
The use of the role transitive closure onto the parent role allows all descendant
sections of the unit section to be included in the assignment and shown to the worker.
Also, regular expressions may be used to restrict the value of data -type roles. Such is
the case of the value of the header role in T1 (Section_T1 header : “/^Article/”).
4.2 The Task-Definition UI Templates
In the Catalan constitution-making scenario all tasks are solved by human workers
(volunteers). Volunteers contribute by adopting two different profiles: non-experts or
experts. Non-experts are the large majority of citizens who sign into the platform to
complete tasks in T1; experts are those volunteers designated by the requesters with
an editing role of the outputs produced by non-experts (classification, collation,
amendments). In both cases, the workflow-definition includes an UI (User Interface)
template. The UI template of T1 presents the unit section, its parent section, and all its
descendant sections to the non-expert volunteer. The volunteer is then invited to
evaluate the contents of the section (an article of the constitution) and assess whether
it needs to be: (i) updated (rewritten), (ii) removed or (ii) accepted as it is. Volunteers
can access the complete initial constitutional draft at any time to situate their
assignment into the broader picture of the full text.
The UI template of T2 presents the unit section to expert workers in the same way
as T1, including any modifications of the constitutional text by non-experts in T1. The
expert volunteer is then asked to submit a new revised section with all outputs
collected T1 classified and, if necessary, edited and collated.
Finally, the UI template of T3 presents each of the previously submitted sections
(during T2), along with the original section. In T3, all volunteers are requested to
select the best version. Fig. 7 below offers an example of the UI template of T1 as
presented to non-expert volunteers.
Fig. 7. Example assignment with the UI template of T1.
5 Conclusions and Future Work
Crowdsourcing the writing of a constitution to a large number of citizens is a complex
task that can be addressed by subdividing it to smaller units (micro-tasks). While
there are a number of examples of participatory constitution-making that involve
online deliberation, none of them offers a platform for citizens to edit the articles of
the text. Rather, their focus on eliciting and collecting opinions from public
deliberation, generally via social media, makes crowdsourcing initiatives accessory to
the drafting process developed elsewhere (e.g. in constitutional commissions).
Ultimately, this contingent aspect of crowdsourcing makes it difficult to assess the
impact of online participation on both the drafting process and the final outcome.
In our approach, writing a constitution becomes the core task. We rely here on two
well-researched conditions in the literature on the “wisdom of the crowd effect”: (i)
independence of judgment and (ii) heterogeneity of the crowd [21–23]. When these
two conditions are met, the crowd can perform better than individual experts.
To date, the platform has been tested by a reduced group of 8 experts who have
provided useful feedback. Future work involves expanding the testing to larger groups
of volunteers and refine the following issues: (i) identification of sub-topics within an
article and further division of micro-tasks; (ii) credentials and role of experts; (iii)
aggregation mechanisms in T3 (e.g. ratings, rankings) to avoid inconsistencies, and
(iv) generally, mechanisms to detect and resolve conflicts between different sections
in a constitution.
Beyond addressing these different issues dealing with coordination mechanisms,
further research will also be required to tackle substantive issues on how to coordinate
the crowd itself: (i) motivation; (ii) incentives to participate; (ii) relevance and quality
of the contributions; (iii) monitoring spam and sabotage attempts, etc. The ultimate
challenge is how to engage the crowds' collective wisdom in drafting such a high-
impact legal document as a national constitution.
Acknowledgments. This work is part-funded by FEDER Funds, by the ERDF
(European Regional Development Fund) through the COMPETE Programme
(Operational Programme for Competitiveness) and by National Funds through the
FCT (Portuguese Foundation for Science and Technology) within the project
FCOMP-01-0124-FEDER-028980 (PTDC/EEI-SII/1386/2012). The work of Nuno
Luz is supported by the doctoral grant SFRH/BD/70302/2010. The work of Marta
Poblet draws from previous research within the framework of the project
“Crowdsourcing: instrumentos semánticos para el desarrollo de la participación y la
mediación online” (DER 2012- 39492 -C02 -01) by the Spanish Ministry of Economy
1. United Nations Rule of Law (2009)- Guidance Note of the Secretary-General:
United Nations Assistance to Constitution-making Processes.
2. Klein C, Sajo A (2012) Constitution-Making: Process and substance. The Oxford
Handbook of Comparative Constitutional Law 419.
3. Elster J (2012) The optimal design of a constituent assembly. Collective Wisdom:
Principles and Mechanisms 148–172.
4. Howe J (2006) The rise of crowdsourcing. Wired magazine 14:1–4.
5. Luz N, Silva N, Novais P (2014) A survey of task-oriented crowdsourcing.
Artificial Intelligence Review 1–27.
6. Von Ahn L (2009) Human Computation. 46th ACM IEEE Design Automation
Conference. pp 418–419
7. Chklovski T (2003) Learner: A System for Acquiring Commonsense Knowledge
by Analogy. Proceedings of the 2nd ACM International Conference on
Knowledge Capture. Sanibel Island, FL, USA, pp 4–12
8. Singh P, Lin T, Mueller ET, et al. (2002) Open Mind Common Sense: Knowledge
Acquisition from the General Public. On the Move to Meaningful Internet
Systems 2002: CoopIS, DOA, and ODBASE. Springer, pp 1223–1237
9. Orozco D (2014) Democratizing the Law: Legal Crowdsourcing (Lawsourcing) as
a Means to Achieve Legal, Regulatory and Policy Objectives. Regulatory and
Policy Objectives (November 7, 2014)
10. Poblet M (2013) Visualizing the law: crisis mapping as an open tool for legal
practice. Journal of Open Access to Law 1:
11. Aitamurto T (2012) Crowdsourcing for Democracy: New Era In Policy–Making.
Committee for the Future, Parliament of Finland 1:2012.
12. Casanovas P (2012) Legal crowdsourcing and relational law: What the semantic
web can do for legal education. Journal of the Australasian Law Teachers
13. Landemore H (2014) Inclusive Constitution-Making: The Icelandic Experiment.
Journal of Political Philosophy
14. Gluck J, Ballou B (2014) New Technologies in Constitution Making.
15. Deely S, Nesh-Nash T (2014) The Future of Democratic Participation: An Online
Constitution Making Platform. pp 43–62
16. Obrst L, Liu H, Wray R (2003) Ontologies for Corporate Web Applications. AI
17. Hornung T, Koschmider A, Mendling J (2006) Integration of heterogeneous BPM
Schemas: The Case of XPDL and BPEL. CAiSE Forum 231:
18. Luz N, Pereira C, Silva N, et al. (2014) An Ontology for Human-Machine
Computation Workflow Specification. Lecture Notes in Artificial Intelligence
19. Constitute Project. https://www.constituteproject.org/.
20. (2010) Constitucio de Catalunya. Reagrupament.cat
21. Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2014) When is a crowd
wise? Decision 1:79.
22. Levine SS, Prietula MJ (2013) The Hazards of Interaction: Why Isolation Can
Benefit Performance. In Academy of Management Proceedings 2013:10736.
23. Ober J (2013) Democracy’s Wisdom: An Aristotelian Middle Way for Collective
Judgment. American Political Science Review 107:104–122.