Content uploaded by Humberto Rodríguez Avila
Author content
All content in this area was uploaded by Humberto Rodríguez Avila on May 14, 2019
Content may be subject to copyright.
Sparrow: A DSL for Coordinating Large Groups of
Heterogeneous Actors
Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
Vrije Universiteit Brussel
Brussels, Belgium
{rhumbert,jdekoste,wdmeuter}@vub.be
Abstract
Actor-based programming is a well-established program-
ming model for the development of concurrent and parallel
systems. However, due to the asynchronous nature of its
communication mechanism, it is often dicult to express
coordination between multiple groups of actors. As a result,
complex synchronization patterns are often dicult to ex-
press from within the actor model. The goal of this paper
is to introduce a novel coordination model for actor-based
programs. The model proposes advanced join-patterns over
messages that improve the state of the art with four addi-
tional features: aggregation, sequencing, timing constraints,
and transformations of sequence of messages. This paper also
presents a prototype implementation of our model through
a domain-specic language in Elixir.
CCS Concepts •Software and its engineering →Do-
main specic languages
;
Concurrent programming
languages;Functional languages;
Keywords
Join-Patterns, Coordination, Actors, Concur-
rency
ACM Reference Format:
Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter.
2019. Sparrow: A DSL for Coordinating Large Groups of Hetero-
geneous Actors. In Proceedings of ACM Conference (Conference’17).
ACM, New York, NY, USA, 10 pages. https://doi
.
org/10
.
1145/nnnn
nnn.nnnnnnn
1 Introduction
In the last two decades, multiple solutions have been pro-
posed to solve coordination problems in actor-based pro-
grams [
1
,
2
,
8
,
10
–
12
]. We can group these solutions into
two categories: tuple-spaces [
7
] and join-patterns [
5
]. Most
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for prot or commercial advantage and that copies bear
this notice and the full citation on the rst page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specic permission and/or a fee. Request
permissions from permissions@acm.org.
Conference’17, July 2017, Washington, DC, USA
©2019 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
https://doi.org/10.1145/nnnnnnn .nnnnnnn
of the modern solutions [
2
,
8
,
10
] target join-patterns as the
base of their coordination model. However, these solutions
lack the support of a scalable and modular way to capture
the whole coordination process. First, joins are declared stat-
ically, and their reactions (a.k.a. methods) are bound upon
pattern denition. As result, developers can only dene a
static actor interface and static reactions. Second, there is
almost no support for aggregation, sequencing, and timing
constraints of the messages involved in a join pattern. Fur-
thermore, the transformation and ltering of these messages
is limited.
This paper focuses on solving the aforementioned lim-
itations of join-based approaches. We propose Sparrow, a
domain-specic language for coordinating large groups of
heterogeneous actors in Elixir. Sparrow extends the tradi-
tional
receive
primitive of the actor-model [
9
] using a declar-
ative and modular way to dene the interface of an actor.
The main contributions of this paper are:
•
A set of six primary challenges for modern join-
patterns implementations to coordinate large groups
of heterogeneous actors (Section 2).
•
A declarative and modular coordination model for
actor-based programs supported by advanced join-
patterns (Section 3) and dynamic registration of re-
actions (Section 4).
•
A macro-based library for supporting rst class join-
patterns in Elixir (Section 5).
•
A comparison between our approach and its related
work (Section 6).
2 Motivation
This section gives an overview of the coordination problems
that can be found in actor-based programs. Furthermore,
we investigate why modern join-language implementations
fail to solve these coordination problems for large groups of
heterogeneous actors.
The actor model is a well-established concurrency model
proposed 44 years ago by Hewitt et al. [
9
]. In this paradigm,
actors are isolated entities that communicate with each other
through asynchronous messages. This asynchronous nature
of the communication mechanism makes actors very suit-
able for the development of concurrent and parallel systems.
However, at the same time, this mechanism increases the
complexity to express coordination between multiple actors.
Conference’17, July 2017, Washington, DC, USA Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
In other words, developers often face diculties to express
complex synchronization patterns from within the model.
To illustrate the kind of synchronization problems exhibited
by actor-based systems, we use Elixir
1
to solve the classic
Santa Claus problem [
13
]. We begin by introducing some
basic Elixir concepts.
Elixir is a modern functional language that runs on top
of the Erlang
2
virtual machine (BEAM). Like Erlang, Elixir
is a process/actor based language. Actors in Elixir can send
asynchronous messages using the primitive
send(id, msg)
,
which takes two arguments, namely the recipient id, and the
message. Messages are idiomatically conceived as tuples e.g.
{:register, a, b}
. In doing so, usually the rst element is an
atom that represents the message name. The other elements
play the role of the arguments. They can be static values
(atoms, strings, integers, booleans, etc.) or dynamic values
as logic variables. An actor can scan its mailbox for newly
received messages with the primitive
receive
. This function
tries to match the pattern of the received message with the
ones dened in its clauses. These clauses dene the interface
of an actor. Elixir like Erlang provides a powerful pattern
matching mechanism with support for non-linear patterns
3
.
Non-linear patterns allow developers to use the same logic
variable multiple times in a message, and therefore synchro-
nize on their bindings.
1def lo op ( ) do
2re c ei ve d o
3{: m sg 1 , a , 5, a , si d } - >
4#some code here
5se n d ( si d , : ok )
6{: ms g2 , b, t rue , s id } - >
7#some code here
8se n d ( si d , : ok )
9after
10 10 _ 0 00 -> r ai s e " No m e s s ag e s in 1 0 s "
11 end
12
13 lo op ( )
14 end
Listing 1. Basic actor interface in Elixir
Listing 1exemplies the concepts described in this section.
The
loop
function denes the interface of an actor (lines 2-11)
with two patterns or clauses. Lines 3 and 6 show how Elixir’s
pattern matching and non-linear patterns work. For example,
the rst pattern (line 3) will match against a message with
identier
:msg1
followed by four attributes. The rst and
third attribute, identied by the logic variable
a
must have
the same value, while the second one must have 5 as value.
The last attribute
sid
represents the sender of the message.
In the body of both clauses, a task will be executed, after
that a
:ok
message is sent back to the sender of the original
message. Finally, if after 10 seconds the actor does not receive
1https://elixir-lang.org
2http://erlang.org
3https://elixir-lang.org/getting-started/pattern-matching.html
any message, it will be stopped by an exception raised inside
the primitive after (lines 9-11).
2.1 A Classic Coordination Problem
The Santa Claus problem is a classic coordination problem
introduced by Trono [
13
]. It states that Santa sleeps at the
North Pole waiting to be awakened by either all of his nine
reindeer or a group of three of his ten elves. When one of the
two groups is ready, Santa will perform some work (toy R&D
with the elves and toy delivery with the reindeer) with them.
In case both groups become available at the same time, Santa
must give priority to the group of reindeer.
Even for this simple coordination problem, the solution is
far from trivial in current mainstream actor languages like
Scala, Erlang or Elixir. To illustrate this, we are assuming
that every reindeer,elf, and Santa is represented as an actor.
Available online solutions of this problem show the diculty
to express such coordination patterns.
1defp sa nt a () d o
2{ sp ec i es , gr o up } =
3re c ei ve d o
4{: r ei nd e er , g } -> { : re in de er , g}
5after 0 ->
6re c ei ve d o
7{:reindeer , g} -> {:reindeer , g }
8{: e lv es , g } -> {: e l ve s , g}
9end
10 end
11
12 case sp e ci es d o
13 : r ei n d ee r - > IO . p u ts ( " L et 's d el i ve r to ys ! ")
14 : el v e s -> IO . p u ts ( " L et 's m e et i n th e . .. !" )
15 end
16
17 for pi d <- gr oup , do: s e nd ( p id , se l f ( ) )
18 for _p id <- g ro u p do
19 re c ei ve do { : le av e , _ pi d } -> : ok end
20 end
21
22 sa nt a ()
23 end
Listing 2. A Santa Claus solution in Elixir
Listing 2shows a fragment of our solution to the Santa
Claus problem in Elixir
4
. This implementation is based on
the one proposed by Richard A. O’Keefe in Erlang
5
. This frag-
ment only shows the Santa actor interface (lines 3-10). In this
example, the actor interface is changed using nested
receive
statements to give priority to the group of reindeer. To do
this, Santa actor rst checks for the availability of a group
of nine reindeer (line 4). Otherwise, he changes his interface
(lines 6-9) to wait for one of the two groups, increasing in
this way the complexity of the program. However, all the
hard work to keep track of the number of helpers available
for each group is done by two extra actors, called secretaries.
Each secretary noties the Santa Claus actor when her group
4https://github.com/rhumbertgz/santaclaus/
5http://www.cs.otago.ac.nz/staffpriv/ok/santa/santa.erl
Sparrow: A DSL for Coordinating Large Groups of Heterogeneous Actors Conference’17, July 2017, Washington, DC, USA
is ready. In other words, the coordination logic is split across
multiple actors, making its code harder to understand and
reason about. In Elixir like in most actor-based languages,
the interface of an actor can not pattern match over mul-
tiple messages. This limitation usually leads to a complex
set of nested
receive
statements to enforce certain sequence
of messages. To solve this kind of synchronization issues
in actors, join-pattern prototypes have been designed for
languages like Erlang [
10
] and Scala [
2
,
8
]. For example,
JErlang [
10
] takes advantage of the robust Erlang’s pattern-
matching mechanism, and its support for non-linear patterns
to incorporate them in its join-pattern implementation. The
following example shows a fragment of the solution to the
Santa Claus problem using JErlang abstractions ported to
Elixir.
1defp sa nt a () d o
2gr o up =
3re c ei ve d o
4{:reindeer , pid1 } and {: reindeer , pid2} and
5{:reindeer , pid3 } and {: reindeer , pid4} and
6{:reindeer , pid5 } and {: reindeer , pid6} and
7{:reindeer , pid7 } and {: reindeer , pid8} and
8{:reindeer , pid9} ->
9IO . p u ts ( " L et 's d el i ve r to ys ! ")
10 [ pid 1 , p id2 , p id3 , p id4 , pi d5 ,
11 pi d6 , pi d7 , pi d8 , pi d9 ]
12
13 {: e lf , p i d1 } and { : el f , pi d 2 } and
14 {: e lf , p id 3 } ->
15 IO . p u ts ( " L et 's m e et i n th e . .. !" )
16 [ pid 1 , p id2 , p id 3]
17 end
18
19 for pi d <- gr oup , do: s e nd ( p id , : o k )
20
21 sa nt a ()
22 end
Listing 3. A Santa Claus JErlang-like solution in Elixir
The coordination code of this example is much more un-
derstandable. The rst join in the Santa actor interface (lines
4-8) expresses that he is waiting for a group of nine reindeer.
This is accomplished through the operator
and
to connect
patterns. On the other hand, the second join (lines 13-14)
expects a group of three elves. In JErlang, like in Elixir and
Erlang, the
receive
primitive tries to match patterns from top
to bottom. Thus, the reindeer join-pattern has priority over
the elf one, as required by the problem description.
2.2 Extended Santa Claus Problem
JErlang-like joins allow developers to declare the coordina-
tion code in one single actor, and without nested
receive
statements. Although these kind of join abstractions can be
sucient to solve simple problems (Subsection 2.1), they
have limitations for expressing coordination patterns be-
tween large groups of heterogeneous actors. The following
Extended Santa Claus problem exemplies these limitations.
Several centuries have passed since Santa Claus started his
rst journey. Nowadays, Santa is an old man who is no longer
capable of serving millions of children in one night. Therefore
Santa has invested in no less than 900 reindeer and 1000 elves.
Furthermore, he has hired 99 helpers that look and behave
exactly like him. The 100 Santas share the 900 reindeer and the
1000 elves and all display the above behaviour. But a reindeer
and an elf cannot help more than one Santa at a time. Also,
sometimes Santa and his helpers can not wait to long for a
group of reindeer/elves. If a group is not ready on time, those
members that had already arrived must be released. In this
way, they can help other Santas. Although Santa is an old man,
he is up to date with current gamication
6
techniques to im-
prove the productivity of his labor. Reindeer and elves receive
rewards for their good work. These rewards can be used by
all Santas as lters to prefer members. Nowadays, even Santa
and his helpers sometimes are aected by trac jams. In these
situations, Santas have the capabilities to change their delivery
routes to deliver children’s toys on time. On the other hand,
occasionally Santas are only interested in toy R&D meetings
with groups of elves if all their members became available con-
secutively. In other words, any reindeer can become available
while a Santa is waiting to complete his group of three elves.
The extremely complex coordination logic of the Extended
Santa Claus problem shows new challenges for join-patterns
implementations. For example:
•
Developers have to enumerate the patterns of the rein-
deer and the elves joins exhaustively even though they
are very similar (Listing 3, lines 4-8 and 13-14). This
approach suers from a lack of syntactic scalability to
dene joins for coordinating hundreds or thousands of
actors. To solve this limitation, join languages have to
support aggregation of messages (see Subsection 3.2.1).
The need of this capability can be also observed in
more realistic server-side applications like participa-
tory campaigns, routing and crew scheduling in public
transportation just to name a few.
•
Modern implementations limit the constraints of their
join-patterns to message attributes, and they do not
support timing constraints. The support of timing con-
straints (see Subsection 3.2.2) in joins allows devel-
opers to dene more dynamic and complex patterns
with timing constraints. Like in our academic example,
daily life activities of our society and economy such
as airline baggage handling, trading, and nancial ser-
vices require this kind of timing capabilities in their
environment.
•
The denition of traditional joins does not specify
any order in their messages. However, there are situa-
tions like the one in our extended Santa Claus problem
where an actor (Santa) requires an accurate sequencing
of messages (see Subsection 3.2.3). Realistic use cases
6https://en.wikipedia.org/wiki/Gamification
Conference’17, July 2017, Washington, DC, USA Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
for this requirement include patient ow monitoring
in hospitals, network surveillance and intrusion detec-
tion.
•
Current join languages do not support abstractions
to apply transformations (Subsection 3.3.1 ) and l-
ter (Subsection 3.3.2 ) operations over an incoming
sequence of messages. Sometimes programs need to
execute actions based on the overall composition of
that sequences instead of its constituents patterns. As
in our academic example, online retailers such as Ama-
zon, eBay, or media companies like Netix and Spotify
need these kinds of abstractions to provide a better
user experience based on the overall behavior of their
consumers.
•
The reuse of join-patterns denitions is not supported.
This limitation leads developers having to repeat com-
plex patterns for every join instance. In other words,
this lack increases the amount of work of program-
mers and gives them very primitive coordination ab-
stractions. Composition and abstraction of patterns (see
Subsection 3.4) are a needed abstraction to face current
development challenges.
•
Finally, a limitation of most of the contemporary ap-
proaches is that join reactions (Listing 3, lines 9-11 and
15-16) are statically bound to the denition of the join
pattern itself. Dynamic registration of reactions (see
Section 4) is almost never supported. This coupling
between the join-pattern denition and its reaction
reduces the expressiveness and modularity of the lan-
guage.
To the best of our knowledge, all these features together
have been not integrated in a join-pattern language.
3 The Sparrow Actor Coordination Model
This section gives an overview of the main features of Spar-
row to provide a modular and extensible coordination model
for actors. We introduce small dierences between the se-
mantic and syntax of Sparrow and its host language. Sparrow
was conceived as a macro-based library for the Elixir pro-
gramming language. Later, we describe the coordination
artifacts supported by Sparrow.
3.1 Basic Semantics and Syntax
In Sparrow like in Elixir actors are isolated entities that com-
municate with each other through messages. Messages are
represented as tuples, where the rst element is the message
id, followed by a set of attributes. The message id is always an
atom. These tuples are called message-patterns. Sparrow man-
ages the coordination between local and distributed actors
through an advanced join-pattern model. This model is inte-
grated into a concurrent and distributed functional language,
instead of a separate meta-level coordination language. In
Sparrow, the coordination and computational concerns are
integrated within the same programming paradigm. In this
rst proposal, as with other join-pattern languages, Spar-
row focuses only on the synchronization of messages on the
receiver side.
A traditional join-pattern is a combination of multiple
message-patterns that are connected by logic operators
(AND, OR, NOT) together with a reaction function. A reac-
tion represents the code to be executed when a full message-
pattern match occurs. In Sparrow, reactions are represented
as anonymous and regular functions, and they will be exe-
cuted asynchronously. If a regular function is used, develop-
ers have to use the capture operator
&7
to add it as reaction
to a particular join. Section 4will discuss reactions in more
in detail. The following example illustrates the denition of
a traditional join pattern that will accept three messages of
100 years old elves from the same forest. In this example, we
omitted the code of the reaction (line 8).
1{: el f , i dA , fo re st A , ag eA } and
2{: el f , i dB , fo re st B , ag eB } and
3{: el f , i dC , fo re st C , ag eC } ->
4if ( f or e st A = = fo r es tB and
5fo r es tB = = f or es t A and
6ag eA = = 10 0 and ageA == ageB and
7ageA == ageC) do
8# re a ct i on h e re
9end
Listing 4. Example of a traditional join-pattern
In this traditional join approach developers have to check
the values of the attributes forest and age of each pattern man-
ually. This type of solution becomes more complicated when
the number of constraints grows. To simply the denition of
joins, Sparrow uses the Elixir’s non-linear pattern-matching
mechanism. The following example illustrates an alterna-
tive version of Listing 4that relies on non-linear pattern-
matching. First, the example uses the same logic variable
forest in each message to guarantee that all the elves come
from the same forest. Second, it substitutes the age attribute
for its value in each message to accept only 100 years old
elves.
1{: el f , i dA , fo re st , 1 00 } and
2{: el f , i dB , fo re st , 1 00 } and
3{: el f , i dC , fo re st , 1 00 }
Listing 5.
Example of a join-pattern with non-linear
pattern-matching support
Sparrow classies joins according to their complexity in
simple and complex. The former represents the primary form
of join in Sparrow using non-linear pattern-matching like
the ones dened in Listing 5. The latter category extends
the rst one to dene more complex joins using Pattern Sets
(Subsection 3.2) and Pattern Set Transformers (Subsection 3.3).
However, all these patterns are represented as (anonymous)
7https://hexdocs.pm/elixir/Kernel.SpecialForms.html#&/1
Sparrow: A DSL for Coordinating Large Groups of Heterogeneous Actors Conference’17, July 2017, Washington, DC, USA
literals in the join expression. The key for pattern modular-
ity and reusability of our model relies on named patterns.
Named patterns can be used to compose others join-patterns
(Subsection 3.4). To dene named patterns we have to use
the construct
defpattern as
. Consequently, Sparrow’s joins
can be reused to compose more complex patterns. Listing 6
shows the main grammar behind of our model.
1 <m es sa ge > : : = '{'<t e rm > { , <t e rm > } '}'
2 < p a t t e r n > : : = < m es s ag e > { < p a t t e r n _ s e t s > <c o m p l e x _ a c t i o n s > }
3 < p a t t e r n _ s e t s > : : = '['< ps _ op > { , < p s_ o p > } ']'
4 <p s _o p > : : = < a g g r e g a t i o n > | <w in do wi ng >
5 < a g g r e g a t i o n > : : = c o u n t : < i n t e g e r >
6 <w in dow in g > : : = wind ow : '{'< i n t e g e r > , < t i m e _ u n i t > '}'
7 < t i m e _ u n i t > : : = : s e c s | : m i ns | : h o u rs | : d a y s | : we ek s
8 < p s _ t r a n s f o r m e r s > : : = | > < p st _ o p > { | > <p s t _ op >}
9 <p s t _o p > : : = <m ap _e xp r > | <f o l d r _ e x p > | < l e t _ e x p r >
10 <m ap _ex pr > : : = map ( < fu n > )
11 < f o l d r _ e x p > : : = f o l d r ( < fu n >)
12 < l e t _ e x p r > : : = l e t ( < va r > )
13 < o p e r a t o r > : : = a nd | o r | n ot
14 < c o m p o s i t e _ p a t t e r n > : : = < p a t t e r n > { < o p e r a t o r > <p a t t e r n > }
Listing 6. Sparrow’s Grammar
3.2 Pattern Sets
Pattern Sets represents the rst of four main features of Spar-
row to face the challenges described at the end of the Subsec-
tion 2.2. This feature includes abstractions for aggregation,
timing constraints, and sequencing of messages.
3.2.1 Messages Aggregation
Sparrow increases the syntactic scalability of its join-patterns
by using aggregation of messages. When a developer needs to
match a group of messages of the same type, he can use the
aggregation abstraction to avoid a lengthy representation of
all message-patterns. Sparrow provides the keyword count:
to specify the number of messages to aggregate. For example,
Listing 5could be rewritten as follows:
1{: e lf , i d , fo re s t , 1 00 } [ count: 3 ]
Listing 7. Example of a join-pattern with aggregation
This new implementation will be expanded to a join pat-
tern that is semantically equivalent to the one dened in
Listing 5. However, this naive solution has a problem be-
cause the join-pattern will only match if Santa receives three
identical elf messages. To solve this pattern-matching issue
during the aggregation of messages, Sparrow introduces the
operators ! must be distinct, and @ may be distinct. These
two operators increase the expressiveness of our model to
conceive more complex joins. The following code shows the
correct version of the join shown in Listing 7.
1{: e lf , ! id , f o re st , 10 0 }[ count: 3]
Listing 8. Example of a join using the ! operator
In this new solution, we explicitly specied that the at-
tribute
id
must be dierent in all the messages. The original
semantic for the attributes forest and age remains the same.
The expanded join of this code will be like the one in List-
ing 5.
3.2.2 Timing Constraints
Most actor-based languages do not provide bounded mail-
boxes. This fact could be a source of memory leaks for actor-
based systems. If there are no messages that match the cur-
rent actor interface, the actor mailbox grows until the mem-
ory is exhausted. To avoid this kind of issues, we limit the
lifetime of messages that sit in an actor’s mailbox. To sup-
port timing constraints, Sparrow injects a time attribute into
every message that arrives in the actor’s mailbox. Sparrow
runtime uses this implicit time-stamp to remove expired mes-
sages from the working memory of its join-patterns engine
(Subsection 5.1). The following example extends Listing 8
to illustrate the use of a timing constraint where messages
older than 30 minutes will be discarded by the join pattern.
1{: e lf , ! id , f o re st , 10 0 }
2[count: 3, w i nd o w: { 30 , : m in } ]
Listing 9. Example of a timing constraint
Notice the use of the keyword
window:
to specify the timing
constraint. This primitive receives a tuple with two attributes.
The rst one is an integer value that represents the time.
The second one is the unit of time. Our model supports the
following unit of time: sec,min,hour,day,week.
3.2.3 Messages Sequencing
Actors are not always interested in reacting to reception of
single messages. There are scenarios (Subsection 2.2 ) where
the sequencing of messages is an essential requirement. To
support this kind of behavior, we introduce
andThen
, as a se-
quencing primitive. This primitive establishes a sequential
order between messages. The following join will only accept
three elves messages followed by a reindeer message. Ad-
ditionally, both elves and reindeer must be from the same
forest and with the same age.
1{: e lf , ! id , f or es t , a ge } [ count: 3]
2an d Th en
3{: r e in d ee r , i dr , f or e st , ag e }
Listing 10. Example of messages sequencing
This example also shows that in Sparrow aggregated mes-
sages (line 1) can be sequenced (line 2) with non-aggregated
messages (line 3).
3.3 Pattern Set Transformers
Pattern sets allow developers to specify patterns that will
match as soon as the actor has received entire sequence
messages. However, they do not provide any mechanism
to transform and lter these messages. In this subsection,
we introduce Pattern Set Transformers or Transformers for
Conference’17, July 2017, Washington, DC, USA Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
short. Transformers allow developers to apply transformation
and lter abstractions to a sequence of incoming messages.
This second feature complements pattern sets to solve other
challenges dened in the Subsection 2.2.
Syntactically, transformers are dened following the pat-
tern sets (if the join has any). To express transformers devel-
opers create a ow using the pipe
|>
operator. This operator
relies on the same semantics of the Elixir’s operator
8
. In
Sparrow, the pipe operator rst requests the sequence of
messages from the pattern sets or a list with a single element
from the message-pattern of the join. This rst output will
pass through the transformer call appearing on the right
hand side. This process is repeated for every output of a
transformer function. Developers can create a big ow of
transformers using multiple instances of transformation, l-
ter abstractions, and the
|>
operator. The execution of trans-
formers follows the same order in which they were dened.
The following listing exemplies the general form of a trans-
former ow in Sparrow.
1(< me ss age - pa tt ern > | <p at ter n -s ets > )
2|> < t r an s fo r m er - 1 >
3|> < t r an s fo r m er - 2 >
4...
5|> <t r a ns f o rm e r - n >
Listing 11. General form of a transformer ow
3.3.1 Transformation Abstractions
Sparrow supports two transformation abstractions
map
and
foldr
. These abstractions are well-known functions, and they
follow the standard semantics. The following examples ex-
tend the join dened in Listing 8to exemplify the afore-
mentioned transformation abstractions. In these extended
versions there is no restriction about the age of the elves,
notice the use of the operator @.
1{: e lf , ! id , f o re st , @a g e }[ count: 3 ]
2|> map (fn { m , id , f , a } ->
3{m , i d , " S he r w oo d " , a }
4end )
Listing 12. Example of a map transformation
This rst example uses a
map
function (lines 2-4) to set
Sherwood as the value of the attribute forest in all the elves
messages. On the other hand, the next example uses a
foldr
function (lines 2-4) to sum the age of the three elves.
1{: e lf , ! id , f o re st , @a g e }[ count: 3 ]
2|> foldr (fn ( { _ ,_ , _ , a g e }, a c c ) ->
3age + acc
4end)
Listing 13. Example of a foldr transformation
8
https://elixir-lang
.
org/getting-started/enumerables- and-streams
.
html#t
he-pipe- operator
3.3.2 Filter Abstractions
Message ltering was another of the challenges described in
the Subsection 2.2. Sparrow supports two lter abstractions
(
filter
and
when
) and one auxiliary primitive (
let
) to face this
limitation. The
filter
primitive like
map
and
foldr
receives
a sequence of incoming messages and an anonymous func-
tion. However, it returns only those messages for which the
given function returns a truthy value. Like in the previous
subsection, the following examples extend the join dened
in Listing 8to exemplify the lter abstractions. Also, there is
no restriction about the age of the elves, notice the use of the
operator @. The following example denes a join that will
only accept a group of three elves younger than 100 years
old. In this example, the
filter
primitive (lines 2-4) takes a
sequence of three messages and a predicate to discard elves
older than 100 years.
1{: e lf , ! id , f o re st , @a g e }[ count: 3 ]
2|> filter (fn { _ , _ ,_ , a g e } ->
3age < 100
4end)
Listing 14. Example of use of the lter primitive
Sometimes it is useful to refer to intermediate results in
a pipe of transformer functions. To provide this functional-
ity, Sparrow introduces the
let
auxiliary primitive. The
let
primitive also takes an implicit value as its rst argument,
but the second argument is an logic variable. The primitive
will bind the value of the rst argument to the logic variable,
and return the original value. The scope of the logic variable
is the remaining denition of the join. By default, the pipe
of transformers delivers one single result to the reaction of
the join (Section 4). Nevertheless,
let
instances open a way
to provide multiple results to reactions.
Sparrow includes support for guard expressions to expand
joins with more complex logic which transcend the capabili-
ties of pattern-matching. A guard is dened by the primitive
when
, and it relies on the semantics of Elixir’s guards
9
. How-
ever, guards are limited to particular scenarios, and only a
reduced list of expressions are allowed in them. Guards can
only have access to logic variables in simple join-patterns or
logic variables dened by the
let
primitive. The following
join example uses a guard to express that the rst elf has
to be older than the second one. Although in this case, the
guard has a clause or predicate, multiple clauses can be added
using boolean operators.
1{: el f , i d1 , f1 , a1 } and
2{: el f , i d2 , f2 , a2 } when a1 > a 2
Listing 15. Example of a simple join with a guard
The next example extends Listing 13 to accept a group
of elves if their total age is greater than 300 years. In this
solution, we use the
let
abstraction to refer to the value that
9https://hexdocs.pm/elixir/master/guards.html
Sparrow: A DSL for Coordinating Large Groups of Heterogeneous Actors Conference’17, July 2017, Washington, DC, USA
returns the foldr call. After the binding of that result to the
logic variable ages (line 5), we use that variable to dene the
guard-clause (line 6) of the join.
1{: e lf , ! id , f o re st , @a g e }[ count: 3 ]
2|> foldr (fn ( { _ ,_ , _ , a g e }, a c c ) ->
3age + acc
4end)
5|> let ( ag e s )
6when ag es > 3 0 0
Listing 16. Example of a complex join with a guard
3.4 Composite Join-Patterns
Modularity and reuse of complex patterns constitute one
of the fundamental lack in current join-pattern languages
(Subsection 2.2).
Sparrow solves this problem using composite join-patterns.
This third main feature is supported by the use of named
patterns. As was introduced in the subsection 3.1, our model
provide the primitive
defpattern as
to dene such kind of
patterns. The following example shows an example of join
compositions in Sparrow. The join-pattern
elves
of this ex-
ample will match after receiving three messages from the
same elf. In line 1 of this example we dene a named-pattern
elf
. Then, we compose a join
elves
(line 2) based on the rst
one. In this example, the
elf
pattern act as a constituent
message-pattern.
1defpattern elf as { : el f , id , f or es t , ag e }
2defpattern elves as e lf [ count: 3 ]
Listing 17.
Composite joins using
defpattern as
primitive
The following example shows three more complex com-
posite join-patterns. The rst one expects three dierent
elves (line 2). The second one awaits for three dierent elves
younger than 100 years old (lines 3-4). The last join-pattern
extends the second one changing the value of the forest at-
tribute to Sherwood (lines 5-6). In the example, line 1 remains
equals to the one from Listing 17. However, the rst compos-
ite join (line 2) introduces new concepts. This join-pattern
species that the values of attribute
id
must be dierent,
while the values of
age
can be equal or not. In the line 3 a
second join is composed based on the
elves
join, and a lter
function is dened on line 4. Finally, the third join (line 5)
extends the second one (lines 3-4) adding a map function
(line 6).
1defpattern elf as { : el f , id , f or es t , ag e }
2defpattern elves as e lf { ! id , @a ge } [ count: 3 ]
3defpattern youngElves as elves
4|> filter (fn { _ , _ ,_ , a g e } -> a ge < 1 00 end )
5defpattern youngElvesFromSherwood as youngElves
6|> map (fn { m , id , f , a } -> { m ,i d , " S h er w o od " , a } end )
Listing 18. Complex composite joins in Sparrow
Composite join-patterns can be created by mixing named
and anonymous patterns. In the following example, an
anonymous
reindeer
pattern and a named
elf
pattern are
used to compose the join mix.
1defpattern elf as { : el f , id , f or e st , ag e }
2defpattern mix as e lf { ! id , @a ge } [ count: 3 ] and
3{: r e in d e er , rid , f , r a }
Listing 19. Complex composite joins in Sparrow
Developers have to be careful when composing patterns
from dierent constituent patterns. Constituents can share
the same logic variables. In these cases, the natural non-linear
patterns-matching of Sparrow can generate wrong results.
To solve possible conicts during the composition of joins,
Sparrow supports aliases for attributes. Listing 20 shows
a join example of how to use aliases in Sparrow. The join-
pattern
santaTeam
will only accept a group of three dierent
elves and a group of nine dierent reindeer from the same
forest
1defpattern elf as { : el f , id , f or e st , ag e }
2defpattern reindeer as { : r ei nd e er , id , f or es t , a ge }
3defpattern santaTeam as
4el f { ! id , @ a ge } [ count: 3 ] and
5re i n de e r { id ~ > ! ri d , a ~ > @r _ a ge } [ count: 9 ]
Listing 20. Aliases in composite joins
In this example, the santaTeam join is composed by using
two named joins dened in lines 1-2. A particular property of
both named-joins is that they share the same logic variables.
However, the problem states that the attribute values of
elf
and
reindeer
messages with exception of the
forest
attribute
have to be dierent. To solve this problem, the
santaTeam
join
species that the
id
attribute value must be dierent and
the
age
attribute value can have any value in the three
elf
messages (line 4). The same specications must be applied
to
reindeer
messages. However, as they share the same logic
variables than
elf
messages, attributes aliases are used to
avoid conicts (line 5). The alias
~>
operator takes in its left
side an attribute identier and renamed it to the identier
in its right side. During alias operations, developers can use
! and @operators.
4 Dynamic Registration of Reactions
Our exposition of Sparrow so far has been focusing on the
denition of join-patterns, instead of the reaction function
to be executed when a full message-pattern match occurs.
However, as we expressed in the subsection 2.2 the registry
of reactions is one the limitations of the expressiveness and
modularity of modern join languages. This section gives an
overview of how Sparrow solves this limitation using a dy-
namic registry of reactions. This last feature of our model
allows a total decoupling between the join denition and
the registry of its reaction. In Sparrow, developers can even
register multiple reactions to a join-pattern. Furthermore,
Conference’17, July 2017, Washington, DC, USA Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
a reaction function can be reused by multiple joins. Reac-
tions are dened using the
react_to with:
construct. This
construct takes as rst argument a reference to a particular
join-pattern (that was composed with the abstractions ex-
plained in Section 3), while the second argument represents
the action to bind to the join. In the same way, reactions
can be removed by the use of the
remove_reaction from:
and
remove_all_reactions
constructs. The former removes a partic-
ular reaction from a specic join-pattern. The latter removes
all reactions from a particular join-pattern.
The interface of an actor in languages like Scala, Erlang,
Elixir, and JErlang is statically dened, and the order of check-
ing the patterns follows the order in which they were de-
ned (top-down). However, in Sparrow, the interface of an
actor is dynamically dened using the
defpattern as
primi-
tive. This behavior implies that if at a certain point of the
execution of a program multiple patterns can be selected, the
runtime will choose one of them in a non-deterministic way.
Although Sparrow follows a non-deterministic selection pro-
cess by default, developers can specify a particular order
using the
match
primitive. This construct takes a sequence
of join-patterns references separated by the logic operator
or
(e.g.
match firstJoin or secondJoin or thirdJoin
). If the
match
primitive is used, all join-patterns must be dened at compi-
lation time. In other words, new joins can not be added at
runtime.
Listing 21 shows a solution to the classic Santa Claus prob-
lem (Subsection 2.1) in Sparrow, which exemplies most of
the main features of our model. In the solution, we dene a
module
Santa
that imports the join abstractions of our model
(line 2). The module
JoinPatterns
allows developers to dene
an actor using the Sparrow abstractions. This module fol-
lows some of the design principles of generic behaviours
frequently used in the Erlang/OTP
10
framework. To imple-
ment the logic of the traditional Santa Claus problem, we
dene two composite patterns (lines 9-12). As in the original
implementation (Subsection 2.1), in both cases the join reac-
tion only needs the member
ids
of the group. To obtain the
ids
from the message-patterns, we use a map function (line
10 and 12). Using the
react_to with:
primitive (lines 15-16)
we add the reactions functions (lines 22-13) to both join-
patterns. Finally, to give the priority to the group of reindeer
we use the primitive match (line 19).
1defmodule Santa do
2use Sp a rr o w . Jo i nP a tt e rn s
3
4# ba se pa t te r ns
5defpattern elf as { : el f , i d }
6defpattern reindeer as { : r ei n de e r , i d }
7
8# composite joins
9defpattern el f Te am a s el f { ! id } [ count: 3]
10 |> map (fn { _ , id } - > id end )
11 defpattern reindeerTeam as r e in d ee r { ! id } [ count: 9]
12 |> map (fn { _ , id } - > id end )
10http://learnyousomeerlang.com/what-is-otp
13
14 # join reactions
15 react_to elfTeam , with: &discuss_toys_rd /1
16 react_to reindeerTeam , with: & d el i ve r _t o ys / 1
17
18 # se t m at c hi n g or d er
19 match reindeerTeam or e lf T ea m
20
21 # re a ct i on f u nc t io n s
22 def de l i ve r _ to y s ( i ds ) , do: # co de l og i c he re
23 def discuss_toys_rd(ids), do: # c od e lo g ic h er e
24
25 end
Listing 21.
Solution to the Santa Claus problem in
Sparrow
5 Prototype Implementation
This section discusses an implementation of our model using
meta-programming techniques and the native non-linear
pattern-matching of Elixir. The meta-programming approach
of Sparrow has two main objectives. First, we want to provide
a seamless integration with our host language. In this way,
developers do not need of specic compilers or extensions
to use our coordination model. Second, meta-programming
allows us to optimize our code by static transformations.
Consequently, our implementation can be used as a library,
but it is implemented as a compile-time transformation.
Sparrow denes ve primary constructs to perform coordi-
nations actions (
defpattern as
,
react_to with:
,
remove_reaction
from:
,
remove_all_reactions
,
match
). Sparrow constructs were
implemented using the meta-programming techniques sup-
ported by Elixir. Although the implementation of our model
takes advantage of the resources available in Elixir, its coor-
dination abstractions can be integrated into other existing
actor languages and frameworks. However, in some cases,
the expressiveness of the join-patterns can be limited by the
absence of the support of non-linear pattern matching.
In summary, by leveraging on the powerful Elixir’s macro
system, we aim to nd a sweet-spot between a dedicated
compile approach, and the exibility of libraries.
5.1 Join-Patterns Engine
In this subsection, we give an overview of how we solved in
Sparrow the limitation to match multiple messages in the
receive
primitive. To address this issue, Sparrow does not
modify the BEAM. Instead, it incorporates a virtual mailbox
to each actor. Sparrow uses a directed acyclic graph (DAG) for
its execution engine to support the matching mechanism of
the virtual mailbox, which is based on the ideas of the RETE
[
4
] algorithm initially designed for an ecient resolution
of production rules systems. Sparrow extends the original
approach and creates an optimized RETE-like network to
represent its join-patterns.
The virtual mailbox represents the knowledge base (work-
ing memory) of our RETE-based algorithm. Every time a new
Sparrow: A DSL for Coordinating Large Groups of Heterogeneous Actors Conference’17, July 2017, Washington, DC, USA
message arrive in the Elixir actor’s mailbox, it is transfered
to the virtual one. Our implementation preserves the order
in which the messages arrived to the original mailbox, and
it respects the no-shared memory principle of actor model.
Each message in the virtual mailbox has a lifetime. This tim-
ing constraint is twofold. First, it solves problems related to
overow of messages found in most actor’s mailbox imple-
mentations. Second, this approach allows new joins-patterns
to match with previously arrived messages. Developers can
establish a default lifetime for the messages according to the
requirements of their systems.
Sparrow’s engine uses four main types of nodes to rep-
resent the join-patterns. Like the original RETE algorithm,
it has alpha,beta and terminal nodes. Alpha nodes perform
matching tests over the messages available in the working
memory. Whereas, beta nodes perform matching tests for
consistency in join-patterns. Finally, terminal nodes repre-
sent the reactions of a particular join. A particular feature
of our algorithm is that terminal nodes can contain multiple
reactions. Also, our algorithm includes a new type of node,
called omega. Omega nodes are responsible for executing
pattern sets and pattern set transformers operations. The
reduction rules of both alpha and beta nodes of our network
are performed using anonymous functions (lambdas). The
header of a lambda represents a message-pattern. However,
headers of beta nodes lambdas contain multiple message-
patterns. In this way, we take advantage of the native pattern
matching mechanism of Elixir to execute the reduction phase
of our algorithm. The following example shows the internal
representation of the simple a join-pattern in Sparrow.
1# joi n - p a tt er n
2{: e lf , n e , f , ae } and {: re i n de e r , nr , f , a r }
3--------------------------------------------------
4# al ph a - n od e s
5fn { : el f , ne , f , ae } - > tr u e end
6fn { : r ei n de e r , nr , f , a r } -> t ru e end
7# bet a - n o de
8fn ( {: e lf , n e , f, a e} , { : re in d ee r , nr , f , a r }) - >
9true
10 end
Listing 22. Internal representation of a Join-Pattern
To sum up, Sparrow engine follows a RETE-based approach
to optimize the representation of its join-patterns. Further-
more, the use of lambdas as test conditions of our join net-
work, allow us to represent complex patterns in a simple
way.
6 Related Work
This section provides a comparison between Sparrow and
related coordination approaches. These approaches can be
classied in two categories: receiver and sender side. The
former category focuses on abstractions to synchronize in-
coming messages generated by one or multiple actors in a
consumer-producer style (e.g., Sparrow, [
2
,
3
,
8
,
10
,
14
]). The
latter category, instead, focuses on abstractions to synchro-
nize reply messages generated by one or multiple actors as
a response to a request of a sender actor (e.g., [5,6]).
Haller and Van Cutsem [
8
] introduce a join-pattern library
for synchronizing multiple synchronous events in Scala Ac-
tors. Dolev [
2
] also targets Scala as the host language for
his macro-based library. This library focus on coordination
of asynchronous stream of events for Rx. Although both
implementations embrace an extensible pattern-matching
approach, they do not support non-linear patterns. Further-
more, they do not provide any kind of pattern sets and pat-
tern set transformers available in Sparrow. However, Dolev’s
work has two main contributions. First, the support both de-
terministic and nondeterministic join pattern-matching selec-
tion. Second, his library provides a choice to specify the use
of a bounded (xed amount of messages) or an unbounded
mailbox (default behaviour). Sparrow supports bounded mail-
boxes by time, but not by the number of messages. We agree
that both approaches have particular advantages and disad-
vantages. However, we are targeting scenarios where timing
constraints are needed.
JErlang [
10
] targets alternately the Erlang VM for its join-
patterns implementation. JErlang provides both a library and
a dedicated compiler approach. Like in JErlang, we exploit
the native pattern-matching mechanism and the support of
non-linear patterns available in our host language. However,
we go a step further to support aggregations and sequencing
of messages, timing constraints, and transformations of an
incoming sequence of messages. Although Sparrow does not
support message propagation [
10
], a message can be reused
for multiple joins while it is available in the engine working
memory.
Eugster and Jayaram [
3
] extend Java to support generic
event-based distributed programming. EventJava does not
target an actor-based language, but it supports primitives for
aggregation and sequencing of messages. EventJava does not
follow a pattern-matching approach, but it supports guards.
Furthermore, it supports unicast and broadcast messages.
While Sparrow only provides to developers the traditional
unicast actor communication mechanism. However, Sparrow
provides a more robust and declarative way for aggregation
and transformation of messages than EventJava. Addition-
ally, EventJava does not implement any feature for timing
constraints.
Van Ham et al.[
14
] introduce JEscala, as a modular and
declarative event system with join abstractions. JEscala uses
concepts from aspect-oriented programming and join pat-
terns to capture the coordination logic of a thread-based
system. Furthermore, this implementation adds a dynamic
registration of reactions. This key feature is not supported by
any of the approaches above, except for Sparrow. However,
events and their event reactions in JEscala can be dened
in dierent classes. A limitation of JEscala events, is that
Conference’17, July 2017, Washington, DC, USA Humberto Rodriguez Avila, Joeri De Koster, Wolfgang De Meuter
they only represent a state. Furthermore, operations like ag-
gregation, sequencing, and timing constraints of messages
are not supported. Also, the transformation of an incoming
sequence of messages is limited.
Frølund and Agha [
6
] introduce an object-oriented com-
munication model to support synchronization over a set of
messages based on activators. Activators allow both receiver
and sender synchronization of messages. However, they do
not support non-linear pattern matching, pattern sets, and
pattern set transformation. Furthermore, since activators are
anonymous expressions, they cannot be reused to compose
other activators.
SALSA [
15
] provides three sender-side coordination ab-
stractions based on continuations: token-passing, joins, and
rst-class continuations. Future work on Sparrow could sup-
port the abstractions above. By doing this, they could benet
from aggregation, transformation, and timing constraints,
which are features that are currently unavailable for contin-
uations in SALSA.
7 Conclusion & Future Work
The combination of operations like aggregation, sequencing,
and timing constraints of messages increase the syntactic
scalability of the coordination constructs of a join-based lan-
guage. Furthermore, a number of complex messages transfor-
mation and lter functions can complement the aforemen-
tioned operations to conceive more expressive coordination
abstractions. Named join-patterns and a dynamic registry of
its reactions allows developers the denition of an expres-
sive and modular interface of actors. The meta-programming
techniques available in Elixir decreased the time and eort
to implement these coordination abstractions in our DSL.
Our future work will involve the utilization of the pro-
posed model in the development of a more realistic appli-
cation. Furthermore, we plan to realize a set a benchmarks
to evaluate the scalability of our implementation. We are
also interested in exploring the use of our join-patterns in
scenarios further than the reception of messages by an actor,
like the emission of broadcast messages between actors.
Acknowledgments
We would like to thank the reviewers of this paper for their
comments. Humberto Rodriguez Avila is supported by the
FWO-SBO-SMILE-IT project (8B02), funded by the Research
Foundation Flanders (FWO).
References
[1]
Gul Agha and Christian J. Callsen. 1993. ActorSpace: An Open Dis-
tributed Programming Paradigm. In Proceedings of the Fourth ACM
SIGPLAN Symposium on Principles and Practice of Parallel Programming
(PPOPP ’93). ACM, New York, NY, USA, 23–32.
[2]
Alon Dolev. 2016. Scalable Join-Pattern Matching for Observables.
Master’s thesis. Department of Software Technology, University of
Technology Delft.
[3]
Patrick Eugster and K. R. Jayaram. 2009. EventJava: An Extension of
Java for Event Correlation. Springer Berlin Heidelberg, Berlin, Heidel-
berg, 570–594.
[4]
Charles L. Forgy. 1982. Rete: A fast algorithm for the many pattern/-
many object pattern match problem. Articial Intelligence 19, 1 (1982),
17–37.
[5]
Cédric Fournet and Georges Gonthier. 1996. The Reexive CHAM and
the Join-calculus. In Proceedings of the 23rd ACM SIGPLAN-SIGACT
Symposium on Principles of Programming Languages (POPL ’96). ACM,
New York, NY, USA, 372–385.
[6]
Svend Frølund and Gul Agha. 1995. Abstracting interactions based on
message sets. In Object-Based Models and Languages for Concurrent
Systems, Paolo Ciancarini, Oscar Nierstrasz, and Akinori Yonezawa
(Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 107–124.
[7]
David Gelernter. 1985. Generative communication in Linda. ACM
Transactions on Programming Languages and Systems (TOPLAS) 7, 1
(Jan. 1985), 80–112.
[8]
Philipp Haller and Tom Van Cutsem. 2008. Implementing Joins Using
Extensible Pattern Matching. In Coordination Models and Languages.
Springer Berlin Heidelberg, Berlin, Heidelberg, 135–152.
[9]
Carl Hewitt, Peter Bishop, and Richard Steiger. 1973. A Universal
Modular ACTOR Formalism for Articial Intelligence. In IJCAI’73:
Proceedings of the 3rd International Joint Conference on Articial Intel-
ligence. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA,
235–245.
[10]
Hubert Plociniczak and Susan Eisenbach. 2010. JErlang: Erlang with
Joins. In Coordination Models and Languages: 12th International Con-
ference, COORDINATION 2010, Amsterdam, The Netherlands, June 7-9,
2010. Proceedings, Dave Clarke and Gul Agha (Eds.). Springer Berlin
Heidelberg, Berlin, Heidelberg, 61–75.
[11]
Christophe Scholliers, Elisa Gonzalez Boix, and Wolfgang De Meuter.
2009. Totam: Scoped tuples for the ambient. Electronic Communications
of the EASST 19 (2009).
[12]
Ignacio Solla Paula. 2010. JCThorn: Extending Thorn with joins and
chords. Master’s thesis. Department of Computing, Imperial College
London.
[13]
John A. Trono. 1994. A New Exercise in Concurrency. SIGCSE Bull.
26, 3 (Sept. 1994), 8–10.
[14]
Jurgen M Van Ham, Guido Salvaneschi, Mira Mezini, and Jacques Noyé.
2014. JEScala: Modular Coordination with Declarative Events and Joins.
In Proceedings of the 13th International Conference on Modularity. ACM,
New York, NY, USA, 205–216.
[15]
Carlos Varela and Gul Agha. 2001. Programming Dynamically Recon-
gurable Open Systems with SALSA. SIGPLAN Not. 36, 12 (Dec. 2001),
20–34.