Content uploaded by Anke Reinschluessel
Author content
All content in this area was uploaded by Anke Reinschluessel on Mar 20, 2018
Content may be subject to copyright.
Beyond Math Manipulatives: Smart
Tangible Objects for Algebra Learning
Anke V. Reinschluessel
University of Bremen
Digital Media Lab
avr@uni-bremen.de
Danny Thieme
University of Bremen
danny1@uni-bremen.de
Tanja Döring
University of Bremen
Digital Media Lab
tanja.doering@uni-bremen.de
Rainer Malaka
University of Bremen
Digital Media Lab
malaka@tzi.de
Dmitry Alexandrovsky
University of Bremen
Digital Media Lab
dimi@uni-bremen.de
Copyright © 2018 for this paper held by its author(s). Copying permitted for private
and academic purposes.
Abstract
This workshop position paper presents ongoing research
on using smart tangible objects for algebra learning. While
mathematical manipulatives have played an important role
in children’s mathematics development for decades, em-
ploying tangible objects in the classroom has been rarely
explored yet. In our work, we investigate the potentials of
using smart objects for algebra learning. Our smart tiles
are based on traditional algebra tiles, passive mathematical
manipulatives used in many schools in Northern America,
and we currently extend these by 1.) multimodal input and
output capabilities, 2.) dynamic constraints and 3.) adap-
tivity and feedback. In this paper, we give an overview on
the overall system concept, the interaction with the tangible
objects and their current design, as well as on the potentials
of actuated smart objects for future interaction.
Author Keywords
Tangible user interface; smart objects; tabletop interaction;
embodied interaction; multimodal feedback; collaborative
learning; adaptive system.
ACM Classification Keywords
H.5.2 [Information Interfaces and Presentation (e.g. HCI)]:
User Interfaces
Introduction
In math education, simple passive manipulatives provide
valuable “hands-on” approaches to teach students ab-
stract concepts, especially when the students start to learn
a novel unit of math, e.g., arithmetic, geometry, or alge-
bra. These approaches are in-line with models from didac-
tics like Bruner’s concrete-representational-abstract ap-
proach [3] or the constructivistic objects-to-think-with ap-
proach [17] that suggest to use physical objects for abstract
concepts, especially for beginners. While a considerable
body of research on using tangible user interfaces (TUIs)
for learning has been conducted, more research efforts
are needed to address the question how tangible user in-
terfaces can be made smarter in order to facilitate a better
learning environment and better support for learners.
In our research, we investigate the potentials of smart ob-
jects for learning. The objects are based on traditional al-
gebra tiles, which are passive mathematical manipulatives
(see Fig. 1) as used in many schools in Northern America
to support algebra learning. We are extending these tiles to
smart “tiles” by 1.) multimodal input and output capabilities,
e.g., light and display, 2.) dynamic constraints, e.g., electro-
magnets attracting or repelling objects, and 3.) adaptivity
and feedback, e.g., user support and hints.
In educational research tactile models are common, as
for example the ones by Bruner [3] or Kieran [10]. They
showed that by using physical objects it is possible to teach
already small children mathematical, in particular alge-
braic concepts. Common digital learning platforms, such
as Dragonbox1lack the haptic and tactile components
and therefore the benefits that come with tangibility. By
transforming the algebra tiles to smart objects we want to
1http://dragonbox.com/
Figure 1: Algebra Tiles as commonly used in Northern America.
The set consists of three types of objects: small squares represent
constant values of 1and −1; elongated rectangles stand for
positive and negative variables (x); large squares represent
squared variables (x2). The sign of an object is shown through
color. All objects consist of one red surface, representing the
negative value and a unique color for each object type depicting
the positive value.
combine the richer feedback that can be provided by digi-
tal platforms with the benefits of tactile interaction, where
the smart tiles themselves create dynamic constraints and
allow for multimodal input and output, also in combination
with a touch screen.
Figure 2: Left: early system version with simple tiles. The tiles are made of transparent acrylic glass with reacTIVision markers; Right: latest
version with smart tiles with electromagnets and visual output.
Related Work
Figure 3: Solving steps of task
3 + (−x)−5 = xwith algebra
tiles.
Tangible user interfaces emerged in the 1990s, as Ishii and
Ullmer [9] describe their vision for “tangible bits”. The gap
between the physical world and the digital world should be
closed by allowing users to directly manipulate these bits,
which can be everyday physical objects. Early examples
are the metaDesk [20], the transBOARD [8] and the Urban
Planning Workbench [21]. Since then, a popular applica-
tion domain for TUIs has been learning. Examples for using
tangibles for math learning have been provided by Falc¯
ao
et al. [5], Girouard et al. [6], Manches and O’Malley [12],
and Marichal et al. [13], amongst many others. Others like
Rick [19] incorporate touch to be able to directly manipu-
late math objects presented on a screen. Research about
how tangibles can support learning or how learning theories
can inform tangible development is for example presented
by the “Tangible Interaction Framework” by Hornecker and
Buur [7] or the “Tangible Learning Design Framework” by
Antle and Wise [1]. They propose design principles for tan-
gibles and a taxonomy about the relationship between TUIs,
interactions and learning. Fur thermore Marshall [14] and
Marshall, Price and Rogers [15] critically discuss how tangi-
bles can support learning.
Examples for technological approaches for smart objects
are Sifteos (previously Siftables) [16], small objects which
have all technology needed inside that react to each other
and encourage interaction with the objects themselves. Sif-
teo cubes were launched as product in 2011 but are not
available anymore. A newer approach are the Actibles [4],
which are tangibles with a smartwatch core and light feed-
back. They allow a variety of interactions, including shaking,
tilting, stacking and neighbouring.
Multimodal Algebra Learning with Tiles
Algebra tiles as shown in Fig. 1 consist of three types of
tiles: single units used as “ones”, x-tiles and x2-tiles. Each
tile has a positive and a negative side, whereby the neg-
ative side is colored red and the positive value is repre-
sented by a unique color of that object. The tiles are typi-
cally placed on a 2×2 area (compare Fig. 2), where the two
squares on the left represent the left side of an equation
and the two squares on the right represent the right side
of the equation. The top square on both sides is the “ad-
dition zone”, i.e., all tiles there are connected by addition,
while the lower areas are the “subtraction zones”, i.e., all
tiles there are subtracted from the top ones. An equation,
for example 3+(−x)−5 = x, is put up in tiles (see top
image in Fig. 3). There would be three positive ones and
one negative x-tile in the addition section on the left side,
and five positive ones in the subtraction zone. On the right
side there is just one positive x-tile in the addition zone.
The model comes with a set of possible actions that cor-
respond to typical algebraic manipulations, when dealing
with linear equations (addition, subtraction, multiplication
and division). Generally, the goal is to apply a sequence of
legal actions in order to transform the equation to another
form or to isolate the x-tile on one side and the one-tiles on
the other side. With the traditional tiles, a student can trans-
form the equation, but does not get any feedback about the
correctness of the actions. A teacher is still necessary to
verify them. On the contrary, in our approach, the algebra
tiles themselves are aware of the equation they are part of
and give feedback or hints about the steps. They can verify
the result of a student’s actions or support grouping of tiles
with visual feedback (see subsection Multimodal Input and
Output for more details). In combination with a touchscreen
and the capacity to track the tiles, multimodal algebra learn-
ing is possible, as the benefits of the tangible tiles are com-
bined with the rich feedback such a system can provide.
Figure 4: Smart Tiles unattached
with magnets and connection
areas
Figure 5: Smart Tiles attached
with with magnets and light
feedback
Interaction with Smart Tangible Objects
In our current setup, the smart tangible objects are placed
on an interactive tabletop, where they are identified, located
and tracked with regard to orientation. This way, they are
integrated into a system that also uses the tabletop surface
for input and output capabilities (e.g., visual feedback and
multi-touch interaction). While we used passive tiles in early
versions of our systems (see Fig. 2, left side), the smart
tiles (see Fig. 4,5,6,7) are designed to provide rich interac-
tion capabilities themselves as this enhances the learning
environment and supports learners. Furthermore, this ap-
proach would also allow for a setting in which the smart tiles
are used fully functional on their own without a multi-touch
table. The intelligence situated in our tangible objects and
their surrounding system currently improves the interaction
by three approaches: multimodal input and output capabili-
ties,dynamic constraints, and adaptivity and feedback.
Multimodal Input and Output
Multimodal input and output facilities of learning systems
can enhance learning experiences, as more senses are
involved in the learning process. Our tangibles follow this
approach by allowing direct haptic interaction with the tiles,
which can be moved around and placed in different areas
of the system for input, i.e., to perform algebraic opera-
tions. Moreover, the objects directly give visual feedback,
for which we developed a model with two kinds of visual
feedback with central display and edge light feedback (see
also [4, 2] for a similar approach with low-resolution edge
displays). With this approach, the smart tiles can display
their current state in the center, e.g., the current value a
tile presents (see Fig. 6 and Fig. 7). At the same time they
communicate feedback on the performed operations via
edge light animations, e.g., if the moves were correct or
about current relations to surrounding tiles such as group-
ing of tiles where tiles are combined to one unit. Other out-
put modalities such as sound and vibration feedback are
currently tested.
Dynamic Constraints
Our smart objects are designed to provide dynamic con-
straints that guide the interaction. In our system, the dy-
namic constraints can direct the grouping of tiles, which is
an essential action for performing operations and thus for
transforming and solving equations, they can also prevent
wrong combination of tiles (e.g., placing a tile of unit one
along the long side of an x-tile). The dynamic constraints
change according to the current value of each tile and the
possible combinations. We realized the dynamic constraints
by adding neodymium magnets and electromagnets to the
sides of the objects. When the electromagnets are switched
off, the neodymium magnets repel two objects. In case the
electromagnets are switched on, realized by closed circuits
when two fitting objects have contact, the objects attract
each other (see next section for further information). With
this approach, we can also stack objects. Furthermore, this
physical grouping also gives a great physical representation
of grouped units and enhances the haptic interaction.
Figure 6: Smart tile with display
showing value +1
Figure 7: Smart tile with display
showing value −1
Adaptivity and Feedback
Among the advantages of our system is that it can be adapted
and, to some degree, can automatically adapt to learners
with different levels and needs with regard to feedback and
hints provided. Partly, this is directly provided by the smart
tiles themselves via display and light feedback as well as
magnetic hints, partly this is currently communicated by
the surrounding system, in our case the interactive table-
top system. In order to make the system smart and allow
for good feedback and hints, we have integrated and fur-
ther employ a number of approaches. One of these is us-
ing Wolfram Alpha as computation knowledge engine2that
computes algebraic transformations and allows for feedback
if an operation is a useful move towards the result for exam-
ple. Moreover, machine learning approaches can support
2https://www.wolfram.com/engine/
the identification of typical errors and the integration of use-
ful feedback, which will also improve the feedback the tiles
communicate directly. A central challenge lies in finding a
good balance of learner level and adequate feedback and
hints.
Current Design of the Smart Tangible Objects
For our tangible learning system with algebra tiles we started
with passive tiles and optical tracking. Using the reacTIVi-
sion3framework we had an early setup with tracking from
below to avoid occlusion problems. Currently we are work-
ing on capacitive tracking in combination with motion sen-
sors to enable working on touch screens like the Microsoft
Surface Hub4. The underlying software on the touchscreen
is programmed in Unity5and for supporting the equation
solving we are using the Wolfram Alpha API.
Figures 6 and 7 show the current design of our smart tiles
with center display, edge RGB lights, and electromagnetic
dynamic constraints. Currently, the tiles have a size of
7x7x5cm (width ×depth ×height) and contain Arduino
Atmega328P CH340 boards, which communicate via WIFI
with the central system. Next to a matrix LED display they
contain 12 RGB LEDS for edge light feedback. The dy-
namic constraints are realized by a combination of eight
neodymium magnets in den corners of the tiles and four
electromagnets at the edges. By default, the electromag-
nets are switched off, so that the neodymium magnets re-
pel objects placed next to each other. When two tiles fit
together, the electromagnets at the edges of the tiles (see
Figure 5) attract each other (being stronger than the re-
pelling neodymium magnets), as they are switched on when
3http://reactivision.sourceforge.net/
4https://www.microsoft.com/en-us/surface/devices/
surface-hub/overview
5https://unity3d.com/
the tiles have contact and close a circuit through conductive
contacts. These contacts only close a circuit when the pair-
ing is allowed, which can be changed dynamically. If com-
bined, the two objects stick together and represent a unit.
Through the neodymium magnets stacking tiles is generally
also possible.
A concrete example for the use case of the magnets at-
tracting would be two tiles of opposite value, e.g., +1 and
−1, which can be paired as an “zero pair” and be removed
from the working area, as they cancel out. Additionally we
have light feedback to support the same pairing process
and grouping. With the light feedback one smart tile can
support more advanced learners, which already know that
+1 on the left side on an equation can be moved to right
side and then results in an −1. Smart tiles can automati-
cally change color to show this sign change.
Envisioning Future Smart Tangible Objects
The current setup and underlying model have some lim-
itations that could be addressed by actuated tangibles.
During solving, especially children tend to compare the
resulting x-tile in size with the one-tiles. For the future,
smart x-tiles that could change their shape regarding size
would be beneficial to this step, after the equation is solved.
Additionally it may occur the case that the user is able to
see the solution even though there is still more than one
x-tile on the area. Being able to change the size for all x-
tiles synchronously would enhance the effect that all x are
the same. Thus, exploring shape change (c.f. the design
space of shape-changing interfaces by [18]) for tangible
presentations of variables in algebraic expressions would
be valuable. Another case is that especially after a divi-
sion it happens that multiple objects need to be removed at
once, which tends to be bothersome for the user. Here ac-
tuated smart tiles would do the trick and after performing a
division, they could automatically remove themselves from
the working area. Vice versa, for multiplication they would
automatically enter the working area. Also in case of pro-
viding help they could rearrange to give another view on the
equation. Approaches like used in small swarm robots as
the Zooid Swarm Robots [11] could be applied to make the
tiles and the interactions smarter.
Conclusion
In this paper, we presented our approach for smart tangible
objects to support algebra learning. Starting from traditional
passive math manipulatives, we developed a concept to
make these learning objects more intelligent in order to pro-
vide better learning environments that allow for rich and
multimodal interactions, dynamic constraints, as well as
adaptivity and feedback. We presented our current proto-
type, including the overall math system and the design of
the smart objects. Furthermore, we discussed strategies to
design even smarter tangible objects, which would address
some of the current limitations by realizing actuation, both
with regard to shape-change as well as to self-moving tiles.
Overall, while our ongoing work focuses on providing a con-
tribution to designing smart tangible objects for learning
scenarios, our approaches to make the interaction and the
objects smarter could also be valuable for other application
contexts.
REFERENCES
1. Alissa N. Antle and Alyssa F. Wise. 2013. Getting Down
to Details: Using Theories of Cognition and Learning to
Inform Tangible User Interface Design. Interacting with
Computers 25, 1 (Jan. 2013), 1–20. DOI:
http://dx.doi.org/10.1093/iwc/iws007
2. Ahmed Sabbir Arif, Brien East, Sean DeLong, Roozbeh
Manshaei, Apurva Gupta, Manasvi Lalwani, and Ali
Mazalek. 2017. Extending the Design Space of
Tangible Objects via Low-Resolution Edge Displays.
ACM Press, 481–488. DOI:
http://dx.doi.org/10.1145/3024969.3025078
3. Jerome Seymour Bruner. 1966. Toward a theory of
instruction. Vol. 59. Harvard University Press.
4. Brien East, Sean DeLong, Roozbeh Manshaei, Ahmed
Arif, and Ali Mazalek. 2016. Actibles: Open Source
Active Tangibles. In Proceedings of the 2016 ACM
International Conference on Interactive Surfaces and
Spaces (ISS ’16). ACM, New York, NY, USA, 469–472.
DOI:http://dx.doi.org/10.1145/2992154.2996874
5. Taciana Pontual Falcao, Luciano Meira, and
Alex Sandro Gomes. 2007. Designing tangible
interfaces for mathematics learning in elementary
school. In Proceedings of III Latin American conference
on human-computer interaction.
6. Audrey Girouard, Erin Treacy Solovey, Leanne M.
Hirshfield, Stacey Ecott, Orit Shaer, and Robert J. K.
Jacob. 2007. Smart Blocks: A Tangible Mathematical
Manipulative. In Proceedings of the 1st International
Conference on Tangible and Embedded Interaction
(TEI ’07). ACM, New York, NY, USA, 183–186. DOI:
http://dx.doi.org/10.1145/1226969.1227007
7. Eva Hornecker and Jacob Buur. 2006. Getting a Grip
on Tangible Interaction: A Frameowrk on Physical
Space and Social Interaction. In CHI 2006. ACM Press,
Montréal, Québec, Canada.
8. Hiroshi Ishii. 2008. The Tangible User Interface and Its
Evolution. Commun. ACM 51, 6 (June 2008), 32–36.
DOI:http://dx.doi.org/10.1145/1349026.1349034
9. Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits:
towards seamless interfaces between people, bits and
atoms. In Proceedings of the ACM SIGCHI Conference
on Human factors in computing systems. ACM,
234–241.
10. C Kieran. 2007. Learning and teaching algebra at the
middle school through college levels. I: Lester, FK Jr.
Second Handbook of Research on Mathematics
Teaching and Learning (2007).
11. Mathieu Le Goc, Lawrence H Kim, Ali Parsaei,
Jean-Daniel Fekete, Pierre Dragicevic, and Sean
Follmer. 2016. Zooids: Building blocks for swarm user
interfaces. In Proceedings of the 29th Annual
Symposium on User Interface Software and
Technology. ACM, 97–109.
12. Andrew Manches and Claire O’Malley. 2012. Tangibles
for Learning: A Representational Analysis of Physical
Manipulation. Personal Ubiquitous Comput. 16, 4 (April
2012), 405–419. DOI:
http://dx.doi.org/10.1007/s00779-011-0406-0
13. Sebastián Marichal, Anadrea Rosales,
Fernando Gonzalez Perilli, Ana Cristina Pires, Ewelina
Bakala, Gustavo Sansone, and Josep Blat. 2017.
CETA: Designing Mixed-reality Tangible Interaction to
Enhance Mathematical Learning. In Proceedings of the
19th International Conference on Human-Computer
Interaction with Mobile Devices and Services
(MobileHCI ’17). ACM, New York, NY, USA, Article 29,
13 pages. DOI:
http://dx.doi.org/10.1145/3098279.3098536
14. Paul Marshall. 2007. Do tangible interfaces enhave
learning. In TEI ’07: Proceedings of the 1st
international conference on Tangible and embedded
interaction, Brygg Ulmer and Association for
Computing Machinery (Eds.). Curran, Red Hook, NY.
OCLC: 254584537.
15. Paul Marshall, Sara Price, and Yvonne Rogers. 2004.
Conceptualising tangibles to support learning. In
Proceedings of the 2003 Conference on Interaction
Design and Children: 2003, Preston, England, July
01-03, 2003, Conference on Interaction Design and
Children, Stuart MacFarlane, ACM Digital Library, and
Association for Computing Machinery (Eds.).
Association for Computing Machinery, New York, N.Y.
OCLC: 612478047.
16. David Merrill, Jeevan Kalanithi, and Pattie Maes. 2007.
Siftables: Towards Sensor Network User Interfaces. In
Proceedings of the 1st International Conference on
Tangible and Embedded Interaction (TEI ’07). ACM,
New York, NY, USA, 75–78. DOI:
http://dx.doi.org/10.1145/1226969.1226984
17. Seymour Papert and Idit Harel. 1991. Constructionism.
Norwood. (1991).
18. Majken K. Rasmussen, Esben W. Pedersen,
Marianne G. Petersen, and Kasper Hornbæk. 2012.
Shape-changing Interfaces: A Review of the Design
Space and Open Research Questions. In Proceedings
of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’12). ACM, New York, NY,
USA, 735–744. DOI:
http://dx.doi.org/10.1145/2207676.2207781
19. Jochen Rick. 2010. Quadratic: Manipulating algebraic
expressions on an interactive tabletop. In Proceedings
of the 9th International Conference on Interaction
Design and Children. ACM, 304–307.
20. Brygg Ullmer and Hiroshi Ishii. 1997. The metaDESK:
models and prototypes for tangible user interfaces. In
Proceedings of the 10th annual ACM symposium on
User interface software and technology. ACM,
223–232.
21. John Underkoffler and Hiroshi Ishii. 1999. Urp: a
luminous-tangible workbench for urban planning and
design. In Proceedings of the SIGCHI conference on
Human Factors in Computing Systems. ACM, 386–393.
