Content uploaded by Anke Reinschluessel

Author content

All content in this area was uploaded by Anke Reinschluessel on Mar 20, 2018

Content may be subject to copyright.

Beyond Math Manipulatives: Smart

Tangible Objects for Algebra Learning

Anke V. Reinschluessel

University of Bremen

Digital Media Lab

avr@uni-bremen.de

Danny Thieme

University of Bremen

danny1@uni-bremen.de

Tanja Döring

University of Bremen

Digital Media Lab

tanja.doering@uni-bremen.de

Rainer Malaka

University of Bremen

Digital Media Lab

malaka@tzi.de

Dmitry Alexandrovsky

University of Bremen

Digital Media Lab

dimi@uni-bremen.de

Copyright © 2018 for this paper held by its author(s). Copying permitted for private

and academic purposes.

Abstract

This workshop position paper presents ongoing research

on using smart tangible objects for algebra learning. While

mathematical manipulatives have played an important role

in children’s mathematics development for decades, em-

ploying tangible objects in the classroom has been rarely

explored yet. In our work, we investigate the potentials of

using smart objects for algebra learning. Our smart tiles

are based on traditional algebra tiles, passive mathematical

manipulatives used in many schools in Northern America,

and we currently extend these by 1.) multimodal input and

output capabilities, 2.) dynamic constraints and 3.) adap-

tivity and feedback. In this paper, we give an overview on

the overall system concept, the interaction with the tangible

objects and their current design, as well as on the potentials

of actuated smart objects for future interaction.

Author Keywords

Tangible user interface; smart objects; tabletop interaction;

embodied interaction; multimodal feedback; collaborative

learning; adaptive system.

ACM Classiﬁcation Keywords

H.5.2 [Information Interfaces and Presentation (e.g. HCI)]:

User Interfaces

Introduction

In math education, simple passive manipulatives provide

valuable “hands-on” approaches to teach students ab-

stract concepts, especially when the students start to learn

a novel unit of math, e.g., arithmetic, geometry, or alge-

bra. These approaches are in-line with models from didac-

tics like Bruner’s concrete-representational-abstract ap-

proach [3] or the constructivistic objects-to-think-with ap-

proach [17] that suggest to use physical objects for abstract

concepts, especially for beginners. While a considerable

body of research on using tangible user interfaces (TUIs)

for learning has been conducted, more research efforts

are needed to address the question how tangible user in-

terfaces can be made smarter in order to facilitate a better

learning environment and better support for learners.

In our research, we investigate the potentials of smart ob-

jects for learning. The objects are based on traditional al-

gebra tiles, which are passive mathematical manipulatives

(see Fig. 1) as used in many schools in Northern America

to support algebra learning. We are extending these tiles to

smart “tiles” by 1.) multimodal input and output capabilities,

e.g., light and display, 2.) dynamic constraints, e.g., electro-

magnets attracting or repelling objects, and 3.) adaptivity

and feedback, e.g., user support and hints.

In educational research tactile models are common, as

for example the ones by Bruner [3] or Kieran [10]. They

showed that by using physical objects it is possible to teach

already small children mathematical, in particular alge-

braic concepts. Common digital learning platforms, such

as Dragonbox1lack the haptic and tactile components

and therefore the beneﬁts that come with tangibility. By

transforming the algebra tiles to smart objects we want to

1http://dragonbox.com/

Figure 1: Algebra Tiles as commonly used in Northern America.

The set consists of three types of objects: small squares represent

constant values of 1and −1; elongated rectangles stand for

positive and negative variables (x); large squares represent

squared variables (x2). The sign of an object is shown through

color. All objects consist of one red surface, representing the

negative value and a unique color for each object type depicting

the positive value.

combine the richer feedback that can be provided by digi-

tal platforms with the beneﬁts of tactile interaction, where

the smart tiles themselves create dynamic constraints and

allow for multimodal input and output, also in combination

with a touch screen.

Figure 2: Left: early system version with simple tiles. The tiles are made of transparent acrylic glass with reacTIVision markers; Right: latest

version with smart tiles with electromagnets and visual output.

Related Work

Figure 3: Solving steps of task

3 + (−x)−5 = xwith algebra

tiles.

Tangible user interfaces emerged in the 1990s, as Ishii and

Ullmer [9] describe their vision for “tangible bits”. The gap

between the physical world and the digital world should be

closed by allowing users to directly manipulate these bits,

which can be everyday physical objects. Early examples

are the metaDesk [20], the transBOARD [8] and the Urban

Planning Workbench [21]. Since then, a popular applica-

tion domain for TUIs has been learning. Examples for using

tangibles for math learning have been provided by Falc¯

ao

et al. [5], Girouard et al. [6], Manches and O’Malley [12],

and Marichal et al. [13], amongst many others. Others like

Rick [19] incorporate touch to be able to directly manipu-

late math objects presented on a screen. Research about

how tangibles can support learning or how learning theories

can inform tangible development is for example presented

by the “Tangible Interaction Framework” by Hornecker and

Buur [7] or the “Tangible Learning Design Framework” by

Antle and Wise [1]. They propose design principles for tan-

gibles and a taxonomy about the relationship between TUIs,

interactions and learning. Fur thermore Marshall [14] and

Marshall, Price and Rogers [15] critically discuss how tangi-

bles can support learning.

Examples for technological approaches for smart objects

are Sifteos (previously Siftables) [16], small objects which

have all technology needed inside that react to each other

and encourage interaction with the objects themselves. Sif-

teo cubes were launched as product in 2011 but are not

available anymore. A newer approach are the Actibles [4],

which are tangibles with a smartwatch core and light feed-

back. They allow a variety of interactions, including shaking,

tilting, stacking and neighbouring.

Multimodal Algebra Learning with Tiles

Algebra tiles as shown in Fig. 1 consist of three types of

tiles: single units used as “ones”, x-tiles and x2-tiles. Each

tile has a positive and a negative side, whereby the neg-

ative side is colored red and the positive value is repre-

sented by a unique color of that object. The tiles are typi-

cally placed on a 2×2 area (compare Fig. 2), where the two

squares on the left represent the left side of an equation

and the two squares on the right represent the right side

of the equation. The top square on both sides is the “ad-

dition zone”, i.e., all tiles there are connected by addition,

while the lower areas are the “subtraction zones”, i.e., all

tiles there are subtracted from the top ones. An equation,

for example 3+(−x)−5 = x, is put up in tiles (see top

image in Fig. 3). There would be three positive ones and

one negative x-tile in the addition section on the left side,

and ﬁve positive ones in the subtraction zone. On the right

side there is just one positive x-tile in the addition zone.

The model comes with a set of possible actions that cor-

respond to typical algebraic manipulations, when dealing

with linear equations (addition, subtraction, multiplication

and division). Generally, the goal is to apply a sequence of

legal actions in order to transform the equation to another

form or to isolate the x-tile on one side and the one-tiles on

the other side. With the traditional tiles, a student can trans-

form the equation, but does not get any feedback about the

correctness of the actions. A teacher is still necessary to

verify them. On the contrary, in our approach, the algebra

tiles themselves are aware of the equation they are part of

and give feedback or hints about the steps. They can verify

the result of a student’s actions or support grouping of tiles

with visual feedback (see subsection Multimodal Input and

Output for more details). In combination with a touchscreen

and the capacity to track the tiles, multimodal algebra learn-

ing is possible, as the beneﬁts of the tangible tiles are com-

bined with the rich feedback such a system can provide.

Figure 4: Smart Tiles unattached

with magnets and connection

areas

Figure 5: Smart Tiles attached

with with magnets and light

feedback

Interaction with Smart Tangible Objects

In our current setup, the smart tangible objects are placed

on an interactive tabletop, where they are identiﬁed, located

and tracked with regard to orientation. This way, they are

integrated into a system that also uses the tabletop surface

for input and output capabilities (e.g., visual feedback and

multi-touch interaction). While we used passive tiles in early

versions of our systems (see Fig. 2, left side), the smart

tiles (see Fig. 4,5,6,7) are designed to provide rich interac-

tion capabilities themselves as this enhances the learning

environment and supports learners. Furthermore, this ap-

proach would also allow for a setting in which the smart tiles

are used fully functional on their own without a multi-touch

table. The intelligence situated in our tangible objects and

their surrounding system currently improves the interaction

by three approaches: multimodal input and output capabili-

ties,dynamic constraints, and adaptivity and feedback.

Multimodal Input and Output

Multimodal input and output facilities of learning systems

can enhance learning experiences, as more senses are

involved in the learning process. Our tangibles follow this

approach by allowing direct haptic interaction with the tiles,

which can be moved around and placed in different areas

of the system for input, i.e., to perform algebraic opera-

tions. Moreover, the objects directly give visual feedback,

for which we developed a model with two kinds of visual

feedback with central display and edge light feedback (see

also [4, 2] for a similar approach with low-resolution edge

displays). With this approach, the smart tiles can display

their current state in the center, e.g., the current value a

tile presents (see Fig. 6 and Fig. 7). At the same time they

communicate feedback on the performed operations via

edge light animations, e.g., if the moves were correct or

about current relations to surrounding tiles such as group-

ing of tiles where tiles are combined to one unit. Other out-

put modalities such as sound and vibration feedback are

currently tested.

Dynamic Constraints

Our smart objects are designed to provide dynamic con-

straints that guide the interaction. In our system, the dy-

namic constraints can direct the grouping of tiles, which is

an essential action for performing operations and thus for

transforming and solving equations, they can also prevent

wrong combination of tiles (e.g., placing a tile of unit one

along the long side of an x-tile). The dynamic constraints

change according to the current value of each tile and the

possible combinations. We realized the dynamic constraints

by adding neodymium magnets and electromagnets to the

sides of the objects. When the electromagnets are switched

off, the neodymium magnets repel two objects. In case the

electromagnets are switched on, realized by closed circuits

when two ﬁtting objects have contact, the objects attract

each other (see next section for further information). With

this approach, we can also stack objects. Furthermore, this

physical grouping also gives a great physical representation

of grouped units and enhances the haptic interaction.

Figure 6: Smart tile with display

showing value +1

Figure 7: Smart tile with display

showing value −1

Adaptivity and Feedback

Among the advantages of our system is that it can be adapted

and, to some degree, can automatically adapt to learners

with different levels and needs with regard to feedback and

hints provided. Partly, this is directly provided by the smart

tiles themselves via display and light feedback as well as

magnetic hints, partly this is currently communicated by

the surrounding system, in our case the interactive table-

top system. In order to make the system smart and allow

for good feedback and hints, we have integrated and fur-

ther employ a number of approaches. One of these is us-

ing Wolfram Alpha as computation knowledge engine2that

computes algebraic transformations and allows for feedback

if an operation is a useful move towards the result for exam-

ple. Moreover, machine learning approaches can support

2https://www.wolfram.com/engine/

the identiﬁcation of typical errors and the integration of use-

ful feedback, which will also improve the feedback the tiles

communicate directly. A central challenge lies in ﬁnding a

good balance of learner level and adequate feedback and

hints.

Current Design of the Smart Tangible Objects

For our tangible learning system with algebra tiles we started

with passive tiles and optical tracking. Using the reacTIVi-

sion3framework we had an early setup with tracking from

below to avoid occlusion problems. Currently we are work-

ing on capacitive tracking in combination with motion sen-

sors to enable working on touch screens like the Microsoft

Surface Hub4. The underlying software on the touchscreen

is programmed in Unity5and for supporting the equation

solving we are using the Wolfram Alpha API.

Figures 6 and 7 show the current design of our smart tiles

with center display, edge RGB lights, and electromagnetic

dynamic constraints. Currently, the tiles have a size of

7x7x5cm (width ×depth ×height) and contain Arduino

Atmega328P CH340 boards, which communicate via WIFI

with the central system. Next to a matrix LED display they

contain 12 RGB LEDS for edge light feedback. The dy-

namic constraints are realized by a combination of eight

neodymium magnets in den corners of the tiles and four

electromagnets at the edges. By default, the electromag-

nets are switched off, so that the neodymium magnets re-

pel objects placed next to each other. When two tiles ﬁt

together, the electromagnets at the edges of the tiles (see

Figure 5) attract each other (being stronger than the re-

pelling neodymium magnets), as they are switched on when

3http://reactivision.sourceforge.net/

4https://www.microsoft.com/en-us/surface/devices/

surface-hub/overview

5https://unity3d.com/

the tiles have contact and close a circuit through conductive

contacts. These contacts only close a circuit when the pair-

ing is allowed, which can be changed dynamically. If com-

bined, the two objects stick together and represent a unit.

Through the neodymium magnets stacking tiles is generally

also possible.

A concrete example for the use case of the magnets at-

tracting would be two tiles of opposite value, e.g., +1 and

−1, which can be paired as an “zero pair” and be removed

from the working area, as they cancel out. Additionally we

have light feedback to support the same pairing process

and grouping. With the light feedback one smart tile can

support more advanced learners, which already know that

+1 on the left side on an equation can be moved to right

side and then results in an −1. Smart tiles can automati-

cally change color to show this sign change.

Envisioning Future Smart Tangible Objects

The current setup and underlying model have some lim-

itations that could be addressed by actuated tangibles.

During solving, especially children tend to compare the

resulting x-tile in size with the one-tiles. For the future,

smart x-tiles that could change their shape regarding size

would be beneﬁcial to this step, after the equation is solved.

Additionally it may occur the case that the user is able to

see the solution even though there is still more than one

x-tile on the area. Being able to change the size for all x-

tiles synchronously would enhance the effect that all x are

the same. Thus, exploring shape change (c.f. the design

space of shape-changing interfaces by [18]) for tangible

presentations of variables in algebraic expressions would

be valuable. Another case is that especially after a divi-

sion it happens that multiple objects need to be removed at

once, which tends to be bothersome for the user. Here ac-

tuated smart tiles would do the trick and after performing a

division, they could automatically remove themselves from

the working area. Vice versa, for multiplication they would

automatically enter the working area. Also in case of pro-

viding help they could rearrange to give another view on the

equation. Approaches like used in small swarm robots as

the Zooid Swarm Robots [11] could be applied to make the

tiles and the interactions smarter.

Conclusion

In this paper, we presented our approach for smart tangible

objects to support algebra learning. Starting from traditional

passive math manipulatives, we developed a concept to

make these learning objects more intelligent in order to pro-

vide better learning environments that allow for rich and

multimodal interactions, dynamic constraints, as well as

adaptivity and feedback. We presented our current proto-

type, including the overall math system and the design of

the smart objects. Furthermore, we discussed strategies to

design even smarter tangible objects, which would address

some of the current limitations by realizing actuation, both

with regard to shape-change as well as to self-moving tiles.

Overall, while our ongoing work focuses on providing a con-

tribution to designing smart tangible objects for learning

scenarios, our approaches to make the interaction and the

objects smarter could also be valuable for other application

contexts.

REFERENCES

1. Alissa N. Antle and Alyssa F. Wise. 2013. Getting Down

to Details: Using Theories of Cognition and Learning to

Inform Tangible User Interface Design. Interacting with

Computers 25, 1 (Jan. 2013), 1–20. DOI:

http://dx.doi.org/10.1093/iwc/iws007

2. Ahmed Sabbir Arif, Brien East, Sean DeLong, Roozbeh

Manshaei, Apurva Gupta, Manasvi Lalwani, and Ali

Mazalek. 2017. Extending the Design Space of

Tangible Objects via Low-Resolution Edge Displays.

ACM Press, 481–488. DOI:

http://dx.doi.org/10.1145/3024969.3025078

3. Jerome Seymour Bruner. 1966. Toward a theory of

instruction. Vol. 59. Harvard University Press.

4. Brien East, Sean DeLong, Roozbeh Manshaei, Ahmed

Arif, and Ali Mazalek. 2016. Actibles: Open Source

Active Tangibles. In Proceedings of the 2016 ACM

International Conference on Interactive Surfaces and

Spaces (ISS ’16). ACM, New York, NY, USA, 469–472.

DOI:http://dx.doi.org/10.1145/2992154.2996874

5. Taciana Pontual Falcao, Luciano Meira, and

Alex Sandro Gomes. 2007. Designing tangible

interfaces for mathematics learning in elementary

school. In Proceedings of III Latin American conference

on human-computer interaction.

6. Audrey Girouard, Erin Treacy Solovey, Leanne M.

Hirshﬁeld, Stacey Ecott, Orit Shaer, and Robert J. K.

Jacob. 2007. Smart Blocks: A Tangible Mathematical

Manipulative. In Proceedings of the 1st International

Conference on Tangible and Embedded Interaction

(TEI ’07). ACM, New York, NY, USA, 183–186. DOI:

http://dx.doi.org/10.1145/1226969.1227007

7. Eva Hornecker and Jacob Buur. 2006. Getting a Grip

on Tangible Interaction: A Frameowrk on Physical

Space and Social Interaction. In CHI 2006. ACM Press,

Montréal, Québec, Canada.

8. Hiroshi Ishii. 2008. The Tangible User Interface and Its

Evolution. Commun. ACM 51, 6 (June 2008), 32–36.

DOI:http://dx.doi.org/10.1145/1349026.1349034

9. Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits:

towards seamless interfaces between people, bits and

atoms. In Proceedings of the ACM SIGCHI Conference

on Human factors in computing systems. ACM,

234–241.

10. C Kieran. 2007. Learning and teaching algebra at the

middle school through college levels. I: Lester, FK Jr.

Second Handbook of Research on Mathematics

Teaching and Learning (2007).

11. Mathieu Le Goc, Lawrence H Kim, Ali Parsaei,

Jean-Daniel Fekete, Pierre Dragicevic, and Sean

Follmer. 2016. Zooids: Building blocks for swarm user

interfaces. In Proceedings of the 29th Annual

Symposium on User Interface Software and

Technology. ACM, 97–109.

12. Andrew Manches and Claire O’Malley. 2012. Tangibles

for Learning: A Representational Analysis of Physical

Manipulation. Personal Ubiquitous Comput. 16, 4 (April

2012), 405–419. DOI:

http://dx.doi.org/10.1007/s00779-011-0406-0

13. Sebastián Marichal, Anadrea Rosales,

Fernando Gonzalez Perilli, Ana Cristina Pires, Ewelina

Bakala, Gustavo Sansone, and Josep Blat. 2017.

CETA: Designing Mixed-reality Tangible Interaction to

Enhance Mathematical Learning. In Proceedings of the

19th International Conference on Human-Computer

Interaction with Mobile Devices and Services

(MobileHCI ’17). ACM, New York, NY, USA, Article 29,

13 pages. DOI:

http://dx.doi.org/10.1145/3098279.3098536

14. Paul Marshall. 2007. Do tangible interfaces enhave

learning. In TEI ’07: Proceedings of the 1st

international conference on Tangible and embedded

interaction, Brygg Ulmer and Association for

Computing Machinery (Eds.). Curran, Red Hook, NY.

OCLC: 254584537.

15. Paul Marshall, Sara Price, and Yvonne Rogers. 2004.

Conceptualising tangibles to support learning. In

Proceedings of the 2003 Conference on Interaction

Design and Children: 2003, Preston, England, July

01-03, 2003, Conference on Interaction Design and

Children, Stuart MacFarlane, ACM Digital Library, and

Association for Computing Machinery (Eds.).

Association for Computing Machinery, New York, N.Y.

OCLC: 612478047.

16. David Merrill, Jeevan Kalanithi, and Pattie Maes. 2007.

Siftables: Towards Sensor Network User Interfaces. In

Proceedings of the 1st International Conference on

Tangible and Embedded Interaction (TEI ’07). ACM,

New York, NY, USA, 75–78. DOI:

http://dx.doi.org/10.1145/1226969.1226984

17. Seymour Papert and Idit Harel. 1991. Constructionism.

Norwood. (1991).

18. Majken K. Rasmussen, Esben W. Pedersen,

Marianne G. Petersen, and Kasper Hornbæk. 2012.

Shape-changing Interfaces: A Review of the Design

Space and Open Research Questions. In Proceedings

of the SIGCHI Conference on Human Factors in

Computing Systems (CHI ’12). ACM, New York, NY,

USA, 735–744. DOI:

http://dx.doi.org/10.1145/2207676.2207781

19. Jochen Rick. 2010. Quadratic: Manipulating algebraic

expressions on an interactive tabletop. In Proceedings

of the 9th International Conference on Interaction

Design and Children. ACM, 304–307.

20. Brygg Ullmer and Hiroshi Ishii. 1997. The metaDESK:

models and prototypes for tangible user interfaces. In

Proceedings of the 10th annual ACM symposium on

User interface software and technology. ACM,

223–232.

21. John Underkofﬂer and Hiroshi Ishii. 1999. Urp: a

luminous-tangible workbench for urban planning and

design. In Proceedings of the SIGCHI conference on

Human Factors in Computing Systems. ACM, 386–393.