Content uploaded by Michele Romani
Author content
All content in this area was uploaded by Michele Romani on Dec 04, 2020
Content may be subject to copyright.
AR Notes: Augmenting Collaborative Tools
Rusna Rusna
Université Paris-Saclay
Orsay, 91400, France
rusnatan@gmail.com
Michele Romani
Université Paris-Saclay
Orsay, 91400, France
michele.romani.gzl0@gmail.com
Brendan Hanke
Université Paris-Saclay
Orsay, 91400, France
brndnhanke@gmail.com
Abstract
Over the last decades of HCI research, it has become
clear that humans interact more naturally when in
presence of a tangible interface that recalls them of a
functionality they already know from a previous non-
digital experience. However digitalization and
automation of many tools we use on a daily basis have
made complicated integrating tangibles and software in
a way that takes advantage of both physical and digital
world without affecting the user experience with the
disadvantages that either one or both might carry.
Author Keywords
AR; tangibles; augmented; Kanban; agile; tasks;
notes; sticky notes; paper; professional; collaboration;
collaborative.
CSS Concepts
• Human-centered computing~Human computer
interaction (HCI); Collaborative and social computing
systems and tools; Asynchronous editors; Human-
centered computing; Collaborative and social
computing; Collaborative and social computing systems
and tools; Asynchronous editors; Interaction
techniques; Gestural input.
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all other
uses, contact the owner/author(s).
CHI 2020 Extended Abstracts, April 25–30, 2020, Honolulu, HI, USA.
© 2020 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-6819-3/20/04.
Introduction
AR Notes is an augmented reality task organizer
designed to improve teamwork in a shared coworking
space. It features a combination of software
technologies and physical interactions on a physical
Kanban board to take advantage of the strengths of
both the virtual and the real world.The initial idea of
our project is to create a tool that could take advantage
of the physical world to improve the user engagement
in daily tasks and their collaboration in a shared
environment. We identified different groups that could
take advantage of this tool: students, workers, families,
and teams. Our target as tangible interface were sticky
notes: many people daily use papers to take notes and
reminders and stick them where they can easily be
found again. However, paper notes have limited space,
the content can hardly be edited and even worse they
can be completely lost.
Related Works
We also identified different tangibles objects that could
be used for an intuitive interaction: cards, marbles,
cardboard shapes, colored shapes. These inspirations
came from different other ideas below.
The marble answering machine [1] is a prototype of an
answering machine that uses tangible interaction. Each
recorded message is represented by a single marble.
Users can manipulate the messages by placing the
marbles onto different parts of the device, to listen to
or delete a message for example. This idea is influential
in tangible user interface [TUI] history, and to our
project, because of its use of a single tangible (one
marble) to represent one-to-one a piece of digital
information (one voice message). We used this concept
as the basis for our ideas, using a single tangible object
to represent a single digital note.
Another inspiration came from Virtual Reminders [2].
In our brainstorming we found this concept by
RealiTease. The idea consists of digital sticky notes that
can be seen using AR through a smart phone. This
concept gave us inspiration to use paper as our
medium for the tangibles. The concept is interesting as
a basis, but we wanted to push it further and to cover a
use case that is based in user research.
System Design
First Concept
Our first concept was a post-it board in which everyone
could attach an augmented post-it for the others to
see, easily copied and shared. On top of it, the users
could also have the digital version always available on
their devices.
Interviews
We created an interview guide and interviewed
participants. The goal was to talk to participants in
different user groups - students, those living in a
residence and who may use notes to communicate
amongst each other and working professionals. We
talked to five students, one of which used notes in their
household and one working in an office as PhD. We also
reiterated the process with a university professor that
gave us a professional point of view. Some key points
emerged that made us realize we needed to reiterate
our process and modify our first concept, in particular:
• Post-it gets easily lost, they should be easy to
replace
• There are very user-friendly apps, such as
Google Keep, to take notes and share
Figure 1 - The Marble Answering
machine by Durrell Bishop (1995)
Figure 2 - Virtual Reminders, a
concept by Augmented Realitease
• Physical reminders are better in a shared
environment, you may “lose” track in a group
chat
• Paper on the desk is a better reminder than a
notes app
Second Concept
The new concept we came out after reiterating our idea
is an AR task board for professional teams, like a
Kanban board. The board and the tasks themselves will
be physical, and each task on the Kanban board will act
as a marker which can be used to display
supplementary information about the task in AR. Tasks
could be duplicated and shared by using duplicate
markers. We experimented with other interactivity of
the board and markers, like showing task progress and
changing it accordingly. The complete board can also
be accessed digitally on a web interface on any type of
device while the AR client can be used on any mobile
device equipped with a camera.
Paper Prototype
We built our first prototype using paper and food
images as the markers. We used this prototype to test
the augmentation with Vuforia development kit and
explore possible interactions with the physical cards
and between different users.
Figure 3 - The first paper prototype that was later augmented
using markers
The main concepts we learned from this prototype
were:
• The possibility to add extra text/photos/videos
to a paper task
• The possibility to have a community version of
each card on the board and a duplicate that is
given to the task owner. Having the same
marker, it means that every printed copy of a
card is also a digital copy of its content
• The possibility to use a projector to
synchronize the physical board with its digital
counterpart.
Software Prototype
Building a complete software system out of a paper
concept has not been an easy task. The first step was
identifying the entities of our system and how they
needed to interact. We used DBDesigner to build an
Entity-Relationship schema for our application. We then
exported the SQL Create statement and imported it into
PostgreSQL.
Figure 4 - The ER schema of the application
We also implemented a Java application with RESTful
API that manages the layer between clients and the
database, and we modelled all the entities, both on
server and clients, to match the schema.
Figure 5 - Screenshot of the mobile client ARNotes while
pointing at the wooden Kanban board with augmented tasks
and markers
Figure 6 - Screenshot of ARNotesWeb, the web client that
synchronizes with the physical board
System Architecture
The system is a distributed virtual environment
composed of 3 different modules that communicate
with each other with REST calls. It's based on a classic
centralized client-server architecture.
1. ARNotes: An Android client built in Unity and
powered by Vuforia AR library. It’s the main
interaction point between the users and the
application physical objects.
2. ARNotesServer: A Java application that
provides all the core logic and functionalities of
the backend.
3. ARNotesWeb: A web interface for managing
the digital version of the application objects.
To build the system we used several technologies,
languages and libraries according to our requirements:
• Unity Engine (C#) – Game engine that we use
for the AR clients.
• Vuforia – AR library we used to render the
augmented tasks.
• Java Spring (Java 13) – Last version of the
OOP language we used to build the server.
• Maven – Package manager for Java libraries
and dependencies.
• JavaScript EC6 – Last version of the web
language we used to manipulate objects on the
interface.
• Bootstrap – CSS/JS library for building
consistent interfaces.
• PostgreSQL – Database management system.
• Swagger – API viewer we used for testing the
application APIs.
Implemented Functionalities
In the current prototype the application supports both
co-located and remote collaboration, thanks to the
server and the persistence of the database. We tried to
keep the virtual environment always consistent with the
continuous changes that the users may perform. The
responsiveness of the system is decent when
interacting with a limited number of virtual objects in
the same scenario, however the nature of the Kanban
board allows us to be flexible on this. The following
functionalities have been fully implemented:
1. CRUD methods for managing entities (Task,
User, etc.) through API.
2. A claim method to assign a Task to the current
user.
3. An assign method to assign a Task to a specific
user through tangible interactions.
4. A state recognizer that changes the state of a
Task according to its position on the Kanban
board.
5. Real-time editing of the content of a Task.
6. Synchronization of the AR Notes clients using
the API.
7. Synchronization of the digital web board using
the API.
8. Copy the content of an existing task on a new
task by colliding on it.
Known Issues
The main issues we have with our system are related to
markers recognition and synchronization (WYSIAWIS).
We noticed that having different objects in the same
close space requires more computational effort and this
usually leads the application to a lack of precision or
eventually to not being able to identify the object.
Another problem due to the client-server architecture
and the physical-digital duality is the synchronization. A
distributed system like ours requires the clients to keep
“polling” the server with requests to update the current
state of the system. If the polling happens in a too
short interval of time, even with a few clients, the
server gets flooded by requests and eventually is
unable to answer all of them. An opposite situation
happens when the clients do not update their status
enough frequently to be updated about changes
between them, leading to inconsistency of the contents
or worse to overlap changes made by another user. The
main synchronization problem is between the physical
and the digital board: when updating the physical
board, it is quite simple to update the server and then
synchronize the digital board accordingly, but doing the
opposite is a hard challenge. In the following section we
discuss a possible solution.
Future Developments
The project has a lot of potential and we are planning
on developing further to improve existing functionalities
and add new ones. A main functionality we had no time
to implement is user authentication and content
visibility as we want our user to be able to keep some
of their notes “private” while sharing others with the
team.
For the physical synchronization problem we were
thinking to use a combinations of cameras and a
projector that are always watching the physical board
and whenever the state changes a visual notification
will appear on the physical Task asking the users to
reposition it accordingly to the new state.
We also aim to improve synchronization, maybe using a
system or rules, locks and ownership to prevent users
from having conflicts while editing the same Task.
Finally we would like to use a magnetic board with
magnets instead of paper and wood, that would
improve the physical interaction with the board and at
the same time make our system easy to setup on any
office board.
Proposed Evaluation
For evaluating our system, we planned to assign a
small project using ARNotes as task planner to a
development team of 3-4 people. The project will
consist in developing a small web application, in this
way we can assign each member a role in the team.
One of the members will be elected as team leader and
will be responsible for the first assignment of the tasks.
The project will be developed over a period of 3 days,
for 3 hours a day, to avoid stressing too much the
team.
On the first day, the team will follow a 30 minutes
tutorial to learn how to use and interact with the
system. The team leader will be given a set of Tasks
defined by us according to the project requirements.
He/she will then distribute a copy of each task among
the team members and stick the originals on the
Kanban board under the TODO state and officially start
the development.
At the beginning of each day, the team can have a 15
minutes meeting in which the team leader will move
away “completed” Tasks and add new ones in the
TODO state. Every member will be able to add new
tasks as they prefer, even after the meeting.
On the last day, they will be singularly interviewed or
given a questionnaire to gather data about the usage
and feedbacks on the experience.
We did not have time nor the users availability to run
our experiment, however during the annual exhibition
day of the HCI master in Université Paris-Saclay, we let
visitors (mostly IT students and professors) try our
prototype and experiment with the interactions. We
also were able to explain our use case, so in the
following section we report the feedbacks we got during
our live demo.
User Feedbacks
In general, user feedbacks during the exhibition were
positive, highlighting robust synchronization between
the physical tools to the online version. There were also
some specific inputs and suggestions shared by the
visitors quoted as followed:
• “I don't see the advantage in using the phone
to augment paper when I can simply use a task
app on the phone”
• “The content of the tasks should be always
visible like in the web page”
• “It would be really useful having the possibility
to add links, pictures and media to the tasks”
• “It would be great to recognize handwritten
text and make it the digital text of the task.
Otherwise it would be useful as well having a
picture of the handwritten content attached to
the task”
• “It would be better to have a different identifier
or card design to differentiate public card from
the personal copy”
• “It is confusing to have the screen still moving
following the phone’s camera movement during
text input. It would be better to stop the
screen from moving during the text input,
maybe freeze the screen using a buffer when
edit button is clicked”
• “With augmented reality, there is always a
problem when it comes to scalability to ensure
objects could still be properly tracked when the
number of tasks increase”
Discussion
With the user feedbacks from the exhibition, we
understand that the tool could be further improved in
several aspects. In terms of functionality, the tool could
be equipped with additional features such as storing
media contents and screen freeze during content input.
As for the usability, the presentation of the tool could
be modified to increase learnability, such as having
physical text written on the physical task card and a
different layout or design to show the difference
between a personal copy and a public copy of the task.
The list of improvements mentioned above are
achievable and some have been considered during the
tool design and discussed in future developments, such
as the use of a magnetic board with magnets, where
the magnets would be in the form of a writable and
erasable white board or a frame with writable inserts.
This would allow users to have an overview idea of the
particular task for each task with the extension of
information presented in the augmented form.
Some concerns which require further exploration are
the scalability of the tool and the usability in real life
scenarios. These would be ideally reviewed during an
evaluation session as proposed and extended to a
larger scale to test the scalability aspect.
Conclusions
The collaboration tool proposed has the potential to
improve the real-life situation of teamwork, especially
for team that are adopting agile methodology or lean
manufacturing, by providing support on real time
discussion collaboration with the advantage of digital
management of content. It also leverages the use of
tangible reminders that are commonly used and
visually obvious. Further improvements and evaluations
are needed to assess the usability and scalability of the
tool as well as alternatives for the visualization of the
augmented content for example installing a permanent
camera and projector on the physical board. It’s worth
mentioning that the rapid development of new AR
technologies could soon be a valid alternative to
smartphones. It’s not hard imagining in the near future
a slim iteration of MS HoloLens being actively used in
the enterprise: that could be the perfect way to scale
this tangibles-based technology and make the
transition between the physical and the digital world
completely seamless for the users.
References
[1] Kaltenbrunner, Martin. (2018). An Abstraction
Framework for Tangible Interactive Surfaces.
[2] DDACorp. (2018). Virtual Reminders. Retrieved
December 19, 2019 from
https://augmentedrealitease.com/post-it.cfm
[3] Jacob, Robert & Ishii, Hiroshi & Pangaro, Gian &
Patten, James. (2002). A Tangible Interface for
Organizing Information Using a Grid.
10.1145/503376.503437.
[4] Sagar, Ravi. (2017). Sync your physical Agile
board with JIRA. Retrieved December 19, 2019
from https://blog.valiantys.com/en/jira-
software/sync-physical-agile-board-jira/
[5] Aylward, Bruce. (2018). PsodaVision launches to
help you sync physical and digital kanban boards.
Retrieved December 19, 2019 from
https://www.psoda.com/global/blog/psodavision-
sync-physical-and-digital-kanban-boards/
