Content uploaded by Unai Lopez-Novoa
Author content
All content in this area was uploaded by Unai Lopez-Novoa
Content may be subject to copyright.
B. Abdulrazak et al. (Eds.): ICOST 2011, LNCS 6719, pp. 266–270, 2011.
© Springer-Verlag Berlin Heidelberg 2011
BlindShopping: Enabling Accessible Shopping for
Visually Impaired People through Mobile Technologies
Diego López-de-Ipiña, Tania Lorido, and Unai López
Deusto Institute of Technology – DeustoTech, University of Deusto
Avda. Universidades 24, 48007 Bilbao, Spain
{dipina,tania.lorido,unai.lopez}@deusto.es
Abstract. BlindShopping is a mobile low-cost easily-deployable system de-
vised to allow visually impaired people to do shopping autonomously within a
supermarket. Its main contributions are: a) a user navigation component
combining an RFID reader on the tip of a white cane and mobile technology,
and b) a product recognition component that uses embossed QR codes placed
on product shelves and an Android phone camera for their identification.
Furthermore, it provides a web-based management component to easily config-
ure the system, generating and binding barcode tags for product shelves and
RFID tag markers attached to the supermarket floor.
Keywords: Blind, Navigation, Mobile Computing, QR codes, Web-Services.
1 Introduction
Accessibility describes the degree to which a device or environment is available for
every person. Nowadays, this term is more and more present in our society, as it is
considered a fundamental right. On the other hand, technology seems to be invading
every aspect of our lives, but it is also moving away from or not giving service to
those collectives which most need it.
The PIRAmIDE project[1] addresses this issue by taking advantage of smartphones
potential to behave as disabled users’ sensorial complement. Its aim is to enable the
smartphone-mediated interaction of a user with the ecosystem of services populating
an environment (e.g. home or supermarket). Thus, PIRAmIDE allows disabled people
to perform daily life tasks autonomously and independently of their disability (e.g.
blind, deaf or elderly people). Mobile devices are transformed into sense enhancers
giving a 6th sense to those who already enjoy their five functional senses, but more
importantly, complementing those which have some sensorial impairment.
One of the concrete application domains targeted by PIRAmIDE is overcoming the
difficulties blind people usually encounter whilst shopping in a supermarket without
the help of someone else. This paper presents an inexpensive easily-deployable solu-
tion which makes use of off-the-shelf technology (mainly smartphones).
From our point of view, an accessible shopping solution has to fulfil the following
requirements if a feasible and flexible wide deployment wants to be achieved:
BlindShopping: Enabling Accessible Shopping for Visually Impaired People 267
• Conventional shopping behaviour should not be altered. Many available solu-
tions require the user to establish her shopping list before initiating the purchas-
ing process (planned shopping). In most cases, such an approach is more efficient
but it is less enjoyable since the user, in this case the blind person, cannot actually
browse the products of the supermarket, discover new brands or new product
types, i.e. carry out opportunistic shopping. In fact, what we actually wish to do is
mixed shopping, i.e. something in between planned and opportunistic shopping.
• Minimal additional off-the-shelf infrastructure should be introduced in super-
markets. Supermarkets are reluctant to introduce complex changes in their inter-
nal information management systems. Furthermore, only simple low-cost easily
maintainable physical instrumentation of their purchasing surface including aisles
and shelves is acceptable. Any feasible solution should leave products as they
are, i.e. such solution must be able of recognizing and deal with the standard
UPC barcodes utilized in worldwide retail. It is a must that accessible shopping
systems operate in actual supermarkets with all their restrictions.
• Blind people should use their usual devices. A blind person carries with her a
white cane and a mobile phone. Therefore, if any, those are the elements that may
be modified or enhanced in order to allow a blind person to safely and effectively
carry out her shopping. Only inexpensive off-the-shelf already known technology
by the blind should be considered to ensure wider acceptance.
2 The BlindShopping Platform
Our solution aims to offer eyes-free technological support for blind people to shop
around as if they saw, without altering conventional shopping patterns. It is designed
to avoid overloading the visually-impaired person with additional new gadgets and
enforcing a supermarket to go through heavy and costly, both in price and time, in-
stallation and maintenance processes.
The assumptions taken by BlindShopping regarding a supermarket organization are
as follows. First, it is considered that all products are grouped into different product
categories (e.g. drinks), and these are divided into product types (e.g. drinks/cola)
which again are divided into concrete brand products (e.g. Pepsi can). Apart from
that, the supermarket is divided into cells of two main types: cells containing shelves
and passageway cells. Thus, internally, BlindShopping maps the IDs of the RFID tags
within a cell to navigation and product location information such as the type of a
given cell, its neighbour cell types, and in case of being shelf type cells, the product
category, types and concrete products located in that area.
BlindShopping offers infrastructural support for the whole purchasing process
within a supermarket, understood as a four step cyclic process: product category
navigation/product search/product identification/product selection. Such cycle is
broken when the user decides to go to the cash till to pay for her purchases. Conse-
quently, BlindShopping offers a navigation component driving the user through voice
messages to the aisle where a product category previously dictated to her smartphone
is located. Once there, BlindShopping also offers support for product recognition by
either shelf section identification or product own identification by means of QR or
UPC code scanning, respectively.
268 D. López-de-Ipiña, T. Lorido, and U. López
Fig. 1. Navigation system (left), UPC code recognition (middle) and QR-code recognition
2.1 BlindShopping Architecture
The distributed component architecture of BlindShopping is composed of the follow-
ing three components:
1. Navigation system. It is in charge of guiding the blind user inside the supermarket.
It provides through a headphone connected to her smartphone simple verbal navi-
gation instructions. It combines a white cane with a portable RFID reader attached
to its tip, a set of road mark-like RFID tag lines distributed throughout the
corridors of the supermarket (see left hand side of Fig. 1 and top part of Fig. 2,
respectively) and a smartphone application processing the RFID readings received
though Bluetooth and generating user navigation verbal commands as result.
Fig. 2. RFID tag marking (top), Motorola Milestone and HTC Desire Android devices (left),
Baracoda’s Pencil2 barcode recognizer and IDBlue RFID reader (middle), NFC 6131 NFC
device (right) and QR-Code and standard UPC barcode (bottom centre)
2. Product recognition. Once the user reaches the target product section, she points
with her camera phone (see Fig. 1) to an embossed QR [2] or UPC code attached
to a shelf section or product. The smartphone camera recognizes that code and
then informs verbally about the product main features. Note that a QR code can
encode up-to 4296 alphanumeric characters, and its redundancy makes successful
reading possible even when partial images of them are captured.
3. System management: BlindShopping includes a web front-end for BlindShop-
ping RFID and QR code infrastructure management. It allows the registration of
BlindShopping: Enabling Accessible Shopping for Visually Impaired People 269
the collection of RFID tags scattered though the supermarket floor and the QR-
codes attached to products or shelf sections.
2.2 Implementation Details
A Nokia 6131 NFC was used, initially, for reading RFID tag floor markings and de-
liver them through a Java ME Bluetooth application to a user-carrying Android
phone. An alternative implementation using the autonomous Baracoda Tagrunner1
RFID Bluetooth reader has then been used.
The mobile application in an Android phone allows the blind person to choose an
action through a gesture interface or by issuing a voice command. Concretely, the
navigation system operation is requested by drawing an “L” or issuing the “Location”
voice command (see Fig. 3). Drawing a “P” or issuing a “Product” voice command,
the user will access the product recognition component that allows obtaining informa-
tion about a product.
A backend server contains the system data and business logic of the BlindShopping
platform. In a real deployment, this back-end should be integrated with the inventory
management system of the supermarket.
Fig. 3. User drawing a “P” (left) on Motorola Milestone, Nokia 6131 NFC to read HF RFID tag
for navigation and supermarket mock-up for testing (right)
3 Related Work
An exhaustive comparative study of existing accessible shopping systems for blind
and visually impaired people was carried out by Kulyukin and Kutiyanawala [3] in
2010. ShopTalk [4] is a wearable solution that requires the user to carry a barcode
scanner and a UMPC in a backpack. Verbal route instructions are issued through a
headphone connected to the UMPC at the blind person’s backpack. Although the
supermarket does not need to install and maintain any hardware, the system requires
access to the supermarket’s inventory control system and binding of product barcodes
into supermarket locations so that guiding can be accomplished. The advantage of
BlindShopping is that it only demands a lightweight smartphone equipped with a
camera to read QR Codes attached to shelf sections and the very blind person’s white
cane enhanced with an off-the-shelf RFID reader.
1 http://www.baracoda.com/baracoda/product/ p_48_TagRunners.html
270 D. López-de-Ipiña, T. Lorido, and U. López
The Tinetra[5] project offers the possibility of detecting products via a barcode or
RFID reader, and then it obtains related information via GPRS from the server. How-
ever, it does not include a guiding system as BlindShopping. Similarly to us they use
Baracoda’s Pencil2 to scan barcodes and IDBlue to scan RFID tags.
4 Conclusions and Further Work
A basic usability study with a blind person has been carried out. She was requested to
navigate through different sections of our emulated supermarket surface by using her
white cane with an attached BT RFID reader and an Android application on an HTC
Desire. Her main comment was that navigation was very intuitive since locating the
RFID tag markings was very easy and the navigation vocal commands very useful to
reach the desired target. She was then requested to assess whether locating embossed
UPC barcodes and using a Baracoda Pencil2 device to recognize them (see centre of
Fig. 1) was easier or harder than using the Android phone camera to point to em-
bossed QR codes located over products (see right hand side of Fig. 1). She judged that
the latter was much more plausible. Besides, QR code recognition using a camera
phone is much faster and reliable.
BlindShopping is a low-cost easily deployable solution which makes a supermarket
accessible to visually-impaired people through two core components: a) an RFID and
mobile phone based indoor navigation system and b) a mobile QR-code based product
recognizer. Future work will undertake a fully fledge evaluation in a real supermarket.
Acknowledgments
This work has been supported by project grant TSI-020301-2008-2 (PIRAmIDE),
funded by the Spanish Ministry of Industry, Tourism and Trade.
References
1. PIRAmIDE platform official site, http://www.piramidepse.com
2. QR Code ISO specification,
http://www.iso.org/iso/catalogue_detail.htm?csnumber=43655
3. Kulyukin, V., Kutiyanawala, A.: Accessible Shopping Systems for Blind and Visually Im-
paired Individuals: Design Requirements and the State of the Art. The Open Rehabilitation
Journal (to appear)
4. Nicholson, J., Kulyukin, V.: ShopTalk: Independent blind shopping = verbal route direc-
tions + barcode scans. In: Proceedings of the 30th Rehabilitation Engineering and Assistive
Technology Society of North America (RESNA), Phoenix, Arizona (2007)
5. Lanigan, P.E., Paulos, A.M., Williams, A.W., Rossi, D., Narasimhan, P.: Trinetra: Assistive
technologies for grocery shopping for the blind. In: International IEEE-BAIS Symposium
on Research on Assistive Technologies (RAT), Dayton, OH (2007)
