SIGN LANGUAGE TO SPEECH CONVERSION USING ARDUINO

Abstract

A huge population in India alone is of the dumb and deaf people. So the system is working on a glove based device which will be used for conversion of sign language (ASL) to speech .The basic system consists of a two parts; sign language recognition and conversion to text and further to speech. The sign language glove consist of a simple hand gloves fitted with flex sensors which is being used for the monitoring the amount of bend on the fingers. Flex means bend, this is the sensors that change the resistance depending on the amount of bend on the sensor. Data from the sensors is send to the Control unit which is the Arduino Nano the analog signals from the sensors are digitally converted and compared with the stored value for the recognition of sign and then displayed as a text on the 16x2 LCD. Further the text output is wirelessly transmitted to a cellular phone or a PC which consist of a test to speech conversion software. Currently we are working on a simple prototype that will convert the basic Alphabets and numeric which will be further extended for recognition of words.

Full text

International Journal of Recent Innovation in Engineering and Research
Scientific Journal Impact Factor - 3.605 by SJIF
e- ISSN: 2456 – 2084
@IJRIER-All rights Reserved -2017 Page 37
SIGN LANGUAGE TO SPEECH CONVERSION USING ARDUINO
Yash Jhunjhunwala1, Pooja Shah2, Pradnya Patil3 and Jyoti Waykule4
1,2,3,4 Sou. Sushila Danchand Ghodawat Charitable Trust’s Sanjay Ghodawat group of Institutions
Abstract- A huge population in India alone is of the dumb and deaf people. So the system is
working on a glove based device which will be used for conversion of sign language (ASL) to
speech .The basic system consists of a two parts; sign language recognition and conversion to text
and further to speech. The sign language glove consist of a simple hand gloves fitted with flex
sensors which is being used for the monitoring the amount of bend on the fingers. Flex means bend,
this is the sensors that change the resistance depending on the amount of bend on the sensor. Data
from the sensors is send to the Control unit which is the Arduino Nano the analog signals from the
sensors are digitally converted and compared with the stored value for the recognition of sign and
then displayed as a text on the 16x2 LCD. Further the text output is wirelessly transmitted to a
cellular phone or a PC which consist of a test to speech conversion software. Currently we are
working on a simple prototype that will convert the basic Alphabets and numeric which will be
further extended for recognition of words.
Key words- Sign language, Arduino, ASL, Sound Glove, Dumb, Gesture recognition, Deaf, Flex.
I. INTRODUCTION
Sign language is a natural way of communication between normal and dumb people. Sign
language is mostly dependent on hand gesture recognition. It is sometimes not easy for normal
people to recognize the signs properly and understand what they want to say. So the intension of the
gloves is to make the life style of the dumb and deaf people easy. The gloves translates the hand
gestures to text and further speech so that the normal people can read the recognized gesture and hear
to the voice and understand what that person wants to tell, which will make the communication more
efficient. The system consists of both physical and non-physical communication. Sign language
differ from country to country it is not universally same. America developed American Sign
Language (ASL); British developed British Sign Language and so on. Most of the countries follow
the American Sign Language and our system is also based on the same. The gloves convert the
specific gestures to text then to speech using Arduino as heart of the system. The flex sensors are
used in the system which is attached onto the gloves which convert the gesture into resistance which
is further converted to the text through Arduino nano. The flex sensors come from flexible sensors
family, which are flexible enough. Along with flex sensors accelerometer and contact sensors are
also used for accurate output. The accelerometer is used to monitor the motion of the hand and to
monitor the contact between the fingers the contact sensors are used. The selection of the sensors is
based on the singes the language is consisting. Some singes are dependent on the movement of palm
so to monitor that accelerometer is used while some signs are dependent on the contact of the fingers
so to get the specific output if that signs contact sensors are used. The output of the sensors is
processed on Arduino nano to get text as an output displayed on LCD. Further that text is sent via
Bluetooth module to mobile phones/computers. Further that data is converted to speech via text to
speech conversion software. There is no such commercial system available in the market to convert
sign language into speech. However research is being made to convert sign to speech and make it
portable, efficient and highly accurate.

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 38
Figure 1:- American Sign Language[7]
.
II. LITREATURE REVIEW
Thomas Pryor and Navid Azodi are UG students who made the Gloves that translate sign
language into text and speech known as signaloud. They had won Lemelson-MIT student price for
this project. Which inspired me to make a system of my own which can be used for public welfare
[1].The 1st Hand Talk glove was designed by Ryan Patterson in the year 2001. This model had
limitations that a computer or a laptop was always required for its functioning which made it less
portable. n 1620, Juan Pablo Bonet published R, education of letters and art for teaching mute people
to speak which is considered as the first modern treatise of sign language phonetics, setting out a
method of oral education for deaf people and a manual alphabet [4]. Interactive Acclerometric Glove
For Hearing Impaired: Kuldeep Singh Rajput, Shashank Deshpande, UmaMudenagudi: The main
aim is to set an interface between the Hearing Impaired people and normal person to improve the
communication efficiency so that they can communicate handily with others.[5] Hand Gesture
Recognition System :Swapnil D. Badgujar , GourabTalukdar ,Omkar Gondhalekar, Feb. 2014.
Implemented by real time gesture recognition a user can control a computer by doing a decided
gesture in front of a video camera which is linked to the computer. [5]
III. DESIGN METHODOLOGY
A) Block Diagram
Figure 2:- Block Diagram
The overall functioning of the System is explained through the block diagram shown in
figure. It represents the general order and hierarchy of various working blocks of the project. The

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 39
person wears the glove which has flex sensors, contact sensors and accelerometer stitched to it and
makes the gesture according to the American Sign Language (ASL). Arduino Nano is used to gather
signals from the flex sensors and accelerometer placed on the glove. Then the processed output is
send over the LCD to display the text output and also via a Bluetooth link to an Android Smartphone
or a Personal Computer consisting of test to speech software (application) and speech output is
obtained.
B) Flow Chart
Figure above shows the flow of algorithm used in the system. The sign language gloves are
stitched with 5 flex sensors over the thumb, index, middle, ring, pinkey fingers on the hand, contact
sensor and the accelerometer so recognize the exact movement of the fingers. Each flex sensor is
supplied with the initial voltage as per the movement of the fingers there will be a voltage drop
because of the change in the resistance due to the bending of the flex sensor on the fingers. The
voltage thus obtained will be analog in nature. This analog voltage is then converted to digital
voltage using an analog to digital converter (ADC) using the Arduino Nano.
Figure 3:- System Flow Chart
The Digital output is then compared with the preloaded values of sensor in the system for the
Alphabets and numbers, if the digital output matches the preloaded values then the recognition takes
place of the Gestures and the compared output is given as a text output on the Liquid Crystal Display
(LCD) which will help a normal person to read and understand. Further the same output is
transmitted over a Bluetooth module and then via Bluetooth link output is sent to android
Smartphone or a personal computer which is connected to the Bluetooth module and consisting of
Text to Speech (TTS) conversion software (application) the serial input is received via the Bluetooth
link on the connected device and a text output is received on the application with speech output from
the speakers. This will help normal people to read the output and understand the gesture or else listen
to the audio and communicate easily with the dumb and deaf people.

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 40
IV. COMPONENTS USED
A) Flex Sensor
Flex sensor means flexible sensors, flexible sensor are sensors which change their resistance
depending upon the bend on the sensor. The more the bend the resistance is also more.
Figure 4: - Flex Sensor
Figure 5:- Voltage Divider (Flex Sensor connection)[3]
Flex sensor also work as variable analog voltage divider. Flex sensor consists of carbon
resistive element within a thin flexible substrate. When the substrate is bent the resistive element
produces a resistive output relative to the bend radius. The system consists of 5 flex sensors which
are stitched on the fingers of the gloves because the main part of the gestures are fingers so for each
finger a separate sensor is required. The hand gesture is inputted to the system via flex sensors the
bent of each finger describes as shown in American Sign Language. As the fingers bends the sensors
also starts bending and as the sensors bends the resistances also changes accordingly and that
resistance value is inputted to the Arduino Nano.
B) Accelerometer (ADXL338)
Figure 6: - ADXL-338
The accelerometer is a thin, small, low power. The accelerometer consists of 3-axis X, Y, Z
the change in position of the accelerometer changes the output of the X, Y, and Z axis according to
that output of the axis the gesture is recognized. The sensor measures the static acceleration of
gravity in tilt-sensing applications and dynamic acceleration resulting from motion, vibration. The
accelerometer contains a Polysilicon surface semicromachined structure built on top of silicon wafer.
The sensor measures the deflection of the surface and gives the deflection corresponding to particular
axis. The accelerometer is placed on the wrist to sense the bending motion. Connection of
accelerometer to the Arduino
 Output pin connected to one of the analog pin on Arduino
 Vin pin is connected to 3.3V pin on Arduino
 GND pin is connected to ground pin on Arduino
C) Liquid Crystal Display (LCD)
Figure 7:- LCD

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 41
A 16x2 LCD (Liquid Crystal Display) is used in the system to display 16 characters. Each
line displays 8 characters. Each character is further divided into 5x7 pixels matrix format. LCD
contains tow registers namely command register and data register to send command to the LCD
command register is used whereas to send data on the LCD screen to be displayed is saved on data
register. The intensity of the screen can also be controlled by using 10k pot connected to V0 pin of
LCD. The LCD works on 4 bit mode as well as on 8 bit mode according to system requirement.
D) Bluetooth module (HC 05)
Figure 8: - HC-05
The Bluetooth module is used to transfer text data from Arduino to mobile phone/computer.
The Bluetooth module sends serial data to the target device and the target device displays that data
on the screen. The data send to the Bluetooth module is the alphabets, numerical values, etc these
data is then serially transferred to the target device and displays the output and further to software of
speech conversion.
E) Arduino Nano
Figure 9: - Arduino Nano Board
Arduino is an open-source computer hardware and software company, project and user
community that designs and manufactures kits for building digital devices and interactive objects
that can sense and control the physical world.
F) MIT App Inventor
It is an application which is used to prepare android applications. In the inventor application
all the components that an android phone consists are available like buttons, text boxes, etc, Sensors
like accelerometer, barcode detector, location sensor, etc, social things like email, texting, twitter,
etc, connectivity like Bluetooth, activity starter, web, etc. The app is used develop the application for
text to speech conversion. In that application the device like android phone or computer is connected
to the Bluetooth module of the device through Bluetooth connectivity of the device. After connecting
to the system serial data is received and we get text to speech output.
V. RESULTS
Figure 10: - Initial Setup

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 42
The above system provides with conversion of Sign gestures to a limited scope; recognition
of alphabets from A-Z and numeric’s from 0-9.It displays the output on a LCD screen and sound
output is obtained on the android Smartphone which is still in process. Hardware setup is show in the
figure which shows the general components used in the system i.e. Arduino Nano, flex sensors,
LCD, Bluetooth device HC-05, Accelerometer.
Figure 11: - System Prototype
Figure 12(a): - Final Prototype
Figure 12(b): - Final Prototype
The figure 12(a) & 12(b) shows the final prototype whereas the figure 11 shows the initial
prototype of the system. Initially the glove was made of net fabric and the sensors were stitched on it.
The glove was not giving the accurate value as it should be. And as we have stitched the sensor on
the net the fabric started tearing so due to which the output was not accurate. The net gloves were
also having a problem of fitting. To overcome to the problem we have replaced the net gloves with

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 43
cotton gloves and stitched the sensors and Arduino nano board with LCD display and Bluetooth
module is stitched on the cotton glove. The whole PCB of Aurdino nano and Bluetooth module is
fitted on the glove with the help of belt stitched on the glove as shown in figure 12(a). The belt on
the glove provides perfect fitting to the user and also fits the whole circuit on the glove.
Figure 13: - System with android application
The above figure shows the whole system (gloves, LCD, android application on android
phone). As shown in the figure the gloves are connected and software is installed on the android
phone which looks as shown in figure. On the application the block “Select BT module” is selected
and then the list of available Bluetooth device is displayed select the Bluetooth device having name
“HC-05” and connect the android phone to the circuit. The output is then received on the screen in
red color and also in the audio format.
The figure 14 shows the output of alphabet “B” as shown in figure 1 the following gesture
indicates the letter B. the sensor (flex) value for fingers index, middle, ring, pinky is 255 whereas the
sensor(flex) valve for thumb is below 100. When these conditions are satisfied then only the alphabet
B will be displayed on LCD and via Bluetooth on android phone and then speech output is obtained.
Accordingly the output of letters “A, C, D, E, F, H, I, M, W” if successfully achieved.
Figure 14: - System with android application
VI. FUTURE WORK
1. The device can be made more portable by using a small lipo battery which can be charged and
used again and again.
2. The user interface of the system can be improved.
3. The other glove can be developed so that both hand movements can be done and sentence
formation can be made.

Volume: 02 Issue: 01 January – 2017 (IJRIER)
Available Online at : www.ijrier.com Page 44
VII. CONCLUSION
The system is a bridge between normal and dumb\deaf people; it fills the gaps of
communication between dumb\deaf people and normal people. The gloves are independent and it is
portable with minimum weight and low power. The system converts the hand gestures to the text and
further to speech. If in some conditions the person can’t hear the sound produced there is a provision
provided into the system of text due to which the person can read and understand what the another
person wants to convey.
REFERENCES
[1] http://lemelson.mit.edu/winners/thomas-pryor-and-navid- azodi.
[2] https://www.arduino.cc/en/uploads/Main/ArduinoNanoMa nual23.pdf
[3] https://www.sparkfun.com/datasheets/Sensors/Flex/flex22 .pdf
[4] http://www.in.techradar.com/news/wearables/These-gloves-literally-turn-sign-language-into-speech/articleshow/5181
0332.cms
[5] http://www.ijste.org/articles/IJSTEV2I9089.pdf
[6] Solanki Krunal M, “Indian Sign Languages using Flex Senser Glove,” International Journal of Engineering Trends
and Technology (IJETT) - Volume4 Issue6- June 2013 ISSN: 2231
[7] http://www.statesymbolsusa.org/symbol-official-item/maine/state-language-poetry/american-sign-language