A Real Time Air Mouse Using Video Processing

Abstract

Hand gesture recognition is a simplest and innovative way to connect with computer ,since interactions with the computer can be increased through multi dimensional use of hand gestures .hand gesture recognition is one of the active region of research in computer vision. When user doesn't have technical knowledge about the system then human computer interaction enable the user to use system without any problem. They still will be able to use the system with their normal hands. Gestures communicate the meaning of statement said by the human being. Hand gesture has been one of the most common and natural communication media among human being. Hand gesture recognition research has gained a lot of attentions because of its applications for interactive human-machine interface and virtual environments.

Full text

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4635
A Real Time Air Mouse Using Video Processing.
Aviral Gupta Dr. Neeta Sharma
M.Tech. Scholar, Assistant Professor
Software Engineering (SE) Software Engineering (SE)
Noida International University Noida international University
Abstract
Hand gesture recognition is a simplest and innovative way to connect with computer ,since interactions
with the computer can be increased through multi dimensional use of hand gestures .hand gesture
recognition is one of the active region of research in computer vision. When user doesn’t have technical
knowledge about the system then human computer interaction enable the user to use system without any
problem. They still will be able to use the system with their normal hands. Gestures communicate the
meaning of statement said by the human being. Hand gesture has been one of the most common and
natural communication media among human being. Hand gesture recognition research has gained a lot
of attentions because of its applications for interactive human-machine interface and virtual
environments.
Keywords: gesture recognition, vision based hand gesture recognition, human computer interaction.
I. INTRODUCTION
1.1 Overview
Human Computer Interaction today greatly emphasizes on developing more spontaneous and natural
interfaces. The Graphical User Interface (GUI) on Personal Computers (PCs) is quiet developed, well
defined an efficient interface for a user to interact with the computer and access the various applications
effortlessly with the help of mice, track pad, etc. In the present day scenario most of the mobile phones
are using touch screen technology to interact with the user. But this technology is still not cheap to be
used in desktops and laptops. Our objective was to create a virtue mouse system using Web camera to
interact with the computer in a more user friendly manner that can be alternative approach for the touch
screen.
Touch screen are also a good control interface and nowadays it is used globally in many applications.
However, touch screen cannot be applied to desktop systems because of cost and other hardware
limitations. By applying vision technology and controlling the mouse by natural hand gestures, reduce the
work space required. On developing this approach that uses a video device to control the mouse system.
This mouse system can control all mouse tasks, such as clicking (right and left), double clicking and
scrolling.

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4636
Many researchers in the human computer interaction and robotics fields have tried to control mouse
movement using video devices. However, all of them used different methods to make Clicking event. One
approach, by Erdem et al, used fingertip tracking to control the motion of the mouse. A click of the mouse
button was implemented by defining a screen such that a click occurred when a user’s hand passed over
the region [1, 3]. Another approach was developed by Chu-Feng Lien [4]. He used only the finger-tips to
control the mouse cursor and click. His clicking method was based on image density, and required the
user to hold the mouse cursor on the desired spot for a short period of time. Paul et al, used still another
method to click. They used the motion of the thumb (from a ‘thumbs-up’ position to a fist) to mark a
clicking event thumb. Movement of the hand while making a special hand sign moved the mouse pointer.
Our project was inspired by a paper of Kamran Niyazi [8] et al used Web camera to detect color tapes for
cursor movement. The clicking actions were performed by calculating the distance between two colored
tapes in the fingers. In their paper [5] K N Shah et al have represented some of the innovative methods of
the finger tracking used to interact with a computer system using computer vision. They have divided the
approaches used in Human Computer Interaction (HCI) in two categories: 1. HCI without using interface
and 2. HCI using interface. Moreover they have mentioned some useful applications using finger tracking
through computer vision. This method uses gesture based interactive experiment using finger movements
to stimulate mouse operations. As compared with the traditional segmentation method this method has
two benefits one is that it uses colored tape and another is that it requires no special object model with
relative high performance. These two benefits make the system applicable to the augmented reality
systems or other real-time systems. Since the system is based on image capture through a webcam, it is
dependent on illumination to a certain extent. Furthermore the presence of other colored objects in the
background might cause the system to give an erroneous response. Although by configuring the threshold
values and other parameters of the system this problem can be reduced but still that the operating
background be light and no bright colored objects be present. The system might run slower on certain
computers with low computational capabilities because it involves a lot of complex calculations in a very
small amount of time. However a standard pc or laptop has the required computational power for
optimum performance of the system. Another fact is that if the resolution of the camera is too high then
the system might run slow. However this problem can be solved by reducing the resolution of the image
by making changes in the system.
1.2 Objective
The great challenge lies in developing an economically feasible and hardware independent system so that
humans can interact without having any physical connection with the computer.
The main objective is to develop an object tracking application to interact with the computer, and a virtual
human computer interaction device, where in no physical contact with the device is required and also to
provide an easier human-machine interaction routine.
1.3 Purpose
The purpose of the project is to create a virtual mouse that works with the help of a web camera. In this
project a camera continuously takes images of hand movements of a user, which is then mapped into
mouse inputs. This means that we can give inputs to computer without having any physical connection
with the computer and without having any hardware movements.

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4637
1.4 Problem Definition
To develop a software solution to a problem, the first step is to understand the problem. The problem here
is to develop a way so that humans can interact with a computer without having any physical connection
with the computer. Many ideas were put forward but they all required physical movement of hardware.
Another idea put forward was to use the principle of photoelectric effect. But for that a special hardware
is needed and it is not economically feasible. So the final decision is to develop a virtual mouse which
uses simple and cheap image processing techniques.
II. PROPOSED WORK
The proposed system is a real time video processing that is based on a real time application system. This
can replace one of the traditionally used input device i.e. mouse so that simply by using the hand gestures
the user will be able to interact naturally with their computer. The basic block diagram of the overall
proposed system is as shown below in the figure.
Basic block diagram As soon as the palm is detected by the camera, it starts capturing the images and
takes out the current images and starts processing. The processed image is given as input to the next phase
of processing where it starts detecting the palm following the centers and edges. These all extracted
features are used for the further processing such as detecting hand gestures and working according to it.
The flowchart of the proposed system above is shown for the single frame which is captured. The system
is a real time system so the flowchart is a continuous process for each and every frame that is captured by
the web camera.
In the protest following application one of the principle issues is question identification. Rather than
fingertips, a shading pointer has been utilized to make the question recognition simple and quick. A circle
blue sticker is utilized as a shading pointer in this review. To mimic the snap occasions of the mouse three
fingers with three shading pointers has been utilized.
The basic block diagram of the system

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4638
Figure- 1 The Basic Block Diagram
System Approach
 Capturing real time video using Web-Camera.
 Processing the individual image frame.
 Flipping of each image frame.
 Conversion of each frame to a grey scale image.
 Color detection and extraction of the different colors (RGB) from flipped gray scale image.
 Conversion of the detected image into a binary image.
 Finding the region of the image and calculating its centroid.
 Tracking the mouse pointer using the coordinates obtained from the centroid.
 Simulating the left click and the right click events of the mouse by assigning different color
pointers.
III. RESERCH METHODOLOGY
3.1Capturing the real time video
For the framework to work we require a sensor to distinguish the hand developments of the client. The
webcam of the PC is utilized as a sensor. The webcam catches the constant video at a settled casing rate
and determination which is controlled by the equipment of the camera. The casing rate and determination
can be changed in the framework if required. Computer Webcam is utilized to catch the Real Time Video
Video is partitioned into Image outlines in light of the FPS (Frames every second) of the camera
Processing of individual Frames

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4639
Figure-2 Capturing the video
3.2 Flipping of Images
At the point when the camera catches a picture, it is upset. This implies on the off chance that we move
the shading pointer towards the left, the picture of the pointer moves towards the privilege and the other
way around. It's like a picture acquired when we remain before a mirror (Left is distinguished as right and
right is recognized as cleared out). To maintain a strategic distance from this issue we have to vertically
flip the picture. The picture caught is a RGB picture and flipping activities can't be straightforwardly
performed on it. So the individual shading channels of the picture are isolated and afterward they are
flipped independently. In the wake of flipping the red, blue and green hued channels separately, they are
connected and a flipped RGB picture is gotten.
Figure-3 Flipping of an Image

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4640
The following images show the entire flipping process in real time.
Figure-4 Flipping of a Real time Image
3.3 Conversion of Flipped Image into Gray scale Image
When contrasted with a shaded picture, computational many-sided quality is diminished in a dim scale
picture. In this way the flipped picture is changed over into a dim scale picture. All the fundamental
operations were performed in the wake of changing over the picture into dark scale.
3.4 Colour Detection
This is the most critical stride in the entire procedure. The red, green and blue shading article is
recognized by subtracting the flipped shading smothered channel from the flipped Gray-Scale Image. This
makes a picture which contains the recognized protest as a fix of dim encompassed by dark space.
Figure-5 Detection of Red Color

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4641
3.5 Performing Clicking Actions
The control activities of the mouse are performed by controlling the banners related with the mouse
catches. JAVA is utilized to get to these banners. The client needs to perform hand motions keeping in
mind the end goal to make the control activities. Because of the utilization of shading pointers, the
calculation time required is lessened. Besides the framework ends up noticeably impervious to foundation
clamor and low light conditions. The discovery of green and blue hues takes after a similar strategy
examined previously. Clicking activity depends on concurrent recognition of two hues.
If Red alongside Green shading is recognized, Left clicking activity is performed
If Red alongside Blue shading is identified, Right clicking activity is performed
Figure-6 flowchart

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4642
3.6 Color Detection
To recognize the blue shade of the pointer, MATLAB's implicit "imsubtract" work has been utilized.
Imsubtract capacity can be utilized as;
Z = imsubtract(X,Y)
Where; it subtracts every component in cluster Y from the relating component in exhibit X and returns the
distinction in the comparing component of the yield cluster Z. X and Y are genuine, non-inadequate
numeric varieties of a similar size and class, or Y is a twofold scalar. The cluster returned, Z, has an
indistinguishable size and class from X unless X is coherent, in which case Z is twofold. Be that as it
may, this approach is for all intents and purposes hard to apply particularly in Live video because of the
surrounding light. There is one more arrangement where one can recognize a specific shading say red,
blue or green. In spite of the fact that this is not adaptable for all hues, but rather it effortlessly wipes out
the issue of surrounding light. Here, we will utilize a similar approach so as to recognize red shading.
Fig :7 Color Detection
IV. EXPERIMENTAL RESULTS
The necessary steps to be followed are as follows:
 Firstly, we need to interface the camera using MATLAB.
 Start the camera to capture the images continuously.
 Capturing the real time video using the started up camera.

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4643
 Processing the individual image frame obtained from the video.
 Flipping of each image frame.
 Conversion of each frame to a GRAY-SCALE image.
 Color detection and extraction of different colors.
 Conversion of each frame into BINARY-SCALE image.
 Find the center and draw the bounding box.
 Track mouse pointer using coordinates of centroid.
Different mouse actions are performed based on the steps below.
a. First, we need to detect red color in the image.
b. If Red color pointer is detected, we need to track the mouse pointer.
c. If a single Blue color pointer is detected, we need to perform Left click.
d. If two Blue pointers are detected, we need to perform Right clicking action.
e. If three Blue pointers are detected, we need to perform Double clicking action.
f. If none of the color has been detected again we need to try tracking different colors from the image.
Fig: 8 Double click event
Fig:-9 Right click event

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4644
Fig :-10 Left click event
Fig:-11 Mouse movement event.
Fig:12 Mouse scrolling event.
V. CONCLUSION
The proposed system is used to control the mouse cursor and implement its function using a real-time
camera. We implemented mouse movement, selection of the icons and its functions like right, left, double
click and scrolling. This system is based on image comparison and motion detection technology to do
mouse pointer movements and selection of icon. From the results, we can expect that if the algorithms can
work in all environments then our system will work more efficiently. This system could be useful in
presentations and to reduce work space. In the future, we plan to add more features such as enlarging and
shrinking windows, closing window, etc. by using the palm and multiple fingers.
FUTURE WORK
There are as yet numerous changes that can be made to our framework like enhancing the execution of the
present framework and including components, for example, broadening and contracting windows,

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4645
shutting window, and so forth by utilizing the palm and different fingers. The present framework is
variation to reflection and scale changes and requires legitimate hand signals, great enlightenment
innovation and intense camera for the execution of mouse capacities. Accuracy can simply be expanded at
the cost of review by including more stages, however each progressive stage sets aside twice as much
opportunity to discover harder negative examples and the applications which advantage from this
innovation. We introduce a picture seeing application for instance of where this innovation could prompt
a more characteristic UI. The same could be said for exploring something like Google Maps or perusing
envelopes on a screen. In any case, the applications reach a long ways past that. They are especially
convincing in circumstances where touch screens are not relevant or not as much as perfect. For instance,
with projection frameworks, there is no screen to touch. Here vision-based innovation would give a
perfect substitution to touch screen innovation. Likewise out in the open terminals, consistent utilize
brings about the spread of soil and germs. Vision-based frameworks would evacuate the need to touch
such setups, and would bring about enhanced connection.
REFERENCES
[1] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. “Computer vision based mouse”, Acoustics,
Speech, and Signal Processing, Proceedings. (ICASS). IEEE International Conference, 2012
[2] Hojoon Park, “A Method for Controlling the Mouse Movement using a Real Time Camera”, Brown
University, Providence, RI, USA, Department of computer science, 2013
[3] Chu-Feng Lien, “Portable Vision-Based HCI – A Realtime Hand Mouse System on Handheld
Devices”, National Taiwan University, Computer Science and Information Engineering Department
[4] Kamran Niyazi, Vikram Kumar, Swapnil Mahe, Swapnil Vyawahare, “Mouse Simulation Using
Two Coloured Tapes”, Department of Computer Science, University of Pune, India, International
Journal of Information Sciences and Techniques (IJIST) Vol.2, No.2, March 2012
[5] K N. Shah, K R. Rathod and S. J. Agravat, “A survey on Human Computer Interaction Mechanism
Using Finger Tracking” International Journal of Computer Trends and Technology, 7(3), 2014, 174-177
[6] Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, 2nd edition, Prentice Hall,
Upper Saddle River, New Jersey, 07458
[7] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project
Report, December 18, 2013
[8] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. “Computer vision based mouse”, Acoustics,
Speech, and Signal Processing, 2012. Proceedings. (ICASS). IEEE International Conference
[9]Hojoon Park. A Method for Controlling Mouse Movement using a Real-Time Camera.
ww.cs.brown.edu/research/pubs/theses/masters/2010/park.pdf2010.
[10]Computer vision based mouse, A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. Acoustics,
Speech, and Signal Processing, 2002. Proceedings. (ICASS). IEEE International Conference.

International Journal of Advanced Science and Technology
Vol. 29, No. 2, (2020), pp. 4635 - 4646
`
ISSN: 2005-4238 IJAST
Copyright ⓒ 2020 SERSC 4646
[11] Chu-Feng Lien, “Portable Vision-Based HCI – A Real-time Hand Mouse System on Handheld
Devices”, National Taiwan University, Computer Science and Information Engineering Department
[12] Chu-Feng Lien, Portable Vision-Based HCI - A Real-time Hand Mouse System on Handheld
Devices.
[13] Hojoon Park, “A Method For Controlling The MouseMovement using a Real Time Camera”, 2008,
Brown University,Providence ,RI ,USA, Department of computer science
[14] Chris Solomon and Toby Breckon, “Fundamentals Of Digital Image Processing”, Third edition,
John Wiley & Sons, Ltd,2011