ArticlePDF Available

Selective neutralisation and deterring of cockroaches with laser automated by machine vision

Authors:

Abstract and Figures

Controlling insect pests still relies on the extensive usage of generic and established methods, such as pesticides, which utilise broad spectrum chemicals or toxins persisting in the environment and targeting non-pest insect species. Therefore, more effective and environmental friendly approaches are needed to counteract these damaging effects. Since a laser can be remotely directed to neutralise undesirable targets, this approach could be highly promising for controlling insect pests in a selective and ecofriendly fashion. In this study, we present a laser system automated by machine vision for neutralising and influencing the behaviour of insect pests. By performing experiments on domicili-ary cockroaches, Blattella germanica, we demonstrate that our approach enables the immediate and selective neutrali-sation of individual insects at a distance up to 1.2 m. We further show the possibility to deter cockroaches by training them not to hide under a dark shelter through aversive heat conditioning with a low power-laser. Parameters of our prototype system can readily be tuned for applications in various situations and on different pest species like mosquitoes, locusts, and caterpillars. The prospect of this study is to pursue the creation of a standalone, safe for the environment , compact, low-cost, and energy-efficient device system for pest control. ARTICLE HISTORY
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=toin20
Oriental Insects
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/toin20
Selective neutralisation and deterring of
cockroaches with laser automated by machine
vision
Ildar Rakhmatulin, Mathieu Lihoreau & Jose Pueyo
To cite this article: Ildar Rakhmatulin, Mathieu Lihoreau & Jose Pueyo (2022): Selective
neutralisation and deterring of cockroaches with laser automated by machine vision, Oriental
Insects, DOI: 10.1080/00305316.2022.2121777
To link to this article: https://doi.org/10.1080/00305316.2022.2121777
© 2022 The Author(s). Published by Informa
UK Limited, trading as Taylor & Francis
Group.
Published online: 21 Sep 2022.
Submit your article to this journal
View related articles
View Crossmark data
Selective neutralisation and deterring of cockroaches
with laser automated by machine vision
Ildar Rakhmatulin
a
, Mathieu Lihoreau
b
and Jose Pueyo
c
a
School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh, UK;
b
Research
Center on Animal Cognition (CRCA), Center for Integrative Biology (CBI); CNRS, University Paul
Sabatier, Toulouse, France;
c
Brighton and Sussex Medical School, University of Sussex, Brighton, UK
ABSTRACT
Controlling insect pests still relies on the extensive usage of
generic and established methods, such as pesticides, which
utilise broad spectrum chemicals or toxins persisting in the
environment and targeting non-pest insect species.
Therefore, more eective and environmental friendly
approaches are needed to counteract these damaging
eects. Since a laser can be remotely directed to neutralise
undesirable targets, this approach could be highly promising
for controlling insect pests in a selective and ecofriendly
fashion. In this study, we present a laser system automated
by machine vision for neutralising and inuencing the beha-
viour of insect pests. By performing experiments on domicili-
ary cockroaches, Blattella germanica, we demonstrate that
our approach enables the immediate and selective neutrali-
sation of individual insects at a distance up to 1.2 m. We
further show the possibility to deter cockroaches by training
them not to hide under a dark shelter through aversive heat
conditioning with a low power-laser. Parameters of our pro-
totype system can readily be tuned for applications in various
situations and on dierent pest species like mosquitoes,
locusts, and caterpillars. The prospect of this study is to
pursue the creation of a standalone, safe for the environ-
ment, compact, low-cost, and energy-ecient device system
for pest control.
ARTICLE HISTORY
Received 17 October 2021
Accepted 2 September 2022
KEYWORDS
Laser; pest control; German
cockroaches; Blattella
germanica; laser insect
control
Introduction
Pest control is a major issue for the food industry and public health. Current
agrochemical practices for controlling harmful insects are generally proble-
matic because they lead to resistance and often impact non-targeted species.
This is the case of the massive utilisation of non-specific insecticides for crop
protection that largely contributes to pollinator declines (Colin 2020). To
overcome these problems, biocontrol solutions have been developed using
natural predators or chemical traps releasing molecules involved in the
CONTACT Ildar Rakhmatulin i.rakhmatulin@hw.ac.uk School of Engineering and Physical Sciences,
Heriot-Watt University, Edinburgh, UK
ORIENTAL INSECTS
https://doi.org/10.1080/00305316.2022.2121777
© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives
License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduc-
tion in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.
Published online 21 Sep 2022
communication of targeted pests. However, even when these practices are
possible, not only a detailed knowledge of the biology of species is required
but also their impact has to be taken in into consideration prior any efficient
implementation. Thus, a more generic approach that can be targeted to
specific animal pests without damaging the environment should be
developed.
Laser (amplification of light by means of stimulated emission) may
provide an alternative approach for the selective neutralisation of unwanted
targets, such as animal pests and weeds. For instance, Brown et al. (2021)
discussed the possibility of using a laser for birds to protect sweet corn fields.
Furthermore, Xiong et al. (2017) tested a robot with a static laser for
weeding during strawberry harvesting. However, a major drawback of this
approach for large-scale deployment is that it requires specific types of
equipments that are costly and complex to implement. Recently, we success-
fully tested the possibility of using a laser to kill mosquitoes (Rakhmatulin
2021), weeds (Rakhmatulin 2020) and created a simulation program for the
destruction of locusts and caterpillars (Rakhmatulin 2021). Despite in these
previous studies an economical and energy-efficient laser prototype was
developed, we envisioned major health and safety risks that could be
triggered by the use of high laser power, such as eye damage and fire
ignition, which prevented the large-scale expansion of our prototype.
Therefore, further laser theory considerations were needed in order to
develop robust and safe biocontrol approaches.
The idea of using a galvanometer (i.e., an electromechanical instru-
ment for measuring electric current) was previously used for directing
the laser in other applications. For instance, Hegna et al. (2010) used
a laser galvanometer to tilt the beam from a laser range finder along an
object of interest to determine its geometric properties and therefore
developing a more effective 3-D scanning system. In addition, Huang
et al. (2020) considered the magnetic properties of the rotor and con-
cluded that the energy consumption of the motor can be reduced by
reducing the total number of changes in the position of the mirrors
per second. In this paper, we have tested the efficiency of our improved
automated vision driven Laser prototype to with the German cockroach
Blattella germanica (Linnaeus 1767), which is one of the major urban
pest species worldwide (Lihoreau et al. 2012). These domiciliary cock-
roaches can live in populations of millions of individuals, potentially
spoiling food, household, and electrical appliances (Nasirian 2017), but
also being a serious health hazard triggering allergic reactions and the
development of asthma. Despite various methods are used to control
cockroaches, there is still no perfect solution (Pan 2020). The most
efficient approaches so far are mechanical (e.g., sticky traps) and chemi-
cals (insecticidal gels and pastes). These methods have the disadvantage
2I. RAKHMATULIN ET AL.
of having a limited catching range and therefore many individuals may
not visit the trap areas. In addition, long-term chemical treatments may
lead to problematic resistances.
We have improved our previous prototype by using galvanometer mir-
rors driven by neural networks algorithms to control the laser beam direc-
tion to efficiently target moving cockroaches in a controlled environment.
This work demonstrates that our prototype is able to neutralise and deter
cockroaches efficiently at specific distances depending on the laser power
utilised. Thus, our approach (laser-targeted) could offer an effective, eco-
friendly, and low-cost alternative to control the number of cockroaches and
possibly other insects in different environments (i.e., domestic households,
fields).
Methods
We have improved our previous approach (Rakhmatulin 2021) by develop-
ing the laser system automated by machine vision for neutralising and
deterring moving insect pests. Guidance of the laser by machine vision
allows for faster and more selective usage of the laser to locate objects
more precisely, therefore decreasing associated risks of off-target laser
exposure. In this device, the laser beam is controlled by galvanometer
mirrors driven by a Jetson Nano single-board computer which uses deep
learning algorithms (neural networks) for detecting moving objects and
training.
Setup overview
We developed a laser device automated by machine vision (Fig. 1).
The single-board computer Jetson nano 1 (CPU Quad-core ARM
Cortex-A57 MPCore processor, NVIDIA, USA) processes the digital signal
from two cameras (IMX219, Sony, Japan) – 2 and determines the position of
the target insect −3 in the 3-D space. After calculations, the Jetson nano
transmits the analog signal in the 0–5 V range to the board with an
operational amplifier. This board converts the signal into a bipolar voltage
(± 5 V), which powers the boards with a motor driver for the galvanometer –
4 (Unbranded/Generic, style DMX Stage Ligh type: DHR814494, the pro-
tocol for laser control: ILDA DB25, AUCD, China). The galvanometer, with
the help of mirrors, changes the direction of the laser – 6 to shoot the target
(see details in Supplementary materials). We used generally available equip-
ment at a low-price range (all devices cost not more than 250 $US).
Programming was made in python 3.7. The device algorithm is presented
in Fig. 2.
ORIENTAL INSECTS 3
Machine vision
Machine vision is the limiting element of the entire system since its accuracy
depends on many factors: hardware, neural network model, dataset, etc. To
reach high tracking accuracy, we used external focusing systems
(Rakhmatuliun 2021). We used camera IMX219 with an 8-megapixel matrix
to allow for optimal detection of objects. To implement the neural network,
we used Yolov4-tiny to search for 1 object in real-time video with
a resolution of 416 × 416 pixels. We chose Yolo because it has high speed
and accuracy of declaration (Kuznetsova et al. 2021). Images for training the
neural network were labelled manually using https://github.com/tzutalin/
labelImg. The resulting dataset is publicly available at https://www.kaggle.
com/ildaron/tracking-a-cockroach-at-home. We used 1000 images that
were prepared with different lighting to make the research more effective
at any time of the day. We made 1000 images with cockroaches in our test
box that we then manually labelled by marking the position of each cock-
roach with a red rectangle on each image (Fig. 3A). The detection of
cockroaches in the red rectangles was made by YoloV4-tiny realised in the
Darknet framework (https://pjreddie.com/darknet/) on Jetson nano
(Fig. 3B). The speed of the neural network can be greatly improved. We
tested our model on various frameworks and obtained the following results
(Table 1).
Figure 1. Summary diagram of the laser setup: 1 – transparent box containing cockroaches, 2 –
Pi cameras, 3 – Jetson nano, 4 – laser, 5 – galvanometer, 6 – laser beam, L – distance between
laser device and target.
4I. RAKHMATULIN ET AL.
Laser operation
Liu et al. (2020) described in detail the physics of the galvanometer opera-
tion process. The galvanometer scanner system consists of two orthogonal
mirrors (X mirror and Y mirror) driven by two motors. Considering the
voltages applied to the X and Y mirrors, the mirrors create certain angles of
rotation. Both rotation angles are proportional to the input voltages. It is
important to note that the position of the mirrors only needs to be changed
100 times instead of 20,000 times. The accuracy of the galvanometer is
controlled by the use of iterative learning as proposed by Dai et al. (2019),
Figure 2. Algorithm of the laser operation for the neutralisation of cockroaches.
ORIENTAL INSECTS 5
which achieves high speed, linear, and accurate bi-directional scanning for
laser microscopy. In addition, the galvanometer is able to operate at differ-
ent temperatures.
The positions of the mirrors were calculated as follows. For the tasks of
stereo vision, we developed a stereo camera, and followed a detailed calibra-
tion as described in Rakhmatulin (2021b). Stereo vision permits the deter-
mination of the distance to the object along the z-axis. Then, the angle of the
mirror rotation is calculated using the laws of optics (Fig. 5A). Once the
object coordinates are determined then, the required angle of rotation for
the mirror is calculated through the tangent of the angle α, which is the ratio
of the opposite leg to the adjacent leg. The used galvanometer had a step
width of 30°, but as it turned out we have only 20° (Fig. 4A-D).
The principle of the determination of the distance to the object and the
dimensions of the device is shown in Fig. 5. The laser beam obeys all the
optical laws of physics. Therefore, depending on the design of the galvan-
ometer, the required angle of inclination of the mirror – α, can be calculated
through the geometrical formulas. In our case, through the tangent of the
angle α, where it is equal to the ratio of the opposing side – X(Y) (position
calculated by deep learning) to the adjacent side Z (calculated by stereo
vision). The galvanometer is a fairly accurate device and can direct the laser
to tens of metres with an accuracy of several centimetres (Belosludtsev
Figure 3. Machine vision for cockroach detection: A, labelling process of cockroaches; B, an
example of a neural network YoloV4 for detecting a cockroach.
Table 1. Frames per second (FPS) for the Yolov4-tiny model when running on the
Jetson Nano.
Methods FPS Source
Keras 4–5 https://github.com/jkjung-avt/tensorrt_demos
Darknet 12–15 https://github.com/AlexeyAB/darknet
Tensor RT 24–27 https://github.com/jkjung-avt/tensorrt_demos
DeepStream 23–26 https://github.com/marcoslucianops/DeepStream-Yolo
tkDNN 30–35 https://github.com/ceccocats/tkDNN
6I. RAKHMATULIN ET AL.
2021). In the general case, we experimentally established that the mechan-
ical part of the device allows directing the laser to 10 m with an accuracy of
up to 1 cm. To calibrate the system, we set the coordinates programmatically
through the OpenCV library and checked whether the galvanometer directs
the laser beam to the given point. Eight points were arbitrarily chosen for
1000 mm, 2000 mm, and 3000 mm (Ruler Accuracy: 0.2 mm; Line
Expansion, Tritec, Japan).
Cockroaches husbandry
We used German cockroaches Blattela germanica for testing the viabi-
lity of our laser device for pest control. A population of 150 individuals
composed of nymphs and adults (sex ratio 1:1) was obtained from a pet
Figure 4. Maximum and minimum positions of galvanometer mirrors: A, lower position – 35° for
x mirror; B, upper position 55° for x mirror; C, lower position for y mirror; D, upper
position – 25° for y mirror.
Figure 5. The developed device (dimensions in mm): A, determination of the distance to the
object, z and z coordinates when the mirror is located at an angle of 35°; B, side view: 1 PI
cameras, 2 – galvanometer, 3 – Jetson nano, 4 – adjusting the position to the object, 5 – laser
device, 6 – power supply, 7 – galvanometer driver boards, 8 – analog conversion boards.
ORIENTAL INSECTS 7
store (https://herasato-s.business.site/). We indiscriminately used
nymphs and adults ( and ) during the experiments. Cockroaches
were placed in the clear plastic box (height 8 cm × length 24 cm ×
depth 6 cm) without food or water for 24 hours before the experiments.
We carried out the experiments in a room maintained at 24°C, under
65% humidity (temperature sensor with an accuracy 0.1 C and humidity
sensor with accuracy of 1%, TDJstudio, Japan) and with the lighting of
150 lux (Digital Lux Metre BT-881D Illumination Metre, BTMETER,
China). Depending on the experiment, the transparent box containing
the cockroaches was placed on a table at a distance of either 600 mm or
1200 mm from the laser.
Laser neutralisation approach
We used three types of lasers:
- Power laser, 300 mW, 450 nm Oxlasers, China;
- Power laser, 1600 mW, 808 nm Oxlasers, China;
- Power laser, 100 mW, 660 nm Oxlasers, China.
To test the accuracy of our laser prototype in neutralising freely moving
cockroaches in the box, we used a group of four cockroaches per box, as it is
the limiting number of objects that can efficiently been tracked at high speed
with the neural network algorithm. Five groups of cockroaches were used as
controls (N = 20 cockroaches). In these control groups, cockroaches could
move freely and interact in the box for 10 min. In the experimental groups
(N = 15 groups, 60 cockroaches) cockroaches were exposed to the laser. We
compared two laser power (300 mV and 1600 mW) and two distances
between the laser and the box containing cockroaches (600 mm and
1200 mm). Different laser spot sizes (2 mm, 3 mm, 5 mm) for the
1600 mW were also tested to ascertain the efficacy of the laser intensity.
We compared the area of action in the box of the cockroaches in the control
and laser groups over time. We measured cockroach movements (travelled
distance) using a library OpenCV cv2.TrackerCSRT_create, the laser detec-
tion accuracy, and efficiency of neutralisation.
Laser deterring approach
For deterring cockroaches from aggregating in a dark shelter upon light
exposure, we used a laser with a lower power of 100 mW that triggers
fleeing. The shelter was an opaque plastic plate (dimensions 500 × 300 ×
10 mm) (see images in Appendix 2). The box containing cockroaches was
placed at a distance of 300 mm from the laser. We ran experiments with five
control groups (N = 20) that could move freely for 12 h. We also observed
five experimental groups (N = 20), in which we shot cockroaches with a low-
8I. RAKHMATULIN ET AL.
power laser every time they were detected hiding under the shelter. For
measuring the deterring effect of the laser, a virtual region of interest (ROI)
in the test box with the OpenCV library was created (see Fig. 6) and then we
counted the number of times cockroaches entered the ROI area using the
Figure 6. The process of experiments for Cockroach training: A, ROI for neutralising laser and
control group; B, laser group; C, demonstration of the operation of a powerful laser – 100 mW.
ORIENTAL INSECTS 9
OpenCV function and compared data at the beginning of the experiments
and after 12 h of laser exposure.
Results
Neutralising cockroaches
To test the efficacy of our laser system in neutralising freely moving cock-
roaches in an enclosed chamber (box), we used two different laser power
settings, each targeting specimens at two different distances and measured
cockroach movements, and neutralisation efficacy (Table 2; see methods).
On average, the cockroaches ran at a speed of 1.3 metres per second
(4.8 km/h) with an error of no more than 10% (speed error is average error
found in the movements of cockroaches, N = 20). Overall, cockroaches in
the laser groups covered much more distance than cockroaches in the
control groups as the latter were undisturbed from the thermal effect
induced by the laser.
There was an increase in the movement speed of cockroaches targeted
with the 300 mW laser in comparison with those shot with the 1.6 W laser.
This speed increment in the lower laser group is because to neutralise the
individuals generally the laser needs to shoot the target continuously for
about 2–3 seconds (Stopwatch – 8RDA55-002 Digital Stopwatch 055 Water
Resistant Rainproof Black, Accuracy 0.01 s, Citizen CITIZEN, Japan) and at
this lower laser power, cockroaches do not die instantly, but instead moved
quickly outside the laser beam.
Examples of the thermal effects of the laser on cockroaches can be found
in Fig. 7. Overall, the laser 1.6 W was more efficient for neutralising
cockroaches than the laser 300 mW. Laser neutralisation was also signifi-
cantly faster with 1.6 W than with 300 mW.
Next, we searched for the relationship between the laser distance and the
laser spot diameters with the neutralisation efficacy. First, we found out that
the higher the distance between the laser and the box, the longer it took for
the lasers to neutralise the cockroaches. Detection and shot accuracy were
also greatly reduced with distance from laser. Second, we tested the different
diameters of laser spots for high 1.6 W laser at 300 mm to target. As it is
shown in Fig. 8, the dose–effect curve for laser neutralisation of cockroaches
for different laser spots indicates that the optimal laser spot size is 3 mm.
Theoretically, reducing the size spot of the laser to 1 mm should be more
efficient in neurtralisation, especially if the laser hits the cockroach’s abdo-
men. However, the spot 1 mm is extremely difficult to implement experi-
mentally. In addition, it is important to note that when a laser spot touches
the most distal part of the leg (arolium an pulvilli) containing heat receptors,
it triggers the activation of the escape response thus provoking only slight
10 I. RAKHMATULIN ET AL.
Table 2. Technical parameters during neutralising cockroaches process with sample size – 5.
Group
Distance – camera and
cockroaches, mm, (error –
0.05 mm)
Covered distance, cm
(error – 10 mm)
Cockroaches
neutralised,%
(error – 0)
Neutralisation time,
sec (error 0.5 sec)
Detection accuracy,%
(by Yolov4-tiny)
Shot accuracy, %
(error – visually)
Laser 300 mW Control 600 130 - - 83 -
Laser 1950 100 265 85 86
Laser 1600 mW Control 150 - - 88 -
Laser 300 100 5 82 90
Laser 300 mW Control 1200 144 - - 62 -
Laser 2450 100 350 64 29
Laser 1600 mW Control 130 - - 60 -
Laser 1550 100 65 62 35
ORIENTAL INSECTS 11
damage and sparing the individual from lethal exposure. Thus, although the
spot size concentration directly affects the efficiency of the laser, our experi-
ments show that for effective neutralisation a laser power of 1600 mW and
a spot size of 3 mm must hit its body area, otherwise hitting the appendages
triggers the escape response, increases speed and reduces neutralisation.
Since we cannot control which part of the body is hit by the laser in each
neutralisation attempt, and technical gaps preclude us for targeting the laser
in a specific area, then further improvements are needed in our targeted
laser prototype for effective neutralisation.
Cockroaches are most active at night whereas during daytime they rest in
secluded places, such as shelters (Lihoreau et al. 2012). Therefore, in the
presence of light, cockroaches have a strong tendency to aggregate under
Figure 8. Dose-effect curve for laser 1600 mW control of cockroaches where 1 laser spot
3 mm, 2 – laser spot 2 mm, 3 – laser spot 5 mm. Cockroaches were placed 300 mm to the laser
device.
Figure 7. Result of laser exposure during 0.5 sec: A, detected cockroaches; B, neutralised
cockroaches by laser 300 mv; C, neutralised cockroaches by laser 1.6 W; D, cockroach box for
the control and laser group.
12 I. RAKHMATULIN ET AL.
these dark shelters (Halloy 2007). Having demonstrated the ability of the
high-power laser to neutralise cockroaches, we tested whether our system
could successfully deter cockroaches, by training them not to aggregate
under a preferred dark shelter based on negative reinforcement with low-
power laser. The laser was located 300 mm to the experimental box contain-
ing cockroaches. The results are shown in the Table 3.
Cockroaches in the control group spent more time under the dark shelter
than in other areas in the box throughout the experiments, whereas in the
experimental laser group cockroaches showed a strikingly different trend. In
the latter, cockroaches tended to aggregate under the shelter at the begin-
ning of the experiment similar to the control ones. After 12 hours of laser
treatment, these specimens were no longer in the shelter. Instead, they were
resting in initially non-preferred areas under the light. Similar changes in
behaviour (shelter avoidance) were observed when measuring first the
number of times cockroaches entered under the shelters, with the experi-
mental group showing a lower number of entries than the control one
(Table 3), and second by measuring the time spent in the different areas
of the box, in which the experimental group showed a cumulative time
increase in non-preferred areas and a subsequent time reduction in shelter
area (Fig. 9).
Interestingly, the laser-induced shelter avoidance seems to be effective
from 8 hr onwards when cockroaches in the laser groups showed a steeply
Figure 9. Heatmap of movement of cockroaches (average of 5 groups over 12 h). X-axis
distance in millimetres. Y-axis time in seconds. A, laser groups; B, control groups. The black
rectangle is region of interest.
Table 3. Technical parameters during deterring cockroaches process.
Group
Sample
size
(groups)
Time in shelter,
min (Accuracy
0.5 sec)
Cockroaches
entered the shelter
(Accuracy 5%)
Laser operation,
times (Accuracy
0.5 sec)
Tracking
accuracy, %
(by Yolov4-
tiny)
Laser
precision, %
(error –
visually)
Control
group
5 45 96 - 88 -
Laser
group
5 2 45 80 89 85
ORIENTAL INSECTS 13
reduced ability to enter the shelter (Fig. 10). Such decline in ability was not
observed in control groups. These differences cannot be explained by
differences in the automated tracking accuracy, which was similar in control
and laser groups throughout the experiments (Table 3). However, it is
important to note that even do the data presented is an average of the
results obtained, we have not controlled for the number of laser shots being
received by each cockroach during the experiments. Despite this, these
results indicate that our laser set-up is able to deter sheltering of cock-
roaches and therefore could be implemented for conditioning insect pest
behaviours.
Discussion
In this paper, we have described the development of a safe, compact, low-
cost, and energy-efficient laser device automated by machine vision, which
is able to neutralise and influence the behaviour behaviour of insect pests in
an environmentally friendly fashion. We have included a GitHub database
with all technical details in open-source with GPL-3.0 licence. It is very
important to note that the usage of this device must be restricted to areas
where the possibility of the laser beam hitting another non-targeted living
specimen is minimised. The aim of this work is to demonstrate the feasi-
bility of using a laser to control insects pest, but we also understand that
Figure 10. Number of times cockroaches entered the shelter (i.e., the region of interest: ROI).
The X-axis is the time in hours. On the y-axis of attempted entries. Green line – laser group, blue
line – control group.
14 I. RAKHMATULIN ET AL.
there exist safety issues of using lasers in insect control processes and they
would require further study.
Here, we have improved a previous prototype by leveraging deep neural
networks through the use of the Jetson nano, an on-pay computer-office
suite, and its CUDA, a hardware-software architecture for parallel comput-
ing. We studied the effect of laser size and capability on the performance of
a laser. We also significantly increased the working distance of the galvan-
ometer. This device is a high-quality effective, with low energy consump-
tion, destruction of one pest will cost a few joules.
By using B. germanica as a pest model, our work has shown that at higher
laser power settings this device can selectively neutralise freely moving
cockroaches at a distance beyond 1 m. In addition, in a low power setting,
this laser device is able to precisely and persistently deploy heat at 300 mm
distance inducing cockroach heat escape response and therefore influencing
their shelter behaviour. Despite these findings, our program which num-
bered cockroaches periodically failed and did not allow us to calculate how
many times each individual cockroach needed to be hit with the laser to
avoid hiding. To resolve this issue, we could either use only one single
cockroach per experiment or improve the tracker program. Further
improvements in increasing the tracking range can be achieved by replacing
the camera with a higher width and focus. At first with simple means, like
the Haar cascade by OpenCV library, we can find an area with an object and
then focus on this area using a telephoto lens and get an image for a neural
network. Thus, our approach provides a plausible alternative solution to
mechanical traps and chemicals (pesticides) for the selective neutralisation
and deterring of not only cockroach populations but other insect pests as
this system is readily tunable and could be applied for tracking and con-
trolling other targets for instance to protect against mosquitoes, bees from
predatory hornets, bees from ectoparasites.
Although this prototype is suitable for academic research, several factors
must be considered before exploring the large-scale deployment of this
technology. One of the main limiting factors is the danger of the laser
beam hitting the eyes. The laser can enter a blood vessel and clog it, get
into a blind spot where nerves from all over the eye go to the brain, burn out
a line of ‘pixels’ and then the damaged retina can begin to flake off, and this
is the path to complete and irreversible loss of vision (Schirmacher 2010).
The danger of lasers is considered based on whether it can cause damage
before the eye reflexively blinks and it is considered not too dangerous
a power of 5 mW for visible radiation. As we have shown with cockroaches,
5 mW is not enough to control pests. To overcome this problem, we can
develop additional security systems, such as human detection and audio
sensors. But in any case, we are not able to make the installation 100% safe,
since even a laser can be reflected and damage the eye of a person who is not
ORIENTAL INSECTS 15
in the field of view of the device and at a distant distance. Therefore, this
technology should not be used at home (for a review of eye safety of laser see
Sliney 2009).
However, this technology could still be useful in non-human-driven
applications (robot-driven) or when high laser speed is not required, for
example, for weed control. Our prototype can be tuned for this purpose by
using a microcontroller instead of a computer, which will make the system
more reliable and energy-efficient. We can also utilise the remote control of
IP camera for stereo vision task and neural networks can be used to
determine the distance (Zhang 2021). As a processor, any low-cost single-
board computer such as RaspberryPI, BananaPI, Google Coral Edge TPU
and ASUS Tinker Board can be incorporated. However, in all these board
computers there is no possibility of using the GPU due to the hardware and
software architecture of parallel computing – CUDA.
These devices can be used for other possible applications, such as real-time
detection of objects with a bright background with OpenCV; or transfer images
over the network and carry out calculations remotely. For either application
out-of-the-box solutions, such as AWS IoT Greengrass Platform, which extends
Amazon Web Services AWS capabilities to edge devices can be used to allow
them to work locally with the data they create. Furthermore, Google Colab is
also a service like Jupyter-Notebook that has been offering free access for a long
time to GPU instances. Colab GPUs have been updated to the new NVIDIA T4
GPUs. This update unlocks new software packages, which means we can now
experiment with RAPIDS for free on Colab. Based on CUDA-X AI, the
RAPIDS suite of software libraries gives the freedom to run end-to-end data
processing and analysis pipelines entirely on GPUs. In addition, despite the
Jetson Nano has been used in our prototype, another more expensive devices
such as Jetson TX2, Jetson Xavier NX, and Jetson AGX Xavier can also be
considered. Other considerations to take into account are power supply, size,
and periphery, which users can vary depending on the specifics of the project.
Finally, while we used YoloV4 in this work through Keras, it is advisable to
switch to YOLOv5 in PyTorch. YOLOv5-7, which conceptually gives a much
better percentage of recognition on the COCO dataset than previous versions.
One of the advantages of our laser prototypes is the cost of installation
which does not exceed several hundred dollars, and can even be reduced to
$30 when using a stm32 series microcontroller instead of Jetsonnano with
a CDMI protocol with an OpenCV library and also by replacing the fitted
motor driver for less powerful ones with a mirror rotation angle of no more
than 100 times per second. These improvements will allow us to create
a ‘pocket version’ of the device, which could be easily fitted with other
apparatus for the using this laser technology in the study of complex pro-
cesses at the cellular and tissue levels. Finally, it has been proposed that ARM
and FPGA processors for embedded laser marking controllers can operate
16 I. RAKHMATULIN ET AL.
outdoors even in harsh environmental conditions (Wang et al. 2014). In our
prototype, a standard galvanometer was installed, but it can be upgraded for
various specific purposes such as operating in outdoor settings.
Thus, this study is a proof of principle that our automated laser targeting
prototype is an effective, and low-cost approach for controlling freely mov-
ing insects. Further improvements will allow the development of a more
efficient device for targeting insect pests, weeds, and other desirable speci-
mens in different environments.
Disclosure statement
No potential conflict of interest was reported by the authors.
References
Belosludtsev A, Buinovskis D. 2021. Significant increase of UV reflectivity of SiC galvan-
ometer mirror scanners for the high-power laser applications. Optics & Laser
Technology. 140. doi:10.1016/j.optlastec.2021.107027.
Brown R, Brown D. 2021. Robotic laser scarecrows: a tool for controlling bird damage in
sweet corn. Crop Protection. 146:105652. doi:10.1016/j.cropro.2021.105652.
Colin T, Coline M, Lihoreau M, Barron A. 2020. Pesticide dosing must be guided by
ecological principles. Nature Ecology and Evolution. 4(12):1575–1577. doi:10.1038/
s41559-020-01302-1.
Dai W, Zhang W, Zheng Z, Li J. 2019. Investigation of micron-sized fish-scale surface
structures on tool steel surfaces using laser galvanometer scanning. Applied Surface
Science. 470(15):1111–1121. doi:10.1016/j.apsusc.2018.10.264.
Halloy J, Sempo G, Caprari G, Rivault C, Asadpour M, Tache F, Said I, Durier V, Canonge S,
Ame JM. 2007. Cockroaches to control self-organized choices. Science. 318(5853):1155.
doi:10.1126/science.1144259.
Hegna H, Pettersson K. 2010. Inexpensive 3-D laser scanner system based on
a galvanometer scan head. International Archives of Photogrammetry, Remote Sensing
and Spatial Information Sciences; Newcastle upon Tyne, UK. Vol. XXXVIII, Part 5
Commission V Symposium.
Huang Y, Zhang S, Gao L, Zheng Y. 2020. Angle measurement technology based on
magnetization vector for narrow space applications. Nanotechnology and Precision
Engineering. 3(3):167–173. doi:10.1016/j.npe.2020.09.001.
Kuznetsova A, Maleva T, Soloviev V. 2021. YOLOv5 versus YOLOv3 for apple detection.
Cyber-Physical Systems: Modelling and Intelligent Control. doi:10.1007/978-3-030-
66077-2_28
Lihoreau M, Costa J, Rivault C. 2012. The social biology of domiciliary cockroaches: colony
structure, kin recognition and collective decisions. Insect Soc. 59:445–452. doi:10.1007/
s00040-012-0234-x.
Liu C, Zang C, Zhou B. 2020. Extension of continuous scanning laser Doppler vibrometry
measurement for complex structures with curved surfaces. Chinese Journal of
Aeronautics. 7:145–150.
ORIENTAL INSECTS 17
Nasirian H. 2017. Infestation of cockroaches (Insecta: Blattaria) in the human dwelling
environments. A systematic review and meta-analysis. Acta Tropica. 167:86–98. doi:10.
1016/j.actatropica.2016.12.019.
Pan X, Zhang F. 2020. Advances in biological control of the German cockroach, Blattella
germanica (L.). Biological Control. 142:104104. doi:10.1016/j.biocontrol.2019.104104.
Rakhmatulin I. 2021a. Raspberry PI for kill mosquitoes by laser. Preprints. 2021010412.
doi:10.20944/preprints202101.0412.v1.
Rakhmatulin I. 2021b. Detect caterpillar, grasshopper, aphid and simulation program for
neutralizing them by laser. doi:10.21203/rs.3.rs-242641/v1
Rakhmatulin I, Andreasen C. 2020. A concept of a compact and inexpensive device for
controlling weeds with laser beams. Agronomy. 10(10):1616. doi:10.3390/
agronomy10101616.
Rakhmatulin I, Pomazov E. 2021. Low-cost stereovision system (Disparity Map) for few
dollars. Preprints. 2021040282. doi:10.20944/preprints202104.0282.v1.
Rakhmatuliun I. 2021. Machine vision for low-cost remote control of mosquitoes by power
laser. Journal of Real-Time Image Processing. doi:10.1007/s11554-021-01079-x
Schirmacher A. 2010. Eye protection for short and ultra-short pulsed laser systems
Augenschutz für Kurz- und Ultrakurzpulslasereinrichtungen. Medical Laser
Application. 25(2):93–98. doi:10.1016/j.mla.2010.01.004.
Sliney D. 2009. Chapter 24 - eye safety of laser and light-based devices. In: Cosmetics
applications of laser & light-based systems. Personal care & cosmetic technology. p. 499–516
Wang D, Xinpiao Y. 2014. An embedded laser marking controller based on ARM and FPGA
processors. The Scientific World Journal. 7:1–8. doi:10.1155/2014/716046.
Xiong Y, Ge Y, Grimstad L. 2017. An autonomous strawberry-harvesting robot: design,
development, integration, and field evaluation. Hoboken (NJ): Wiley Online Library.
doi:10.1002/rob.21889
Zhang C, Wu J, Chen Z, Liu W, Li M. 2021. Dense-CNN: dense convolutional neural
network for stereo matching using multiscale feature connection. Signal Processing:
Image Communication. 95:116285.
18 I. RAKHMATULIN ET AL.
ResearchGate has not been able to resolve any citations for this publication.
Preprint
Full-text available
The paper presents an analysis of the latest developments in the field of stereo vision in the low-cost segment, both for prototypes and for industrial designs. We described the theory of stereo vision and presented information about cameras and data transfer protocols and their compatibility with various devices. The theory in the field of image processing for stereo vision processes is considered and the calibration process is described in detail. Ultimately, we presented the developed stereo vision system and provided the main points that need to be considered when developing such systems. The final, we presented software for adjusting stereo vision parameters in real-time in the python language in the Windows operating system.
Chapter
Full-text available
The use of the YOLOv3 and YOLOv5 algorithms for apple detection in fruit-harvesting robots are compared. It is shown that the YOLOv5 algorithm could detect apples in orchards without additional pre- and post-processing with 97.8% Recall (fruit detection rate), and 3.5% False Positive Rate (FPR). It is much better than YOLOv3 that gives 90.8% Recall and 7.8 FPR when combined with special pre- and post-processing procedures, and then 9.1% Recall and 10.0% FPR without pre- and post-processing.
Preprint
Full-text available
More than 700 thousand human deaths from mosquito bites are observed annually in the world. It is more than 2 times the number of annual murders in the world. In this regard, the invention of new more effective methods of protection against mosquitoes is necessary. In this article for the first time, comprehensive studies of mosquito neutralization using machine vision and a 1 W power laser are considered. Developed laser installation with Raspberry Pi that changing the direction of the laser with a galvanometer. We developed a program for mosquito tracking in real. The possibility of using deep neural networks, Haar cascades, machine learning for mosquito recognition was considered. We considered in detail the classification problems of mosquitoes in images. A recommendation is given for the implementation of this device based on a microcontroller for subsequent use as part of an unmanned aerial vehicle. Any harmful insects in the fields can be used as objects for control.
Preprint
Full-text available
The protection of crops from pests is relevant for any cultivated crop. But modern methods of pest control by pesticides carry many dangers for humans. Therefore, research into the development of safe and effective pest control methods is promising. This manuscript presents a new method of pest control. We used neural networks for pest detection and developed a powerful laser device (5 W) for their neutralization. In the manuscript methods of processing images with pests to extract the most useful feature are described in detail. Using the following pets as an example: aphids, grasshopper, cabbage caterpillar, we analyzed various neural network models and selected the optimal models and characteristics for each insect. In the paper the principle of operation of the developed laser device is described in detail. We created the program to search a pest in the video stream calculation of their coordinates and transmission data with coordinates to the device with the laser.
Article
Full-text available
In this paper, we present an innovative and effective method for remote monitoring of mosquitoes and their neutralization. We explain in detail how we leverage modern advances in neural networks to use a powerful laser to neutralize mosquitoes. The paper presented the experimental low-cost prototype for mosquito control, which uses a powerful laser to thermally neutralize the mosquitoes. The developed device is controlled by a single-board computer based on the neural network. The paper demonstrated experimental research for mosquito neutralization during which, to maximize approximation to natural conditions, simulation of various working conditions was conducted. The manuscript showed that a low-cost device can be used to kill mosquitoes with a powerful laser.
Article
Full-text available
With the wide application of laser in the field of skin plastic surgery, micro laser galvanometer scanner has made great progress in this field with its portability. However, the measurement method used to measure the deflection angle of laser galvanometer in the narrow space of scanner with high precision remains to be studied. In this paper, an angle measurement method based on magnetic field is proposed, and the effect of the shapes of permanent magnets (PMs) on the measurement is studied by theoretical and experimental study under the condition that the maximum available space for the PMs is a 10 mm side cube. An angle measuring experimental device is set up, and the contrast experiment is carried out with different PMs which are the same as simulation. The experimental results show that cylindrical PM is more suitable than other PMs, which is consistent with the simulation results, and the maximum nonlinearity error is 0.562%. This method has the advantages of small volume, non-contact measurement, small moment of inertia, good dynamic response and no external excitation for the PMs, so it has a broad application prospect in micro laser galvanometer scanner.
Article
Full-text available
A prototype of a relatively cheap laser-based weeding device was developed and tested on couch grass (Elytrigia repens (L.) Desv. ex Nevski) mixed with tomatoes. Three types of laser were used (0.3 W, 1 W, and 5 W). A neural network was trained to identify the weed plants, and a laser guidance system estimated the coordinates of the weed. An algorithm was developed to estimate the energy necessary to harm the weed plants. We also developed a decision model for the weed control device. The energy required to damage a plant depended on the diameter of the plant which was related to plant length. The 1 W laser was not sufficient to eliminate all weed plants and required too long exposure time. The 5 W laser was more efficient but also harmed the crop if the laser beam became split into two during the weeding process. There were several challenges with the device, which needs to be improved upon. In particular, the time of exposure needs to be reduced significantly. Still, the research showed that it is possible to develop a concept for laser weeding using relatively cheap equipment, which can work in complicated situations where weeds and crop are mixed.
Article
In spite of the fact that convolutional neural network-based stereo matching models have shown good performance in both accuracy and robustness, the issue of image feature loss in regions of texture-less, complex scenes and occlusions remains. In this paper, we present a dense convolutional neural network-based stereo matching method with multiscale feature connection, named Dense-CNN. First, we construct a novel densely connected network with multiscale convolutional layers to extract rich image features, in which the merged multiscale features with context information are utilized to estimate the cost volume for stereo matching. Second, we plan a novel loss-function strategy to learn the network parameters more reasonably, which can develop the performance of the proposed Dense-CNN model on disparity computation. Finally, we run our Dense-CNN model on the Middlebury and KITTI databases to conduct a comprehensive comparison with several state-of-the-art approaches. The experimental results demonstrate that the proposed method achieved superior performance on computational accuracy and robustness of disparity estimation, especially achieving the significant benefit of feature preservation in ill-posed regions.
Article
Sweet corn (Zea mays L.) is an important direct retail and local wholesale crop in the United States. Birds have long been recognized as a pest of corn, which they damage by shredding the husks and eating the kernels. This study was performed to evaluate moving green laser beams as a bird control strategy in sweet corn. A portable, battery-powered robotic scarecrow was designed to continuously move a 14 mm diameter, 532 nm wavelength beam from a 50 mW laser. The scarecrow was tested in sweet corn fields in Rhode Island, USA over three years using a split-field design where half of each field was covered by the laser and the other half served as a control. . In 2017, the control sections averaged 48.4 ± 6.9 damaged ears per plot, while the sections covered by the laser averaged 14.6 ±5.5 damaged ears (P = 0.0002). In 2018, the control sections averaged 23.8% ± 4.1% damaged ears while 13.7% ±2.1% of ears were damaged in sections covered by the laser (P = 0.0046). In 2019, mean damage values across all planting blocks were 14.9% ± 4.1% for the protected plots and 20.3% ±5.8% for the unprotected plots (P = 0.0332). The results of this study show that automated laser scarecrows can reduce bird damage to sweet corn under field conditions. Further research, including aviary studies with controlled populations of individually identified birds, are needed to measure efficacy and test for habituation.
Article
Silicon carbide (SiC) galvanometer mirror scanners are widely used for high-speed laser scanning systems engineering. For bare SiC galvo-scanners, the working range and laser induced damage threshold (LIDT) are limited. In order to overcome these limits special surface enhancement should be done. In present research, an optimized multilayer coating approach was suggested. This method ensures a significant increase in SiC ultraviolet (UV) range reflectivity. The SiC reflectivity was enhanced from 25% up to the 98.5% at the 355 nm wavelength for the angles of reflection from 34 ° to 56 °. Moreover, the multilayer coating designs suitable for the high-power laser systems was optimized and deposited on SiC. What is more important, it was demonstrated how to reach better flatness of the SiC without affecting the coating roughness. Finally, the coating has passed the high-temperature annealing, humidity chamber and scotch-tape tests needed for final application.