ArticlePDF Available

Abstract and Figures

In this work, we show the success of unsupervised transfer learning between Electroencephalographic (brainwave) classification and Electromyographic (muscular wave) domains with both MLP and CNN methods. To achieve this, signals are measured from both the brain and forearm muscles and EMG data is gathered from a 4-class gesture classification experiment via the Myo Armband, and a 3-class mental state EEG dataset is acquired via the Muse EEG Headband. A hyperheuristic multi-objective evolutionary search method is used to find the best network hyperparameters. We then use this optimised topology of deep neural network to classify both EMG and EEG signals, attaining results of 84.76% and 62.37% accuracy, respectively. Next, when pre-trained weights from the EMG classification model are used for initial distribution rather than random weight initialisation for EEG classification, 93.82%(+29.95) accuracy is reached. When EEG pre-trained weights are used for initial weight distribution for EMG, 85.12% (+0.36) accuracy is achieved. When the EMG network attempts to classify EEG, it outperforms the EEG network even without any training (+30.25% to 82.39% at epoch 0), and similarly the EEG network attempting to classify EMG data outperforms the EMG network (+2.38% at epoch 0). All transfer networks achieve higher pre-training abilities, curves, and asymptotes, indicating that knowledge transfer is possible between the two signal domains. In a second experiment with CNN transfer learning, the same datasets are projected as 2D images and the same learning process is carried out. In the CNN experiment, EMG to EEG transfer learning is found to be successful but not vice-versa, although EEG to EMG transfer learning did exhibit a higher starting classification accuracy. The significance of this work is due to the successful transfer of ability between models trained on two different biological signal domains, reducing the need for building more computationally complex models in future research.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.DOI
Cross-domain MLP and CNN Transfer
Learning for Biological Signal
Processing: EEG and EMG
JORDAN J. BIRD1, JHONATAN KOBYLARZ2, DIEGO R. FARIA1, ANIKÓ EKÁRT1, AND
EDUARDO P. RIBEIRO2
1School of Engineering and Applied Science, Aston University, Birmingham, United Kingdom
2Department of Electrical Engineering, Federal University of Parana, Curitiba, Brazil
Emails: {birdj1, d.faria, a.ekart}@aston.ac.uk1, jhonatankobylarz@gmail.com2, edu@eletrica.ufpr.br2
JJB and JK are co-first authors
ABSTRACT In this work, we show the success of unsupervised transfer learning between Electroen-
cephalographic (brainwave) classification and Electromyographic (muscular wave) domains with both MLP
and CNN methods. To achieve this, signals are measured from both the brain and forearm muscles and EMG
data is gathered from a 4-class gesture classification experiment via the Myo Armband, and a 3-class mental
state EEG dataset is acquired via the Muse EEG Headband. A hyperheuristic multi-objective evolutionary
search method is used to find the best network hyperparameters. We then use this optimised topology
of deep neural network to classify both EMG and EEG signals, attaining results of 84.76% and 62.37%
accuracy, respectively. Next, when pre-trained weights from the EMG classification model are used for
initial distribution rather than random weight initialisation for EEG classification, 93.82%(+29.95) accuracy
is reached. When EEG pre-trained weights are used for initial weight distribution for EMG, 85.12% (+0.36)
accuracy is achieved. When the EMG network attempts to classify EEG, it outperforms the EEG network
even without any training (+30.25% to 82.39% at epoch 0), and similarly the EEG network attempting
to classify EMG data outperforms the EMG network (+2.38% at epoch 0). All transfer networks achieve
higher pre-training abilities, curves, and asymptotes, indicating that knowledge transfer is possible between
the two signal domains. In a second experiment with CNN transfer learning, the same datasets are projected
as 2D images and the same learning process is carried out. In the CNN experiment, EMG to EEG transfer
learning is found to be successful but not vice-versa, although EEG to EMG transfer learning did exhibit
a higher starting classification accuracy. The significance of this work is due to the successful transfer of
ability between models trained on two different biological signal domains, reducing the need for building
more computationally complex models in future research.
INDEX TERMS Applied Machine Learning, Biological Signal Processing, EEG, EMG, Knowledge
Adaptation, Neural Networks, Transfer Learning
I. INTRODUCTION
It is no secret that the hardware requirements of Deep
Learning are far outgrowing the average consumer level
of resource availability, even when a distributed processing
device such as a GPU is considered [1]. In addition to this,
limited data availability often hampers the machine learning
process. It is for these reasons that researchers often find
similar domains to transfer the learning between, effectively
saving computational resources through said similarities by
applying cross-domain interpretation. By doing so, once
impossible tasks become possible, despite limited resources.
A well-known example is VGG (Visual Geometry Group), a
set of 16 and 19 hidden-layer Convolutional Neural Networks
(CNNs) which have been trained to the extreme on a large
image dataset [2]. Useful recognisable features from images
such as points, lines, curves, and geometric shapes can be
transferred over to a differing CNN task since these features
always exist within the domain. Thus, cross-domain transfer
learning is enabled in order to interpret new data [3], [4].
VOLUME 4, 2016 1
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
Electrical biological signals show a similarly non-
juxtapose pattern of behaviour [5], [6], and thus the domain-
transfer may be possible, although it is currently not yet well-
understood. If it is possible, then to what extent and effects
are those possibilities?
Here, we study for the first time whether cross-domain
transfer learning can impact the classification ability of mod-
els when trained on Electroencephalographic (brainwave)
and Electromyographic (muscular wave) data. This is per-
formed through the transfer of initial weights via the best
models of each, and learning is continued from this ini-
tial starting point. When compared to the classical method
of random weight distribution initialisation, we argue that
knowledge can be transferred from EMG to EEG and vice-
versa, succesfully. We also compare the results to a model
fine-tuned by ImageNet weights in order to discern that use-
ful domain-related knowledge is actually being transferred
rather than simply general image rules, which could be learnt
from any range of sources and domains.
With better classification results come higher impact ap-
plications. In the domain of Human-Robot Interaction, the
control of prosthetic devices [7]–[9], enabling telepresence
within settings such as care assistance [10], [11], as well
as within hazardous settings such as bomb disposal [12],
and remote environments [13], as well as risk of potential
injury [14]–[16] are just a few of many possible fields that
successful knowledge transfer could potentially advance,
through both improved classification ability and lower com-
putational expense required to train models.
The most notable scientific contributions of this work are
the following:
1) The collection of an original EMG dataset of hand
gestures gathered from the left and right forearms.
2) Derivation of a strong set of neural network hyper-
parameters through an evolutionary search algorithm,
via a multi-objective fitness function towards the best
interpretation and classification ability of both EEG
and EMG data.
3) Successful transfer of knowledge between the two do-
mains through unsupervised transfer learning, enabling
increased classification ability of the neural networks
when weights are transferred between them as opposed
to traditional random initial weight distribution. Bet-
ter starting abilities, learning curves, and asymptotes
of the network learning process are observed when
knowledge is transferred.
4) To the authors’ knowledge, cross-domain transfer
learning is performed between differing biological sig-
nals (EEG and EMG) for the first time.
The remainder of this article is structured as follows: In
Section II important background on Artificial Neural Net-
works and the domains of EEG and EMG are described,
and also the concept of Transfer Learning is introduced. In
Section III the experimental setup is described. The MLP and
CNN transfer learning experiments are described in Sections
IV and V, and subsequent findings are then presented in Sec-
tion VI. Finally, future work and conclusions are presented in
Section VII.
II. BACKGROUND
In this section, the important scientific concepts as the basis
of this work, as well as notable state-of-the-art research
within Transfer Learning and Biological Signal Processing
are considered.
A. MULTILAYER PERCEPTRON ARTIFICIAL NEURAL
NETWORKS
A Multilayer Perceptron (MLP) is an Artificial Neural Net-
work (ANN) trained via validation, backpropagation of er-
rors [17] and a subsequent gradient descent optimisation al-
gorithm [18] in order to perform a classification or regression
prediction task [19]. The task in the context of this work is to
label a wave, i.e., what class the wave data belong to, based
on statistical descriptions of the wave behaviour. The goal of
the learning process is to reduce the error rate of the output
of the network when compared to the ground truth; the loss
function minimised in the experiments reported here is the
cross-entropy loss [20], [21] of the networks:
M
X
c=1
yo,c log(po,c),(1)
where Mis the number of classes (3 for EEG, 4 for EMG),
yis a binary indicator of whether the prediction that class c
is the class of observed data ois correct, and finally pis the
probability that aforementioned data obelongs to the class
label c.
Learning by backpropagation is performed through a gra-
dient descent optimisation algorithm that drives the updating
of the weights within the neural network. In this study,
the Adaptive Moment Estimation (ADAM) algorithm is ap-
plied [22]. Inspired by RMSProp [23] and Momentum [24],
the main steps of ADAM are the following:
1) The exponentially weighted average of past gradients,
vdW is calculated.
2) The exponentially weighted averages of the squares of
past gradient, sdW is calculated.
3) The bias towards zero in the previous are corrected,
deriving vcorrected
dW and scorrected
dW respectively.
4) The network parameters are updated through the fol-
lowing process:
2VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
vdW =β1vdW + (1 β1)J
∂W
sdW =β2sdW + (1 β2)J
∂W 2
vcorrected
dW =vdW
1(β1)t
scorrected
dW =sdW
1(β2)t
W=Wαvcorrected
dW
qscorrected
dW +ε,
(2)
in which β1and β1are tunable hyperparameters, J
∂W is a
cost gradient of the current network layer to be tuned, Wis a
matrix of weights, αis the learning rate, and εis a small value
introduced to prevent the possibility of division by zero.
B. CONVOLUTIONAL NEURAL NETWORKS
A Convolutional Neural Network (CNN) is a Deep Learning
algorithm capable of collecting an input matrix and ascribing
weights and bias in parallel under the constraints of a pre-
dictive problem [25], [26], resulting in specific features. A
Convolutional layer performs a dot product between two ma-
trices, where one matrix is the set of learnable parameters and
the other one is known as a kernel, producing an Activation
Map, as shown below:
G[m, n]=(fh)[m, n] = X
jX
k
h[j, k]f[mj, n k],
(3)
where the input matrix is fand the kernel is denoted as h.
It has been previously shown that CNNs succeed with
Biological Signal interpretation. Using EEG-based mental
state, Tripathi and Acharya [27] performed two different clas-
sifications using both Deep (DNN) and Convolutional (CNN)
Neural Networks, where DNN achieved 75.78% accuracy
for valence using the 4040 values as input vector, and CNN
achieved 81.41% accuracy in two dimensions, respectively.
Following the Convolutional layers, interpretation layers are
used to learn from convolutional inputs through the same
process as MLP (see Subsection II-A). Additionally, filters
are employed to extract statistics from the input matrix and a
single interpretation layer concludes the classification into K
possible labels.
C. EEG AND EMG
In this subsection, the necessary scientific background
on Electroencephalography (EEG) and Electromyography
(EMG) is provided.
EEG is the process of measuring and recording electrical
signals produced by the brain through electrodes placed ei-
ther upon or within the cranium, that is, the nervous impulses
produced by the neurological structure of the brain [28,
p. 31]. Since the activities of a person can largely be summed
up by brain activity [29], to classify an EEG pattern would
provide useful information, and is thus used as a point of
input and control for a Brain Computer Interface (BCI).
BCIs have been successfully operated in a variety of
clinical situations such as evaluation of seizures, epilepsy,
confused states and coma [30]. Classification of the brain
activity preceding a stroke showed that a stroke could be
predicted before actual occurrence based on abnormalities
in brain activity [31], through a Random Forest algorithm
applied for statistical classification of extracted feature sets.
Successful rehabilitation of motor functions within stroke
patients is aided in a process of measurement and classi-
fication of brain activity along with robotic feedback [32].
Classification of brain activity has also been successful,
similarly to stroke, in the preemptive detection of epileptic
seizures before they occur [33], [34].
EMG is a measure of the electrical potential difference
between two points whose origin are individual or groups
of muscle fibres [35]. Similarly to EEG, the activity of the
muscle can largely be summed up by electrical impulses
produced, and can thus form a point of control in a Muscle-
Computer Interface (muCI) [36]. Similarly to the Muse head-
band operated in many EEG studies, due to its consumer-
friendliness and future potential based on its low-cost yet
high-performing nature, the Myo armband is a prominent
device used in muCI systems, frameworks, and applications.
For example, researchers collaborating from multiple fields
found that accurate gesture classification could lead to a
new standard for New Interfaces for Musical Expression
(NIME) [37].
In the Human-Machine Interaction community, Myo has
been succesfully employed in Sign Language Recognition,
with the classification of 20 Brazilian Sign Language letters
with 96-99% accuracy [38]. Following Myo’s proprietary
system of classification boasting around 83% accuracy, re-
searchers found that through the application of K-Nearest
Neighbour (KNN) and Dynamic Time Warping (DTW)
algorithms, the classification of five Myo gestures could be
improved to around 86% (+3%) [39].
D. TRANSFER LEARNING
Transfer Learning, as the name suggests, consists in trans-
ferring something learnt in one problem or task to another.
Oftentimes, Transfer Learning is the application of a model
trained on source data to unseen data of the same domain
called target data [40]. The model trained on the source data
can be further trained on the target data, before its deploy-
ment on the target data. Cross-domain transfer learning is a
similar application of a pre-trained model from one domain
to another domain of different nature; for example, in this
study, models trained on two different datasets of biological
waves from the brain and forearm muscles are applied to one
another’s data for further training.
Transfer of knowledge is considered successful from one
domain to the other when the starting point, the learning
curves and asymptotes are higher than those of the traditional
VOLUME 4, 2016 3
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
Accuracy
Time
FIGURE 1: Example of a successful Transfer Learning ex-
periment. Transfer Learning (top line) has a higher starting
point, steeper curve, and higher asymptote in comparison to
learning via random weight distribution (bottom line).
source-train source-classify approach [41]. A visual repre-
sentation of a successful Transfer Learning experiment can
be seen in Figure 1, where the starting point is higher for
transfer learning compared to random distribution, and sub-
sequently the learning curve is also steeper and the asymptote
is higher.
Generally, there are two main reasons for the application
of Transfer Learning [40]. Firstly, pre-trained models and
computational resources have become easily accessible [42],
there are countless available models trained over many
hours on extremely powerful hardware. Examples include
VGG [2], Inception [43], and MobileNet [44]. Secondly, the
lack of a large enough dataset for learning is often negated by
transferring previous knowledge to the domain at hand [40].
Pan and Yang [45] define three main types of Transfer
Learning as follows:
1) Inductive Transfer Learning is knowledge transfer
when the source and target domains are identical but
a new task is to be learned. For example, if five EMG
gestures are classified and further learning enables
the model to learn to recognise additional gestures,
based on the current knowledge, then inductive transfer
learning takes place.
2) Unsupervised Transfer Learning is the transfer of
knowledge between two differing domains and like-
wise differing tasks. In this study, unsupervised trans-
fer learning finds success in sharing knowledge be-
tween EEG and EMG domains through the mental state
and gesture recognition tasks.
3) Transductive Transfer Learning is the process of shar-
ing knowledge between differing domains but for the
same task. For example, if an EEG headband is to
be calibrated to a subject’s data (a slightly different
domain) to complete the same mental state recognition
task, then transductive transfer learning takes place.
Recently, many Transfer learning techniques have been ap-
plied successfully in real-world problems, for example, can-
cer subtype discovery [46], building-space optimisation [47],
[48], text-mining [49], [50], and reinforcement-learning for
videogame AI [51], [52].
Cross-domain transfer learning has been given relatively
little attention in the field of biological signal processing,
with research almost exclusively opting for same-domain
personalisation, or calibration. EEG and EMG signals are ex-
cellent candidates for cross-domain transfer learning, given
their similarities, yet this idea has not been investigated. In
this study we aim to fill this gap and establish cross-domain
transfer learning between EEG and EMG domains.
It has been shown that models do not generalise well
between subjects, thus there is a need for transfer learning
to achieve accurate classification results [53], [54].
A highly promising proposal [55] consists of a two-step
ensemble of filter-bank classification of EEG data via two
models, one for the original dataset, and another for a small
dataset collected from a new subject. The baseline classifi-
cation ability for nine individual subjects improves by ap-
proximately 10%. Similarly, the kernel principal component
analysis (kernel PCA) approach in leads to an improvement
from 58.95% to 79.83% (+20.88) classification accuracy
when transfer learning from the original dataset is performed
for a new subject [56].
Similarly to EEG, transfer learning in EMG is most of-
ten concerned with cross-subject learning rather than cross-
domain application [57]. Researchers gathered and combined
datasets of EMG data measured from a total of 36 subjects
via the Myo armband (as also used in this study). The dataset
was split into sets of 19 and 17 subjects. Transfer learning
of learnt features of a Convolutional Neural Network led to
a classification accuracy improvement of 3.5%. Though this
improvement is small, the achieved accuracy is the state-of-
the-art for the dataset.
Transfer Learning in EMG has been successful in calibrat-
ing to electrode shift, change of posture, and disturbances due
to sweat and fatigue [58] through small calibration recordings
that subsequently require fewer than 60 seconds of training
time. Prahm et al. found that the exercises increased in
accuracy after disturbance from 74.6% to 97.1%.
Motivated by the small successes of cross-subject transfer
learning within EEG and EMG domains independently, as
well as the similar nature and behaviour of these biological
signals, we propose to explore the potential of applying learnt
knowledge from one biological signal domain to the other
and vice versa.
III. EXPERIMENTAL SETUP
In this section,we describe the data acquisition, feature
extraction, topology selection, learning, and transfer learning
4VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
(a) Muse EEG Headband
(InteraXon)
(b) MYO EMG Armband
(Thalmic Labs)
FIGURE 2: The devices used to acquire data for this experi-
ment.
experiments.
In terms of hardware, all models are trained on Tensor-
Flow via an NVidia GTX980Ti Graphics Processing Unit.
For topology selection, our previously proposed DEvo [59]
algorithm is executed for 15 generations. Hard limits of a
maximum of 5 hidden layers and 512 neurons were set. Evo-
lutionary topology optimisation allowed for 100 epochs of
training and transfer learning was observed with 30 epochs of
training. These values were chosen based on the observation
that in preliminary experiments there was little or no further
improvement after these numbers of generations and epochs,
respectively. Training validation is enabled through 10-fold
cross validation, where the ten folds are shuffled. Other
hyperparameters that were chosen were the ReLu activation
function for the hidden layers, and the ADAM optimisation
algorithm [22] for tuning of weights during training.
A. DATA ACQUISITION
Two datasets are used in this experiment, EEG and EMG.
Here we describe the acquisition of the EEG and EMG
datasets. The devices used to acquire datasets can be seen
in Figure 2.
The EEG dataset was collected in a previous study [60].
This dataset was obtained from four subjects, two male and
two female. The subjects performed three tasks, while the
sensors were recording the data. The tasks involved three
different states of brain activity: concentration, relaxation
and neutral. The EEG data were acquired using the Muse
Headband, which is a commercial EEG device with four dry
electrodes (TP9, AF7, AF8 and TP10). The EMG dataset
was gathered via the Myo armband, a commercial elec-
tromyograph monitoring device with 8 dry electrodes for the
measurement of electrical current produced by the skeletal
muscles within the arm. The Myo armband is capable of
recording EMG data at sample rate of 200Hz. In addition,
it also has a nine axis Inertial Measurement Unit (IMU)
operating at a sample rate of 50Hz. For this project, only
the EMG data are used and so the inertia of the arm is not
considered.
Ten subjects contributed to the EMG dataset, six male
and four female all aged 22-40. The subjects performed
four different gestures for 60 seconds each, and the sensors
FIGURE 3: Data collection from a male subject via the Myo
armband on their left arm.
recorded EMG data produced by the muscles in the forearm.
The gestures performed were, clenching and relaxation of the
fist, spreading and relaxation of fingers, swiping right, and
swiping left. The observations were performed twice, once
for the right arm and once for the left. Figure 3 shows the
experimental setup of a subject wearing the Myo armband.
B. FEATURE EXTRACTION
Biological signal processing encounters issues in classifi-
cation due to their non-stationary, nonlinear and random
nature. Since the wave class is observed over time, temporal
observations must be made rather than single points. For
this reason, short segments of the wave are considered and
a statistical feature matrix is generated based on observation
of the temporal segment. Previous studies identified a set of
statistics to be extracted from EEG brainwaves [59], [60].
In this study, a set of statistics was extracted from both the
EMG and EEG signals of a temporal nature. Initially, the
sampling rate was reduced to a uniform 200Hz based on fast
Fourier transformations along a given axis due to the variable
nature of a Bluetooth Low Energy (BLE) connection. The
signal was assumed to be periodic since a Fourier method
is used. This led to a realistic down-sampling since the
dominant energy was concentrated in the range of 20-500Hz
as observed in [60], even though the frequency range of
the EEG sensor was superior to this. The Muse headband
performed Notch filtering at 50Hz since the experiment was
performed in the United Kingdom.
Both EEG and EMG signals are non-stationary and ran-
dom waves and it is for this reason that feature extraction
must be performed when a non-temporal learning process is
followed. Feature extraction was performed by introducing
sliding windows of length of 1 second at an overlap of 0.5
seconds to segment the data. Algorithm 1 describes the hand-
crafted features extracted in this work. Considering how we
split the 1 second window, the two 0.5 second half-windows
produced due to the differences between two 1 second win-
dows, we compute: (i) the change in both the sample means
VOLUME 4, 2016 5
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
and in the sample standard deviations between the first and
second half-window; (ii) the change in both the maximum
and minimum values between the first and second half-
windows. Then, considering the two 0.25 second quarter-
windows produced due to further differences between com-
puted windows, we then compute (i) the sample mean of each
quarter-window; (ii) all paired differences of sample means
between the quarter-windows; (iii) the maximum (minimum)
values of each quarter-window, plus all paired differences of
maximum (minimum) values between the quarter-windows.
A Discrete Fourier Transform is performed on the time
windows and all resultant values are treated as attributes.
Result: Features extracted from raw data for every wt
User defined the size of the sliding window wt= 1s;
Input: sequence of raw data (EEG or EMG);
Initialisation of variables init = 1,wt= 0;
while getting sequence of raw data from sensor (>1min)do
if init then
prev_lag = 0;
post_lag = 1
end
init = 0;
;
for each slide window (wtprev_lag) to (wt+post_lag )
do
Compute the mean of all wtvalues y1, y2, y3...yn;
¯yk=1
NPN
i=1 yki ;
;
Compute the asymmetry and waves peakedness
represented by 3rd and 4th order moments skewness and
kurtosis g1,k =PN
i=1(yki ¯yk)3
Ns3
k
and
g2,k =PN
i=1(yki ¯yk)4
Ns4
k
3;
;
Compute the maximum and minimum value of each
signal wt
max =max(wt)and wt
min =min(wt);
;
Compute the sample variances K×Kmatrix Sof each
signal, plus the sample covariances of all signal pairs
sk` =1
N1PN
i=1 (yki ¯yk) (y`i ¯y`) ;
k, ` [1, K];
;
Compute the Eigenvalues of the covariance matrix S,
which are the λsolutions to: det (SλIK)=0, where
IKis the K×Kidentity matrix, and det(·)is the
determinant of a matrix;
;
Compute the upper triangular elements of the matrix
logarithm of the covariance matrix S, where the matrix
exponential for Sis defined via Taylor expansion
eB=IK+P
n=1
Sn
n!, then BCK×Kis a matrix
logarithm of S;
;
Compute the magnitude of the frequency components of
each signal, obtained using a Fast Fourier Transform
(FFT), magFFT(wt);
;
Get the frequency values of the ten most energetic
components of the FFT, for each signal, getFFT(wt, 10);
end
wt=wt+ 1s;
prev_lag = 0.5s;post_lag = 1.5s;
Output Features F wtextracted within the current wt
end
Algorithm 1: Feature extraction algorithm for a se-
quence of data (EEG or EMG signals).
A change in attribute values is also considered via a
lag window, in which each window is passed the previous
extracted value vector from the preceding window. The
first window does not receive this vector since no window
preceded it. Due to the redundancies of the maximum, mean,
and minimum values of quarter-windows due to the overlaps,
these values are not considered within lag window attribute
vectors.
The features extend beyond the usual values read by clin-
icians due to the lower quality of the Muse headband as
opposed to what can be expected from clinical EEG, with
dimensionality reduction often then used in order to select a
set of useful features from a large dataset in which weaker
features may exist. In this work, identification of weak fea-
tures has to be left to the deep learning algorithms since
identical inputs are needed for the networks in order to enable
the possibility of transfer learning (preventing a mismatch in
topology, or neuron representation etc.). This is a limitation
discussed in Section VII, and future experiments are outlined
in order to produce a synchronised dimensionality reduction
algorithm between the two datasets concurrently in the form
of a combinatorial optimisation problem. Dimensionality
reduction was not performed here since it was not needed for
the exploration of transfer learning between biological signal
domains.
A vector representation of the wave behaviour is formed
through the above process and 988 numerical features are
created. Since the same process is followed for the same
number of electrodes, the datasets describing EMG and EEG
have the same dimensionality and thus transfer learning is
possible with these features as inputs to an ANN.
IV. METHOD I: MLP TRANSFER LEARNING
A. DERIVATION OF BEST MLP TOPOLOGY
Although many studies focus on grid search of topologies
[61]–[63], this study applies a multi-objective evolutionary
algorithm in order to select the best neural network architec-
ture for both classification problems. We apply an evolution-
ary algorithm instead of a classical grid search for two main
reasons [64]–[66]:
1) Evolutionary search allows for exploration within
promising areas of the problem space at a finer level.
Previous experiments, such as speech recognition [67],
found complex best solutions for the problem, e.g. a
combination of three deep layers of 599, 1197, and 436
neurons. Including such multiples within a grid search
would increase computational complexity of the search
beyond realistic possibility.
2) With multi-objective optimisation through mean ac-
curacy via equal scalarisation (see equation 4), the
algorithm was able to search for a best solution for
both of the problems rather than having to be executed
twice, followed by statistical analyses to calculate a
best topology.
6VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
Deep Evolutionary Optimisation (DEvo) is an evolution-
ary algorithm, inspired by Darwinian evolution in Nature,
used to search the problem space of neural network topology
in order to select a best-performing structure of hidden
layers and neurons i.e. a selection of network hyperpa-
rameters. DEvo has been successfully used in benchmark
problems [68] as well as in phoneme classification [67] and
EEG classification [59].
Result: Output the best neural network configurations discovered
per generation
User defined population size p, maximum number of hidden layers
h, and maximum number of neurons n
Initialise prandom solutions with hidden layers in the range [1..h],
each hidden layer is assigned [1..n] neurons ;
Train all neural networks through forward-pass and
backpropagation;
Calculate fitness Fof all solutions;
Sort solutions by F;
while Termination not met do
for For each solution parent1 do
Select a random solution parent2;
Select network depth drandomly from parent1,parent2 or
mutate
while d not met do
Randomly select layer from parent1,parent2 or
mutate
if NOT mutate then
if Selected layer exists within selected parent
then
Amend layer to solution offspring
else
if Selected layer exists within other parent
then
Amend layer to solution offspring
else
Assign random value;
\\since none exist
end
end
end
end
end
Calculate fitness Fof all solutions;
Sort solutions by F;
Keep the top psolutions from all parents and offspring;
Amend best solution to best solutions b;
end
Output all best solutions bof each generation;
Algorithm 2: Generalised process of the evolutionary
search of Neural Network topology
As illustrated by the flow diagram in Figure 4, the general
idea of single-objective DEvo is as follows:
1) Create an initial population of random solutions.
2) Simulate the following until a user-defined termina-
tion:
a) Select parent networks for the current generation
of the evolutionary cycle.
b) Optionally, alter the depth or width of the network
via random mutation, to prevent premature con-
vergence to local minima within the population.
c) Train the neural networks by performing forward-
pass and backpropagation of errors through a
given optimisation function for a user-defined
number of epochs, and calculate the fitness.
A more detailed process of the evolutionary search can be
observed in Algorithm 2. In particular, the re-combinations
of two parent networks is shown in detail.
Since the search must derive a ’best of both worlds’
solution for both the EMG and EEG problems, a new fitness
function is introduced to score a solution:
F(s)=0.5A(EMG)
100 + 0.5A(EEG)
100 ,(4)
where A(EM G)and A(E EG)are the mean accuracy scores
of the networks when trained with EMG and EEG data
respectively through shuffled 10-fold cross validation. Equal
weights are allocated to the two components as EEG and
EMG training are equally important. Only hidden layers are
to be optimised, therefore the input and output layers of the
network are simply hard-coded.
B. BENCHMARKING OF TRANSFER LEARNING
For transfer learning, the following process is followed:
1) A neural network with randomly distributed weights is
trained to classify the EMG dataset.
2) A neural network with randomly distributed weights is
trained to classify the EEG dataset.
3) The best weights from the EMG network are applied to
a third neural network, which is then trained to classify
the EEG dataset.
4) Mirroring step 3, the best weights from the EEG net-
work (step 2) are initialised to a fourth neural network,
which is then trained to classify the EMG dataset.
The four networks are then compared. EEG to EMG-EEG
and EMG to EEG-EMG in order to discern whether knowl-
edge has been transferred. If higher starts, curves, and asymp-
totes are observed, then knowledge is considered succesfully
transferred between the two domains.
V. METHOD II: CNN TRANSFER LEARNING
A. REPRESENTING BIOLOGICAL WAVES AS IMAGES
In order to generate a square matrix, after the feature extrac-
tion process, the final 28 attributes are removed from each
dataset. This is done because 961 is the closest square number
within the attribute set (31x31) and the final attributes are
chosen in order to retain identical inputs to the networks for
both datasets. After normalisation of all attributes between
the values of 0 and 255, they are then projected as 31px
square images. Examples of waves projected into visual
space can be observed in Figures 5 and 6. Though padding
would be applied in the situation where a square reshape is
not possible (if square input is considered), this is not needed
in this experiment since 961 attributes are selected (31x31
reshape).
VOLUME 4, 2016 7
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
START
Generateprandom
neuralnetwork
architectures
Trainallnetworks
throughforward-pass
andbackpropagation
CalculateFitnessFiof
eachneuralnetworki
Sortneuralnetworksin
descendingorderofFi
Selectparentnetworks
forthegeneration
Termination?
Foreachparent1,
chooseparent2and
produceoffspring
Applyrandommutation
Present[b1..bn]best
solutionsobservedat
eachgeneration
END
NO
YES
FIGURE 4: Flow Diagram of the evolutionary search of Neural Network topology. Population size is given as pand fitness
calculation is given as F. Set {b1..bn}denotes the best solution presented at each generation
FIGURE 5: 30 Samples of EEG as 31x31 Images. Top row shows relaxed, middle row shows neutral, and bottom row shows
concentrating.
FIGURE 6: 40 Samples of EMG as 31x31 Images. Top row shows close fist, second row shows open fingers, third row shows
wave in and bottom row shows wave out.
8VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
TABLE 1: Network topology and parameters found to be best
in a previous work [69]
Layer Output Parameters
Conv2d (ReLu) (0, 14, 14, 32) 320
Conv2d (ReLu) (0, 12, 12, 64) 18,496
Max Pooling (0, 6, 6, 64) 0
Dropout (0.25) (0, 6, 6, 64) 0
Flatten (0, 2304) 0
Dense (ReLu) (0, 512) 1,180,160
Dropout (0.5) (0, 512) 0
Dense (Softmax) (0, 3) 1,539
B. DERIVATION OF HYPERPARAMETERS
Due to the computational costs of hyperparameter optimisa-
tion used in the MLP experiment when a CNN is consid-
ered, CNN topology is instead inspired by Ashford et al.’s
work [69]. In this work, the topology observed in Table 1
was found to be best after empirical exploration. Dropout is
introduced in order to prevent overfitting after two convolu-
tional operations (and subsequent pooling) and again after an
interpretation layer of 512 densely connected neurons.
C. BENCHMARKING OF TRANSFER LEARNING
The benchmark of the CNN transfer learning follows the
same process as detailed in Section IV-B, except the weight
transfer applies to input, convolutional, and hidden interpre-
tation layers.
The hypothesis of this experiment ie. that transfer learning
has occurred cross-domain, not simply through deep learn-
ing, is tested by comparison to a popular pre-trained model.
For this purpose, the ResNet50 architecture and weights [70]
are used when trained on the ImageNet dataset. This archi-
tecture is chosen based on its aptitude for smaller images as
opposed to the previously mentioned VGG16 model, more
fitting to the nature of the images generated by the algorithm.
The experiments are given unlimited time to train in order
to explore this, with an early stop executing after 10 epochs
with no observed improvement of validation accuracy. Other
model hyperparameters are identical to their transfer learning
counterparts.
VI. EXPERIMENTAL RESULTS
In this section, the results from the two experiments are dis-
cussed. Firstly, an MLP network topology is derived through
the previously described DEvo method before transfer learn-
ing capabilities are benchmarked. Initially, the models are
trained starting from random weight distribution (baseline) in
order to provide the baseline. Secondly, the model trained on
EMG dataset is used to transfer knowledge to a model train-
ing to classify the EEG dataset and then vice-versa. These are
then compared to their baseline non-transfer learning coun-
terparts. This is carried out a second time with Convolutional
Neural Networks (without evolutionary search) where signals
have been projected as raster images.
The MLP experiments are presented and discussed in Sub-
section VI-A and the CNN experiments are then presented
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0
0.2
0.4
0.6
0.8
1
Generation
Best Fitness Score
FIGURE 7: Highest (best) fitness observed per generation of
the combined and normalised fitnesses of EEG and EMG data
classification. The two fitness components are considered
equally weighted to produce the same topology in order to
allow direct transfer of weights.
TABLE 2: Comparison of the MLP Training Processes of
EMG and EEG with random weight distribution compared
to weight transfer learning between EMG and EEG
Experiment Training Accuracy (%)
Epoch 0 Final Epoch Best Epoch
EMG 62.84 83.57 84.76
EEG 54.7 62.1 62.73
Transfer Learning
(EEG to EMG) 65.22 (+2.38) 85 85.12 (+0.36)
Transfer Learning
(EMG to EEG) 84.95 (+30.25) 93.28 93.82 (+29.95)
and discussed in Subsection VI-B.
A. EXPERIMENT 1: MLP TRANSFER LEARNING
1) Hyperparameter Selection for Initial Random Distribution
Learning
Figure 7 shows the fitness evolution (Equation 4) of neural
network topologies for the two datasets, where each point
is the combined mean fitness for EEG and EMG and the
best topology. The best result was found to be a network of
5 hidden layers, with neuron counts 206,226,298,167,363
respectively at a combined fitness of 0.74. This network
topology is thus taken forward in the experiments towards
transfer learning capability between the networks of EEG to
EMG and vice-versa.
2) MLP Transfer Results
Finally, the transfer learning experiment is executed fol-
lowing the process described in section IV-B. Figure 8 and
Table 2 detail the learning processes of both EMG and
EEG as well as the transfer learning experiments, from one
VOLUME 4, 2016 9
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
FIGURE 8: Test and Training Accuracies of EMG, EEG, and transfer between EMG and EEG. ’EEG Transfer’ denotes EMG
to EEG and likewise for ’EMG Transfer’.
domain to the other and vice-versa. Transfer learning was
most successful when EMG data was used to fine tune the
EEG problem, with an increase of best classification accuracy
from 62.37% to 93.82% (+29.95). A very slight increase
was also observed in reverse, when EEG network weights
were used as the initial distribution for the EEG problem,
with best accuracy rising from 84.76% to 85.12% (+0.36). In
terms of starting accuracy, that is, accuracy of classification
with no training at all, a success of knowledge transfer also
occured; EEG classification increased from 54.7% to 84.95%
(+30.25), and thus even prior to any training the network
outperformed the network initially trained on EEG data.
Likewise, the EMG classification prior to any training at
epoch 0 increased from 62.84% to 65.22% (+2.38). It was
observed that learning had ceased prior to epoch 30 being
reached.
The epoch zero results are particularly interesting since
transfer learning has occurred between two completely dif-
ferent domains, from EMG gesture classification to EEG
mental state recognition. This shows that knowledge transfer
TABLE 3: Comparison of the CNN Training Processes of
EMG and EEG with random weight distribution compared to
weight transfer learning between EMG and EEG
Experiment Training Accuracy (%)
Epoch 0 Final Epoch Best Epoch
EMG 52.4 88 88.55
EEG 72.5 95.3 96.24
Transfer Learning
(EMG to EEG) 82.39 (+9.89) 96.4 97.18 (+0.94)
Transfer Learning
(EEG to EMG) 58.18 (+5.78) 84.24 85.18 (-3.37)
is possible even without training being required.
B. EXPERIMENT 2: CNN TRANSFER LEARNING
Figure 9 shows the learning processes for the four networks.
It was observed that learning was still occurring at epoch
30 (unlike in the MLPs in Experiment 1), and due to this,
learning time was increased to 100 epochs. Table 3 shows
the outcome of the experiments. Some transfer learning suc-
cesses were achieved, with higher starts in TL experiments,
10 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
FIGURE 9: Test and Training Accuracies of EMG, EEG, and transfer between EMG and EEG with a Convolutional Neural
Network, over 100 epochs. As with the previous figure, ’EEG Transfer’ denotes EMG to EEG and likewise for ’EMG Transfer’.
TABLE 4: Best CNN accuracy observed for ResNet50, Base-
line (Non-Transfer) Learning, and Transfer Learning
Best CNN Accuracy Observed (%)
ResNet50 Baseline Transfer Learning
EEG EMG EEG EMG EEG EMG
92.34 74.92 96.24 88.55 97.18 85.18
of +9.89% and +5.78% for EEG and EMG, respectively. The
best classification accuracy of EEG was improved by 0.94%
whereas this was not the case for EMG, which actually de-
creased by 3.37%. Thus, the CNN transfer learning approach
is only successful in the case of EMG to EEG but not vice-
versa.
It is important to note that previously, the One Rule Ran-
dom Forest approach [60] gained 87.16% accuracy and the
image representation and CNN approach [69] gained 89.38%
accuracy on EEG data. Our network is competitive at 82.39%
accuracy on the same dataset with no training whatsoever,
using simply the weights from the EMG network. Similarly,
it is also important to note that the final accuracy of 97.18%
substantially outperforms these previous approaches.
1) Comparison to ResNet50
For comparison of transfer quality, the ResNet 50 CNN ar-
chitecture is used. Table 4 shows that the ResNet50 achieves
weaker results for both problems. The ResNet50 architecture
was observed to stop improving after 35 and 39 epochs for
EEG and EMG respectively, similarly to the behaviour of our
architecture shown in Figure 9.
VII. FUTURE WORK AND CONCLUSION
This study demonstrated that cross-domain transfer learning
is possible between the domains of electroencephalography
and electromyography, between the electrical signals pro-
duced by the frontal lobes of the brain and forearm mus-
cles.To the best of our knowledge, cross-domain transfer
learning with EMG to EEG and vice-versa has not been ex-
plored elsewhere. Firstly, models were trained via a train/test
split and with some hard-coded hyperparameters such as acti-
vation function and gradient descent optimisation algorithm;
future work should explore the effects and implications of
differing hyperparameter sets.
Limited selection of network topologies was performed
through a single multi-objective evolutionary search. With
the possibility of a local minimum being encountered and
stagnation occurring, further executions of the search should
be performed in a subsequent study in order to explore the
problem space and thus reduce the chance of stagnation.
VOLUME 4, 2016 11
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
Scalarisation was considered equal between the two datasets,
though the EMG dataset was more diverse and much larger
than the EEG dataset and thus alternative scalars with prefer-
ence to either dataset should also be benchmarked. Future
work could also involve the possibilities of cross-domain
transfer learning in multiple biological signal domains, such
as including other areas of the muscular system and brain,
and additionally, other domains such as electrocardiography.
The potential for transfer learning between these domains
should be applied in Human-machine interaction in the
future, since the application of a framework as described
here shows not only the advantage of improved accuracy
of classification, but additionally, the derivation of a less
computationally expensive process compared to learning
from scratch.
In this study, dimensionality reduction or feature selection
was not performed. Although previous studies have shown
the effectiveness of removing certain features in favour of
others (both for increased classification ability and reduction
of complexity), it was important that the neural networks
were identical in structure to one another. In doing this,
transfer of weights between networks was directly compa-
rable to the original network to be transferred. In future, a
strategy of synchronised feature selection could be explored
in order to match network inputs. This could be done in a
relatively simple manner, by ranking a defined number of
attributes (by Information Gain, Symmetrical Uncertainty,
etc.) and choosing those which both algorithms would choose
individually. In a far more complex experiment, the se-
lection of attributes could be presented as a combinatorial
optimisation problem. Analogously to the famous knapsack
problem, transfer learning could provide a fitness function
to the combination of selected attributes similar to that in
Equation 4. A set of attributes useful to both problems could
then be derived to present a synchronised set of inputs for a
final transfer learning benchmark. These suggestions could
be benchmarked in an ’optimisation extension’ to this study.
Additionally, dimensional-reshape techniques other than the
traditional 2D image reshape considered in this study could
also be performed.
The final comparison experiment was designed specifi-
cally to assess whether the cross-domain transfer of knowl-
edge has occurred as opposed to simply CNN transfer
learning. Success in cross-domain transfer of knowledge
was shown compared to the ResNet50 model. In the future,
further CNN architectures could be explored. For example,
InceptionV3 and VGG16 models outperform the ResNet50
model in the state-of-the-art, but their minimum input di-
mensions are above those of the data in this experiment. With
larger feature selection bounds for the CNN input, in order to
generate larger images (which are also to be benchmarked
in a related experiment prior to discern the best feature
extraction processes), other models such as these could be
implemented and provide further evidence for cross-domain
transfer of knowledge.
To conclude, we argue that, through initial weight distribu-
tion, cross-domain transfer learning between two biological
signal domains is possible and, in some cases, to a great pos-
itive effect. Identical mathematical features were extracted
from the waves to provide a stationary description fit for
classification, and transfer between features was also noted.
Initial abilities pre-training were higher than random weight
distribution, the learning curves and final classification abili-
ties for both domains were also better, indicating that useful
knowledge had been shared between both domains during the
transfer learning process. The exploration of the possibility
of transfer of knowledge from/to other biological signal
domains such as ECG is also an exciting topic for future
study.
ACKNOWLEDGEMENT
This work was partially supported by “UK ACADEMIES -
RESEARCH MOBILITY” PI 02/2018 in Partnership with
CONFAP-Brazil and Fundação Araucária Pr-Brazil (protocol
50715.538.33202.06072018), through the project: "Stepping-
stones to Transhumanism: Merging EMG and EEG signals to
control a low-cost robotic hand" awarded to Diego R. Faria
(Aston University, UK) and Eduardo P. Ribeiro (Federal
University of Parana, Brazil).
REFERENCES
[1] S. Shi, Q. Wang, P. Xu, and X. Chu, “Benchmarking state-of-the-art deep
learning software tools,” in 2016 7th International Conference on Cloud
Computing and Big Data (CCBD), pp. 99–104, IEEE, 2016.
[2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[3] H. Qassim, A. Verma, and D. Feinzimer, “Compressed residual-vgg16 cnn
model for big data places image recognition,” in 2018 IEEE 8th Annual
Computing and Communication Workshop and Conference (CCWC),
pp. 169–175, IEEE, 2018.
[4] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time
style transfer and super-resolution,” in European conference on computer
vision, pp. 694–711, Springer, 2016.
[5] M. A. Oskoei and H. Hu, “Myoelectric control systems—a survey,
Biomedical signal processing and control, vol. 2, no. 4, pp. 275–294, 2007.
[6] D. P. Subha, P. K. Joseph, R. Acharya, and C. M. Lim, “EEG signal
analysis: a survey,” Journal of medical systems, vol. 34, no. 2, pp. 195–
212, 2010.
[7] K. Tatarian, M. S. Couceiro, E. P. Ribeiro, and D. R. Faria, “Stepping-
stones to transhumanism: An EMG-controlled low-cost prosthetic hand for
academia,” in 2018 International Conference on Intelligent Systems (IS),
pp. 807–812, IEEE, 2018.
[8] C. Kast, B. Rosenauer, H. Meissner, W. Aramphianlert, M. Krenn,
C. Hofer, O. C. Aszmann, and W. Mayr, “Development of a modular
bionic prototype arm prosthesis integrating a closed-loop control system,”
in World Congress on Medical Physics and Biomedical Engineering 2018,
pp. 751–753, Springer, 2019.
[9] J. Edwards, “Prosthetics’ signal processing connection: Sophisticated
prosthetic controls allow amputees to engage more fully in everyday
life [special reports],” IEEE Signal Processing Magazine, vol. 36, no. 4,
pp. 10–172, 2019.
[10] F. Michaud, P. Boissy, D. Labonte, H. Corriveau, A. Grant, M. Lauria,
R. Cloutier, M.-A. Roux, D. Iannuzzi, and M.-P. Royer, “Telepresence
robot for home care assistance.,” in AAAI spring symposium: multidisci-
plinary collaboration for socially assistive robotics, pp. 50–55, California,
USA, 2007.
[11] J. Broekens, M. Heerink, H. Rosendal, et al., “Assistive social robots in
elderly care: a review,” Gerontechnology, vol. 8, no. 2, pp. 94–103, 2009.
12 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
[12] J. Scholtz, M. Theofanos, and B. Antonishek, “Development of a test bed
for evaluating human-robot performance for explosive ordnance disposal
robots,” in Proceedings of the 1st ACM SIGCHI/SIGART conference on
Human-robot interaction, pp. 10–17, ACM, 2006.
[13] E. Welburn, T. Wright, C. Marsh, S. Lim, A. Gupta, B. Crowther, and
S. Watson, “A mixed reality approach to robotic inspection of remote
environments,
[14] V. F. Annese, M. Crepaldi, D. Demarchi, and D. De Venuto, “A digital
processor architecture for combined EEG/EMG falling risk prediction,”
in Proceedings of the 2016 Conference on Design, Automation & Test in
Europe, pp. 714–719, EDA Consortium, 2016.
[15] A. Heydari, A. V. Nargol, A. P. Jones, A. R. Humphrey, and C. G.
Greenough, “EMG analysis of lumbar paraspinal muscles as a predictor
of the risk of low-back pain,” European Spine Journal, vol. 19, no. 7,
pp. 1145–1152, 2010.
[16] D. De Venuto, V. Annese, M. de Tommaso, E. Vecchio, and A. S.
Vincentelli, “Combining EEG and EMG signals in a wireless system for
preventing fall in neurodegenerative diseases,” in Ambient assisted living,
pp. 317–327, Springer, 2015.
[17] P. Werbos, “Beyond regression:" new tools for prediction and analysis in
the behavioral sciences,” Ph. D. dissertation, Harvard University, 1974.
[18] S. Ruder, “An overview of gradient descent optimization algorithms,”
arXiv preprint arXiv:1609.04747, 2016.
[19] S. Haykin, Neural networks: a comprehensive foundation. Prentice Hall
PTR, 1994.
[20] K. P. Murphy, Machine learning: a probabilistic perspective. MIT press,
2012.
[21] S. Kullback and R. A. Leibler, “On information and sufficiency,” The
annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951.
[22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[23] T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop, coursera: Neural net-
works for machine learning,” University of Toronto, Technical Report,
2012.
[24] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of
initialization and momentum in deep learning,” in International conference
on machine learning, pp. 1139–1147, 2013.
[25] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural informa-
tion processing systems, pp. 1097–1105, 2012.
[26] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural
networks, vol. 61, pp. 85–117, 2015.
[27] S. Tripathi, S. Acharya, R. D. Sharma, S. Mittal, and S. Bhattacharya,
“Using deep and convolutional neural networks for accurate emotion
classification on deap dataset.,” in Twenty-Ninth IAAI Conference, 2017.
[28] D. Purves, G. Augustine, D. Fitzpatrick, W. Hall, A. LaMantia, J. McNa-
mara, and S. Williams, Neuroscience. Sinauer Associates, 2004.
[29] E. Niedermeyer and F. L. da Silva, Electroencephalography: basic prin-
ciples, clinical applications, and related fields. Lippincott Williams &
Wilkins, 2005.
[30] R. Mercelis, “Practical approach to electroencephalography,” Spinal Cord,
vol. 48, pp. 840 EP –, Nov 2010. Book Review.
[31] K. G. Jordan, “Emergency EEG and continuous EEG monitoring in acute
ischemic stroke,” Journal of Clinical Neurophysiology, vol. 21, no. 5,
pp. 341–352, 2004.
[32] K. K. Ang, C. Guan, K. S. G. Chua, B. T. Ang, C. Kuah, C. Wang,
K. S. Phua, Z. Y. Chin, and H. Zhang, “Clinical study of neurorehabili-
tation in stroke using EEG-based motor imagery brain-computer interface
with robotic feedback,” in Engineering in Medicine and Biology Society
(EMBC), 2010 Annual International Conference of the IEEE, pp. 5549–
5552, IEEE, 2010.
[33] A. T. Tzallas, M. G. Tsipouras, and D. I. Fotiadis, “Epileptic seizure
detection in EEGs using time–frequency analysis,” IEEE transactions on
information technology in biomedicine, vol. 13, no. 5, pp. 703–710, 2009.
[34] A. Aarabi, R. Grebe, and F. Wallois, “A multistage knowledge-based
system for EEG seizure detection in newborn infants,” Clinical Neuro-
physiology, vol. 118, no. 12, pp. 2781–2797, 2007.
[35] G. Kamen and D. Gabriel, Essentials of Electromyography. Human
Kinetics 10%.
[36] A. Chowdhury, R. Ramadas, and S. Karmakar, “Muscle computer inter-
face: a review,” in ICoRD’13, pp. 411–421, Springer, 2013.
[37] K. Nymoen, M. R. Haugen, and A. R. Jensenius, “Mumyo–evaluating and
exploring the myo armband for musical interaction,” 2015.
[38] J. G. Abreu, J. M. Teixeira, L. S. Figueiredo, and V. Teichrieb, “Evalu-
ating sign language recognition using the myo armband,” in 2016 XVIII
Symposium on Virtual and Augmented Reality (SVR), pp. 64–70, IEEE,
2016.
[39] M. E. Benalcázar, A. G. Jaramillo, A. Zea, A. Páez, V. H. Andaluz,
et al., “Hand gesture recognition using machine learning and the myo arm-
band,” in 2017 25th European Signal Processing Conference (EUSIPCO),
pp. 1040–1044, IEEE, 2017.
[40] L. Torrey and J. Shavlik, “Transfer learning,” in Handbook of research
on machine learning applications and trends: algorithms, methods, and
techniques, pp. 242–264, IGI Global, 2010.
[41] U. Kamath, J. Liu, and J. Whitaker, “Transfer learning: Domain adapta-
tion,” in Deep Learning for NLP and Speech Recognition, pp. 495–535,
Springer, 2019.
[42] W. Van Der Aalst, “Data science in action,” in Process Mining, pp. 3–23,
Springer, 2016.
[43] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
the inception architecture for computer vision,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, pp. 2818–2826,
2016.
[44] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convo-
lutional neural networks for mobile vision applications,” arXiv preprint
arXiv:1704.04861, 2017.
[45] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions
on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2009.
[46] E. Hajiramezanali, S. Z. Dadaneh, A. Karbalayghareh, M. Zhou, and
X. Qian, “Bayesian multi-domain learning for cancer subtype discovery
from next-generation sequencing count data,” in Advances in Neural
Information Processing Systems, pp. 9115–9124, 2018.
[47] I. B. Arief-Ang, F. D. Salim, and M. Hamilton, “Da-hoc: semi-supervised
domain adaptation for room occupancy prediction using co 2 sensor data,”
in Proceedings of the 4th ACM International Conference on Systems for
Energy-Efficient Built Environments, p. 1, ACM, 2017.
[48] I. B. Arief-Ang, M. Hamilton, and F. D. Salim, “A scalable room oc-
cupancy prediction with transferable time series decomposition of co 2
sensor data,” ACM Transactions on Sensor Networks (TOSN), vol. 14,
no. 3-4, p. 21, 2018.
[49] C. B. Do and A. Y. Ng, “Transfer learning for text classification,” in
Advances in Neural Information Processing Systems, pp. 299–306, 2006.
[50] P. H. Calais Guerra, A. Veloso, W. Meira Jr, and V. Almeida, “From bias
to opinion: a transfer-learning approach to real-time sentiment analysis,”
in Proceedings of the 17th ACM SIGKDD international conference on
Knowledge discovery and data mining, pp. 150–158, ACM, 2011.
[51] M. Sharma, M. P. Holmes, J. C. Santamaría, A. Irani, C. L. Isbell Jr,
and A. Ram, “Transfer learning in real-time strategy games using hybrid
cbr/rl.,” in IJCAI, vol. 7, pp. 1041–1046, 2007.
[52] M. Thielscher, “General game playing in ai research and education,” in
Annual Conference on Artificial Intelligence, pp. 26–37, Springer, 2011.
[53] D. Wu, B. Lance, and V. Lawhern, “Transfer learning and active trans-
fer learning for reducing calibration data in single-trial classification of
visually-evoked potentials,” in 2014 IEEE International Conference on
Systems, Man, and Cybernetics (SMC), pp. 2801–2807, IEEE, 2014.
[54] H. Kang, Y. Nam, and S. Choi, “Composite common spatial pattern for
subject-to-subject transfer,” IEEE Signal Processing Letters, vol. 16, no. 8,
pp. 683–686, 2009.
[55] W. Tu and S. Sun, “A subject transfer framework for EEG classification,”
Neurocomputing, vol. 82, pp. 109–116, 2012.
[56] W.-L. Zheng, Y.-Q. Zhang, J.-Y. Zhu, and B.-L. Lu, “Transfer components
between subjects for EEG-based emotion recognition,” in 2015 Inter-
national Conference on Affective Computing and Intelligent Interaction
(ACII), pp. 917–922, IEEE, 2015.
[57] U. Côté-Allard, C. L. Fall, A. Drouin, A. Campeau-Lecours, C. Gosselin,
K. Glette, F. Laviolette, and B. Gosselin, “Deep learning for electromyo-
graphic hand gesture signal classification using transfer learning,” IEEE
Transactions on Neural Systems and Rehabilitation Engineering, vol. 27,
no. 4, pp. 760–771, 2019.
[58] C. Prahm, B. Paassen, A. Schulz, B. Hammer, and O. Aszmann, “Transfer
learning for rapid re-calibration of a myoelectric prosthesis after electrode
shift,” in Converging clinical and engineering research on neurorehabilita-
tion II, pp. 153–157, Springer, 2017.
[59] J. J. Bird, D. R. Faria, L. J. Manso, A. Ekart, and C. D. Buckingham, “A
deep evolutionary approach to bioinspired classifier optimisation for brain-
machine interaction,” Complexity, vol. 2019, 2019.
VOLUME 4, 2016 13
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2979074, IEEE Access
JJ Bird et al.: Cross-domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
[60] J. J. Bird, L. J. Manso, E. P. Ribeiro, A. Ekárt, and D. R. Faria, “A study
on mental state classification using EEG-based brain-machine interface,”
in 2018 International Conference on Intelligent Systems (IS), pp. 795–800,
IEEE, 2018.
[61] Z. Xiong and J. Zhang, “Neural network model-based on-line re-
optimisation control of fed-batch processes using a modified iterative
dynamic programming algorithm,” Chemical Engineering and Processing:
Process Intensification, vol. 44, no. 4, pp. 477–484, 2005.
[62] H. Huttunen, F. S. Yancheshmeh, and K. Chen, “Car type recognition
with deep neural networks,” in 2016 IEEE Intelligent Vehicles Symposium
(IV), pp. 1115–1120, IEEE, 2016.
[63] J. Zahavi and N. Levin, “Applying neural computing to target marketing,
Journal of direct marketing, vol. 11, no. 1, pp. 5–22, 1997.
[64] R. F. Albrecht, C. R. Reeves, and N. C. Steele, Artificial neural nets
and genetic algorithms: proceedings of the International conference in
Innsbruck, Austria, 1993. Springer Science & Business Media, 2012.
[65] M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programming
approach to designing convolutional neural network architectures,” in
Proceedings of the Genetic and Evolutionary Computation Conference,
pp. 497–504, ACM, 2017.
[66] V. Maniezzo, “Genetic evolution of the topology and weight distribution
of neural networks,” IEEE Transactions on neural networks, vol. 5, no. 1,
pp. 39–53, 1994.
[67] J. J. Bird, , E. Wanner, A. Ekart, and D. R. Faria, “Phoneme aware
speech recognition through evolutionary optimisation,” in The Genetic and
Evolutionary Computation Conference, GECCO, 2019.
[68] J. J. Bird, A. Ekart, and D. R. Faria, “Evolutionary optimisation of
fully connected artificial neural network topology,” in SAI Computing
Conference 2019, SAI, 2019.
[69] J. Ashford, J. J. Bird, F. Campelo, and D. R. Faria, “Classification of EEG
signals based on image representation,” in UK Workshop on Computa-
tional Intelligence, Springer, 2019.
[70] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, pp. 770–778, 2016.
JORDAN J. BIRD achieved a first class Bache-
lor’s Degree with Honours in Computer Science
at Aston University in the United Kingdom, be-
fore continuing with PhD studies at the same
institution in 2018 with an awarded scholarship.
Garnered through a deep Scientific passion from
an early age, his research interests exist largely
within the field of Human-robot Interaction; these
include, the Emergence of Artificial Intelligence,
Intelligent Social Frameworks, Turing’s Imitation
Game, Deep Machine Learning, and Transfer Learning. Jordan is a founding
member of the Aston Robotics, Vision and Intelligent Systems (ARVIS)
laboratory at Aston University.
JHONATAN KOBYLARZ is an Electronic Engi-
neering student at Universidade Federal do Paraná
(UFPR), Brazil. His research interests include
Deep Machine Learning towards Social Robotic
Interaction, Bioengineering and Computer Vision.
DIEGO R. FARIA is a Senior Lecturer in Com-
puter Science. He is with the School of Engineer-
ing and Applied Science, Aston University, Birm-
ingham (UK). He is the coordinator and founder
of the ARVIS Lab (Aston Robotics, Vision and
Intelligent Systems Lab). Currently (2019-2022)
he is the project coordinator of the EU CHIST-
ERA InDex project (Robot In-hand Dexterous ma-
nipulation by extracting data from human manipu-
lation of objects to improve robotic autonomy and
dexterity) funded by EPSRC UK. Dr Faria is also PI and Co-I (2020-2022) of
two projects with industry (KTP-Innovate UK scheme) related to perception
and autonomous systems applied to autonomous vehicles, and NLP and
image processing for multimedia retrieval. He received his Ph.D. degree
in electrical and computer engineering from the University of Coimbra,
Portugal, in 2014. He holds an M.Sc. degree in computer science from the
Federal University of Parana, Brazil, in 2005. In 2001, he earned a bachelor’s
degree in informatics technology (data computing & information) and he has
finished a computer science specialization in 2002 at the State University of
Londrina, Brazil. From 2014 to 06/2016 Dr. D. Faria was a post-doctoral
fellow at the Institute of Systems and Robotics, University of Coimbra
where he collaborated on different projects funded by EU commission
and the Portuguese government in areas of Robot Grasping and Dexterous
Manipulation, Artificial Perception, Cognitive Robotics and Assisted Living.
His research interests are: assisted living, perception systems, intelligent and
autonomous systems, and cognitive robotics.
ANIKÓ EKÁRT is a reader in Computer Science
at Aston University, where she is also Associate
Dean for Postgraduate Studies for the School of
Engineering and Applied Science. After obtaining
a PhD from Eötvös Loránd University, Budapest,
Hungary, she held research positions at the Com-
puter and Automation Research Institute, Hun-
garian Academy of Sciences in Budapest, Hun-
gary and a lectureship in Computer Science at
University of Birmingham, UK. Her research is
focused on theory and application of artificial intelligence techniques, and
in particular genetic programming and evolutionary computation. Areas of
application range from health to engineering, art and economics.
EDUARDO P. RIBEIRO is a Professor at Univer-
sidade Federal do Parana, Brazil, since 1997. He
received Bachelor’s degree in Electrical Engineer-
ing (1990), M.Sc. degree (1992) and Ph.D. degree
(1996) from Pontifical Catholic University of Rio
de Janeiro. He did research stage at Vanderbilt
University, USA (1995) and at The University of
British Columbia, Canada (2005). His research
interests include machine learning, signal process-
ing, instrumentation and communication.
14 VOLUME 4, 2016
... In addition, Demir et al. [106] applied AlexNet that are pre-trained in computer vision tasks to fine-tune the sEMG images. Bird et al. [107] investigated FT between sEMG and EEG signals, and observed that the knowledge could be successfully transferred between two modalities. To further enhance the effectiveness of FT, Chen et al. [108] constructed the source gestures composed of elbow, wrist, and finger joints. ...
... By contrast, sufficient but unlabelled D T data are available in UTL. For instance, FT approaches [67,[103][104][105][106][107][108] typically belong to STL, whilst the DANNbased approaches [34,112,113] exploit UTL. According to investigations on both conventional TL [96,115] and deep TL [104], the supervised versions usually perform significantly better than unsupervised ones. ...
Article
Full-text available
To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.
... Most current EEG research [35], [36], [37], [38] only takes into account the data distributions across subjects, which still ignores the distribution shift between datasets. Some modern models [21], [22], [23], [39], [40] have evolved to ease the distribution shifts across both subjects and datasets. In other words, these models can be applied in multiple datasets rather than a single dataset. ...
... In other words, these models can be applied in multiple datasets rather than a single dataset. For example, in the field of gesture classification, Bird et al. [39] proposed to transfer knowledge between an EEG dataset recording brainwaves and an Electromyographic (EMG) dataset recording muscular waves based on the pre-trained networks. They achieved a high accuracy of 85.12% and 93.82% by transferring EEG-to-EMG and EMG-to-EEG knowledge, respectively. ...
Preprint
EEG-based tinnitus classification is a valuable tool for tinnitus diagnosis, research, and treatments. Most current works are limited to a single dataset where data patterns are similar. But EEG signals are highly non-stationary, resulting in model's poor generalization to new users, sessions or datasets. Thus, designing a model that can generalize to new datasets is beneficial and indispensable. To mitigate distribution discrepancy across datasets, we propose to achieve Disentangled and Side-aware Unsupervised Domain Adaptation (DSUDA) for cross-dataset tinnitus diagnosis. A disentangled auto-encoder is developed to decouple class-irrelevant information from the EEG signals to improve the classifying ability. The side-aware unsupervised domain adaptation module adapts the class-irrelevant information as domain variance to a new dataset and excludes the variance to obtain the class-distill features for the new dataset classification. It also align signals of left and right ears to overcome inherent EEG pattern difference. We compare DSUDA with state-of-the-art methods, and our model achieves significant improvements over competitors regarding comprehensive evaluation criteria. The results demonstrate our model can successfully generalize to a new dataset and effectively diagnose tinnitus.
... Most neuroscience studies use multiple electrodes that are costly and not readily available, although they give a detailed results [11,12]. Recent advances in commercial EEG devices have seen the use of low-cost [13], non-invasive EEG headsets in various applications [14,15]. These headsets are easy to use, comfortable to wear and readily available for people to purchase and rarely require technical expertise in electrode handling and installation [16]. ...
Article
Full-text available
Electroencephalography (EEG) is a technique of Electrophysiology used in a wide variety of scientific studies and applications. Inadequately, many commercial devices that are available and used worldwide for EEG monitoring are expensive that costs up to thousands of dollars. Over the past few years, because of advancements in technology, different cost-effective EEG recording devices have been made. One such device is a non-invasive single electrode commercial EEG headset called MindWave 002 (MW2), created by NeuroSky Inc that cost less than 100 USD. This work contributes in four distinct ways, first, how mental states such as a focused and relaxed can be identified based on EEG signals recorded by inexpensive MW2 is demonstrated for accurate information extraction. Second, MW2 is considered because apart from cost, the user's comfort level is enhanced due to non-invasive operation, low power consumption, portable small size, and a minimal number of detecting locations of MW2. Third, 2 situations were created to stimulate focus and relaxation states. Prior to analysis, the acquired brain signals were pre-processed to discard artefacts and noise, and band-pass filtering was performed for delta, theta, alpha, beta, and gamma wave extraction. Fourth, analysis of the shapes and nature of extracted waves was performed with power spectral density (PSD), mean amplitude values, and other parameters in LabVIEW. Finally, with comprehensive experiments, the mean values of the focused and relaxed signal EEG signals were found to be 30.23 µV and 15.330 µV respectively. Similarly, average PSD values showed an increase in theta wave value and a decrease in beta wave value related to the focus and relaxed state, respectively. We also analyzed the involuntary and intentional number of blinks recorded by the MW2 device. Our study can be used to check mental health wellness and could provide psychological treatment effects by training the mind to quickly enter a relaxed state and improve the person's ability to focus. In addition, this study can open new avenues for neurofeedback and brain control applications. Supplementary information: The online version contains supplementary material available at 10.1007/s11277-022-09731-w.
... There are many kinds of methods to analyze the signals. One of them is neuro-fuzzy matrix modifier [1]. It calculates user's hand force vector based on the estimated joint torque to obtain hand acceleration to estimate the user's hand trajectory. ...
Preprint
This work introduces a method for high-accuracy EMG based gesture identification. A newly developed deep learning method, namely, deep residual shrinkage network is applied to perform gesture identification. Based on the feature of EMG signal resulting from gestures, optimizations are made to improve the identification accuracy. Finally, three different algorithms are applied to compare the accuracy of EMG signal recognition with that of DRSN. The result shows that DRSN excel traditional neural networks in terms of EMG recognition accuracy. This paper provides a reliable way to classify EMG signals, as well as exploring possible applications of DRSN.
... have been vigorously adopted in deep learning studies. [32][33][34] The classification session in this study was conducted based on traditional machine learning techniques, which are advantageous as they allow an easy and simple design compared with their complex and sophisticated counterparts. However, as a trade-off, it is restricted in terms of flexibility and functionality to accommodate diverse options or customise internal structures associated with performance enhancement. ...
Article
Purpose This study aimed to evaluate the efficacy of a new automated method for the evaluation of histopathological images of pterygium using artificial intelligence. Methods An in-house software for automated grading of histopathological images was developed. Histopathological images of pterygium (400 images from 40 patients) were analysed using our newly developed software. Manual grading (I–IV), labelled based on an established scoring system, served as the ground truth for training the four-grade classification models. Region of interest segmentation was performed before the classification of grades, which was achieved by the combination of expectation-maximisation and k -nearest neighbours. Fifty-five radiomic features extracted from each image were analysed with feature selection methods to examine the significant features. Five classifiers were evaluated for their ability to predict quantitative grading. Results Among the classifier models applied for automated grading in this study, the bagging tree showed the best performance, with a 75.9% true positive rate (TPR) and 75.8% positive predictive value (PPV) in internal validation. In external validation, the method also demonstrated reproducibility, with an 81.3% TPR and 82.0% PPV for the average of four classification grades. Conclusions Our newly developed automated method for quantitative grading of histopathological images of pterygium may be a reliable method for quantitative analysis of histopathological evaluation of pterygium.
Article
Intelligent Transport System should be renovated in many aspects in post-pandemic situation like COVID-19. The passenger-count inside a car will be restricted based on the vehicle capacity and the COVID-19 hot-spot zone. Traffic rules will be impacted to align with a similar contagious outbreak. The on-road ‘Yellow-Vulture’ cameras need to incorporate such surveillance rules to monitor related anomalies for preventing contamination. To maintain safe-distance, an automatic surveillance system will be preferred by the Government very soon. Moreover, facial mask usage during the journey has become an essential habit to stop the spread of the infection. In this article, we have proposed a deep-Learning based framework that employs an augmented image data set to provide proper surveillance in the transport system to maintain the health protocols. Fast and accurate detection of the number of passengers inside a car and their face masks from the traffic inspection camera feed has been demonstrated. We have exploited the advantages of the popular Transfer Learning approach with novel variations of images while performing the training. To the best of our knowledge, this is the first attempt to watch over in-vehicle social-distancing in post-pandemic circumstances through deep-Learning based image analysis. The superiority of the proposed framework has been established over several state-of-the-art techniques using different numerical metrics and visual comparisons along with a support of statistical hypothesis test. Our technique has achieved 98.5% testing accuracy in various adverse conditions. Zero-shot evaluation has been explored for the Real-Time-Medical-Mask-Detection data set Wang et al. (Real-Time-Medical-Mask-Detection, 2020a https://github.com/TheSSJ2612/Real-Time-Medical-Mask-Detection/, Accessed 14 Nov 2020), where we have attained 96.4% accuracy that manifests the generalization of the network.
Article
Efficient human-robot collaboration during physical interaction requires estimating the human state for optimal role allocation and load sharing. In this study, we present a transfer learning approach based on convolution neural networks (CNN) to predict motor control difficulty from muscle activities of subject (surface EMG) data in a physical human-robot interaction (pHRI) task. Twenty six individuals participated in a pHRI experiment where a subject guides the end-effector with different levels of motor control difficulty. The motor control difficulty is varied by changing the damping parameter of the robot from low to high and constraining the motion to gross and fine movements. A CNN network with raw EMG as input is used to classify the motor control difficulty. The CNN transfer learning is compared against Riemann geometry-based Procrustes analysis (RPA). With very few labeled samples from new subjects, we demonstrate that the CNN-based transfer learning approach (avg. 69.77%) outperforms the RPA transfer learning (avg. 59.20%). Moreover, we observe that the subject's skill level in the pre-trained model has no significant effect on the transfer learning performance of the new users.
Article
Objective: A body movement involves the complicated information exchange between the central and peripheral systems, which is characterized by the dynamical coupling patterns between the multiple brain areas and multiple muscle units. How the central and peripheral nerves coordinate multiple internal brain regions and muscle groups is very important when accomplishing the action. Approach: In this study, we extend the adaptive directed transfer function to construct the time-varying networks between multiple corticomuscular regions and divide the movement duration into different stages by the time-varying corticomuscular network patterns. Main results: The inter dynamical corticomuscular network demonstrated the different interaction patterns between the central and peripheral systems during the different hand movement stages. The muscles transmit bottom-up movement information in the preparation stage, but the brain issues top-down control commands and dominates in the execution stage, and finally, the brain's dominant advantage gradually weakens in the relaxation stage. When classifying the different movement stages based on time-varying corticomuscular network indicators, an average accuracy above 74% could be reliably achieved. Significance: The findings of this study help deepen our knowledge of central-peripheral nerve pathways and coordination mechanisms, and also provide opportunities for monitoring and regulating movement disorders.
Article
Electroencephalographic (EEG) and electromyography (EMG) signal classification seem to be a modulus topic in engineering and the medical field. The nature of the EEG and EMG signal is non‐stationary, noisy and high dimensional. The intrusion of noise in the signal may distress movement recognition. A novel methodology is being developed in this research to deal with these issues. Here, the EEG and EMG signals are recorded using the BCI2000 system. The proposed model comprises three phases: pre‐processing, feature extraction (FE), and motion classification. The pre‐processing method can be used to enhance visual appearances and the quality of the signal. The hybrid discrete wavelet based delayed error normalized least mean square error (DWT‐DENLMS) is introduced to eliminate the presence of motion artifacts in the recorded EEG and EMG signal. After pre‐processing, the recorded signals are combined and forwarded to the FE stage. The hybrid Dual tree complex wavelet transforms based on Walsh Hadamard transform (DTCWT‐WHT) is proposed to extract the indispensable time domain features from the combined biological signal. The hybrid capsule transient autoencoder (HCTAE) algorithm is proposed to classify the motion recognition (T0‐rest, T1‐left fist and both fists, and T2‐right fist, both feet). The error in the network model is diminished by transient search optimization (TSO) strategy. The Python platform is used to implement the developed approach, and the performance of the optimized classification approach yields accuracy, precision, recall, F1 score and specificity of 98.51%, 97.25%, 97.94%, 97.58% and 98.94%, respectively.
Thesis
Full-text available
In modern Human-Robot Interaction, much thought has been given to accessibility regarding robotic locomotion, specifically the enhancement of awareness and lowering of cognitive load. On the other hand, with social Human-Robot Interaction considered, published research is far sparser given that the problem is less explored than pathfinding and locomotion. This thesis studies how one can endow a robot with affective perception for social awareness in verbal and non-verbal communication. This is possible by the creation of a Human-Robot Interaction framework which abstracts machine learning and artificial intelligence technologies which allow for further accessibility to non-technical users compared to the current State-of-the-Art in the field. These studies thus initially focus on individual robotic abilities in the verbal, non-verbal and multimodality domains. Multimodality studies show that late data fusion of image and sound can improve environment recognition, and similarly that late fusion of Leap Motion Controller and image data can improve sign language recognition ability. To alleviate several of the open issues currently faced by researchers in the field, guidelines are reviewed from the relevant literature and met by the design and structure of the framework that this thesis ultimately presents. The framework recognises a user's request for a task through a chatbot-like architecture. Through research in this thesis that recognises human data augmentation (paraphrasing) and subsequent classification via language transformers, the robot's more advanced Natural Language Processing abilities allow for a wider range of recognised inputs. That is, as examples show, phrases that could be expected to be uttered during a natural human-human interaction are easily recognised by the robot. This allows for accessibility to robotics without the need to physically interact with a computer or write any code, with only the ability of natural interaction (an ability which most humans have) required for access to all the modular machine learning and artificial intelligence technologies embedded within the architecture. Following the research on individual abilities, this thesis then unifies all of the technologies into a deliberative interaction framework, wherein abilities are accessed from long-term memory modules and short-term memory information such as the user's tasks, sensor data, retrieved models, and finally output information. In addition, algorithms for model improvement are also explored, such as through transfer learning and synthetic data augmentation and so the framework performs autonomous learning to these extents to constantly improve its learning abilities. It is found that transfer learning between electroencephalographic and electromyographic biological signals improves the classification of one another given their slight physical similarities. Transfer learning also aids in environment recognition, when transferring knowledge from virtual environments to the real world. In another example of non-verbal communication, it is found that learning from a scarce dataset of American Sign Language for recognition can be improved by multi-modality transfer learning from hand features and images taken from a larger British Sign Language dataset. Data augmentation is shown to aid in electroencephalographic signal classification by learning from synthetic signals generated by a GPT-2 transformer model, and, in addition, augmenting training with synthetic data also shows improvements when performing speaker recognition from human speech. Given the importance of platform independence due to the growing range of available consumer robots, four use cases are detailed, and examples of behaviour are given by the Pepper, Nao, and Romeo robots as well as a computer terminal. The use cases involve a user requesting their electroencephalographic brainwave data to be classified by simply asking the robot whether or not they are concentrating. In a subsequent use case, the user asks if a given text is positive or negative, to which the robot correctly recognises the task of natural language processing at hand and then classifies the text, this is output and the physical robots react accordingly by showing emotion. The third use case has a request for sign language recognition, to which the robot recognises and thus switches from listening to watching the user communicate with them. The final use case focuses on a request for environment recognition, which has the robot perform multimodality recognition of its surroundings and note them accordingly. The results presented by this thesis show that several of the open issues in the field are alleviated through the technologies within, structuring of, and examples of interaction with the framework. The results also show the achievement of the three main goals set out by the research questions; the endowment of a robot with affective perception and social awareness for verbal and non-verbal communication, whether we can create a Human-Robot Interaction framework to abstract machine learning and artificial intelligence technologies which allow for the accessibility of non-technical users, and, as previously noted, which current issues in the field can be alleviated by the framework presented and to what extent.
Conference Paper
Full-text available
Phoneme awareness provides the path to high resolution speech recognition to overcome the difficulties of classical word recognition. Here we present the results of a preliminary study on Artificial Neural Network (ANN) and Hidden Markov Model (HMM) methods of classification for Human Speech Recognition through Diphthong Vowel sounds in the English Phonetic Alphabet, with a specific focus on evolutionary optimisation of bio-inspired classification methods. A set of audio clips are recorded by subjects from the United Kingdom and Mexico. For each recording, the data were pre-processed, using Mel-Frequency Cepstral Coefficients (MFCC) at a sliding window of 200ms per data object, as well as a further MFCC timeseries format for forecast-based models, to produce the dataset. We found that an evolutionary optimised deep neural network achieves 90.77% phoneme classification accuracy as opposed to the best HMM of 150 hidden units achieving 86.23% accuracy. Many of the evolutionary solutions take substantially longer to train than the HMM, however one solution scoring 87.5% (+1.27%) requires fewer resources than the HMM.
Conference Paper
Full-text available
This work presents an image classification approach to EEG brainwave classification. The proposed method is based on the representation of temporal and statistical features as a 2D image, which is then classified using a deep Convolutional Neural Network. A three-class mental state problem is investigated, in which subjects experience either relaxation, concentration, or neutral states. Using publicly available EEG data from a Muse Electroencephalography headband, a large number of features describing the wave are extracted, and subsequently reduced to 256 based on the Information Gain measure. These 256 features are then normalised and reshaped into a 16 × 16 grid, which can be expressed as a grayscale image. A deep Convolutional Neural Network is then trained on this data in order to classify the mental state of subjects. The proposed method obtained an out-of-sample classification accuracy of 89.38%, which is competitive with the 87.16% of the current best method from a previous work.
Article
Full-text available
This study suggests a new approach to EEG data classification by exploring the idea of using evolutionary computation to both select useful discriminative EEG features and optimise the topology of Artificial Neural Networks. An evolutionary algorithm is applied to select the most informative features from an initial set of 2550 EEG statistical features. Optimisation of a Multilayer Perceptron (MLP) is performed with an evolutionary approach before classification to estimate the best hyperparameters of the network. Deep learning and tuning with Long Short-Term Memory (LSTM) are also explored, and Adaptive Boosting of the two types of models is tested for each problem. Three experiments are provided for comparison using different classifiers: one for attention state classification, one for emotional sentiment classification, and a third experiment in which the goal is to guess the number a subject is thinking of. The obtained results show that an Adaptive Boosted LSTM can achieve an accuracy of 84.44%, 97.06%, and 9.94% on the attentional, emotional, and number datasets, respectively. An evolutionary-optimised MLP achieves results close to the Adaptive Boosted LSTM for the two first experiments and significantly higher for the number-guessing experiment with an Adaptive Boosted DEvo MLP reaching 31.35%, while being significantly quicker to train and classify. In particular, the accuracy of the nonboosted DEvo MLP was of 79.81%, 96.11%, and 27.07% in the same benchmarks. Two datasets for the experiments were gathered using a Muse EEG headband with four electrodes corresponding to TP9, AF7, AF8, and TP10 locations of the international EEG placement standard. The EEG MindBigData digits dataset was gathered from the TP9, FP1, FP2, and TP10 locations.
Conference Paper
Full-text available
This paper proposes an approach to selecting the amount of layers and neurons contained within Multilayer Perceptron hidden layers through a single-objective evolutionary approach with the goal of model accuracy. At each generation, a population of Neural Network architectures are created and ranked by their accuracy. The generated solutions are combined in a breeding process to create a larger population, and at each generation the weakest solutions are removed to retain the population size inspired by a Darwinian 'survival of the fittest'. Multiple datasets are tested, and results show that architectures can be successfully improved and derived through a hyper-heuristic evolutionary approach, in less than 10% of the exhaustive search time. The evolutionary approach was further optimised through population density increase as well as gradual solution max complexity increase throughout the simulation.
Conference Paper
Full-text available
Over the past decades, humans have redesign themselves with the intent to evolve beyond their current physical and mental limitations. This phenomenon has been known as transhumanism, wherein robotics and biomimetics have been exploiting the unique designs of the human body with the intent to develop disruptive anthropomorphic artificial appendages. Nevertheless, while lower extremity prosthetics have evolved to the point at which lower leg amputees may be competitive with professional runners in the world, there is still a gap between upper extremity prosthetics and real hands. This work intends to be a pioneer into developing a low-cost multipurpose robotic hand for research and academia. This paper describes the robotic hand, including its electromechanical development and full ROS integration. Moreover, the paper also presents a MatLab framework designed to introduce sequence data classification, namely providing the ability to control the robotic hand using electromyography (EMG) signals from the forearm. This paper expects to contribute to an ever-increasing human-robot symbiosis by motivating students to engage in transhumanism studies using more sophisticated technologies and methods.
Conference Paper
Full-text available
This work aims to find discriminative EEG-based features and appropriate classification methods that can categorise brainwave patterns based on their level of activity or frequency for mental state recognition useful for human-machine interaction. By using the Muse headband with four EEG sensors (TP9, AF7, AF8, TP10), we categorised three possible states such as relaxing, neutral and concentrating based on a few states of mind defined by cognitive behavioural studies. We have created a dataset with five individuals and sessions lasting one minute for each class of mental state in order to train and test different methods. Given the proposed set of features extracted from the EEG headband five signals (alpha, beta, theta, delta, gamma), we have tested a combination of different features selection algorithms and classifier models to compare their performance in terms of recognition accuracy and number of features needed. Different tests such as 10-fold cross validation were performed. Results show that only 44 features from a set of over 2100 features are necessary when used with classical classifiers such as Bayesian Networks, Support Vector Machines and Random Forests, attaining an overall accuracy over 87%.
Article
Full-text available
Human occupancy counting is crucial for both space utilisation and building energy optimisation. In the current article, we present a semi-supervised domain adaptation method for carbon dioxide - Human Occupancy Counter Plus Plus (DA-HOC++), a robust way to estimate the number of people within in one room by using data from a carbon dioxide sensor. In our previous work, the proposed Seasonal Decomposition for Human Occupancy Counting (SD-HOC) model can accurately predict the number of individuals when the training and labelled data are adequately available. DA-HOC++ is able to predict the number of occupancy with minimal training data, as little as one-day data. DA-HOC++ accurately predicts indoor human occupancy for five different rooms across different countries using a model trained from a small room and adapted to the other rooms. We evaluate DA-HOC++ with two baseline methods - support vector regression technique and SD-HOC model. The results demonstrate that DA-HOC++'s performance on average is better by 10.87% in comparison to SVR and 8.65% in comparison to SD-HOC.
Article
It is an exciting time to be involved in prosthetic technology research and development. The era is distinguished by quantum leaps in the understanding of how human bodies and brains work. Individuals who are not familiar with the current state of prosthetics engineering may not be aware of its impressive advancements. Improved materials and technologies are enabling many people with limb-related disabilities to return to the activities they previously enjoyed.
Chapter
Domain adaptation is a form of transfer learning, in which the task remains the same, but there is a domain shift or a distribution change between the source and the target. As an example, consider a model that has learned to classify reviews on electronic products for positive and negative sentiments, and is used for classifying the reviews for hotel rooms or movies. The task of sentiment analysis remains the same, but the domain (electronics and hotel rooms) has changed. The application of the model to a separate domain poses many problems because of the change between the training data and the unseen testing data, typically known as domain shift. For example, sentences containing phrases such as “loud and clear” will be mostly considered positive in electronics whereas negative in hotel room reviews. Similarly, usage of keywords such as “lengthy” or “boring” which may be prevalent in domains such as book reviews might be completely absent in domains such as kitchen equipment reviews.