Multiobjective Hybrid Optimization and Training of Recurrent Neural Networks

Department of Computer Science and Artificial Intelligence, University of Grenada, Grenada, Spain.
IEEE TRANSACTIONS ON CYBERNETICS (Impact Factor: 6.22). 05/2008; 38(2):381-403. DOI: 10.1109/TSMCB.2007.912937
Source: PubMed


The application of neural networks to solve a problem involves tasks with a high computational cost until a suitable network is found, and these tasks mainly involve the selection of the network topology and the training step. We usually select the network structure by means of a trial-and-error procedure, and we then train the network. In the case of recurrent neural networks (RNNs), the lack of suitable training algorithms sometimes hampers these procedures due to vanishing gradient problems. This paper addresses the simultaneous training and topology optimization of RNNs using multiobjective hybrid procedures. The proposal is based on the SPEA2 and NSGA2 algorithms for making hybrid methods using the Baldwinian hybridization strategy. We also study the effects of the selection of the objectives, crossover, and mutation in the diversity during evolution. The proposals are tested in the experimental section to train and optimize the networks in the competition on artificial time-series (CATS) benchmark.

22 Reads
  • Source
    • "Also, a general framework using GA for designing neural network ensembles was presented in [14]. The authors in [5] proposed a hybrid MOGA method based on the SPEA2 and NSGA2 algorithms to optimize the training and the topology of the Recurrent Neural Network (RNN) simultaneously in time-series prediction problems. Another approach presented by [15] used to generalized multi-layer perceptrons (MLP) improved the performance of the evolutionary model. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Evolutionary Algorithms (EAs) are population based algorithms, which allow for simultaneous exploration of different parts in the Pareto optimal set. This paper presents Memetic Elitist Pareto Evolutionary Algorithm of Three-Term Backpropagation Network for Classification Problems. This memetic elitist Pareto evolutionary algorithm is called METBP and used to evolve Three-term Backpropagation (TBP) network, which are optimal with respect to connection weight, error rates and architecture complexity simultaneously. METPB is based on NSGA-II benefit from the local search algorithm that used to enhance the individuals in the population of the algorithm. The numerical results of METPB show the advantages of the combination of the local search algorithm, and it is able to obtain a TBP network with better classification accuracy and simpler structure when compared with a multiobjective genetic algorithm based TBP network (MOGATBP) and some methods found in the literature, the results indicate that the proposed method is a potentially useful classifier for enhancing classification process ability.
    International Journal of Advances in Soft Computing and its Applications 11/2014; 6(3):2074-8523.
  • Source
    • "Another method used by Generalized Multi-layer Perceptrons also improve the performance of the evolutionary model. The (Delgado et al., 2008) proposed a hybrid MOGA method based on the Strength Pareto Evolutionary Algorithm-2 (SPEA2) and NSGA-II algorithms to optimize the training and the topology of the Recurrent Neural Network simultaneously. (Jin et al., 2005) used MOGA and focused on the problem of multi objective optimization for feed-forward ANNs as a solution for the regularization problems in the network's complexity. "
  • Source
    • "The proposed algorithm is applied here to the problem of space-time neural receivers for DS-CDMA systems, even though, the algorithm is general and can be used for other applications of neural networks. It should also be remarked that some very recent contributions [17]-[24], which constitute the state-of-the-art in the area of RNN, can be considered for further investigation in the context of interference mitigation in communication systems such as CDMA, OFDM and MIMO [25]. However, we have opted to focus here on a RNN scheme combined with FIR filters in a decision feedback architecture, its application to space-time CDMA systems and the development of a novel estimation technique. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A space-time adaptive decision feedback (DF) receiver using recurrent neural networks (RNNs) is proposed for joint equalization and interference suppression in direct-sequence code-division multiple-access (DS-CDMA) systems equipped with antenna arrays. The proposed receiver structure employs dynamically driven RNNs in the feedforward section for equalization and multiaccess interference (MAI) suppression and a finite impulse response (FIR) linear filter in the feedback section for performing interference cancellation. A data selective gradient algorithm, based upon the set-membership (SM) design framework, is proposed for the estimation of the coefficients of RNN structures and is applied to the estimation of the parameters of the proposed neural receiver structure. Simulation results show that the proposed techniques achieve significant performance gains over existing schemes.
    IEEE Transactions on Neural Networks 12/2008; 19(11):1887-95. DOI:10.1109/TNN.2008.2003286 · 2.95 Impact Factor
Show more