Dynamic tunneling technique for efficient training of multilayer perceptrons

Defence Terraom Res. Lab., Delhi
IEEE Transactions on Neural Networks (Impact Factor: 2.95). 02/1999; 10(1):48 - 55. DOI: 10.1109/72.737492
Source: IEEE Xplore


A new efficient computational technique for training of multilayer
feedforward neural networks is proposed. The proposed algorithm consists
of two learning phases. The first phase is a local search which
implements gradient descent, and the second phase is a direct search
scheme which implements dynamic tunneling in weight space avoiding the
local trap and thereby generates the point of next descent. The repeated
application of these two phases alternately forms a new training
procedure which results in a global minimum point from any arbitrary
initial choice in the weight space. The simulation results are provided
for five test examples to demonstrate the efficiency of the proposed
method which overcomes the problem of initialization and local minimum
point in multilayer perceptrons

Download full-text


Available from: Pinaki Roy Chowdhury, Oct 25, 2013
  • Source
    • "In other words, these two approaches do not communicate well between each other. Approaches that might resemble our methodology are TRUST [6] and dynamic tunneling [23]. These methods attempt to move out of the local minimum in a stochastic manner. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Efficient training in a neural network plays a vital role in deciding the network architecture and the accuracy of these classifiers. Most popular local training algorithms tend to be greedy and hence get stuck at the nearest local minimum of the error surface and this corresponds to suboptimal network model. Stochastic approaches in combination with local methods are used to obtain an effective set of training parameters. Due to the lack of effective fine-tuning capability, these algorithms often fail to obtain such an optimal set of parameters and are computationally expensive. As a trade-off between computational expense and accuracy required, a novel method to improve the local search capability of training algorithms is proposed in this paper. This approach takes advantage of TRUST-TECH (TRansformation Under STability-reTaining Equilibrium CHaracterization) to compute neighborhood local minima on the error surface surrounding the current solution in a systematic manner. Empirical results on different real world datasets indicate that the proposed algorithm is computationally effective in obtaining promising solutions.
    Neural Networks, 2007. IJCNN 2007. International Joint Conference on; 09/2007
  • Source
    • "al. in [14]. The computational scheme described in [14] and used by us in our experiments is described below. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Over time, neural networks have proven to be extremely powerful tools for data exploration with the capability to discover previously unknown dependencies and relationships in the data sets. However, the sheer volume of available data and its dimensionality makes data exploration a challenge. Employing neural network training paradigms in such domains can prove to be prohibitively expensive. An algorithm, originally proposed for supervised on-line learning, has been improvised upon to make it suitable for deployment in large volume, high-dimensional domains. The basic strategy is to divide the data into manageable subsets or blocks and maintain multiple copies of a neural network with each copy training on a different block. A method to combine the results has been defined in such a way that convergence towards stationary points of the global error function can be guaranteed. A parallel algorithm has been implemented on a Linux-based cluster. Experimental results on popular benchmarks have been included to endorse the efficacy of our implementation.
    Proceedings of the International Joint Conference on Neural Networks, IJCNN 2006, part of the IEEE World Congress on Computational Intelligence, WCCI 2006, Vancouver, BC, Canada, 16-21 July 2006; 01/2006
  • Source
    • "Only after several consecutive failed attempts in escaping from a local minimum, will we allow the network to grow by adding a hidden-layer neuron. We note that the weight scaling process employed here may be substituted by any other techniques for attempting to escape from a local minimum, e.g., the dynamic tunneling technique [14]. However, the weight scaling process is much simpler to implement and works very well in our simulation. "
    [Show abstract] [Hide abstract]
    ABSTRACT: We develop, in this brief, a new constructive learning algorithm for feedforward neural networks. We employ an incremental training procedure where training patterns are learned one by one. Our algorithm starts with a single training pattern and a single hidden-layer neuron. During the course of neural network training, when the algorithm gets stuck in a local minimum, we will attempt to escape from the local minimum by using the weight scaling technique. It is only after several consecutive failed attempts in escaping from a local minimum that will we allow the network to grow by adding a hidden-layer neuron. At this stage, we employ an optimization procedure based on quadratic/linear programming to select initial weights for the newly added neuron. Our optimization procedure tends to make the network reach the error tolerance with no or little training after adding a hidden-layer neuron. Our simulation results indicate that the present constructive algorithm can obtain neural networks very close to minimal structures (with the least possible number of hidden-layer neurons) and that convergence (to a solution) in neural network training can be guaranteed. We tested our algorithm extensively using a widely used benchmark problem, i.e., the parity problem.
    IEEE Transactions on Circuits and Systems I Fundamental Theory and Applications 01/2003; 49(12):1876- 1879. DOI:10.1109/TCSI.2002.805733
Show more