Degenerated simplex search method to optimize neural network error function

Ahmed, Shamsuddin (2013) Degenerated simplex search method to optimize neural network error function. Kybernetes, 42 (1). pp. 106-124. ISSN 0368-492X

PDF - Accepted Version Restricted to Repository staff only Download (154kB)

Abstract

Purpose 13By repeatedly reflecting and expanding a simplex, the centroid property of the simplex changes the location of the simplex vertices. The proposed algorithm selects the location of the centroid of a simplex as the possible minimum point of an artificial neural network (ANN) error function. The algorithm continually changes the shape of the simplex to move multiple directions in error function space. Each movement of the simplex in search space generates local minimum. Simulating the simplex geometry, the algorithm generates random vertices to train ANN error function. It is easy to solve problems in lower dimension. The algorithm is reliable and locates minimum function value at the early stage of training. It is appropriate for classification, forecasting and optimization problems.

Design/methodology/approach 13 Adding more neurons in ANN structure, the terrain of the error function becomes complex and the hessian matrix of the error function tends to be positive semi-definite. As a result,derivative based training method faces convergence difficulty. If the error function contains several local minimum or if the error surface is almost flat, then the algorithm faces convergence difficulty. The proposed algorithm is an alternate method in such case. This paper presents a non degenerate simplex training algorithm. It improves convergence by maintaining irregular shape of the simplex geometry during degenerated stage. A randomized simplex geometry is introduced to maintain irregular contour of a degenerated simplex during training.

Findings 13 Simulation results show that the new search is efficient and improves the function convergence. Classification and statistical time series problems in higher dimensions are solved. Experimental results show that the new algorithm (degenerated simplex algorithm, DSA) works better than the random simplex algorithm (RSM) and back propagation training method (BPM). Experimental results confirm algorithm 19s robust performance.

Research limitations/implications 13 The algorithm is expected to face convergence complexity for optimization problems in higher dimensions. Good quality suboptimal solution is available at the early stage of training and the locally optimized function value is not far off the global optimal solution, determined by the algorithm.

Practical implications 13 Traditional simplex faces convergence difficulty to train ANN error function since
during training simplex can 19t maintain irregular shape to avoid degeneracy. Simplex size becomes extremely small. Hence convergence difficulty is common. Steps are taken to redefine simplex so that the algorithm avoids the local minimum. The proposed ANN training method is derivative free. There is no demand for first order or second order derivative information hence making it simple to train ANN error function.

Originality/value 13 The algorithm optimizes ANN error function, when the hessian matrix of error function is ill conditioned. Since no derivative information is necessary, the algorithm is appealing for instances where it is hard to find derivative information. It is robust and is considered a benchmark algorithm for unknown optimization problems.

Item Type:	Journal Article
Uncontrolled Keywords:	Simplex search, Derivative free, ANN, Training Algorithm, Robust, Benchmark solution, Paper type - Research paper
Subjects:	H Social Sciences > HA Statistics
Divisions:	Faculty of Business and Economics (FBE) > Graduate School of Business
Depositing User:	Professor Shamsuddin Ahmed
Date Deposited:	01 May 2013 22:58
Last Modified:	12 Jun 2016 21:43
URI:	https://repository.usp.ac.fj/id/eprint/5784

Actions (login required)

View Item