Principles of convergent rate and generalization enhancement for feedforward sigmoid-like network

Abstract

This dissertation demonstrates how the p-recursive piecewise polynomial (p-RPP) generators and their derivatives are constructed. The feedforward computational time of a multilayer feedforward network can be reduced by using these functions as the activation functions. Three modifications of training algorithms are proposed. First, the modified error function is used so that the sigmoid prime factor for the updating rule of the output units is eliminated. Second, the input patterns are normalized in order to balance the dynamic range of the inputs. Third, a new penalty function is introduced to the hidden layer to get the anti-Hebbian rules providing information when the activation functions have zero sigmoid prime factor. The three modifications are combined with two versions of Rprop (Resilient propagation) algorithm. The proposed procedures achieved the excellent results without the need for careful selection of the training parameters. Not only the algorithm but also the shape of the activation function has important influence on the training performance.