A Sensitivity-based Approach For Pruning Architecture Of Madalines

Architecture design is a very important issue in neural network research. One popular way to find proper size of a network for a given task is to prune an oversize trained network to a smaller one while keeping established performance. This paper presents a sensitivity-based approach that prunes hidden Adalines from a Madaline with causing as little as possible performance loss and tries to compensate for the loss. The approach is novel in setting up a relevance measure, by means of an Adalines' sensitivity measure, to locate the least relevant Adaline in a Madaline. The sensitivity measure is the probability of an Adaline's output inversions due to input variation with respect to overall input patterns, and the relevance measure is defined as the multiplication of the Adaline's sensitivity value by the summation of the absolute value of the Adaline's outgoing weights. Based on the relevance measure, a pruning algorithm can be simply programmed, which iteratively prunes an Adaline with the least relevance value from hidden layer of a given Madaline and then conducts some compensations until no more Adalines can be removed under a given performance requirement. The effectiveness of the pruning approach is verified by some experimental results. Besides, the paper also discusses the trial of improving Madalines' learning performance by using the Adalines' sensitivity measure.