Theory of noisy learning in nonlinear perceptrons: The cavity approach

The cavity method is applied to study the supervised learning of noisy examples in nonlinear perceptrons. The advantage of this mean field method is demonstrated in this study on the unique characteristics of nonlinear perceptrons.; Mean field equations of this complex system are obtained. The information conflict inherent in noisy examples gives rise to instability of the perturbative calculation if the weight decay to the student weight vector is weak. The occurrence of rough energy landscape with many metastable states and preferentially learning some examples in the cost of sacrificing others are investigated. It is found that the distribution of activation to examples shows band gaps when information conflict is serious, which is the case when both the training set and the noise temperature are large.; The generalization ability of a nonlinear perceptron trained with noisy examples is studied and compared with linear case. It is found that the student is able to decipher the rule of teacher in the large training set limit if it has the knowledge of the weight length of the teacher. Phase transition from a good generalization state to a poor one when the weight decay strength or the size of training set varies is predicted in theory and confirmed by simulation results. The full phase diagram of the system is plotted. The origin of discrepancy between theory and simulation is mainly attributed to the effect of rough energy landscape beside the finite size effect.