Research On Self-adaptively Constructing CNNs Based On Mutual Information

Convolutional Neural Networks(CNNs) play an important role in deep learning models, which are widely used in computer vision, speech recognition, natural language processing, etc. CNNs not only own strong ability for the pattern recognition tasks, but also model distributed feature representations for the input dataset. The advantages of CNNs are benefited from its sparse and local structure, which mainly consists of local receptive field and pooling groups. However, to acquire good results identified, the structure of CNNs always has to be artificially designed carefully, which may require researchers to have deep understanding of the datasets. In this case, it is always impossible to apply CNNs to many other datasets that don’t show statistical property obviously. Whether the structure of CNNs could be totally decided by the dataset? To solve this problem, some researches have been finished, which would be introduced in this article below.(1) A new method is put forward for self-adaptively constructing convolutional neural network structure(called SAC-CNNs), which mainly consists of two phases: constructing the convolution layer and arranging the pooling layer. A local receptive field is constructed for each input neuron. SAC-CNNs firstly select the members for each local receptive field, and initialize their ranking order by the mutual information(MI) between each pair of input neurons. Then, SAC-CNNs employs a greedy algorithm to select the most informative parts of neurons as pooling centers for each pooling group, and defines the other neurons using MI. In order to verify SAC-CNNs experimentally, the target dataset is shuffled randomly where any obvious prior knowledge won’t be made use while constructing the network. The results showed that the structure constructed by SAC-CNNs is reasonable, and SAC-CNNs could achieve comparable performance with the best CNNs currently. Except for the classification accuracy, the difference of the local receptive fields and pooling groups between SAC-CNNs and the artificially designed CNNs, is evaluated. The results showed that the structure decided by the dataset is considerably similar with the artificially designed network.(2) SAC-CNNs introduce ranking order inconsistence among different local receptive fields. Aiming at this problem, two optimization strategies are proposed, i.e., confidence-weighting and ranking order self-adjusting. Confidence-weighting strategy weighted each local receptive field’s activations with the normalized MI values before the convolution operation, which helps mitigate the impact of inconsistence over the model’s classification performance. The ranking order self-adjusting strategy updated the ranking order within each local receptive field with the trained filters, which attempts to reduce the level of inconsistence substantially. The experiment results’ shown that both of these two strategies are reasonable and could improve the classification ability of SAC-CNNs. Besides, the distance with the best CNNs is further reduced after these two strategies are combined into SAC-CNNs.