Research On Neural Networks Based On Vectorized Neurons And Application Of Attention Mechanism

As the main research content of artificial intelligence,artificial neural network has been widely concerned because of its excellent performance in the fields such as computer vision and natural language processing since it was proposed in the 1940s.The neuron model of the traditional neural network was proposed by McCulloch and Pitts in 1943(MP neurons),But MP neurons cannot be selectively activated like biological neurons.Based on this,this paper studies the attention mechanism and proposes vectorized neuron and its activation function.First,we propose vectorized neurons,then dynamically generate connection weights between vectorized neurons through the attention mechanism,finally,we construct a novel neural network with vectorized neurons,called neural functional group(NFG).The vectorized neuron and its activation function solve the problem that traditional MP neurons cannot be selectively activated.In addition,vectorized neuron has the potential to representing complex biological neuron,which is difficult for MP neuron.We tested the proposed neural functional group model on two tasks:image classification and few-shot learning.The experimental results show that it can achieve higher accuracy with fewer parameters than convolutional neural networks(CNN)and capsule networks in image classification task;it also competitive to CNN based feature extractor in few-shot learning task.Secondly,we have studied the application of the attention mechanism and proposed a hyperspectral image classification algorithm HSI-BERT based on the attention mechanism.The algorithm uses the attention mechanism to learn the global dependence of the input area and capture global context information.The richer global context information enables HSI-BERT to extract more valuable information,and makes the model converge more accurately.Our comparative experiments on three widely used hyperspectral image datasets show that HSI-BERT is superior to the latest CNN based spatial spectral algorithm in terms of classification accuracy and training time.In addition,since the attention mechanism makes no assumptions about the temporal and spatial relationships across the input data,HSI-BERT can also flexibly select input areas of various shapes and sizes.