| The traditional neural network has problems such as electrical signal interference and excessive power consumption,as well as the limitation of calculation speed caused by the Von Neumann architecture.Therefore,the optical neural network combining light and neural network has the advantages of large photon bandwidth,low transmission loss,low power consumption and fast processing speed,etc.,which has attracted great attention.With the rapid development of the field of artificial intelligence,multi-modality information processing is becoming more and more common.Therefore,it is important to study the more difficult and complex modeling and learning of information across different modalities.In this paper,the optical neural network is applied to the multi-modal fusion task,which not only expands the application of the optical neural network,but also provides a new idea for the realization of multi-modal fusion.The main research contents of this paper are as follows:1.In view of the lack of modal information that may exist in a single modality,based on the advantages of photonic high bandwidth,low power consumption and low crosstalk,this paper proposes a parallel fusion mechanism based on heterogeneous optical neural networks,and realizes the multimodal MNIST data set of classification tasks.The heterogeneous optical neural network is constructed by combining the optical convolutional neural network and the optical artificial neural network of different dimensions,processing data of different modalities through a parallel mechanism,and inputting the processed feature vectors into the fusion.Fusion decisions are made in layers.This paper also discusses the effects of learning rate,optimizer,and random Gaussian noise on network performance.In order to further verify the feasibility of the proposed model,architecture and algorithm,this paper made a heterogeneous optical neural network structure based on the MZI array on the Interconnect simulation software,and carried out digital identification according to the output optical power of the network to determine the design scheme.Finally,the classification accuracy of the heterogeneous optical network proposed in this paper is 95.75%on the multimodal MNIST dataset,and it is 1.73 times faster in terms of computational speed compared to the electronic heterogeneous network.2.In order to further improve the classification accuracy of the task,this paper proposes a heterogeneous optical neural network based on the fusion of attention mechanism,which introduces the attention mechanism in the fusion stage,and assigns different weights to the information of each modality in the fusion stage.The score further improves the fusion effect of the heterogeneous optical neural network.Similar to the above work,the heterogeneous optical network was built on the Interconnect simulation software to verify its effectiveness.In addition,on the classification task of multimodal MNIST dataset,the classification accuracy of the model is 98.31%,which is an accuracy improvement of 1.13%～5.80%compared with many current single-modal photonic neural networks;it is comparable to the accuracy of electronic heterogeneous networks. |
