Parallel Computing Of Fully Connected And Convolutional Neural Networks Using COStream

Neural network is a research hotspot in the field of artificial intelligence.One of the disadvantages of neural networks is the huge amount of computation caused by iteration,and speeding up the training of neural network models can save a lot of time.The COStream data stream programming model is a data-driven parallel model.The parallelization of the neural network model with COStream has a practical meaning for improving its training speed.In view of the huge computing amount in neural networks,the parallelization method provided by COStream programming model,consist of the characteristics of data parallelism and pipeline parallelism in neural network model,which is used for implementing the three-layer fully connected neural network parallel model and LeNet-5 convolutional neural network parallel model implemented.The three-layer fully connected neural network model defines the operation of each layer node in the “composite”,and expands nodes of the layer through the “splitjoin” structure,each layer is connected by a “pipeline” structure.The LeNet-5 convolutional neural network model connects multiple layers of fully connected layers after multi-layer convolution and pooling,where the convolutional layer uses a two-layer “splitjoin” nested structure to expand different convolution kernel groups and kernels on different channels.The convolutional layer,pooled layer,and fully connected layer are connected by a “pipeline” structure.Cause the COStream uses phased pipeline parallelism,for the data-dependent computing model between neural networks,there will be inconsistencies between parameters and data batches.The “synchronization in group and asynchronization between group” scheduling method is proposed to solve it.The experiment uses the X86-64 architecture multi-core processor as the target platform,which tests the execution effect of the three-layer fully connected neural network program and the LeNet-5 convolutional neural network program under different core numbers.The experimental results proves that COStream is effective for speeding up the training of neural network models.