Design And Implementation Of Text Classifier Based On Neural Network With Spark

In recent years,AI technology has become more and more popular,especially in the field of deep learning,which is developing rapidly with the wave of big data,but training deep learning networks is a very time-consuming process,so it is important to make full use of cluster resources for distributed parallel training.However,some popular distributed computing frameworks such as MapReduce and Spark do not support dense network IO and asynchronous calls in distributed deep learning work.Therefore,this article implements a deep learning system that is used in Spark clusters.Distributed deep network model training implements the design and implementation of distributed text classifiers.Text classification is currently a relatively heavy business for many enterprises,and the application scenarios are very extensive.Therefore,the distributed deep learning system in this paper mainly focuses on text classification.This paper will mainly research and explore the implementation of distributed text classifiers and the operation scheme of deep learning models for distributed text classification on the distributed computing platform Spark.The main work is as follows:(1)Implemented a deep learning system for text classification based on Spark.Users can create,train,and classify distributed classifiers through the web interface of the platform.The implemented distributed deep learning system can support the distributed training and parallel classification of classifier models on Spark clusters.The deep learning system also implements a variety of parallel algorithms required for distributed training for users to choose.(2)A solution for simple porting of a stand-alone model to a distributed environment has been implemented.At present,there are already many excellent deep learning models in the field of natural language processing.Users can implement these models based on various deep learning frameworks,or propose their own deep learning model.In order to facilitate users to work existing stand-alone deep learning models on Spark,this article proposes a set of solutions for user transplantation.Users only need to implement specific methods based on this solution and submit it to the deep learning system,and model can work on the Spark environment.(3)Implemented some commonly used distributed deep learning classifiers for text classification.In order to facilitate the use of users and reduce the cost of creating models for ordinary users.The thesis implements distributed versions of some of the popular text classification deep learning models through deep learning systems.Users can choose freely when using this system.Some distributed text classifiers implemented by the Spark-based text classification deep learning system in this article can meet the text classification needs of some companies.The system also provides users with a feasible solution for distributed and parallel operation of deep learning models.