Design And Implementation Of Natural Language Processing Capability Open Platform

With the gradual arrival of Web 3.0-era,de-centralized Internet applications put forward higher requirements for artificial intelligence and search engine technology.As one of the basic steps of artificial intelligence and search engine,natural language processing needs to ensure the correctness,rapidity and stability.However,natural language processing is essentially a multi-lateral interdisciplinary subject,which has considerable complexity in both theory and practice.This greatly increases the difficulty of the development of related applications.Referring to open platform,extracting algorithms with high reuse,forming abilities with distinct function,and offering as Open API(Application Programming Interface)to third-party developers,can significantly lower the threshold of the development of natural language processing applications,and thus raise the productivity of industry.By combining natural language processing with open platform,this thesis designs and implements a natural language processing capability open platform.This open platform is divided into service access layer,platform layer,algorithm layer,communication layer and portal systems.In service access layer,authentication mechanism is implemented based on Oauth 2.0 standard,and secure network communication is guaranteed by HTTPS protocol.To solve flow control,leaky bucket/token bucket algorithm's basic idea and use cases are emphatically compared,and frequency-limit strategy,time-limit strategy and whitelist-filter strategy on service/client dimension are implemented based on token bucket algorithm.In platform layer,firstly,capability subsystem is constructed by REST(Representational State Transfer)to dock with clients,and caching mechanism of response results is applied in reducing response time.Secondly,log subsystem is implemented to guarantee the stability of the platform in running time,which consists of six processes,log collection,log caching,log analysis,log storage,log query and log monitor/alarm.In algorithm layer,several algorithm model's loading strategies and threading models are studied from perspective of engineering,and efficient algorithm services are provided based on Reactor pattern.In communication layer,connection pool,load balance and failover are employed to optimize performance in response time and reliability.Lastly,self-service portal and management portal are established.Following the discipline of software engineering,this thesis firstly introduces the research background and key technologies of the platform,then analyses the functional requirements and the non-functional requirements of the platform,afterwards designs and implements the platform according to the requirements,and finally accomplishes the functional tests and the non-functional tests.Some experimental results have shown the correctness of design and implementation.