Research And Implementation On Data Service Of Heterogeneous Data Source In High Concurrency Environment

With the development of science and technology as well as social progress,more and more enterprises have started Internet business,which results in the needs of having their own data storage and management systems.Consequently,these enterprises have stored massive heterogeneous data sources with various data structures and formats relevant to their business needs.It leads to the era of big data.However,the heterogeneity of massive data becomes a big problem in using these big data.Thus,finding a way to block the heterogeneity of these data sources is the premise of utilizing big data.With rapid increasing of Internet users,huge user visits require high performance of enterprises'servers.A single server,no matter how high the performance is,can never meet the needs of increasing user visits,so the message bus based heterogeneous data service is a good way to solve the above two problems.This paper presents a method of integrating heterogeneous data sources so as to solve the heterogeneity problem of heterogeneous data sources.Besides,this paper also proposes a load-balancing algorithm to help reasonably optimize task assignment of server cluster,so as to optimize the performance,which allows enterprises to effectively deal with high-concurrency of massive user visits.This paper proposes a middleware-based parsing method of heterogeneous data,that is,users query data aiming at the unified data model provided by middleware,which decomposes the query statement into ones for different data sources and then gets data from encapsulator.Among them,integration of unified data type is the core problem of middleware method.Therefore,this paper will elaborate on the mapping methods of turning heterogeneous data type into intermediate type,and then into unified data type.In addition,this paper addresses the problem of uneven task assignment of server cluster through load balancing technology based on message bus:connect all servers in the cluster to message bus,and each server can get others' loading condition,so as to conduct dynamic task assignment through message bus,better solving the problem of high concurrency.In the last part,this paper conducts simulation of heterogeneous data source integration and integration of a relational data source and an XML data source is achieved.Besides,it implements message bus by using ActiveMQ message-oriented middleware and simulates load balancing of cluster server by connecting four servers to message bus.