Design And Implementation Of Distributed Expansion System For Enterprise-level Data

The data storage technology and application are one of the important topics in the field of the Internet and the basic guarantee of the development of the Internet enterprises, which has great theoretical significance and application value. With the arrival of the era of Internet plus, small and medium-sized Internet companies have encountered the problem of the rapid growth of data. The data storage architecture based on traditional RDBMS(relational database management system) and single point file is increasingly exposed with problems regarding to data security, access efficiency and scalability, which impedes the development of the enterprise. Thus, how we can make full use of open-source resources to distribute existing data storage architecture on the premise that not changing the existing Internet application system and provide enterprise with a data migration scheme which has low cost, simple adapters and is convenient to use and easy to expand, has great theoretical and practical value.In view of the problems mentioned above, combined with the universal data storage solution adopted by the small and medium-sized Internet enterprises, this paper designed and implemented the system from database and file system two aspects,and devide the system into database distributed expansion subsystem and file distributed expansion subsystem. In aspect of the database, aiming at retaining the enterprise's original software architecture based on relational database, this work designed a new data segmentation scheme combining the transverse segmentation on application layer based on dynamic datasource switching with the horizontal segmentation based on middle layer based on Mycat. In aspect of the file system,we put forward a "single user request in serial processing, multi user requests in parallel processing" multi thread allocation management strategy which solved the problem caused by a large number of requests packed into the system by a single user in a short time; and relying on Solr, we implemented the semantic file accessing by building a file identification index; through the analysis of the application characteristics of the system, we designed a read/write splitting module for file acquisition, which supports HTTP protocol providing the system with efficient, quick and diversified file obtaining capacities together with load balancing and cache controlling in reverse proxy module. At the end of this paper, we developed the overall system by Java based on the above design and completed the work about the enterprise data storage distributed migration, to achieve the requirements related to the architecture about securestorage, efficient access and easy expansion.In this paper, we tested the system with the original system's data set(database structured data and file documents) about the functionality and performance and compared the results with the original system. The new system not only owns the characteristics of good data security, high access efficiency, favorable expansibility, but also provides better functional extension, friendly interaction, developmental adaptation, which further enhances the value of the system. The system has been run on-line smoothly for about half a year in industrialization base of “Product Quality Traceability Technology” associated lab, providing the enterprise with more stable, flexible, efficient data storage architecture.