Research And Implementation On The Key Problems Of Data Storage In Video Big Data Analysis System

With the popularization of Internet application,more and more applications collected a large amount of data,along with the growth of these data,system of visits and concurrency are in increased dramatically,system performance problems increasingly serious.With the arrival of the era of big data,whether it is a traditional enterprise or a new type of Internet companies,are inevitably faced with the challenge of big data.How cheap storage and efficient access to massive data,has been a major problem faced by enterprises.Video big data analysis system referred to in this thesis is an information system based on Web SpringMVC and MyBatis framework which designed and developed by our laboratory.The database architecture design is a typical single storage system.With the video big data analysis system support more and more business,the amount of data in the table is also growing.Massive data storage and access to the database cause a considerable load,data processing capacity and access efficiency are also encountered bottlenecks.According to the characteristics of video big data rapid growth,starting from the perspective of distributed database,explore how the single storage system is extended to distributed storage system.The main work of this paper is focused on the following aspects:first,introducing the background of big data and the existing large scale distributed storage system,the existing NoSQL database is classified and summarized,and the design idea of the existing database is studied.Second,based on MySQL Sharding technology,propose database vertical split,table shard,partition three layer split mode,and integrated use of vertical and horizontal shard solution to split data hash to different databases or tables.At the same time,the distributed data access layer is designed on the ORM framework,which can effectively alleviate the pressure on the database and improve the data access efficiency.Third,analyzing the data characteristics of each node and using the database replication technology design high availability clustering scheme;To solve single point of failure,using the amoeba open-source framework achieve Read/Write Splitting and Load Balance.In summary,this article from the scalability,access efficiency,concurrent read and write,high availability and other aspects,discusses the key problem facing big data storage,and puts forward the corresponding solutions.On the basis of the video big data analysis system,the efficient storage and access solution based on the distributed relational database in this thesis is verified in practice.