Study Of The Performance Optimization Of E-Commerce Database On Flash Hybrid Storage

The current development trend of e-commerce is in full swing. The increasing scale of e-commerce transactions result in a massive e-commerce data, witch makes the performance improvement of e-commerce database to read and write, update and query more and more difficult. The main solutions have been close to the limit of the performance optimization of database systems. However, flash storage technology development to bring new opportunities to solve the above problem. The I/O performance of flash is100times over disks, which is a way to break through the above bottleneck. This thesis intends to start from the underlying storage technology research, taking into account the upper software algorithms to enhance the performance of e-commerce database running in the massive data environment.The study found that the speed of sequential read and write, random read of flash chip is much faster than the disk, but the random write and update performance of flash are lower than disks, and the price of flash chip is much higher. So, this thesis proposes to use a hybrid storage system of flash and disk. Witch comprehensively utilizes of the advantages of both, and also takes into account the cost of equipment prices. But there is no system study on database deployment scheme based on hybrid storage systems. This thesis explores the performance optimization of e-commerce database based on hybrid storage system. The research results enrich the content of existing e-commerce database, and based on the development of e-commerce enterprises run by the e-commerce database. And the research has important practical significance for the development of e-commerce businesses based on the e-commerce database.The main contents of the thesis consist of following four aspects:(1) To build differentiated storage model based on hybrid storage system.From the point of view of e-commerce enterprise applications, the thesis creates differentiated storage model from two levels of data tables and data attributes, designes the threshold algorithm to determine the reading and writing sensitivity and hot or cold characteristic for table and attribute, realizes the analysis the reading and writing sensitivity and hot or cold characteristic for table and attribute and proposes migration storage solutions.(2) The study of query performance optimization of the differentiated storage model on the hybrid storage systems.Using e-commerce platform, consumers need to execute a large number of database query operations. To ensure and improve the e-commerce database query performance is essential to ensure the operational quality of the e-commerce businesses. Firstly, the thesis designed to study the HS-Join algorithm, used to improve the join query performance of the multi-table stored in different storage media which storage unit is table. Secondly, the thesis design to study the PHS-Join algorithm to establish a child connection table on flash and then reconstruct to generate the overall connection table based on the child table. The experiments show that the HS-Join and PHS-Join algorithm can significantly improve database query performance.(3) The study of transaction processing performance optimization of the differentiated storage model on the hybrid storage systems.The performance of e-commerce database transaction processing is an important technical foundation for e-commerce enterprises to improve their ability to respond, to maintain more number of consumers, and to meet the individual needs of each consumer at the same time. Taking Oracle database as an example, the thesis firstly analyzes the reading and writing sensitivity and hot or cold characteristic for the data of temporary table and log file, and migrate these two types of file to the flash device. Secondly, based on the rapid growth of the log file, when the flash lack for space, the thesis proposes Dynamic Logging method, logging dispersed stored in the flash space. The experiments show that the management method can significantly improve the performance of transaction concurrency of the e-commerce database.(4) The study of performance optimization of the data warehouse on the differentiated storage model on the hybrid storage systems.Optimize the query analysis performance of e-commerce data warehouse is the important technical foundation for e-commerce business to make businesses decisions more quickly and accurately. The thesis proposes the column migration storage algorithm on using column storage solution, which improved query efficiency of the decision-making attribute data. In addition, this thesis presents the hybrid storage of row and column method to reduce connection cost by solidifying associated attributes in advance. Finally, this thesis proposes differentiated data compression scheme, witch not only protects the data query efficiency, but also takes effectively use of the disk space.The innovative points and the main contributions of this paper are:(1) Considering from the economic feasibility and technical feasibility of the e-commerce enterprises, the thesis proposed to use hybrid storage system of flash and disk for the e-commerce database and build differentiated storage model from the data table and attribute granularity level, which is not only conducive to achieve the application of e-commerce businesses, but also to meet the response speed requirements by the user.(2) The thesis proposed the HS-Join circulation algorithm and PHS-Join algorithm suitable for query join of large-scale e-commerce database, which takes full advantage of the random read performance of flash, improves the query performance of the system。 Through reasonable deployment of the database file in a hybrid storage system, the study greatly improves the performance of concurrent transaction processing of the database and enhanced the business processing capabilities of the e-commerce business, and makes it possible that e-commerce enterprises quickly and easily applying the flash device and directly using of existed disk-based database management systems.(3) The thesis proposes to use hybrid storage technology of row and column for performance optimization of the e-commerce data warehouse, to solidify associated attributes of high connection frequency in advance, which improves the speed and efficiency of the decision-making analysis. Additionally, utilization differentiated data compression scheme according on hybrid storage model. The compression scheme not only improves the storage efficiency of two types of storage devices, but also to ensure the query speed of data warehouse.