An Unstructured Data Multi-Store System Based On Data Channel

As the result of the rapid development of Internet, data types and applications are becoming more and more complicated. One traditional RDBMS or NoSQL database has been unable to meet the demand of data storage and processing. Multi-store system becomes a new choice of data storage framework, therefore how to synchronize data between different storage components becomes the key issue. Traditional ETL or user-driven dual writes are incompetent for data synchronization in unstructured data multi-store system, which makes its development limited.According to the issues above, this paper proposes an unstructured data multi-store system based on data channel, the main contents include the following aspects:Firstly, in order to solve the problem of data synchronization in multi-store system, we present the data channel, which uses coprocessor and WAL files to capture data changes of HBase. The changes are serialized by serializer and saved in the cluster. Other storage components synchronize the data via querying to the data channel.Secondly, this paper builds an unstructured data multi-store system based on data channel. Around the interaction between the data channel and the storage components, we describe the role and the work contents of data channel in the multi-store system. Then demonstrate the convenience and feasibility of data channel in solving the synchronization problem in multi-store system by an example of a practical application of Internet.Finally, we design the experiments to test the data channel’s performance considering of availability, scalability, and verify the data channel having the advantage talked above by comparing with traditional synchronization methods, like ETL and user-driven dual writes. We demonstrate the practicability of our data channel and provide a new idea for constructing unstructured data multi-store system.