Research And Implement Of Multi-mainpage Service Framework Based On Mapreduce

With the rapid development of Internet technology and the continuousimprovement of the Cloud storage technology, large-scale data processingmechanism has undergone tremendous changes. With the face of large-scale datagenerated every day by the Internet applications, we propose a new service modelnamed BIRIS (BIlateral Resource Integration Service, BIRIS), and we alsointroduce a fourth party service platform—BIRIS-Cloud. Based on this platform,we propose a multi-mainpage service framework. The framework is based on theframework of the Hadoop MapReduce programming model, it provides service toseveral service centers of the BIRIS-Cloud platform. With the support of theframework, the system supports large-scale data organization, storage, and access,which from different service centers. And it also can provide better service for theservice center’s users.By reading a lot of literature, I study the organization of large-scale data. Theplatform adopts the HDFS from the open-source framework Hadoop as thelarge-scale data storage system. Based on the storage system, establish index for thelarge-scale data which is stored on it. And also the index file is stored on each node,which can be found concurrency.By combining multi-mainpage service framework with the characteristics oflarge-scale data storage, this paper analyzes the system requirements. Themulti-mainpage service framework is designed to provide mainpage service todifferent service centers. By defining and analyzing the mainpage model under themainpage framework, the function of the multi-mainpage service framework isbetter expressed. The users make this form of service implement ed by subscripingthe mainpage models. The paper also studies how the large unstructured data isstored, and it provides appropriate strategy to solve the problem. To make differentindividual’s requirements better meet who has the demand to the large-scale datastorage.How to access large-scale data has been a hot research. For the data storedform on the HDFS, this system adopts the inverted index mechanism to comply the index established on the data needed. On the basis of the inverted index, thetwo-level index creation is completed. And finally achieve the positioning of therequired data.At last, the multi-mainpage service framework is set up. On the basis of theframework, we achieve the target of effective management of large-scale data. Weapply the studied theoretical and the key techno logies on the framework, themanagement of the large-scale data. After the framework is set up, we take theadvantage of the mainpage model gathered on the framework to provide service fordifferent service center. On the basis of this framework, constantly improve theservice mechanism, making the framework run perfectly. The value of theframework is also well reflected.