Research On Big Data Sets Of Co-Processing System With Community-Based LBS System

Collaborative processing system, which often contain a variety of algorithms for one application, usually means a data process with many ways. With the rapid development of modern information technology, particularly the wide spread of Internet technology, information growing explosively. The traditional way of information process has reached its limit when the data sets are much bigger. Nowadays, many data-sets has risen from MB, GB level to TB, PB level, the scale of these data-sets make the data process by traditional way a mission that cannot be accomplished. Therefore, we use the big data technology, combine the data process technology and the big data technology, to achieve multi-node parallel correlation analysis, to improve the limit of the data size and the data type of traditional data process system. With the actual needs of community-based LBS system, we use this system to provide big data analysis service for the managers of the community.Based on the above analysis, this paper completed the following main work:First, this paper analyzed the characteristics of community data, learned the current data preprocessing, knowledge and algorithms associated with big data storage and analysis, defined the meaning of community-based LBS system, co-processing of big data sets storage and analysis.Second, this paper designed a quantitative methods of community data, designed the data storage model based on HDFS distributing file system. With the study of the traditional correlation analysis algorithms, we propose a model based on multi-dimensional correlation analysis division. Third, with the design of the program, we simulate an actual procedure of data process, from data preprocessing to data mining, and use the field-related knowledge, to verify the accuracy of the system.