Design And Implementation Of Web Data Acquisition System For Wireless City

With the rapid development of information technology in China, lots of kinds of communication tools (such as mobile phones, Internet, mobile Internet) have been introduced. These tools bring convenience, while they quietly change the work and life style of humans. Then, Demand for information processing increases. For those who are lost in all kinds of mass information, the way of information acquisition and transmission with more efficiency, more accuracy and more convenience has been the important solution of information providers. The concept of Wireless City has been raised for this situation. Wireless City includes two aspects, which are wireless coverage and wireless applications. Wireless applications refer to the scenario that the citizens obtain all kinds of public services (services provides by government, information about personal life) with mobile phones and other wireless terminals, truly anytime, anywhere, on demand. Therefore, the service information acquisition is an important part of wireless city service platform.This paper bases on Hadoop, the open-source distributed computing platform, and introduces the design and implementation of the distributed crawler system. First, the paper presents two core technology of Hadoop, HDFS (Hadoop Distributed File System) and MapReduce (Distributed Computing Framework of Hadoop). After analyzing the requirements of Wireless City Data Acquisition System over Internet, the author comes up with the design scheme of this system. Then, the author illustrates the logic architecture of the whole system, the physical deployment architecture, the work flow, the function module architecture, and data structures used in the system. Based on the design scheme, this paper introduces the key function modules of the whole system, and mainly presents the distributed processing procedure of the key function modules. At last, the paper elaborates the test work on the system, including function test and performance test in distributed computing clusters of different scales, to verify the usability of the system. Finally, the author points out the future work plan according to the imperfection of the current system.