Design And Realization Of Oriented Acquisition System For Internet Meteorological And Hydrological Data

With the development of international meteorological and hydrological exchange and sharing technology and Internet data oriented crawling technology,and according to our country’s demand for meteorological and hydrological observation data,it is particularly important to make full use of the Internet meteorological and hydrological data resources to make up for the lack of meteorological and hydrological data in the relevant hot spots.Based on the advanced technology at home and abroad,this paper has designed the system scheme of the oriented acquisition of internet meteorological and hydrological data,and developed the prototype software.The system has widened the collecting channels of meteorological and hydrological data,solved the problem of lack of meteorological and hydrological data in the related areas,and provided an important support for the development of the operational collection of the meteorological hydrological data of the Internet.The system adopts the automatic acquisition and transmission process to ferry data from the Internet download area to the internal network for distribution and sharing.The System consists of five subsystems,which are Comprehensive setting of Internet data oriented acquisition subsystem,Task scheduling of Internet data oriented acquisition subsystem,Web page analysis and Data acquisition subsystem,Internet data preprocessing subsystem and Network ferry subsystem.For each sub-system and function,this paper analyzes the use of related technologies in detail and summarizes three key technologies,which contain the page structure analysis technology based on coordinate tree,the page content extraction technology based on WEB page feature and the data acquisition technology based on IP address pool and proxy pool.These key technologies lay a solid foundation for the successful development of the system.After the successful design of the prototype software,a instance test for data acquisition is carried out,and the test results show that,the system can be used to realize the Internet Oriented collection of meteorological hydrological data,under the background of the rapid development of information network.