Design And Implementation Of Guangdong POI Data Crawler Program Based On Amap.com

With the continuous and rapid development of network information technology,POI data has become more and more important as a very important network data resource.It has been widely used by planning designers in the land and space planning industry in the new century Although the POI data application is very convenient,due to the particularity of the POI data,the acquisition process is not easy.How to use low-cost,high-efficiency methods to obtain POI has been plaguing.This paper realizes the rapid and automatic capture of POI data from Internet maps,which provides planners with a good solution to the problems encountered at present,and also provides strong data support for the preparation and research of land space planning.After analyzing and researching the literature and the POI of many map data service providers,this paper selects the amap.com's POI as the research object.After researching the key technologies of POI data capture,the acquisition process of POI in Guangdong Province was finally developed.The main contents of the thesis include the following aspects.(1)Analyzed the POI usage requirements in the planning industry,and for the feasibility,the program based on the Python will be a fully automated crawling program.(2)By using the the Scrapy framework,we design the POI crawling program based on the amap.com of Guangdong Province and collected more than 5 million data.(3)We studied 7 secondary aspects including:?the acquisition and application of prefecture-level cities in Guangdong Province;? custom polygon search scope using recursive dichotomy method to divide city boundary;?studying on URL change method;? research to alter key of amap;? research on GCJ-02 coordinate system transform to WGS-84 coordinate system;?Storage test on tens millions data using Redis database and the Mongodb database;?research on remove duplicates in Mongodb database data,achieving such works on the POI data in Guangdong Province.(4)Design a Crawler program intent to collect POI data,compared with using third party software,and analysis the difference of those POI data.The results show that our program will collect a much more complete POI data database than the third-party software.