The Study And Implementation Of Intelligent Information Collection System For Mobile APP Stores

With the rapid development of mobile Internet,mobile Internet has came into the era of big data,the demand of analysis on mobile app imformation become more and more apparent,thus the standard of mobile app information collection puted forward higher.Due to the large number of applications,almost all Third-party app stores only show little part of the applications in hyperlinks which can be reached,and majority information hidden in Deep Web database after the query form.The existing crawler strategy can not meet the demand.In response to this problem,researchers have applied Deep Web collection technology to mobile app stores,but it has not taken into account the characteristics of mobile app store website itself,which has leaded to the acquisition of application information coverage and efficiency is still low.At the same time,it's also time-sensitive like a regular web page,that is the structure of the web will change periodically,resulting in the failure of the crawler.In order to solve the above problems,this paper studied the information collection technology and a large number of mainstream mobile app stores,and completed the following work:1.This paper analyzes the characteristics of the web structure of mobile app store,information distribution,studies the search mechanism,summarizes the application information collection process and the technology used in different app stores.Put forward by building a rule base,the add and maintain a specific app store to the operation of the rule base,thereby improve the scalability and maintainability of the system;2.Propose a collection method based on category keywords query,by extractting a certain proportion of high weighting words which calculated by the tf-idf algorithm from application name +application description information as the search keyword to construct the search form of the app store,to make the applications which hidden in the Deep Web exposed.And then,combine with the Surface Web information collection technology,these applications information is collected,the coverage and efficiency of the application information collected are improved.3.Analyze the failure cause and effect of information collection system,and put forward a system failure warning strategy based on runtime and post-run data statistical analysis.To determine whether a system failure by analyzing the system run-time failure proportion of access to web pages,extraction of the fields,the proportion of successful access to web pages and successful extraction fields,issue a corresponding alarm to improve the timeliness and maintainability of the system.4.Designed a intelligent information collection system for mobile app stores,structure information collection code through rule base,after collecting the Surface Web application information,continue to collect Deep Web application information,and alarm the system through data statistics;5.Using the python programming language,Scrapy web crawler frame and so on to realize the intelligent information collection system for mobile app stores,and has carried on the experiment on the mainstream app stores.