Abnormal Data Detection In Agriculture Search Engine

With the rapid development of IT application in agriculture, information of agriculture on the Internet is increasing rapidly. According to incomplete statistics, China has more than 30,000 agriculture-related sites. But much of the information on these sites is unorganized, multi-structure and dynamic, and this will reduce the personalized service quality of agricultural information. A vertical search engine for agriculture"Sounong"is developed with the support of project which is granted N0.2006BAD10A1410 named"an agriculture search engine based on Ontology". This research was supported by the national science and technology. Practice shows that good personalized service of agricultural information is provided by this vertical search engine.With the explosive growth of agricultural information, abnormal data is also greatly increased. These abnormal data include fuzzy data, incomplete data and abnormal data values. For these abnormal information, (Liu Feng, etc. 2009) proposed a treatment of incomplete data. We apply this treatment to the vertical search engine; the actual application shows that this method could work well. This article will focus on how to identify abnormal agricultural price data.This paper focuses on the outlier detection method for agricultural price data; introduces common outlier detection methods briefly, and applies these methods to the agricultural price data. This paper also features a detailed analysis on the characteristics of the agricultural price data. According to these characteristics a new method of abnormal data detection is presented. Also, we carried out experiments of these methods. According to these experiments, the proposed outlier detection algorithm for agricultural information can work well.How to apply outlier detection algorithm to vertical search engines is discussed in this paper too. This paper analyzes the system architecture of vertical search engines, and focuses on the data flow of the search engine. Based on the analysis, we present how to integrate this subsystem to the vertical search engine system. The application shows that the system which has abnormal data detection system can provide high quality personalized service.