Design And Implementation Of A Meta-search Data Acquisition System For Quality Safety

Nowadays, quality and safety (QS) issues explode frequently, and with the popularityof the Internet, more and more QS issues are discussed by the public on the Internet. Thecomments about QS issues from the public and the reports from the online media can beused as text corpus for products QS analysis. Therefore, Internet can be a source of QSdata acquisition, and data base of QS analysis.This thesis aims to design and implement the meta-search-based data acquisitionsystem which is responsible for the QS-related web pages collection. In this paper,meta-search engine is used to collect data through query words setting by users instead oftraditional ways. This system can mainly be divided into three functional modules: querymodule, web extraction module and text categorization module. In query module, thesystem encapsulates a number of independent search engines and uses a priority queue tomanage query words. In web extraction module, the system uses this following two ways:template-based analysis which extracts links from web pages returned by search engineand statistical-based analysis which extracts text from web pages. In text categorizationmodule, a support vector machine (SVM) classification algorithm is used to filter theQS-related data. At last, the result of web extraction and text categorization is tested andthe statistical data is shown.Since QS-related web pages are dispersed on the Internet and QS-related data hasobvious features, meta-search-based data acquisition system is used instead of webcrawler in this paper. The research in this paper can be used in other similar fields.