Research On Key Technologies Of Deep Web Data Integration

With the rapid development of the web, more and more information has been transferred from static web pages (that is Surface Web) into web databases (that is Deep Web) managed by web servers. Public information on the Deep Web is currently 400 to 500 times larger than that of Surface Web, compared with Surface Web, Deep Web not only contains higher quality information, but also constitutes the fastest growing web data carrier, therefore, it has become an urgent task to accelerate research on Deep Web.The purpose of integrating Deep Web databases is to make full use of information to a multitude of online databases as much as possible. In the oriented-domain information retrieval, the relevant domain knowledge generally helps to improve the search results. Ontology as the foundation of knowledge processing, a modeling tool which describes concepts and concept relationships in semantic level, is widely used in information retrieval. Therefore, this paper studies Deep Web searching techniques by systematically employing ontology, the main contents are as follows:(1) Construct domain ontology: Ontologies use the hierarchical tree structure to describe logical relationships between concepts, user's queries and relevant data can be mapped to the concept model. Domain ontology can be seen as a knowledge system which describes domain-specific concepts and concept relationships. Because of the different considerations for various subject areas and specific projects, the process of constructing domain ontology is different. Generally, one of the most influential is T.R. Gruber proposed five criterias in 1995: clarity and objective, coherence, extendibility, minimal encoding bias and minimal ontological commitment. In order to improve the efficiency of ontology construction, and ensure the quality of domain ontology to some extent, this paper proposes a method of semi-automatic ontology construction. Firstly, construct a core ontology by ontology tool, then judge the semantic relationships between domain terms and ontology concepts by matching method, and add the matching concepts to the core ontology to finish the ontology expansion automatically. Lastly, the domain ontology is represented in the Web Ontology Language(OWL). To operate ontology is equivalent to operate the OWL file.(2) The discovery of Deep Web entries: The quality and quantity of databases appertain directly to the quality of Deep Web data integration. In Deep Web, a significant amount of information can only be accessed through the query interface of a back-end database, and query interface is the door to get information from web databases, therefore, the discovery of Deep Web entries can be translated into the issue of distinguishing query interfaces. Traditional crawlers use a strategy of breadth-first to access web pages, in this way, the crawlers will download a large number of irrelevant pages, it is not only a high cost, but also low efficiency. Therefore, this paper constructs web page classifier, form structure classifier and form content classifier by importing focused crawling and ontology, which are used to access these pages that may link to Deep Web query interfaces. This method can avoid accessing useless links and implement automatic discovery of domain-specific Deep Web entries.a) Web page classifier: namely ontology-based focused crawling, which is used to discover topic pages. The knowledge base of domain ontology is a key component of web page classifier, essentially an object set of topic knowledge. During the process of focused crawling, web page classifier calculates the similarity between pages and domain topic to filter the useless pages.b) Form structure classifier: It is used to parse the related pages and judge whether these pages subsume query forms by Decision Tree algorithm, and then add these relevant pages to form database. It is directly related for form structure classifier to the existence of real databases.c) Form content classifier: It is used to recognize the query entries of domain-specific Deep Web in the semantic level, and store these URLs of domain-specific query interfaces to a database which will be called by other modules.(3) Schema extraction of Deep Web query interfaces: From the observation of HTML, we find that web designers usually merge a number of visual features to make the form structure more clearly and better, such as position features, layout features and appearance features, which are very important to associate attribute labels and query controls. Therefore, this paper proposes an algorithm of schema extraction based on visual features of pages automatically to obtain the interface schemas. The process of schema extraction for Deep Web query interfaces is as follows:a) Parse query interface region and obtain the form elements of query interface region. Firstly, analyze query interface region by calling HTML parser to self-define the process of parsing. During the process of parsing, we capture information for four categories, which are named start tag, end tag, simple tag and text tag. The information which does not belong to the four categories will be filtered automatically to obtain the form elements of query interface region.b) Attribute extraction: During parsing query interface region, we assign name, label, type and value for each category tag. However, when the designer codes the form, there is usually no information between visual tag