Mongolia Web Spider, Text Encoding Recognition And Conversion Research

With the rapid development of the Internet technology, people have been changing the way of obtaining information from traditional books gradually transferred to the network, resulting in the rapid growth of network information and websites. People search their real wanted information more difficult from the Internet.This make the Information Retrieval (IR) tools, which is also called as search engine, very important in human life. By using the search engine will enable people get information, products and services more quicker and easier than by traditional ways.With several years of mongolian information development, the mongolian website gradually keep with the rapid growth, and we can get more information about Mongolian, but we can get our information which is our wanted one become more difficult. Although at present the search engine play a big role in information retrieval. But mongolian search engine have not been developed. There are various problem, but the mainly problem is mongolian text encoding diverse and no relationship between those text encoding. If we get an unknown encoding web pages to obtain its correct semantics, we need to judge it and determine which is encoded, and then can we correctly interpreted its semantic. Therefore, crawled Mongolian web pages, coding and identification, and converted it to a unification intermediate coding become our research subject.